iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Let a llama session be shown a picture where the model reads one

Browse Source 
A multimodal model is loaded with the `mmproj` found beside its weights --
which is how a repository publishes the pair -- and an attached image rides
in the request as an `image_url` data URI, so it reaches a model on another
machine without the file going there. Nothing is done for a model without a
projector: no captioning, no OCR, no second model.

Whether a session takes pictures is measured rather than assumed:
`/props`'s `modalities.vision` from the server that loaded the model, in
three states, because a model still coming off disk has genuinely not said.
Unknown is offered rather than refused -- a control withheld because nobody
could ask goes missing from sessions that would have taken it. The answer
reaches the phone twice per model as `Event::Images`, so the photo button is
withdrawn the moment a model with vision is left rather than at whatever
later point the session row is fetched again.

A message carrying an image a model cannot read is stopped rather than
stripped: `llama-server` refuses the whole request over one image part, and
a message sent without its picture would be answered as though the picture
had never been mentioned. The phone will not attach one, and the driver
refuses it again at the three moments the answer can first exist -- at the
door, when a message queued behind a loading model is read, and at the tool
boundary a steer enters by. An earlier turn's image folds into a line of
words for a model without vision, so switching a conversation onto one does
not end it.

A projector is filtered out of the models a provider *offers*, since a
session started on one is a server that cannot load it; it stays in the
machine's own model list, where a file on a disk is managed.

Verified against ggml-org/SmolVLM-256M-Instruct-GGUF, local and over ssh:
"In this picture there is a red circle." Switching that session to
Qwen3-0.6B reports `refused`, refuses the next picture with the reason, and
still answers an ordinary message.
This commit is contained in:
iris-ai committed 2026-09-20 16:19:53 -04:00
1 parent b7fd18b195
commit bd9596d782
17 files changed
+835 -97
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.claude/skills/ai-app-rigs/SKILL.md
+8
View File
@@ -166,6 +166,14 @@ Two models are downloaded under `~/.local/share/ai-app/models`:
head. It is the rig for anything about `loading` being a state of its own,
since 20s is long enough to send into.
- `ggml-org/SmolVLM-256M-Instruct-GGUF/SmolVLM-256M-Instruct-Q8_0.gguf`,
175 MB, plus the `mmproj-…` beside it, downloaded 2026-09-20 as the rig for
**vision**: it is the only model here that reads pictures, it loads in
seconds on the CPU, and it described a red circle correctly. The pair is
also what exercises the projector being found beside the weights, and the
projector being kept out of the models a provider offers. Qwen3-0.6B beside
it is the other half of that rig -- the model that answers `refused`.
**Do not test with a 2-bit quant**: the IQ2_XXS of the 0.6B produces fluent
nonsense, which reads exactly like a broken driver — `llama-cli` produces the
same from the file directly, which is how to tell the two apart in a hurry.
AGENTS.md
+17
View File
@@ -84,6 +84,23 @@ Module-by-module intent is in PLAN.md's "Backend layout".
decides whether the MTP draft head is a 50% speed-up or a 33% loss; and
`spec-type = draft-mtp` is conditional on the file actually having a head,
because asking for one that is not there makes `llama-server` **exit**.
**A llama session takes a picture only where the model natively reads one**
(2026-09-20): a multimodal model is loaded with the `mmproj` found beside
its weights (overridable per model, `off` included), an attachment rides in
the request as an `image_url` data URI, and nothing at all is done for a
model without a projector. Four things fall out of it and are easy to get
wrong again -- whether a session takes pictures is `/props`'s
`modalities.vision` from the loaded server and never a guess from this side,
with three states because a loading model has not answered yet
(`Images::Unknown` is *offered*, since a control withheld because nobody
could ask is missing from sessions that would have taken it); a message
carrying an image a model cannot read is **stopped rather than stripped**,
refused at the door, at the queue and at the steering boundary, because
`llama-server` refuses the whole request over one part and a message sent
without its picture is a different message; an earlier turn's image folds
into a line of words for a model without vision, so switching models does
not end the conversation; and a projector is filtered out of the models a
provider *offers*, while staying in the machine's own model list.
**A llama session's thinking is drawn** (2026-09-19): `reasoning_content`
becomes `Event::Thinking` deltas closed by an `Event::ThinkingDone` carrying
the span the *driver* measured, and the phone draws a card that spins while
PLAN.md
+35
View File
@@ -521,6 +521,41 @@ deliberate and easy to undo by accident:
of the file — on the machine that will serve it, in the round trip the spawn
was already making — and `params["speculative"] = "off"` is the way out.
- **A picture goes to a model that natively reads one, and nowhere else**
(2026-09-20). Vision here is a multimodal model loaded with its projector:
`mmproj` in that model's preset section, found beside the weights because
that is how a repository publishes the pair, and overridable per model —
another file name, or `off` — in the machines tab's provider view. Nothing
is done for a model without one: no captioning, no OCR, no second model.
An attached image rides in the request as an `image_url` content part with
a data URI, which is what reaches a model on another machine without
shipping the file there.
- **Whether a session takes pictures is measured, not assumed.**
`/props`'s `modalities.vision` is the loaded server's own answer, and it
is the only one worth having: the projector is loaded over there, and
this side cannot see whether it worked. Three states, because a model
still coming off disk has genuinely not said — `Images::Unknown` is
offered rather than refused, since withholding the control on a session
nobody could ask about hides it on models that read pictures perfectly
well. The answer reaches the phone twice per model, as `Event::Images`:
unknown the moment the old model is left, then the new server's answer.
- **A message carrying an image a model cannot read is stopped, not
stripped.** `llama-server` refuses the whole request over one image part,
and a message sent without its picture would be answered as though the
picture had never been mentioned. The phone will not attach one (the
photo item is disabled with the reason on it, and the share sheet and
file chooser refuse the same), and the driver refuses it again where it
arrives — at the door, when a message that queued behind a loading model
is read, and at the tool boundary a steer enters by, because those are
the three moments the answer can first exist.
- **An earlier turn's image becomes a line of words for a model without
vision**, so that switching a conversation onto one does not end it. It
is a statement that there was an attachment, not a description of it.
- **A projector is not a model.** It is filtered out of what a provider
offers a spawn or a model change, since a session started on one is a
server that cannot load it, and it stays in the machine's own model list,
which is where a file on a disk is managed.
### Models (2026-08-28, rebuilt per machine 2026-09-19)
- **A download belongs to the model, not to the request.** Keyed by
app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt
+10
View File
@@ -219,6 +219,15 @@ data class SessionSummary(
* server because that is where a provider's kind is known.
*/
val maxImageEdge: Int?,
/**
* Whether a picture can be sent here at all.
*
* A snapshot, like the rest of this row: the live answer arrives as [SessionEvent.Images],
* which is what a session screen watches. Asked of the server rather than worked out here
* because for a local model it is a property of what is loaded on the serving machine, which
* this app cannot see.
*/
val images: ImageSupport,
/**
* Which of `GET /usage`'s snapshots is about this session, and null where nothing meters it.
*
@@ -263,6 +272,7 @@ private fun parseSession(session: JSONObject) =
contextLimit = if (session.has("contextLimit")) session.getLong("contextLimit") else null,
params = session.optJSONObject("params").stringMap(),
maxImageEdge = session.optInt("maxImageEdge", 0).takeIf { it > 0 },
images = imageSupport(session.optString("images")),
usageProvider = session.optString("usageProvider").ifEmpty { null },
status = session.getString("status"),
lastActivity = session.getDouble("lastActivity"),
app/androidApp/src/main/kotlin/com/example/aiapp/Attachment.kt
+24
View File
@@ -11,6 +11,30 @@ import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
/**
* Whether a session can be sent a picture, as the server answers it.
*
* Three states rather than a switch, because for a local model the answer belongs to the server
* that loaded it: one still coming off disk genuinely has not said. [UNKNOWN] is offered -- a
* control withheld because nobody could ask is a photo button missing from a session that would
* have read the photo perfectly well, and the send path says so if the guess was wrong.
*/
enum class ImageSupport {
ACCEPTED,
REFUSED,
UNKNOWN,
}
/**
* What the server called it; anything else -- an older server, a newer word -- is not an answer.
*/
fun imageSupport(word: String): ImageSupport =
when (word) {
"accepted" -> ImageSupport.ACCEPTED
"refused" -> ImageSupport.REFUSED
else -> ImageSupport.UNKNOWN
}
/**
* Whether [ref] names an image the server stored as one -- `<hex>.<extension>`, with an extension
* from the list it writes -- rather than a file kept under its own name. Mirrors the server's
app/androidApp/src/main/kotlin/com/example/aiapp/Attachments.kt
+11
View File
@@ -38,6 +38,11 @@ suspend fun uploadPickedImage(
* Uploads whatever [uri] names, the way its kind needs. An image goes through [uploadPickedImage]
* and is shrunk; anything else goes whole, under the name the other app or the file chooser gave
* it, because the session is told that name rather than shown the bytes.
*
* A picture is refused here, before anything is read or sent, when [images] says this session's
* model cannot read one. Here rather than beside the photo button because this is where every way
* of attaching meets: the picker, the file chooser, and another app's share sheet -- and only the
* first of those has a button to disable.
*/
suspend fun uploadPicked(
context: Context,
@@ -45,10 +50,16 @@ suspend fun uploadPicked(
sessionId: String,
uri: Uri,
maxEdge: Int?,
images: ImageSupport,
): String {
val resolver = context.contentResolver
val mime = resolver.getType(uri)
if (mime != null && mime.startsWith("image/")) {
if (images == ImageSupport.REFUSED) {
throw ApiException(
"this session's model can't read pictures, so that one wasn't attached"
)
}
return uploadPickedImage(context, settings, sessionId, uri, maxEdge)
}
// Opened before the request starts, so a provider that refuses says so here and not from inside
app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt
+11
View File
@@ -168,6 +168,16 @@ sealed class SessionEvent {
*/
data class Settings(val model: String?, val permissionMode: String?) : SessionEvent()
/**
* Whether a picture can be sent to this session now, as the thing serving its model answered.
*
* Only a llama.cpp session says this, and it says it twice per model: unknown the moment the
* old one is left, then the loaded server's answer. It carries no row -- it is what the
* composer's photo button is drawn from, and a line in the transcript about a control is not
* something anybody asked after.
*/
data class Images(val images: ImageSupport) : SessionEvent()
/**
* What a turn cost, and how much the model was holding when it ended.
*
@@ -328,6 +338,7 @@ fun parseSeqEvent(json: String): SeqEvent {
model = body.optString("model").ifEmpty { null },
permissionMode = body.optString("permissionMode").ifEmpty { null },
)
"images" -> SessionEvent.Images(imageSupport(body.optString("images")))
"contextWindow" -> SessionEvent.ContextWindow(body.getLong("tokens"))
"usageDelta" ->
SessionEvent.UsageDelta(
app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt
+51 -3
View File
@@ -306,6 +306,16 @@ fun SessionScreen(
// reads as a rename that did not take.
var title by remember(summary.id) { mutableStateOf(summary.title) }
var model by remember { mutableStateOf(summary.model) }
// Whether this session takes pictures. Held here rather than read from the row that opened the
// screen for the reason [model] is, and it moves with the model: a llama.cpp session changed
// onto a model without vision stops offering the photo button at that moment, not at whatever
// later point the session list is fetched again.
var images by remember { mutableStateOf(summary.images) }
// Pictures attached to a message this session can no longer be sent -- the model was changed
// under them. The message is stopped here as well as at the server, which refuses it whichever
// device sent it: a picture stripped out on the way would be answered as though it had never
// been mentioned.
val strandedImages = images == ImageSupport.REFUSED && pendingAttachments.any(::isImageRef)
// The thinking level, held here for the same reason [title] is: the settings dialog can change
// it, and the row this screen was opened from is a snapshot taken before that. Read straight
// from the summary, a level set here was drawn as the old one again the next time the dialog
@@ -517,6 +527,7 @@ fun SessionScreen(
event.model?.let { model = it }
event.permissionMode?.let { permissionMode = it }
}
if (event is SessionEvent.Images) images = event.images
if (event is SessionEvent.Status) {
// The event's own timestamp, so a compaction that began before this screen
// opened is timed from when it actually began -- and a compaction worth asking
@@ -853,6 +864,7 @@ fun SessionScreen(
if (!isSubagent) {
model = summary.model
permissionMode = summary.permissionMode ?: "auto"
images = summary.images
}
if (status != "compacting") compactingSince = null
// Nothing to put back, so these rows are the screen and the probe can return under
@@ -1397,7 +1409,14 @@ fun SessionScreen(
// An image is shrunk to what this session's provider takes before it is
// uploaded, so a twelve-megapixel photo does not cross the tunnel to be
// rejected at the far end; a file goes whole.
uploadPicked(context, settings, summary.id, uri, summary.maxImageEdge)
uploadPicked(
context,
settings,
summary.id,
uri,
summary.maxImageEdge,
images,
)
}
pendingAttachments = pendingAttachments + id
actionError = null
@@ -2058,6 +2077,18 @@ fun SessionScreen(
refs = pendingAttachments,
onRemove = { pendingAttachments = pendingAttachments - it },
)
// Only reachable by the model changing under something already attached, since
// attaching is refused where the answer is already known. Said beside the
// pictures it is about and above the button it disables, which is where
// somebody can act on it: removing them is what makes this sendable.
if (strandedImages) {
Text(
"This model can't read pictures. Remove them to send this.",
color = MaterialTheme.colorScheme.error,
style = MaterialTheme.typography.bodySmall,
modifier = Modifier.padding(bottom = 4.dp),
)
}
OutlinedTextField(
value = input,
onValueChange = {
@@ -2091,8 +2122,23 @@ fun SessionScreen(
properties = PopupProperties(clippingEnabled = false),
shape = BubbleMenuShape,
) {
// Disabled rather than dropped: an item that comes and goes
// makes its own absence the message, and the reason belongs on
// the item, where somebody wondering why is looking.
val takesPictures = images != ImageSupport.REFUSED
DropdownMenuItem(
text = { Text("Photo") },
text = {
Column {
Text("Photo")
if (!takesPictures) {
Text(
"this model can't read them",
style = MaterialTheme.typography.labelSmall,
)
}
}
},
enabled = takesPictures,
onClick = {
attaching = false
pickImage.launch(
@@ -2200,7 +2246,9 @@ fun SessionScreen(
// reason above.
CircleButton(
onClick = { send() },
enabled = input.text.isNotBlank() || pendingAttachments.isNotEmpty(),
enabled =
(input.text.isNotBlank() || pendingAttachments.isNotEmpty()) &&
!strandedImages,
fill = if (running) queueColor else sendColor,
) {
Glyph(
app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt
+3
View File
@@ -600,6 +600,9 @@ fun foldEvent(items: List<TranscriptItem>, entry: SeqEvent): List<TranscriptItem
// nothing it belongs above.
is SessionEvent.MessageDropped -> items
is SessionEvent.Settings -> items
// Neither carries a row: both are about what the session can do rather than about anything
// said in it, and the composer is where they are drawn.
is SessionEvent.Images -> items
is SessionEvent.BackgroundTasks -> items
is SessionEvent.Status -> settleReply(items, event.state)
is SessionEvent.AuthenticationRequired ->
server/src/config.rs
+29
View File
@@ -21,6 +21,8 @@ use serde::{Deserialize, Serialize};
use wg_app_link::format;
use crate::session::driver::Images;
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct Config {
@@ -230,6 +232,21 @@ impl DriverKind {
}
}
/// What a session of this kind makes of an image attached to a message.
///
/// The provider's own answer, which for every CLI here is that it takes
/// them. llama.cpp is [`Images::Unknown`] because the question is not the
/// provider's to answer: a local model reads images only if its file has a
/// projector loaded beside it, and the server that loaded it is the only
/// thing that knows -- `LlamaDriver::images` gives the live answer once
/// there is one, and this is what a session with no process yet reports.
pub fn images(self) -> Images {
match self {
DriverKind::Echo | DriverKind::ClaudeCli | DriverKind::CodexCli => Images::Accepted,
DriverKind::LlamaCpp => Images::Unknown,
}
}
/// Which paid service meters a session of this kind, and `None` for one
/// that costs nothing.
///
@@ -542,6 +559,18 @@ pub const LLAMA_MODEL_PARAMS: &[ParamSpec] = &[
},
restart: true,
},
ParamSpec {
key: "mmproj",
label: "Vision projector",
// Found rather than asked for, because a repository publishing a
// vision model publishes the projector beside it -- so this is for the
// two cases finding it cannot cover: a repository that published more
// than one, and a model whose projector somebody would rather not
// spend the memory on.
unset: "the mmproj file beside the model -- a file name, or \"off\" for none",
kind: ParamKind::Text,
restart: true,
},
ParamSpec {
key: "specDraftNMax",
label: "Tokens drafted ahead",
server/src/machines.rs
+9 -1
View File
@@ -153,7 +153,15 @@ pub async fn provider_models(
) -> Result<Vec<OfferedModel>> {
if provider.kind == DriverKind::LlamaCpp {
let dir = crate::models::dir_on(transport, models_dir);
let found = crate::models::on_machine(transport, &dir).await?;
let mut found = crate::models::on_machine(transport, &dir).await?;
// A vision model's projector is a file beside it rather than a model,
// and the session that reads pictures is the one on the model: offered
// here it is a chip that starts a server which cannot load it. It is
// still in the machine's own model list, which is where a file on a
// disk is managed and deleted.
found.retain(|model| {
!crate::session::llama::is_projector(model.key.rsplit('/').next().unwrap_or_default())
});
let labels = crate::models::labels(&found);
return Ok(found
.into_iter()
server/src/session/claude.rs
+1 -8
View File
@@ -1150,14 +1150,7 @@ fn missing_conversation(detail: &str) -> bool {
/// one this server would have written, so a crafted id cannot name a file
/// outside the session.
fn attachment_path(session_dir: &Path, id: &str) -> Result<PathBuf> {
if !id
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-' || c == '_')
|| id.contains("..")
{
anyhow::bail!("invalid attachment id");
}
let path = session_dir.join("attachments").join(id);
let path = super::driver::attachment_path(session_dir, id)?;
// A file copied to the session's own machine is named where it landed there
// -- `routes::upload_attachment` writes that down beside it -- because the
// path has to be one the CLI can open, not one this server can.
server/src/session/codex.rs
+1 -12
View File
@@ -540,7 +540,7 @@ fn input_for(inner: &Inner, message: &Waiting) -> Result<Vec<Value>> {
let mut files = Vec::new();
let mut input = Vec::new();
for attachment in &message.attachments {
let path = attachment_path(&inner.session_dir, attachment)?;
let path = super::driver::attachment_path(&inner.session_dir, attachment)?;
if let Some(media_type) = crate::media::media_type_for(attachment) {
if matches!(inner.transport, Transport::Here) {
input.push(json!({"type": "localImage", "path": path}));
@@ -1319,17 +1319,6 @@ fn protocol_error(inner: &Inner, message: String) {
});
}
fn attachment_path(session_dir: &Path, id: &str) -> Result<PathBuf> {
if id.contains("..")
|| !id
.chars()
.all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_'))
{
anyhow::bail!("invalid attachment id");
}
Ok(session_dir.join("attachments").join(id))
}
pub(super) fn read_thread(session_dir: &Path) -> Option<String> {
serde_json::from_str::<Value>(&std::fs::read_to_string(session_dir.join(THREAD_FILE)).ok()?)
.ok()?
server/src/session/driver.rs
+69
View File
@@ -89,6 +89,52 @@ pub(in crate::session) fn message_body(
/// `crate::media::media_type_for`.
pub type AttachmentRef = String;
/// Whether a session can be sent an image.
///
/// Asked before one is attached rather than when it is sent, because that is
/// the only moment at which the answer is worth anything: a photo refused
/// after it has been picked, shrunk and uploaded over the tunnel is a refusal
/// that cost everything sending it would have.
///
/// Three states and not a boolean, for the reason
/// [`crate::session::llama`] has to have them: a local model's answer is the
/// loaded server's to give, so a session whose model is still coming off disk
/// -- or has never been started -- genuinely does not know yet. "We could not
/// find out" drawn as "no" is a control that is never offered on a model that
/// reads images perfectly well.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub enum Images {
/// The model reads them, so attaching one is offered.
Accepted,
/// It does not, and one sent anyway is refused where it arrives.
Refused,
/// Nobody has been able to ask. Offered, because the alternative is
/// withholding the control on every session that has not started yet.
Unknown,
}
/// Where an uploaded attachment sits in a session's directory.
///
/// Refused rather than resolved when the id is not one this server would have
/// written: the id reaches here from a phone and from transcript lines, and a
/// `..` in one would name a file outside the session. Every driver that opens
/// an attachment goes through this, so there is one answer to what an id may
/// hold.
pub(in crate::session) fn attachment_path(
session_dir: &std::path::Path,
id: &str,
) -> anyhow::Result<std::path::PathBuf> {
if id.contains("..")
|| !id
.chars()
.all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_'))
{
anyhow::bail!("invalid attachment id");
}
Ok(session_dir.join("attachments").join(id))
}
/// One choice offered in answer to a [`Event::Question`]. More than a label
/// because the reader is deciding rather than confirming: what an option
/// means, and what picking it would produce, are what decide it.
@@ -391,6 +437,18 @@ pub enum Event {
#[serde(default, skip_serializing_if = "Option::is_none")]
permission_mode: Option<String>,
},
/// Whether a picture can be sent here, as the thing serving the model
/// answered -- see [`Images`].
///
/// Its own event rather than a field on [`Event::Settings`] because it is
/// not something anybody set: it is measured, twice per model, and by a
/// driver rather than asked for by a phone. Emitted by llama.cpp alone,
/// where a model change can take the answer away -- so the phone withdraws
/// the control the moment the model that could read pictures is left,
/// rather than at whatever later point the session row is fetched again.
Images {
images: Images,
},
/// How much context this session's model has to hold a conversation in.
///
/// The denominator the phone draws [`Event::UsageDelta`]'s `context`
@@ -783,6 +841,17 @@ pub trait Driver: Send + Sync {
/// the transcript, so a driver that never sends it drops the message from
/// the conversation entirely.
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>);
/// What this session makes of an image *now*, where that is not simply a
/// property of the provider -- `None` leaves the answer to
/// [`crate::config::DriverKind::images`], which is the whole answer for a
/// CLI that takes them whatever it is talking to.
///
/// Overridden by llama.cpp alone, and it has to be: which model a session
/// is on is changeable while it runs, and whether that model reads images
/// is the loaded server's answer rather than the provider's.
fn images(&self) -> Option<Images> {
None
}
/// Takes back a message that is still waiting, named by the id its
/// [`Event::MessageQueued`] carried.
///
server/src/session/llama/mod.rs
+467 -72
View File
@@ -70,7 +70,7 @@ use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use super::driver::{
AttachmentRef, Driver, Event, EventSink, QuestionOption, SessionStatus, Unqueued,
AttachmentRef, Driver, Event, EventSink, Images, QuestionOption, SessionStatus, Unqueued,
};
use super::process;
use super::transport::{Launch, Transport};
@@ -173,7 +173,7 @@ const MAX_STEPS: usize = 32;
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Message {
role: String,
content: String,
content: Content,
/// What the assistant asked to run, in the wire's own shape so it goes
/// back exactly as it came.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
@@ -183,16 +183,54 @@ struct Message {
tool_call_id: Option<String>,
}
/// What one message's `content` holds.
///
/// Words alone for all but one case, and that case is why this is not a
/// `String`: an image reaches a model as a *part* of a user message, beside
/// the words, and `llama-server` reads the OpenAI shape for that. Untagged,
/// so a message without images serializes exactly as it did before images
/// existed -- which is what keeps every turn in a long conversation
/// byte-identical to the one the server already has cached.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(untagged)]
enum Content {
Text(String),
Parts(Vec<Value>),
}
impl Message {
fn new(role: &str, content: impl Into<String>) -> Self {
Self {
role: role.to_string(),
content: content.into(),
content: Content::Text(content.into()),
tool_calls: Vec::new(),
tool_call_id: None,
}
}
/// What somebody sent: their words, and whatever images went with them.
///
/// The images come first, as they do in the Claude driver, because a
/// question asked before the picture it is about reads as a question about
/// nothing. A message with no readable image is an ordinary one rather
/// than a one-part list -- the two render differently through some chat
/// templates, and only the first has ever been tested by everything else
/// here.
fn from_user(text: impl Into<String>, images: Vec<Value>) -> Self {
let text = text.into();
if images.is_empty() {
return Self::new("user", text);
}
let mut parts = images;
if !text.is_empty() {
parts.push(json!({"type": "text", "text": text}));
}
Self {
content: Content::Parts(parts),
..Self::new("user", String::new())
}
}
fn result_of(call: &str, content: impl Into<String>) -> Self {
Self {
tool_call_id: Some(call.to_string()),
@@ -304,9 +342,13 @@ impl Serves {
#[derive(Default)]
struct Turns {
running: bool,
waiting: std::collections::VecDeque<(String, String)>,
waiting: std::collections::VecDeque<Waiting>,
}
/// A message accepted from the phone that nothing has read yet: the id its
/// [`Event::MessageQueued`] announced, the words, and what was attached.
type Waiting = (String, String, Vec<AttachmentRef>);
/// What it takes to put this session on a different model.
///
/// Kept whole rather than reduced to the two fields that change, because
@@ -391,6 +433,10 @@ struct Shared {
/// so about a model nobody has asked yet is the one mistake here that
/// looks exactly like an answer.
thinking_options: Mutex<Option<Vec<String>>>,
/// Whether the loaded model reads images ([`vision_of`]), `None` until it
/// has been asked -- the same three states, and for the same reason, as
/// `thinking_options` above.
vision: Mutex<Option<bool>>,
}
pub struct LlamaDriver {
@@ -458,6 +504,7 @@ impl LlamaDriver {
watching: AtomicBool::new(false),
thinking: Mutex::new(chosen_thinking(&meta.params)),
thinking_options: Mutex::new(None),
vision: Mutex::new(None),
}),
respawn: Respawn {
meta: meta.clone(),
@@ -500,9 +547,16 @@ impl LlamaDriver {
// as loading until the model is in memory, rather than looking ready
// and refusing the first message.
*shared.serving.lock().unwrap() = Serving::Loading;
// The answer belonged to whatever was loaded before this; the model
// These answers belonged to whatever was loaded before this; the model
// starting now gets asked for itself.
*shared.thinking_options.lock().unwrap() = None;
*shared.vision.lock().unwrap() = None;
// Said as it is taken away, not only when the next answer arrives: a
// session leaving a model with vision must stop offering the control
// now, while the picture that would be refused is still unattached.
let _ = shared.sink.send(Event::Images {
images: Images::Unknown,
});
let _ = shared.sink.send(Event::Status {
state: SessionStatus::Loading,
});
@@ -561,6 +615,13 @@ impl LlamaDriver {
// otherwise surface as the model simply not doing it.
*shared.thinking_options.lock().unwrap() = Some(thinking_options(&serves));
shared.note_unusable_thinking();
// Asked for the same reason again: whether a model reads
// images is whether its projector loaded, which only the
// server that loaded it can see.
*shared.vision.lock().unwrap() = vision_of(&serves);
shared.emit(Event::Images {
images: shared.images(),
});
Serving::Ready {
serves,
tools: Arc::new(tools),
@@ -627,6 +688,51 @@ impl Shared {
});
}
/// What this session makes of a picture now, which is the loaded model's
/// answer and changes with it.
fn images(&self) -> Images {
match *self
.vision
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
{
Some(true) => Images::Accepted,
Some(false) => Images::Refused,
None => Images::Unknown,
}
}
/// Whether an image sent now would be read. "We could not ask" is not a
/// refusal: the message goes, and the server says so itself -- loudly, as
/// a refused request -- if that was wrong.
fn reads_images(&self) -> bool {
self.images() != Images::Refused
}
/// Says so, and answers true, when these attachments cannot go to the
/// loaded model.
///
/// The whole message is stopped rather than stripped of its images:
/// `llama-server` refuses the entire request over one image part, so a
/// message sent without them is a different message from the one somebody
/// wrote, answered as though the picture had never been mentioned.
///
/// Asked in three places because there are three moments at which the
/// answer can first exist: when a message arrives, when one that queued
/// behind a loading model is read, and when one typed during a turn is
/// steered into it.
fn refuse_images(&self, attachments: &[AttachmentRef]) -> bool {
if attachments.is_empty() || self.reads_images() {
return false;
}
self.emit(Event::Error {
message: "this model can't read images, so that message wasn't sent. Put the \
session on a model with vision, or send it without the picture."
.to_string(),
});
true
}
/// Whether the model is still on its way, which is what a message sent
/// now has to wait behind.
fn loading(&self) -> bool {
@@ -911,8 +1017,8 @@ impl LlamaDriver {
}
}
};
if let Some((id, text)) = next {
Self::run_turn(shared, Some(id), text);
if let Some((id, text, images)) = next {
Self::run_turn(shared, Some(id), text, images);
}
}
@@ -922,7 +1028,12 @@ impl LlamaDriver {
/// model takes to generate, and a tool call inside it blocks for as long
/// as the tool takes, which for a shell command is unbounded by anything
/// this server knows.
fn run_turn(shared: &Arc<Shared>, queued: Option<String>, text: String) {
fn run_turn(
shared: &Arc<Shared>,
queued: Option<String>,
text: String,
images: Vec<AttachmentRef>,
) {
let shared = Arc::clone(shared);
std::thread::spawn(move || {
shared.cancel.store(false, Ordering::SeqCst);
@@ -932,9 +1043,14 @@ impl LlamaDriver {
// this message -- and it is then sent twice, once folded out of
// the transcript and once appended to it. Certain rather than
// racy across a load, which is where it was found.
let ready = shared
.await_ready()
.map(|(serves, tools)| (serves, tools, conversation(&shared.transcript)));
let ready = shared.await_ready().map(|(serves, tools)| {
// Which of the two an earlier message's attachments fold back
// as is this model's answer, not the one they were sent to:
// the session may have been moved onto a model without vision
// since.
let seen = shared.reads_images().then(|| shared.session_dir.as_path());
(serves, tools, conversation(&shared.transcript, seen))
});
// The message goes into the transcript here, at the moment it is
// read -- see `MessageTaken`. `queued` names the bubble this
// resolves, and is `None` for one that never waited. Recorded
@@ -943,21 +1059,27 @@ impl LlamaDriver {
shared.emit(Event::MessageTaken {
id: queued,
text: text.clone(),
// Never any: this driver refuses them where they arrive, so
// nothing is carried this far.
attachments: Vec::new(),
attachments: images.clone(),
});
match ready {
Ok((serves, tools, mut messages)) => {
shared.emit(Event::Status {
state: SessionStatus::Running,
});
messages.push(Message::new("user", text));
if let Err(err) = converse(&shared, &serves, &tools, messages) {
shared.emit(Event::Error {
message: format!("{err:#}"),
// Refused here as well as at the door, because a message
// that queued behind a loading model was accepted before
// anything knew what that model could read.
if !shared.refuse_images(&images) {
shared.emit(Event::Status {
state: SessionStatus::Running,
});
messages.push(Message::from_user(
text,
image_parts(&shared.session_dir, &images),
));
if let Err(err) = converse(&shared, &serves, &tools, messages) {
shared.emit(Event::Error {
message: format!("{err:#}"),
});
}
}
shared.emit(Event::Status {
state: SessionStatus::Idle,
@@ -1054,18 +1176,23 @@ fn take_steers(shared: &Arc<Shared>, messages: &mut Vec<Message>) {
return;
}
let waiting = std::mem::take(&mut shared.turns.lock().unwrap().waiting);
for (id, text) in waiting {
for (id, text, images) in waiting {
// The message enters the transcript here, below the calls that had
// not read it and above the ones that will -- the same rule the
// Claude driver announces a steer by.
shared.emit(Event::MessageTaken {
id: Some(id),
text: text.clone(),
attachments: Vec::new(),
attachments: images.clone(),
});
messages.push(Message::new(
"user",
// The same refusal the other two paths make, for the third moment at
// which it can first be known -- see `Shared::refuse_images`.
if shared.refuse_images(&images) {
continue;
}
messages.push(Message::from_user(
super::driver::message_body(&text, &[], true),
image_parts(&shared.session_dir, &images),
));
}
}
@@ -1171,11 +1298,34 @@ fn permitted(shared: &Arc<Shared>, call: &Call) -> bool {
impl Driver for LlamaDriver {
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>) {
if !attachments.is_empty() {
// An image rides in the request itself, which is how it reaches a
// model on another machine. Any other file has nowhere to go: this
// driver hands the model no path it could open, and the upload is on
// this side of the tunnel rather than on the serving machine.
let (images, files): (Vec<_>, Vec<_>) = attachments
.into_iter()
.partition(|id| crate::media::media_type_for(id).is_some());
if !files.is_empty() {
self.shared.emit(Event::Error {
message: "this model can't be sent attachments or files".to_string(),
message: format!(
"a llama.cpp session can be shown a picture but not handed a file, so {} \
wasn't sent.",
// The name it was attached under, which is what the reader
// recognises; the hex before it is this server's.
super::names(
files
.iter()
.map(|id| id.split_once('-').map_or(&id[..], |(_, name)| name))
),
),
});
}
// Refused while somebody is still looking at the screen, where the
// model has loaded and cannot read them. While it is loading nothing
// knows yet, and the message queues below with its images.
if self.shared.refuse_images(&images) {
return;
}
// A message written during a turn does not start a second
// conversation against the same server: it waits for the turn's next
// tool boundary and steers it from there (`take_steers`), or for the
@@ -1193,22 +1343,24 @@ impl Driver for LlamaDriver {
// while the model loaded is the one that opens the first turn.
if turns.running || !turns.waiting.is_empty() || self.shared.loading() {
let id = super::random_hex();
turns.waiting.push_back((id.clone(), text.clone()));
turns
.waiting
.push_back((id.clone(), text.clone(), images.clone()));
drop(turns);
self.shared.emit(Event::MessageQueued {
id,
text,
// Refused above, so there are none to wait with it. Said
// as an empty list rather than the caller's, which would
// draw a thumbnail on a bubble whose message will arrive
// without it.
attachments: Vec::new(),
attachments: images,
});
return;
}
turns.running = true;
}
Self::run_turn(&self.shared, None, text);
Self::run_turn(&self.shared, None, text, images);
}
fn images(&self) -> Option<Images> {
Some(self.shared.images())
}
fn unqueue(&self, id: &str) -> Unqueued {
@@ -1377,6 +1529,34 @@ impl Driver for LlamaDriver {
}
}
/// The uploaded images among `attachments`, as the content parts a request
/// carries them in.
///
/// Inline data URIs rather than paths, because the model is on the serving
/// machine as often as not and the upload is on this one -- the same reason
/// the Claude and Codex drivers base64 an image into the message instead of
/// naming it.
///
/// Anything that is not an image, or that cannot be read, is left out: the
/// send path has already said so about both, and a turn is a worse place to
/// learn it than the moment the thing was attached.
fn image_parts(session_dir: &Path, attachments: &[AttachmentRef]) -> Vec<Value> {
use base64::Engine as _;
attachments
.iter()
.filter_map(|id| {
let media_type = crate::media::media_type_for(id)?;
let path = super::driver::attachment_path(session_dir, id).ok()?;
let bytes = std::fs::read(path).ok()?;
let data = base64::engine::general_purpose::STANDARD.encode(bytes);
Some(json!({
"type": "image_url",
"image_url": {"url": format!("data:{media_type};base64,{data}")},
}))
})
.collect()
}
/// The conversation so far, folded out of the transcript.
///
/// Consecutive `AssistantText` deltas are one assistant turn, closed by the
@@ -1392,12 +1572,17 @@ impl Driver for LlamaDriver {
/// request or renders a conversation where the model asked for something and
/// nothing came back, and the second is worse than the first.
///
/// `images` is the session directory, for a model that reads them, and `None`
/// for one that does not -- see [`sent`].
///
/// This must stay a pure function of the transcript and must never re-render
/// earlier turns. llama.cpp caches the prompt prefix, so a growing conversation
/// reprocesses almost nothing -- but only while every turn is byte-identical to
/// last time. Changing how an old turn is rendered silently reprocesses the
/// whole history on every message.
fn conversation(path: &Path) -> Vec<Message> {
/// whole history on every message. (Which of the two an attachment renders as
/// is settled by the model, so it changes only when the model does -- and that
/// throws the prefix away regardless.)
fn conversation(path: &Path, images: Option<&Path>) -> Vec<Message> {
let Ok(events) = crate::session::transcript::read_after(path, 0) else {
return Vec::new();
};
@@ -1412,9 +1597,11 @@ fn conversation(path: &Path) -> Vec<Message> {
let mut fold = Fold::default();
for event in events.iter().cloned() {
match event.event {
Event::UserMessage { text, .. } => {
Event::UserMessage {
text, attachments, ..
} => {
fold.close();
fold.messages.push(Message::new("user", text));
fold.messages.push(sent(images, &text, &attachments));
}
Event::AssistantText { delta } => {
// Text after a call belongs to the reply that follows it, not
@@ -1445,6 +1632,35 @@ fn conversation(path: &Path) -> Vec<Message> {
fold.messages
}
/// One message somebody sent, as the model is to read it.
///
/// `images` is [`conversation`]'s: the session directory for a model with
/// vision, `None` for one without. Without it an attachment becomes a line
/// saying there was one, because a question about a picture the model was
/// never shown is worse read as a question about nothing.
fn sent(images: Option<&Path>, text: &str, attachments: &[AttachmentRef]) -> Message {
let parts = images.map_or_else(Vec::new, |dir| image_parts(dir, attachments));
if !parts.is_empty() {
return Message::from_user(text, parts);
}
let count = attachments.len();
if count == 0 {
return Message::new("user", text);
}
let note = format!(
"[{count} attachment{} sent with this message, which you cannot read.]",
if count == 1 { "" } else { "s" }
);
Message::new(
"user",
if text.is_empty() {
note
} else {
format!("{text}\n\n{note}")
},
)
}
/// The running state of [`conversation`]: the assistant turn being assembled,
/// and what is known about the calls in it.
#[derive(Default)]
@@ -1482,6 +1698,24 @@ impl Fold {
}
}
/// The projector lying in `dir`, absolute, or `None` for a directory with
/// none.
///
/// The first by name where there are several, which is a choice rather than a
/// guess: a repository publishing two publishes them at different precisions
/// (`mmproj-F16`, `mmproj-F32`), and the smaller sorts first. The model
/// setting is how to ask for the other one.
fn projector_in(dir: &Path) -> Option<String> {
let first = std::fs::read_dir(dir)
.ok()?
.filter_map(|entry| {
let name = entry.ok()?.file_name().into_string().ok()?;
is_projector(&name).then_some(name)
})
.min()?;
Some(dir.join(first).to_string_lossy().into_owned())
}
/// Where a model key resolves to on disk, refusing anything that climbs
/// out of the models directory -- the key arrives from a phone.
fn model_path(models_dir: &Path, key: &str) -> Result<PathBuf> {
@@ -1509,6 +1743,32 @@ pub struct Model {
/// in the "yes" direction is a server that exits rather than one that runs
/// slightly differently.
mtp: bool,
/// The multimodal projector lying beside it, absolute on that machine, and
/// `None` where there is none -- which is most models.
///
/// Found rather than configured because of how these arrive: a repository
/// that publishes a vision model publishes its projector in the same
/// directory, so downloading both is all somebody should have to do to be
/// able to send the model a picture. The model setting of the same name
/// overrides this, including with "off" -- see
/// [`crate::config::LLAMA_MODEL_PARAMS`].
mmproj: Option<String>,
}
/// Whether a file lying beside a model is a multimodal projector for it.
///
/// By name, which is the only thing both sides of this can see cheaply: the
/// two conventions in the wild are `mmproj-<something>.gguf` and
/// `<model>-mmproj-<something>.gguf`, so the test is the word anywhere in a
/// `.gguf` name. A file that is not really one is a server that fails to
/// start and says so, rather than a model that quietly answers wrongly.
///
/// Said twice, because the same question is asked of two filesystems: the
/// other half is the `grep` in [`model_on`]'s remote script, and the two have
/// to keep meaning the same thing.
pub fn is_projector(name: &str) -> bool {
let name = name.to_ascii_lowercase();
name.ends_with(".gguf") && name.contains("mmproj")
}
/// The model file **on the machine that will serve it**, confirmed to be
@@ -1531,9 +1791,11 @@ fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result<Model
let mtp = std::fs::File::open(&path)
.map(|mut file| crate::gguf::has_mtp_head(&mut file))
.unwrap_or(false);
let mmproj = path.parent().and_then(projector_in);
return Ok(Model {
path: path.to_string_lossy().into_owned(),
mtp,
mmproj,
});
};
// The same directory the spawn screen listed for this machine, and one
@@ -1557,7 +1819,9 @@ fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result<Model
let script = format!(
"p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${{p#\"~/\"}};; esac; \
[ -f \"$p\" ] || {{ printf 'missing\\n'; exit 0; }}; \
printf 'at\\t%s\\t%s\\n' \"$(head -c {prefix} \"$p\" | base64 | tr -d '\\n')\" \"$p\"",
m=$(ls \"${{p%/*}}\" 2>/dev/null | grep -i 'mmproj.*\\.gguf$' | head -n 1); \
printf 'at\\t%s\\t%s\\t%s\\n' \"$(head -c {prefix} \"$p\" | base64 | tr -d '\\n')\" \
\"$m\" \"$p\"",
prefix = crate::gguf::PREFIX_BYTES,
);
let launch = Launch::new(
@@ -1568,16 +1832,23 @@ fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result<Model
let answer = transport
.capture_blocking(&launch)
.with_context(|| format!("couldn't ask {name} where its models are"))?;
// The path last, so a `\t` in it survives; the middle field is base64,
// whose alphabet has no tab.
let found = answer
.trim()
.strip_prefix("at\t")
.and_then(|rest| rest.split_once('\t'));
// The path last, so a `\t` in it survives; the fields before it are
// base64, whose alphabet has no tab, and a file name from the same
// directory the path names.
let found = answer.trim().strip_prefix("at\t").and_then(|rest| {
let mut fields = rest.splitn(3, '\t');
Some((fields.next()?, fields.next()?, fields.next()?))
});
match found {
Some((head, resolved)) => Ok(Model {
path: resolved.to_string(),
Some((head, projector, resolved)) => Ok(Model {
mtp: mtp_in_prefix(head),
// Joined here rather than over there, so that only one side has an
// opinion about what "beside the model" means.
mmproj: (!projector.is_empty()).then(|| {
let dir = resolved.rsplit_once('/').map_or("", |(dir, _)| dir);
format!("{dir}/{projector}")
}),
path: resolved.to_string(),
}),
None => bail!(
"{name} has no model at {path}. A llama.cpp session serves the file from the \
@@ -1613,6 +1884,25 @@ fn context_window(serves: &Serves) -> Option<u64> {
.and_then(Value::as_u64)
}
/// Whether the loaded model reads images, from the server that loaded it.
///
/// `/props` reports the modalities of what is in memory -- vision is there
/// when a projector was loaded beside the weights, which is the preset's
/// `mmproj` and not anything the model file says on its own. `None` is a
/// server that would not answer or is too old to say, which is "we did not
/// find out" rather than "no": the phone withholds a control on the second
/// and not on the first.
fn vision_of(serves: &Serves) -> Option<bool> {
ureq::get(serves.query("/props"))
.call()
.ok()?
.body_mut()
.read_json::<Value>()
.ok()?
.pointer("/modalities/vision")
.and_then(Value::as_bool)
}
/// What the loaded model's chat template takes for thinking, asked of it.
///
/// Two different questions, because they are two different template arguments.
@@ -2031,6 +2321,21 @@ mod tests {
use super::*;
use crate::session::transcript::Transcript;
impl Content {
/// What a message says, for the assertions that are about the words.
/// A message carrying images keeps them beside its text, so this is
/// the text part among them.
fn words(&self) -> &str {
match self {
Content::Text(text) => text,
Content::Parts(parts) => parts
.iter()
.find_map(|part| part.get("text").and_then(Value::as_str))
.unwrap_or_default(),
}
}
}
/// Writes a transcript the way the pump does, so the fold is tested against
/// the real file format rather than a hand-built vector.
fn transcript_with(events: &[Event]) -> (tempfile::TempDir, PathBuf) {
@@ -2069,11 +2374,11 @@ mod tests {
delta: "yes".into(),
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
assert_eq!(
messages
.iter()
.map(|m| (m.role.as_str(), m.content.as_str()))
.map(|m| (m.role.as_str(), m.content.words()))
.collect::<Vec<_>>(),
[
("user", "hello"),
@@ -2121,9 +2426,9 @@ mod tests {
delta: "one two".into(),
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
assert_eq!(messages.len(), 2);
assert_eq!(messages[1].content, "one two");
assert_eq!(messages[1].content.words(), "one two");
}
#[test]
@@ -2159,11 +2464,11 @@ mod tests {
delta: "right".into(),
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
assert_eq!(
messages
.iter()
.map(|m| (m.role.as_str(), m.content.as_str()))
.map(|m| (m.role.as_str(), m.content.words()))
.collect::<Vec<_>>(),
[
("user", "read it"),
@@ -2194,9 +2499,9 @@ mod tests {
state: SessionStatus::Idle,
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
assert_eq!(messages.len(), 2);
assert_eq!(messages[1].content, "one two");
assert_eq!(messages[1].content.words(), "one two");
}
#[test]
@@ -2226,10 +2531,10 @@ mod tests {
prefill_ms: None,
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
assert_eq!(messages.len(), 2);
assert_eq!(messages[0].content, "hello");
assert_eq!(messages[1].content, "still here");
assert_eq!(messages[0].content.words(), "hello");
assert_eq!(messages[1].content.words(), "still here");
}
#[test]
@@ -2256,10 +2561,10 @@ mod tests {
delta: "cheaply".into(),
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
assert_eq!(messages.len(), 2);
assert_eq!(messages[0].content, "a fresh start");
assert_eq!(messages[1].content, "cheaply");
assert_eq!(messages[0].content.words(), "a fresh start");
assert_eq!(messages[1].content.words(), "cheaply");
}
#[test]
@@ -2285,9 +2590,9 @@ mod tests {
attachments: Vec::new(),
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
assert_eq!(messages.len(), 1);
assert_eq!(messages[0].content, "three");
assert_eq!(messages[0].content.words(), "three");
}
#[test]
@@ -2319,10 +2624,10 @@ mod tests {
delta: "It says alpha and beta.".into(),
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
let shape: Vec<(&str, &str)> = messages
.iter()
.map(|m| (m.role.as_str(), m.content.as_str()))
.map(|m| (m.role.as_str(), m.content.words()))
.collect();
assert_eq!(
shape,
@@ -2371,12 +2676,12 @@ mod tests {
output: "first".into(),
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
assert_eq!(messages[1].tool_calls.len(), 2);
assert_eq!(
messages[2..]
.iter()
.map(|m| (m.tool_call_id.as_deref(), m.content.as_str()))
.map(|m| (m.tool_call_id.as_deref(), m.content.words()))
.collect::<Vec<_>>(),
[(Some("a"), "first"), (Some("b"), "second")],
);
@@ -2404,10 +2709,10 @@ mod tests {
state: SessionStatus::Idle,
},
]);
let messages = conversation(&path);
let messages = conversation(&path, None);
assert_eq!(messages[1].tool_calls.len(), 1);
assert_eq!(messages[2].role, "tool");
assert_eq!(messages[2].content, tools::UNFINISHED);
assert_eq!(messages[2].content.words(), tools::UNFINISHED);
}
#[test]
@@ -2437,10 +2742,10 @@ mod tests {
delta: "it is linux".into(),
},
]);
let messages = conversation(&path);
assert_eq!(messages[1].content, "checking");
let messages = conversation(&path, None);
assert_eq!(messages[1].content.words(), "checking");
assert_eq!(messages[1].tool_calls.len(), 1);
assert_eq!(messages[3].content, "it is linux");
assert_eq!(messages[3].content.words(), "it is linux");
assert!(messages[3].tool_calls.is_empty());
}
@@ -2535,4 +2840,94 @@ mod tests {
);
}
}
/// An image sent to a model that reads them goes into the message as a
/// part beside the words, and one sent to a model that does not is said
/// in words -- never dropped, which would leave a question about a picture
/// reading as a question about nothing.
#[test]
fn a_picture_is_a_part_for_a_model_that_reads_them_and_a_line_for_one_that_does_not() {
let dir = tempfile::tempdir().expect("tempdir");
std::fs::create_dir(dir.path().join("attachments")).expect("attachments");
std::fs::write(dir.path().join("attachments").join("ab12.png"), b"PNG")
.expect("write image");
let refs = ["ab12.png".to_string()];
let seen = sent(Some(dir.path()), "what is this?", &refs);
assert_eq!(
seen.content,
Content::Parts(vec![
json!({
"type": "image_url",
"image_url": {"url": "data:image/png;base64,UE5H"},
}),
json!({"type": "text", "text": "what is this?"}),
]),
);
let unseen = sent(None, "what is this?", &refs);
assert_eq!(
unseen.content,
Content::Text(
"what is this?\n\n[1 attachment sent with this message, which you cannot \
read.]"
.to_string()
),
);
// Nothing attached is an ordinary message either way, and in
// particular not a one-part list: that renders differently through
// some chat templates, and every turn in this file is the other shape.
assert_eq!(
sent(Some(dir.path()), "plain", &[]).content,
Content::Text("plain".to_string()),
);
}
/// A file that cannot be read is left out rather than failing the turn:
/// the send path has already said so, and a turn is a worse place to learn
/// it.
#[test]
fn an_unreadable_attachment_is_not_a_picture() {
let dir = tempfile::tempdir().expect("tempdir");
assert!(image_parts(dir.path(), &["ab12.png".to_string()]).is_empty());
// Not an image at all, and an id this server would never have written.
assert!(image_parts(dir.path(), &["ab12-trace.txt".to_string()]).is_empty());
assert!(image_parts(dir.path(), &["../../etc/passwd.png".to_string()]).is_empty());
}
/// Both conventions repositories publish a projector under, and nothing
/// else -- a model file that merely sits beside one is not one.
#[test]
fn a_projector_is_known_by_its_name() {
assert!(is_projector("mmproj-SmolVLM-256M-Instruct-Q8_0.gguf"));
assert!(is_projector("Qwen2.5-VL-7B-mmproj-F16.GGUF"));
assert!(!is_projector("SmolVLM-256M-Instruct-Q8_0.gguf"));
assert!(!is_projector("mmproj-notes.txt"));
}
/// The projector found beside a model is the one it loads with, and the
/// smaller of two sorts first.
#[test]
fn the_projector_beside_a_model_is_found() {
let dir = tempfile::tempdir().expect("tempdir");
for name in [
"SmolVLM-256M-Instruct-Q8_0.gguf",
"mmproj-F32.gguf",
"mmproj-F16.gguf",
] {
std::fs::write(dir.path().join(name), b"").expect("write");
}
assert_eq!(
projector_in(dir.path()),
Some(
dir.path()
.join("mmproj-F16.gguf")
.to_string_lossy()
.into_owned()
),
);
let empty = tempfile::tempdir().expect("tempdir");
assert_eq!(projector_in(empty.path()), None);
}
}
server/src/session/llama/router.rs
+73
View File
@@ -707,9 +707,37 @@ fn section(found: &Model, settings: &BTreeMap<String, String>) -> String {
if found.mtp && settings.get("speculative").map(String::as_str) != Some("off") {
lines.push("spec-type = draft-mtp".to_string());
}
// What makes a model able to read pictures, and the one flag here that is
// found rather than defaulted: the projector is a second file published
// beside the weights, so a model that has one is loaded with it unless
// this model's settings name another or turn it off.
if let Some(projector) = projector(found, settings) {
lines.push(format!("mmproj = {projector}"));
}
lines.join("\n")
}
/// Which projector this model is loaded with: what its settings say, else
/// whatever was found beside it, and nothing for `"off"`.
///
/// A setting naming a bare file name means one in the model's own directory,
/// since that is where the alternatives to the file found there are; anything
/// with a `/` in it is taken as the path it is, absolute or not -- the serving
/// machine resolves it, and this side does not know its working directory.
fn projector(found: &Model, settings: &BTreeMap<String, String>) -> Option<String> {
let chosen = settings.get("mmproj").map(|value| value.trim());
match chosen {
Some("off") => None,
Some("") => found.mmproj.clone(),
Some(name) if name.contains('/') => Some(name.to_string()),
Some(name) => {
let dir = found.path.rsplit_once('/').map_or("", |(dir, _)| dir);
Some(format!("{dir}/{name}"))
}
None => found.mmproj.clone(),
}
}
/// The preset file with `name`'s section replaced by `body`, added at the end
/// if it was not there.
///
@@ -782,6 +810,7 @@ mod tests {
#[test]
fn a_section_names_the_file_and_the_flags_that_were_set() {
let found = Model {
mmproj: None,
path: "/models/a.gguf".to_string(),
mtp: true,
};
@@ -810,6 +839,50 @@ mod tests {
assert!(!section(&found, &settings(&[("speculative", "off")])).contains("spec-type"));
}
/// The projector found beside a model is loaded with it; the setting names
/// another where a repository published several, or turns it off.
#[test]
fn a_vision_model_is_loaded_with_its_projector() {
let found = Model {
path: "/models/repo/a.gguf".to_string(),
mtp: false,
mmproj: Some("/models/repo/mmproj-F16.gguf".to_string()),
};
let line = |settings: &[(&str, &str)]| {
section(&found, &self::settings(settings))
.lines()
.find_map(|line| line.strip_prefix("mmproj = "))
.map(str::to_string)
};
assert_eq!(line(&[]), Some("/models/repo/mmproj-F16.gguf".to_string()));
assert_eq!(
line(&[("mmproj", " ")]),
Some("/models/repo/mmproj-F16.gguf".to_string())
);
assert_eq!(line(&[("mmproj", "off")]), None);
// A bare name is one of the model's own neighbours; anything with a
// separator in it is the path it says it is.
assert_eq!(
line(&[("mmproj", "mmproj-F32.gguf")]),
Some("/models/repo/mmproj-F32.gguf".to_string()),
);
assert_eq!(
line(&[("mmproj", "/elsewhere/p.gguf")]),
Some("/elsewhere/p.gguf".to_string()),
);
// A model with none, and nothing asked for, loads without one.
assert!(
!section(
&Model {
mmproj: None,
..found.clone()
},
&settings(&[])
)
.contains("mmproj")
);
}
#[test]
fn a_section_replaces_its_own_and_leaves_every_other_line_alone() {
let first = upsert("", "repo/a.gguf", "model = /models/a.gguf");
server/src/session/mod.rs
+16 -1
View File
@@ -36,7 +36,7 @@ use crate::config::{
use claude::ClaudeDriver;
use codex::CodexDriver;
use driver::{
AttachmentRef, BackgroundTask, Driver, Event, EventSink, SessionCommand, SessionStatus,
AttachmentRef, BackgroundTask, Driver, Event, EventSink, Images, SessionCommand, SessionStatus,
Unqueued, context_after, context_limit_after,
};
use echo::EchoDriver;
@@ -235,6 +235,10 @@ pub struct SessionInfo {
/// that happens to be big" are different answers.
#[serde(skip_serializing_if = "Option::is_none")]
pub max_image_edge: Option<u32>,
/// Whether an image can be sent here at all -- see [`Images`]. Reported
/// so the phone can withhold the control rather than take a photo,
/// shrink it, upload it over the tunnel and then be told.
pub images: Images,
/// Which of `GET /usage`'s snapshots reports on this session, and
/// absent where nothing meters it -- see
/// [`DriverKind::usage_provider`].
@@ -610,6 +614,13 @@ impl LiveSession {
auto_resume_message: resume_message(current),
resume_at: current.resume.map(|scheduled| scheduled.at),
max_image_edge: kind.and_then(DriverKind::max_image_edge),
// The driver's answer where it has one -- llama.cpp's depends on
// the model it is on now, which is not the provider's to say.
images: self
.driver()
.and_then(|driver| driver.images())
.or_else(|| kind.map(DriverKind::images))
.unwrap_or(Images::Unknown),
usage_provider: kind.and_then(DriverKind::usage_provider),
imported,
keeps_own_transcript: kind.is_some_and(DriverKind::keeps_own_transcript),
@@ -1156,6 +1167,10 @@ impl SessionManager {
params: meta.params.clone(),
max_image_edge: kind_of(&inner.config, &meta.machine, &meta.provider)
.and_then(DriverKind::max_image_edge),
// No process, so nothing but the kind can answer; for
// llama.cpp that is deliberately "not known yet".
images: kind_of(&inner.config, &meta.machine, &meta.provider)
.map_or(Images::Unknown, DriverKind::images),
usage_provider: kind_of(&inner.config, &meta.machine, &meta.provider)
.and_then(DriverKind::usage_provider),
notify: meta.notify,