diff --git a/.claude/skills/ai-app-rigs/SKILL.md b/.claude/skills/ai-app-rigs/SKILL.md index f87641e..cd1b57d 100644 --- a/.claude/skills/ai-app-rigs/SKILL.md +++ b/.claude/skills/ai-app-rigs/SKILL.md @@ -166,6 +166,14 @@ Two models are downloaded under `~/.local/share/ai-app/models`: head. It is the rig for anything about `loading` being a state of its own, since 20s is long enough to send into. +- `ggml-org/SmolVLM-256M-Instruct-GGUF/SmolVLM-256M-Instruct-Q8_0.gguf`, + 175 MB, plus the `mmproj-…` beside it, downloaded 2026-09-20 as the rig for + **vision**: it is the only model here that reads pictures, it loads in + seconds on the CPU, and it described a red circle correctly. The pair is + also what exercises the projector being found beside the weights, and the + projector being kept out of the models a provider offers. Qwen3-0.6B beside + it is the other half of that rig -- the model that answers `refused`. + **Do not test with a 2-bit quant**: the IQ2_XXS of the 0.6B produces fluent nonsense, which reads exactly like a broken driver — `llama-cli` produces the same from the file directly, which is how to tell the two apart in a hurry. diff --git a/AGENTS.md b/AGENTS.md index 27e05be..1a78c93 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -84,6 +84,23 @@ Module-by-module intent is in PLAN.md's "Backend layout". decides whether the MTP draft head is a 50% speed-up or a 33% loss; and `spec-type = draft-mtp` is conditional on the file actually having a head, because asking for one that is not there makes `llama-server` **exit**. + **A llama session takes a picture only where the model natively reads one** + (2026-09-20): a multimodal model is loaded with the `mmproj` found beside + its weights (overridable per model, `off` included), an attachment rides in + the request as an `image_url` data URI, and nothing at all is done for a + model without a projector. Four things fall out of it and are easy to get + wrong again -- whether a session takes pictures is `/props`'s + `modalities.vision` from the loaded server and never a guess from this side, + with three states because a loading model has not answered yet + (`Images::Unknown` is *offered*, since a control withheld because nobody + could ask is missing from sessions that would have taken it); a message + carrying an image a model cannot read is **stopped rather than stripped**, + refused at the door, at the queue and at the steering boundary, because + `llama-server` refuses the whole request over one part and a message sent + without its picture is a different message; an earlier turn's image folds + into a line of words for a model without vision, so switching models does + not end the conversation; and a projector is filtered out of the models a + provider *offers*, while staying in the machine's own model list. **A llama session's thinking is drawn** (2026-09-19): `reasoning_content` becomes `Event::Thinking` deltas closed by an `Event::ThinkingDone` carrying the span the *driver* measured, and the phone draws a card that spins while diff --git a/PLAN.md b/PLAN.md index 32dd79b..398e372 100644 --- a/PLAN.md +++ b/PLAN.md @@ -521,6 +521,41 @@ deliberate and easy to undo by accident: of the file — on the machine that will serve it, in the round trip the spawn was already making — and `params["speculative"] = "off"` is the way out. +- **A picture goes to a model that natively reads one, and nowhere else** + (2026-09-20). Vision here is a multimodal model loaded with its projector: + `mmproj` in that model's preset section, found beside the weights because + that is how a repository publishes the pair, and overridable per model — + another file name, or `off` — in the machines tab's provider view. Nothing + is done for a model without one: no captioning, no OCR, no second model. + An attached image rides in the request as an `image_url` content part with + a data URI, which is what reaches a model on another machine without + shipping the file there. + - **Whether a session takes pictures is measured, not assumed.** + `/props`'s `modalities.vision` is the loaded server's own answer, and it + is the only one worth having: the projector is loaded over there, and + this side cannot see whether it worked. Three states, because a model + still coming off disk has genuinely not said — `Images::Unknown` is + offered rather than refused, since withholding the control on a session + nobody could ask about hides it on models that read pictures perfectly + well. The answer reaches the phone twice per model, as `Event::Images`: + unknown the moment the old model is left, then the new server's answer. + - **A message carrying an image a model cannot read is stopped, not + stripped.** `llama-server` refuses the whole request over one image part, + and a message sent without its picture would be answered as though the + picture had never been mentioned. The phone will not attach one (the + photo item is disabled with the reason on it, and the share sheet and + file chooser refuse the same), and the driver refuses it again where it + arrives — at the door, when a message that queued behind a loading model + is read, and at the tool boundary a steer enters by, because those are + the three moments the answer can first exist. + - **An earlier turn's image becomes a line of words for a model without + vision**, so that switching a conversation onto one does not end it. It + is a statement that there was an attachment, not a description of it. + - **A projector is not a model.** It is filtered out of what a provider + offers a spawn or a model change, since a session started on one is a + server that cannot load it, and it stays in the machine's own model list, + which is where a file on a disk is managed. + ### Models (2026-08-28, rebuilt per machine 2026-09-19) - **A download belongs to the model, not to the request.** Keyed by diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt index d6f4300..515959b 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt @@ -219,6 +219,15 @@ data class SessionSummary( * server because that is where a provider's kind is known. */ val maxImageEdge: Int?, + /** + * Whether a picture can be sent here at all. + * + * A snapshot, like the rest of this row: the live answer arrives as [SessionEvent.Images], + * which is what a session screen watches. Asked of the server rather than worked out here + * because for a local model it is a property of what is loaded on the serving machine, which + * this app cannot see. + */ + val images: ImageSupport, /** * Which of `GET /usage`'s snapshots is about this session, and null where nothing meters it. * @@ -263,6 +272,7 @@ private fun parseSession(session: JSONObject) = contextLimit = if (session.has("contextLimit")) session.getLong("contextLimit") else null, params = session.optJSONObject("params").stringMap(), maxImageEdge = session.optInt("maxImageEdge", 0).takeIf { it > 0 }, + images = imageSupport(session.optString("images")), usageProvider = session.optString("usageProvider").ifEmpty { null }, status = session.getString("status"), lastActivity = session.getDouble("lastActivity"), diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Attachment.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Attachment.kt index b53d82e..af78c85 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/Attachment.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Attachment.kt @@ -11,6 +11,30 @@ import androidx.compose.ui.text.font.FontFamily import androidx.compose.ui.text.style.TextOverflow import androidx.compose.ui.unit.dp +/** + * Whether a session can be sent a picture, as the server answers it. + * + * Three states rather than a switch, because for a local model the answer belongs to the server + * that loaded it: one still coming off disk genuinely has not said. [UNKNOWN] is offered -- a + * control withheld because nobody could ask is a photo button missing from a session that would + * have read the photo perfectly well, and the send path says so if the guess was wrong. + */ +enum class ImageSupport { + ACCEPTED, + REFUSED, + UNKNOWN, +} + +/** + * What the server called it; anything else -- an older server, a newer word -- is not an answer. + */ +fun imageSupport(word: String): ImageSupport = + when (word) { + "accepted" -> ImageSupport.ACCEPTED + "refused" -> ImageSupport.REFUSED + else -> ImageSupport.UNKNOWN + } + /** * Whether [ref] names an image the server stored as one -- `.`, with an extension * from the list it writes -- rather than a file kept under its own name. Mirrors the server's diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Attachments.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Attachments.kt index 2a00cf2..5abfe69 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/Attachments.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Attachments.kt @@ -38,6 +38,11 @@ suspend fun uploadPickedImage( * Uploads whatever [uri] names, the way its kind needs. An image goes through [uploadPickedImage] * and is shrunk; anything else goes whole, under the name the other app or the file chooser gave * it, because the session is told that name rather than shown the bytes. + * + * A picture is refused here, before anything is read or sent, when [images] says this session's + * model cannot read one. Here rather than beside the photo button because this is where every way + * of attaching meets: the picker, the file chooser, and another app's share sheet -- and only the + * first of those has a button to disable. */ suspend fun uploadPicked( context: Context, @@ -45,10 +50,16 @@ suspend fun uploadPicked( sessionId: String, uri: Uri, maxEdge: Int?, + images: ImageSupport, ): String { val resolver = context.contentResolver val mime = resolver.getType(uri) if (mime != null && mime.startsWith("image/")) { + if (images == ImageSupport.REFUSED) { + throw ApiException( + "this session's model can't read pictures, so that one wasn't attached" + ) + } return uploadPickedImage(context, settings, sessionId, uri, maxEdge) } // Opened before the request starts, so a provider that refuses says so here and not from inside diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt index a300a0c..c3d1478 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt @@ -168,6 +168,16 @@ sealed class SessionEvent { */ data class Settings(val model: String?, val permissionMode: String?) : SessionEvent() + /** + * Whether a picture can be sent to this session now, as the thing serving its model answered. + * + * Only a llama.cpp session says this, and it says it twice per model: unknown the moment the + * old one is left, then the loaded server's answer. It carries no row -- it is what the + * composer's photo button is drawn from, and a line in the transcript about a control is not + * something anybody asked after. + */ + data class Images(val images: ImageSupport) : SessionEvent() + /** * What a turn cost, and how much the model was holding when it ended. * @@ -328,6 +338,7 @@ fun parseSeqEvent(json: String): SeqEvent { model = body.optString("model").ifEmpty { null }, permissionMode = body.optString("permissionMode").ifEmpty { null }, ) + "images" -> SessionEvent.Images(imageSupport(body.optString("images"))) "contextWindow" -> SessionEvent.ContextWindow(body.getLong("tokens")) "usageDelta" -> SessionEvent.UsageDelta( diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt index 8ef7619..50bb87f 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt @@ -306,6 +306,16 @@ fun SessionScreen( // reads as a rename that did not take. var title by remember(summary.id) { mutableStateOf(summary.title) } var model by remember { mutableStateOf(summary.model) } + // Whether this session takes pictures. Held here rather than read from the row that opened the + // screen for the reason [model] is, and it moves with the model: a llama.cpp session changed + // onto a model without vision stops offering the photo button at that moment, not at whatever + // later point the session list is fetched again. + var images by remember { mutableStateOf(summary.images) } + // Pictures attached to a message this session can no longer be sent -- the model was changed + // under them. The message is stopped here as well as at the server, which refuses it whichever + // device sent it: a picture stripped out on the way would be answered as though it had never + // been mentioned. + val strandedImages = images == ImageSupport.REFUSED && pendingAttachments.any(::isImageRef) // The thinking level, held here for the same reason [title] is: the settings dialog can change // it, and the row this screen was opened from is a snapshot taken before that. Read straight // from the summary, a level set here was drawn as the old one again the next time the dialog @@ -517,6 +527,7 @@ fun SessionScreen( event.model?.let { model = it } event.permissionMode?.let { permissionMode = it } } + if (event is SessionEvent.Images) images = event.images if (event is SessionEvent.Status) { // The event's own timestamp, so a compaction that began before this screen // opened is timed from when it actually began -- and a compaction worth asking @@ -853,6 +864,7 @@ fun SessionScreen( if (!isSubagent) { model = summary.model permissionMode = summary.permissionMode ?: "auto" + images = summary.images } if (status != "compacting") compactingSince = null // Nothing to put back, so these rows are the screen and the probe can return under @@ -1397,7 +1409,14 @@ fun SessionScreen( // An image is shrunk to what this session's provider takes before it is // uploaded, so a twelve-megapixel photo does not cross the tunnel to be // rejected at the far end; a file goes whole. - uploadPicked(context, settings, summary.id, uri, summary.maxImageEdge) + uploadPicked( + context, + settings, + summary.id, + uri, + summary.maxImageEdge, + images, + ) } pendingAttachments = pendingAttachments + id actionError = null @@ -2058,6 +2077,18 @@ fun SessionScreen( refs = pendingAttachments, onRemove = { pendingAttachments = pendingAttachments - it }, ) + // Only reachable by the model changing under something already attached, since + // attaching is refused where the answer is already known. Said beside the + // pictures it is about and above the button it disables, which is where + // somebody can act on it: removing them is what makes this sendable. + if (strandedImages) { + Text( + "This model can't read pictures. Remove them to send this.", + color = MaterialTheme.colorScheme.error, + style = MaterialTheme.typography.bodySmall, + modifier = Modifier.padding(bottom = 4.dp), + ) + } OutlinedTextField( value = input, onValueChange = { @@ -2091,8 +2122,23 @@ fun SessionScreen( properties = PopupProperties(clippingEnabled = false), shape = BubbleMenuShape, ) { + // Disabled rather than dropped: an item that comes and goes + // makes its own absence the message, and the reason belongs on + // the item, where somebody wondering why is looking. + val takesPictures = images != ImageSupport.REFUSED DropdownMenuItem( - text = { Text("Photo") }, + text = { + Column { + Text("Photo") + if (!takesPictures) { + Text( + "this model can't read them", + style = MaterialTheme.typography.labelSmall, + ) + } + } + }, + enabled = takesPictures, onClick = { attaching = false pickImage.launch( @@ -2200,7 +2246,9 @@ fun SessionScreen( // reason above. CircleButton( onClick = { send() }, - enabled = input.text.isNotBlank() || pendingAttachments.isNotEmpty(), + enabled = + (input.text.isNotBlank() || pendingAttachments.isNotEmpty()) && + !strandedImages, fill = if (running) queueColor else sendColor, ) { Glyph( diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt index 9ce2b20..e6f6101 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt @@ -600,6 +600,9 @@ fun foldEvent(items: List, entry: SeqEvent): List items is SessionEvent.Settings -> items + // Neither carries a row: both are about what the session can do rather than about anything + // said in it, and the composer is where they are drawn. + is SessionEvent.Images -> items is SessionEvent.BackgroundTasks -> items is SessionEvent.Status -> settleReply(items, event.state) is SessionEvent.AuthenticationRequired -> diff --git a/server/src/config.rs b/server/src/config.rs index d3368a7..d02ff3b 100644 --- a/server/src/config.rs +++ b/server/src/config.rs @@ -21,6 +21,8 @@ use serde::{Deserialize, Serialize}; use wg_app_link::format; +use crate::session::driver::Images; + #[derive(Debug, Clone, Default, Serialize, Deserialize)] #[serde(rename_all = "camelCase", default)] pub struct Config { @@ -230,6 +232,21 @@ impl DriverKind { } } + /// What a session of this kind makes of an image attached to a message. + /// + /// The provider's own answer, which for every CLI here is that it takes + /// them. llama.cpp is [`Images::Unknown`] because the question is not the + /// provider's to answer: a local model reads images only if its file has a + /// projector loaded beside it, and the server that loaded it is the only + /// thing that knows -- `LlamaDriver::images` gives the live answer once + /// there is one, and this is what a session with no process yet reports. + pub fn images(self) -> Images { + match self { + DriverKind::Echo | DriverKind::ClaudeCli | DriverKind::CodexCli => Images::Accepted, + DriverKind::LlamaCpp => Images::Unknown, + } + } + /// Which paid service meters a session of this kind, and `None` for one /// that costs nothing. /// @@ -542,6 +559,18 @@ pub const LLAMA_MODEL_PARAMS: &[ParamSpec] = &[ }, restart: true, }, + ParamSpec { + key: "mmproj", + label: "Vision projector", + // Found rather than asked for, because a repository publishing a + // vision model publishes the projector beside it -- so this is for the + // two cases finding it cannot cover: a repository that published more + // than one, and a model whose projector somebody would rather not + // spend the memory on. + unset: "the mmproj file beside the model -- a file name, or \"off\" for none", + kind: ParamKind::Text, + restart: true, + }, ParamSpec { key: "specDraftNMax", label: "Tokens drafted ahead", diff --git a/server/src/machines.rs b/server/src/machines.rs index 215803d..d6cd172 100644 --- a/server/src/machines.rs +++ b/server/src/machines.rs @@ -153,7 +153,15 @@ pub async fn provider_models( ) -> Result> { if provider.kind == DriverKind::LlamaCpp { let dir = crate::models::dir_on(transport, models_dir); - let found = crate::models::on_machine(transport, &dir).await?; + let mut found = crate::models::on_machine(transport, &dir).await?; + // A vision model's projector is a file beside it rather than a model, + // and the session that reads pictures is the one on the model: offered + // here it is a chip that starts a server which cannot load it. It is + // still in the machine's own model list, which is where a file on a + // disk is managed and deleted. + found.retain(|model| { + !crate::session::llama::is_projector(model.key.rsplit('/').next().unwrap_or_default()) + }); let labels = crate::models::labels(&found); return Ok(found .into_iter() diff --git a/server/src/session/claude.rs b/server/src/session/claude.rs index a733b8e..49ed5e0 100644 --- a/server/src/session/claude.rs +++ b/server/src/session/claude.rs @@ -1150,14 +1150,7 @@ fn missing_conversation(detail: &str) -> bool { /// one this server would have written, so a crafted id cannot name a file /// outside the session. fn attachment_path(session_dir: &Path, id: &str) -> Result { - if !id - .chars() - .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-' || c == '_') - || id.contains("..") - { - anyhow::bail!("invalid attachment id"); - } - let path = session_dir.join("attachments").join(id); + let path = super::driver::attachment_path(session_dir, id)?; // A file copied to the session's own machine is named where it landed there // -- `routes::upload_attachment` writes that down beside it -- because the // path has to be one the CLI can open, not one this server can. diff --git a/server/src/session/codex.rs b/server/src/session/codex.rs index f32a6f2..84ea499 100644 --- a/server/src/session/codex.rs +++ b/server/src/session/codex.rs @@ -540,7 +540,7 @@ fn input_for(inner: &Inner, message: &Waiting) -> Result> { let mut files = Vec::new(); let mut input = Vec::new(); for attachment in &message.attachments { - let path = attachment_path(&inner.session_dir, attachment)?; + let path = super::driver::attachment_path(&inner.session_dir, attachment)?; if let Some(media_type) = crate::media::media_type_for(attachment) { if matches!(inner.transport, Transport::Here) { input.push(json!({"type": "localImage", "path": path})); @@ -1319,17 +1319,6 @@ fn protocol_error(inner: &Inner, message: String) { }); } -fn attachment_path(session_dir: &Path, id: &str) -> Result { - if id.contains("..") - || !id - .chars() - .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_')) - { - anyhow::bail!("invalid attachment id"); - } - Ok(session_dir.join("attachments").join(id)) -} - pub(super) fn read_thread(session_dir: &Path) -> Option { serde_json::from_str::(&std::fs::read_to_string(session_dir.join(THREAD_FILE)).ok()?) .ok()? diff --git a/server/src/session/driver.rs b/server/src/session/driver.rs index d026859..6d34060 100644 --- a/server/src/session/driver.rs +++ b/server/src/session/driver.rs @@ -89,6 +89,52 @@ pub(in crate::session) fn message_body( /// `crate::media::media_type_for`. pub type AttachmentRef = String; +/// Whether a session can be sent an image. +/// +/// Asked before one is attached rather than when it is sent, because that is +/// the only moment at which the answer is worth anything: a photo refused +/// after it has been picked, shrunk and uploaded over the tunnel is a refusal +/// that cost everything sending it would have. +/// +/// Three states and not a boolean, for the reason +/// [`crate::session::llama`] has to have them: a local model's answer is the +/// loaded server's to give, so a session whose model is still coming off disk +/// -- or has never been started -- genuinely does not know yet. "We could not +/// find out" drawn as "no" is a control that is never offered on a model that +/// reads images perfectly well. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum Images { + /// The model reads them, so attaching one is offered. + Accepted, + /// It does not, and one sent anyway is refused where it arrives. + Refused, + /// Nobody has been able to ask. Offered, because the alternative is + /// withholding the control on every session that has not started yet. + Unknown, +} + +/// Where an uploaded attachment sits in a session's directory. +/// +/// Refused rather than resolved when the id is not one this server would have +/// written: the id reaches here from a phone and from transcript lines, and a +/// `..` in one would name a file outside the session. Every driver that opens +/// an attachment goes through this, so there is one answer to what an id may +/// hold. +pub(in crate::session) fn attachment_path( + session_dir: &std::path::Path, + id: &str, +) -> anyhow::Result { + if id.contains("..") + || !id + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_')) + { + anyhow::bail!("invalid attachment id"); + } + Ok(session_dir.join("attachments").join(id)) +} + /// One choice offered in answer to a [`Event::Question`]. More than a label /// because the reader is deciding rather than confirming: what an option /// means, and what picking it would produce, are what decide it. @@ -391,6 +437,18 @@ pub enum Event { #[serde(default, skip_serializing_if = "Option::is_none")] permission_mode: Option, }, + /// Whether a picture can be sent here, as the thing serving the model + /// answered -- see [`Images`]. + /// + /// Its own event rather than a field on [`Event::Settings`] because it is + /// not something anybody set: it is measured, twice per model, and by a + /// driver rather than asked for by a phone. Emitted by llama.cpp alone, + /// where a model change can take the answer away -- so the phone withdraws + /// the control the moment the model that could read pictures is left, + /// rather than at whatever later point the session row is fetched again. + Images { + images: Images, + }, /// How much context this session's model has to hold a conversation in. /// /// The denominator the phone draws [`Event::UsageDelta`]'s `context` @@ -783,6 +841,17 @@ pub trait Driver: Send + Sync { /// the transcript, so a driver that never sends it drops the message from /// the conversation entirely. fn send_user_message(&self, text: String, attachments: Vec); + /// What this session makes of an image *now*, where that is not simply a + /// property of the provider -- `None` leaves the answer to + /// [`crate::config::DriverKind::images`], which is the whole answer for a + /// CLI that takes them whatever it is talking to. + /// + /// Overridden by llama.cpp alone, and it has to be: which model a session + /// is on is changeable while it runs, and whether that model reads images + /// is the loaded server's answer rather than the provider's. + fn images(&self) -> Option { + None + } /// Takes back a message that is still waiting, named by the id its /// [`Event::MessageQueued`] carried. /// diff --git a/server/src/session/llama/mod.rs b/server/src/session/llama/mod.rs index 8250c0d..b44ccfe 100644 --- a/server/src/session/llama/mod.rs +++ b/server/src/session/llama/mod.rs @@ -70,7 +70,7 @@ use serde::{Deserialize, Serialize}; use serde_json::{Value, json}; use super::driver::{ - AttachmentRef, Driver, Event, EventSink, QuestionOption, SessionStatus, Unqueued, + AttachmentRef, Driver, Event, EventSink, Images, QuestionOption, SessionStatus, Unqueued, }; use super::process; use super::transport::{Launch, Transport}; @@ -173,7 +173,7 @@ const MAX_STEPS: usize = 32; #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] struct Message { role: String, - content: String, + content: Content, /// What the assistant asked to run, in the wire's own shape so it goes /// back exactly as it came. #[serde(default, skip_serializing_if = "Vec::is_empty")] @@ -183,16 +183,54 @@ struct Message { tool_call_id: Option, } +/// What one message's `content` holds. +/// +/// Words alone for all but one case, and that case is why this is not a +/// `String`: an image reaches a model as a *part* of a user message, beside +/// the words, and `llama-server` reads the OpenAI shape for that. Untagged, +/// so a message without images serializes exactly as it did before images +/// existed -- which is what keeps every turn in a long conversation +/// byte-identical to the one the server already has cached. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(untagged)] +enum Content { + Text(String), + Parts(Vec), +} + impl Message { fn new(role: &str, content: impl Into) -> Self { Self { role: role.to_string(), - content: content.into(), + content: Content::Text(content.into()), tool_calls: Vec::new(), tool_call_id: None, } } + /// What somebody sent: their words, and whatever images went with them. + /// + /// The images come first, as they do in the Claude driver, because a + /// question asked before the picture it is about reads as a question about + /// nothing. A message with no readable image is an ordinary one rather + /// than a one-part list -- the two render differently through some chat + /// templates, and only the first has ever been tested by everything else + /// here. + fn from_user(text: impl Into, images: Vec) -> Self { + let text = text.into(); + if images.is_empty() { + return Self::new("user", text); + } + let mut parts = images; + if !text.is_empty() { + parts.push(json!({"type": "text", "text": text})); + } + Self { + content: Content::Parts(parts), + ..Self::new("user", String::new()) + } + } + fn result_of(call: &str, content: impl Into) -> Self { Self { tool_call_id: Some(call.to_string()), @@ -304,9 +342,13 @@ impl Serves { #[derive(Default)] struct Turns { running: bool, - waiting: std::collections::VecDeque<(String, String)>, + waiting: std::collections::VecDeque, } +/// A message accepted from the phone that nothing has read yet: the id its +/// [`Event::MessageQueued`] announced, the words, and what was attached. +type Waiting = (String, String, Vec); + /// What it takes to put this session on a different model. /// /// Kept whole rather than reduced to the two fields that change, because @@ -391,6 +433,10 @@ struct Shared { /// so about a model nobody has asked yet is the one mistake here that /// looks exactly like an answer. thinking_options: Mutex>>, + /// Whether the loaded model reads images ([`vision_of`]), `None` until it + /// has been asked -- the same three states, and for the same reason, as + /// `thinking_options` above. + vision: Mutex>, } pub struct LlamaDriver { @@ -458,6 +504,7 @@ impl LlamaDriver { watching: AtomicBool::new(false), thinking: Mutex::new(chosen_thinking(&meta.params)), thinking_options: Mutex::new(None), + vision: Mutex::new(None), }), respawn: Respawn { meta: meta.clone(), @@ -500,9 +547,16 @@ impl LlamaDriver { // as loading until the model is in memory, rather than looking ready // and refusing the first message. *shared.serving.lock().unwrap() = Serving::Loading; - // The answer belonged to whatever was loaded before this; the model + // These answers belonged to whatever was loaded before this; the model // starting now gets asked for itself. *shared.thinking_options.lock().unwrap() = None; + *shared.vision.lock().unwrap() = None; + // Said as it is taken away, not only when the next answer arrives: a + // session leaving a model with vision must stop offering the control + // now, while the picture that would be refused is still unattached. + let _ = shared.sink.send(Event::Images { + images: Images::Unknown, + }); let _ = shared.sink.send(Event::Status { state: SessionStatus::Loading, }); @@ -561,6 +615,13 @@ impl LlamaDriver { // otherwise surface as the model simply not doing it. *shared.thinking_options.lock().unwrap() = Some(thinking_options(&serves)); shared.note_unusable_thinking(); + // Asked for the same reason again: whether a model reads + // images is whether its projector loaded, which only the + // server that loaded it can see. + *shared.vision.lock().unwrap() = vision_of(&serves); + shared.emit(Event::Images { + images: shared.images(), + }); Serving::Ready { serves, tools: Arc::new(tools), @@ -627,6 +688,51 @@ impl Shared { }); } + /// What this session makes of a picture now, which is the loaded model's + /// answer and changes with it. + fn images(&self) -> Images { + match *self + .vision + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + { + Some(true) => Images::Accepted, + Some(false) => Images::Refused, + None => Images::Unknown, + } + } + + /// Whether an image sent now would be read. "We could not ask" is not a + /// refusal: the message goes, and the server says so itself -- loudly, as + /// a refused request -- if that was wrong. + fn reads_images(&self) -> bool { + self.images() != Images::Refused + } + + /// Says so, and answers true, when these attachments cannot go to the + /// loaded model. + /// + /// The whole message is stopped rather than stripped of its images: + /// `llama-server` refuses the entire request over one image part, so a + /// message sent without them is a different message from the one somebody + /// wrote, answered as though the picture had never been mentioned. + /// + /// Asked in three places because there are three moments at which the + /// answer can first exist: when a message arrives, when one that queued + /// behind a loading model is read, and when one typed during a turn is + /// steered into it. + fn refuse_images(&self, attachments: &[AttachmentRef]) -> bool { + if attachments.is_empty() || self.reads_images() { + return false; + } + self.emit(Event::Error { + message: "this model can't read images, so that message wasn't sent. Put the \ + session on a model with vision, or send it without the picture." + .to_string(), + }); + true + } + /// Whether the model is still on its way, which is what a message sent /// now has to wait behind. fn loading(&self) -> bool { @@ -911,8 +1017,8 @@ impl LlamaDriver { } } }; - if let Some((id, text)) = next { - Self::run_turn(shared, Some(id), text); + if let Some((id, text, images)) = next { + Self::run_turn(shared, Some(id), text, images); } } @@ -922,7 +1028,12 @@ impl LlamaDriver { /// model takes to generate, and a tool call inside it blocks for as long /// as the tool takes, which for a shell command is unbounded by anything /// this server knows. - fn run_turn(shared: &Arc, queued: Option, text: String) { + fn run_turn( + shared: &Arc, + queued: Option, + text: String, + images: Vec, + ) { let shared = Arc::clone(shared); std::thread::spawn(move || { shared.cancel.store(false, Ordering::SeqCst); @@ -932,9 +1043,14 @@ impl LlamaDriver { // this message -- and it is then sent twice, once folded out of // the transcript and once appended to it. Certain rather than // racy across a load, which is where it was found. - let ready = shared - .await_ready() - .map(|(serves, tools)| (serves, tools, conversation(&shared.transcript))); + let ready = shared.await_ready().map(|(serves, tools)| { + // Which of the two an earlier message's attachments fold back + // as is this model's answer, not the one they were sent to: + // the session may have been moved onto a model without vision + // since. + let seen = shared.reads_images().then(|| shared.session_dir.as_path()); + (serves, tools, conversation(&shared.transcript, seen)) + }); // The message goes into the transcript here, at the moment it is // read -- see `MessageTaken`. `queued` names the bubble this // resolves, and is `None` for one that never waited. Recorded @@ -943,21 +1059,27 @@ impl LlamaDriver { shared.emit(Event::MessageTaken { id: queued, text: text.clone(), - // Never any: this driver refuses them where they arrive, so - // nothing is carried this far. - attachments: Vec::new(), + attachments: images.clone(), }); match ready { Ok((serves, tools, mut messages)) => { - shared.emit(Event::Status { - state: SessionStatus::Running, - }); - messages.push(Message::new("user", text)); - if let Err(err) = converse(&shared, &serves, &tools, messages) { - shared.emit(Event::Error { - message: format!("{err:#}"), + // Refused here as well as at the door, because a message + // that queued behind a loading model was accepted before + // anything knew what that model could read. + if !shared.refuse_images(&images) { + shared.emit(Event::Status { + state: SessionStatus::Running, }); + messages.push(Message::from_user( + text, + image_parts(&shared.session_dir, &images), + )); + if let Err(err) = converse(&shared, &serves, &tools, messages) { + shared.emit(Event::Error { + message: format!("{err:#}"), + }); + } } shared.emit(Event::Status { state: SessionStatus::Idle, @@ -1054,18 +1176,23 @@ fn take_steers(shared: &Arc, messages: &mut Vec) { return; } let waiting = std::mem::take(&mut shared.turns.lock().unwrap().waiting); - for (id, text) in waiting { + for (id, text, images) in waiting { // The message enters the transcript here, below the calls that had // not read it and above the ones that will -- the same rule the // Claude driver announces a steer by. shared.emit(Event::MessageTaken { id: Some(id), text: text.clone(), - attachments: Vec::new(), + attachments: images.clone(), }); - messages.push(Message::new( - "user", + // The same refusal the other two paths make, for the third moment at + // which it can first be known -- see `Shared::refuse_images`. + if shared.refuse_images(&images) { + continue; + } + messages.push(Message::from_user( super::driver::message_body(&text, &[], true), + image_parts(&shared.session_dir, &images), )); } } @@ -1171,11 +1298,34 @@ fn permitted(shared: &Arc, call: &Call) -> bool { impl Driver for LlamaDriver { fn send_user_message(&self, text: String, attachments: Vec) { - if !attachments.is_empty() { + // An image rides in the request itself, which is how it reaches a + // model on another machine. Any other file has nowhere to go: this + // driver hands the model no path it could open, and the upload is on + // this side of the tunnel rather than on the serving machine. + let (images, files): (Vec<_>, Vec<_>) = attachments + .into_iter() + .partition(|id| crate::media::media_type_for(id).is_some()); + if !files.is_empty() { self.shared.emit(Event::Error { - message: "this model can't be sent attachments or files".to_string(), + message: format!( + "a llama.cpp session can be shown a picture but not handed a file, so {} \ + wasn't sent.", + // The name it was attached under, which is what the reader + // recognises; the hex before it is this server's. + super::names( + files + .iter() + .map(|id| id.split_once('-').map_or(&id[..], |(_, name)| name)) + ), + ), }); } + // Refused while somebody is still looking at the screen, where the + // model has loaded and cannot read them. While it is loading nothing + // knows yet, and the message queues below with its images. + if self.shared.refuse_images(&images) { + return; + } // A message written during a turn does not start a second // conversation against the same server: it waits for the turn's next // tool boundary and steers it from there (`take_steers`), or for the @@ -1193,22 +1343,24 @@ impl Driver for LlamaDriver { // while the model loaded is the one that opens the first turn. if turns.running || !turns.waiting.is_empty() || self.shared.loading() { let id = super::random_hex(); - turns.waiting.push_back((id.clone(), text.clone())); + turns + .waiting + .push_back((id.clone(), text.clone(), images.clone())); drop(turns); self.shared.emit(Event::MessageQueued { id, text, - // Refused above, so there are none to wait with it. Said - // as an empty list rather than the caller's, which would - // draw a thumbnail on a bubble whose message will arrive - // without it. - attachments: Vec::new(), + attachments: images, }); return; } turns.running = true; } - Self::run_turn(&self.shared, None, text); + Self::run_turn(&self.shared, None, text, images); + } + + fn images(&self) -> Option { + Some(self.shared.images()) } fn unqueue(&self, id: &str) -> Unqueued { @@ -1377,6 +1529,34 @@ impl Driver for LlamaDriver { } } +/// The uploaded images among `attachments`, as the content parts a request +/// carries them in. +/// +/// Inline data URIs rather than paths, because the model is on the serving +/// machine as often as not and the upload is on this one -- the same reason +/// the Claude and Codex drivers base64 an image into the message instead of +/// naming it. +/// +/// Anything that is not an image, or that cannot be read, is left out: the +/// send path has already said so about both, and a turn is a worse place to +/// learn it than the moment the thing was attached. +fn image_parts(session_dir: &Path, attachments: &[AttachmentRef]) -> Vec { + use base64::Engine as _; + attachments + .iter() + .filter_map(|id| { + let media_type = crate::media::media_type_for(id)?; + let path = super::driver::attachment_path(session_dir, id).ok()?; + let bytes = std::fs::read(path).ok()?; + let data = base64::engine::general_purpose::STANDARD.encode(bytes); + Some(json!({ + "type": "image_url", + "image_url": {"url": format!("data:{media_type};base64,{data}")}, + })) + }) + .collect() +} + /// The conversation so far, folded out of the transcript. /// /// Consecutive `AssistantText` deltas are one assistant turn, closed by the @@ -1392,12 +1572,17 @@ impl Driver for LlamaDriver { /// request or renders a conversation where the model asked for something and /// nothing came back, and the second is worse than the first. /// +/// `images` is the session directory, for a model that reads them, and `None` +/// for one that does not -- see [`sent`]. +/// /// This must stay a pure function of the transcript and must never re-render /// earlier turns. llama.cpp caches the prompt prefix, so a growing conversation /// reprocesses almost nothing -- but only while every turn is byte-identical to /// last time. Changing how an old turn is rendered silently reprocesses the -/// whole history on every message. -fn conversation(path: &Path) -> Vec { +/// whole history on every message. (Which of the two an attachment renders as +/// is settled by the model, so it changes only when the model does -- and that +/// throws the prefix away regardless.) +fn conversation(path: &Path, images: Option<&Path>) -> Vec { let Ok(events) = crate::session::transcript::read_after(path, 0) else { return Vec::new(); }; @@ -1412,9 +1597,11 @@ fn conversation(path: &Path) -> Vec { let mut fold = Fold::default(); for event in events.iter().cloned() { match event.event { - Event::UserMessage { text, .. } => { + Event::UserMessage { + text, attachments, .. + } => { fold.close(); - fold.messages.push(Message::new("user", text)); + fold.messages.push(sent(images, &text, &attachments)); } Event::AssistantText { delta } => { // Text after a call belongs to the reply that follows it, not @@ -1445,6 +1632,35 @@ fn conversation(path: &Path) -> Vec { fold.messages } +/// One message somebody sent, as the model is to read it. +/// +/// `images` is [`conversation`]'s: the session directory for a model with +/// vision, `None` for one without. Without it an attachment becomes a line +/// saying there was one, because a question about a picture the model was +/// never shown is worse read as a question about nothing. +fn sent(images: Option<&Path>, text: &str, attachments: &[AttachmentRef]) -> Message { + let parts = images.map_or_else(Vec::new, |dir| image_parts(dir, attachments)); + if !parts.is_empty() { + return Message::from_user(text, parts); + } + let count = attachments.len(); + if count == 0 { + return Message::new("user", text); + } + let note = format!( + "[{count} attachment{} sent with this message, which you cannot read.]", + if count == 1 { "" } else { "s" } + ); + Message::new( + "user", + if text.is_empty() { + note + } else { + format!("{text}\n\n{note}") + }, + ) +} + /// The running state of [`conversation`]: the assistant turn being assembled, /// and what is known about the calls in it. #[derive(Default)] @@ -1482,6 +1698,24 @@ impl Fold { } } +/// The projector lying in `dir`, absolute, or `None` for a directory with +/// none. +/// +/// The first by name where there are several, which is a choice rather than a +/// guess: a repository publishing two publishes them at different precisions +/// (`mmproj-F16`, `mmproj-F32`), and the smaller sorts first. The model +/// setting is how to ask for the other one. +fn projector_in(dir: &Path) -> Option { + let first = std::fs::read_dir(dir) + .ok()? + .filter_map(|entry| { + let name = entry.ok()?.file_name().into_string().ok()?; + is_projector(&name).then_some(name) + }) + .min()?; + Some(dir.join(first).to_string_lossy().into_owned()) +} + /// Where a model key resolves to on disk, refusing anything that climbs /// out of the models directory -- the key arrives from a phone. fn model_path(models_dir: &Path, key: &str) -> Result { @@ -1509,6 +1743,32 @@ pub struct Model { /// in the "yes" direction is a server that exits rather than one that runs /// slightly differently. mtp: bool, + /// The multimodal projector lying beside it, absolute on that machine, and + /// `None` where there is none -- which is most models. + /// + /// Found rather than configured because of how these arrive: a repository + /// that publishes a vision model publishes its projector in the same + /// directory, so downloading both is all somebody should have to do to be + /// able to send the model a picture. The model setting of the same name + /// overrides this, including with "off" -- see + /// [`crate::config::LLAMA_MODEL_PARAMS`]. + mmproj: Option, +} + +/// Whether a file lying beside a model is a multimodal projector for it. +/// +/// By name, which is the only thing both sides of this can see cheaply: the +/// two conventions in the wild are `mmproj-.gguf` and +/// `-mmproj-.gguf`, so the test is the word anywhere in a +/// `.gguf` name. A file that is not really one is a server that fails to +/// start and says so, rather than a model that quietly answers wrongly. +/// +/// Said twice, because the same question is asked of two filesystems: the +/// other half is the `grep` in [`model_on`]'s remote script, and the two have +/// to keep meaning the same thing. +pub fn is_projector(name: &str) -> bool { + let name = name.to_ascii_lowercase(); + name.ends_with(".gguf") && name.contains("mmproj") } /// The model file **on the machine that will serve it**, confirmed to be @@ -1531,9 +1791,11 @@ fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result Result/dev/null | grep -i 'mmproj.*\\.gguf$' | head -n 1); \ + printf 'at\\t%s\\t%s\\t%s\\n' \"$(head -c {prefix} \"$p\" | base64 | tr -d '\\n')\" \ + \"$m\" \"$p\"", prefix = crate::gguf::PREFIX_BYTES, ); let launch = Launch::new( @@ -1568,16 +1832,23 @@ fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result Ok(Model { - path: resolved.to_string(), + Some((head, projector, resolved)) => Ok(Model { mtp: mtp_in_prefix(head), + // Joined here rather than over there, so that only one side has an + // opinion about what "beside the model" means. + mmproj: (!projector.is_empty()).then(|| { + let dir = resolved.rsplit_once('/').map_or("", |(dir, _)| dir); + format!("{dir}/{projector}") + }), + path: resolved.to_string(), }), None => bail!( "{name} has no model at {path}. A llama.cpp session serves the file from the \ @@ -1613,6 +1884,25 @@ fn context_window(serves: &Serves) -> Option { .and_then(Value::as_u64) } +/// Whether the loaded model reads images, from the server that loaded it. +/// +/// `/props` reports the modalities of what is in memory -- vision is there +/// when a projector was loaded beside the weights, which is the preset's +/// `mmproj` and not anything the model file says on its own. `None` is a +/// server that would not answer or is too old to say, which is "we did not +/// find out" rather than "no": the phone withholds a control on the second +/// and not on the first. +fn vision_of(serves: &Serves) -> Option { + ureq::get(serves.query("/props")) + .call() + .ok()? + .body_mut() + .read_json::() + .ok()? + .pointer("/modalities/vision") + .and_then(Value::as_bool) +} + /// What the loaded model's chat template takes for thinking, asked of it. /// /// Two different questions, because they are two different template arguments. @@ -2031,6 +2321,21 @@ mod tests { use super::*; use crate::session::transcript::Transcript; + impl Content { + /// What a message says, for the assertions that are about the words. + /// A message carrying images keeps them beside its text, so this is + /// the text part among them. + fn words(&self) -> &str { + match self { + Content::Text(text) => text, + Content::Parts(parts) => parts + .iter() + .find_map(|part| part.get("text").and_then(Value::as_str)) + .unwrap_or_default(), + } + } + } + /// Writes a transcript the way the pump does, so the fold is tested against /// the real file format rather than a hand-built vector. fn transcript_with(events: &[Event]) -> (tempfile::TempDir, PathBuf) { @@ -2069,11 +2374,11 @@ mod tests { delta: "yes".into(), }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); assert_eq!( messages .iter() - .map(|m| (m.role.as_str(), m.content.as_str())) + .map(|m| (m.role.as_str(), m.content.words())) .collect::>(), [ ("user", "hello"), @@ -2121,9 +2426,9 @@ mod tests { delta: "one two".into(), }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); assert_eq!(messages.len(), 2); - assert_eq!(messages[1].content, "one two"); + assert_eq!(messages[1].content.words(), "one two"); } #[test] @@ -2159,11 +2464,11 @@ mod tests { delta: "right".into(), }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); assert_eq!( messages .iter() - .map(|m| (m.role.as_str(), m.content.as_str())) + .map(|m| (m.role.as_str(), m.content.words())) .collect::>(), [ ("user", "read it"), @@ -2194,9 +2499,9 @@ mod tests { state: SessionStatus::Idle, }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); assert_eq!(messages.len(), 2); - assert_eq!(messages[1].content, "one two"); + assert_eq!(messages[1].content.words(), "one two"); } #[test] @@ -2226,10 +2531,10 @@ mod tests { prefill_ms: None, }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); assert_eq!(messages.len(), 2); - assert_eq!(messages[0].content, "hello"); - assert_eq!(messages[1].content, "still here"); + assert_eq!(messages[0].content.words(), "hello"); + assert_eq!(messages[1].content.words(), "still here"); } #[test] @@ -2256,10 +2561,10 @@ mod tests { delta: "cheaply".into(), }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); assert_eq!(messages.len(), 2); - assert_eq!(messages[0].content, "a fresh start"); - assert_eq!(messages[1].content, "cheaply"); + assert_eq!(messages[0].content.words(), "a fresh start"); + assert_eq!(messages[1].content.words(), "cheaply"); } #[test] @@ -2285,9 +2590,9 @@ mod tests { attachments: Vec::new(), }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); assert_eq!(messages.len(), 1); - assert_eq!(messages[0].content, "three"); + assert_eq!(messages[0].content.words(), "three"); } #[test] @@ -2319,10 +2624,10 @@ mod tests { delta: "It says alpha and beta.".into(), }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); let shape: Vec<(&str, &str)> = messages .iter() - .map(|m| (m.role.as_str(), m.content.as_str())) + .map(|m| (m.role.as_str(), m.content.words())) .collect(); assert_eq!( shape, @@ -2371,12 +2676,12 @@ mod tests { output: "first".into(), }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); assert_eq!(messages[1].tool_calls.len(), 2); assert_eq!( messages[2..] .iter() - .map(|m| (m.tool_call_id.as_deref(), m.content.as_str())) + .map(|m| (m.tool_call_id.as_deref(), m.content.words())) .collect::>(), [(Some("a"), "first"), (Some("b"), "second")], ); @@ -2404,10 +2709,10 @@ mod tests { state: SessionStatus::Idle, }, ]); - let messages = conversation(&path); + let messages = conversation(&path, None); assert_eq!(messages[1].tool_calls.len(), 1); assert_eq!(messages[2].role, "tool"); - assert_eq!(messages[2].content, tools::UNFINISHED); + assert_eq!(messages[2].content.words(), tools::UNFINISHED); } #[test] @@ -2437,10 +2742,10 @@ mod tests { delta: "it is linux".into(), }, ]); - let messages = conversation(&path); - assert_eq!(messages[1].content, "checking"); + let messages = conversation(&path, None); + assert_eq!(messages[1].content.words(), "checking"); assert_eq!(messages[1].tool_calls.len(), 1); - assert_eq!(messages[3].content, "it is linux"); + assert_eq!(messages[3].content.words(), "it is linux"); assert!(messages[3].tool_calls.is_empty()); } @@ -2535,4 +2840,94 @@ mod tests { ); } } + + /// An image sent to a model that reads them goes into the message as a + /// part beside the words, and one sent to a model that does not is said + /// in words -- never dropped, which would leave a question about a picture + /// reading as a question about nothing. + #[test] + fn a_picture_is_a_part_for_a_model_that_reads_them_and_a_line_for_one_that_does_not() { + let dir = tempfile::tempdir().expect("tempdir"); + std::fs::create_dir(dir.path().join("attachments")).expect("attachments"); + std::fs::write(dir.path().join("attachments").join("ab12.png"), b"PNG") + .expect("write image"); + let refs = ["ab12.png".to_string()]; + + let seen = sent(Some(dir.path()), "what is this?", &refs); + assert_eq!( + seen.content, + Content::Parts(vec![ + json!({ + "type": "image_url", + "image_url": {"url": "data:image/png;base64,UE5H"}, + }), + json!({"type": "text", "text": "what is this?"}), + ]), + ); + + let unseen = sent(None, "what is this?", &refs); + assert_eq!( + unseen.content, + Content::Text( + "what is this?\n\n[1 attachment sent with this message, which you cannot \ + read.]" + .to_string() + ), + ); + + // Nothing attached is an ordinary message either way, and in + // particular not a one-part list: that renders differently through + // some chat templates, and every turn in this file is the other shape. + assert_eq!( + sent(Some(dir.path()), "plain", &[]).content, + Content::Text("plain".to_string()), + ); + } + + /// A file that cannot be read is left out rather than failing the turn: + /// the send path has already said so, and a turn is a worse place to learn + /// it. + #[test] + fn an_unreadable_attachment_is_not_a_picture() { + let dir = tempfile::tempdir().expect("tempdir"); + assert!(image_parts(dir.path(), &["ab12.png".to_string()]).is_empty()); + // Not an image at all, and an id this server would never have written. + assert!(image_parts(dir.path(), &["ab12-trace.txt".to_string()]).is_empty()); + assert!(image_parts(dir.path(), &["../../etc/passwd.png".to_string()]).is_empty()); + } + + /// Both conventions repositories publish a projector under, and nothing + /// else -- a model file that merely sits beside one is not one. + #[test] + fn a_projector_is_known_by_its_name() { + assert!(is_projector("mmproj-SmolVLM-256M-Instruct-Q8_0.gguf")); + assert!(is_projector("Qwen2.5-VL-7B-mmproj-F16.GGUF")); + assert!(!is_projector("SmolVLM-256M-Instruct-Q8_0.gguf")); + assert!(!is_projector("mmproj-notes.txt")); + } + + /// The projector found beside a model is the one it loads with, and the + /// smaller of two sorts first. + #[test] + fn the_projector_beside_a_model_is_found() { + let dir = tempfile::tempdir().expect("tempdir"); + for name in [ + "SmolVLM-256M-Instruct-Q8_0.gguf", + "mmproj-F32.gguf", + "mmproj-F16.gguf", + ] { + std::fs::write(dir.path().join(name), b"").expect("write"); + } + assert_eq!( + projector_in(dir.path()), + Some( + dir.path() + .join("mmproj-F16.gguf") + .to_string_lossy() + .into_owned() + ), + ); + let empty = tempfile::tempdir().expect("tempdir"); + assert_eq!(projector_in(empty.path()), None); + } } diff --git a/server/src/session/llama/router.rs b/server/src/session/llama/router.rs index 1de021d..c08f5f2 100644 --- a/server/src/session/llama/router.rs +++ b/server/src/session/llama/router.rs @@ -707,9 +707,37 @@ fn section(found: &Model, settings: &BTreeMap) -> String { if found.mtp && settings.get("speculative").map(String::as_str) != Some("off") { lines.push("spec-type = draft-mtp".to_string()); } + // What makes a model able to read pictures, and the one flag here that is + // found rather than defaulted: the projector is a second file published + // beside the weights, so a model that has one is loaded with it unless + // this model's settings name another or turn it off. + if let Some(projector) = projector(found, settings) { + lines.push(format!("mmproj = {projector}")); + } lines.join("\n") } +/// Which projector this model is loaded with: what its settings say, else +/// whatever was found beside it, and nothing for `"off"`. +/// +/// A setting naming a bare file name means one in the model's own directory, +/// since that is where the alternatives to the file found there are; anything +/// with a `/` in it is taken as the path it is, absolute or not -- the serving +/// machine resolves it, and this side does not know its working directory. +fn projector(found: &Model, settings: &BTreeMap) -> Option { + let chosen = settings.get("mmproj").map(|value| value.trim()); + match chosen { + Some("off") => None, + Some("") => found.mmproj.clone(), + Some(name) if name.contains('/') => Some(name.to_string()), + Some(name) => { + let dir = found.path.rsplit_once('/').map_or("", |(dir, _)| dir); + Some(format!("{dir}/{name}")) + } + None => found.mmproj.clone(), + } +} + /// The preset file with `name`'s section replaced by `body`, added at the end /// if it was not there. /// @@ -782,6 +810,7 @@ mod tests { #[test] fn a_section_names_the_file_and_the_flags_that_were_set() { let found = Model { + mmproj: None, path: "/models/a.gguf".to_string(), mtp: true, }; @@ -810,6 +839,50 @@ mod tests { assert!(!section(&found, &settings(&[("speculative", "off")])).contains("spec-type")); } + /// The projector found beside a model is loaded with it; the setting names + /// another where a repository published several, or turns it off. + #[test] + fn a_vision_model_is_loaded_with_its_projector() { + let found = Model { + path: "/models/repo/a.gguf".to_string(), + mtp: false, + mmproj: Some("/models/repo/mmproj-F16.gguf".to_string()), + }; + let line = |settings: &[(&str, &str)]| { + section(&found, &self::settings(settings)) + .lines() + .find_map(|line| line.strip_prefix("mmproj = ")) + .map(str::to_string) + }; + assert_eq!(line(&[]), Some("/models/repo/mmproj-F16.gguf".to_string())); + assert_eq!( + line(&[("mmproj", " ")]), + Some("/models/repo/mmproj-F16.gguf".to_string()) + ); + assert_eq!(line(&[("mmproj", "off")]), None); + // A bare name is one of the model's own neighbours; anything with a + // separator in it is the path it says it is. + assert_eq!( + line(&[("mmproj", "mmproj-F32.gguf")]), + Some("/models/repo/mmproj-F32.gguf".to_string()), + ); + assert_eq!( + line(&[("mmproj", "/elsewhere/p.gguf")]), + Some("/elsewhere/p.gguf".to_string()), + ); + // A model with none, and nothing asked for, loads without one. + assert!( + !section( + &Model { + mmproj: None, + ..found.clone() + }, + &settings(&[]) + ) + .contains("mmproj") + ); + } + #[test] fn a_section_replaces_its_own_and_leaves_every_other_line_alone() { let first = upsert("", "repo/a.gguf", "model = /models/a.gguf"); diff --git a/server/src/session/mod.rs b/server/src/session/mod.rs index 92d1e07..5df98cb 100644 --- a/server/src/session/mod.rs +++ b/server/src/session/mod.rs @@ -36,7 +36,7 @@ use crate::config::{ use claude::ClaudeDriver; use codex::CodexDriver; use driver::{ - AttachmentRef, BackgroundTask, Driver, Event, EventSink, SessionCommand, SessionStatus, + AttachmentRef, BackgroundTask, Driver, Event, EventSink, Images, SessionCommand, SessionStatus, Unqueued, context_after, context_limit_after, }; use echo::EchoDriver; @@ -235,6 +235,10 @@ pub struct SessionInfo { /// that happens to be big" are different answers. #[serde(skip_serializing_if = "Option::is_none")] pub max_image_edge: Option, + /// Whether an image can be sent here at all -- see [`Images`]. Reported + /// so the phone can withhold the control rather than take a photo, + /// shrink it, upload it over the tunnel and then be told. + pub images: Images, /// Which of `GET /usage`'s snapshots reports on this session, and /// absent where nothing meters it -- see /// [`DriverKind::usage_provider`]. @@ -610,6 +614,13 @@ impl LiveSession { auto_resume_message: resume_message(current), resume_at: current.resume.map(|scheduled| scheduled.at), max_image_edge: kind.and_then(DriverKind::max_image_edge), + // The driver's answer where it has one -- llama.cpp's depends on + // the model it is on now, which is not the provider's to say. + images: self + .driver() + .and_then(|driver| driver.images()) + .or_else(|| kind.map(DriverKind::images)) + .unwrap_or(Images::Unknown), usage_provider: kind.and_then(DriverKind::usage_provider), imported, keeps_own_transcript: kind.is_some_and(DriverKind::keeps_own_transcript), @@ -1156,6 +1167,10 @@ impl SessionManager { params: meta.params.clone(), max_image_edge: kind_of(&inner.config, &meta.machine, &meta.provider) .and_then(DriverKind::max_image_edge), + // No process, so nothing but the kind can answer; for + // llama.cpp that is deliberately "not known yet". + images: kind_of(&inner.config, &meta.machine, &meta.provider) + .map_or(Images::Unknown, DriverKind::images), usage_provider: kind_of(&inner.config, &meta.machine, &meta.provider) .and_then(DriverKind::usage_provider), notify: meta.notify,