Let a llama session be shown a picture where the model reads one

A multimodal model is loaded with the `mmproj` found beside its weights --
which is how a repository publishes the pair -- and an attached image rides
in the request as an `image_url` data URI, so it reaches a model on another
machine without the file going there. Nothing is done for a model without a
projector: no captioning, no OCR, no second model.

Whether a session takes pictures is measured rather than assumed:
`/props`'s `modalities.vision` from the server that loaded the model, in
three states, because a model still coming off disk has genuinely not said.
Unknown is offered rather than refused -- a control withheld because nobody
could ask goes missing from sessions that would have taken it. The answer
reaches the phone twice per model as `Event::Images`, so the photo button is
withdrawn the moment a model with vision is left rather than at whatever
later point the session row is fetched again.

A message carrying an image a model cannot read is stopped rather than
stripped: `llama-server` refuses the whole request over one image part, and
a message sent without its picture would be answered as though the picture
had never been mentioned. The phone will not attach one, and the driver
refuses it again at the three moments the answer can first exist -- at the
door, when a message queued behind a loading model is read, and at the tool
boundary a steer enters by. An earlier turn's image folds into a line of
words for a model without vision, so switching a conversation onto one does
not end it.

A projector is filtered out of the models a provider *offers*, since a
session started on one is a server that cannot load it; it stays in the
machine's own model list, where a file on a disk is managed.

Verified against ggml-org/SmolVLM-256M-Instruct-GGUF, local and over ssh:
"In this picture there is a red circle." Switching that session to
Qwen3-0.6B reports `refused`, refuses the next picture with the reason, and
still answers an ordinary message.
This commit is contained in:
iris-ai committed 2026-09-20 16:19:53 -04:00
1 parent b7fd18b195
commit bd9596d782
17 files changed
+835 -97

No files matched your search

+69
View File
@@ -89,6 +89,52 @@ pub(in crate::session) fn message_body(
/// `crate::media::media_type_for`.
pub type AttachmentRef = String;
/// Whether a session can be sent an image.
///
/// Asked before one is attached rather than when it is sent, because that is
/// the only moment at which the answer is worth anything: a photo refused
/// after it has been picked, shrunk and uploaded over the tunnel is a refusal
/// that cost everything sending it would have.
///
/// Three states and not a boolean, for the reason
/// [`crate::session::llama`] has to have them: a local model's answer is the
/// loaded server's to give, so a session whose model is still coming off disk
/// -- or has never been started -- genuinely does not know yet. "We could not
/// find out" drawn as "no" is a control that is never offered on a model that
/// reads images perfectly well.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub enum Images {
/// The model reads them, so attaching one is offered.
Accepted,
/// It does not, and one sent anyway is refused where it arrives.
Refused,
/// Nobody has been able to ask. Offered, because the alternative is
/// withholding the control on every session that has not started yet.
Unknown,
}
/// Where an uploaded attachment sits in a session's directory.
///
/// Refused rather than resolved when the id is not one this server would have
/// written: the id reaches here from a phone and from transcript lines, and a
/// `..` in one would name a file outside the session. Every driver that opens
/// an attachment goes through this, so there is one answer to what an id may
/// hold.
pub(in crate::session) fn attachment_path(
session_dir: &std::path::Path,
id: &str,
) -> anyhow::Result<std::path::PathBuf> {
if id.contains("..")
|| !id
.chars()
.all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_'))
{
anyhow::bail!("invalid attachment id");
}
Ok(session_dir.join("attachments").join(id))
}
/// One choice offered in answer to a [`Event::Question`]. More than a label
/// because the reader is deciding rather than confirming: what an option
/// means, and what picking it would produce, are what decide it.
@@ -391,6 +437,18 @@ pub enum Event {
#[serde(default, skip_serializing_if = "Option::is_none")]
permission_mode: Option<String>,
},
/// Whether a picture can be sent here, as the thing serving the model
/// answered -- see [`Images`].
///
/// Its own event rather than a field on [`Event::Settings`] because it is
/// not something anybody set: it is measured, twice per model, and by a
/// driver rather than asked for by a phone. Emitted by llama.cpp alone,
/// where a model change can take the answer away -- so the phone withdraws
/// the control the moment the model that could read pictures is left,
/// rather than at whatever later point the session row is fetched again.
Images {
images: Images,
},
/// How much context this session's model has to hold a conversation in.
///
/// The denominator the phone draws [`Event::UsageDelta`]'s `context`
@@ -783,6 +841,17 @@ pub trait Driver: Send + Sync {
/// the transcript, so a driver that never sends it drops the message from
/// the conversation entirely.
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>);
/// What this session makes of an image *now*, where that is not simply a
/// property of the provider -- `None` leaves the answer to
/// [`crate::config::DriverKind::images`], which is the whole answer for a
/// CLI that takes them whatever it is talking to.
///
/// Overridden by llama.cpp alone, and it has to be: which model a session
/// is on is changeable while it runs, and whether that model reads images
/// is the loaded server's answer rather than the provider's.
fn images(&self) -> Option<Images> {
None
}
/// Takes back a message that is still waiting, named by the id its
/// [`Event::MessageQueued`] carried.
///