Let a llama session be shown a picture where the model reads one

A multimodal model is loaded with the `mmproj` found beside its weights --
which is how a repository publishes the pair -- and an attached image rides
in the request as an `image_url` data URI, so it reaches a model on another
machine without the file going there. Nothing is done for a model without a
projector: no captioning, no OCR, no second model.

Whether a session takes pictures is measured rather than assumed:
`/props`'s `modalities.vision` from the server that loaded the model, in
three states, because a model still coming off disk has genuinely not said.
Unknown is offered rather than refused -- a control withheld because nobody
could ask goes missing from sessions that would have taken it. The answer
reaches the phone twice per model as `Event::Images`, so the photo button is
withdrawn the moment a model with vision is left rather than at whatever
later point the session row is fetched again.

A message carrying an image a model cannot read is stopped rather than
stripped: `llama-server` refuses the whole request over one image part, and
a message sent without its picture would be answered as though the picture
had never been mentioned. The phone will not attach one, and the driver
refuses it again at the three moments the answer can first exist -- at the
door, when a message queued behind a loading model is read, and at the tool
boundary a steer enters by. An earlier turn's image folds into a line of
words for a model without vision, so switching a conversation onto one does
not end it.

A projector is filtered out of the models a provider *offers*, since a
session started on one is a server that cannot load it; it stays in the
machine's own model list, where a file on a disk is managed.

Verified against ggml-org/SmolVLM-256M-Instruct-GGUF, local and over ssh:
"In this picture there is a red circle." Switching that session to
Qwen3-0.6B reports `refused`, refuses the next picture with the reason, and
still answers an ordinary message.
This commit is contained in:
iris-ai committed 2026-09-20 16:19:53 -04:00
1 parent b7fd18b195
commit bd9596d782
17 files changed
+835 -97

No files matched your search

+73
View File
@@ -707,9 +707,37 @@ fn section(found: &Model, settings: &BTreeMap<String, String>) -> String {
if found.mtp && settings.get("speculative").map(String::as_str) != Some("off") {
lines.push("spec-type = draft-mtp".to_string());
}
// What makes a model able to read pictures, and the one flag here that is
// found rather than defaulted: the projector is a second file published
// beside the weights, so a model that has one is loaded with it unless
// this model's settings name another or turn it off.
if let Some(projector) = projector(found, settings) {
lines.push(format!("mmproj = {projector}"));
}
lines.join("\n")
}
/// Which projector this model is loaded with: what its settings say, else
/// whatever was found beside it, and nothing for `"off"`.
///
/// A setting naming a bare file name means one in the model's own directory,
/// since that is where the alternatives to the file found there are; anything
/// with a `/` in it is taken as the path it is, absolute or not -- the serving
/// machine resolves it, and this side does not know its working directory.
fn projector(found: &Model, settings: &BTreeMap<String, String>) -> Option<String> {
let chosen = settings.get("mmproj").map(|value| value.trim());
match chosen {
Some("off") => None,
Some("") => found.mmproj.clone(),
Some(name) if name.contains('/') => Some(name.to_string()),
Some(name) => {
let dir = found.path.rsplit_once('/').map_or("", |(dir, _)| dir);
Some(format!("{dir}/{name}"))
}
None => found.mmproj.clone(),
}
}
/// The preset file with `name`'s section replaced by `body`, added at the end
/// if it was not there.
///
@@ -782,6 +810,7 @@ mod tests {
#[test]
fn a_section_names_the_file_and_the_flags_that_were_set() {
let found = Model {
mmproj: None,
path: "/models/a.gguf".to_string(),
mtp: true,
};
@@ -810,6 +839,50 @@ mod tests {
assert!(!section(&found, &settings(&[("speculative", "off")])).contains("spec-type"));
}
/// The projector found beside a model is loaded with it; the setting names
/// another where a repository published several, or turns it off.
#[test]
fn a_vision_model_is_loaded_with_its_projector() {
let found = Model {
path: "/models/repo/a.gguf".to_string(),
mtp: false,
mmproj: Some("/models/repo/mmproj-F16.gguf".to_string()),
};
let line = |settings: &[(&str, &str)]| {
section(&found, &self::settings(settings))
.lines()
.find_map(|line| line.strip_prefix("mmproj = "))
.map(str::to_string)
};
assert_eq!(line(&[]), Some("/models/repo/mmproj-F16.gguf".to_string()));
assert_eq!(
line(&[("mmproj", " ")]),
Some("/models/repo/mmproj-F16.gguf".to_string())
);
assert_eq!(line(&[("mmproj", "off")]), None);
// A bare name is one of the model's own neighbours; anything with a
// separator in it is the path it says it is.
assert_eq!(
line(&[("mmproj", "mmproj-F32.gguf")]),
Some("/models/repo/mmproj-F32.gguf".to_string()),
);
assert_eq!(
line(&[("mmproj", "/elsewhere/p.gguf")]),
Some("/elsewhere/p.gguf".to_string()),
);
// A model with none, and nothing asked for, loads without one.
assert!(
!section(
&Model {
mmproj: None,
..found.clone()
},
&settings(&[])
)
.contains("mmproj")
);
}
#[test]
fn a_section_replaces_its_own_and_leaves_every_other_line_alone() {
let first = upsert("", "repo/a.gguf", "model = /models/a.gguf");