Let a llama session be shown a picture where the model reads one

A multimodal model is loaded with the `mmproj` found beside its weights --
which is how a repository publishes the pair -- and an attached image rides
in the request as an `image_url` data URI, so it reaches a model on another
machine without the file going there. Nothing is done for a model without a
projector: no captioning, no OCR, no second model.

Whether a session takes pictures is measured rather than assumed:
`/props`'s `modalities.vision` from the server that loaded the model, in
three states, because a model still coming off disk has genuinely not said.
Unknown is offered rather than refused -- a control withheld because nobody
could ask goes missing from sessions that would have taken it. The answer
reaches the phone twice per model as `Event::Images`, so the photo button is
withdrawn the moment a model with vision is left rather than at whatever
later point the session row is fetched again.

A message carrying an image a model cannot read is stopped rather than
stripped: `llama-server` refuses the whole request over one image part, and
a message sent without its picture would be answered as though the picture
had never been mentioned. The phone will not attach one, and the driver
refuses it again at the three moments the answer can first exist -- at the
door, when a message queued behind a loading model is read, and at the tool
boundary a steer enters by. An earlier turn's image folds into a line of
words for a model without vision, so switching a conversation onto one does
not end it.

A projector is filtered out of the models a provider *offers*, since a
session started on one is a server that cannot load it; it stays in the
machine's own model list, where a file on a disk is managed.

Verified against ggml-org/SmolVLM-256M-Instruct-GGUF, local and over ssh:
"In this picture there is a red circle." Switching that session to
Qwen3-0.6B reports `refused`, refuses the next picture with the reason, and
still answers an ordinary message.
This commit is contained in:
iris-ai committed 2026-09-20 16:19:53 -04:00
1 parent b7fd18b195
commit bd9596d782
17 files changed
+835 -97

No files matched your search

+35
View File
@@ -521,6 +521,41 @@ deliberate and easy to undo by accident:
of the file — on the machine that will serve it, in the round trip the spawn
was already making — and `params["speculative"] = "off"` is the way out.
- **A picture goes to a model that natively reads one, and nowhere else**
(2026-09-20). Vision here is a multimodal model loaded with its projector:
`mmproj` in that model's preset section, found beside the weights because
that is how a repository publishes the pair, and overridable per model —
another file name, or `off` — in the machines tab's provider view. Nothing
is done for a model without one: no captioning, no OCR, no second model.
An attached image rides in the request as an `image_url` content part with
a data URI, which is what reaches a model on another machine without
shipping the file there.
- **Whether a session takes pictures is measured, not assumed.**
`/props`'s `modalities.vision` is the loaded server's own answer, and it
is the only one worth having: the projector is loaded over there, and
this side cannot see whether it worked. Three states, because a model
still coming off disk has genuinely not said — `Images::Unknown` is
offered rather than refused, since withholding the control on a session
nobody could ask about hides it on models that read pictures perfectly
well. The answer reaches the phone twice per model, as `Event::Images`:
unknown the moment the old model is left, then the new server's answer.
- **A message carrying an image a model cannot read is stopped, not
stripped.** `llama-server` refuses the whole request over one image part,
and a message sent without its picture would be answered as though the
picture had never been mentioned. The phone will not attach one (the
photo item is disabled with the reason on it, and the share sheet and
file chooser refuse the same), and the driver refuses it again where it
arrives — at the door, when a message that queued behind a loading model
is read, and at the tool boundary a steer enters by, because those are
the three moments the answer can first exist.
- **An earlier turn's image becomes a line of words for a model without
vision**, so that switching a conversation onto one does not end it. It
is a statement that there was an attachment, not a description of it.
- **A projector is not a model.** It is filtered out of what a provider
offers a spawn or a model change, since a session started on one is a
server that cannot load it, and it stays in the machine's own model list,
which is where a file on a disk is managed.
### Models (2026-08-28, rebuilt per machine 2026-09-19)
- **A download belongs to the model, not to the request.** Keyed by