Let a llama session be shown a picture where the model reads one
A multimodal model is loaded with the `mmproj` found beside its weights -- which is how a repository publishes the pair -- and an attached image rides in the request as an `image_url` data URI, so it reaches a model on another machine without the file going there. Nothing is done for a model without a projector: no captioning, no OCR, no second model. Whether a session takes pictures is measured rather than assumed: `/props`'s `modalities.vision` from the server that loaded the model, in three states, because a model still coming off disk has genuinely not said. Unknown is offered rather than refused -- a control withheld because nobody could ask goes missing from sessions that would have taken it. The answer reaches the phone twice per model as `Event::Images`, so the photo button is withdrawn the moment a model with vision is left rather than at whatever later point the session row is fetched again. A message carrying an image a model cannot read is stopped rather than stripped: `llama-server` refuses the whole request over one image part, and a message sent without its picture would be answered as though the picture had never been mentioned. The phone will not attach one, and the driver refuses it again at the three moments the answer can first exist -- at the door, when a message queued behind a loading model is read, and at the tool boundary a steer enters by. An earlier turn's image folds into a line of words for a model without vision, so switching a conversation onto one does not end it. A projector is filtered out of the models a provider *offers*, since a session started on one is a server that cannot load it; it stays in the machine's own model list, where a file on a disk is managed. Verified against ggml-org/SmolVLM-256M-Instruct-GGUF, local and over ssh: "In this picture there is a red circle." Switching that session to Qwen3-0.6B reports `refused`, refuses the next picture with the reason, and still answers an ordinary message.
This commit is contained in:
1 parent
b7fd18b195
commit
bd9596d782
17 files changed
+835
-97
No files matched your search
@@ -36,7 +36,7 @@ use crate::config::{
|
||||
use claude::ClaudeDriver;
|
||||
use codex::CodexDriver;
|
||||
use driver::{
|
||||
AttachmentRef, BackgroundTask, Driver, Event, EventSink, SessionCommand, SessionStatus,
|
||||
AttachmentRef, BackgroundTask, Driver, Event, EventSink, Images, SessionCommand, SessionStatus,
|
||||
Unqueued, context_after, context_limit_after,
|
||||
};
|
||||
use echo::EchoDriver;
|
||||
@@ -235,6 +235,10 @@ pub struct SessionInfo {
|
||||
/// that happens to be big" are different answers.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub max_image_edge: Option<u32>,
|
||||
/// Whether an image can be sent here at all -- see [`Images`]. Reported
|
||||
/// so the phone can withhold the control rather than take a photo,
|
||||
/// shrink it, upload it over the tunnel and then be told.
|
||||
pub images: Images,
|
||||
/// Which of `GET /usage`'s snapshots reports on this session, and
|
||||
/// absent where nothing meters it -- see
|
||||
/// [`DriverKind::usage_provider`].
|
||||
@@ -610,6 +614,13 @@ impl LiveSession {
|
||||
auto_resume_message: resume_message(current),
|
||||
resume_at: current.resume.map(|scheduled| scheduled.at),
|
||||
max_image_edge: kind.and_then(DriverKind::max_image_edge),
|
||||
// The driver's answer where it has one -- llama.cpp's depends on
|
||||
// the model it is on now, which is not the provider's to say.
|
||||
images: self
|
||||
.driver()
|
||||
.and_then(|driver| driver.images())
|
||||
.or_else(|| kind.map(DriverKind::images))
|
||||
.unwrap_or(Images::Unknown),
|
||||
usage_provider: kind.and_then(DriverKind::usage_provider),
|
||||
imported,
|
||||
keeps_own_transcript: kind.is_some_and(DriverKind::keeps_own_transcript),
|
||||
@@ -1156,6 +1167,10 @@ impl SessionManager {
|
||||
params: meta.params.clone(),
|
||||
max_image_edge: kind_of(&inner.config, &meta.machine, &meta.provider)
|
||||
.and_then(DriverKind::max_image_edge),
|
||||
// No process, so nothing but the kind can answer; for
|
||||
// llama.cpp that is deliberately "not known yet".
|
||||
images: kind_of(&inner.config, &meta.machine, &meta.provider)
|
||||
.map_or(Images::Unknown, DriverKind::images),
|
||||
usage_provider: kind_of(&inner.config, &meta.machine, &meta.provider)
|
||||
.and_then(DriverKind::usage_provider),
|
||||
notify: meta.notify,
|
||||
|
||||
Reference in new issue
Block a user