Let a llama session be shown a picture where the model reads one
A multimodal model is loaded with the `mmproj` found beside its weights -- which is how a repository publishes the pair -- and an attached image rides in the request as an `image_url` data URI, so it reaches a model on another machine without the file going there. Nothing is done for a model without a projector: no captioning, no OCR, no second model. Whether a session takes pictures is measured rather than assumed: `/props`'s `modalities.vision` from the server that loaded the model, in three states, because a model still coming off disk has genuinely not said. Unknown is offered rather than refused -- a control withheld because nobody could ask goes missing from sessions that would have taken it. The answer reaches the phone twice per model as `Event::Images`, so the photo button is withdrawn the moment a model with vision is left rather than at whatever later point the session row is fetched again. A message carrying an image a model cannot read is stopped rather than stripped: `llama-server` refuses the whole request over one image part, and a message sent without its picture would be answered as though the picture had never been mentioned. The phone will not attach one, and the driver refuses it again at the three moments the answer can first exist -- at the door, when a message queued behind a loading model is read, and at the tool boundary a steer enters by. An earlier turn's image folds into a line of words for a model without vision, so switching a conversation onto one does not end it. A projector is filtered out of the models a provider *offers*, since a session started on one is a server that cannot load it; it stays in the machine's own model list, where a file on a disk is managed. Verified against ggml-org/SmolVLM-256M-Instruct-GGUF, local and over ssh: "In this picture there is a red circle." Switching that session to Qwen3-0.6B reports `refused`, refuses the next picture with the reason, and still answers an ordinary message.
This commit is contained in:
iris-ai committed
1 parent
b7fd18b195
commit
bd9596d782
17 files changed
+835
-97
No files matched your search
@@ -1150,14 +1150,7 @@ fn missing_conversation(detail: &str) -> bool {
|
||||
/// one this server would have written, so a crafted id cannot name a file
|
||||
/// outside the session.
|
||||
fn attachment_path(session_dir: &Path, id: &str) -> Result<PathBuf> {
|
||||
if !id
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-' || c == '_')
|
||||
|| id.contains("..")
|
||||
{
|
||||
anyhow::bail!("invalid attachment id");
|
||||
}
|
||||
let path = session_dir.join("attachments").join(id);
|
||||
let path = super::driver::attachment_path(session_dir, id)?;
|
||||
// A file copied to the session's own machine is named where it landed there
|
||||
// -- `routes::upload_attachment` writes that down beside it -- because the
|
||||
// path has to be one the CLI can open, not one this server can.
|
||||
|
||||
@@ -540,7 +540,7 @@ fn input_for(inner: &Inner, message: &Waiting) -> Result<Vec<Value>> {
|
||||
let mut files = Vec::new();
|
||||
let mut input = Vec::new();
|
||||
for attachment in &message.attachments {
|
||||
let path = attachment_path(&inner.session_dir, attachment)?;
|
||||
let path = super::driver::attachment_path(&inner.session_dir, attachment)?;
|
||||
if let Some(media_type) = crate::media::media_type_for(attachment) {
|
||||
if matches!(inner.transport, Transport::Here) {
|
||||
input.push(json!({"type": "localImage", "path": path}));
|
||||
@@ -1319,17 +1319,6 @@ fn protocol_error(inner: &Inner, message: String) {
|
||||
});
|
||||
}
|
||||
|
||||
fn attachment_path(session_dir: &Path, id: &str) -> Result<PathBuf> {
|
||||
if id.contains("..")
|
||||
|| !id
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_'))
|
||||
{
|
||||
anyhow::bail!("invalid attachment id");
|
||||
}
|
||||
Ok(session_dir.join("attachments").join(id))
|
||||
}
|
||||
|
||||
pub(super) fn read_thread(session_dir: &Path) -> Option<String> {
|
||||
serde_json::from_str::<Value>(&std::fs::read_to_string(session_dir.join(THREAD_FILE)).ok()?)
|
||||
.ok()?
|
||||
|
||||
@@ -89,6 +89,52 @@ pub(in crate::session) fn message_body(
|
||||
/// `crate::media::media_type_for`.
|
||||
pub type AttachmentRef = String;
|
||||
|
||||
/// Whether a session can be sent an image.
|
||||
///
|
||||
/// Asked before one is attached rather than when it is sent, because that is
|
||||
/// the only moment at which the answer is worth anything: a photo refused
|
||||
/// after it has been picked, shrunk and uploaded over the tunnel is a refusal
|
||||
/// that cost everything sending it would have.
|
||||
///
|
||||
/// Three states and not a boolean, for the reason
|
||||
/// [`crate::session::llama`] has to have them: a local model's answer is the
|
||||
/// loaded server's to give, so a session whose model is still coming off disk
|
||||
/// -- or has never been started -- genuinely does not know yet. "We could not
|
||||
/// find out" drawn as "no" is a control that is never offered on a model that
|
||||
/// reads images perfectly well.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub enum Images {
|
||||
/// The model reads them, so attaching one is offered.
|
||||
Accepted,
|
||||
/// It does not, and one sent anyway is refused where it arrives.
|
||||
Refused,
|
||||
/// Nobody has been able to ask. Offered, because the alternative is
|
||||
/// withholding the control on every session that has not started yet.
|
||||
Unknown,
|
||||
}
|
||||
|
||||
/// Where an uploaded attachment sits in a session's directory.
|
||||
///
|
||||
/// Refused rather than resolved when the id is not one this server would have
|
||||
/// written: the id reaches here from a phone and from transcript lines, and a
|
||||
/// `..` in one would name a file outside the session. Every driver that opens
|
||||
/// an attachment goes through this, so there is one answer to what an id may
|
||||
/// hold.
|
||||
pub(in crate::session) fn attachment_path(
|
||||
session_dir: &std::path::Path,
|
||||
id: &str,
|
||||
) -> anyhow::Result<std::path::PathBuf> {
|
||||
if id.contains("..")
|
||||
|| !id
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_'))
|
||||
{
|
||||
anyhow::bail!("invalid attachment id");
|
||||
}
|
||||
Ok(session_dir.join("attachments").join(id))
|
||||
}
|
||||
|
||||
/// One choice offered in answer to a [`Event::Question`]. More than a label
|
||||
/// because the reader is deciding rather than confirming: what an option
|
||||
/// means, and what picking it would produce, are what decide it.
|
||||
@@ -391,6 +437,18 @@ pub enum Event {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
permission_mode: Option<String>,
|
||||
},
|
||||
/// Whether a picture can be sent here, as the thing serving the model
|
||||
/// answered -- see [`Images`].
|
||||
///
|
||||
/// Its own event rather than a field on [`Event::Settings`] because it is
|
||||
/// not something anybody set: it is measured, twice per model, and by a
|
||||
/// driver rather than asked for by a phone. Emitted by llama.cpp alone,
|
||||
/// where a model change can take the answer away -- so the phone withdraws
|
||||
/// the control the moment the model that could read pictures is left,
|
||||
/// rather than at whatever later point the session row is fetched again.
|
||||
Images {
|
||||
images: Images,
|
||||
},
|
||||
/// How much context this session's model has to hold a conversation in.
|
||||
///
|
||||
/// The denominator the phone draws [`Event::UsageDelta`]'s `context`
|
||||
@@ -783,6 +841,17 @@ pub trait Driver: Send + Sync {
|
||||
/// the transcript, so a driver that never sends it drops the message from
|
||||
/// the conversation entirely.
|
||||
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>);
|
||||
/// What this session makes of an image *now*, where that is not simply a
|
||||
/// property of the provider -- `None` leaves the answer to
|
||||
/// [`crate::config::DriverKind::images`], which is the whole answer for a
|
||||
/// CLI that takes them whatever it is talking to.
|
||||
///
|
||||
/// Overridden by llama.cpp alone, and it has to be: which model a session
|
||||
/// is on is changeable while it runs, and whether that model reads images
|
||||
/// is the loaded server's answer rather than the provider's.
|
||||
fn images(&self) -> Option<Images> {
|
||||
None
|
||||
}
|
||||
/// Takes back a message that is still waiting, named by the id its
|
||||
/// [`Event::MessageQueued`] carried.
|
||||
///
|
||||
|
||||
+467
-72
@@ -70,7 +70,7 @@ use serde::{Deserialize, Serialize};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use super::driver::{
|
||||
AttachmentRef, Driver, Event, EventSink, QuestionOption, SessionStatus, Unqueued,
|
||||
AttachmentRef, Driver, Event, EventSink, Images, QuestionOption, SessionStatus, Unqueued,
|
||||
};
|
||||
use super::process;
|
||||
use super::transport::{Launch, Transport};
|
||||
@@ -173,7 +173,7 @@ const MAX_STEPS: usize = 32;
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
struct Message {
|
||||
role: String,
|
||||
content: String,
|
||||
content: Content,
|
||||
/// What the assistant asked to run, in the wire's own shape so it goes
|
||||
/// back exactly as it came.
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
@@ -183,16 +183,54 @@ struct Message {
|
||||
tool_call_id: Option<String>,
|
||||
}
|
||||
|
||||
/// What one message's `content` holds.
|
||||
///
|
||||
/// Words alone for all but one case, and that case is why this is not a
|
||||
/// `String`: an image reaches a model as a *part* of a user message, beside
|
||||
/// the words, and `llama-server` reads the OpenAI shape for that. Untagged,
|
||||
/// so a message without images serializes exactly as it did before images
|
||||
/// existed -- which is what keeps every turn in a long conversation
|
||||
/// byte-identical to the one the server already has cached.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(untagged)]
|
||||
enum Content {
|
||||
Text(String),
|
||||
Parts(Vec<Value>),
|
||||
}
|
||||
|
||||
impl Message {
|
||||
fn new(role: &str, content: impl Into<String>) -> Self {
|
||||
Self {
|
||||
role: role.to_string(),
|
||||
content: content.into(),
|
||||
content: Content::Text(content.into()),
|
||||
tool_calls: Vec::new(),
|
||||
tool_call_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// What somebody sent: their words, and whatever images went with them.
|
||||
///
|
||||
/// The images come first, as they do in the Claude driver, because a
|
||||
/// question asked before the picture it is about reads as a question about
|
||||
/// nothing. A message with no readable image is an ordinary one rather
|
||||
/// than a one-part list -- the two render differently through some chat
|
||||
/// templates, and only the first has ever been tested by everything else
|
||||
/// here.
|
||||
fn from_user(text: impl Into<String>, images: Vec<Value>) -> Self {
|
||||
let text = text.into();
|
||||
if images.is_empty() {
|
||||
return Self::new("user", text);
|
||||
}
|
||||
let mut parts = images;
|
||||
if !text.is_empty() {
|
||||
parts.push(json!({"type": "text", "text": text}));
|
||||
}
|
||||
Self {
|
||||
content: Content::Parts(parts),
|
||||
..Self::new("user", String::new())
|
||||
}
|
||||
}
|
||||
|
||||
fn result_of(call: &str, content: impl Into<String>) -> Self {
|
||||
Self {
|
||||
tool_call_id: Some(call.to_string()),
|
||||
@@ -304,9 +342,13 @@ impl Serves {
|
||||
#[derive(Default)]
|
||||
struct Turns {
|
||||
running: bool,
|
||||
waiting: std::collections::VecDeque<(String, String)>,
|
||||
waiting: std::collections::VecDeque<Waiting>,
|
||||
}
|
||||
|
||||
/// A message accepted from the phone that nothing has read yet: the id its
|
||||
/// [`Event::MessageQueued`] announced, the words, and what was attached.
|
||||
type Waiting = (String, String, Vec<AttachmentRef>);
|
||||
|
||||
/// What it takes to put this session on a different model.
|
||||
///
|
||||
/// Kept whole rather than reduced to the two fields that change, because
|
||||
@@ -391,6 +433,10 @@ struct Shared {
|
||||
/// so about a model nobody has asked yet is the one mistake here that
|
||||
/// looks exactly like an answer.
|
||||
thinking_options: Mutex<Option<Vec<String>>>,
|
||||
/// Whether the loaded model reads images ([`vision_of`]), `None` until it
|
||||
/// has been asked -- the same three states, and for the same reason, as
|
||||
/// `thinking_options` above.
|
||||
vision: Mutex<Option<bool>>,
|
||||
}
|
||||
|
||||
pub struct LlamaDriver {
|
||||
@@ -458,6 +504,7 @@ impl LlamaDriver {
|
||||
watching: AtomicBool::new(false),
|
||||
thinking: Mutex::new(chosen_thinking(&meta.params)),
|
||||
thinking_options: Mutex::new(None),
|
||||
vision: Mutex::new(None),
|
||||
}),
|
||||
respawn: Respawn {
|
||||
meta: meta.clone(),
|
||||
@@ -500,9 +547,16 @@ impl LlamaDriver {
|
||||
// as loading until the model is in memory, rather than looking ready
|
||||
// and refusing the first message.
|
||||
*shared.serving.lock().unwrap() = Serving::Loading;
|
||||
// The answer belonged to whatever was loaded before this; the model
|
||||
// These answers belonged to whatever was loaded before this; the model
|
||||
// starting now gets asked for itself.
|
||||
*shared.thinking_options.lock().unwrap() = None;
|
||||
*shared.vision.lock().unwrap() = None;
|
||||
// Said as it is taken away, not only when the next answer arrives: a
|
||||
// session leaving a model with vision must stop offering the control
|
||||
// now, while the picture that would be refused is still unattached.
|
||||
let _ = shared.sink.send(Event::Images {
|
||||
images: Images::Unknown,
|
||||
});
|
||||
let _ = shared.sink.send(Event::Status {
|
||||
state: SessionStatus::Loading,
|
||||
});
|
||||
@@ -561,6 +615,13 @@ impl LlamaDriver {
|
||||
// otherwise surface as the model simply not doing it.
|
||||
*shared.thinking_options.lock().unwrap() = Some(thinking_options(&serves));
|
||||
shared.note_unusable_thinking();
|
||||
// Asked for the same reason again: whether a model reads
|
||||
// images is whether its projector loaded, which only the
|
||||
// server that loaded it can see.
|
||||
*shared.vision.lock().unwrap() = vision_of(&serves);
|
||||
shared.emit(Event::Images {
|
||||
images: shared.images(),
|
||||
});
|
||||
Serving::Ready {
|
||||
serves,
|
||||
tools: Arc::new(tools),
|
||||
@@ -627,6 +688,51 @@ impl Shared {
|
||||
});
|
||||
}
|
||||
|
||||
/// What this session makes of a picture now, which is the loaded model's
|
||||
/// answer and changes with it.
|
||||
fn images(&self) -> Images {
|
||||
match *self
|
||||
.vision
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
{
|
||||
Some(true) => Images::Accepted,
|
||||
Some(false) => Images::Refused,
|
||||
None => Images::Unknown,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether an image sent now would be read. "We could not ask" is not a
|
||||
/// refusal: the message goes, and the server says so itself -- loudly, as
|
||||
/// a refused request -- if that was wrong.
|
||||
fn reads_images(&self) -> bool {
|
||||
self.images() != Images::Refused
|
||||
}
|
||||
|
||||
/// Says so, and answers true, when these attachments cannot go to the
|
||||
/// loaded model.
|
||||
///
|
||||
/// The whole message is stopped rather than stripped of its images:
|
||||
/// `llama-server` refuses the entire request over one image part, so a
|
||||
/// message sent without them is a different message from the one somebody
|
||||
/// wrote, answered as though the picture had never been mentioned.
|
||||
///
|
||||
/// Asked in three places because there are three moments at which the
|
||||
/// answer can first exist: when a message arrives, when one that queued
|
||||
/// behind a loading model is read, and when one typed during a turn is
|
||||
/// steered into it.
|
||||
fn refuse_images(&self, attachments: &[AttachmentRef]) -> bool {
|
||||
if attachments.is_empty() || self.reads_images() {
|
||||
return false;
|
||||
}
|
||||
self.emit(Event::Error {
|
||||
message: "this model can't read images, so that message wasn't sent. Put the \
|
||||
session on a model with vision, or send it without the picture."
|
||||
.to_string(),
|
||||
});
|
||||
true
|
||||
}
|
||||
|
||||
/// Whether the model is still on its way, which is what a message sent
|
||||
/// now has to wait behind.
|
||||
fn loading(&self) -> bool {
|
||||
@@ -911,8 +1017,8 @@ impl LlamaDriver {
|
||||
}
|
||||
}
|
||||
};
|
||||
if let Some((id, text)) = next {
|
||||
Self::run_turn(shared, Some(id), text);
|
||||
if let Some((id, text, images)) = next {
|
||||
Self::run_turn(shared, Some(id), text, images);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -922,7 +1028,12 @@ impl LlamaDriver {
|
||||
/// model takes to generate, and a tool call inside it blocks for as long
|
||||
/// as the tool takes, which for a shell command is unbounded by anything
|
||||
/// this server knows.
|
||||
fn run_turn(shared: &Arc<Shared>, queued: Option<String>, text: String) {
|
||||
fn run_turn(
|
||||
shared: &Arc<Shared>,
|
||||
queued: Option<String>,
|
||||
text: String,
|
||||
images: Vec<AttachmentRef>,
|
||||
) {
|
||||
let shared = Arc::clone(shared);
|
||||
std::thread::spawn(move || {
|
||||
shared.cancel.store(false, Ordering::SeqCst);
|
||||
@@ -932,9 +1043,14 @@ impl LlamaDriver {
|
||||
// this message -- and it is then sent twice, once folded out of
|
||||
// the transcript and once appended to it. Certain rather than
|
||||
// racy across a load, which is where it was found.
|
||||
let ready = shared
|
||||
.await_ready()
|
||||
.map(|(serves, tools)| (serves, tools, conversation(&shared.transcript)));
|
||||
let ready = shared.await_ready().map(|(serves, tools)| {
|
||||
// Which of the two an earlier message's attachments fold back
|
||||
// as is this model's answer, not the one they were sent to:
|
||||
// the session may have been moved onto a model without vision
|
||||
// since.
|
||||
let seen = shared.reads_images().then(|| shared.session_dir.as_path());
|
||||
(serves, tools, conversation(&shared.transcript, seen))
|
||||
});
|
||||
// The message goes into the transcript here, at the moment it is
|
||||
// read -- see `MessageTaken`. `queued` names the bubble this
|
||||
// resolves, and is `None` for one that never waited. Recorded
|
||||
@@ -943,21 +1059,27 @@ impl LlamaDriver {
|
||||
shared.emit(Event::MessageTaken {
|
||||
id: queued,
|
||||
text: text.clone(),
|
||||
// Never any: this driver refuses them where they arrive, so
|
||||
// nothing is carried this far.
|
||||
attachments: Vec::new(),
|
||||
attachments: images.clone(),
|
||||
});
|
||||
|
||||
match ready {
|
||||
Ok((serves, tools, mut messages)) => {
|
||||
shared.emit(Event::Status {
|
||||
state: SessionStatus::Running,
|
||||
});
|
||||
messages.push(Message::new("user", text));
|
||||
if let Err(err) = converse(&shared, &serves, &tools, messages) {
|
||||
shared.emit(Event::Error {
|
||||
message: format!("{err:#}"),
|
||||
// Refused here as well as at the door, because a message
|
||||
// that queued behind a loading model was accepted before
|
||||
// anything knew what that model could read.
|
||||
if !shared.refuse_images(&images) {
|
||||
shared.emit(Event::Status {
|
||||
state: SessionStatus::Running,
|
||||
});
|
||||
messages.push(Message::from_user(
|
||||
text,
|
||||
image_parts(&shared.session_dir, &images),
|
||||
));
|
||||
if let Err(err) = converse(&shared, &serves, &tools, messages) {
|
||||
shared.emit(Event::Error {
|
||||
message: format!("{err:#}"),
|
||||
});
|
||||
}
|
||||
}
|
||||
shared.emit(Event::Status {
|
||||
state: SessionStatus::Idle,
|
||||
@@ -1054,18 +1176,23 @@ fn take_steers(shared: &Arc<Shared>, messages: &mut Vec<Message>) {
|
||||
return;
|
||||
}
|
||||
let waiting = std::mem::take(&mut shared.turns.lock().unwrap().waiting);
|
||||
for (id, text) in waiting {
|
||||
for (id, text, images) in waiting {
|
||||
// The message enters the transcript here, below the calls that had
|
||||
// not read it and above the ones that will -- the same rule the
|
||||
// Claude driver announces a steer by.
|
||||
shared.emit(Event::MessageTaken {
|
||||
id: Some(id),
|
||||
text: text.clone(),
|
||||
attachments: Vec::new(),
|
||||
attachments: images.clone(),
|
||||
});
|
||||
messages.push(Message::new(
|
||||
"user",
|
||||
// The same refusal the other two paths make, for the third moment at
|
||||
// which it can first be known -- see `Shared::refuse_images`.
|
||||
if shared.refuse_images(&images) {
|
||||
continue;
|
||||
}
|
||||
messages.push(Message::from_user(
|
||||
super::driver::message_body(&text, &[], true),
|
||||
image_parts(&shared.session_dir, &images),
|
||||
));
|
||||
}
|
||||
}
|
||||
@@ -1171,11 +1298,34 @@ fn permitted(shared: &Arc<Shared>, call: &Call) -> bool {
|
||||
|
||||
impl Driver for LlamaDriver {
|
||||
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>) {
|
||||
if !attachments.is_empty() {
|
||||
// An image rides in the request itself, which is how it reaches a
|
||||
// model on another machine. Any other file has nowhere to go: this
|
||||
// driver hands the model no path it could open, and the upload is on
|
||||
// this side of the tunnel rather than on the serving machine.
|
||||
let (images, files): (Vec<_>, Vec<_>) = attachments
|
||||
.into_iter()
|
||||
.partition(|id| crate::media::media_type_for(id).is_some());
|
||||
if !files.is_empty() {
|
||||
self.shared.emit(Event::Error {
|
||||
message: "this model can't be sent attachments or files".to_string(),
|
||||
message: format!(
|
||||
"a llama.cpp session can be shown a picture but not handed a file, so {} \
|
||||
wasn't sent.",
|
||||
// The name it was attached under, which is what the reader
|
||||
// recognises; the hex before it is this server's.
|
||||
super::names(
|
||||
files
|
||||
.iter()
|
||||
.map(|id| id.split_once('-').map_or(&id[..], |(_, name)| name))
|
||||
),
|
||||
),
|
||||
});
|
||||
}
|
||||
// Refused while somebody is still looking at the screen, where the
|
||||
// model has loaded and cannot read them. While it is loading nothing
|
||||
// knows yet, and the message queues below with its images.
|
||||
if self.shared.refuse_images(&images) {
|
||||
return;
|
||||
}
|
||||
// A message written during a turn does not start a second
|
||||
// conversation against the same server: it waits for the turn's next
|
||||
// tool boundary and steers it from there (`take_steers`), or for the
|
||||
@@ -1193,22 +1343,24 @@ impl Driver for LlamaDriver {
|
||||
// while the model loaded is the one that opens the first turn.
|
||||
if turns.running || !turns.waiting.is_empty() || self.shared.loading() {
|
||||
let id = super::random_hex();
|
||||
turns.waiting.push_back((id.clone(), text.clone()));
|
||||
turns
|
||||
.waiting
|
||||
.push_back((id.clone(), text.clone(), images.clone()));
|
||||
drop(turns);
|
||||
self.shared.emit(Event::MessageQueued {
|
||||
id,
|
||||
text,
|
||||
// Refused above, so there are none to wait with it. Said
|
||||
// as an empty list rather than the caller's, which would
|
||||
// draw a thumbnail on a bubble whose message will arrive
|
||||
// without it.
|
||||
attachments: Vec::new(),
|
||||
attachments: images,
|
||||
});
|
||||
return;
|
||||
}
|
||||
turns.running = true;
|
||||
}
|
||||
Self::run_turn(&self.shared, None, text);
|
||||
Self::run_turn(&self.shared, None, text, images);
|
||||
}
|
||||
|
||||
fn images(&self) -> Option<Images> {
|
||||
Some(self.shared.images())
|
||||
}
|
||||
|
||||
fn unqueue(&self, id: &str) -> Unqueued {
|
||||
@@ -1377,6 +1529,34 @@ impl Driver for LlamaDriver {
|
||||
}
|
||||
}
|
||||
|
||||
/// The uploaded images among `attachments`, as the content parts a request
|
||||
/// carries them in.
|
||||
///
|
||||
/// Inline data URIs rather than paths, because the model is on the serving
|
||||
/// machine as often as not and the upload is on this one -- the same reason
|
||||
/// the Claude and Codex drivers base64 an image into the message instead of
|
||||
/// naming it.
|
||||
///
|
||||
/// Anything that is not an image, or that cannot be read, is left out: the
|
||||
/// send path has already said so about both, and a turn is a worse place to
|
||||
/// learn it than the moment the thing was attached.
|
||||
fn image_parts(session_dir: &Path, attachments: &[AttachmentRef]) -> Vec<Value> {
|
||||
use base64::Engine as _;
|
||||
attachments
|
||||
.iter()
|
||||
.filter_map(|id| {
|
||||
let media_type = crate::media::media_type_for(id)?;
|
||||
let path = super::driver::attachment_path(session_dir, id).ok()?;
|
||||
let bytes = std::fs::read(path).ok()?;
|
||||
let data = base64::engine::general_purpose::STANDARD.encode(bytes);
|
||||
Some(json!({
|
||||
"type": "image_url",
|
||||
"image_url": {"url": format!("data:{media_type};base64,{data}")},
|
||||
}))
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The conversation so far, folded out of the transcript.
|
||||
///
|
||||
/// Consecutive `AssistantText` deltas are one assistant turn, closed by the
|
||||
@@ -1392,12 +1572,17 @@ impl Driver for LlamaDriver {
|
||||
/// request or renders a conversation where the model asked for something and
|
||||
/// nothing came back, and the second is worse than the first.
|
||||
///
|
||||
/// `images` is the session directory, for a model that reads them, and `None`
|
||||
/// for one that does not -- see [`sent`].
|
||||
///
|
||||
/// This must stay a pure function of the transcript and must never re-render
|
||||
/// earlier turns. llama.cpp caches the prompt prefix, so a growing conversation
|
||||
/// reprocesses almost nothing -- but only while every turn is byte-identical to
|
||||
/// last time. Changing how an old turn is rendered silently reprocesses the
|
||||
/// whole history on every message.
|
||||
fn conversation(path: &Path) -> Vec<Message> {
|
||||
/// whole history on every message. (Which of the two an attachment renders as
|
||||
/// is settled by the model, so it changes only when the model does -- and that
|
||||
/// throws the prefix away regardless.)
|
||||
fn conversation(path: &Path, images: Option<&Path>) -> Vec<Message> {
|
||||
let Ok(events) = crate::session::transcript::read_after(path, 0) else {
|
||||
return Vec::new();
|
||||
};
|
||||
@@ -1412,9 +1597,11 @@ fn conversation(path: &Path) -> Vec<Message> {
|
||||
let mut fold = Fold::default();
|
||||
for event in events.iter().cloned() {
|
||||
match event.event {
|
||||
Event::UserMessage { text, .. } => {
|
||||
Event::UserMessage {
|
||||
text, attachments, ..
|
||||
} => {
|
||||
fold.close();
|
||||
fold.messages.push(Message::new("user", text));
|
||||
fold.messages.push(sent(images, &text, &attachments));
|
||||
}
|
||||
Event::AssistantText { delta } => {
|
||||
// Text after a call belongs to the reply that follows it, not
|
||||
@@ -1445,6 +1632,35 @@ fn conversation(path: &Path) -> Vec<Message> {
|
||||
fold.messages
|
||||
}
|
||||
|
||||
/// One message somebody sent, as the model is to read it.
|
||||
///
|
||||
/// `images` is [`conversation`]'s: the session directory for a model with
|
||||
/// vision, `None` for one without. Without it an attachment becomes a line
|
||||
/// saying there was one, because a question about a picture the model was
|
||||
/// never shown is worse read as a question about nothing.
|
||||
fn sent(images: Option<&Path>, text: &str, attachments: &[AttachmentRef]) -> Message {
|
||||
let parts = images.map_or_else(Vec::new, |dir| image_parts(dir, attachments));
|
||||
if !parts.is_empty() {
|
||||
return Message::from_user(text, parts);
|
||||
}
|
||||
let count = attachments.len();
|
||||
if count == 0 {
|
||||
return Message::new("user", text);
|
||||
}
|
||||
let note = format!(
|
||||
"[{count} attachment{} sent with this message, which you cannot read.]",
|
||||
if count == 1 { "" } else { "s" }
|
||||
);
|
||||
Message::new(
|
||||
"user",
|
||||
if text.is_empty() {
|
||||
note
|
||||
} else {
|
||||
format!("{text}\n\n{note}")
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
/// The running state of [`conversation`]: the assistant turn being assembled,
|
||||
/// and what is known about the calls in it.
|
||||
#[derive(Default)]
|
||||
@@ -1482,6 +1698,24 @@ impl Fold {
|
||||
}
|
||||
}
|
||||
|
||||
/// The projector lying in `dir`, absolute, or `None` for a directory with
|
||||
/// none.
|
||||
///
|
||||
/// The first by name where there are several, which is a choice rather than a
|
||||
/// guess: a repository publishing two publishes them at different precisions
|
||||
/// (`mmproj-F16`, `mmproj-F32`), and the smaller sorts first. The model
|
||||
/// setting is how to ask for the other one.
|
||||
fn projector_in(dir: &Path) -> Option<String> {
|
||||
let first = std::fs::read_dir(dir)
|
||||
.ok()?
|
||||
.filter_map(|entry| {
|
||||
let name = entry.ok()?.file_name().into_string().ok()?;
|
||||
is_projector(&name).then_some(name)
|
||||
})
|
||||
.min()?;
|
||||
Some(dir.join(first).to_string_lossy().into_owned())
|
||||
}
|
||||
|
||||
/// Where a model key resolves to on disk, refusing anything that climbs
|
||||
/// out of the models directory -- the key arrives from a phone.
|
||||
fn model_path(models_dir: &Path, key: &str) -> Result<PathBuf> {
|
||||
@@ -1509,6 +1743,32 @@ pub struct Model {
|
||||
/// in the "yes" direction is a server that exits rather than one that runs
|
||||
/// slightly differently.
|
||||
mtp: bool,
|
||||
/// The multimodal projector lying beside it, absolute on that machine, and
|
||||
/// `None` where there is none -- which is most models.
|
||||
///
|
||||
/// Found rather than configured because of how these arrive: a repository
|
||||
/// that publishes a vision model publishes its projector in the same
|
||||
/// directory, so downloading both is all somebody should have to do to be
|
||||
/// able to send the model a picture. The model setting of the same name
|
||||
/// overrides this, including with "off" -- see
|
||||
/// [`crate::config::LLAMA_MODEL_PARAMS`].
|
||||
mmproj: Option<String>,
|
||||
}
|
||||
|
||||
/// Whether a file lying beside a model is a multimodal projector for it.
|
||||
///
|
||||
/// By name, which is the only thing both sides of this can see cheaply: the
|
||||
/// two conventions in the wild are `mmproj-<something>.gguf` and
|
||||
/// `<model>-mmproj-<something>.gguf`, so the test is the word anywhere in a
|
||||
/// `.gguf` name. A file that is not really one is a server that fails to
|
||||
/// start and says so, rather than a model that quietly answers wrongly.
|
||||
///
|
||||
/// Said twice, because the same question is asked of two filesystems: the
|
||||
/// other half is the `grep` in [`model_on`]'s remote script, and the two have
|
||||
/// to keep meaning the same thing.
|
||||
pub fn is_projector(name: &str) -> bool {
|
||||
let name = name.to_ascii_lowercase();
|
||||
name.ends_with(".gguf") && name.contains("mmproj")
|
||||
}
|
||||
|
||||
/// The model file **on the machine that will serve it**, confirmed to be
|
||||
@@ -1531,9 +1791,11 @@ fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result<Model
|
||||
let mtp = std::fs::File::open(&path)
|
||||
.map(|mut file| crate::gguf::has_mtp_head(&mut file))
|
||||
.unwrap_or(false);
|
||||
let mmproj = path.parent().and_then(projector_in);
|
||||
return Ok(Model {
|
||||
path: path.to_string_lossy().into_owned(),
|
||||
mtp,
|
||||
mmproj,
|
||||
});
|
||||
};
|
||||
// The same directory the spawn screen listed for this machine, and one
|
||||
@@ -1557,7 +1819,9 @@ fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result<Model
|
||||
let script = format!(
|
||||
"p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${{p#\"~/\"}};; esac; \
|
||||
[ -f \"$p\" ] || {{ printf 'missing\\n'; exit 0; }}; \
|
||||
printf 'at\\t%s\\t%s\\n' \"$(head -c {prefix} \"$p\" | base64 | tr -d '\\n')\" \"$p\"",
|
||||
m=$(ls \"${{p%/*}}\" 2>/dev/null | grep -i 'mmproj.*\\.gguf$' | head -n 1); \
|
||||
printf 'at\\t%s\\t%s\\t%s\\n' \"$(head -c {prefix} \"$p\" | base64 | tr -d '\\n')\" \
|
||||
\"$m\" \"$p\"",
|
||||
prefix = crate::gguf::PREFIX_BYTES,
|
||||
);
|
||||
let launch = Launch::new(
|
||||
@@ -1568,16 +1832,23 @@ fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result<Model
|
||||
let answer = transport
|
||||
.capture_blocking(&launch)
|
||||
.with_context(|| format!("couldn't ask {name} where its models are"))?;
|
||||
// The path last, so a `\t` in it survives; the middle field is base64,
|
||||
// whose alphabet has no tab.
|
||||
let found = answer
|
||||
.trim()
|
||||
.strip_prefix("at\t")
|
||||
.and_then(|rest| rest.split_once('\t'));
|
||||
// The path last, so a `\t` in it survives; the fields before it are
|
||||
// base64, whose alphabet has no tab, and a file name from the same
|
||||
// directory the path names.
|
||||
let found = answer.trim().strip_prefix("at\t").and_then(|rest| {
|
||||
let mut fields = rest.splitn(3, '\t');
|
||||
Some((fields.next()?, fields.next()?, fields.next()?))
|
||||
});
|
||||
match found {
|
||||
Some((head, resolved)) => Ok(Model {
|
||||
path: resolved.to_string(),
|
||||
Some((head, projector, resolved)) => Ok(Model {
|
||||
mtp: mtp_in_prefix(head),
|
||||
// Joined here rather than over there, so that only one side has an
|
||||
// opinion about what "beside the model" means.
|
||||
mmproj: (!projector.is_empty()).then(|| {
|
||||
let dir = resolved.rsplit_once('/').map_or("", |(dir, _)| dir);
|
||||
format!("{dir}/{projector}")
|
||||
}),
|
||||
path: resolved.to_string(),
|
||||
}),
|
||||
None => bail!(
|
||||
"{name} has no model at {path}. A llama.cpp session serves the file from the \
|
||||
@@ -1613,6 +1884,25 @@ fn context_window(serves: &Serves) -> Option<u64> {
|
||||
.and_then(Value::as_u64)
|
||||
}
|
||||
|
||||
/// Whether the loaded model reads images, from the server that loaded it.
|
||||
///
|
||||
/// `/props` reports the modalities of what is in memory -- vision is there
|
||||
/// when a projector was loaded beside the weights, which is the preset's
|
||||
/// `mmproj` and not anything the model file says on its own. `None` is a
|
||||
/// server that would not answer or is too old to say, which is "we did not
|
||||
/// find out" rather than "no": the phone withholds a control on the second
|
||||
/// and not on the first.
|
||||
fn vision_of(serves: &Serves) -> Option<bool> {
|
||||
ureq::get(serves.query("/props"))
|
||||
.call()
|
||||
.ok()?
|
||||
.body_mut()
|
||||
.read_json::<Value>()
|
||||
.ok()?
|
||||
.pointer("/modalities/vision")
|
||||
.and_then(Value::as_bool)
|
||||
}
|
||||
|
||||
/// What the loaded model's chat template takes for thinking, asked of it.
|
||||
///
|
||||
/// Two different questions, because they are two different template arguments.
|
||||
@@ -2031,6 +2321,21 @@ mod tests {
|
||||
use super::*;
|
||||
use crate::session::transcript::Transcript;
|
||||
|
||||
impl Content {
|
||||
/// What a message says, for the assertions that are about the words.
|
||||
/// A message carrying images keeps them beside its text, so this is
|
||||
/// the text part among them.
|
||||
fn words(&self) -> &str {
|
||||
match self {
|
||||
Content::Text(text) => text,
|
||||
Content::Parts(parts) => parts
|
||||
.iter()
|
||||
.find_map(|part| part.get("text").and_then(Value::as_str))
|
||||
.unwrap_or_default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Writes a transcript the way the pump does, so the fold is tested against
|
||||
/// the real file format rather than a hand-built vector.
|
||||
fn transcript_with(events: &[Event]) -> (tempfile::TempDir, PathBuf) {
|
||||
@@ -2069,11 +2374,11 @@ mod tests {
|
||||
delta: "yes".into(),
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(
|
||||
messages
|
||||
.iter()
|
||||
.map(|m| (m.role.as_str(), m.content.as_str()))
|
||||
.map(|m| (m.role.as_str(), m.content.words()))
|
||||
.collect::<Vec<_>>(),
|
||||
[
|
||||
("user", "hello"),
|
||||
@@ -2121,9 +2426,9 @@ mod tests {
|
||||
delta: "one two".into(),
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(messages.len(), 2);
|
||||
assert_eq!(messages[1].content, "one two");
|
||||
assert_eq!(messages[1].content.words(), "one two");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -2159,11 +2464,11 @@ mod tests {
|
||||
delta: "right".into(),
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(
|
||||
messages
|
||||
.iter()
|
||||
.map(|m| (m.role.as_str(), m.content.as_str()))
|
||||
.map(|m| (m.role.as_str(), m.content.words()))
|
||||
.collect::<Vec<_>>(),
|
||||
[
|
||||
("user", "read it"),
|
||||
@@ -2194,9 +2499,9 @@ mod tests {
|
||||
state: SessionStatus::Idle,
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(messages.len(), 2);
|
||||
assert_eq!(messages[1].content, "one two");
|
||||
assert_eq!(messages[1].content.words(), "one two");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -2226,10 +2531,10 @@ mod tests {
|
||||
prefill_ms: None,
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(messages.len(), 2);
|
||||
assert_eq!(messages[0].content, "hello");
|
||||
assert_eq!(messages[1].content, "still here");
|
||||
assert_eq!(messages[0].content.words(), "hello");
|
||||
assert_eq!(messages[1].content.words(), "still here");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -2256,10 +2561,10 @@ mod tests {
|
||||
delta: "cheaply".into(),
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(messages.len(), 2);
|
||||
assert_eq!(messages[0].content, "a fresh start");
|
||||
assert_eq!(messages[1].content, "cheaply");
|
||||
assert_eq!(messages[0].content.words(), "a fresh start");
|
||||
assert_eq!(messages[1].content.words(), "cheaply");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -2285,9 +2590,9 @@ mod tests {
|
||||
attachments: Vec::new(),
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(messages.len(), 1);
|
||||
assert_eq!(messages[0].content, "three");
|
||||
assert_eq!(messages[0].content.words(), "three");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -2319,10 +2624,10 @@ mod tests {
|
||||
delta: "It says alpha and beta.".into(),
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
let shape: Vec<(&str, &str)> = messages
|
||||
.iter()
|
||||
.map(|m| (m.role.as_str(), m.content.as_str()))
|
||||
.map(|m| (m.role.as_str(), m.content.words()))
|
||||
.collect();
|
||||
assert_eq!(
|
||||
shape,
|
||||
@@ -2371,12 +2676,12 @@ mod tests {
|
||||
output: "first".into(),
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(messages[1].tool_calls.len(), 2);
|
||||
assert_eq!(
|
||||
messages[2..]
|
||||
.iter()
|
||||
.map(|m| (m.tool_call_id.as_deref(), m.content.as_str()))
|
||||
.map(|m| (m.tool_call_id.as_deref(), m.content.words()))
|
||||
.collect::<Vec<_>>(),
|
||||
[(Some("a"), "first"), (Some("b"), "second")],
|
||||
);
|
||||
@@ -2404,10 +2709,10 @@ mod tests {
|
||||
state: SessionStatus::Idle,
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(messages[1].tool_calls.len(), 1);
|
||||
assert_eq!(messages[2].role, "tool");
|
||||
assert_eq!(messages[2].content, tools::UNFINISHED);
|
||||
assert_eq!(messages[2].content.words(), tools::UNFINISHED);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -2437,10 +2742,10 @@ mod tests {
|
||||
delta: "it is linux".into(),
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
assert_eq!(messages[1].content, "checking");
|
||||
let messages = conversation(&path, None);
|
||||
assert_eq!(messages[1].content.words(), "checking");
|
||||
assert_eq!(messages[1].tool_calls.len(), 1);
|
||||
assert_eq!(messages[3].content, "it is linux");
|
||||
assert_eq!(messages[3].content.words(), "it is linux");
|
||||
assert!(messages[3].tool_calls.is_empty());
|
||||
}
|
||||
|
||||
@@ -2535,4 +2840,94 @@ mod tests {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// An image sent to a model that reads them goes into the message as a
|
||||
/// part beside the words, and one sent to a model that does not is said
|
||||
/// in words -- never dropped, which would leave a question about a picture
|
||||
/// reading as a question about nothing.
|
||||
#[test]
|
||||
fn a_picture_is_a_part_for_a_model_that_reads_them_and_a_line_for_one_that_does_not() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
std::fs::create_dir(dir.path().join("attachments")).expect("attachments");
|
||||
std::fs::write(dir.path().join("attachments").join("ab12.png"), b"PNG")
|
||||
.expect("write image");
|
||||
let refs = ["ab12.png".to_string()];
|
||||
|
||||
let seen = sent(Some(dir.path()), "what is this?", &refs);
|
||||
assert_eq!(
|
||||
seen.content,
|
||||
Content::Parts(vec![
|
||||
json!({
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "data:image/png;base64,UE5H"},
|
||||
}),
|
||||
json!({"type": "text", "text": "what is this?"}),
|
||||
]),
|
||||
);
|
||||
|
||||
let unseen = sent(None, "what is this?", &refs);
|
||||
assert_eq!(
|
||||
unseen.content,
|
||||
Content::Text(
|
||||
"what is this?\n\n[1 attachment sent with this message, which you cannot \
|
||||
read.]"
|
||||
.to_string()
|
||||
),
|
||||
);
|
||||
|
||||
// Nothing attached is an ordinary message either way, and in
|
||||
// particular not a one-part list: that renders differently through
|
||||
// some chat templates, and every turn in this file is the other shape.
|
||||
assert_eq!(
|
||||
sent(Some(dir.path()), "plain", &[]).content,
|
||||
Content::Text("plain".to_string()),
|
||||
);
|
||||
}
|
||||
|
||||
/// A file that cannot be read is left out rather than failing the turn:
|
||||
/// the send path has already said so, and a turn is a worse place to learn
|
||||
/// it.
|
||||
#[test]
|
||||
fn an_unreadable_attachment_is_not_a_picture() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
assert!(image_parts(dir.path(), &["ab12.png".to_string()]).is_empty());
|
||||
// Not an image at all, and an id this server would never have written.
|
||||
assert!(image_parts(dir.path(), &["ab12-trace.txt".to_string()]).is_empty());
|
||||
assert!(image_parts(dir.path(), &["../../etc/passwd.png".to_string()]).is_empty());
|
||||
}
|
||||
|
||||
/// Both conventions repositories publish a projector under, and nothing
|
||||
/// else -- a model file that merely sits beside one is not one.
|
||||
#[test]
|
||||
fn a_projector_is_known_by_its_name() {
|
||||
assert!(is_projector("mmproj-SmolVLM-256M-Instruct-Q8_0.gguf"));
|
||||
assert!(is_projector("Qwen2.5-VL-7B-mmproj-F16.GGUF"));
|
||||
assert!(!is_projector("SmolVLM-256M-Instruct-Q8_0.gguf"));
|
||||
assert!(!is_projector("mmproj-notes.txt"));
|
||||
}
|
||||
|
||||
/// The projector found beside a model is the one it loads with, and the
|
||||
/// smaller of two sorts first.
|
||||
#[test]
|
||||
fn the_projector_beside_a_model_is_found() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
for name in [
|
||||
"SmolVLM-256M-Instruct-Q8_0.gguf",
|
||||
"mmproj-F32.gguf",
|
||||
"mmproj-F16.gguf",
|
||||
] {
|
||||
std::fs::write(dir.path().join(name), b"").expect("write");
|
||||
}
|
||||
assert_eq!(
|
||||
projector_in(dir.path()),
|
||||
Some(
|
||||
dir.path()
|
||||
.join("mmproj-F16.gguf")
|
||||
.to_string_lossy()
|
||||
.into_owned()
|
||||
),
|
||||
);
|
||||
let empty = tempfile::tempdir().expect("tempdir");
|
||||
assert_eq!(projector_in(empty.path()), None);
|
||||
}
|
||||
}
|
||||
@@ -707,9 +707,37 @@ fn section(found: &Model, settings: &BTreeMap<String, String>) -> String {
|
||||
if found.mtp && settings.get("speculative").map(String::as_str) != Some("off") {
|
||||
lines.push("spec-type = draft-mtp".to_string());
|
||||
}
|
||||
// What makes a model able to read pictures, and the one flag here that is
|
||||
// found rather than defaulted: the projector is a second file published
|
||||
// beside the weights, so a model that has one is loaded with it unless
|
||||
// this model's settings name another or turn it off.
|
||||
if let Some(projector) = projector(found, settings) {
|
||||
lines.push(format!("mmproj = {projector}"));
|
||||
}
|
||||
lines.join("\n")
|
||||
}
|
||||
|
||||
/// Which projector this model is loaded with: what its settings say, else
|
||||
/// whatever was found beside it, and nothing for `"off"`.
|
||||
///
|
||||
/// A setting naming a bare file name means one in the model's own directory,
|
||||
/// since that is where the alternatives to the file found there are; anything
|
||||
/// with a `/` in it is taken as the path it is, absolute or not -- the serving
|
||||
/// machine resolves it, and this side does not know its working directory.
|
||||
fn projector(found: &Model, settings: &BTreeMap<String, String>) -> Option<String> {
|
||||
let chosen = settings.get("mmproj").map(|value| value.trim());
|
||||
match chosen {
|
||||
Some("off") => None,
|
||||
Some("") => found.mmproj.clone(),
|
||||
Some(name) if name.contains('/') => Some(name.to_string()),
|
||||
Some(name) => {
|
||||
let dir = found.path.rsplit_once('/').map_or("", |(dir, _)| dir);
|
||||
Some(format!("{dir}/{name}"))
|
||||
}
|
||||
None => found.mmproj.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The preset file with `name`'s section replaced by `body`, added at the end
|
||||
/// if it was not there.
|
||||
///
|
||||
@@ -782,6 +810,7 @@ mod tests {
|
||||
#[test]
|
||||
fn a_section_names_the_file_and_the_flags_that_were_set() {
|
||||
let found = Model {
|
||||
mmproj: None,
|
||||
path: "/models/a.gguf".to_string(),
|
||||
mtp: true,
|
||||
};
|
||||
@@ -810,6 +839,50 @@ mod tests {
|
||||
assert!(!section(&found, &settings(&[("speculative", "off")])).contains("spec-type"));
|
||||
}
|
||||
|
||||
/// The projector found beside a model is loaded with it; the setting names
|
||||
/// another where a repository published several, or turns it off.
|
||||
#[test]
|
||||
fn a_vision_model_is_loaded_with_its_projector() {
|
||||
let found = Model {
|
||||
path: "/models/repo/a.gguf".to_string(),
|
||||
mtp: false,
|
||||
mmproj: Some("/models/repo/mmproj-F16.gguf".to_string()),
|
||||
};
|
||||
let line = |settings: &[(&str, &str)]| {
|
||||
section(&found, &self::settings(settings))
|
||||
.lines()
|
||||
.find_map(|line| line.strip_prefix("mmproj = "))
|
||||
.map(str::to_string)
|
||||
};
|
||||
assert_eq!(line(&[]), Some("/models/repo/mmproj-F16.gguf".to_string()));
|
||||
assert_eq!(
|
||||
line(&[("mmproj", " ")]),
|
||||
Some("/models/repo/mmproj-F16.gguf".to_string())
|
||||
);
|
||||
assert_eq!(line(&[("mmproj", "off")]), None);
|
||||
// A bare name is one of the model's own neighbours; anything with a
|
||||
// separator in it is the path it says it is.
|
||||
assert_eq!(
|
||||
line(&[("mmproj", "mmproj-F32.gguf")]),
|
||||
Some("/models/repo/mmproj-F32.gguf".to_string()),
|
||||
);
|
||||
assert_eq!(
|
||||
line(&[("mmproj", "/elsewhere/p.gguf")]),
|
||||
Some("/elsewhere/p.gguf".to_string()),
|
||||
);
|
||||
// A model with none, and nothing asked for, loads without one.
|
||||
assert!(
|
||||
!section(
|
||||
&Model {
|
||||
mmproj: None,
|
||||
..found.clone()
|
||||
},
|
||||
&settings(&[])
|
||||
)
|
||||
.contains("mmproj")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_section_replaces_its_own_and_leaves_every_other_line_alone() {
|
||||
let first = upsert("", "repo/a.gguf", "model = /models/a.gguf");
|
||||
|
||||
@@ -36,7 +36,7 @@ use crate::config::{
|
||||
use claude::ClaudeDriver;
|
||||
use codex::CodexDriver;
|
||||
use driver::{
|
||||
AttachmentRef, BackgroundTask, Driver, Event, EventSink, SessionCommand, SessionStatus,
|
||||
AttachmentRef, BackgroundTask, Driver, Event, EventSink, Images, SessionCommand, SessionStatus,
|
||||
Unqueued, context_after, context_limit_after,
|
||||
};
|
||||
use echo::EchoDriver;
|
||||
@@ -235,6 +235,10 @@ pub struct SessionInfo {
|
||||
/// that happens to be big" are different answers.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub max_image_edge: Option<u32>,
|
||||
/// Whether an image can be sent here at all -- see [`Images`]. Reported
|
||||
/// so the phone can withhold the control rather than take a photo,
|
||||
/// shrink it, upload it over the tunnel and then be told.
|
||||
pub images: Images,
|
||||
/// Which of `GET /usage`'s snapshots reports on this session, and
|
||||
/// absent where nothing meters it -- see
|
||||
/// [`DriverKind::usage_provider`].
|
||||
@@ -610,6 +614,13 @@ impl LiveSession {
|
||||
auto_resume_message: resume_message(current),
|
||||
resume_at: current.resume.map(|scheduled| scheduled.at),
|
||||
max_image_edge: kind.and_then(DriverKind::max_image_edge),
|
||||
// The driver's answer where it has one -- llama.cpp's depends on
|
||||
// the model it is on now, which is not the provider's to say.
|
||||
images: self
|
||||
.driver()
|
||||
.and_then(|driver| driver.images())
|
||||
.or_else(|| kind.map(DriverKind::images))
|
||||
.unwrap_or(Images::Unknown),
|
||||
usage_provider: kind.and_then(DriverKind::usage_provider),
|
||||
imported,
|
||||
keeps_own_transcript: kind.is_some_and(DriverKind::keeps_own_transcript),
|
||||
@@ -1156,6 +1167,10 @@ impl SessionManager {
|
||||
params: meta.params.clone(),
|
||||
max_image_edge: kind_of(&inner.config, &meta.machine, &meta.provider)
|
||||
.and_then(DriverKind::max_image_edge),
|
||||
// No process, so nothing but the kind can answer; for
|
||||
// llama.cpp that is deliberately "not known yet".
|
||||
images: kind_of(&inner.config, &meta.machine, &meta.provider)
|
||||
.map_or(Images::Unknown, DriverKind::images),
|
||||
usage_provider: kind_of(&inner.config, &meta.machine, &meta.provider)
|
||||
.and_then(DriverKind::usage_provider),
|
||||
notify: meta.notify,
|
||||
|
||||
Reference in new issue
Block a user