Let a llama session be shown a picture where the model reads one
A multimodal model is loaded with the `mmproj` found beside its weights -- which is how a repository publishes the pair -- and an attached image rides in the request as an `image_url` data URI, so it reaches a model on another machine without the file going there. Nothing is done for a model without a projector: no captioning, no OCR, no second model. Whether a session takes pictures is measured rather than assumed: `/props`'s `modalities.vision` from the server that loaded the model, in three states, because a model still coming off disk has genuinely not said. Unknown is offered rather than refused -- a control withheld because nobody could ask goes missing from sessions that would have taken it. The answer reaches the phone twice per model as `Event::Images`, so the photo button is withdrawn the moment a model with vision is left rather than at whatever later point the session row is fetched again. A message carrying an image a model cannot read is stopped rather than stripped: `llama-server` refuses the whole request over one image part, and a message sent without its picture would be answered as though the picture had never been mentioned. The phone will not attach one, and the driver refuses it again at the three moments the answer can first exist -- at the door, when a message queued behind a loading model is read, and at the tool boundary a steer enters by. An earlier turn's image folds into a line of words for a model without vision, so switching a conversation onto one does not end it. A projector is filtered out of the models a provider *offers*, since a session started on one is a server that cannot load it; it stays in the machine's own model list, where a file on a disk is managed. Verified against ggml-org/SmolVLM-256M-Instruct-GGUF, local and over ssh: "In this picture there is a red circle." Switching that session to Qwen3-0.6B reports `refused`, refuses the next picture with the reason, and still answers an ordinary message.
This commit is contained in:
1 parent
b7fd18b195
commit
bd9596d782
17 files changed
+835
-97
No files matched your search
@@ -219,6 +219,15 @@ data class SessionSummary(
|
||||
* server because that is where a provider's kind is known.
|
||||
*/
|
||||
val maxImageEdge: Int?,
|
||||
/**
|
||||
* Whether a picture can be sent here at all.
|
||||
*
|
||||
* A snapshot, like the rest of this row: the live answer arrives as [SessionEvent.Images],
|
||||
* which is what a session screen watches. Asked of the server rather than worked out here
|
||||
* because for a local model it is a property of what is loaded on the serving machine, which
|
||||
* this app cannot see.
|
||||
*/
|
||||
val images: ImageSupport,
|
||||
/**
|
||||
* Which of `GET /usage`'s snapshots is about this session, and null where nothing meters it.
|
||||
*
|
||||
@@ -263,6 +272,7 @@ private fun parseSession(session: JSONObject) =
|
||||
contextLimit = if (session.has("contextLimit")) session.getLong("contextLimit") else null,
|
||||
params = session.optJSONObject("params").stringMap(),
|
||||
maxImageEdge = session.optInt("maxImageEdge", 0).takeIf { it > 0 },
|
||||
images = imageSupport(session.optString("images")),
|
||||
usageProvider = session.optString("usageProvider").ifEmpty { null },
|
||||
status = session.getString("status"),
|
||||
lastActivity = session.getDouble("lastActivity"),
|
||||
|
||||
@@ -11,6 +11,30 @@ import androidx.compose.ui.text.font.FontFamily
|
||||
import androidx.compose.ui.text.style.TextOverflow
|
||||
import androidx.compose.ui.unit.dp
|
||||
|
||||
/**
|
||||
* Whether a session can be sent a picture, as the server answers it.
|
||||
*
|
||||
* Three states rather than a switch, because for a local model the answer belongs to the server
|
||||
* that loaded it: one still coming off disk genuinely has not said. [UNKNOWN] is offered -- a
|
||||
* control withheld because nobody could ask is a photo button missing from a session that would
|
||||
* have read the photo perfectly well, and the send path says so if the guess was wrong.
|
||||
*/
|
||||
enum class ImageSupport {
|
||||
ACCEPTED,
|
||||
REFUSED,
|
||||
UNKNOWN,
|
||||
}
|
||||
|
||||
/**
|
||||
* What the server called it; anything else -- an older server, a newer word -- is not an answer.
|
||||
*/
|
||||
fun imageSupport(word: String): ImageSupport =
|
||||
when (word) {
|
||||
"accepted" -> ImageSupport.ACCEPTED
|
||||
"refused" -> ImageSupport.REFUSED
|
||||
else -> ImageSupport.UNKNOWN
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether [ref] names an image the server stored as one -- `<hex>.<extension>`, with an extension
|
||||
* from the list it writes -- rather than a file kept under its own name. Mirrors the server's
|
||||
|
||||
@@ -38,6 +38,11 @@ suspend fun uploadPickedImage(
|
||||
* Uploads whatever [uri] names, the way its kind needs. An image goes through [uploadPickedImage]
|
||||
* and is shrunk; anything else goes whole, under the name the other app or the file chooser gave
|
||||
* it, because the session is told that name rather than shown the bytes.
|
||||
*
|
||||
* A picture is refused here, before anything is read or sent, when [images] says this session's
|
||||
* model cannot read one. Here rather than beside the photo button because this is where every way
|
||||
* of attaching meets: the picker, the file chooser, and another app's share sheet -- and only the
|
||||
* first of those has a button to disable.
|
||||
*/
|
||||
suspend fun uploadPicked(
|
||||
context: Context,
|
||||
@@ -45,10 +50,16 @@ suspend fun uploadPicked(
|
||||
sessionId: String,
|
||||
uri: Uri,
|
||||
maxEdge: Int?,
|
||||
images: ImageSupport,
|
||||
): String {
|
||||
val resolver = context.contentResolver
|
||||
val mime = resolver.getType(uri)
|
||||
if (mime != null && mime.startsWith("image/")) {
|
||||
if (images == ImageSupport.REFUSED) {
|
||||
throw ApiException(
|
||||
"this session's model can't read pictures, so that one wasn't attached"
|
||||
)
|
||||
}
|
||||
return uploadPickedImage(context, settings, sessionId, uri, maxEdge)
|
||||
}
|
||||
// Opened before the request starts, so a provider that refuses says so here and not from inside
|
||||
|
||||
@@ -168,6 +168,16 @@ sealed class SessionEvent {
|
||||
*/
|
||||
data class Settings(val model: String?, val permissionMode: String?) : SessionEvent()
|
||||
|
||||
/**
|
||||
* Whether a picture can be sent to this session now, as the thing serving its model answered.
|
||||
*
|
||||
* Only a llama.cpp session says this, and it says it twice per model: unknown the moment the
|
||||
* old one is left, then the loaded server's answer. It carries no row -- it is what the
|
||||
* composer's photo button is drawn from, and a line in the transcript about a control is not
|
||||
* something anybody asked after.
|
||||
*/
|
||||
data class Images(val images: ImageSupport) : SessionEvent()
|
||||
|
||||
/**
|
||||
* What a turn cost, and how much the model was holding when it ended.
|
||||
*
|
||||
@@ -328,6 +338,7 @@ fun parseSeqEvent(json: String): SeqEvent {
|
||||
model = body.optString("model").ifEmpty { null },
|
||||
permissionMode = body.optString("permissionMode").ifEmpty { null },
|
||||
)
|
||||
"images" -> SessionEvent.Images(imageSupport(body.optString("images")))
|
||||
"contextWindow" -> SessionEvent.ContextWindow(body.getLong("tokens"))
|
||||
"usageDelta" ->
|
||||
SessionEvent.UsageDelta(
|
||||
|
||||
@@ -306,6 +306,16 @@ fun SessionScreen(
|
||||
// reads as a rename that did not take.
|
||||
var title by remember(summary.id) { mutableStateOf(summary.title) }
|
||||
var model by remember { mutableStateOf(summary.model) }
|
||||
// Whether this session takes pictures. Held here rather than read from the row that opened the
|
||||
// screen for the reason [model] is, and it moves with the model: a llama.cpp session changed
|
||||
// onto a model without vision stops offering the photo button at that moment, not at whatever
|
||||
// later point the session list is fetched again.
|
||||
var images by remember { mutableStateOf(summary.images) }
|
||||
// Pictures attached to a message this session can no longer be sent -- the model was changed
|
||||
// under them. The message is stopped here as well as at the server, which refuses it whichever
|
||||
// device sent it: a picture stripped out on the way would be answered as though it had never
|
||||
// been mentioned.
|
||||
val strandedImages = images == ImageSupport.REFUSED && pendingAttachments.any(::isImageRef)
|
||||
// The thinking level, held here for the same reason [title] is: the settings dialog can change
|
||||
// it, and the row this screen was opened from is a snapshot taken before that. Read straight
|
||||
// from the summary, a level set here was drawn as the old one again the next time the dialog
|
||||
@@ -517,6 +527,7 @@ fun SessionScreen(
|
||||
event.model?.let { model = it }
|
||||
event.permissionMode?.let { permissionMode = it }
|
||||
}
|
||||
if (event is SessionEvent.Images) images = event.images
|
||||
if (event is SessionEvent.Status) {
|
||||
// The event's own timestamp, so a compaction that began before this screen
|
||||
// opened is timed from when it actually began -- and a compaction worth asking
|
||||
@@ -853,6 +864,7 @@ fun SessionScreen(
|
||||
if (!isSubagent) {
|
||||
model = summary.model
|
||||
permissionMode = summary.permissionMode ?: "auto"
|
||||
images = summary.images
|
||||
}
|
||||
if (status != "compacting") compactingSince = null
|
||||
// Nothing to put back, so these rows are the screen and the probe can return under
|
||||
@@ -1397,7 +1409,14 @@ fun SessionScreen(
|
||||
// An image is shrunk to what this session's provider takes before it is
|
||||
// uploaded, so a twelve-megapixel photo does not cross the tunnel to be
|
||||
// rejected at the far end; a file goes whole.
|
||||
uploadPicked(context, settings, summary.id, uri, summary.maxImageEdge)
|
||||
uploadPicked(
|
||||
context,
|
||||
settings,
|
||||
summary.id,
|
||||
uri,
|
||||
summary.maxImageEdge,
|
||||
images,
|
||||
)
|
||||
}
|
||||
pendingAttachments = pendingAttachments + id
|
||||
actionError = null
|
||||
@@ -2058,6 +2077,18 @@ fun SessionScreen(
|
||||
refs = pendingAttachments,
|
||||
onRemove = { pendingAttachments = pendingAttachments - it },
|
||||
)
|
||||
// Only reachable by the model changing under something already attached, since
|
||||
// attaching is refused where the answer is already known. Said beside the
|
||||
// pictures it is about and above the button it disables, which is where
|
||||
// somebody can act on it: removing them is what makes this sendable.
|
||||
if (strandedImages) {
|
||||
Text(
|
||||
"This model can't read pictures. Remove them to send this.",
|
||||
color = MaterialTheme.colorScheme.error,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
modifier = Modifier.padding(bottom = 4.dp),
|
||||
)
|
||||
}
|
||||
OutlinedTextField(
|
||||
value = input,
|
||||
onValueChange = {
|
||||
@@ -2091,8 +2122,23 @@ fun SessionScreen(
|
||||
properties = PopupProperties(clippingEnabled = false),
|
||||
shape = BubbleMenuShape,
|
||||
) {
|
||||
// Disabled rather than dropped: an item that comes and goes
|
||||
// makes its own absence the message, and the reason belongs on
|
||||
// the item, where somebody wondering why is looking.
|
||||
val takesPictures = images != ImageSupport.REFUSED
|
||||
DropdownMenuItem(
|
||||
text = { Text("Photo") },
|
||||
text = {
|
||||
Column {
|
||||
Text("Photo")
|
||||
if (!takesPictures) {
|
||||
Text(
|
||||
"this model can't read them",
|
||||
style = MaterialTheme.typography.labelSmall,
|
||||
)
|
||||
}
|
||||
}
|
||||
},
|
||||
enabled = takesPictures,
|
||||
onClick = {
|
||||
attaching = false
|
||||
pickImage.launch(
|
||||
@@ -2200,7 +2246,9 @@ fun SessionScreen(
|
||||
// reason above.
|
||||
CircleButton(
|
||||
onClick = { send() },
|
||||
enabled = input.text.isNotBlank() || pendingAttachments.isNotEmpty(),
|
||||
enabled =
|
||||
(input.text.isNotBlank() || pendingAttachments.isNotEmpty()) &&
|
||||
!strandedImages,
|
||||
fill = if (running) queueColor else sendColor,
|
||||
) {
|
||||
Glyph(
|
||||
|
||||
@@ -600,6 +600,9 @@ fun foldEvent(items: List<TranscriptItem>, entry: SeqEvent): List<TranscriptItem
|
||||
// nothing it belongs above.
|
||||
is SessionEvent.MessageDropped -> items
|
||||
is SessionEvent.Settings -> items
|
||||
// Neither carries a row: both are about what the session can do rather than about anything
|
||||
// said in it, and the composer is where they are drawn.
|
||||
is SessionEvent.Images -> items
|
||||
is SessionEvent.BackgroundTasks -> items
|
||||
is SessionEvent.Status -> settleReply(items, event.state)
|
||||
is SessionEvent.AuthenticationRequired ->
|
||||
|
||||
Reference in new issue
Block a user