iris-ai
bd9596d782
A multimodal model is loaded with the `mmproj` found beside its weights -- which is how a repository publishes the pair -- and an attached image rides in the request as an `image_url` data URI, so it reaches a model on another machine without the file going there. Nothing is done for a model without a projector: no captioning, no OCR, no second model. Whether a session takes pictures is measured rather than assumed: `/props`'s `modalities.vision` from the server that loaded the model, in three states, because a model still coming off disk has genuinely not said. Unknown is offered rather than refused -- a control withheld because nobody could ask goes missing from sessions that would have taken it. The answer reaches the phone twice per model as `Event::Images`, so the photo button is withdrawn the moment a model with vision is left rather than at whatever later point the session row is fetched again. A message carrying an image a model cannot read is stopped rather than stripped: `llama-server` refuses the whole request over one image part, and a message sent without its picture would be answered as though the picture had never been mentioned. The phone will not attach one, and the driver refuses it again at the three moments the answer can first exist -- at the door, when a message queued behind a loading model is read, and at the tool boundary a steer enters by. An earlier turn's image folds into a line of words for a model without vision, so switching a conversation onto one does not end it. A projector is filtered out of the models a provider *offers*, since a session started on one is a server that cannot load it; it stays in the machine's own model list, where a file on a disk is managed. Verified against ggml-org/SmolVLM-256M-Instruct-GGUF, local and over ssh: "In this picture there is a red circle." Switching that session to Qwen3-0.6B reports `refused`, refuses the next picture with the reason, and still answers an ordinary message.
430 lines
20 KiB
Kotlin
430 lines
20 KiB
Kotlin
package com.example.aiapp
|
|
|
|
import org.json.JSONObject
|
|
|
|
// The common event model, mirrored from server/src/session/driver.rs -- the app renders purely from
|
|
// this stream (replayed from the transcript by cursor, then live), so there is no separate "load
|
|
// history" shape to keep in sync with it.
|
|
|
|
/** One transcript line: the event plus its resume cursor and time. */
|
|
data class SeqEvent(val seq: Long, val ts: Double, val event: SessionEvent)
|
|
|
|
/**
|
|
* One choice offered in answer to a question.
|
|
*
|
|
* More than a label because the reader is deciding rather than confirming. Both are absent on a
|
|
* permission, whose Allow and Deny mean exactly what they say.
|
|
*/
|
|
data class QuestionOption(val label: String, val description: String?, val preview: String?)
|
|
|
|
sealed class SessionEvent {
|
|
data class UserMessage(
|
|
val text: String,
|
|
/**
|
|
* The [MessageQueued] this resolves, or null when it never waited.
|
|
*
|
|
* Matched on rather than the text, because the same message sent twice is two waiting
|
|
* bubbles and clearing whichever one matched first would leave the wrong one on screen.
|
|
*/
|
|
val id: String?,
|
|
/**
|
|
* What was attached to it, by the ref the files route serves: images, and any file, told
|
|
* apart by [isImageRef].
|
|
*
|
|
* On the message rather than beside it: these arrived as separate image events until
|
|
* 2026-08-30, which drew somebody's screenshot as a row floating above the bubble that sent
|
|
* it, and left this app deciding from adjacency which message an image went with.
|
|
*/
|
|
val attachments: List<String>,
|
|
) : SessionEvent()
|
|
|
|
/**
|
|
* A message the server has accepted and the session has not read yet.
|
|
*
|
|
* From the server, not from this app's memory of what it sent. The pending bubble used to be
|
|
* screen state, so leaving the session drew nothing waiting while the message was still queued
|
|
* -- and nothing waiting is what "there is nothing" looks like.
|
|
*
|
|
* Resolved by the [UserMessage] carrying the same id.
|
|
*/
|
|
data class MessageQueued(val id: String, val text: String, val attachments: List<String>) :
|
|
SessionEvent()
|
|
|
|
/**
|
|
* A queued message taken back before the session read it.
|
|
*
|
|
* Recorded by the server for the same reason [MessageQueued] is: a phone that reconnects
|
|
* replays both, and without this one it would put back a bubble for a message that is never
|
|
* coming.
|
|
*/
|
|
data class MessageDropped(val id: String) : SessionEvent()
|
|
|
|
data class AssistantText(val delta: String) : SessionEvent()
|
|
|
|
/** The durable value of the open assistant message, replacing its provisional deltas. */
|
|
data class AssistantTextFinal(val text: String) : SessionEvent()
|
|
|
|
/**
|
|
* The model's working, streamed the way its reply is: its own card, and deliberately not part
|
|
* of what the session said. Only a provider that actually streams its reasoning sends it.
|
|
*/
|
|
data class Thinking(val delta: String) : SessionEvent()
|
|
|
|
/**
|
|
* The thinking above this finished, having taken [ms].
|
|
*
|
|
* Measured by the driver, because only it can see when the model stopped: this app knows when
|
|
* an event *arrived*, and the last fragment of a block followed by a slow tool call looks
|
|
* exactly like thinking that went on that long.
|
|
*/
|
|
data class ThinkingDone(val ms: Long) : SessionEvent()
|
|
|
|
data class ToolStart(val id: String, val tool: String, val input: String) : SessionEvent()
|
|
|
|
data class ToolUpdate(val id: String, val output: String) : SessionEvent()
|
|
|
|
data class ToolEnd(val id: String, val output: String) : SessionEvent()
|
|
|
|
data class Image(
|
|
val ref: String,
|
|
/** The tool call whose result carried it, or null for a person's own attachment. */
|
|
val about: String?,
|
|
) : SessionEvent()
|
|
|
|
data class Question(
|
|
val id: String,
|
|
val prompt: String,
|
|
/** A few words naming what the question is about, when the asker offered one. */
|
|
val header: String?,
|
|
val options: List<QuestionOption>,
|
|
/** Whether several options may be chosen at once. */
|
|
val multiSelect: Boolean,
|
|
/** The tool call this is permission for, or null when it is not about one. */
|
|
val about: String?,
|
|
) : SessionEvent()
|
|
|
|
/** Everything chosen for one question, in the order it was offered. */
|
|
data class Answered(val id: String, val answers: List<String>) : SessionEvent()
|
|
|
|
/**
|
|
* A message another agent sent this session.
|
|
*
|
|
* Not a [UserMessage]: nobody holding the phone said it, and drawing it in their voice would
|
|
* claim they had. It is also the explanation for a session that starts working on something
|
|
* this device never asked for.
|
|
*/
|
|
data class PeerMessage(
|
|
val from: String,
|
|
val text: String,
|
|
/**
|
|
* Where the turn this started begins, when the server could say.
|
|
*
|
|
* The live Claude Code path only learns a turn was somebody else's when the turn ends, so
|
|
* the event arrives below everything it caused; this is what puts it back above it. Null
|
|
* for a message read out of a session file, and for one that started no turn.
|
|
*/
|
|
val turnStart: Long? = null,
|
|
) : SessionEvent()
|
|
|
|
/**
|
|
* A line in the transcript this build cannot read: a kind a newer server wrote, or one an older
|
|
* server wrote that has since been dropped.
|
|
*
|
|
* [kind] is the word the line called itself, so the row can say what is missing rather than
|
|
* that something is. The server makes these when reading; no driver sends one.
|
|
*/
|
|
data class Unreadable(val kind: String) : SessionEvent()
|
|
|
|
/**
|
|
* Retired on 2026-09-06, hours after it was added: a background task finishing, which turned
|
|
* out to be a screenful of notices about work nobody was asking after.
|
|
*
|
|
* Kept because a transcript is append-only -- the sessions that ran a background task in that
|
|
* window have these lines for ever. It draws no row, which is the whole reason it is still
|
|
* named here rather than left to fall through to [Unknown]: that would draw a placeholder per
|
|
* background task, which is the same wall the row was removed for.
|
|
*/
|
|
object RetiredTaskNote : SessionEvent()
|
|
|
|
/**
|
|
* A command the session was asked to run on itself and cannot run yet. Resolved by
|
|
* [CommandSent] with the same id; a command that ran straight away has only that one.
|
|
*/
|
|
data class CommandQueued(val id: String, val text: String) : SessionEvent()
|
|
|
|
/** The same command, handed to the session. */
|
|
data class CommandSent(val id: String, val text: String) : SessionEvent()
|
|
|
|
/** Provider-reported number of background tasks alive now. */
|
|
data class BackgroundTasks(val count: Int) : SessionEvent()
|
|
|
|
data class Status(val state: String) : SessionEvent()
|
|
|
|
/**
|
|
* What the session is set to, as the session itself reports it.
|
|
*
|
|
* Either field alone: the two are confirmed separately. Asking for a change is not having one,
|
|
* so this -- not the request -- is what the pickers show.
|
|
*/
|
|
data class Settings(val model: String?, val permissionMode: String?) : SessionEvent()
|
|
|
|
/**
|
|
* Whether a picture can be sent to this session now, as the thing serving its model answered.
|
|
*
|
|
* Only a llama.cpp session says this, and it says it twice per model: unknown the moment the
|
|
* old one is left, then the loaded server's answer. It carries no row -- it is what the
|
|
* composer's photo button is drawn from, and a line in the transcript about a control is not
|
|
* something anybody asked after.
|
|
*/
|
|
data class Images(val images: ImageSupport) : SessionEvent()
|
|
|
|
/**
|
|
* What a turn cost, and how much the model was holding when it ended.
|
|
*
|
|
* [context] is prompt plus both cache figures. Carried on the event rather than summed by the
|
|
* reader, because it is not a sum: a conversation's context drops at a compaction and a clear,
|
|
* so adding turns up would report a figure the session stopped being true of. Null where the
|
|
* dialect did not say, which leaves the context unmeasured rather than unchanged.
|
|
*/
|
|
data class UsageDelta(
|
|
val tokens: Long,
|
|
val context: Long?,
|
|
/**
|
|
* How fast the reply came out, where the provider measured it -- null everywhere else,
|
|
* which is most of them. Never worked out here: the time this app watched a reply arrive
|
|
* over includes the network and whatever the server was doing between tokens.
|
|
*/
|
|
val tokensPerSecond: Double? = null,
|
|
/**
|
|
* How long the provider spent reading the prompt before it began answering; null where
|
|
* nothing measured it. The same rule as [tokensPerSecond]: the provider's own figure, or
|
|
* nothing at all.
|
|
*/
|
|
val prefillMs: Long? = null,
|
|
) : SessionEvent()
|
|
|
|
/** How much context this session's model has, which is what [UsageDelta.context] is out of. */
|
|
data class ContextWindow(val tokens: Long) : SessionEvent()
|
|
|
|
/**
|
|
* A compaction that finished, and how much context it recovered.
|
|
*
|
|
* The counts are nullable because the server sends them only when it was told them: a zero here
|
|
* would read as "recovered nothing" and a made-up number would read as a measurement.
|
|
*/
|
|
data class Compacted(
|
|
val preTokens: Long?,
|
|
val postTokens: Long?,
|
|
/** What asked for it, in the CLI's own word; `auto` is the one worth naming. */
|
|
val trigger: String?,
|
|
) : SessionEvent()
|
|
|
|
/**
|
|
* The conversation was cleared. Everything above this is still here to read and is no longer in
|
|
* the session's context. An object rather than a class because what it means is entirely its
|
|
* position in the transcript.
|
|
*/
|
|
data object Cleared : SessionEvent()
|
|
|
|
/**
|
|
* The session stopped because its account's usage limit was reached.
|
|
*
|
|
* Its own event rather than an [Error] carrying the CLI's sentence, because it is a state
|
|
* rather than something that went wrong -- and because the raw sentence is `Claude AI usage
|
|
* limit reached|1788546972`, which is not readable by the person it is shown to.
|
|
*
|
|
* [resetsAt] is epoch seconds and null where the session was told nothing. Only the server acts
|
|
* on it; what this draws it as is a time, not a countdown, because nothing here re-measures it.
|
|
*/
|
|
data class LimitReached(val resetsAt: Double?) : SessionEvent()
|
|
|
|
data class AuthenticationRequired(val message: String) : SessionEvent()
|
|
|
|
data class Error(val message: String) : SessionEvent()
|
|
|
|
/**
|
|
* An event type this app build doesn't know -- a newer server. Kept rather than thrown so one
|
|
* new event kind degrades to a placeholder row instead of killing the stream.
|
|
*/
|
|
data class Unknown(val type: String) : SessionEvent()
|
|
}
|
|
|
|
/**
|
|
* A JSON array of strings under [name], empty when the field is absent -- the ordinary case, since
|
|
* the server omits the field rather than sending an empty list.
|
|
*/
|
|
private fun JSONObject.stringList(name: String): List<String> {
|
|
val array = optJSONArray(name) ?: return emptyList()
|
|
return (0 until array.length()).map { array.getString(it) }
|
|
}
|
|
|
|
fun parseSeqEvent(json: String): SeqEvent {
|
|
val body = JSONObject(json)
|
|
val event =
|
|
when (val type = body.getString("type")) {
|
|
"userMessage" ->
|
|
SessionEvent.UserMessage(
|
|
body.getString("text"),
|
|
body.optString("id").ifEmpty { null },
|
|
body.stringList("attachments"),
|
|
)
|
|
"messageQueued" ->
|
|
SessionEvent.MessageQueued(
|
|
body.getString("id"),
|
|
body.getString("text"),
|
|
body.stringList("attachments"),
|
|
)
|
|
"messageDropped" -> SessionEvent.MessageDropped(body.getString("id"))
|
|
"assistantText" -> SessionEvent.AssistantText(body.getString("delta"))
|
|
"assistantTextFinal" -> SessionEvent.AssistantTextFinal(body.getString("text"))
|
|
"thinking" -> SessionEvent.Thinking(body.getString("delta"))
|
|
"thinkingDone" -> SessionEvent.ThinkingDone(body.getLong("ms"))
|
|
"toolStart" ->
|
|
SessionEvent.ToolStart(
|
|
id = body.getString("id"),
|
|
tool = body.getString("tool"),
|
|
// Kept as raw JSON text: the input shape is the tool's own business, and the UI
|
|
// only ever shows it verbatim.
|
|
input = body.get("input").toString(),
|
|
)
|
|
"toolUpdate" -> SessionEvent.ToolUpdate(body.getString("id"), body.getString("output"))
|
|
"toolEnd" -> SessionEvent.ToolEnd(body.getString("id"), body.getString("output"))
|
|
"image" ->
|
|
SessionEvent.Image(
|
|
ref = body.getString("ref"),
|
|
about = body.optString("about").ifEmpty { null },
|
|
)
|
|
"question" ->
|
|
SessionEvent.Question(
|
|
id = body.getString("id"),
|
|
prompt = body.getString("prompt"),
|
|
header = body.optString("header").ifEmpty { null },
|
|
options =
|
|
body.getJSONArray("options").let { options ->
|
|
(0 until options.length()).map { at ->
|
|
val option = options.getJSONObject(at)
|
|
QuestionOption(
|
|
label = option.getString("label"),
|
|
description = option.optString("description").ifEmpty { null },
|
|
preview = option.optString("preview").ifEmpty { null },
|
|
)
|
|
}
|
|
},
|
|
multiSelect = body.optBoolean("multiSelect", false),
|
|
about = body.optString("about").ifEmpty { null },
|
|
)
|
|
"answered" ->
|
|
SessionEvent.Answered(
|
|
body.getString("id"),
|
|
body.getJSONArray("answers").let { answers ->
|
|
(0 until answers.length()).map { answers.getString(it) }
|
|
},
|
|
)
|
|
"peerMessage" ->
|
|
SessionEvent.PeerMessage(
|
|
body.getString("from"),
|
|
body.getString("text"),
|
|
if (body.has("turnStart")) body.getLong("turnStart") else null,
|
|
)
|
|
"unreadable" -> SessionEvent.Unreadable(body.getString("kind"))
|
|
"taskNote" -> SessionEvent.RetiredTaskNote
|
|
"commandQueued" ->
|
|
SessionEvent.CommandQueued(body.getString("id"), body.getString("text"))
|
|
"commandSent" -> SessionEvent.CommandSent(body.getString("id"), body.getString("text"))
|
|
"backgroundTasks" -> SessionEvent.BackgroundTasks(body.getInt("count"))
|
|
"status" -> SessionEvent.Status(body.getString("state"))
|
|
"settings" ->
|
|
SessionEvent.Settings(
|
|
model = body.optString("model").ifEmpty { null },
|
|
permissionMode = body.optString("permissionMode").ifEmpty { null },
|
|
)
|
|
"images" -> SessionEvent.Images(imageSupport(body.optString("images")))
|
|
"contextWindow" -> SessionEvent.ContextWindow(body.getLong("tokens"))
|
|
"usageDelta" ->
|
|
SessionEvent.UsageDelta(
|
|
body.getLong("tokens"),
|
|
if (body.has("context")) body.getLong("context") else null,
|
|
if (body.has("tokensPerSecond")) body.getDouble("tokensPerSecond") else null,
|
|
if (body.has("prefillMs")) body.getLong("prefillMs") else null,
|
|
)
|
|
"compacted" ->
|
|
SessionEvent.Compacted(
|
|
preTokens = if (body.has("preTokens")) body.getLong("preTokens") else null,
|
|
postTokens = if (body.has("postTokens")) body.getLong("postTokens") else null,
|
|
trigger = body.optString("trigger").ifEmpty { null },
|
|
)
|
|
"cleared" -> SessionEvent.Cleared
|
|
"limitReached" ->
|
|
SessionEvent.LimitReached(
|
|
if (body.has("resetsAt")) body.getDouble("resetsAt") else null
|
|
)
|
|
"authenticationRequired" ->
|
|
SessionEvent.AuthenticationRequired(body.getString("message"))
|
|
"error" -> SessionEvent.Error(body.getString("message"))
|
|
else -> SessionEvent.Unknown(type)
|
|
}
|
|
return SeqEvent(seq = body.getLong("seq"), ts = body.getDouble("ts"), event = event)
|
|
}
|
|
|
|
/**
|
|
* Whether [state] is one the session is doing work in -- the states a turn is still open under.
|
|
*
|
|
* One predicate because two readers have to agree on the list: the session screen's working
|
|
* indicator, and the fold's decision that the newest reply is finished. Two copies would drift the
|
|
* first time the server grows a state, and the drift would be a reply that never splits or one
|
|
* split mid-stream.
|
|
*/
|
|
fun sessionWorking(state: String): Boolean =
|
|
state == "running" || state == "compacting" || state == "loading" || state == "reading"
|
|
|
|
/** Whether the latest events still say this session needs an explicit provider login. */
|
|
internal fun authenticationPromptAfter(open: Boolean, event: SessionEvent): Boolean =
|
|
when (event) {
|
|
is SessionEvent.AuthenticationRequired -> true
|
|
// A later provider response proves an older authentication failure in a replayed page is
|
|
// no longer current. Without this, one old failure reopened sign-in after every later
|
|
// successful turn.
|
|
is SessionEvent.AssistantText,
|
|
is SessionEvent.AssistantTextFinal,
|
|
is SessionEvent.ToolStart -> false
|
|
else -> open
|
|
}
|
|
|
|
/**
|
|
* The context after [event], given what it was before.
|
|
*
|
|
* The same rule the server folds with, because the screen has to keep up between page loads: the
|
|
* summary it opened with is a measurement from before this stream started.
|
|
*
|
|
* The two that lower it are the point. A clear takes the conversation away and a compaction
|
|
* replaces it with a summary, so a figure measured before either stopped being true at that moment
|
|
* -- and carrying it forward is how a session that had just been cleared went on reporting the
|
|
* context it no longer had.
|
|
*
|
|
* Null is "we don't know", which each of them can reach.
|
|
*/
|
|
fun contextAfter(current: Long?, event: SessionEvent): Long? =
|
|
when (event) {
|
|
// Falls back to what we had, so a turn the dialect reported no usage for is stale by a turn
|
|
// -- which every context figure is -- rather than unknown.
|
|
is SessionEvent.UsageDelta -> event.context ?: current
|
|
is SessionEvent.Compacted -> event.postTokens
|
|
is SessionEvent.Cleared -> null
|
|
else -> current
|
|
}
|
|
|
|
/**
|
|
* The context window after [event], mirroring the server's `context_limit_after` for the same
|
|
* reason [contextAfter] mirrors its neighbour: the screen has to keep up between page loads.
|
|
*
|
|
* A window belongs to the process, so a session whose process has exited has none — left standing,
|
|
* a session restarted on a different model would draw its occupancy against the old model's
|
|
* ceiling.
|
|
*/
|
|
fun contextLimitAfter(current: Long?, event: SessionEvent): Long? =
|
|
when (event) {
|
|
is SessionEvent.ContextWindow -> event.tokens
|
|
is SessionEvent.Status -> if (event.state == "exited") null else current
|
|
else -> current
|
|
}
|