iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Give llama.cpp sessions tools, web search and a model picker

Browse Source 
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.

Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.

Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.

Also here, because tools made each of them matter:

- **Loading is a state.** A 12 GB model takes twenty seconds to reach
  memory and refuses everything until it has; the session used to report
  `running` for that whole time, and a message sent meanwhile came back as
  an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
  stops it and starts another. The conversation survives because it was
  never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
  itself -- over ssh too, in the round trip the spawn was already making.
  Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
  the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
  slot, and 28 with it at four -- speculating against a split KV cache is
  worse than not speculating. The flag is conditional because asking for a
  head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
  an overrun context is now ordinary; it was "http status: 400" and is now
  the server's own "exceeds the available context size, try increasing it".

`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.

Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
iris-aiandClaude Opus 5 committed 2026-09-19 08:11:49 -04:00
1 parent 392cc5413d
commit ac476ab0c9
23 files changed
+3627 -1096
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt
+24 -20
View File
@@ -1334,7 +1334,18 @@ fun deleteSession(settings: ServerSettings, sessionId: String, deleteForeign: Bo
// Browsing is proxied by the server rather than done here, because this app trusts exactly one
// certificate and has no general internet trust to spend on huggingface.co.
data class LocalModel(val key: String, val repo: String, val file: String, val bytes: Long)
data class LocalModel(
val key: String,
val repo: String,
val file: String,
val bytes: Long,
/**
* What the file itself says it is called, or null when it does not say. Not what to draw: see
* the server's `models::labels`, which needs the whole list to decide -- two quantisations of
* one model share a name.
*/
val name: String?,
)
/**
* A download in flight or finished. [total] is null when the server never said how big the file is
@@ -1371,36 +1382,28 @@ private fun parseDownload(o: JSONObject) =
)
/**
* The models on one machine, which is the list a llama.cpp session there can choose from.
* One model a picker can offer.
*
* Not [fetchModels], which is what the *backend* has downloaded. A session serves its model from
* the machine it runs on, so for a machine reached over ssh those are two different lists -- and
* offering the backend's would name files that are not there, turning a choice that cannot work
* into a session that fails when it tries to load one.
* Two fields because for one provider they differ: a llama.cpp session names its model by the path
* it lives at and reads it as the name its own metadata gives it. Every other provider's [id] is
* already what a person calls it, and the server says so by repeating it -- which is what keeps
* every picker here free of a branch on the session kind.
*/
fun fetchMachineModels(settings: ServerSettings, machineId: String): List<LocalModel> =
requestFromServer(settings, "/machines/${machineId.urlEncoded()}/models") { connection ->
JSONArray(connection.inputStream.bufferedReader().readText()).mapObjects { m ->
LocalModel(
key = m.getString("key"),
repo = m.getString("repo"),
file = m.getString("file"),
bytes = m.getLong("bytes"),
)
}
}
data class OfferedModel(val id: String, val label: String)
/** The current model catalog for one CLI provider on the machine where it runs. */
/** The current model catalog for one provider on the machine where it runs. */
fun fetchProviderModels(
settings: ServerSettings,
machineId: String,
provider: String,
): List<String> =
): List<OfferedModel> =
requestFromServer(
settings,
"/machines/${machineId.urlEncoded()}/providers/${provider.urlEncoded()}/models",
) { connection ->
JSONArray(connection.inputStream.bufferedReader().readText()).strings()
JSONArray(connection.inputStream.bufferedReader().readText()).mapObjects { m ->
OfferedModel(id = m.getString("id"), label = m.getString("label"))
}
}
fun fetchModels(settings: ServerSettings): Models =
@@ -1414,6 +1417,7 @@ fun fetchModels(settings: ServerSettings): Models =
repo = m.getString("repo"),
file = m.getString("file"),
bytes = m.getLong("bytes"),
name = m.optString("name").ifEmpty { null },
)
},
downloads = body.getJSONArray("downloads").mapObjects(::parseDownload),
app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt
+2 -1
View File
@@ -325,7 +325,8 @@ fun parseSeqEvent(json: String): SeqEvent {
* first time the server grows a state, and the drift would be a reply that never splits or one
* split mid-stream.
*/
fun sessionWorking(state: String): Boolean = state == "running" || state == "compacting"
fun sessionWorking(state: String): Boolean =
state == "running" || state == "compacting" || state == "loading"
/** Whether the latest events still say this session needs an explicit provider login. */
internal fun authenticationPromptAfter(open: Boolean, event: SessionEvent): Boolean =
app/androidApp/src/main/kotlin/com/example/aiapp/ModelName.kt
+13
View File
@@ -21,12 +21,25 @@ const val DEFAULT_MODEL = "default"
* one model rather than one model from another. Anything that does not look like that is returned
* untouched.
*
* A llama.cpp session's model is not an identifier at all -- it is `owner/repo/file.gguf`, where
* the file was downloaded from -- so what is kept is the file, which is the part that tells two
* models apart, and the extension goes with the directories. The model's *own* name is better still
* and is not derivable here: it is inside the file, and only the server has ever opened it. Where a
* screen has the server's answer it should prefer it; this is the floor under every screen that
* does not.
*
* A display decision, not a correction: the full name is what the session reports.
*/
fun modelLabel(model: String?): String {
val name = model?.takeIf { it.isNotBlank() } ?: return DEFAULT_MODEL
if (name.endsWith(GGUF)) {
return name.substringAfterLast('/').removeSuffix(GGUF)
}
return name.removePrefix("claude-").replace(DATED_SUFFIX, "")
}
/** A trailing `-YYYYMMDD`, which is how these identifiers carry their release date. */
private val DATED_SUFFIX = Regex("""-\d{8}$""")
/** What every model a llama.cpp session can run is stored as. */
private const val GGUF = ".gguf"
app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt
+24 -7
View File
@@ -302,7 +302,7 @@ fun SessionScreen(
var permissionMode by remember { mutableStateOf(summary.permissionMode ?: "auto") }
// The models this provider actually offers, asked of the server rather than listed here: a
// hardcoded list is a claim about a machine.
var offeredModels by remember { mutableStateOf<List<String>>(emptyList()) }
var offeredModels by remember { mutableStateOf<List<OfferedModel>>(emptyList()) }
var offeredPermissionModes by remember { mutableStateOf<List<String>>(emptyList()) }
val lifecycleOwner = LocalLifecycleOwner.current
// The resume cursor, written from the stream's IO thread.
@@ -1194,6 +1194,17 @@ fun SessionScreen(
}
}
/**
* What to call a model on screen.
*
* The provider's own answer where it has one, because only the server can have it: a llama
* model is identified by the path it lives at and named by what is written inside the file, and
* the phone has never opened that file. [modelLabel] is the fallback and the right one for the
* rest -- a coding CLI's identifier already is its name.
*/
fun label(id: String?): String =
offeredModels.firstOrNull { it.id == id }?.label ?: modelLabel(id)
// Only for the model picker, which a subagent does not have.
if (!isSubagent) {
LaunchedEffect(summary.machine, summary.provider) {
@@ -1889,8 +1900,8 @@ fun SessionScreen(
) {
pendingModel?.let { chosen ->
ModelSwitchWarning(
from = modelLabel(model),
to = modelLabel(chosen),
from = label(model),
to = label(chosen),
onDismiss = { pendingModel = null },
onConfirm = {
pendingModel = null
@@ -2013,22 +2024,28 @@ fun SessionScreen(
) {
if (offeredModels.isNotEmpty()) {
PickerButton(
current = modelLabel(model),
current = label(model),
// What the machine offers, plus the state a session is in when
// it has chosen none of them. The button has always been able
// to
// say "default"; until this the list could not, so leaving it
// was a one-way trip.
options = listOf(DEFAULT_MODEL) + offeredModels,
options =
listOf(DEFAULT_MODEL) + offeredModels.map { it.label },
// Not set here. The button follows what the session reports it
// is set to, which arrives a moment later and is sometimes a
// different answer -- a name the CLI resolved, or no change at
// all on a provider whose model is fixed. Asked about first,
// unless there is nothing to lose by it -- see
// [ModelSwitchWarning].
onPick = { chosen ->
onPick = { picked ->
// Back to the id, because that is what the server resolves
// and it is not always the word on the chip.
val chosen =
offeredModels.firstOrNull { it.label == picked }?.id
?: picked
if (
modelLabel(chosen) == modelLabel(model) ||
label(chosen) == label(model) ||
!worthWarningAbout(status, contextTokens, items)
) {
act { setSessionModel(settings, summary.id, chosen) }
app/androidApp/src/main/kotlin/com/example/aiapp/SessionStatusWords.kt
+7
View File
@@ -22,6 +22,10 @@ fun sessionStatusWord(status: String, subagent: Boolean = false): String =
"idle" -> "idle"
"running" -> "running"
"compacting" -> "compacting"
// Not "running": a model coming off disk is not a model answering, and the difference is
// minutes. Said in its own word so a first message that waits is explained rather than
// looking like a session that has stopped responding. See `SessionStatus::Loading`.
"loading" -> "loading"
// Its own word, because the state it is easily mistaken for means the opposite: "idle"
// invites the reader to type something, and a waiting session is going to carry on without
// them. See `SessionStatus::Waiting`.
@@ -53,6 +57,9 @@ fun sessionStatusColour(status: String): Color =
"awaitingInput" -> awaitingColor
"running" -> runningColor
"compacting" -> commandColor
// The same accent as the other states that are busy on their own account, because that is
// what this is: something is happening and nothing is wanted from the reader.
"loading" -> commandColor
"waiting" -> waitingColor
else -> MaterialTheme.colorScheme.onSurfaceVariant
}
app/androidApp/src/main/kotlin/com/example/aiapp/SpawnScreen.kt
+128 -93
View File
@@ -58,7 +58,7 @@ fun SpawnScreen(
var providerName by remember { mutableStateOf<String?>(null) }
var title by remember { mutableStateOf("") }
var model by remember { mutableStateOf("") }
var providerModels by remember { mutableStateOf<List<String>>(emptyList()) }
var providerModels by remember { mutableStateOf<List<OfferedModel>>(emptyList()) }
var providerModelsLoading by remember { mutableStateOf(false) }
var providerModelsError by remember { mutableStateOf<String?>(null) }
var cwd by remember { mutableStateOf("") }
@@ -73,12 +73,13 @@ fun SpawnScreen(
// Only the spawn's own failure. The fetch's lives in `options`: this one leaves a filled-in
// form worth keeping, and that one leaves nothing to fill in.
var spawnError by remember { mutableStateOf<String?>(null) }
// GGUFs on the chosen machine, for a llama provider to choose between. Kept separate from a
// coding CLI's provider catalog and refetched when the machine changes.
var models by remember { mutableStateOf<List<LocalModel>>(emptyList()) }
var modelKey by remember { mutableStateOf<String?>(null) }
var contextSize by remember { mutableStateOf("") }
var temperature by remember { mutableStateOf("") }
// Whether a model that carries a multi-token-prediction head drafts with it. Left to the
// server by default, which turns it on exactly where the file has one -- see `SPECULATIVE` in
// the llama driver. Here so a machine where drafting does not pay has a way out that is not
// an edit to config.ron.
var speculative by remember { mutableStateOf(SPECULATIVE_AUTO) }
LaunchedEffect(Unit) {
// Separate from the machines fetch below and deliberately not fatal: failing to learn the
@@ -126,17 +127,6 @@ fun SpawnScreen(
is LoadState.Loaded -> state.value
}
val machine = machines.firstOrNull { it.name == machineName }
// Whichever machine is chosen now, asked again when that changes. The old machine's list
// is dropped first rather than left on screen: a file name from another machine looks
// exactly like one from this one.
LaunchedEffect(machine?.id) {
models = emptyList()
modelKey = null
val id = machine?.id ?: return@LaunchedEffect
models =
runCatching { withContext(Dispatchers.IO) { fetchMachineModels(settings, id) } }
.getOrDefault(emptyList())
}
val current = machine?.providers?.firstOrNull { it.name == providerName }
// Coding CLIs take a working directory, model, permission mode and thinking level. Keying
// the extra fields on the kind rather than the provider name keeps a second installation
@@ -145,25 +135,35 @@ fun SpawnScreen(
val isCodex = current?.kind == "codex_cli"
val isCodingCli = isClaude || isCodex
val isLlama = current?.kind == "llama_cpp"
// Echo is the only kind with nothing to choose between.
val offersModels = isCodingCli || isLlama
// Where a session's tools act, which is the only thing a working directory decides.
val takesCwd = isCodingCli || isLlama
// Whichever machine and provider are chosen now, asked again when either changes. The
// previous answer is dropped first rather than left on screen: a model name from another
// machine looks exactly like one from this one.
LaunchedEffect(machine?.id, current?.name) {
model = ""
providerModels = emptyList()
providerModelsError = null
permissionMode = current?.defaultPermissionMode.orEmpty()
if (isCodingCli) {
providerModelsLoading = true
try {
providerModels =
withContext(Dispatchers.IO) {
fetchProviderModels(settings, machine.id, current.name)
}
} catch (e: ApiException) {
providerModelsError = e.message
} finally {
providerModelsLoading = false
}
} else {
// Every kind that offers models at all, not only the coding CLIs: a llama provider
// answers with the GGUFs on the machine it runs on, through the same call. One
// question with one answer is what keeps the picker free of a branch on the kind.
if (machine == null || current == null || !offersModels) {
providerModelsLoading = false
return@LaunchedEffect
}
providerModelsLoading = true
try {
providerModels =
withContext(Dispatchers.IO) {
fetchProviderModels(settings, machine.id, current.name)
}
} catch (e: ApiException) {
providerModelsError = e.message
} finally {
providerModelsLoading = false
}
}
@@ -220,29 +220,70 @@ fun SpawnScreen(
modifier = Modifier.fillMaxWidth(),
)
if (isLlama) {
// A llama session names one of the models on the machine it will run on, so the
// choice is that list rather than free text -- a name that is not on that machine's
// disk is a session that cannot start.
if (models.isEmpty()) {
Text(
"No models on ${machine.name}. The Models screen downloads " +
"to the backend; another machine needs the file put there itself.",
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
} else {
ChipGroup(
label = "Model",
// The file, not the whole key: the repository is the same for every
// quantisation of a model, so the file name is what tells two of them apart.
options = models.map { it.file },
selected = models.firstOrNull { it.key == modelKey }?.file,
onSelect = { file -> modelKey = models.first { it.file == file }.key },
)
if (offersModels) {
when {
providerModelsLoading ->
Text(
"Loading model choices…",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
providerModelsError != null ->
Text(
"Model choices unavailable: $providerModelsError",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.error,
)
// A llama session cannot start without one, so this says what to do about it
// rather than only that there is nothing -- the models it needs are on the
// machine that will serve them, which is not always this backend.
providerModels.isEmpty() && isLlama ->
Text(
"No models on ${machine?.name}. The Models screen downloads " +
"to the backend; another machine needs the file put there itself.",
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
providerModels.isEmpty() ->
Text(
"This machine reported no selectable models.",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
else -> {
Spacer(Modifier.height(16.dp))
ChipGroup(
label = "Model",
// The label, and the id is what is sent: for a llama model those differ,
// since it is chosen by path and named by what is inside the file.
options = providerModels.map { it.label },
selected = providerModels.firstOrNull { it.id == model }?.label,
onSelect = { chosen ->
val id = providerModels.first { it.label == chosen }.id
// A llama session has to have one, so choosing the same chip twice
// must not clear it -- there is nothing to fall back to.
model = if (model == id && !isLlama) "" else id
},
)
}
}
Spacer(Modifier.height(16.dp))
}
if (isCodingCli) {
// Free text as well as the chips above: the catalog is a shortcut, and a CLI will
// take a name it did not list.
OutlinedTextField(
value = model,
onValueChange = { model = it },
label = { Text("Model (blank = the CLI's default)") },
singleLine = true,
modifier = Modifier.fillMaxWidth(),
)
Spacer(Modifier.height(16.dp))
}
if (isLlama) {
OutlinedTextField(
value = contextSize,
onValueChange = { contextSize = it },
@@ -260,48 +301,21 @@ fun SpawnScreen(
modifier = Modifier.fillMaxWidth(),
)
Spacer(Modifier.height(16.dp))
}
if (isCodingCli) {
when {
providerModelsLoading ->
Text(
"Loading model choices…",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
providerModelsError != null ->
Text(
"Model choices unavailable: $providerModelsError",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.error,
)
providerModels.isEmpty() ->
Text(
"This machine reported no selectable models.",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
else -> {
Spacer(Modifier.height(16.dp))
ChipGroup(
label = "Model",
options = providerModels,
selected = model.ifEmpty { null },
onSelect = { chosen -> model = if (model == chosen) "" else chosen },
)
}
}
Spacer(Modifier.height(8.dp))
OutlinedTextField(
value = model,
onValueChange = { model = it },
label = { Text("Model (blank = the CLI's default)") },
singleLine = true,
modifier = Modifier.fillMaxWidth(),
// Said as what it is rather than as "MTP": the reader is choosing whether the session
// goes faster, and most models have nothing to turn on here at all.
ChipGroup(
label = "Speculative decoding (models that carry a draft head)",
options = listOf(SPECULATIVE_AUTO, SPECULATIVE_OFF),
selected = speculative,
onSelect = { speculative = it },
)
Spacer(Modifier.height(16.dp))
}
// Every session whose tools act on files needs one, which is both kinds that have
// tools -- a llama session's built-in tools run in it exactly as a CLI's do.
if (takesCwd) {
OutlinedTextField(
value = cwd,
onValueChange = { cwd = it },
@@ -311,7 +325,11 @@ fun SpawnScreen(
modifier = Modifier.fillMaxWidth(),
)
Spacer(Modifier.height(16.dp))
}
// Offered wherever the provider has modes, rather than where this screen believes it
// does: the server is what knows, and llama.cpp grew them without this line changing.
if (current != null && current.permissionModes.isNotEmpty()) {
ChipGroup(
label = "Permissions",
options = current.permissionModes,
@@ -319,7 +337,9 @@ fun SpawnScreen(
onSelect = { permissionMode = it },
)
Spacer(Modifier.height(16.dp))
}
if (isCodingCli) {
// Says what it does to *later* spawns as well, because it does: the level chosen here
// is stored as the default, which is the whole way that default is set. A picker that
// quietly changed a global would be the same control with the fact left out.
@@ -363,11 +383,9 @@ fun SpawnScreen(
machine = machine.id,
provider = chosen.name,
title = title.trim(),
model =
if (isLlama) modelKey
else model.trim().takeIf { isCodingCli },
cwd = cwd.trim().takeIf { isCodingCli },
permissionMode = permissionMode.takeIf { isCodingCli },
model = model.trim().takeIf { offersModels },
cwd = cwd.trim().takeIf { takesCwd },
permissionMode = permissionMode.takeIf { it.isNotEmpty() },
effort = effort.takeIf { isCodingCli },
// Sent only when set, so blank means "whatever llama.cpp does
// by default" rather than a zero.
@@ -382,6 +400,13 @@ fun SpawnScreen(
.trim()
.takeIf { it.isNotEmpty() }
?.let { put("temperature", it) }
// Only the choice that changes anything: "auto"
// is the absence of the setting, not a value of
// it, so a session spawned without an opinion
// carries none.
if (speculative == SPECULATIVE_OFF) {
put("speculative", "off")
}
}
},
)
@@ -393,13 +418,23 @@ fun SpawnScreen(
}
}
},
enabled = !busy && current != null && !(isLlama && modelKey == null),
// A llama session names the file to load, so there is nothing to spawn without one.
enabled = !busy && current != null && !(isLlama && model.isEmpty()),
) {
Text(if (busy) "Spawning..." else "Spawn")
}
}
}
/**
* Leave the draft head to the server, which uses one wherever the model file has one. Spelled the
* same as the absence of the `speculative` parameter, because that is what it means.
*/
private const val SPECULATIVE_AUTO = "auto"
/** The `speculative` parameter's only other value; see the llama driver's `SPECULATIVE`. */
private const val SPECULATIVE_OFF = "off"
/**
* A labeled row of choices that wraps onto as many lines as it needs.
*