iris-aiandClaude Opus 5
ac476ab0c9
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.
Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.
Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.
Also here, because tools made each of them matter:
- **Loading is a state.** A 12 GB model takes twenty seconds to reach
memory and refuses everything until it has; the session used to report
`running` for that whole time, and a message sent meanwhile came back as
an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
stops it and starts another. The conversation survives because it was
never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
itself -- over ssh too, in the round trip the spawn was already making.
Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
slot, and 28 with it at four -- speculating against a split KV cache is
worse than not speculating. The flag is conditional because asking for a
head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
an overrun context is now ordinary; it was "http status: 400" and is now
the server's own "exceeds the available context size, try increasing it".
`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.
Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
46 lines
2.1 KiB
Kotlin
46 lines
2.1 KiB
Kotlin
package com.example.aiapp
|
|
|
|
/**
|
|
* What a session with no model of its own is called, in the button and in the list it opens.
|
|
*
|
|
* One constant rather than a literal in each place, because the two have to agree: a picker whose
|
|
* options cannot say every state its button can display is one you can leave and not get back to.
|
|
* It is also the Claude CLI's own word for "whatever is configured".
|
|
*/
|
|
const val DEFAULT_MODEL = "default"
|
|
|
|
/**
|
|
* A model's name as a person reads it.
|
|
*
|
|
* Providers answer with their own full identifier -- Claude Code resolves `haiku` to `claude-
|
|
* haiku-4-5-20251001` and reports that, which is the honest answer to "what is this session using"
|
|
* and far too long for a button in a row that also holds Stop and Send.
|
|
*
|
|
* So the two ends that identify nothing are dropped and nothing else is: the vendor prefix, which
|
|
* is the same on every model this app can show, and the release date, which distinguishes builds of
|
|
* one model rather than one model from another. Anything that does not look like that is returned
|
|
* untouched.
|
|
*
|
|
* A llama.cpp session's model is not an identifier at all -- it is `owner/repo/file.gguf`, where
|
|
* the file was downloaded from -- so what is kept is the file, which is the part that tells two
|
|
* models apart, and the extension goes with the directories. The model's *own* name is better still
|
|
* and is not derivable here: it is inside the file, and only the server has ever opened it. Where a
|
|
* screen has the server's answer it should prefer it; this is the floor under every screen that
|
|
* does not.
|
|
*
|
|
* A display decision, not a correction: the full name is what the session reports.
|
|
*/
|
|
fun modelLabel(model: String?): String {
|
|
val name = model?.takeIf { it.isNotBlank() } ?: return DEFAULT_MODEL
|
|
if (name.endsWith(GGUF)) {
|
|
return name.substringAfterLast('/').removeSuffix(GGUF)
|
|
}
|
|
return name.removePrefix("claude-").replace(DATED_SUFFIX, "")
|
|
}
|
|
|
|
/** A trailing `-YYYYMMDD`, which is how these identifiers carry their release date. */
|
|
private val DATED_SUFFIX = Regex("""-\d{8}$""")
|
|
|
|
/** What every model a llama.cpp session can run is stored as. */
|
|
private const val GGUF = ".gguf"
|