Give llama.cpp sessions tools, web search and a model picker
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.
Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.
Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.
Also here, because tools made each of them matter:
- **Loading is a state.** A 12 GB model takes twenty seconds to reach
memory and refuses everything until it has; the session used to report
`running` for that whole time, and a message sent meanwhile came back as
an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
stops it and starts another. The conversation survives because it was
never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
itself -- over ssh too, in the round trip the spawn was already making.
Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
slot, and 28 with it at four -- speculating against a split KV cache is
worse than not speculating. The flag is conditional because asking for a
head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
an overrun context is now ordinary; it was "http status: 400" and is now
the server's own "exceeds the available context size, try increasing it".
`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.
Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
392cc5413d
commit
ac476ab0c9
23 files changed
+3627
-1096
No files matched your search
@@ -1334,7 +1334,18 @@ fun deleteSession(settings: ServerSettings, sessionId: String, deleteForeign: Bo
|
||||
// Browsing is proxied by the server rather than done here, because this app trusts exactly one
|
||||
// certificate and has no general internet trust to spend on huggingface.co.
|
||||
|
||||
data class LocalModel(val key: String, val repo: String, val file: String, val bytes: Long)
|
||||
data class LocalModel(
|
||||
val key: String,
|
||||
val repo: String,
|
||||
val file: String,
|
||||
val bytes: Long,
|
||||
/**
|
||||
* What the file itself says it is called, or null when it does not say. Not what to draw: see
|
||||
* the server's `models::labels`, which needs the whole list to decide -- two quantisations of
|
||||
* one model share a name.
|
||||
*/
|
||||
val name: String?,
|
||||
)
|
||||
|
||||
/**
|
||||
* A download in flight or finished. [total] is null when the server never said how big the file is
|
||||
@@ -1371,36 +1382,28 @@ private fun parseDownload(o: JSONObject) =
|
||||
)
|
||||
|
||||
/**
|
||||
* The models on one machine, which is the list a llama.cpp session there can choose from.
|
||||
* One model a picker can offer.
|
||||
*
|
||||
* Not [fetchModels], which is what the *backend* has downloaded. A session serves its model from
|
||||
* the machine it runs on, so for a machine reached over ssh those are two different lists -- and
|
||||
* offering the backend's would name files that are not there, turning a choice that cannot work
|
||||
* into a session that fails when it tries to load one.
|
||||
* Two fields because for one provider they differ: a llama.cpp session names its model by the path
|
||||
* it lives at and reads it as the name its own metadata gives it. Every other provider's [id] is
|
||||
* already what a person calls it, and the server says so by repeating it -- which is what keeps
|
||||
* every picker here free of a branch on the session kind.
|
||||
*/
|
||||
fun fetchMachineModels(settings: ServerSettings, machineId: String): List<LocalModel> =
|
||||
requestFromServer(settings, "/machines/${machineId.urlEncoded()}/models") { connection ->
|
||||
JSONArray(connection.inputStream.bufferedReader().readText()).mapObjects { m ->
|
||||
LocalModel(
|
||||
key = m.getString("key"),
|
||||
repo = m.getString("repo"),
|
||||
file = m.getString("file"),
|
||||
bytes = m.getLong("bytes"),
|
||||
)
|
||||
}
|
||||
}
|
||||
data class OfferedModel(val id: String, val label: String)
|
||||
|
||||
/** The current model catalog for one CLI provider on the machine where it runs. */
|
||||
/** The current model catalog for one provider on the machine where it runs. */
|
||||
fun fetchProviderModels(
|
||||
settings: ServerSettings,
|
||||
machineId: String,
|
||||
provider: String,
|
||||
): List<String> =
|
||||
): List<OfferedModel> =
|
||||
requestFromServer(
|
||||
settings,
|
||||
"/machines/${machineId.urlEncoded()}/providers/${provider.urlEncoded()}/models",
|
||||
) { connection ->
|
||||
JSONArray(connection.inputStream.bufferedReader().readText()).strings()
|
||||
JSONArray(connection.inputStream.bufferedReader().readText()).mapObjects { m ->
|
||||
OfferedModel(id = m.getString("id"), label = m.getString("label"))
|
||||
}
|
||||
}
|
||||
|
||||
fun fetchModels(settings: ServerSettings): Models =
|
||||
@@ -1414,6 +1417,7 @@ fun fetchModels(settings: ServerSettings): Models =
|
||||
repo = m.getString("repo"),
|
||||
file = m.getString("file"),
|
||||
bytes = m.getLong("bytes"),
|
||||
name = m.optString("name").ifEmpty { null },
|
||||
)
|
||||
},
|
||||
downloads = body.getJSONArray("downloads").mapObjects(::parseDownload),
|
||||
|
||||
Reference in new issue
Block a user