Give llama.cpp sessions tools, web search and a model picker
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.
Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.
Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.
Also here, because tools made each of them matter:
- **Loading is a state.** A 12 GB model takes twenty seconds to reach
memory and refuses everything until it has; the session used to report
`running` for that whole time, and a message sent meanwhile came back as
an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
stops it and starts another. The conversation survives because it was
never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
itself -- over ssh too, in the round trip the spawn was already making.
Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
slot, and 28 with it at four -- speculating against a split KV cache is
worse than not speculating. The flag is conditional because asking for a
head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
an overrun context is now ordinary; it was "http status: 400" and is now
the server's own "exceeds the available context size, try increasing it".
`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.
Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
392cc5413d
commit
ac476ab0c9
23 files changed
+3627
-1096
No files matched your search
@@ -302,7 +302,7 @@ fun SessionScreen(
|
||||
var permissionMode by remember { mutableStateOf(summary.permissionMode ?: "auto") }
|
||||
// The models this provider actually offers, asked of the server rather than listed here: a
|
||||
// hardcoded list is a claim about a machine.
|
||||
var offeredModels by remember { mutableStateOf<List<String>>(emptyList()) }
|
||||
var offeredModels by remember { mutableStateOf<List<OfferedModel>>(emptyList()) }
|
||||
var offeredPermissionModes by remember { mutableStateOf<List<String>>(emptyList()) }
|
||||
val lifecycleOwner = LocalLifecycleOwner.current
|
||||
// The resume cursor, written from the stream's IO thread.
|
||||
@@ -1194,6 +1194,17 @@ fun SessionScreen(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* What to call a model on screen.
|
||||
*
|
||||
* The provider's own answer where it has one, because only the server can have it: a llama
|
||||
* model is identified by the path it lives at and named by what is written inside the file, and
|
||||
* the phone has never opened that file. [modelLabel] is the fallback and the right one for the
|
||||
* rest -- a coding CLI's identifier already is its name.
|
||||
*/
|
||||
fun label(id: String?): String =
|
||||
offeredModels.firstOrNull { it.id == id }?.label ?: modelLabel(id)
|
||||
|
||||
// Only for the model picker, which a subagent does not have.
|
||||
if (!isSubagent) {
|
||||
LaunchedEffect(summary.machine, summary.provider) {
|
||||
@@ -1889,8 +1900,8 @@ fun SessionScreen(
|
||||
) {
|
||||
pendingModel?.let { chosen ->
|
||||
ModelSwitchWarning(
|
||||
from = modelLabel(model),
|
||||
to = modelLabel(chosen),
|
||||
from = label(model),
|
||||
to = label(chosen),
|
||||
onDismiss = { pendingModel = null },
|
||||
onConfirm = {
|
||||
pendingModel = null
|
||||
@@ -2013,22 +2024,28 @@ fun SessionScreen(
|
||||
) {
|
||||
if (offeredModels.isNotEmpty()) {
|
||||
PickerButton(
|
||||
current = modelLabel(model),
|
||||
current = label(model),
|
||||
// What the machine offers, plus the state a session is in when
|
||||
// it has chosen none of them. The button has always been able
|
||||
// to
|
||||
// say "default"; until this the list could not, so leaving it
|
||||
// was a one-way trip.
|
||||
options = listOf(DEFAULT_MODEL) + offeredModels,
|
||||
options =
|
||||
listOf(DEFAULT_MODEL) + offeredModels.map { it.label },
|
||||
// Not set here. The button follows what the session reports it
|
||||
// is set to, which arrives a moment later and is sometimes a
|
||||
// different answer -- a name the CLI resolved, or no change at
|
||||
// all on a provider whose model is fixed. Asked about first,
|
||||
// unless there is nothing to lose by it -- see
|
||||
// [ModelSwitchWarning].
|
||||
onPick = { chosen ->
|
||||
onPick = { picked ->
|
||||
// Back to the id, because that is what the server resolves
|
||||
// and it is not always the word on the chip.
|
||||
val chosen =
|
||||
offeredModels.firstOrNull { it.label == picked }?.id
|
||||
?: picked
|
||||
if (
|
||||
modelLabel(chosen) == modelLabel(model) ||
|
||||
label(chosen) == label(model) ||
|
||||
!worthWarningAbout(status, contextTokens, items)
|
||||
) {
|
||||
act { setSessionModel(settings, summary.id, chosen) }
|
||||
|
||||
Reference in new issue
Block a user