Give llama.cpp sessions tools, web search and a model picker
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.
Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.
Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.
Also here, because tools made each of them matter:
- **Loading is a state.** A 12 GB model takes twenty seconds to reach
memory and refuses everything until it has; the session used to report
`running` for that whole time, and a message sent meanwhile came back as
an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
stops it and starts another. The conversation survives because it was
never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
itself -- over ssh too, in the round trip the spawn was already making.
Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
slot, and 28 with it at four -- speculating against a split KV cache is
worse than not speculating. The flag is conditional because asking for a
head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
an overrun context is now ordinary; it was "http status: 400" and is now
the server's own "exceeds the available context size, try increasing it".
`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.
Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
392cc5413d
commit
ac476ab0c9
23 files changed
+3627
-1096
No files matched your search
@@ -47,6 +47,21 @@ Module-by-module intent is in PLAN.md's "Backend layout".
|
||||
readiness poll watches the process as well as the port, since a model that
|
||||
will not load exits in a second and was being reported as "gave up after
|
||||
300s". See PLAN.md's "Transport" and "llama-server management".
|
||||
**A llama session has tools and runs the loop itself** (2026-09-19):
|
||||
`--tools all` gives it `llama-server`'s built-in set, which that server also
|
||||
*runs* (`GET /tools` for the definitions, `POST /tools` to call one), while
|
||||
web search comes from an MCP server this backend connects to directly
|
||||
(`session/llama/mcp.rs`, Exa preset in a discovered provider's
|
||||
`mcpServers`). Driving the loop is what makes the permission gate ours:
|
||||
`manual` asks before every call and remembers a tool you answer
|
||||
"Always allow …" to, `bypassPermissions` never asks, and the allowances are
|
||||
folded back out of the transcript. Three more things fall out of it and are
|
||||
easy to get wrong again — a model change **reloads the server** rather than
|
||||
being refused, since the conversation lives in the transcript rather than in
|
||||
`llama-server`; `-np 1` is always passed, and it is what decides whether the
|
||||
MTP draft head is a 50% speed-up or a 33% loss; and `--spec-type draft-mtp`
|
||||
is conditional on the file actually having a head, because asking for one
|
||||
that is not there makes `llama-server` **exit**.
|
||||
Codex is one persistent `codex app-server --stdio` process per session; its
|
||||
driver uses native turn steering and interruption, persists the protocol
|
||||
state and thread id, and reads subscription limits through the same CLI
|
||||
@@ -310,6 +325,15 @@ written, and the fold uses that same predicate to decide a reply is settled.
|
||||
|
||||
## Things that have bitten
|
||||
|
||||
- **A llama session reports `loading`, and a message sent into it waits.**
|
||||
Before 2026-09-19 the session showed `running` from the moment the process
|
||||
started, so a minute of reading a model off disk was indistinguishable from
|
||||
a minute of thinking -- and anything sent in that window came back as an
|
||||
error, because `llama-server` refuses everything until the model is in
|
||||
memory. `SessionStatus::Loading` is the state and `Shared::await_ready` is
|
||||
the waiting. A driver that reports `Loading` owes the holding as well as the
|
||||
word.
|
||||
|
||||
- **A transcript outlives the enum.** Removing `Event::TaskNote` hours after
|
||||
adding it made every transcript that had recorded one unreadable, so
|
||||
`launch` failed for those sessions and `SessionManager::new` skipped them —
|
||||
|
||||
Reference in new issue
Block a user