Give llama.cpp sessions tools, web search and a model picker
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.
Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.
Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.
Also here, because tools made each of them matter:
- **Loading is a state.** A 12 GB model takes twenty seconds to reach
memory and refuses everything until it has; the session used to report
`running` for that whole time, and a message sent meanwhile came back as
an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
stops it and starts another. The conversation survives because it was
never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
itself -- over ssh too, in the round trip the spawn was already making.
Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
slot, and 28 with it at four -- speculating against a split KV cache is
worse than not speculating. The flag is conditional because asking for a
head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
an overrun context is now ordinary; it was "http status: 400" and is now
the server's own "exceeds the available context size, try increasing it".
`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.
Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
392cc5413d
commit
ac476ab0c9
23 files changed
+3627
-1096
No files matched your search
@@ -345,6 +345,55 @@ deliberate and easy to undo by accident:
|
||||
out the 300s timeout turned the server's own account of the problem into
|
||||
"gave up". The failure carries the tail of `llama-server.log`, which on a
|
||||
remote session is the only copy anybody reading the phone can see.
|
||||
- **Loading is a state of its own** (2026-09-19, `SessionStatus::Loading`).
|
||||
A multi-gigabyte model takes tens of seconds to reach memory and refuses
|
||||
everything until it has, and the session used to report `running` for that
|
||||
whole time — indistinguishable from a model thinking, with the added
|
||||
detail that any message sent meanwhile came back as an error. It is now
|
||||
`loading` on both screens, and a message sent into a load **waits** for it
|
||||
rather than failing. The waiting is the driver's (`Shared::await_ready`, a
|
||||
condvar on a three-state `Serving`), because "there is a process and it is
|
||||
not ready" is a fact only a driver can have. The third state matters as
|
||||
much as the first two: a model that will never load has to answer a waiting
|
||||
message with what went wrong rather than holding it for ever.
|
||||
- **The driver runs the agent loop, and therefore owns the permission gate**
|
||||
(2026-09-19). `llama-server --tools all` *hosts* the built-in tools —
|
||||
`GET /tools` is their definitions, `POST /tools` runs one — but it does not
|
||||
drive a conversation: a completion comes back with tool calls in it and
|
||||
stops. So the loop is here, which is what puts "may I run this?" somewhere
|
||||
a phone can answer it. Two modes, `manual` and `bypassPermissions`, which
|
||||
is what the mechanism actually has: llama.cpp's own web UI asks before
|
||||
every call and remembers the tools you said "always" to, and a third mode
|
||||
between them would have to invent a rule about which tools count as edits.
|
||||
The allowances are folded out of the transcript's `Answered` events, like
|
||||
everything else this driver remembers, which is why the answer carries the
|
||||
tool's name in it.
|
||||
- **Tools run where the model does; MCP runs here** (2026-09-19). The
|
||||
built-in tools are the far machine's, for the same reason the model file
|
||||
is — they act on that machine's disk. An MCP server is reached from *this*
|
||||
backend instead (`session/llama/mcp.rs`), which is both what llama.cpp's
|
||||
own web UI does (it connects to `https://mcp.exa.ai/mcp` from the browser)
|
||||
and the right side to be on: a web search wants the machine with a route
|
||||
out, not the machine with the GPU. `llama-server`'s own `--mcp-servers-json`
|
||||
is deliberately not used — it can only spawn local commands, so a remote
|
||||
server would mean a Node bridge on whichever machine serves the model.
|
||||
- **A model change reloads the server rather than being refused** (2026-09-19).
|
||||
A `llama-server` holds one model, so switching stops it and starts another;
|
||||
the conversation survives because the conversation was never in the server.
|
||||
What is lost is the prompt cache, which is exactly what the phone already
|
||||
warns about before a switch.
|
||||
- **One slot, and the draft head where the file has one** (measured
|
||||
2026-09-19). `-np 1` always: a session is one conversation making one
|
||||
request at a time, so the other three slots `llama-server` picks on its own
|
||||
are context this session could have had. It is also what decides whether
|
||||
multi-token prediction pays — on the 27B here, **41.5 tok/s** plain at any
|
||||
slot count, **61.4** with `--spec-type draft-mtp` at one slot, and **28**
|
||||
with the head at four. Speculating against a split KV cache is worse than
|
||||
not speculating, and it reads exactly like the head being broken.
|
||||
The flag is conditional because it must be: asked for on a model without a
|
||||
head, `llama-server` exits. `crate::gguf::has_mtp_head` reads the answer out
|
||||
of the file — on the machine that will serve it, in the round trip the spawn
|
||||
was already making — and `params["speculative"] = "off"` is the way out.
|
||||
|
||||
### Models (2026-08-28)
|
||||
|
||||
@@ -1386,8 +1435,10 @@ verified by running it, matching dev-updater's posture.
|
||||
it truncates old KV cache entries, which is silent forgetting with no
|
||||
summary, and it corrupts the harness's view of what the model knows. Fine
|
||||
as a server-side safety net; not memory management.
|
||||
- **Remote llama-server** needs its port forwarded (`ssh -L`) and is not
|
||||
built; such a session is refused rather than misdirected.
|
||||
- **MCP servers are configured in `config.ron`, not from the phone**
|
||||
(2026-09-19). `mcpServers` on a llama provider, with Exa preset on a newly
|
||||
discovered one. A phone screen for them is the obvious next step and was
|
||||
deliberately left out of the change that added them.
|
||||
- **Claude sessions over ssh need the remote machine logged in to Claude.**
|
||||
Usage reporting reads each machine's own credentials, and the Machines tab
|
||||
can run that machine's CLI login without requiring an interactive SSH shell.
|
||||
|
||||
Reference in new issue
Block a user