A llama.cpp session had its own `llama-server`: two sessions on one model
held two copies of it in memory, a model change bought a load only that
session benefited from, and the process was a session's to end. A machine's
models are now served by one `llama-server` in **router mode** -- no `-m`,
a preset file naming models and their flags, a child server per model asked
for, and each request routed by its `model` field. So one server per model
with that model's own settings is what a machine runs, while this backend
has one process, one port and one record per machine to keep track of.
The record is the mechanism every other driver already uses, so a restart
adopts it; a session records the same pid in its own directory as
`Detail::Shared`, and `process::signal` refuses to signal one of those --
which is what keeps stopping, deleting or cleaning up after one session
from unloading a model every other session is using. Nothing stops a router
on its own. That is deliberate (a loaded model is minutes of disk) and it is
why the machines tab now has a card per provider that opens its own screen:
how each model is loaded, how many stay in memory, Unload, and Stop.
How a model is *loaded* therefore belongs to the model on its machine rather
than to a session -- context size, GPU layers, threads, slots, speculative
decoding -- written into the preset as llama-server's own argument names.
Saving them re-reads that file, which unloads the model; that is the change
taking effect, and the dialog says so before you save. What stays a
session's is everything that rides on a request, including which tools it
offers: the router hosts one set for the machine and the choice is a filter
applied here, so it costs no reload (2,181 tokens of prompt with all seven,
698 with none).
Verified end to end against the scratch backend and the emulator: two
sessions sharing one loaded model with one child process, a second session
joining it with a 26ms prefill, a backend restart adopting the router and
answering with the prompt cache intact, the same over ssh to this VM, a
model's settings reaching the running server, Unload, and Stop leaving every
session `exited` with no error line.
A turn has two waits in front of the first token and they were one word.
`SessionStatus::Loading` was already the model coming off disk; this adds
`SessionStatus::Reading` for llama-server processing the prompt -- emitted when
the request goes out, cleared by the first thing the model says of any kind, so
it covers every generate in a tool loop rather than only the first.
Prefill is the expensive half on this machine: measured 9.5s for 6,068 tokens
and 22s for 14,068 on the 27B with the GPU to itself. Reported as `running`
that was indistinguishable from a model thinking, which is the thing the reader
is waiting for. The phone draws both with the working spinner and its own
words -- "loading model" and "reading prompt" -- and the session screen's
status row now spins for all three busy states instead of only `running`,
which is also how `loading` stops being a bare word with nothing moving.
Measured while checking the tok/s figure, and recorded in the rigs skill: the
27B holds 55.5 to 50.3 tok/s between 1.5k and 14k of context, so decode decays
gently, while the 0.6B on the CPU falls 30.1 to 11.5 over 6k. A shared GPU is a
different failure -- the model does not load at all.
Verified on the emulator against a real llama session: "loading model" while
the server started, then "reading prompt" with the spinner through prompt
processing, then the thinking card.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A llama.cpp session's `reasoning_content` becomes `Event::Thinking` deltas
closed by an `Event::ThinkingDone` carrying the span the driver measured, and
the phone draws it as a card of its own: "Thinking" with the spinner a running
command has, then "Thought for 12.4s". Deliberately not a tool call, so a run
of calls cannot collapse the reasoning into "Called 6 tools"; the reasoning is
also kept out of the next prompt, which `conversation` already ignored.
`UsageDelta` gains `tokensPerSecond`, the provider's own figure or nothing --
llama.cpp reports `timings.predicted_per_second` and the coding CLIs report no
such thing -- and a finished reply carries a small line under it saying when it
was sent and, where there is one, how fast it came out: "3:00 PM · 149 tok/s".
The compact usage bar drops the provider's name for the window and puts its
length after the time left instead: "42% · 3h 20m left / 5h".
Three things that had to come with it: the transcript coalesces runs of
thinking deltas as it does reply deltas, so one block is one row of a page
rather than a page of its own; `joinPages` welds a block cut by a page boundary
(`healSplitThinking`), since the half with no ending spun for ever; and
`UsageDelta` now reaches the fold, which is what carries the rate to the reply.
Verified on the emulator against a real Qwen3-0.6B session and the echo rig's
new `/think [seconds]`: the spinner while it runs, "Thought for 1.4s" and
"2:54 PM · 149 tok/s" after, the reasoning on tapping the card, and the usage
bar reading "42% · 3h 19m left / 5h".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.
Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.
Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.
Also here, because tools made each of them matter:
- **Loading is a state.** A 12 GB model takes twenty seconds to reach
memory and refuses everything until it has; the session used to report
`running` for that whole time, and a message sent meanwhile came back as
an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
stops it and starts another. The conversation survives because it was
never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
itself -- over ssh too, in the round trip the spawn was already making.
Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
slot, and 28 with it at four -- speculating against a split KV cache is
worse than not speculating. The flag is conditional because asking for a
head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
an overrun context is now ordinary; it was "http status: 400" and is now
the server's own "exceeds the available context size, try increasing it".
`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.
Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`Config::default_effort` is what a session starts at when nothing chose one,
applied in `spawn_session` rather than filled in by the spawn screen so it
holds for an import and a bare API call too. It is set by the spawn screen's
own picker, whose label says so: one control, where new sessions are made,
rather than a settings page for a single value. Not on a provider, because
providers are discovered and the next rediscovery would erase it; not on the
phone, because a second device would then spawn at a level nobody there
chose. `GET`/`POST /defaults` carry it as a struct, so the permission mode --
still hardcoded to `auto` on the spawn screen -- can move there later without
a second route.
Only drivers that read a level are given one: an echo session was storing a
`--effort` it never passes to anything, which is a config file answering a
question about itself wrongly.
Separately, `AGENTS.md` is 35 KB sent with every request in this repo, and 12
KB of it was rigs and reference measurements that only matter once you are
running one. Those are the `ai-app-rigs` skill now -- the same text, still the
only copy, read when the work touches it. 35,198 -> 20,813 chars.
Verified on the emulator against the sandbox: the spawn screen pre-fills from
the server, picking `low` spawned a session at `low` and left `/defaults` set
to it, and an echo session spawned afterwards took no level at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
iris-ai
iris