Serve a machine's models from one shared llama-server

A llama.cpp session had its own `llama-server`: two sessions on one model
held two copies of it in memory, a model change bought a load only that
session benefited from, and the process was a session's to end. A machine's
models are now served by one `llama-server` in **router mode** -- no `-m`,
a preset file naming models and their flags, a child server per model asked
for, and each request routed by its `model` field. So one server per model
with that model's own settings is what a machine runs, while this backend
has one process, one port and one record per machine to keep track of.

The record is the mechanism every other driver already uses, so a restart
adopts it; a session records the same pid in its own directory as
`Detail::Shared`, and `process::signal` refuses to signal one of those --
which is what keeps stopping, deleting or cleaning up after one session
from unloading a model every other session is using. Nothing stops a router
on its own. That is deliberate (a loaded model is minutes of disk) and it is
why the machines tab now has a card per provider that opens its own screen:
how each model is loaded, how many stay in memory, Unload, and Stop.

How a model is *loaded* therefore belongs to the model on its machine rather
than to a session -- context size, GPU layers, threads, slots, speculative
decoding -- written into the preset as llama-server's own argument names.
Saving them re-reads that file, which unloads the model; that is the change
taking effect, and the dialog says so before you save. What stays a
session's is everything that rides on a request, including which tools it
offers: the router hosts one set for the machine and the choice is a filter
applied here, so it costs no reload (2,181 tokens of prompt with all seven,
698 with none).

Verified end to end against the scratch backend and the emulator: two
sessions sharing one loaded model with one child process, a second session
joining it with a 26ms prefill, a backend restart adopting the router and
answering with the prompt cache intact, the same over ssh to this VM, a
model's settings reaching the running server, Unload, and Stop leaving every
session `exited` with no error line.
This commit is contained in:
iris-ai committed 2026-09-19 17:37:31 -04:00
1 parent 74cda485e5
commit 8c323fc7a9
19 files changed
+2601 -511

No files matched your search

+8 -3
View File
@@ -174,7 +174,10 @@ same from the file directly, which is how to tell the two apart in a hurry.
server loading a model while the 27B holds VRAM fails with `radv/amdgpu:
Failed to allocate a buffer` / `MESA: error: buffer allocation failed` and
exits mid-request. `-ngl 0` runs it on the 8 cores instead, which is the way
to test the driver while something else holds the card.
to test the driver while something else holds the card -- through the app, that
is the model's "Layers on the GPU" set to 0 in the machines tab's provider
view, and `--models-max` above 1 is how two models come to be loaded at once
in the first place.
**Testing tools and MCP without the app**: `llama-server --tools all` publishes
its built-in tools at `GET /tools` and runs one at `POST /tools` with
@@ -253,8 +256,10 @@ where it was instead of half-deleted.
Draft acceptance is 0.530.73 in every case, so the head is working in all
of them: what changes is that speculating against a KV cache split four ways
is slower than not speculating. The driver passes `-np 1` always, so this is
recorded for whoever next sees MTP look broken. `--spec-draft-n-max 2` was
is slower than not speculating. A model's preset gets `parallel = 1` unless
its settings say otherwise (the machines tab's provider view, since
2026-09-19), so this is recorded for whoever next sees MTP look broken or
next raises the slot count to answer two sessions at once. `--spec-draft-n-max 2` was
worth another 7% in a single sample and is deliberately *not* passed — one
sample on a virtualised GPU is not a number to hardcode.