Run GGUF models through llama-server, and stop orphaning them

The second half of the llama.cpp work: a session can now name a downloaded
model and talk to it. `llama-server` is spawned through the same transport
as any other driver, polled until the model is loaded, then driven over its
OpenAI-compatible streaming endpoint and translated into the same events
the Claude driver emits -- so the transcript, the SSE stream and the phone
need to know nothing new.

**The conversation is rebuilt from the transcript, not held in the driver.**
llama-server is stateless between requests, so the whole history goes with
every one, and the obvious place to keep it is a Vec in the driver. That
fails the requirement: memory in a driver is invisible to a second device
and gone on restart, and this app is meant to work across devices. Reading
it back also means the model is prompted with exactly what the phone was
shown -- including a reply that was interrupted half way, which is in the
transcript because the deltas were already emitted.

That leaves the Claude driver as the odd one out rather than this one: the
CLI's memory of a conversation is a cache in front of the same transcript,
not a second truth. Said so at the top of llama.rs, because it is the sort
of inconsistency that gets "fixed" in the wrong direction.

Session settings arrive as a driver-interpreted `params` map rather than
new typed fields, so the shared schema does not grow one dialect's
vocabulary. Context size, gpu layers and threads become server flags;
temperature and the rest ride on each request, so changing them need not
reload a model.

**Also fixes an orphan this feature would have created.** Drivers set
kill_on_drop, which covers a session being deleted -- but nothing drops on
the way out of a SIGTERM, so signalling the server left its children
running. For the Claude CLI that is untidy; for a llama-server holding a
model it is gigabytes belonging to nobody. The server now stops its
sessions on SIGTERM and SIGINT. Found by killing a test server and noticing
two 600 MB processes still resident.

Remote llama sessions are refused rather than half-working: the model is
reached over HTTP, and forwarding that port to an ssh host is the "reach
this port" operation the transport does not have yet.

Verified end to end against a real model: downloaded Qwen3-0.6B Q8_0
through the app's own download route, spawned a session on it, and held a
two-turn conversation -- "my favourite colour is teal" then "what is my
favourite colour?", answered "teal", which is the transcript replay doing
its job. Token counts arrive. An earlier attempt with the IQ2_XXS quant
produced fluent nonsense, which turned out to be the quantisation rather
than the pipeline: llama-cli produces the same from that file directly.
Four unit tests cover the fold and the path guard.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xn8nHw1tw1R6PtiY1eEtw
This commit is contained in:
irisandClaude Opus 5 committed 2026-08-28 05:23:12 -04:00
1 parent e50d1a2bbf
commit 3deeffd1e7
8 files changed
+949 -17

No files matched your search

+2 -2
View File
@@ -10,7 +10,7 @@ path = "src/main.rs"
[dependencies]
axum = { version = "0.8", features = ["json", "multipart"] }
axum-server = { version = "0.8", features = ["tls-rustls"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "sync", "time", "process", "io-util"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "sync", "time", "process", "io-util", "signal"] }
tokio-stream = "0.1"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
@@ -46,7 +46,7 @@ rcgen = { version = "0.14", features = ["pem", "x509-parser"] }
# Outbound HTTPS for the usage endpoint. A small blocking client fits an
# every-few-minutes poll better than pulling in reqwest's tower stack;
# rustls-backed like the rest of the TLS here.
ureq = "3"
ureq = { version = "3", features = ["json"] }
# Direct dependency only to pick the process-level CryptoProvider in main:
# ureq pulls rustls-with-ring, axum-server rustls-with-aws-lc-rs, and with
# both in the graph rustls refuses to auto-select one.