Run GGUF models through llama-server, and stop orphaning them
The second half of the llama.cpp work: a session can now name a downloaded model and talk to it. `llama-server` is spawned through the same transport as any other driver, polled until the model is loaded, then driven over its OpenAI-compatible streaming endpoint and translated into the same events the Claude driver emits -- so the transcript, the SSE stream and the phone need to know nothing new. **The conversation is rebuilt from the transcript, not held in the driver.** llama-server is stateless between requests, so the whole history goes with every one, and the obvious place to keep it is a Vec in the driver. That fails the requirement: memory in a driver is invisible to a second device and gone on restart, and this app is meant to work across devices. Reading it back also means the model is prompted with exactly what the phone was shown -- including a reply that was interrupted half way, which is in the transcript because the deltas were already emitted. That leaves the Claude driver as the odd one out rather than this one: the CLI's memory of a conversation is a cache in front of the same transcript, not a second truth. Said so at the top of llama.rs, because it is the sort of inconsistency that gets "fixed" in the wrong direction. Session settings arrive as a driver-interpreted `params` map rather than new typed fields, so the shared schema does not grow one dialect's vocabulary. Context size, gpu layers and threads become server flags; temperature and the rest ride on each request, so changing them need not reload a model. **Also fixes an orphan this feature would have created.** Drivers set kill_on_drop, which covers a session being deleted -- but nothing drops on the way out of a SIGTERM, so signalling the server left its children running. For the Claude CLI that is untidy; for a llama-server holding a model it is gigabytes belonging to nobody. The server now stops its sessions on SIGTERM and SIGINT. Found by killing a test server and noticing two 600 MB processes still resident. Remote llama sessions are refused rather than half-working: the model is reached over HTTP, and forwarding that port to an ssh host is the "reach this port" operation the transport does not have yet. Verified end to end against a real model: downloaded Qwen3-0.6B Q8_0 through the app's own download route, spawned a session on it, and held a two-turn conversation -- "my favourite colour is teal" then "what is my favourite colour?", answered "teal", which is the transcript replay doing its job. Token counts arrive. An earlier attempt with the IQ2_XXS quant produced fluent nonsense, which turned out to be the quantisation rather than the pipeline: llama-cli produces the same from that file directly. Four unit tests cover the fold and the path guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xn8nHw1tw1R6PtiY1eEtw
This commit is contained in:
1 parent
e50d1a2bbf
commit
3deeffd1e7
8 files changed
+949
-17
No files matched your search
@@ -179,6 +179,11 @@ struct SpawnRequest {
|
||||
cwd: Option<PathBuf>,
|
||||
#[serde(default)]
|
||||
permission_mode: Option<String>,
|
||||
/// Whatever the chosen driver understands -- llama.cpp's context size
|
||||
/// and sampling, for instance. Opaque here on purpose: see
|
||||
/// `SessionConfig::params`.
|
||||
#[serde(default)]
|
||||
params: std::collections::BTreeMap<String, String>,
|
||||
}
|
||||
|
||||
async fn spawn_session(
|
||||
@@ -193,6 +198,7 @@ async fn spawn_session(
|
||||
model: body.model,
|
||||
cwd: body.cwd,
|
||||
permission_mode: body.permission_mode,
|
||||
params: body.params,
|
||||
})
|
||||
.map_err(bad_request)?;
|
||||
tracing::info!(
|
||||
|
||||
Reference in new issue
Block a user