iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Serve a machine's models from one shared llama-server

Browse Source 
A llama.cpp session had its own `llama-server`: two sessions on one model
held two copies of it in memory, a model change bought a load only that
session benefited from, and the process was a session's to end. A machine's
models are now served by one `llama-server` in **router mode** -- no `-m`,
a preset file naming models and their flags, a child server per model asked
for, and each request routed by its `model` field. So one server per model
with that model's own settings is what a machine runs, while this backend
has one process, one port and one record per machine to keep track of.

The record is the mechanism every other driver already uses, so a restart
adopts it; a session records the same pid in its own directory as
`Detail::Shared`, and `process::signal` refuses to signal one of those --
which is what keeps stopping, deleting or cleaning up after one session
from unloading a model every other session is using. Nothing stops a router
on its own. That is deliberate (a loaded model is minutes of disk) and it is
why the machines tab now has a card per provider that opens its own screen:
how each model is loaded, how many stay in memory, Unload, and Stop.

How a model is *loaded* therefore belongs to the model on its machine rather
than to a session -- context size, GPU layers, threads, slots, speculative
decoding -- written into the preset as llama-server's own argument names.
Saving them re-reads that file, which unloads the model; that is the change
taking effect, and the dialog says so before you save. What stays a
session's is everything that rides on a request, including which tools it
offers: the router hosts one set for the machine and the choice is a filter
applied here, so it costs no reload (2,181 tokens of prompt with all seven,
698 with none).

Verified end to end against the scratch backend and the emulator: two
sessions sharing one loaded model with one child process, a second session
joining it with a 26ms prefill, a backend restart adopting the router and
answering with the prompt cache intact, the same over ssh to this VM, a
model's settings reaching the running server, Unload, and Stop leaving every
session `exited` with no error line.
This commit is contained in:
iris-ai committed 2026-09-19 17:37:31 -04:00
1 parent 74cda485e5
commit 8c323fc7a9
19 files changed
+2601 -511
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

server/src/session/llama/router.rs
+857
View File
@@ -0,0 +1,857 @@
//! One `llama-server` per machine, in **router mode**: the front door to every
//! model that machine serves, shared by every session on it.
//!
//! A router holds no weights itself. It reads a preset file naming models and
//! their flags, and starts a child `llama-server` per model that is asked for
//! -- so "one server per model, with that model's own settings" is what a
//! machine ends up running, and one process, one port and one record is what
//! this backend has to keep track of. That is the whole reason it is here:
//! before 2026-09-19 each session started its own `llama-server`, so two
//! sessions on one model held two copies of it in memory and a model change
//! cost a load that only that session benefited from.
//!
//! **A router outlives this backend, and nothing here stops it on its own.**
//! It is recorded the way a session's process is ([`process`]), adopted again
//! on the way back up, and ended only when somebody asks for that in the
//! machine's provider settings. A loaded model is minutes of disk and
//! gigabytes of memory; letting the last session to be closed throw that away
//! would make the shared server pointless.
//!
//! **The preset file is the configuration, and it lives on the serving
//! machine.** Flags that decide how a model is loaded -- context size, layers
//! on the GPU, slots, the draft head -- are per model rather than per session,
//! because one loaded model is what several sessions are now talking to. They
//! are written into a section named by the model's key, which is also the name
//! a request routes by, so nothing has to translate between the two.
//!
//! What is deliberately *not* here: which tools a session offers, how hard it
//! thinks, and the sampling settings. Those ride on each request, so they stay
//! the session's own and need no reload -- see `super`'s module comment.
use std::collections::{BTreeMap, HashMap};
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use anyhow::{Context, Result, bail};
use serde::Serialize;
use serde_json::{Value, json};
use super::Model;
use crate::config::{MachineConfig, ProviderConfig};
use crate::session::process;
use crate::session::transport::{Launch, Streams, Transport};
/// Where a router's own output goes, both streams into one file. For a remote
/// machine this is the local end of the ssh connection, so it carries what the
/// far `llama-server` said -- including what a child model instance said,
/// which is the only account of a model that would not load.
const LOG: &str = "llama-router.log";
/// The preset file's name in the router's own directory, for a router on this
/// machine. One on another machine keeps it over there instead, at
/// [`REMOTE_PRESET`], since that is the only side that can read it.
const PRESET: &str = "models.ini";
/// The preset file's first line, which `llama-server` refuses a file without.
const VERSION: &str = "version = 1\n";
/// How long to wait for a model to load before giving up. Loading is mostly
/// disk, and a large quantised model on a cold cache is genuinely slow, so
/// this is generous -- the failure it exists for is a model that will never
/// answer rather than one that is slow.
const LOAD_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
/// How long to wait for the router process itself. Short: it loads nothing,
/// so anything beyond a second or two is a port it cannot bind or a program
/// too old for one of these flags.
const START_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
/// How often either of those is checked.
const POLL: std::time::Duration = std::time::Duration::from_millis(250);
/// How much of the router's log to carry into a message somebody reads on a
/// phone.
const LOG_TAIL_LINES: usize = 6;
/// How many models a router keeps loaded at once before evicting the least
/// recently used. One by default because the machine serving models usually
/// has one GPU: a second model loaded beside the first is the case where
/// neither fits.
const DEFAULT_MAX_LOADED: u32 = 1;
/// Every machine's router, so that two sessions on one machine reach one
/// process rather than starting two.
///
/// A registry rather than a field on each session: the sharing *is* the
/// point, and a router that two drivers could each own is one that both would
/// start.
pub struct Routers {
dir: PathBuf,
/// The runtime a router is started on and reaped into.
///
/// Captured here because everything that starts one runs on a *blocking*
/// thread -- loading a model is minutes of disk, so it cannot be on the
/// runtime -- and tokio's `Command::spawn` registers the child with the
/// reactor, so calling it outside a runtime context panics. That panic is
/// silent: it kills the loading thread and leaves the session saying
/// "loading" for ever, with nothing in the log, which is exactly how it
/// was found.
runtime: Option<tokio::runtime::Handle>,
inner: Mutex<HashMap<String, Arc<Router>>>,
}
impl Routers {
/// `dir` is where each router's record, log and (for this machine) preset
/// file live -- beside the session directories, since a router is shared
/// by sessions and belongs to none of them.
///
/// Made on the runtime that will outlive it; `None` is a test with no
/// runtime at all, where there is nothing to reap into either.
pub fn new(dir: PathBuf) -> Self {
Self {
dir,
runtime: tokio::runtime::Handle::try_current().ok(),
inner: Mutex::new(HashMap::new()),
}
}
/// This machine-and-provider's router, made if this is the first ask.
///
/// The machine and provider are re-read every time rather than captured,
/// because both are editable while sessions are running: a renamed
/// machine, a re-probed program, a changed `maxLoaded`. What a *running*
/// router was started with is whatever it was started with; the new value
/// reaches the next start, which is the same rule every other launch flag
/// follows.
pub fn of(&self, machine: &MachineConfig, provider: &ProviderConfig) -> Arc<Router> {
let key = format!("{}/{}", machine.id, provider.name);
let spec = Spec {
transport: Transport::for_machine(machine),
program: provider.program().to_string(),
max_loaded: provider.max_loaded.unwrap_or(DEFAULT_MAX_LOADED),
};
let mut routers = self.inner.lock().unwrap_or_else(|e| e.into_inner());
let router = routers.entry(key.clone()).or_insert_with(|| {
Arc::new(Router {
dir: self.dir.join(key.replace('/', "-")),
spec: Mutex::new(spec.clone()),
runtime: self.runtime.clone(),
gate: Mutex::new(()),
})
});
*router.spec.lock().unwrap_or_else(|e| e.into_inner()) = spec;
Arc::clone(router)
}
}
/// What it takes to start a router, as its machine currently describes it.
#[derive(Clone)]
struct Spec {
transport: Transport,
program: String,
max_loaded: u32,
}
pub struct Router {
/// Its record and log, on this machine whichever machine it serves from.
dir: PathBuf,
spec: Mutex<Spec>,
/// See [`Routers::runtime`].
runtime: Option<tokio::runtime::Handle>,
/// Held while a router is started and while the preset file is edited --
/// the two things that go wrong when two sessions do them at once. Never
/// held across a model load, which takes minutes.
gate: Mutex<()>,
}
impl Router {
/// Where its record and log are, for a session that wants to watch the
/// process it is talking through or read what it last said.
pub fn dir(&self) -> &Path {
&self.dir
}
/// Its process record, if one is running -- which is also how a session
/// records the process it reaches its model through.
pub fn record(&self) -> Option<process::Record> {
process::live(&self.dir)
}
/// The same process, recorded as one a session reaches but does not own.
///
/// The one place that conversion happens, so that a session directory can
/// never come to hold a record saying it owns the machine's router -- see
/// [`process::Detail::Shared`].
pub fn shared_record(&self) -> Option<process::Record> {
let record = self.record()?;
match record.detail {
process::Detail::Http { port } => Some(process::Record {
detail: process::Detail::Shared { port },
..record
}),
process::Detail::Stdio { .. } | process::Detail::Shared { .. } => None,
}
}
/// Where to reach it, or `None` when nothing is running.
pub fn endpoint(&self) -> Option<String> {
match self.record()?.detail {
process::Detail::Http { port } => Some(format!("http://127.0.0.1:{port}")),
process::Detail::Stdio { .. } | process::Detail::Shared { .. } => None,
}
}
/// Puts `model` in memory and says where to talk to it, starting the
/// router first if it is not up.
///
/// Blocking, and slow on purpose: a model that has to come off disk takes
/// as long as it takes. The caller is the driver's loading thread, which
/// is what [`super::Serving::Loading`] exists to describe.
///
/// Nothing is held across that wait. Two sessions load through one router,
/// and the second one wanting a model already in memory must not queue
/// behind the first one's cold load of a different model -- which is most
/// of what sharing a server was for.
pub fn load(
&self,
key: &str,
found: &Model,
settings: &BTreeMap<String, String>,
) -> Result<String> {
let endpoint = self.start_if_down()?;
self.describe_model(key, found, settings)?;
// Nothing to ask for where it is already in memory, which is the case
// this whole module exists to produce: a second session naming a model
// somebody else loaded is a round trip rather than a load. Asking
// anyway is not harmless -- `POST /models/load` answers **400** for a
// model that is already loaded, which arrived as a session that
// refused to start next to one happily using that same model.
if !self.is_ready(key) {
let asked = self
.post("/models/load", json!({ "model": key }))
.with_context(|| format!("asking llama-server to load {key}"));
match (asked, self.wait_loaded(key)) {
// Loaded, whatever the request said: something else may have
// asked for it in the meantime, and what is in memory is the
// answer rather than what one request made of being told to
// put it there.
(_, Ok(())) => {}
// It did not load, and a refusal of the request itself says
// more about why than "it never appeared" does.
(Err(refused), Err(_)) => return Err(refused),
(Ok(_), Err(never)) => return Err(never),
}
}
Ok(endpoint)
}
/// Writes what this model is and how to load it into the preset file, and
/// has the router re-read it.
///
/// Also the path a settings change takes, which is why it is separate from
/// [`load`](Self::load): a model whose entry has changed is **unloaded**
/// by the re-read, and that is the change taking effect rather than a
/// side effect -- the sessions using it load it again, with the new
/// settings, on their next message. What must not happen is the same
/// thing to an unrelated model, which is why the file is written and the
/// re-read asked for only when the text actually differs.
pub fn describe_model(
&self,
key: &str,
found: &Model,
settings: &BTreeMap<String, String>,
) -> Result<()> {
// Read, edit, write: under the gate because two of those at once lose
// one of the two sections.
let _one_at_a_time = self.gate.lock().unwrap_or_else(|e| e.into_inner());
let existing = self.preset()?;
let updated = upsert(&existing, key, &section(found, settings));
if updated == existing {
return Ok(());
}
self.write_preset(&updated)?;
// Only meaningful against a running router; one that is down reads the
// file when it starts.
if self.endpoint().is_some() {
self.get("/models?reload=1")
.context("asking llama-server to re-read its models")?;
}
Ok(())
}
/// Whether this model is in memory now.
fn is_ready(&self, key: &str) -> bool {
self.loaded()
.into_iter()
.any(|model| model.model == key && model.ready)
}
/// Every model this router knows about and what each is doing, or an empty
/// list when it is not running.
pub fn loaded(&self) -> Vec<RouterModel> {
let Ok(answer) = self.get("/models") else {
return Vec::new();
};
answer
.get("data")
.and_then(Value::as_array)
.map(|models| models.iter().filter_map(RouterModel::read).collect())
.unwrap_or_default()
}
/// Takes one model out of memory, leaving the router and every other model
/// alone.
pub fn unload(&self, key: &str) -> Result<()> {
self.post("/models/unload", json!({ "model": key }))
.with_context(|| format!("asking llama-server to unload {key}"))?;
Ok(())
}
/// Ends the router and every model it is holding.
///
/// The only thing that does: a session closing, being deleted, or this
/// backend shutting down all leave it running. Sessions using it will see
/// their process go and report `exited`, which is true -- the model they
/// were talking to is no longer in memory.
///
/// Neither the record nor the mark is cleared here, and that is what tells
/// those sessions this was asked for rather than a crash. A session's
/// watcher looks a couple of seconds later, so anything removed now is
/// removed before the only reader of it has looked -- which is how the
/// first version of this put "llama-server exited" in three transcripts
/// belonging to somebody who had just pressed Stop. The record describes a
/// dead process, which every reader already handles, and starting a new
/// router is what clears both.
pub fn stop(&self) -> Result<()> {
let Some(record) = self.record() else {
return Ok(());
};
process::mark_stopping(&self.dir)?;
process::stop(&record, process::STOP_GRACE);
Ok(())
}
/// The end of the router's log, for a failure message.
///
/// Its account of what went wrong is the useful half -- "failed to load
/// model", "bind: Address already in use" -- and on a remote machine it is
/// the only half, since nobody reading the phone can open a file over
/// there. Bounded, because this ends up in an event a phone draws.
pub fn log_tail(&self) -> String {
let Ok(text) = std::fs::read_to_string(self.dir.join(LOG)) else {
return String::new();
};
let tail: Vec<&str> = text.lines().rev().take(LOG_TAIL_LINES).collect();
if tail.is_empty() {
return String::new();
}
format!(
" It last said: {}",
tail.into_iter().rev().collect::<Vec<_>>().join(" / ")
)
}
/// Adopts the running router or starts one, and waits for it to answer.
///
/// Under the gate, and asked again inside it: two sessions starting at once
/// would otherwise both find no record, both start a router, and bind two
/// ports to the same models.
fn start_if_down(&self) -> Result<String> {
if let Some(endpoint) = self.endpoint() {
return Ok(endpoint);
}
let _one_at_a_time = self.gate.lock().unwrap_or_else(|e| e.into_inner());
if let Some(endpoint) = self.endpoint() {
return Ok(endpoint);
}
let spec = self.spec.lock().unwrap_or_else(|e| e.into_inner()).clone();
wg_app_link::private::create_dir(&self.dir)?;
// Whatever the last router left behind, including the mark saying its
// end was asked for -- see [`stop`](Self::stop). From here on, a
// process that goes away is news.
process::clear(&self.dir);
// Written before the router starts, because it is read at startup and
// a router with no preset file at all lists nothing.
let preset = self.write_preset(&self.preset()?)?;
let forward = spec
.transport
.reserve_port()
.context("finding a port for llama-server")?;
let args = vec![
// Loopback there, whichever machine there is: what reaches it from
// outside that machine is the ssh tunnel and nothing else.
"--host".to_string(),
"127.0.0.1".to_string(),
"--port".to_string(),
forward.there.to_string(),
// No `-m`: a `llama-server` given no model is a router.
"--models-preset".to_string(),
preset,
"--models-max".to_string(),
spec.max_loaded.to_string(),
// The built-in agent tools -- read, search, edit, shell. Hosted by
// the router itself, which is what makes them one set for the
// machine rather than one per model. Which of them a *session*
// offers its model is decided here in the backend, per request, so
// there is nothing per-session to pass through: see `super::tools`.
"--tools".to_string(),
"all".to_string(),
];
let launch = Launch::new(&spec.program, args, None).reaching(forward);
// Starting and reaping both happen inside the runtime, though this is
// a blocking thread: tokio's `Command::spawn` registers the child with
// the reactor, so calling it outside a runtime context panics -- and
// that panic kills only this thread, leaving a session that says
// "loading" for ever with nothing in the log. See [`Routers::runtime`].
let _inside = self.runtime.as_ref().map(tokio::runtime::Handle::enter);
// Its output goes to a file, not a pipe. Not only so the process can
// outlive this server: nothing ever read those pipes, so a chatty
// llama-server filled the 64 KB buffer and blocked with no sign of why.
let child = spec.transport.spawn(
&launch,
Streams::Detached {
stdin: std::process::Stdio::null(),
stdout: log_file(&self.dir.join(LOG))?.into(),
stderr: log_file(&self.dir.join(LOG))?.into(),
},
)?;
let pid = child
.id()
.context("llama-server exited before it could be recorded")?;
tracing::info!(
"running {} in router mode {} on 127.0.0.1:{} there, reached at 127.0.0.1:{} here, \
as pid {pid}",
spec.program,
spec.transport.describe(),
forward.there,
forward.here,
);
// Reaped so it does not become a zombie while this server is still
// its parent; the record and the health poll are what say whether it
// is alive, because after a restart there is no `Child` to ask.
if let Some(runtime) = &self.runtime {
runtime.spawn(async move {
let mut child = child;
let _ = child.wait().await;
});
}
// The *near* port, because that is the one anything reaching this
// router has to dial -- including a later run of this backend, which
// adopts the record without knowing which machine it is on. For a
// remote machine the recorded pid is the ssh client's, which is the
// process this machine owns and which holds the tunnel open for
// exactly as long as the far router lives.
let record = process::Record::of(pid, process::Detail::Http { port: forward.here })
.context("llama-server was gone before its start time could be read")?;
process::write(&self.dir, &record);
let endpoint = format!("http://127.0.0.1:{}", forward.here);
self.wait_answering(&endpoint)?;
Ok(endpoint)
}
/// Polls until the router answers, watching the process as well as the
/// port: a port already taken, or a `llama-server` too old for one of
/// these flags, exits within a second and would otherwise be waited out.
fn wait_answering(&self, endpoint: &str) -> Result<()> {
let deadline = std::time::Instant::now() + START_TIMEOUT;
let url = format!("{endpoint}/health");
loop {
if let Ok(response) = ureq::get(&url).call()
&& response.status() == 200
{
return Ok(());
}
match process::recorded(&self.dir) {
Some((_, process::Liveness::Alive | process::Liveness::Unknown)) => {}
Some((_, process::Liveness::Dead)) | None => {
process::clear(&self.dir);
bail!("llama-server exited before it answered.{}", self.log_tail());
}
}
if std::time::Instant::now() > deadline {
process::clear(&self.dir);
bail!(
"llama-server did not answer within {}s.{}",
START_TIMEOUT.as_secs(),
self.log_tail()
);
}
std::thread::sleep(POLL);
}
}
/// Polls until the named model is in memory, or says why it will never be.
///
/// A model that will not load is the common failure and it is fast: the
/// child exits, the router reports it unloaded with an exit code, and this
/// says so rather than waiting out the timeout -- which is what turned "it
/// said the file was corrupt" into "gave up after 300s".
fn wait_loaded(&self, key: &str) -> Result<()> {
let deadline = std::time::Instant::now() + LOAD_TIMEOUT;
loop {
match self.loaded().into_iter().find(|model| model.model == key) {
Some(model) if model.ready => return Ok(()),
Some(model) if model.failed => {
bail!("{key} would not load.{}", self.log_tail())
}
// Still loading, or not listed yet after a reload.
Some(_) | None => {}
}
if process::live(&self.dir).is_none() {
bail!(
"llama-server went away while loading {key}.{}",
self.log_tail()
);
}
if std::time::Instant::now() > deadline {
bail!(
"{key} was still loading after {}s.{}",
LOAD_TIMEOUT.as_secs(),
self.log_tail()
);
}
std::thread::sleep(POLL);
}
}
/// One request to the router's management endpoints. The generation ones
/// are the driver's and stream, so they are not these.
fn get(&self, path: &str) -> Result<Value> {
let url = format!("{}{path}", self.answering()?);
Self::read(
ureq::get(&url)
.call()
.with_context(|| format!("GET {path}"))?,
path,
)
}
fn post(&self, path: &str, body: Value) -> Result<Value> {
let url = format!("{}{path}", self.answering()?);
Self::read(
ureq::post(&url)
.send_json(body)
.with_context(|| format!("POST {path}"))?,
path,
)
}
/// Where to reach a router that is running, or the failure saying it is
/// not -- which is what every one of these requests needs first.
fn answering(&self) -> Result<String> {
self.endpoint().context("no llama-server is running")
}
fn read(mut response: ureq::http::Response<ureq::Body>, path: &str) -> Result<Value> {
response
.body_mut()
.read_json()
.with_context(|| format!("reading what {path} answered"))
}
/// The preset file as it stands on the machine that serves the models, or
/// empty where there is none yet.
///
/// Read back rather than remembered, for one reason that matters after a
/// restart: a router adopted from a previous run is already serving models
/// whose sections this process has never seen, and rewriting the file
/// without them would unload them at the next reload.
fn preset(&self) -> Result<String> {
let spec = self.spec.lock().unwrap_or_else(|e| e.into_inner()).clone();
let text = match &spec.transport {
Transport::Here => {
Ok(std::fs::read_to_string(self.dir.join(PRESET)).unwrap_or_default())
}
Transport::Ssh { name, .. } => {
let script = format!("p={REMOTE_PRESET}; cat \"$p\" 2>/dev/null || true");
let launch = Launch::new("sh", vec!["-c".to_string(), script], None);
spec.transport
.capture_blocking(&launch)
.with_context(|| format!("reading the model settings on {name}"))
}
}?;
// A file that is not there yet reads as a new one rather than as
// nothing: `llama-server` refuses a preset with no version line, so
// "empty" is not a state this can hand back or write.
Ok(if text.trim().is_empty() {
VERSION.to_string()
} else {
text
})
}
/// Writes the preset file where the router will read it, and says where
/// that is -- which is the path the router is given, so the two cannot
/// disagree.
fn write_preset(&self, text: &str) -> Result<String> {
let spec = self.spec.lock().unwrap_or_else(|e| e.into_inner()).clone();
match &spec.transport {
Transport::Here => {
let path = self.dir.join(PRESET);
wg_app_link::private::write_file(&path, text.as_bytes())?;
Ok(path.to_string_lossy().into_owned())
}
Transport::Ssh { name, .. } => {
use base64::Engine as _;
// Base64 rather than a heredoc: the text goes through a shell
// on the far side, and an INI value is not something to trust
// to quoting rules twice over.
let encoded = base64::engine::general_purpose::STANDARD.encode(text);
let script = format!(
"p={REMOTE_PRESET}; mkdir -p \"$(dirname \"$p\")\" && \
printf %s \"$1\" | base64 -d > \"$p\" && printf '%s\\n' \"$p\""
);
let launch = Launch::new(
"sh",
vec!["-c".to_string(), script, "sh".to_string(), encoded],
None,
);
let answer = spec
.transport
.capture_blocking(&launch)
.with_context(|| format!("writing the model settings on {name}"))?;
match answer.trim() {
"" => bail!("{name} did not say where it wrote the model settings"),
path => Ok(path.to_string()),
}
}
}
}
}
/// Where the preset file goes on a machine that is not this one, as a shell
/// word the far side expands: under that machine's state directory, beside
/// whatever else belongs to this app there.
///
/// `$HOME` is resolved over there because only that machine knows what it is.
const REMOTE_PRESET: &str = "\"${XDG_STATE_HOME:-$HOME/.local/state}/ai-app/llama-models.ini\"";
/// One model the router knows about, as the provider view and the load poll
/// both read it.
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct RouterModel {
pub model: String,
/// The router's own word: `unloaded`, `loading`, `loaded`, `sleeping`.
/// Carried through rather than reduced to a boolean, because the phone
/// draws it and llama.cpp is the authority on what states there are.
pub status: String,
pub ready: bool,
/// Unloaded *and* something went wrong, which is not the same as unloaded.
#[serde(skip)]
pub failed: bool,
}
impl RouterModel {
fn read(entry: &Value) -> Option<Self> {
let model = entry.get("id")?.as_str()?.to_string();
let status = entry
.pointer("/status/value")
.and_then(Value::as_str)
.unwrap_or("unknown")
.to_string();
let exited = entry
.get("exit_code")
.and_then(Value::as_i64)
.unwrap_or_default();
Some(Self {
ready: status == "loaded" || status == "sleeping",
failed: status == "unloaded" && exited != 0,
model,
status,
})
}
}
/// What a model's section says, from the file it is and the settings it has.
///
/// The keys are `llama-server`'s own argument names without their dashes,
/// which is what a preset section is: `--n-gpu-layers 20` is
/// `n-gpu-layers = 20`. So adding a setting is a row here and a row in
/// [`crate::config::LLAMA_MODEL_PARAMS`], and nothing in between.
fn section(found: &Model, settings: &BTreeMap<String, String>) -> String {
let mut lines = vec![format!("model = {}", found.path)];
for (key, flag) in [
("contextSize", "ctx-size"),
("gpuLayers", "n-gpu-layers"),
("threads", "threads"),
// How far ahead the draft head guesses. Not defaulted: 2 measured 7%
// faster than llama.cpp's 3 on this machine's GPU, once, which is a
// reason to make the knob reachable and not a reason to move it for
// everybody.
("specDraftNMax", "spec-draft-n-max"),
] {
if let Some(value) = settings.get(key).map(|value| value.trim())
&& !value.is_empty()
{
lines.push(format!("{flag} = {value}"));
}
}
// One slot unless this model is told otherwise. A session is one
// conversation making one request at a time, and a second session's turn
// waits rather than splitting the cache: measured 2026-09-19 on the 27B
// here, 41.5 tok/s plain at any slot count, **61.4** with the draft head
// at one slot, and **28** with the head at four. Speculating against a
// split KV cache is slower than not speculating at all.
let slots = settings
.get("slots")
.map(|value| value.trim())
.filter(|value| !value.is_empty())
.unwrap_or("1");
lines.push(format!("parallel = {slots}"));
// A model carrying a multi-token-prediction head drafts with it, which is
// most of a 50% speed-up for free -- the tensors are in the file whether
// or not they are used. Conditional because it cannot be otherwise: asked
// for on a model without one, `llama-server` **exits** ("context type MTP
// requested but model doesn't contain MTP layers"). See `Model::mtp`.
if found.mtp && settings.get("speculative").map(String::as_str) != Some("off") {
lines.push("spec-type = draft-mtp".to_string());
}
lines.join("\n")
}
/// The preset file with `name`'s section replaced by `body`, added at the end
/// if it was not there.
///
/// Text in and text out, rather than a parsed model, because the file belongs
/// to `llama-server` rather than to this: anything in it that this does not
/// understand -- a `[*]` section, a key added by a later version, a comment
/// somebody wrote -- has to survive being edited.
fn upsert(existing: &str, name: &str, body: &str) -> String {
let header = format!("[{name}]");
let mut out = String::new();
let mut skipping = false;
let mut replaced = false;
for line in existing.lines() {
let trimmed = line.trim();
if trimmed.starts_with('[') && trimmed.ends_with(']') {
skipping = trimmed == header;
if skipping {
replaced = true;
push_section(&mut out, &header, body);
continue;
}
}
if !skipping {
out.push_str(line);
out.push('\n');
}
}
if out.is_empty() {
out.push_str(VERSION);
}
if !replaced {
push_section(&mut out, &header, body);
}
out
}
fn push_section(out: &mut String, header: &str, body: &str) {
if !out.ends_with("\n\n") && !out.is_empty() {
out.push('\n');
}
out.push_str(header);
out.push('\n');
out.push_str(body.trim_end());
out.push_str("\n\n");
}
/// An owner-only log opened for appending, so the two streams pointed at it do
/// not overwrite each other and an adopted router keeps what came before.
fn log_file(path: &Path) -> Result<std::fs::File> {
use std::os::unix::fs::OpenOptionsExt;
std::fs::OpenOptions::new()
.create(true)
.append(true)
.mode(0o600)
.open(path)
.with_context(|| format!("opening {}", path.display()))
}
#[cfg(test)]
mod tests {
use super::*;
fn settings(pairs: &[(&str, &str)]) -> BTreeMap<String, String> {
pairs
.iter()
.map(|(key, value)| ((*key).to_string(), (*value).to_string()))
.collect()
}
#[test]
fn a_section_names_the_file_and_the_flags_that_were_set() {
let found = Model {
path: "/models/a.gguf".to_string(),
mtp: true,
};
let text = section(
&found,
&settings(&[("contextSize", "8192"), ("threads", " 6 ")]),
);
assert_eq!(
text,
"model = /models/a.gguf\nctx-size = 8192\nthreads = 6\nparallel = 1\n\
spec-type = draft-mtp"
);
// A blank is not a value: it is the setting being unset, and passing
// it on is a child that exits on an empty argument.
let text = section(&found, &settings(&[("contextSize", " ")]));
assert!(!text.contains("ctx-size"), "{text}");
// The draft head is asked for only where the file has one, and can be
// turned off for a machine where it does not pay.
let plain = Model {
mtp: false,
..found.clone()
};
assert!(!section(&plain, &settings(&[])).contains("spec-type"));
assert!(!section(&found, &settings(&[("speculative", "off")])).contains("spec-type"));
}
#[test]
fn a_section_replaces_its_own_and_leaves_every_other_line_alone() {
let first = upsert("", "repo/a.gguf", "model = /models/a.gguf");
assert!(first.starts_with("version = 1\n"), "{first}");
assert!(
first.contains("[repo/a.gguf]\nmodel = /models/a.gguf\n"),
"{first}"
);
// A second model is added rather than replacing the first, because a
// reload of a file that lost a section unloads that model -- which
// would be one session taking another's model out of memory.
let both = upsert(&first, "repo/b.gguf", "model = /models/b.gguf");
assert!(both.contains("[repo/a.gguf]"), "{both}");
assert!(both.contains("[repo/b.gguf]"), "{both}");
// Editing one rewrites only its own keys, and keeps what llama.cpp's
// own file has that this does not know about.
let with_global = format!(
"version = 1\n\n[*]\njinja = true\n\n{}",
both.trim_start_matches("version = 1\n")
);
let edited = upsert(
&with_global,
"repo/a.gguf",
"model = /models/a.gguf\nctx-size = 4096",
);
assert!(edited.contains("[*]\njinja = true"), "{edited}");
assert!(edited.contains("ctx-size = 4096"), "{edited}");
assert_eq!(edited.matches("[repo/a.gguf]").count(), 1, "{edited}");
assert!(
edited.contains("[repo/b.gguf]\nmodel = /models/b.gguf"),
"{edited}"
);
}
#[test]
fn writing_the_same_settings_twice_changes_nothing() {
// What keeps an unrelated session's model in memory: the file is only
// written, and the router only told to re-read it, when the text
// actually differs.
let once = upsert("", "repo/a.gguf", "model = /models/a.gguf");
assert_eq!(upsert(&once, "repo/a.gguf", "model = /models/a.gguf"), once);
}
}