iris-ai
bd9596d782
A multimodal model is loaded with the `mmproj` found beside its weights -- which is how a repository publishes the pair -- and an attached image rides in the request as an `image_url` data URI, so it reaches a model on another machine without the file going there. Nothing is done for a model without a projector: no captioning, no OCR, no second model. Whether a session takes pictures is measured rather than assumed: `/props`'s `modalities.vision` from the server that loaded the model, in three states, because a model still coming off disk has genuinely not said. Unknown is offered rather than refused -- a control withheld because nobody could ask goes missing from sessions that would have taken it. The answer reaches the phone twice per model as `Event::Images`, so the photo button is withdrawn the moment a model with vision is left rather than at whatever later point the session row is fetched again. A message carrying an image a model cannot read is stopped rather than stripped: `llama-server` refuses the whole request over one image part, and a message sent without its picture would be answered as though the picture had never been mentioned. The phone will not attach one, and the driver refuses it again at the three moments the answer can first exist -- at the door, when a message queued behind a loading model is read, and at the tool boundary a steer enters by. An earlier turn's image folds into a line of words for a model without vision, so switching a conversation onto one does not end it. A projector is filtered out of the models a provider *offers*, since a session started on one is a server that cannot load it; it stays in the machine's own model list, where a file on a disk is managed. Verified against ggml-org/SmolVLM-256M-Instruct-GGUF, local and over ssh: "In this picture there is a red circle." Switching that session to Qwen3-0.6B reports `refused`, refuses the next picture with the reason, and still answers an ordinary message.
931 lines
39 KiB
Rust
931 lines
39 KiB
Rust
//! One `llama-server` per machine, in **router mode**: the front door to every
|
|
//! model that machine serves, shared by every session on it.
|
|
//!
|
|
//! A router holds no weights itself. It reads a preset file naming models and
|
|
//! their flags, and starts a child `llama-server` per model that is asked for
|
|
//! -- so "one server per model, with that model's own settings" is what a
|
|
//! machine ends up running, and one process, one port and one record is what
|
|
//! this backend has to keep track of. That is the whole reason it is here:
|
|
//! before 2026-09-19 each session started its own `llama-server`, so two
|
|
//! sessions on one model held two copies of it in memory and a model change
|
|
//! cost a load that only that session benefited from.
|
|
//!
|
|
//! **A router outlives this backend, and nothing here stops it on its own.**
|
|
//! It is recorded the way a session's process is ([`process`]), adopted again
|
|
//! on the way back up, and ended only when somebody asks for that in the
|
|
//! machine's provider settings. A loaded model is minutes of disk and
|
|
//! gigabytes of memory; letting the last session to be closed throw that away
|
|
//! would make the shared server pointless.
|
|
//!
|
|
//! **The preset file is the configuration, and it lives on the serving
|
|
//! machine.** Flags that decide how a model is loaded -- context size, layers
|
|
//! on the GPU, slots, the draft head -- are per model rather than per session,
|
|
//! because one loaded model is what several sessions are now talking to. They
|
|
//! are written into a section named by the model's key, which is also the name
|
|
//! a request routes by, so nothing has to translate between the two.
|
|
//!
|
|
//! What is deliberately *not* here: which tools a session offers, how hard it
|
|
//! thinks, and the sampling settings. Those ride on each request, so they stay
|
|
//! the session's own and need no reload -- see `super`'s module comment.
|
|
|
|
use std::collections::{BTreeMap, HashMap};
|
|
use std::path::{Path, PathBuf};
|
|
use std::sync::{Arc, Mutex};
|
|
|
|
use anyhow::{Context, Result, bail};
|
|
use serde::Serialize;
|
|
use serde_json::{Value, json};
|
|
|
|
use super::Model;
|
|
use crate::config::{MachineConfig, ProviderConfig};
|
|
use crate::session::process;
|
|
use crate::session::transport::{Launch, Streams, Transport};
|
|
|
|
/// Where a router's own output goes, both streams into one file. For a remote
|
|
/// machine this is the local end of the ssh connection, so it carries what the
|
|
/// far `llama-server` said -- including what a child model instance said,
|
|
/// which is the only account of a model that would not load.
|
|
const LOG: &str = "llama-router.log";
|
|
|
|
/// The preset file's name in the router's own directory, for a router on this
|
|
/// machine. One on another machine keeps it over there instead, at
|
|
/// [`REMOTE_PRESET`], since that is the only side that can read it.
|
|
const PRESET: &str = "models.ini";
|
|
|
|
/// The preset file's first line, which `llama-server` refuses a file without.
|
|
const VERSION: &str = "version = 1\n";
|
|
|
|
/// How long to wait for a model to load before giving up. Loading is mostly
|
|
/// disk, and a large quantised model on a cold cache is genuinely slow, so
|
|
/// this is generous -- the failure it exists for is a model that will never
|
|
/// answer rather than one that is slow.
|
|
const LOAD_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
|
|
|
|
/// How long to wait for the router process itself. Short: it loads nothing,
|
|
/// so anything beyond a second or two is a port it cannot bind or a program
|
|
/// too old for one of these flags.
|
|
const START_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
|
|
|
/// How often either of those is checked.
|
|
const POLL: std::time::Duration = std::time::Duration::from_millis(250);
|
|
|
|
/// How much of the router's log to carry into a message somebody reads on a
|
|
/// phone.
|
|
const LOG_TAIL_LINES: usize = 6;
|
|
|
|
/// How many models a router keeps loaded at once before evicting the least
|
|
/// recently used. One by default because the machine serving models usually
|
|
/// has one GPU: a second model loaded beside the first is the case where
|
|
/// neither fits.
|
|
const DEFAULT_MAX_LOADED: u32 = 1;
|
|
|
|
/// Every machine's router, so that two sessions on one machine reach one
|
|
/// process rather than starting two.
|
|
///
|
|
/// A registry rather than a field on each session: the sharing *is* the
|
|
/// point, and a router that two drivers could each own is one that both would
|
|
/// start.
|
|
pub struct Routers {
|
|
dir: PathBuf,
|
|
/// The runtime a router is started on and reaped into.
|
|
///
|
|
/// Captured here because everything that starts one runs on a *blocking*
|
|
/// thread -- loading a model is minutes of disk, so it cannot be on the
|
|
/// runtime -- and tokio's `Command::spawn` registers the child with the
|
|
/// reactor, so calling it outside a runtime context panics. That panic is
|
|
/// silent: it kills the loading thread and leaves the session saying
|
|
/// "loading" for ever, with nothing in the log, which is exactly how it
|
|
/// was found.
|
|
runtime: Option<tokio::runtime::Handle>,
|
|
inner: Mutex<HashMap<String, Arc<Router>>>,
|
|
}
|
|
|
|
impl Routers {
|
|
/// `dir` is where each router's record, log and (for this machine) preset
|
|
/// file live -- beside the session directories, since a router is shared
|
|
/// by sessions and belongs to none of them.
|
|
///
|
|
/// Made on the runtime that will outlive it; `None` is a test with no
|
|
/// runtime at all, where there is nothing to reap into either.
|
|
pub fn new(dir: PathBuf) -> Self {
|
|
Self {
|
|
dir,
|
|
runtime: tokio::runtime::Handle::try_current().ok(),
|
|
inner: Mutex::new(HashMap::new()),
|
|
}
|
|
}
|
|
|
|
/// This machine-and-provider's router, made if this is the first ask.
|
|
///
|
|
/// The machine and provider are re-read every time rather than captured,
|
|
/// because both are editable while sessions are running: a renamed
|
|
/// machine, a re-probed program, a changed `maxLoaded`. What a *running*
|
|
/// router was started with is whatever it was started with; the new value
|
|
/// reaches the next start, which is the same rule every other launch flag
|
|
/// follows.
|
|
pub fn of(&self, machine: &MachineConfig, provider: &ProviderConfig) -> Arc<Router> {
|
|
let key = format!("{}/{}", machine.id, provider.name);
|
|
let spec = Spec {
|
|
transport: Transport::for_machine(machine),
|
|
program: provider.program().to_string(),
|
|
max_loaded: provider.max_loaded.unwrap_or(DEFAULT_MAX_LOADED),
|
|
};
|
|
let mut routers = self.inner.lock().unwrap_or_else(|e| e.into_inner());
|
|
let router = routers.entry(key.clone()).or_insert_with(|| {
|
|
Arc::new(Router {
|
|
dir: self.dir.join(key.replace('/', "-")),
|
|
spec: Mutex::new(spec.clone()),
|
|
runtime: self.runtime.clone(),
|
|
gate: Mutex::new(()),
|
|
})
|
|
});
|
|
*router.spec.lock().unwrap_or_else(|e| e.into_inner()) = spec;
|
|
Arc::clone(router)
|
|
}
|
|
}
|
|
|
|
/// What it takes to start a router, as its machine currently describes it.
|
|
#[derive(Clone)]
|
|
struct Spec {
|
|
transport: Transport,
|
|
program: String,
|
|
max_loaded: u32,
|
|
}
|
|
|
|
pub struct Router {
|
|
/// Its record and log, on this machine whichever machine it serves from.
|
|
dir: PathBuf,
|
|
spec: Mutex<Spec>,
|
|
/// See [`Routers::runtime`].
|
|
runtime: Option<tokio::runtime::Handle>,
|
|
/// Held while a router is started and while the preset file is edited --
|
|
/// the two things that go wrong when two sessions do them at once. Never
|
|
/// held across a model load, which takes minutes.
|
|
gate: Mutex<()>,
|
|
}
|
|
|
|
impl Router {
|
|
/// Where its record and log are, for a session that wants to watch the
|
|
/// process it is talking through or read what it last said.
|
|
pub fn dir(&self) -> &Path {
|
|
&self.dir
|
|
}
|
|
|
|
/// Its process record, if one is running -- which is also how a session
|
|
/// records the process it reaches its model through.
|
|
pub fn record(&self) -> Option<process::Record> {
|
|
process::live(&self.dir)
|
|
}
|
|
|
|
/// The same process, recorded as one a session reaches but does not own.
|
|
///
|
|
/// The one place that conversion happens, so that a session directory can
|
|
/// never come to hold a record saying it owns the machine's router -- see
|
|
/// [`process::Detail::Shared`].
|
|
pub fn shared_record(&self) -> Option<process::Record> {
|
|
let record = self.record()?;
|
|
match record.detail {
|
|
process::Detail::Http { port } => Some(process::Record {
|
|
detail: process::Detail::Shared { port },
|
|
..record
|
|
}),
|
|
process::Detail::Stdio { .. } | process::Detail::Shared { .. } => None,
|
|
}
|
|
}
|
|
|
|
/// Where to reach it, or `None` when nothing is running.
|
|
pub fn endpoint(&self) -> Option<String> {
|
|
match self.record()?.detail {
|
|
process::Detail::Http { port } => Some(format!("http://127.0.0.1:{port}")),
|
|
process::Detail::Stdio { .. } | process::Detail::Shared { .. } => None,
|
|
}
|
|
}
|
|
|
|
/// Puts `model` in memory and says where to talk to it, starting the
|
|
/// router first if it is not up.
|
|
///
|
|
/// Blocking, and slow on purpose: a model that has to come off disk takes
|
|
/// as long as it takes. The caller is the driver's loading thread, which
|
|
/// is what [`super::Serving::Loading`] exists to describe.
|
|
///
|
|
/// Nothing is held across that wait. Two sessions load through one router,
|
|
/// and the second one wanting a model already in memory must not queue
|
|
/// behind the first one's cold load of a different model -- which is most
|
|
/// of what sharing a server was for.
|
|
pub fn load(
|
|
&self,
|
|
key: &str,
|
|
found: &Model,
|
|
settings: &BTreeMap<String, String>,
|
|
) -> Result<String> {
|
|
let endpoint = self.start_if_down()?;
|
|
self.describe_model(key, found, settings)?;
|
|
// Nothing to ask for where it is already in memory, which is the case
|
|
// this whole module exists to produce: a second session naming a model
|
|
// somebody else loaded is a round trip rather than a load. Asking
|
|
// anyway is not harmless -- `POST /models/load` answers **400** for a
|
|
// model that is already loaded, which arrived as a session that
|
|
// refused to start next to one happily using that same model.
|
|
if !self.is_ready(key) {
|
|
let asked = self
|
|
.post("/models/load", json!({ "model": key }))
|
|
.with_context(|| format!("asking llama-server to load {key}"));
|
|
match (asked, self.wait_loaded(key)) {
|
|
// Loaded, whatever the request said: something else may have
|
|
// asked for it in the meantime, and what is in memory is the
|
|
// answer rather than what one request made of being told to
|
|
// put it there.
|
|
(_, Ok(())) => {}
|
|
// It did not load, and a refusal of the request itself says
|
|
// more about why than "it never appeared" does.
|
|
(Err(refused), Err(_)) => return Err(refused),
|
|
(Ok(_), Err(never)) => return Err(never),
|
|
}
|
|
}
|
|
Ok(endpoint)
|
|
}
|
|
|
|
/// Writes what this model is and how to load it into the preset file, and
|
|
/// has the router re-read it.
|
|
///
|
|
/// Also the path a settings change takes, which is why it is separate from
|
|
/// [`load`](Self::load): a model whose entry has changed is **unloaded**
|
|
/// by the re-read, and that is the change taking effect rather than a
|
|
/// side effect -- the sessions using it load it again, with the new
|
|
/// settings, on their next message. What must not happen is the same
|
|
/// thing to an unrelated model, which is why the file is written and the
|
|
/// re-read asked for only when the text actually differs.
|
|
pub fn describe_model(
|
|
&self,
|
|
key: &str,
|
|
found: &Model,
|
|
settings: &BTreeMap<String, String>,
|
|
) -> Result<()> {
|
|
// Read, edit, write: under the gate because two of those at once lose
|
|
// one of the two sections.
|
|
let _one_at_a_time = self.gate.lock().unwrap_or_else(|e| e.into_inner());
|
|
let existing = self.preset()?;
|
|
let updated = upsert(&existing, key, §ion(found, settings));
|
|
if updated == existing {
|
|
return Ok(());
|
|
}
|
|
self.write_preset(&updated)?;
|
|
// Only meaningful against a running router; one that is down reads the
|
|
// file when it starts.
|
|
if self.endpoint().is_some() {
|
|
self.get("/models?reload=1")
|
|
.context("asking llama-server to re-read its models")?;
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// Whether this model is in memory now.
|
|
fn is_ready(&self, key: &str) -> bool {
|
|
self.loaded()
|
|
.into_iter()
|
|
.any(|model| model.model == key && model.ready)
|
|
}
|
|
|
|
/// Every model this router knows about and what each is doing, or an empty
|
|
/// list when it is not running.
|
|
pub fn loaded(&self) -> Vec<RouterModel> {
|
|
let Ok(answer) = self.get("/models") else {
|
|
return Vec::new();
|
|
};
|
|
answer
|
|
.get("data")
|
|
.and_then(Value::as_array)
|
|
.map(|models| models.iter().filter_map(RouterModel::read).collect())
|
|
.unwrap_or_default()
|
|
}
|
|
|
|
/// Takes one model out of memory, leaving the router and every other model
|
|
/// alone.
|
|
pub fn unload(&self, key: &str) -> Result<()> {
|
|
self.post("/models/unload", json!({ "model": key }))
|
|
.with_context(|| format!("asking llama-server to unload {key}"))?;
|
|
Ok(())
|
|
}
|
|
|
|
/// Ends the router and every model it is holding.
|
|
///
|
|
/// The only thing that does: a session closing, being deleted, or this
|
|
/// backend shutting down all leave it running. Sessions using it will see
|
|
/// their process go and report `exited`, which is true -- the model they
|
|
/// were talking to is no longer in memory.
|
|
///
|
|
/// Neither the record nor the mark is cleared here, and that is what tells
|
|
/// those sessions this was asked for rather than a crash. A session's
|
|
/// watcher looks a couple of seconds later, so anything removed now is
|
|
/// removed before the only reader of it has looked -- which is how the
|
|
/// first version of this put "llama-server exited" in three transcripts
|
|
/// belonging to somebody who had just pressed Stop. The record describes a
|
|
/// dead process, which every reader already handles, and starting a new
|
|
/// router is what clears both.
|
|
pub fn stop(&self) -> Result<()> {
|
|
let Some(record) = self.record() else {
|
|
return Ok(());
|
|
};
|
|
process::mark_stopping(&self.dir)?;
|
|
process::stop(&record, process::STOP_GRACE);
|
|
Ok(())
|
|
}
|
|
|
|
/// The end of the router's log, for a failure message.
|
|
///
|
|
/// Its account of what went wrong is the useful half -- "failed to load
|
|
/// model", "bind: Address already in use" -- and on a remote machine it is
|
|
/// the only half, since nobody reading the phone can open a file over
|
|
/// there. Bounded, because this ends up in an event a phone draws.
|
|
pub fn log_tail(&self) -> String {
|
|
let Ok(text) = std::fs::read_to_string(self.dir.join(LOG)) else {
|
|
return String::new();
|
|
};
|
|
let tail: Vec<&str> = text.lines().rev().take(LOG_TAIL_LINES).collect();
|
|
if tail.is_empty() {
|
|
return String::new();
|
|
}
|
|
format!(
|
|
" It last said: {}",
|
|
tail.into_iter().rev().collect::<Vec<_>>().join(" / ")
|
|
)
|
|
}
|
|
|
|
/// Adopts the running router or starts one, and waits for it to answer.
|
|
///
|
|
/// Under the gate, and asked again inside it: two sessions starting at once
|
|
/// would otherwise both find no record, both start a router, and bind two
|
|
/// ports to the same models.
|
|
fn start_if_down(&self) -> Result<String> {
|
|
if let Some(endpoint) = self.endpoint() {
|
|
return Ok(endpoint);
|
|
}
|
|
let _one_at_a_time = self.gate.lock().unwrap_or_else(|e| e.into_inner());
|
|
if let Some(endpoint) = self.endpoint() {
|
|
return Ok(endpoint);
|
|
}
|
|
let spec = self.spec.lock().unwrap_or_else(|e| e.into_inner()).clone();
|
|
wg_app_link::private::create_dir(&self.dir)?;
|
|
// Whatever the last router left behind, including the mark saying its
|
|
// end was asked for -- see [`stop`](Self::stop). From here on, a
|
|
// process that goes away is news.
|
|
process::clear(&self.dir);
|
|
// Written before the router starts, because it is read at startup and
|
|
// a router with no preset file at all lists nothing.
|
|
let preset = self.write_preset(&self.preset()?)?;
|
|
let forward = spec
|
|
.transport
|
|
.reserve_port()
|
|
.context("finding a port for llama-server")?;
|
|
let args = vec![
|
|
// Loopback there, whichever machine there is: what reaches it from
|
|
// outside that machine is the ssh tunnel and nothing else.
|
|
"--host".to_string(),
|
|
"127.0.0.1".to_string(),
|
|
"--port".to_string(),
|
|
forward.there.to_string(),
|
|
// No `-m`: a `llama-server` given no model is a router.
|
|
"--models-preset".to_string(),
|
|
preset,
|
|
"--models-max".to_string(),
|
|
spec.max_loaded.to_string(),
|
|
// The built-in agent tools -- read, search, edit, shell. Hosted by
|
|
// the router itself, which is what makes them one set for the
|
|
// machine rather than one per model. Which of them a *session*
|
|
// offers its model is decided here in the backend, per request, so
|
|
// there is nothing per-session to pass through: see `super::tools`.
|
|
"--tools".to_string(),
|
|
"all".to_string(),
|
|
];
|
|
let launch = Launch::new(&spec.program, args, None).reaching(forward);
|
|
// Starting and reaping both happen inside the runtime, though this is
|
|
// a blocking thread: tokio's `Command::spawn` registers the child with
|
|
// the reactor, so calling it outside a runtime context panics -- and
|
|
// that panic kills only this thread, leaving a session that says
|
|
// "loading" for ever with nothing in the log. See [`Routers::runtime`].
|
|
let _inside = self.runtime.as_ref().map(tokio::runtime::Handle::enter);
|
|
// Its output goes to a file, not a pipe. Not only so the process can
|
|
// outlive this server: nothing ever read those pipes, so a chatty
|
|
// llama-server filled the 64 KB buffer and blocked with no sign of why.
|
|
let child = spec.transport.spawn(
|
|
&launch,
|
|
Streams::Detached {
|
|
stdin: std::process::Stdio::null(),
|
|
stdout: log_file(&self.dir.join(LOG))?.into(),
|
|
stderr: log_file(&self.dir.join(LOG))?.into(),
|
|
},
|
|
)?;
|
|
let pid = child
|
|
.id()
|
|
.context("llama-server exited before it could be recorded")?;
|
|
tracing::info!(
|
|
"running {} in router mode {} on 127.0.0.1:{} there, reached at 127.0.0.1:{} here, \
|
|
as pid {pid}",
|
|
spec.program,
|
|
spec.transport.describe(),
|
|
forward.there,
|
|
forward.here,
|
|
);
|
|
// Reaped so it does not become a zombie while this server is still
|
|
// its parent; the record and the health poll are what say whether it
|
|
// is alive, because after a restart there is no `Child` to ask.
|
|
if let Some(runtime) = &self.runtime {
|
|
runtime.spawn(async move {
|
|
let mut child = child;
|
|
let _ = child.wait().await;
|
|
});
|
|
}
|
|
// The *near* port, because that is the one anything reaching this
|
|
// router has to dial -- including a later run of this backend, which
|
|
// adopts the record without knowing which machine it is on. For a
|
|
// remote machine the recorded pid is the ssh client's, which is the
|
|
// process this machine owns and which holds the tunnel open for
|
|
// exactly as long as the far router lives.
|
|
let record = process::Record::of(pid, process::Detail::Http { port: forward.here })
|
|
.context("llama-server was gone before its start time could be read")?;
|
|
process::write(&self.dir, &record);
|
|
let endpoint = format!("http://127.0.0.1:{}", forward.here);
|
|
self.wait_answering(&endpoint)?;
|
|
Ok(endpoint)
|
|
}
|
|
|
|
/// Polls until the router answers, watching the process as well as the
|
|
/// port: a port already taken, or a `llama-server` too old for one of
|
|
/// these flags, exits within a second and would otherwise be waited out.
|
|
fn wait_answering(&self, endpoint: &str) -> Result<()> {
|
|
let deadline = std::time::Instant::now() + START_TIMEOUT;
|
|
let url = format!("{endpoint}/health");
|
|
loop {
|
|
if let Ok(response) = ureq::get(&url).call()
|
|
&& response.status() == 200
|
|
{
|
|
return Ok(());
|
|
}
|
|
match process::recorded(&self.dir) {
|
|
Some((_, process::Liveness::Alive | process::Liveness::Unknown)) => {}
|
|
Some((_, process::Liveness::Dead)) | None => {
|
|
process::clear(&self.dir);
|
|
bail!("llama-server exited before it answered.{}", self.log_tail());
|
|
}
|
|
}
|
|
if std::time::Instant::now() > deadline {
|
|
process::clear(&self.dir);
|
|
bail!(
|
|
"llama-server did not answer within {}s.{}",
|
|
START_TIMEOUT.as_secs(),
|
|
self.log_tail()
|
|
);
|
|
}
|
|
std::thread::sleep(POLL);
|
|
}
|
|
}
|
|
|
|
/// Polls until the named model is in memory, or says why it will never be.
|
|
///
|
|
/// A model that will not load is the common failure and it is fast: the
|
|
/// child exits, the router reports it unloaded with an exit code, and this
|
|
/// says so rather than waiting out the timeout -- which is what turned "it
|
|
/// said the file was corrupt" into "gave up after 300s".
|
|
fn wait_loaded(&self, key: &str) -> Result<()> {
|
|
let deadline = std::time::Instant::now() + LOAD_TIMEOUT;
|
|
loop {
|
|
match self.loaded().into_iter().find(|model| model.model == key) {
|
|
Some(model) if model.ready => return Ok(()),
|
|
Some(model) if model.failed => {
|
|
bail!("{key} would not load.{}", self.log_tail())
|
|
}
|
|
// Still loading, or not listed yet after a reload.
|
|
Some(_) | None => {}
|
|
}
|
|
if process::live(&self.dir).is_none() {
|
|
bail!(
|
|
"llama-server went away while loading {key}.{}",
|
|
self.log_tail()
|
|
);
|
|
}
|
|
if std::time::Instant::now() > deadline {
|
|
bail!(
|
|
"{key} was still loading after {}s.{}",
|
|
LOAD_TIMEOUT.as_secs(),
|
|
self.log_tail()
|
|
);
|
|
}
|
|
std::thread::sleep(POLL);
|
|
}
|
|
}
|
|
|
|
/// One request to the router's management endpoints. The generation ones
|
|
/// are the driver's and stream, so they are not these.
|
|
fn get(&self, path: &str) -> Result<Value> {
|
|
let url = format!("{}{path}", self.answering()?);
|
|
Self::read(
|
|
ureq::get(&url)
|
|
.call()
|
|
.with_context(|| format!("GET {path}"))?,
|
|
path,
|
|
)
|
|
}
|
|
|
|
fn post(&self, path: &str, body: Value) -> Result<Value> {
|
|
let url = format!("{}{path}", self.answering()?);
|
|
Self::read(
|
|
ureq::post(&url)
|
|
.send_json(body)
|
|
.with_context(|| format!("POST {path}"))?,
|
|
path,
|
|
)
|
|
}
|
|
|
|
/// Where to reach a router that is running, or the failure saying it is
|
|
/// not -- which is what every one of these requests needs first.
|
|
fn answering(&self) -> Result<String> {
|
|
self.endpoint().context("no llama-server is running")
|
|
}
|
|
|
|
fn read(mut response: ureq::http::Response<ureq::Body>, path: &str) -> Result<Value> {
|
|
response
|
|
.body_mut()
|
|
.read_json()
|
|
.with_context(|| format!("reading what {path} answered"))
|
|
}
|
|
|
|
/// The preset file as it stands on the machine that serves the models, or
|
|
/// empty where there is none yet.
|
|
///
|
|
/// Read back rather than remembered, for one reason that matters after a
|
|
/// restart: a router adopted from a previous run is already serving models
|
|
/// whose sections this process has never seen, and rewriting the file
|
|
/// without them would unload them at the next reload.
|
|
fn preset(&self) -> Result<String> {
|
|
let spec = self.spec.lock().unwrap_or_else(|e| e.into_inner()).clone();
|
|
let text = match &spec.transport {
|
|
Transport::Here => {
|
|
Ok(std::fs::read_to_string(self.dir.join(PRESET)).unwrap_or_default())
|
|
}
|
|
Transport::Ssh { name, .. } => {
|
|
let script = format!("p={REMOTE_PRESET}; cat \"$p\" 2>/dev/null || true");
|
|
let launch = Launch::new("sh", vec!["-c".to_string(), script], None);
|
|
spec.transport
|
|
.capture_blocking(&launch)
|
|
.with_context(|| format!("reading the model settings on {name}"))
|
|
}
|
|
}?;
|
|
// A file that is not there yet reads as a new one rather than as
|
|
// nothing: `llama-server` refuses a preset with no version line, so
|
|
// "empty" is not a state this can hand back or write.
|
|
Ok(if text.trim().is_empty() {
|
|
VERSION.to_string()
|
|
} else {
|
|
text
|
|
})
|
|
}
|
|
|
|
/// Writes the preset file where the router will read it, and says where
|
|
/// that is -- which is the path the router is given, so the two cannot
|
|
/// disagree.
|
|
fn write_preset(&self, text: &str) -> Result<String> {
|
|
let spec = self.spec.lock().unwrap_or_else(|e| e.into_inner()).clone();
|
|
match &spec.transport {
|
|
Transport::Here => {
|
|
let path = self.dir.join(PRESET);
|
|
wg_app_link::private::write_file(&path, text.as_bytes())?;
|
|
Ok(path.to_string_lossy().into_owned())
|
|
}
|
|
Transport::Ssh { name, .. } => {
|
|
use base64::Engine as _;
|
|
// Base64 rather than a heredoc: the text goes through a shell
|
|
// on the far side, and an INI value is not something to trust
|
|
// to quoting rules twice over.
|
|
let encoded = base64::engine::general_purpose::STANDARD.encode(text);
|
|
let script = format!(
|
|
"p={REMOTE_PRESET}; mkdir -p \"$(dirname \"$p\")\" && \
|
|
printf %s \"$1\" | base64 -d > \"$p\" && printf '%s\\n' \"$p\""
|
|
);
|
|
let launch = Launch::new(
|
|
"sh",
|
|
vec!["-c".to_string(), script, "sh".to_string(), encoded],
|
|
None,
|
|
);
|
|
let answer = spec
|
|
.transport
|
|
.capture_blocking(&launch)
|
|
.with_context(|| format!("writing the model settings on {name}"))?;
|
|
match answer.trim() {
|
|
"" => bail!("{name} did not say where it wrote the model settings"),
|
|
path => Ok(path.to_string()),
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Where the preset file goes on a machine that is not this one, as a shell
|
|
/// word the far side expands: under that machine's state directory, beside
|
|
/// whatever else belongs to this app there.
|
|
///
|
|
/// `$HOME` is resolved over there because only that machine knows what it is.
|
|
const REMOTE_PRESET: &str = "\"${XDG_STATE_HOME:-$HOME/.local/state}/ai-app/llama-models.ini\"";
|
|
|
|
/// One model the router knows about, as the provider view and the load poll
|
|
/// both read it.
|
|
#[derive(Debug, Clone, Serialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
pub struct RouterModel {
|
|
pub model: String,
|
|
/// The router's own word: `unloaded`, `loading`, `loaded`, `sleeping`.
|
|
/// Carried through rather than reduced to a boolean, because the phone
|
|
/// draws it and llama.cpp is the authority on what states there are.
|
|
pub status: String,
|
|
pub ready: bool,
|
|
/// Unloaded *and* something went wrong, which is not the same as unloaded.
|
|
#[serde(skip)]
|
|
pub failed: bool,
|
|
}
|
|
|
|
impl RouterModel {
|
|
fn read(entry: &Value) -> Option<Self> {
|
|
let model = entry.get("id")?.as_str()?.to_string();
|
|
let status = entry
|
|
.pointer("/status/value")
|
|
.and_then(Value::as_str)
|
|
.unwrap_or("unknown")
|
|
.to_string();
|
|
let exited = entry
|
|
.get("exit_code")
|
|
.and_then(Value::as_i64)
|
|
.unwrap_or_default();
|
|
Some(Self {
|
|
ready: status == "loaded" || status == "sleeping",
|
|
failed: status == "unloaded" && exited != 0,
|
|
model,
|
|
status,
|
|
})
|
|
}
|
|
}
|
|
|
|
/// What a model's section says, from the file it is and the settings it has.
|
|
///
|
|
/// The keys are `llama-server`'s own argument names without their dashes,
|
|
/// which is what a preset section is: `--n-gpu-layers 20` is
|
|
/// `n-gpu-layers = 20`. So adding a setting is a row here and a row in
|
|
/// [`crate::config::LLAMA_MODEL_PARAMS`], and nothing in between.
|
|
fn section(found: &Model, settings: &BTreeMap<String, String>) -> String {
|
|
let mut lines = vec![format!("model = {}", found.path)];
|
|
for (key, flag) in [
|
|
("contextSize", "ctx-size"),
|
|
("gpuLayers", "n-gpu-layers"),
|
|
("threads", "threads"),
|
|
// How far ahead the draft head guesses. Not defaulted: 2 measured 7%
|
|
// faster than llama.cpp's 3 on this machine's GPU, once, which is a
|
|
// reason to make the knob reachable and not a reason to move it for
|
|
// everybody.
|
|
("specDraftNMax", "spec-draft-n-max"),
|
|
] {
|
|
if let Some(value) = settings.get(key).map(|value| value.trim())
|
|
&& !value.is_empty()
|
|
{
|
|
lines.push(format!("{flag} = {value}"));
|
|
}
|
|
}
|
|
// One slot unless this model is told otherwise. A session is one
|
|
// conversation making one request at a time, and a second session's turn
|
|
// waits rather than splitting the cache: measured 2026-09-19 on the 27B
|
|
// here, 41.5 tok/s plain at any slot count, **61.4** with the draft head
|
|
// at one slot, and **28** with the head at four. Speculating against a
|
|
// split KV cache is slower than not speculating at all.
|
|
let slots = settings
|
|
.get("slots")
|
|
.map(|value| value.trim())
|
|
.filter(|value| !value.is_empty())
|
|
.unwrap_or("1");
|
|
lines.push(format!("parallel = {slots}"));
|
|
// A model carrying a multi-token-prediction head drafts with it, which is
|
|
// most of a 50% speed-up for free -- the tensors are in the file whether
|
|
// or not they are used. Conditional because it cannot be otherwise: asked
|
|
// for on a model without one, `llama-server` **exits** ("context type MTP
|
|
// requested but model doesn't contain MTP layers"). See `Model::mtp`.
|
|
if found.mtp && settings.get("speculative").map(String::as_str) != Some("off") {
|
|
lines.push("spec-type = draft-mtp".to_string());
|
|
}
|
|
// What makes a model able to read pictures, and the one flag here that is
|
|
// found rather than defaulted: the projector is a second file published
|
|
// beside the weights, so a model that has one is loaded with it unless
|
|
// this model's settings name another or turn it off.
|
|
if let Some(projector) = projector(found, settings) {
|
|
lines.push(format!("mmproj = {projector}"));
|
|
}
|
|
lines.join("\n")
|
|
}
|
|
|
|
/// Which projector this model is loaded with: what its settings say, else
|
|
/// whatever was found beside it, and nothing for `"off"`.
|
|
///
|
|
/// A setting naming a bare file name means one in the model's own directory,
|
|
/// since that is where the alternatives to the file found there are; anything
|
|
/// with a `/` in it is taken as the path it is, absolute or not -- the serving
|
|
/// machine resolves it, and this side does not know its working directory.
|
|
fn projector(found: &Model, settings: &BTreeMap<String, String>) -> Option<String> {
|
|
let chosen = settings.get("mmproj").map(|value| value.trim());
|
|
match chosen {
|
|
Some("off") => None,
|
|
Some("") => found.mmproj.clone(),
|
|
Some(name) if name.contains('/') => Some(name.to_string()),
|
|
Some(name) => {
|
|
let dir = found.path.rsplit_once('/').map_or("", |(dir, _)| dir);
|
|
Some(format!("{dir}/{name}"))
|
|
}
|
|
None => found.mmproj.clone(),
|
|
}
|
|
}
|
|
|
|
/// The preset file with `name`'s section replaced by `body`, added at the end
|
|
/// if it was not there.
|
|
///
|
|
/// Text in and text out, rather than a parsed model, because the file belongs
|
|
/// to `llama-server` rather than to this: anything in it that this does not
|
|
/// understand -- a `[*]` section, a key added by a later version, a comment
|
|
/// somebody wrote -- has to survive being edited.
|
|
fn upsert(existing: &str, name: &str, body: &str) -> String {
|
|
let header = format!("[{name}]");
|
|
let mut out = String::new();
|
|
let mut skipping = false;
|
|
let mut replaced = false;
|
|
for line in existing.lines() {
|
|
let trimmed = line.trim();
|
|
if trimmed.starts_with('[') && trimmed.ends_with(']') {
|
|
skipping = trimmed == header;
|
|
if skipping {
|
|
replaced = true;
|
|
push_section(&mut out, &header, body);
|
|
continue;
|
|
}
|
|
}
|
|
if !skipping {
|
|
out.push_str(line);
|
|
out.push('\n');
|
|
}
|
|
}
|
|
if out.is_empty() {
|
|
out.push_str(VERSION);
|
|
}
|
|
if !replaced {
|
|
push_section(&mut out, &header, body);
|
|
}
|
|
out
|
|
}
|
|
|
|
fn push_section(out: &mut String, header: &str, body: &str) {
|
|
if !out.ends_with("\n\n") && !out.is_empty() {
|
|
out.push('\n');
|
|
}
|
|
out.push_str(header);
|
|
out.push('\n');
|
|
out.push_str(body.trim_end());
|
|
out.push_str("\n\n");
|
|
}
|
|
|
|
/// An owner-only log opened for appending, so the two streams pointed at it do
|
|
/// not overwrite each other and an adopted router keeps what came before.
|
|
fn log_file(path: &Path) -> Result<std::fs::File> {
|
|
use std::os::unix::fs::OpenOptionsExt;
|
|
std::fs::OpenOptions::new()
|
|
.create(true)
|
|
.append(true)
|
|
.mode(0o600)
|
|
.open(path)
|
|
.with_context(|| format!("opening {}", path.display()))
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
fn settings(pairs: &[(&str, &str)]) -> BTreeMap<String, String> {
|
|
pairs
|
|
.iter()
|
|
.map(|(key, value)| ((*key).to_string(), (*value).to_string()))
|
|
.collect()
|
|
}
|
|
|
|
#[test]
|
|
fn a_section_names_the_file_and_the_flags_that_were_set() {
|
|
let found = Model {
|
|
mmproj: None,
|
|
path: "/models/a.gguf".to_string(),
|
|
mtp: true,
|
|
};
|
|
let text = section(
|
|
&found,
|
|
&settings(&[("contextSize", "8192"), ("threads", " 6 ")]),
|
|
);
|
|
assert_eq!(
|
|
text,
|
|
"model = /models/a.gguf\nctx-size = 8192\nthreads = 6\nparallel = 1\n\
|
|
spec-type = draft-mtp"
|
|
);
|
|
|
|
// A blank is not a value: it is the setting being unset, and passing
|
|
// it on is a child that exits on an empty argument.
|
|
let text = section(&found, &settings(&[("contextSize", " ")]));
|
|
assert!(!text.contains("ctx-size"), "{text}");
|
|
|
|
// The draft head is asked for only where the file has one, and can be
|
|
// turned off for a machine where it does not pay.
|
|
let plain = Model {
|
|
mtp: false,
|
|
..found.clone()
|
|
};
|
|
assert!(!section(&plain, &settings(&[])).contains("spec-type"));
|
|
assert!(!section(&found, &settings(&[("speculative", "off")])).contains("spec-type"));
|
|
}
|
|
|
|
/// The projector found beside a model is loaded with it; the setting names
|
|
/// another where a repository published several, or turns it off.
|
|
#[test]
|
|
fn a_vision_model_is_loaded_with_its_projector() {
|
|
let found = Model {
|
|
path: "/models/repo/a.gguf".to_string(),
|
|
mtp: false,
|
|
mmproj: Some("/models/repo/mmproj-F16.gguf".to_string()),
|
|
};
|
|
let line = |settings: &[(&str, &str)]| {
|
|
section(&found, &self::settings(settings))
|
|
.lines()
|
|
.find_map(|line| line.strip_prefix("mmproj = "))
|
|
.map(str::to_string)
|
|
};
|
|
assert_eq!(line(&[]), Some("/models/repo/mmproj-F16.gguf".to_string()));
|
|
assert_eq!(
|
|
line(&[("mmproj", " ")]),
|
|
Some("/models/repo/mmproj-F16.gguf".to_string())
|
|
);
|
|
assert_eq!(line(&[("mmproj", "off")]), None);
|
|
// A bare name is one of the model's own neighbours; anything with a
|
|
// separator in it is the path it says it is.
|
|
assert_eq!(
|
|
line(&[("mmproj", "mmproj-F32.gguf")]),
|
|
Some("/models/repo/mmproj-F32.gguf".to_string()),
|
|
);
|
|
assert_eq!(
|
|
line(&[("mmproj", "/elsewhere/p.gguf")]),
|
|
Some("/elsewhere/p.gguf".to_string()),
|
|
);
|
|
// A model with none, and nothing asked for, loads without one.
|
|
assert!(
|
|
!section(
|
|
&Model {
|
|
mmproj: None,
|
|
..found.clone()
|
|
},
|
|
&settings(&[])
|
|
)
|
|
.contains("mmproj")
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn a_section_replaces_its_own_and_leaves_every_other_line_alone() {
|
|
let first = upsert("", "repo/a.gguf", "model = /models/a.gguf");
|
|
assert!(first.starts_with("version = 1\n"), "{first}");
|
|
assert!(
|
|
first.contains("[repo/a.gguf]\nmodel = /models/a.gguf\n"),
|
|
"{first}"
|
|
);
|
|
|
|
// A second model is added rather than replacing the first, because a
|
|
// reload of a file that lost a section unloads that model -- which
|
|
// would be one session taking another's model out of memory.
|
|
let both = upsert(&first, "repo/b.gguf", "model = /models/b.gguf");
|
|
assert!(both.contains("[repo/a.gguf]"), "{both}");
|
|
assert!(both.contains("[repo/b.gguf]"), "{both}");
|
|
|
|
// Editing one rewrites only its own keys, and keeps what llama.cpp's
|
|
// own file has that this does not know about.
|
|
let with_global = format!(
|
|
"version = 1\n\n[*]\njinja = true\n\n{}",
|
|
both.trim_start_matches("version = 1\n")
|
|
);
|
|
let edited = upsert(
|
|
&with_global,
|
|
"repo/a.gguf",
|
|
"model = /models/a.gguf\nctx-size = 4096",
|
|
);
|
|
assert!(edited.contains("[*]\njinja = true"), "{edited}");
|
|
assert!(edited.contains("ctx-size = 4096"), "{edited}");
|
|
assert_eq!(edited.matches("[repo/a.gguf]").count(), 1, "{edited}");
|
|
assert!(
|
|
edited.contains("[repo/b.gguf]\nmodel = /models/b.gguf"),
|
|
"{edited}"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn writing_the_same_settings_twice_changes_nothing() {
|
|
// What keeps an unrelated session's model in memory: the file is only
|
|
// written, and the router only told to re-read it, when the text
|
|
// actually differs.
|
|
let once = upsert("", "repo/a.gguf", "model = /models/a.gguf");
|
|
assert_eq!(upsert(&once, "repo/a.gguf", "model = /models/a.gguf"), once);
|
|
}
|
|
}
|