Serve a machine's models from one shared llama-server

A llama.cpp session had its own `llama-server`: two sessions on one model
held two copies of it in memory, a model change bought a load only that
session benefited from, and the process was a session's to end. A machine's
models are now served by one `llama-server` in **router mode** -- no `-m`,
a preset file naming models and their flags, a child server per model asked
for, and each request routed by its `model` field. So one server per model
with that model's own settings is what a machine runs, while this backend
has one process, one port and one record per machine to keep track of.

The record is the mechanism every other driver already uses, so a restart
adopts it; a session records the same pid in its own directory as
`Detail::Shared`, and `process::signal` refuses to signal one of those --
which is what keeps stopping, deleting or cleaning up after one session
from unloading a model every other session is using. Nothing stops a router
on its own. That is deliberate (a loaded model is minutes of disk) and it is
why the machines tab now has a card per provider that opens its own screen:
how each model is loaded, how many stay in memory, Unload, and Stop.

How a model is *loaded* therefore belongs to the model on its machine rather
than to a session -- context size, GPU layers, threads, slots, speculative
decoding -- written into the preset as llama-server's own argument names.
Saving them re-reads that file, which unloads the model; that is the change
taking effect, and the dialog says so before you save. What stays a
session's is everything that rides on a request, including which tools it
offers: the router hosts one set for the machine and the choice is a filter
applied here, so it costs no reload (2,181 tokens of prompt with all seven,
698 with none).

Verified end to end against the scratch backend and the emulator: two
sessions sharing one loaded model with one child process, a second session
joining it with a 26ms prefill, a backend restart adopting the router and
answering with the prompt cache intact, the same over ssh to this VM, a
model's settings reaching the running server, Unload, and Stop leaving every
session `exited` with no error line.

server/src/session/mod.rs

@@ -20,7 +20,7 @@ pub mod subagent;
 pub mod transcript;
 pub mod transport;
 
-use std::collections::{HashMap, VecDeque};
+use std::collections::{BTreeMap, HashMap, VecDeque};
 use std::path::{Path, PathBuf};
 use std::sync::{Arc, Mutex, RwLock};
 use std::time::{SystemTime, UNIX_EPOCH};
@@ -45,6 +45,11 @@ use subagent::Subagents;
 use transcript::{SeqEvent, Transcript};
 use transport::Transport;
 
+/// Where each machine's shared `llama-server` keeps its record, log and model
+/// settings: one directory beside the session directories, since a session id
+/// is what everything in there is named by and a router is not a session.
+const ROUTERS: &str = "routers";
+
 /// Fan-out buffer per session. A subscriber further behind than this is
 /// caught up from the transcript file instead, so the size only bounds
 /// memory, not correctness.
@@ -644,6 +649,10 @@ pub struct SessionManager {
     /// every echo driver this manager builds is handed a clone -- see
     /// [`SessionManager::reporting_usage_fixture`].
     usage_fixture: crate::usage::Fixture,
+    /// Every machine's shared `llama-server` -- see [`llama::router`]. On the
+    /// manager because the sharing is the point: a registry each session
+    /// carried its own copy of would be one router per session again.
+    routers: Arc<llama::router::Routers>,
     inner: RwLock<Inner>,
 }
 
@@ -701,6 +710,9 @@ impl SessionManager {
         // this manager builds gets a clone, including the ones built
         // below, so it has to exist before the first session does.
         let usage_fixture = crate::usage::Fixture::new();
+        // Beside the session directories rather than inside one, because a
+        // router belongs to a machine and is shared by every session on it.
+        let routers = Arc::new(llama::router::Routers::new(data_dir.join(ROUTERS)));
         let mut live = HashMap::new();
         for meta in &config.sessions {
             // One unlaunchable session -- a corrupt transcript, an
@@ -715,6 +727,7 @@ impl SessionManager {
                         data_dir: &data_dir,
                         models_dir: &models_dir,
                         usage: &usage_fixture,
+                        routers: &routers,
                     },
                     announce.clone(),
                     // Nothing is started here; see `Launching`.
@@ -737,6 +750,7 @@ impl SessionManager {
             pending: Arc::new(pending::Registry::default()),
             spawn_throwaway: false,
             usage_fixture,
+            routers,
             inner: RwLock::new(Inner { config, live }),
         };
         Ok(manager)
@@ -756,6 +770,7 @@ impl SessionManager {
             data_dir: &self.data_dir,
             models_dir: &self.models_dir,
             usage: &self.usage_fixture,
+            routers: &self.routers,
         }
     }
 
@@ -898,7 +913,17 @@ impl SessionManager {
             if let Some(name) = name {
                 machine.name = name.trim().to_string();
             }
-            if let Some(providers) = providers {
+            if let Some(mut providers) = providers {
+                // A re-probe answers "what is installed here", which is not an
+                // answer about how this machine's models are loaded: settings
+                // kept against a provider survive it, by name. Without this,
+                // pressing Rediscover silently emptied every model's settings.
+                for provider in &mut providers {
+                    if let Some(old) = machine.provider(&provider.name) {
+                        provider.model_settings = old.model_settings.clone();
+                        provider.max_loaded = old.max_loaded;
+                    }
+                }
                 machine.providers = providers;
             }
             Ok(machine.clone())
@@ -1182,6 +1207,72 @@ impl SessionManager {
         self.inner.read().unwrap().config.machines.clone()
     }
 
+    /// The shared `llama-server` behind a machine's provider, for a provider
+    /// that has one.
+    ///
+    /// `None` for every other kind rather than an empty router, because "this
+    /// provider has no server of its own to look at" is what the machine's
+    /// provider view has to say, and a router that exists but is never running
+    /// says something else.
+    pub fn router_for(
+        &self,
+        machine: &MachineConfig,
+        provider: &ProviderConfig,
+    ) -> Option<Arc<llama::router::Router>> {
+        match provider.kind {
+            DriverKind::LlamaCpp => Some(self.routers.of(machine, provider)),
+            DriverKind::Echo | DriverKind::ClaudeCli | DriverKind::CodexCli => None,
+        }
+    }
+
+    /// Records how a machine loads one of its models, or how many it keeps
+    /// loaded at once.
+    ///
+    /// One function for two routes because it is one edit to one provider
+    /// entry, and because the alternative is two `update` closures that have
+    /// to find the same provider the same way. Storage only: telling the
+    /// running server is `llama::apply_model_settings`, deliberately after
+    /// this, so a save that reached the config is one the phone can rely on
+    /// having made whether or not the machine could be reached.
+    pub fn set_provider_settings(
+        &self,
+        machine_id: &str,
+        provider_name: &str,
+        max_loaded: Option<Option<u32>>,
+        model: Option<(String, BTreeMap<String, String>)>,
+    ) -> Result<MachineConfig> {
+        self.update(|config| {
+            let machine = config
+                .machines
+                .iter_mut()
+                .find(|machine| machine.id == machine_id)
+                .with_context(|| format!("no machine with id \"{machine_id}\""))?;
+            let provider = machine
+                .providers
+                .iter_mut()
+                .find(|provider| provider.name == provider_name)
+                .with_context(|| format!("no provider \"{provider_name}\" on that machine"))?;
+            if let Some(max_loaded) = max_loaded {
+                provider.max_loaded = max_loaded;
+            }
+            if let Some((model, params)) = model {
+                // Removed rather than stored empty: an entry of nothing and no
+                // entry mean the same thing, and only one of them leaves the
+                // config file describing models nobody has settings for.
+                let params: BTreeMap<String, String> = params
+                    .into_iter()
+                    .filter(|(_, value)| !value.trim().is_empty())
+                    .collect();
+                if params.is_empty() {
+                    provider.model_settings.remove(&model);
+                } else {
+                    provider.model_settings.insert(model, params);
+                }
+            }
+            Ok(machine.clone())
+        })
+    }
+
     pub fn spawn_session(&self, spec: SpawnSpec) -> Result<SessionInfo> {
         self.spawn_seeded(spec, None)
     }
@@ -2362,6 +2453,9 @@ struct Env<'a> {
     data_dir: &'a Path,
     models_dir: &'a Path,
     usage: &'a crate::usage::Fixture,
+    /// Every machine's shared `llama-server`, so two sessions on one machine
+    /// reach one process -- see [`llama::router`].
+    routers: &'a Arc<llama::router::Routers>,
 }
 
 /// Creates the session directory, opens its transcript (continuing the
@@ -2590,6 +2684,7 @@ fn make_driver(
             provider,
             &Transport::for_machine(machine),
             env.models_dir,
+            env.routers.of(machine, provider),
             transcript_path,
             dir,
             sink.clone(),
@@ -4043,6 +4138,8 @@ mod tests {
                     command: Some(command.to_string_lossy().into_owned()),
                     models: Vec::new(),
                     mcp_servers: Vec::new(),
+                    model_settings: BTreeMap::new(),
+                    max_loaded: None,
                 },
             ])],
             ..Config::default()