Give llama.cpp sessions tools, web search and a model picker

A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.

Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.

Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.

Also here, because tools made each of them matter:

- **Loading is a state.** A 12 GB model takes twenty seconds to reach
  memory and refuses everything until it has; the session used to report
  `running` for that whole time, and a message sent meanwhile came back as
  an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
  stops it and starts another. The conversation survives because it was
  never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
  itself -- over ssh too, in the round trip the spawn was already making.
  Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
  the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
  slot, and 28 with it at four -- speculating against a split KV cache is
  worse than not speculating. The flag is conditional because asking for a
  head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
  an overrun context is now ordinary; it was "http status: 400" and is now
  the server's own "exceeds the available context size, try increasing it".

`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.

Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
iris-aiandClaude Opus 5 committed 2026-09-19 08:11:49 -04:00
1 parent 392cc5413d
commit ac476ab0c9
23 files changed
+3627 -1096

No files matched your search

+103 -12
View File
@@ -50,6 +50,71 @@ pub struct LocalModel {
pub repo: String,
pub file: String,
pub bytes: u64,
/// What the file says it is called (`general.name` in its own metadata),
/// absent when it does not say or could not be read. Not a label: see
/// [`labels`] for what a reader is actually shown, which needs the rest of
/// the list to decide.
#[serde(skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
}
/// What each of these models should be called on screen, in the same order.
///
/// A model's own name is the best answer and is not always an answer at all:
/// two quantisations of one model carry the same `general.name`, and a chip
/// row with two identical chips is one you cannot choose from. So this is a
/// cascade -- the model's own name, else its file name, else its full key --
/// and each model takes the first rung that nothing else on this machine
/// shares. The last rung always terminates it, because the key is what makes
/// these unique in the first place.
///
/// Decided over the whole list rather than per model because ambiguity is a
/// property of the set: the same file is unambiguous on a machine holding one
/// quantisation and not on a machine holding three, and only the list knows
/// which machine this is.
pub fn labels(models: &[LocalModel]) -> Vec<String> {
let rungs = |model: &LocalModel| {
[
model.name.clone(),
Some(model.file.trim_end_matches(".gguf").to_string()),
Some(model.key.clone()),
]
};
let mut taken: Vec<HashMap<String, usize>> = vec![HashMap::new(); 3];
for model in models {
for (rung, candidate) in rungs(model).into_iter().enumerate() {
if let Some(candidate) = candidate {
*taken[rung].entry(candidate).or_insert(0) += 1;
}
}
}
models
.iter()
.map(|model| {
rungs(model)
.into_iter()
.enumerate()
.find_map(|(rung, candidate)| {
let candidate = candidate?;
(taken[rung].get(&candidate) == Some(&1)).then_some(candidate)
})
// Unreachable: the key rung is unique by construction. Said as
// the key rather than as a panic, because a duplicate key would
// mean the same file listed twice and a name is still the
// honest thing to draw for it.
.unwrap_or_else(|| model.key.clone())
})
.collect()
}
/// The model's own name, read out of the file itself.
///
/// Absent for every way of not finding out -- see [`crate::gguf`]. The file is
/// opened and read only as far as the name, which is the first few hundred
/// bytes, so this is affordable once per model per listing.
fn name_of(path: &Path) -> Option<String> {
let mut file = std::fs::File::open(path).ok()?;
crate::gguf::name(&mut file)
}
/// What a run is doing, or did. Flat rather than a tagged enum carrying its
@@ -519,6 +584,7 @@ fn collect(root: &Path, dir: &Path, found: &mut Vec<LocalModel>) {
repo: repo.to_string(),
file: file.to_string(),
bytes: entry.metadata().map(|m| m.len()).unwrap_or(0),
name: name_of(&path),
});
}
}
@@ -566,33 +632,46 @@ pub fn dir_on(transport: &Transport, local: &Path) -> String {
/// directory that is not there is an empty list rather than a failure: a
/// machine that has never had a model put on it is an ordinary state, and
/// the same one as a machine whose directory exists and is empty.
///
/// Each record carries the head of the file as well as its size, because a
/// model's own name is inside it (see [`crate::gguf`]) and the file is on the
/// far machine. The alternative is a second round trip per model, or naming
/// remote models by path while local ones get their proper names -- one
/// machine's models reading differently from another's is exactly the
/// confusion the name was added to remove. The prefix is bounded at
/// [`crate::gguf::PREFIX_BYTES`], which is what keeps this one round trip's
/// worth of bytes.
pub async fn on_machine(transport: &Transport, dir: &str) -> Result<Vec<LocalModel>> {
let script = "p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${p#\"~/\"};; esac; \
[ -d \"$p\" ] || exit 0; \
find \"$p\" -type f -name '*.gguf' -printf '%s\\t%P\\0'";
let script = format!(
"p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${{p#\"~/\"}};; esac; \
[ -d \"$p\" ] || exit 0; cd \"$p\" || exit 0; \
find . -type f -name '*.gguf' -exec sh -c '\
for f do printf \"%s\\t%s\\t%s\\0\" \"$(wc -c < \"$f\")\" \
\"$(head -c {prefix} \"$f\" | base64 | tr -d \"\\n\")\" \"${{f#./}}\"; done\
' sh {{}} +",
prefix = crate::gguf::PREFIX_BYTES,
);
let launch = Launch::new(
"sh",
vec![
"-c".to_string(),
script.to_string(),
"sh".to_string(),
dir.to_string(),
],
vec!["-c".to_string(), script, "sh".to_string(), dir.to_string()],
None,
);
let out = transport.capture(&launch).await?;
let mut found: Vec<LocalModel> = out
.split('\0')
.filter(|record| !record.is_empty())
// Two fields, and the name last, so a `\t` in a filename survives.
.filter_map(|record| record.split_once('\t'))
.filter_map(|(bytes, key)| {
// Three fields, and the name last, so a `\t` in a filename survives.
// The middle one is base64, which has no tab in its alphabet.
.filter_map(|record| {
let (bytes, rest) = record.split_once('\t')?;
let (head, key) = rest.split_once('\t')?;
let (repo, file) = key.rsplit_once('/')?;
Some(LocalModel {
key: key.to_string(),
repo: repo.to_string(),
file: file.to_string(),
bytes: bytes.trim().parse().unwrap_or(0),
name: name_in_prefix(head),
})
})
.collect();
@@ -600,6 +679,18 @@ pub async fn on_machine(transport: &Transport, dir: &str) -> Result<Vec<LocalMod
Ok(found)
}
/// The model's name out of a base64 prefix of its file.
///
/// The remote half of [`name_of`], and `None` for everything that half
/// answers `None` for, plus a prefix that did not survive the trip.
fn name_in_prefix(head: &str) -> Option<String> {
use base64::Engine as _;
let bytes = base64::engine::general_purpose::STANDARD
.decode(head.trim())
.ok()?;
crate::gguf::name(&mut bytes.as_slice())
}
/// A model repository on HuggingFace, as the browse screen shows it.
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]