Give llama.cpp sessions tools, web search and a model picker
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.
Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.
Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.
Also here, because tools made each of them matter:
- **Loading is a state.** A 12 GB model takes twenty seconds to reach
memory and refuses everything until it has; the session used to report
`running` for that whole time, and a message sent meanwhile came back as
an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
stops it and starts another. The conversation survives because it was
never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
itself -- over ssh too, in the round trip the spawn was already making.
Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
slot, and 28 with it at four -- speculating against a split KV cache is
worse than not speculating. The flag is conditional because asking for a
head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
an overrun context is now ordinary; it was "http status: 400" and is now
the server's own "exceeds the available context size, try increasing it".
`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.
Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
392cc5413d
commit
ac476ab0c9
23 files changed
+3627
-1096
No files matched your search
+103
-12
@@ -50,6 +50,71 @@ pub struct LocalModel {
|
||||
pub repo: String,
|
||||
pub file: String,
|
||||
pub bytes: u64,
|
||||
/// What the file says it is called (`general.name` in its own metadata),
|
||||
/// absent when it does not say or could not be read. Not a label: see
|
||||
/// [`labels`] for what a reader is actually shown, which needs the rest of
|
||||
/// the list to decide.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub name: Option<String>,
|
||||
}
|
||||
|
||||
/// What each of these models should be called on screen, in the same order.
|
||||
///
|
||||
/// A model's own name is the best answer and is not always an answer at all:
|
||||
/// two quantisations of one model carry the same `general.name`, and a chip
|
||||
/// row with two identical chips is one you cannot choose from. So this is a
|
||||
/// cascade -- the model's own name, else its file name, else its full key --
|
||||
/// and each model takes the first rung that nothing else on this machine
|
||||
/// shares. The last rung always terminates it, because the key is what makes
|
||||
/// these unique in the first place.
|
||||
///
|
||||
/// Decided over the whole list rather than per model because ambiguity is a
|
||||
/// property of the set: the same file is unambiguous on a machine holding one
|
||||
/// quantisation and not on a machine holding three, and only the list knows
|
||||
/// which machine this is.
|
||||
pub fn labels(models: &[LocalModel]) -> Vec<String> {
|
||||
let rungs = |model: &LocalModel| {
|
||||
[
|
||||
model.name.clone(),
|
||||
Some(model.file.trim_end_matches(".gguf").to_string()),
|
||||
Some(model.key.clone()),
|
||||
]
|
||||
};
|
||||
let mut taken: Vec<HashMap<String, usize>> = vec![HashMap::new(); 3];
|
||||
for model in models {
|
||||
for (rung, candidate) in rungs(model).into_iter().enumerate() {
|
||||
if let Some(candidate) = candidate {
|
||||
*taken[rung].entry(candidate).or_insert(0) += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
models
|
||||
.iter()
|
||||
.map(|model| {
|
||||
rungs(model)
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
.find_map(|(rung, candidate)| {
|
||||
let candidate = candidate?;
|
||||
(taken[rung].get(&candidate) == Some(&1)).then_some(candidate)
|
||||
})
|
||||
// Unreachable: the key rung is unique by construction. Said as
|
||||
// the key rather than as a panic, because a duplicate key would
|
||||
// mean the same file listed twice and a name is still the
|
||||
// honest thing to draw for it.
|
||||
.unwrap_or_else(|| model.key.clone())
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The model's own name, read out of the file itself.
|
||||
///
|
||||
/// Absent for every way of not finding out -- see [`crate::gguf`]. The file is
|
||||
/// opened and read only as far as the name, which is the first few hundred
|
||||
/// bytes, so this is affordable once per model per listing.
|
||||
fn name_of(path: &Path) -> Option<String> {
|
||||
let mut file = std::fs::File::open(path).ok()?;
|
||||
crate::gguf::name(&mut file)
|
||||
}
|
||||
|
||||
/// What a run is doing, or did. Flat rather than a tagged enum carrying its
|
||||
@@ -519,6 +584,7 @@ fn collect(root: &Path, dir: &Path, found: &mut Vec<LocalModel>) {
|
||||
repo: repo.to_string(),
|
||||
file: file.to_string(),
|
||||
bytes: entry.metadata().map(|m| m.len()).unwrap_or(0),
|
||||
name: name_of(&path),
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -566,33 +632,46 @@ pub fn dir_on(transport: &Transport, local: &Path) -> String {
|
||||
/// directory that is not there is an empty list rather than a failure: a
|
||||
/// machine that has never had a model put on it is an ordinary state, and
|
||||
/// the same one as a machine whose directory exists and is empty.
|
||||
///
|
||||
/// Each record carries the head of the file as well as its size, because a
|
||||
/// model's own name is inside it (see [`crate::gguf`]) and the file is on the
|
||||
/// far machine. The alternative is a second round trip per model, or naming
|
||||
/// remote models by path while local ones get their proper names -- one
|
||||
/// machine's models reading differently from another's is exactly the
|
||||
/// confusion the name was added to remove. The prefix is bounded at
|
||||
/// [`crate::gguf::PREFIX_BYTES`], which is what keeps this one round trip's
|
||||
/// worth of bytes.
|
||||
pub async fn on_machine(transport: &Transport, dir: &str) -> Result<Vec<LocalModel>> {
|
||||
let script = "p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${p#\"~/\"};; esac; \
|
||||
[ -d \"$p\" ] || exit 0; \
|
||||
find \"$p\" -type f -name '*.gguf' -printf '%s\\t%P\\0'";
|
||||
let script = format!(
|
||||
"p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${{p#\"~/\"}};; esac; \
|
||||
[ -d \"$p\" ] || exit 0; cd \"$p\" || exit 0; \
|
||||
find . -type f -name '*.gguf' -exec sh -c '\
|
||||
for f do printf \"%s\\t%s\\t%s\\0\" \"$(wc -c < \"$f\")\" \
|
||||
\"$(head -c {prefix} \"$f\" | base64 | tr -d \"\\n\")\" \"${{f#./}}\"; done\
|
||||
' sh {{}} +",
|
||||
prefix = crate::gguf::PREFIX_BYTES,
|
||||
);
|
||||
let launch = Launch::new(
|
||||
"sh",
|
||||
vec![
|
||||
"-c".to_string(),
|
||||
script.to_string(),
|
||||
"sh".to_string(),
|
||||
dir.to_string(),
|
||||
],
|
||||
vec!["-c".to_string(), script, "sh".to_string(), dir.to_string()],
|
||||
None,
|
||||
);
|
||||
let out = transport.capture(&launch).await?;
|
||||
let mut found: Vec<LocalModel> = out
|
||||
.split('\0')
|
||||
.filter(|record| !record.is_empty())
|
||||
// Two fields, and the name last, so a `\t` in a filename survives.
|
||||
.filter_map(|record| record.split_once('\t'))
|
||||
.filter_map(|(bytes, key)| {
|
||||
// Three fields, and the name last, so a `\t` in a filename survives.
|
||||
// The middle one is base64, which has no tab in its alphabet.
|
||||
.filter_map(|record| {
|
||||
let (bytes, rest) = record.split_once('\t')?;
|
||||
let (head, key) = rest.split_once('\t')?;
|
||||
let (repo, file) = key.rsplit_once('/')?;
|
||||
Some(LocalModel {
|
||||
key: key.to_string(),
|
||||
repo: repo.to_string(),
|
||||
file: file.to_string(),
|
||||
bytes: bytes.trim().parse().unwrap_or(0),
|
||||
name: name_in_prefix(head),
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
@@ -600,6 +679,18 @@ pub async fn on_machine(transport: &Transport, dir: &str) -> Result<Vec<LocalMod
|
||||
Ok(found)
|
||||
}
|
||||
|
||||
/// The model's name out of a base64 prefix of its file.
|
||||
///
|
||||
/// The remote half of [`name_of`], and `None` for everything that half
|
||||
/// answers `None` for, plus a prefix that did not survive the trip.
|
||||
fn name_in_prefix(head: &str) -> Option<String> {
|
||||
use base64::Engine as _;
|
||||
let bytes = base64::engine::general_purpose::STANDARD
|
||||
.decode(head.trim())
|
||||
.ok()?;
|
||||
crate::gguf::name(&mut bytes.as_slice())
|
||||
}
|
||||
|
||||
/// A model repository on HuggingFace, as the browse screen shows it.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
|
||||
Reference in new issue
Block a user