iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Give llama.cpp sessions tools, web search and a model picker

Browse Source 
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.

Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.

Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.

Also here, because tools made each of them matter:

- **Loading is a state.** A 12 GB model takes twenty seconds to reach
  memory and refuses everything until it has; the session used to report
  `running` for that whole time, and a message sent meanwhile came back as
  an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
  stops it and starts another. The conversation survives because it was
  never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
  itself -- over ssh too, in the round trip the spawn was already making.
  Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
  the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
  slot, and 28 with it at four -- speculating against a split KV cache is
  worse than not speculating. The flag is conditional because asking for a
  head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
  an overrun context is now ordinary; it was "http status: 400" and is now
  the server's own "exceeds the available context size, try increasing it".

`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.

Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
iris-aiandClaude Opus 5 committed 2026-09-19 08:11:49 -04:00
1 parent 392cc5413d
commit ac476ab0c9
23 files changed
+3627 -1096
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

server/src/session/llama/mcp.rs
+375
View File
@@ -0,0 +1,375 @@
//! An MCP client, for the tools a llama session has that `llama-server` does
//! not provide itself.
//!
//! **Why this is here and not a flag on `llama-server`.** That server can host
//! MCP servers (`--mcp-servers-json`), but only ones it can *spawn*: its
//! configuration is Cursor's, and an entry without a `command` is skipped with
//! "MCP server 'exa' has no command". Exa's is a remote HTTP endpoint with
//! nothing to spawn, so reaching it that way means a local process bridging
//! stdio to HTTP -- a Node install on the machine serving the model, and a
//! package to keep current, for what is three JSON-RPC calls.
//!
//! llama.cpp's own web UI does not do that either. It ships Exa in a
//! "recommended servers" list and connects to `https://mcp.exa.ai/mcp`
//! *itself*, from the browser. This is the same arrangement with this server
//! in the browser's place, and it is the right one for a second reason: it
//! puts the search on the machine running the backend rather than on whichever
//! machine happens to be serving the model, which may have no route out at
//! all.
//!
//! **Only the three calls a tool needs.** `initialize`, `tools/list`,
//! `tools/call`. Nothing here implements resources, prompts, sampling or the
//! server-to-client stream, because nothing here uses them; a session's tools
//! are a list fetched once and a call made on demand. That is why this is a
//! file rather than a dependency on a protocol crate -- there is no spec
//! surface to get subtly wrong, only a request and its reply.
//!
//! Transport is "streamable HTTP": every message is a POST, and the reply is
//! either JSON or a one-event SSE stream carrying the same JSON. Both are
//! accepted because which one arrives is the server's choice, not ours.
use anyhow::{Context, Result, bail};
use serde_json::{Value, json};
/// Identifies this client to an MCP server.
///
/// Not politeness: Exa's endpoint is behind Cloudflare, which answers **403**
/// to a request with no `User-Agent` at all (measured 2026-09-19 -- the same
/// request with one succeeds). A client that omitted it would look exactly
/// like a server that was refusing us.
const USER_AGENT: &str = concat!("ai-server/", env!("CARGO_PKG_VERSION"));
/// The protocol version this speaks. Sent at `initialize`; a server that
/// prefers another says so in its answer and this goes along with whatever it
/// then sends, since none of the three calls here has changed between
/// versions.
const PROTOCOL_VERSION: &str = "2025-06-18";
/// How long any one call may take.
///
/// Generous because a web search is a search: Exa fetches and cleans pages
/// before answering. Bounded at all because this blocks a turn, and a tool
/// that never returns is a session that never speaks again.
const CALL_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(120);
/// A connected MCP server, and the tools it offered.
pub struct McpServer {
/// The name this server is configured under. It prefixes every tool, so
/// two servers offering `search` are two different tools.
name: String,
url: String,
/// What the server called this conversation, when it named one. Sent back
/// on every later request; a server that keeps no session sends no header
/// and this stays `None`.
session: Option<String>,
/// The tool names this server answers to, without the prefix, keyed by the
/// prefixed name the model is given.
tools: Vec<McpTool>,
}
/// One tool an MCP server offers, in both the names it has.
pub struct McpTool {
/// `{server}_{tool}` -- what the model calls it, and what comes back in a
/// tool call. Prefixed the way `llama-server` prefixes the MCP tools it
/// hosts itself, so a reader sees one naming convention whichever side a
/// tool came from.
pub qualified: String,
/// What the server calls it.
bare: String,
/// The OpenAI-shaped function definition sent to the model.
pub definition: Value,
}
impl McpServer {
/// Connects, handshakes, and asks what it can do.
///
/// All three steps or none: a server that answered `initialize` and then
/// failed to list its tools is not a server with no tools, and returning
/// an empty list for it would put a session on screen that silently
/// cannot search.
pub fn connect(name: &str, url: &str) -> Result<Self> {
let mut server = Self {
name: name.to_string(),
url: url.to_string(),
session: None,
tools: Vec::new(),
};
server
.request(
1,
"initialize",
json!({
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {},
"clientInfo": {"name": "ai-server", "title": "AI Sessions", "version": env!("CARGO_PKG_VERSION")},
}),
)
.with_context(|| format!("handshaking with the {name} MCP server at {url}"))?;
// A notification: no id, and the server answers with no body. Sent
// because the specification requires it before any other call, and
// Exa's server does enforce it.
server.notify("notifications/initialized")?;
let listed = server
.request(2, "tools/list", json!({}))
.with_context(|| format!("asking the {name} MCP server what it offers"))?;
server.tools = listed
.get("tools")
.and_then(Value::as_array)
.map(|tools| {
tools
.iter()
.filter_map(|tool| server.describe(tool))
.collect()
})
.unwrap_or_default();
Ok(server)
}
/// Turns one entry of `tools/list` into the function definition a model is
/// given, or `None` for one this cannot name or call.
fn describe(&self, tool: &Value) -> Option<McpTool> {
let bare = tool.get("name").and_then(Value::as_str)?.to_string();
let qualified = format!("{}_{bare}", self.name);
let mut function = serde_json::Map::new();
function.insert("name".into(), json!(qualified));
if let Some(description) = tool.get("description").and_then(Value::as_str) {
function.insert("description".into(), json!(description));
}
// `inputSchema` in MCP, `parameters` in the OpenAI shape: the same
// JSON Schema under two names. A tool that declares none takes no
// arguments, which is an empty object rather than an absent key --
// some templates render the key unconditionally.
function.insert(
"parameters".into(),
tool.get("inputSchema")
.cloned()
.unwrap_or_else(|| json!({"type": "object", "properties": {}})),
);
Some(McpTool {
qualified,
bare,
definition: json!({"type": "function", "function": function}),
})
}
pub fn tools(&self) -> &[McpTool] {
&self.tools
}
/// Runs one of this server's tools, named as the model named it.
///
/// The result is the text a model is shown. A tool the server reports as
/// failing is **not** an error here: `isError` means the tool ran and went
/// wrong -- a search that found nothing, a page that would not fetch --
/// and the model is the one that has to know, so it comes back as its own
/// message. An error is reserved for not having reached the server at all.
pub fn call(&mut self, qualified: &str, arguments: &Value) -> Result<String> {
// The bare name is taken before the call, because the call needs the
// whole of `self` and the tool list is part of it.
let bare = self
.tools
.iter()
.find(|tool| tool.qualified == qualified)
.map(|tool| tool.bare.clone())
.with_context(|| format!("{} does not offer {qualified}", self.name))?;
let result = self.request(
3,
"tools/call",
json!({"name": bare, "arguments": arguments}),
)?;
Ok(rendered(&result))
}
/// One request, and its result.
///
/// `&self` rather than `&mut self` everywhere but the handshake would be
/// tidier and is wrong: the session header is assigned by the server on
/// the first reply and has to be kept.
fn request(&mut self, id: u64, method: &str, params: Value) -> Result<Value> {
let body = json!({"jsonrpc": "2.0", "id": id, "method": method, "params": params});
let answer = self.post(&body)?.with_context(|| {
format!(
"the {} MCP server answered {method} with nothing",
self.name
)
})?;
if let Some(message) = answer.pointer("/error/message").and_then(Value::as_str) {
bail!("{} refused {method}: {message}", self.name);
}
answer
.get("result")
.cloned()
.with_context(|| format!("the {} MCP server's {method} carried no result", self.name))
}
/// A message with no id, which is answered with no body.
fn notify(&mut self, method: &str) -> Result<()> {
self.post(&json!({"jsonrpc": "2.0", "method": method}))?;
Ok(())
}
/// Posts one JSON-RPC message and returns whatever came back, which for a
/// notification is nothing.
fn post(&mut self, body: &Value) -> Result<Option<Value>> {
let mut request = ureq::post(&self.url)
.config()
.timeout_global(Some(CALL_TIMEOUT))
.build()
.header("content-type", "application/json")
// Both, because which one a server replies with is its choice.
.header("accept", "application/json, text/event-stream")
.header("user-agent", USER_AGENT);
if let Some(session) = &self.session {
request = request.header("mcp-session-id", session);
}
let mut response = request
.send_json(body)
.with_context(|| format!("reaching the {} MCP server at {}", self.name, self.url))?;
if let Some(session) = response
.headers()
.get("mcp-session-id")
.and_then(|value| value.to_str().ok())
{
self.session = Some(session.to_string());
}
let streamed = response
.headers()
.get("content-type")
.and_then(|value| value.to_str().ok())
.is_some_and(|value| value.contains("text/event-stream"));
let text = response
.body_mut()
.read_to_string()
.with_context(|| format!("reading the {} MCP server's answer", self.name))?;
Ok(first_message(&text, streamed))
}
}
/// The first JSON-RPC message in a reply body.
///
/// One, not all: every call here carries a single id and the server answers it
/// once. Server-sent events are unwrapped to their payload lines; a plain JSON
/// body is itself.
fn first_message(text: &str, streamed: bool) -> Option<Value> {
if streamed {
return text
.lines()
.filter_map(|line| line.strip_prefix("data: "))
.find_map(|payload| serde_json::from_str(payload).ok());
}
serde_json::from_str(text.trim()).ok()
}
/// A `tools/call` result as the text a model is given.
///
/// MCP answers with a list of content blocks; the text ones are joined and the
/// rest are named rather than dropped, because a model told nothing came back
/// will try again. `structuredContent` is used when there is no text at all,
/// which is how some servers answer entirely.
fn rendered(result: &Value) -> String {
let blocks = result.get("content").and_then(Value::as_array);
let mut parts: Vec<String> = Vec::new();
for block in blocks.into_iter().flatten() {
match block.get("type").and_then(Value::as_str) {
Some("text") => parts.push(
block
.get("text")
.and_then(Value::as_str)
.unwrap_or_default()
.to_string(),
),
Some(kind) => parts.push(format!("[{kind} content, which this session cannot show]")),
None => {}
}
}
if parts.iter().all(|part| part.trim().is_empty())
&& let Some(structured) = result.get("structuredContent")
{
return structured.to_string();
}
parts.join("\n")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_event_stream_body_is_unwrapped_to_its_payload() {
let body =
"event: message\ndata: {\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"ok\":true}}\n\n";
assert_eq!(
first_message(body, true),
Some(json!({"jsonrpc": "2.0", "id": 1, "result": {"ok": true}})),
);
}
#[test]
fn a_plain_json_body_is_the_message() {
let body = " {\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{}}\n";
assert_eq!(
first_message(body, false),
Some(json!({"jsonrpc": "2.0", "id": 1, "result": {}})),
);
}
#[test]
/// A notification's reply, which is nothing at all.
fn an_empty_body_is_no_message() {
assert_eq!(first_message("", false), None);
assert_eq!(first_message("event: ping\n", true), None);
}
#[test]
fn text_blocks_are_joined_and_other_kinds_are_named() {
let result = json!({"content": [
{"type": "text", "text": "first"},
{"type": "image", "data": "…"},
{"type": "text", "text": "second"},
]});
assert_eq!(
rendered(&result),
"first\n[image content, which this session cannot show]\nsecond",
);
}
#[test]
/// A server that answers only in structured form. Rendering "" for it
/// would tell the model the search came back empty, which is a different
/// fact from the one that is true.
fn a_result_with_no_text_falls_back_to_its_structured_form() {
let result = json!({"content": [], "structuredContent": {"hits": 2}});
assert_eq!(rendered(&result), "{\"hits\":2}");
}
#[test]
/// The real endpoint, which is the only thing that can confirm the
/// handshake, the session header and the SSE unwrapping all agree with a
/// server nobody here wrote. Skipped without network rather than failed:
/// `./run-tests.sh` has to pass on a machine with no route out.
fn exa_answers_a_search_over_the_real_protocol() {
let Ok(mut server) = McpServer::connect("exa", super::super::EXA_MCP_URL) else {
eprintln!("skipping: could not reach Exa");
return;
};
assert!(
server
.tools()
.iter()
.any(|tool| tool.qualified == "exa_web_search_exa"),
"Exa offered {:?}",
server
.tools()
.iter()
.map(|tool| &tool.qualified)
.collect::<Vec<_>>(),
);
let answer = server
.call(
"exa_web_search_exa",
&json!({"query": "llama.cpp server", "numResults": 1}),
)
.expect("search");
assert!(!answer.trim().is_empty(), "a search returned nothing");
}
}
server/src/session/llama/mod.rs
+2052
View File
File diff suppressed because it is too large. Load diff
server/src/session/llama/tools.rs
+263
View File
@@ -0,0 +1,263 @@
//! What a llama session can do besides talk, and who runs it.
//!
//! Two sources, one list. `llama-server` started with `--tools` runs a set of
//! its own -- reading, searching, editing, a shell -- and publishes them at
//! `GET /tools` in the shape a model is given, with `POST /tools` to run one.
//! Anything else comes from an MCP server this backend is connected to (see
//! [`super::mcp`]). Both arrive here as a definition to offer and a way to
//! call, and nothing downstream of [`Tools::execute`] knows which a tool was.
//!
//! **The built-in tools run where the model does, and that is the point.** A
//! session on another machine edits that machine's files, because that is the
//! machine `llama-server` is on -- the same rule the model file already
//! follows. MCP tools run here instead, which is right for the opposite
//! reason: a web search wants the machine with a route out, not the one with
//! the GPU.
//!
//! **A tool's failure is a result, not an error.** A missing file, a command
//! that exited non-zero, a search that found nothing: all of those are things
//! the model has to read and act on, so they come back as the tool's output.
//! [`Tools::execute`] returns `Err` only when the tool could not be reached at
//! all, which is a fact about this server rather than about the work.
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use anyhow::{Context, Result};
use serde_json::{Value, json};
use super::mcp::McpServer;
/// How long one tool call may take.
///
/// This is the shell tool's budget as much as anything: a build, a test run,
/// a `find` over a large tree. Bounded because it blocks the turn, and a
/// session stuck behind a command that will never finish cannot even be told
/// to stop.
const EXECUTE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
/// The tools one session has, and how to run each of them.
pub struct Tools {
/// The `llama-server` these belong to. Replaced when the session's model
/// changes, because that is a different server on a different port.
endpoint: String,
/// What the model is given, in the order it is offered: the server's own
/// tools first, then each MCP server's.
definitions: Vec<Value>,
/// The server's tools, and whether each is run relative to a working
/// directory. Only the ones that say so are sent one -- a tool that
/// ignores it would still have its cache keyed on it.
server: HashMap<String, bool>,
/// Connected MCP servers, each of which knows its own tools by the
/// prefixed names they were offered under.
mcp: Vec<Arc<Mutex<McpServer>>>,
}
impl Tools {
/// Asks a ready `llama-server` what it offers and adds what the MCP
/// servers offered.
///
/// A server started without `--tools` answers with an empty list, and a
/// session with only MCP tools is a perfectly good session -- so nothing
/// here treats "no tools" as a failure. What *is* a failure is not being
/// able to ask, because that is the same server the conversation is about
/// to go to.
pub fn discover(endpoint: &str, mcp: Vec<Arc<Mutex<McpServer>>>) -> Result<Self> {
let catalog: Vec<Value> = ureq::get(format!("{endpoint}/tools"))
.call()
.context("asking llama-server which tools it has")?
.body_mut()
.read_json()
.context("reading llama-server's tool list")?;
let mut definitions = Vec::new();
let mut server = HashMap::new();
for entry in &catalog {
let Some(name) = entry
.pointer("/definition/function/name")
.and_then(Value::as_str)
else {
continue;
};
let Some(definition) = entry.get("definition") else {
continue;
};
server.insert(
name.to_string(),
entry
.get("uses_cwd")
.and_then(Value::as_bool)
.unwrap_or(false),
);
definitions.push(definition.clone());
}
for connected in &mcp {
for tool in connected.lock().unwrap().tools() {
definitions.push(tool.definition.clone());
}
}
Ok(Self {
endpoint: endpoint.to_string(),
definitions,
server,
mcp,
})
}
/// What goes in the request's `tools`, or `None` when there is nothing to
/// offer.
///
/// Absent rather than empty for a reason that shows on screen: a chat
/// template branches on whether tools were given, and an empty list
/// renders the whole "you may call one or more functions" preamble with no
/// functions under it.
pub fn offered(&self) -> Option<&[Value]> {
(!self.definitions.is_empty()).then_some(&self.definitions)
}
/// Whether this is a tool at all, which decides what to do about a call
/// naming something else.
pub fn knows(&self, name: &str) -> bool {
self.server.contains_key(name) || self.mcp_for(name).is_some()
}
/// The MCP server that offered `name`, if one did.
fn mcp_for(&self, name: &str) -> Option<&Arc<Mutex<McpServer>>> {
self.mcp.iter().find(|server| {
server
.lock()
.unwrap()
.tools()
.iter()
.any(|tool| tool.qualified == name)
})
}
/// Runs one call and returns what the model should read.
///
/// `cwd` is the session's working directory, sent only to the tools that
/// say they use one. A session with no working directory sends none, and
/// `llama-server` falls back to its own -- which is the honest outcome:
/// this server has no better answer for where "here" is.
pub fn execute(&self, name: &str, arguments: &Value, cwd: Option<&str>) -> Result<String> {
if let Some(server) = self.mcp_for(name) {
return server.lock().unwrap().call(name, arguments);
}
let uses_cwd = *self
.server
.get(name)
.with_context(|| format!("no tool called {name}"))?;
let mut request = ureq::post(format!("{}/tools", self.endpoint))
.config()
.timeout_global(Some(EXECUTE_TIMEOUT))
// The refusal is a sentence the model can act on, so it is read as
// one rather than discarded in favour of its status code -- the
// same reason `super::refusal` exists for generation.
.http_status_as_error(false)
.build()
.header("content-type", "application/json");
if let (true, Some(cwd)) = (uses_cwd, cwd) {
request = request.header("x-tool-cwd", cwd);
}
let mut response = request
.send_json(json!({"tool": name, "params": arguments}))
.with_context(|| format!("asking llama-server to run {name}"))?;
let body = response
.body_mut()
.read_to_string()
.with_context(|| format!("reading what {name} produced"))?;
Ok(match serde_json::from_str::<Value>(&body) {
Ok(answer) => result_text(&answer),
// Not JSON at all: hand over what was said rather than a parse
// error about it, since the model is what has to carry on.
Err(_) => body,
})
}
}
/// `POST /tools`'s answer as the text a model is given.
///
/// The server answers `plain_text_response` for a tool that ran and `error`
/// for one that did not, and both are the model's business -- see this
/// module's note on failures being results. Anything else is handed over as
/// itself rather than discarded, since a tool this build has not seen before
/// is exactly the case where guessing is worst.
fn result_text(answer: &Value) -> String {
if let Some(text) = answer.get("plain_text_response").and_then(Value::as_str) {
return text.to_string();
}
if let Some(message) = answer.get("error").and_then(Value::as_str) {
return message.to_string();
}
answer.to_string()
}
/// How much a session asks before it acts.
///
/// Two, because two is what the mechanism underneath actually has. The web UI
/// that ships with `llama-server` asks before every call and remembers the
/// tools you said "always" to, and that pair -- a prompt and a growing set of
/// exceptions -- is the whole of its permission model. A third mode sitting
/// between them would have to invent a rule about which tools are "edits",
/// and the rule would be this app's opinion rather than anything the tools
/// declare.
pub const MODES: &[&str] = &["manual", "bypassPermissions"];
/// What a new llama session asks by default.
///
/// The cautious one, matching the web UI: a model with a shell on somebody's
/// own machine is the case to be wrong about in this direction, and one tap
/// on "always allow" is what makes it bearable afterwards.
pub const DEFAULT_MODE: &str = "manual";
/// The answer that makes an allowance permanent for the session. The tool's
/// name follows it, which is what makes the transcript alone enough to
/// rebuild the set -- see `super::allowed`.
pub const ALWAYS_PREFIX: &str = "Always allow ";
pub const ALLOW_ONCE: &str = "Allow once";
pub const REFUSE: &str = "Don't allow";
/// What the model is told when a call was refused.
///
/// Addressed to the model, not to the reader: it has to understand that the
/// work did not happen and that trying the same call again is not the way
/// round it, or it retries in a loop.
pub const REFUSED: &str = "The person using this session did not allow this call, so it was not run. Do not try it \
again -- say what you were going to do and why it needed that, and let them decide.";
/// What stands in for a call that never finished, when the transcript is read
/// back into a conversation.
///
/// Every tool call in the history owes a result, because that is the shape a
/// chat template renders; a turn stopped between the call and its result
/// leaves one that has none. Saying so is better than inventing an outcome,
/// and better than dropping the call -- which would tell the model it never
/// asked.
pub const UNFINISHED: &str = "This call was interrupted before it produced anything.";
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_tool_that_ran_reads_as_its_output() {
assert_eq!(
result_text(&json!({"plain_text_response": "hello\n"})),
"hello\n",
);
}
#[test]
/// The model is told what went wrong, because the model is what has to do
/// something about it -- read a different path, fix the command.
fn a_tool_that_failed_reads_as_its_message() {
assert_eq!(
result_text(&json!({"error": "cannot stat file: /tmp/nope"})),
"cannot stat file: /tmp/nope",
);
}
#[test]
fn anything_else_is_handed_over_as_itself() {
assert_eq!(result_text(&json!({"rows": 2})), "{\"rows\":2}");
}
}