iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Give llama.cpp sessions tools, web search and a model picker

Browse Source 
A llama session was a chat box: no tools, a fixed model, no permission
mode, and a model name drawn as the path the file sits at. It now runs the
agent loop itself, which is what the pieces below all hang off.

Tools are `llama-server`'s own (`--tools all`), which that server both
publishes and runs -- `GET /tools` for the definitions, `POST /tools` to
call one. Web search is Exa's MCP server, reached from this backend rather
than from the machine serving the model: that is what llama.cpp's own web
UI does, and it puts the search on the machine with a route out instead of
the one with the GPU. `llama-server`'s `--mcp-servers-json` can only spawn
local commands, so using it would have meant a Node bridge on every
machine that serves a model.

Driving the loop is what makes the permission gate ours. Two modes,
`manual` and `bypassPermissions`, which is what the mechanism has: the web
UI asks before every call and remembers the tools you say "always" to. The
allowances fold back out of the transcript's own answers, so they survive
a restart and a model change without being stored anywhere else.

Also here, because tools made each of them matter:

- **Loading is a state.** A 12 GB model takes twenty seconds to reach
  memory and refuses everything until it has; the session used to report
  `running` for that whole time, and a message sent meanwhile came back as
  an error. It is `loading` now, and the message waits.
- **The model can be changed.** A `llama-server` holds one model, so this
  stops it and starts another. The conversation survives because it was
  never in the server.
- **Models are named, not pathed.** `general.name` read out of the file
  itself -- over ssh too, in the round trip the spawn was already making.
  Where two models share a name the file name breaks the tie.
- **`-np 1`, and the MTP draft head where the file has one.** Measured on
  the 27B here: 41.5 tok/s plain, 61.4 with `--spec-type draft-mtp` at one
  slot, and 28 with it at four -- speculating against a split KV cache is
  worse than not speculating. The flag is conditional because asking for a
  head that is not there makes `llama-server` exit.
- **A refusal says what to do.** Tool results are thousands of tokens, so
  an overrun context is now ordinary; it was "http status: 400" and is now
  the server's own "exceeds the available context size, try increasing it".

`GET /machines/{id}/models` is gone: the provider models route answers the
same question, and two answers to one question is how a picker comes to
offer a model the spawn screen does not.

Verified end to end against real models: a tool call asked and allowed, an
Exa search, a shell command, a 27B loaded while a message waited on it, a
model switch mid-session, a second message queued behind a running turn,
and the whole of it again on a session running over ssh.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
iris-aiandClaude Opus 5 committed 2026-09-19 08:11:49 -04:00
1 parent 392cc5413d
commit ac476ab0c9
23 files changed
+3627 -1096
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

server/src/session/driver.rs
+16
View File
@@ -564,6 +564,22 @@ pub enum SessionStatus {
Running,
AwaitingInput,
Compacting,
/// The session's process is up but cannot be spoken to yet.
///
/// Its own state because the two it would otherwise borrow are both
/// wrong in ways somebody notices. `Running` means the session is
/// answering, so a model taking a minute to load looks like a model
/// thinking for a minute -- and there is no way to tell from the screen
/// that the first message will be refused. `Idle` invites that message
/// and then loses it.
///
/// It exists for `llama-server`, which reads a multi-gigabyte file off
/// disk before it answers anything, and it is general because the
/// condition is: a process that is started and not yet ready is a state
/// any driver may have to report. Nothing is queued *because* of this
/// state -- a driver that reports it is responsible for holding what it
/// is sent until it can deliver it -- but this is what says so on screen.
Loading,
/// The session's own turn is over, but work it started is still going:
/// a backgrounded subagent, or a command left running.
///
server/src/session/llama.rs
-907
View File
@@ -1,907 +0,0 @@
//! The llama.cpp driver: a `llama-server` process per session, spoken to over
//! its OpenAI-compatible HTTP API and translated into the common event model.
//!
//! Two things make this shaped differently from the Claude driver.
//!
//! **It is spawned but not spoken to over stdio.** The process is started
//! through the same [`Transport`] as any other and then reached over HTTP on a
//! loopback port. That is the second half of what a transport is -- "run this"
//! plus "reach this port" -- and it is what lets a session run on another
//! machine: [`Transport::reserve_port`] hands back a port the server binds
//! *there* and one that reaches it *here*, and the ssh connection carrying the
//! command carries the tunnel between them. The far `llama-server` binds
//! loopback only, so a model is never served to that machine's network.
//!
//! **The model file is the far machine's, not this one's.** A remote machine
//! names its own models directory (`SshConfig::models_dir`, defaulting to where
//! this backend keeps its downloads), and the file is looked for *there* -- so
//! a session naming a model that machine does not have says so, instead of
//! starting a server that will never load one. Downloading to another machine
//! is not built; the model gets there however anything else does.
//!
//! **The server is stateless between requests**, so the whole conversation goes
//! with every one. It is rebuilt from the session's transcript rather than kept
//! in this struct, which is not tidiness: a copy in driver memory is invisible
//! to a second device and gone when this process restarts.
//!
//! That leaves the Claude driver as the odd one out rather than this one -- the
//! CLI's own memory of a conversation is a cache in front of the same
//! transcript. Resolve any inconsistency in this direction.
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use anyhow::{Context, Result, bail};
use serde::{Deserialize, Serialize};
use serde_json::json;
use super::driver::{AttachmentRef, Driver, Event, EventSink, SessionStatus};
use super::process;
use super::transport::{Launch, Streams, Transport};
use crate::config::{ProviderConfig, SessionConfig};
/// How long to wait for a model to load before giving up. Loading is mostly
/// disk, and a large quantised model on a cold cache is genuinely slow, so this
/// is generous -- the failure it exists for is a server that will never answer.
const READY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
/// One turn in the conversation this driver keeps on the server's behalf.
#[derive(Debug, Clone, Serialize, Deserialize)]
struct Message {
role: String,
content: String,
}
pub struct LlamaDriver {
sink: EventSink,
/// Where this session's own llama-server answers.
endpoint: String,
/// Where the conversation is read back from, one line per event.
transcript: PathBuf,
/// Sampling settings chosen at spawn, sent with every request.
sampling: serde_json::Map<String, serde_json::Value>,
/// Set by [`Driver::interrupt`]; the streaming loop checks it between
/// chunks and stops, leaving what was generated in the transcript.
cancel: Arc<AtomicBool>,
/// Where this session's process record lives, so [`Driver::stop`] can find
/// the server it has to end.
session_dir: PathBuf,
}
impl LlamaDriver {
/// Takes charge of this session's `llama-server`: the one already loaded if
/// there is one, otherwise a new one.
///
/// One entry point, for the reason `ClaudeDriver::launch` gives, expensive
/// in a different currency: two servers holding the same model is twice the
/// memory, and the second would bind a different port while the phone kept
/// talking to the first.
#[allow(clippy::too_many_arguments)]
pub fn launch(
meta: &SessionConfig,
provider: &ProviderConfig,
transport: &Transport,
models_dir: &Path,
transcript: &Path,
session_dir: &Path,
sink: EventSink,
// llama.cpp has no notion of a Task call, so this is accepted only
// to keep one shape across every driver's launch -- see
// `SUBAGENTS.md`'s "Server layout".
_subagents: Arc<super::subagent::Subagents>,
) -> Result<Self> {
let model = meta.model.as_deref().context(
"a llama.cpp session needs a model -- one of the downloaded ones, by its key",
)?;
let path = model_on(transport, models_dir, model)?;
// Already loaded and still running: keep talking to it. The health poll
// below confirms it is really answering, so adopting a pid whose server
// has wedged still reports as a failure rather than as a session that
// silently never replies.
if let Some(process::Record {
detail: process::Detail::Http { port },
pid,
..
}) = process::live(session_dir)
{
tracing::info!(
"session {} reattaching to the llama-server it left loaded (pid {pid}, port {port})",
meta.id
);
return Ok(Self::attached(
format!("http://127.0.0.1:{port}"),
meta,
model,
transcript,
session_dir,
sink,
));
}
// Where it listens on its own machine, and where that is reached
// from here -- the same number when that machine is this one.
let forward = transport
.reserve_port()
.context("finding a port for llama-server")?;
let mut args: Vec<String> = vec![
"-m".into(),
path.clone(),
// Loopback there, whichever machine there is: what reaches it
// from outside that machine is the ssh tunnel and nothing
// else.
"--host".into(),
"127.0.0.1".into(),
"--port".into(),
forward.there.to_string(),
];
// Settings that belong to the server because they decide how the model
// is loaded; the sampling ones ride on each request instead, so changing
// them later needn't reload anything.
for (key, flag) in [
("contextSize", "-c"),
("gpuLayers", "-ngl"),
("threads", "-t"),
] {
if let Some(value) = meta.params.get(key) {
args.push(flag.to_string());
args.push(value.clone());
}
}
let program = provider.program();
let launch = Launch::new(program, args, meta.cwd.as_deref()).reaching(forward);
// Its output goes to files, not pipes. Not only so the process can
// outlive this server: nothing ever read those pipes, so a chatty
// llama-server filled the 64 KB buffer and blocked mid-load with no sign
// of why.
let child = transport.spawn(
&launch,
Streams::Detached {
stdin: std::process::Stdio::null(),
stdout: log_file(&session_dir.join(SERVER_LOG))?.into(),
stderr: log_file(&session_dir.join(SERVER_LOG))?.into(),
},
)?;
let pid = child
.id()
.context("llama-server exited before it could be recorded")?;
tracing::info!(
"session {} running {program} for {model} {} on 127.0.0.1:{} there, \
reached at 127.0.0.1:{} here, as pid {pid}",
meta.id,
transport.describe(),
forward.there,
forward.here,
);
// Reaped so it does not become a zombie while this server is still its
// parent; the health poll and the record are what say whether the
// session is alive, because after a restart there is no `Child` to ask.
tokio::spawn(async move {
let mut child = child;
let _ = child.wait().await;
});
// The *near* port, because that is the one anything reaching this
// server has to dial -- including a later run of this backend,
// which adopts the record without knowing which machine the server
// is on. For a remote session the recorded pid is the ssh
// client's, which is the process this machine owns and which holds
// the tunnel open for exactly as long as the far server lives.
let record = process::Record::of(pid, process::Detail::Http { port: forward.here })
.context("llama-server was gone before its start time could be read")?;
process::write(session_dir, &record);
Ok(Self::attached(
format!("http://127.0.0.1:{}", forward.here),
meta,
model,
transcript,
session_dir,
sink,
))
}
/// The driver for a `llama-server` at `endpoint`, however it got there.
///
/// Shared by starting one and adopting one, because everything after "there
/// is a server at this address" is identical -- including waiting for it to
/// answer, which an adopted one still owes: a recorded pid says a process
/// exists, not that its model is loaded.
fn attached(
endpoint: String,
meta: &SessionConfig,
model: &str,
transcript: &Path,
session_dir: &Path,
sink: EventSink,
) -> Self {
// Loading is slow enough to be worth saying so: the session shows as
// running until the model is in memory, rather than looking ready and
// refusing the first message.
let _ = sink.send(Event::Status {
state: SessionStatus::Running,
});
{
let sink = sink.clone();
let endpoint = endpoint.clone();
let model = model.to_string();
let session_dir = session_dir.to_path_buf();
std::thread::spawn(move || match wait_until_ready(&endpoint, &session_dir) {
Ok(()) => {
tracing::info!("{model} loaded and answering at {endpoint}");
let _ = sink.send(Event::Status {
state: SessionStatus::Idle,
});
watch(session_dir, sink);
}
Err(err) => {
let _ = sink.send(Event::Error {
message: format!("{model} never became ready: {err:#}"),
});
let _ = sink.send(Event::Status {
state: SessionStatus::Exited,
});
process::clear(&session_dir);
}
});
}
let mut sampling = serde_json::Map::new();
for (key, field) in [
("temperature", "temperature"),
("topP", "top_p"),
("topK", "top_k"),
("maxTokens", "max_tokens"),
] {
if let Some(raw) = meta.params.get(key)
&& let Ok(number) = raw.parse::<f64>()
{
sampling.insert(field.to_string(), json!(number));
}
}
Self {
sink,
endpoint,
transcript: transcript.to_path_buf(),
sampling,
cancel: Arc::new(AtomicBool::new(false)),
session_dir: session_dir.to_path_buf(),
}
}
}
/// Where llama-server's own output goes. One file for both streams: it is
/// diagnostics nobody parses, and interleaving them is how it reads in a
/// terminal anyway.
const SERVER_LOG: &str = "llama-server.log";
/// How often a loaded server is checked for still being there. Slower than the
/// Claude driver's stdout poll because nothing is waiting on it: this only has
/// to notice a server that has gone.
const WATCH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(2);
/// An owner-only log opened for appending, so the two streams pointed at
/// it do not overwrite each other and a reattach keeps what came before.
fn log_file(path: &Path) -> Result<std::fs::File> {
use std::os::unix::fs::OpenOptionsExt;
std::fs::OpenOptions::new()
.create(true)
.append(true)
.mode(0o600)
.open(path)
.with_context(|| format!("opening {}", path.display()))
}
/// Reports the server going away, for as long as the session is there to report
/// it to.
///
/// Polled rather than waited on, for the reason the Claude driver gives: after a
/// restart this server is not the process's parent, so liveness has to be a
/// question asked of the record -- and asking it two different ways is how the
/// two answers come to disagree.
fn watch(session_dir: PathBuf, sink: EventSink) {
std::thread::spawn(move || {
loop {
std::thread::sleep(WATCH_INTERVAL);
match process::recorded(&session_dir) {
Some((_, process::Liveness::Alive)) => {}
// Nothing recorded means the session was stopped or deleted
// deliberately, and whoever did that has already said so.
None => return,
Some((_, process::Liveness::Dead)) => {
if !process::stopping(&session_dir) {
let _ = sink.send(Event::Error {
message: "llama-server exited".to_string(),
});
}
let _ = sink.send(Event::Status {
state: SessionStatus::Exited,
});
process::clear(&session_dir);
return;
}
Some((_, process::Liveness::Unknown)) => {
let _ = sink.send(Event::Status {
state: SessionStatus::Unknown,
});
}
}
if sink.is_closed() {
return;
}
}
});
}
impl Driver for LlamaDriver {
fn send_user_message(&self, text: String, attachments: Vec<AttachmentRef>) {
if !attachments.is_empty() {
let _ = self.sink.send(Event::Error {
message: "this model can't be sent attachments or files".to_string(),
});
}
let sink = self.sink.clone();
let endpoint = self.endpoint.clone();
let transcript = self.transcript.clone();
let sampling = self.sampling.clone();
let cancel = Arc::clone(&self.cancel);
cancel.store(false, Ordering::Relaxed);
// Its own thread: the request blocks for as long as the model takes to
// generate, which is the whole point of streaming it.
std::thread::spawn(move || {
// Nothing is ever held back here -- there is no queue to wait in --
// so the message is taken the moment it arrives. Said anyway,
// because this is what records it: see `MessageTaken`.
let _ = sink.send(Event::MessageTaken {
id: None,
text: text.clone(),
// Never any: this driver refuses attachments above.
attachments: Vec::new(),
});
let _ = sink.send(Event::Status {
state: SessionStatus::Running,
});
// Everything before this message, plus this message. Read rather
// than remembered, and `text` is appended here rather than waited
// for, because the message's own transcript entry is still on its
// way when this runs.
let mut messages = conversation(&transcript);
messages.push(Message {
role: "user".into(),
content: text,
});
// The reply is not stored: the deltas below are the durable record,
// so the next turn reads back exactly what the phone was shown --
// including a partial one that was interrupted.
if let Err(err) = generate(&endpoint, &messages, &sampling, &cancel, &sink) {
let _ = sink.send(Event::Error {
message: format!("{err:#}"),
});
}
let _ = sink.send(Event::Status {
state: SessionStatus::Idle,
});
});
}
fn answer_question(&self, _id: &str, _answers: &[String]) {
// Nothing here asks questions: this driver has no tools.
}
fn interrupt(&self) {
self.cancel.store(true, Ordering::Relaxed);
}
// Nothing to forward: this process has no notion of what the conversation
// is called, and the rename has already happened where the name lives.
fn set_title(&self, _title: &str) {}
fn set_permission_mode(&self, _mode: &str) {
let _ = self.sink.send(Event::Error {
message: "a llama.cpp session runs no tools, so there is nothing for a permission \
mode to govern."
.to_string(),
});
}
fn set_model(&self, _model: &str) {
let _ = self.sink.send(Event::Error {
message: "a llama.cpp session's model is fixed when it starts, because the server \
loads one model into memory. Spawn another session to use a different one."
.to_string(),
});
}
fn run_command(&self, text: &str) {
let _ = self.sink.send(Event::Error {
message: format!(
"a llama.cpp session has no commands of its own, so {text} means nothing to it."
),
});
}
fn compact(&self) {
let _ = self.sink.send(Event::Error {
message: "llama.cpp has no compaction. Clear the session instead, which costs nothing."
.to_string(),
});
}
fn clear(&self) {
// All of it. `conversation` folds from the last of these, so recording
// the marker *is* the reset -- there is no driver state to keep in step
// with it, which is the same property that makes a second device see the
// same conversation this one does.
let _ = self.sink.send(Event::Cleared);
}
/// Stops generating and leaves the server loaded.
///
/// Worth being deliberate about, because the cost points the other way from
/// the Claude driver's: a `llama-server` holds its whole model in memory, so
/// a leaked one is gigabytes nobody is using. It is left anyway, because the
/// alternative is unloading and reloading that model on every backend
/// restart -- minutes of disk, for a session somebody is in the middle of.
/// The record is what keeps it from being *nobody's*.
fn detach(&self) {
self.cancel.store(true, Ordering::Relaxed);
}
fn stop(&self) {
self.cancel.store(true, Ordering::Relaxed);
if let Some(record) = process::live(&self.session_dir) {
process::stop(&record, process::STOP_GRACE);
}
process::clear(&self.session_dir);
}
}
/// The conversation so far, folded out of the transcript.
///
/// Consecutive `AssistantText` deltas are one assistant turn, closed by the next
/// user message -- which is also what makes an interrupted reply come back as
/// the partial text the phone actually saw.
///
/// This must stay a pure function of the transcript and must never re-render
/// earlier turns. llama.cpp caches the prompt prefix, so a growing conversation
/// reprocesses almost nothing -- but only while every turn is byte-identical to
/// last time. Changing how an old turn is rendered silently reprocesses the
/// whole history on every message.
fn conversation(path: &Path) -> Vec<Message> {
let Ok(events) = crate::session::transcript::read_after(path, 0) else {
return Vec::new();
};
let mut messages: Vec<Message> = Vec::new();
let mut pending = String::new();
// Everything before the last clear is still in the transcript and is
// deliberately not in the conversation. Folding from zero would put it back,
// which is the whole of what clearing had to undo.
let events = match events.iter().rposition(|e| e.event == Event::Cleared) {
Some(at) => &events[at + 1..],
None => &events[..],
};
for event in events.iter().cloned() {
match event.event {
Event::UserMessage { text, .. } => {
if !pending.is_empty() {
messages.push(Message {
role: "assistant".into(),
content: std::mem::take(&mut pending),
});
}
messages.push(Message {
role: "user".into(),
content: text,
});
}
Event::AssistantText { delta } => pending.push_str(&delta),
Event::AssistantTextFinal { text } => {
pending = text;
}
_ => {}
}
}
if !pending.is_empty() {
messages.push(Message {
role: "assistant".into(),
content: pending,
});
}
messages
}
/// Where a model key resolves to on disk, refusing anything that climbs
/// out of the models directory -- the key arrives from a phone.
fn model_path(models_dir: &Path, key: &str) -> Result<PathBuf> {
let mut path = models_dir.to_path_buf();
for part in key.split('/') {
if part.is_empty() || part == "." || part == ".." {
bail!("\"{key}\" is not a model key this can resolve");
}
path.push(part);
}
if !path.is_file() {
bail!("no downloaded model called \"{key}\" -- download it first");
}
Ok(path)
}
/// The model file's path **on the machine that will serve it**, confirmed to be
/// there.
///
/// One function rather than a local check and hope for the other case: the same
/// question has to be asked of two filesystems. The remote answer is measured
/// for the reason the local one is -- a missing file otherwise becomes a
/// `llama-server` that starts, fails to load, and reports as a session that
/// never became ready, which reads as the machine being slow.
///
/// One blocking round trip on a remote spawn, which is what the spawn is
/// already paying to start ssh. The alternative is a path built here from a `~`
/// this machine cannot expand.
fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result<String> {
let Transport::Ssh { name, .. } = transport else {
return Ok(model_path(models_dir, key)?.to_string_lossy().into_owned());
};
// The same directory the spawn screen listed for this machine, and one
// function for the same reason: a list from one place and a load from
// another is a model that appears and then fails.
let dir = crate::models::dir_on(transport, models_dir);
// Checked here rather than in the script: `..` in a key would walk out of
// the models directory on a machine this server can start processes on,
// and the phone is where the key comes from.
for part in key.split('/') {
if part.is_empty() || part == "." || part == ".." {
bail!("\"{key}\" is not a model key this can resolve");
}
}
let path = format!("{}/{key}", dir.trim_end_matches('/'));
// `$HOME` on the far side, which is the only machine that knows what it is,
// and the resolved path printed back so the launch hands `llama-server`
// something absolute. "Not there" is answered rather than failed, because a
// machine that could not be asked at all has to say so in its own words --
// it would otherwise arrive as this same sentence about a missing model.
let script = "p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${p#\"~/\"};; esac; \
[ -f \"$p\" ] && printf 'at\\t%s\\n' \"$p\" || printf 'missing\\n'"
.to_string();
let launch = Launch::new(
"sh",
vec!["-c".to_string(), script, "sh".to_string(), path.clone()],
None,
);
let answer = transport
.capture_blocking(&launch)
.with_context(|| format!("couldn't ask {name} where its models are"))?;
match answer.trim().split_once('\t') {
Some(("at", resolved)) => Ok(resolved.to_string()),
_ => bail!(
"{name} has no model at {path}. A llama.cpp session serves the file from the \
machine it runs on, so the model has to be on {name} -- what this backend has \
downloaded is somewhere else."
),
}
}
/// Polls until the server says it is ready, or gives up.
///
/// Watches the process as well as the port, because the two failures need
/// different words and one of them is common: a model that will not load,
/// a port already taken on the far machine, a `llama-server` too old for
/// a flag. All of those exit within a second and none of them will ever
/// answer `/health`, so waiting out the timeout turns a server that said
/// exactly what was wrong into "gave up after 300s".
fn wait_until_ready(endpoint: &str, session_dir: &Path) -> Result<()> {
let deadline = std::time::Instant::now() + READY_TIMEOUT;
let url = format!("{endpoint}/health");
loop {
if let Ok(response) = ureq::get(&url).call()
&& response.status() == 200
{
return Ok(());
}
// `None` is the session having been stopped or deleted while this
// waited, which is nobody's fault and still not worth waiting on.
match process::recorded(session_dir) {
Some((_, process::Liveness::Alive | process::Liveness::Unknown)) => {}
Some((_, process::Liveness::Dead)) | None => {
bail!("it exited before it answered.{}", log_tail(session_dir));
}
}
if std::time::Instant::now() > deadline {
bail!(
"gave up after {}s.{}",
READY_TIMEOUT.as_secs(),
log_tail(session_dir)
);
}
std::thread::sleep(std::time::Duration::from_millis(250));
}
}
/// The end of `llama-server`'s own log, for a failure message.
///
/// Its account of what went wrong is the useful half -- "failed to load
/// model", "bind: Address already in use" -- and on a remote session it
/// is the only half, since nobody reading the phone can open a file on
/// that machine. Bounded, because this ends up in an event a phone draws.
fn log_tail(session_dir: &Path) -> String {
let Ok(text) = std::fs::read_to_string(session_dir.join(SERVER_LOG)) else {
return String::new();
};
let tail: Vec<&str> = text.lines().rev().take(LOG_TAIL_LINES).collect();
if tail.is_empty() {
return String::new();
}
format!(
" It last said: {}",
tail.into_iter().rev().collect::<Vec<_>>().join(" / ")
)
}
/// How much of that log to carry into a message somebody reads on a phone.
const LOG_TAIL_LINES: usize = 6;
/// One streamed completion: posts the conversation, emits each delta as it
/// arrives. Emits rather than returns, because the transcript those events land
/// in is what the next turn reads back.
fn generate(
endpoint: &str,
messages: &[Message],
sampling: &serde_json::Map<String, serde_json::Value>,
cancel: &AtomicBool,
sink: &EventSink,
) -> Result<()> {
let mut body = json!({
"messages": messages,
"stream": true,
"stream_options": {"include_usage": true},
});
let map = body.as_object_mut().expect("built as an object");
for (key, value) in sampling {
map.insert(key.clone(), value.clone());
}
let mut response = ureq::post(format!("{endpoint}/v1/chat/completions"))
.header("Content-Type", "application/json")
.send_json(&body)
.context("asking llama-server to generate")?;
let reader = std::io::BufReader::new(response.body_mut().as_reader());
let mut tokens = 0u64;
// The prompt side only, which is what the model is holding -- the same
// definition the other dialects report, so one word on the phone means one
// thing whichever kind of session it is.
let mut context = None;
for line in std::io::BufRead::lines(reader) {
if cancel.load(Ordering::Relaxed) {
break;
}
let line = line.context("reading the generation stream")?;
// Server-sent events: the payload lines are the ones that matter.
let Some(payload) = line.strip_prefix("data: ") else {
continue;
};
if payload.trim() == "[DONE]" {
break;
}
let Ok(chunk) = serde_json::from_str::<serde_json::Value>(payload) else {
continue;
};
if let Some(usage) = chunk.get("usage") {
if let Some(total) = usage
.get("total_tokens")
.and_then(serde_json::Value::as_u64)
{
tokens = total;
}
if let Some(prompt) = usage
.get("prompt_tokens")
.and_then(serde_json::Value::as_u64)
{
context = Some(prompt);
}
}
let delta = chunk
.get("choices")
.and_then(|c| c.get(0))
.and_then(|c| c.get("delta"))
.and_then(|d| d.get("content"))
.and_then(serde_json::Value::as_str)
.unwrap_or_default();
if !delta.is_empty() {
let _ = sink.send(Event::AssistantText {
delta: delta.to_string(),
});
}
}
if tokens > 0 {
let _ = sink.send(Event::UsageDelta { tokens, context });
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::session::transcript::Transcript;
/// Writes a transcript the way the pump does, so the fold is tested against
/// the real file format rather than a hand-built vector.
fn transcript_with(events: &[Event]) -> (tempfile::TempDir, PathBuf) {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("transcript.jsonl");
let mut transcript = Transcript::open(&path).expect("open");
for event in events {
transcript.append(event.clone(), 0.0).expect("append");
}
(dir, path)
}
#[test]
fn deltas_between_user_messages_are_one_assistant_turn() {
let (_dir, path) = transcript_with(&[
Event::UserMessage {
id: None,
text: "hello".into(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "hi ".into(),
},
Event::AssistantText {
delta: "there".into(),
},
Event::Status {
state: SessionStatus::Idle,
},
Event::UserMessage {
id: None,
text: "again".into(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "yes".into(),
},
]);
let messages = conversation(&path);
assert_eq!(
messages
.iter()
.map(|m| (m.role.as_str(), m.content.as_str()))
.collect::<Vec<_>>(),
[
("user", "hello"),
("assistant", "hi there"),
("user", "again"),
("assistant", "yes")
],
);
}
#[test]
/// The interrupted case, which decides what a resumed conversation is built
/// from: whatever the phone was shown. The deltas that arrived before the
/// stop are in the transcript, so they are in the prompt -- the model is
/// never told it said something the user did not see.
fn an_interrupted_reply_stays_in_the_conversation() {
let (_dir, path) = transcript_with(&[
Event::UserMessage {
id: None,
text: "count".into(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "one two".into(),
},
Event::Status {
state: SessionStatus::Idle,
},
]);
let messages = conversation(&path);
assert_eq!(messages.len(), 2);
assert_eq!(messages[1].content, "one two");
}
#[test]
/// Events this driver does not produce must not disturb the fold: a
/// transcript can carry errors and status changes from a session that
/// was, say, relaunched.
fn other_events_are_not_part_of_the_conversation() {
let (_dir, path) = transcript_with(&[
Event::Status {
state: SessionStatus::Running,
},
Event::UserMessage {
id: None,
text: "hello".into(),
attachments: Vec::new(),
},
Event::Error {
message: "something went wrong".into(),
},
Event::AssistantText {
delta: "still here".into(),
},
Event::UsageDelta {
tokens: 12,
context: Some(12),
},
]);
let messages = conversation(&path);
assert_eq!(messages.len(), 2);
assert_eq!(messages[0].content, "hello");
assert_eq!(messages[1].content, "still here");
}
#[test]
/// Clearing decides what the *model* is given, not just what the phone
/// draws. Everything above the marker stays in the transcript and none of it
/// is sent.
fn the_conversation_starts_after_the_last_clear() {
let (_dir, path) = transcript_with(&[
Event::UserMessage {
id: None,
text: "the long expensive conversation".into(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "at length".into(),
},
Event::Cleared,
Event::UserMessage {
id: None,
text: "a fresh start".into(),
attachments: Vec::new(),
},
Event::AssistantText {
delta: "cheaply".into(),
},
]);
let messages = conversation(&path);
assert_eq!(messages.len(), 2);
assert_eq!(messages[0].content, "a fresh start");
assert_eq!(messages[1].content, "cheaply");
}
#[test]
/// The *last* one, so clearing twice does not resurrect what the
/// first clear dropped.
fn only_the_newest_clear_counts() {
let (_dir, path) = transcript_with(&[
Event::UserMessage {
id: None,
text: "one".into(),
attachments: Vec::new(),
},
Event::Cleared,
Event::UserMessage {
id: None,
text: "two".into(),
attachments: Vec::new(),
},
Event::Cleared,
Event::UserMessage {
id: None,
text: "three".into(),
attachments: Vec::new(),
},
]);
let messages = conversation(&path);
assert_eq!(messages.len(), 1);
assert_eq!(messages[0].content, "three");
}
#[test]
fn a_model_key_cannot_climb_out_of_the_models_directory() {
let dir = tempfile::tempdir().expect("tempdir");
for attempt in ["../../etc/passwd", "unsloth/../../escape.gguf", ""] {
assert!(
model_path(dir.path(), attempt).is_err(),
"{attempt:?} should have been refused",
);
}
}
}
server/src/session/llama/mcp.rs
+375
View File
@@ -0,0 +1,375 @@
//! An MCP client, for the tools a llama session has that `llama-server` does
//! not provide itself.
//!
//! **Why this is here and not a flag on `llama-server`.** That server can host
//! MCP servers (`--mcp-servers-json`), but only ones it can *spawn*: its
//! configuration is Cursor's, and an entry without a `command` is skipped with
//! "MCP server 'exa' has no command". Exa's is a remote HTTP endpoint with
//! nothing to spawn, so reaching it that way means a local process bridging
//! stdio to HTTP -- a Node install on the machine serving the model, and a
//! package to keep current, for what is three JSON-RPC calls.
//!
//! llama.cpp's own web UI does not do that either. It ships Exa in a
//! "recommended servers" list and connects to `https://mcp.exa.ai/mcp`
//! *itself*, from the browser. This is the same arrangement with this server
//! in the browser's place, and it is the right one for a second reason: it
//! puts the search on the machine running the backend rather than on whichever
//! machine happens to be serving the model, which may have no route out at
//! all.
//!
//! **Only the three calls a tool needs.** `initialize`, `tools/list`,
//! `tools/call`. Nothing here implements resources, prompts, sampling or the
//! server-to-client stream, because nothing here uses them; a session's tools
//! are a list fetched once and a call made on demand. That is why this is a
//! file rather than a dependency on a protocol crate -- there is no spec
//! surface to get subtly wrong, only a request and its reply.
//!
//! Transport is "streamable HTTP": every message is a POST, and the reply is
//! either JSON or a one-event SSE stream carrying the same JSON. Both are
//! accepted because which one arrives is the server's choice, not ours.
use anyhow::{Context, Result, bail};
use serde_json::{Value, json};
/// Identifies this client to an MCP server.
///
/// Not politeness: Exa's endpoint is behind Cloudflare, which answers **403**
/// to a request with no `User-Agent` at all (measured 2026-09-19 -- the same
/// request with one succeeds). A client that omitted it would look exactly
/// like a server that was refusing us.
const USER_AGENT: &str = concat!("ai-server/", env!("CARGO_PKG_VERSION"));
/// The protocol version this speaks. Sent at `initialize`; a server that
/// prefers another says so in its answer and this goes along with whatever it
/// then sends, since none of the three calls here has changed between
/// versions.
const PROTOCOL_VERSION: &str = "2025-06-18";
/// How long any one call may take.
///
/// Generous because a web search is a search: Exa fetches and cleans pages
/// before answering. Bounded at all because this blocks a turn, and a tool
/// that never returns is a session that never speaks again.
const CALL_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(120);
/// A connected MCP server, and the tools it offered.
pub struct McpServer {
/// The name this server is configured under. It prefixes every tool, so
/// two servers offering `search` are two different tools.
name: String,
url: String,
/// What the server called this conversation, when it named one. Sent back
/// on every later request; a server that keeps no session sends no header
/// and this stays `None`.
session: Option<String>,
/// The tool names this server answers to, without the prefix, keyed by the
/// prefixed name the model is given.
tools: Vec<McpTool>,
}
/// One tool an MCP server offers, in both the names it has.
pub struct McpTool {
/// `{server}_{tool}` -- what the model calls it, and what comes back in a
/// tool call. Prefixed the way `llama-server` prefixes the MCP tools it
/// hosts itself, so a reader sees one naming convention whichever side a
/// tool came from.
pub qualified: String,
/// What the server calls it.
bare: String,
/// The OpenAI-shaped function definition sent to the model.
pub definition: Value,
}
impl McpServer {
/// Connects, handshakes, and asks what it can do.
///
/// All three steps or none: a server that answered `initialize` and then
/// failed to list its tools is not a server with no tools, and returning
/// an empty list for it would put a session on screen that silently
/// cannot search.
pub fn connect(name: &str, url: &str) -> Result<Self> {
let mut server = Self {
name: name.to_string(),
url: url.to_string(),
session: None,
tools: Vec::new(),
};
server
.request(
1,
"initialize",
json!({
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {},
"clientInfo": {"name": "ai-server", "title": "AI Sessions", "version": env!("CARGO_PKG_VERSION")},
}),
)
.with_context(|| format!("handshaking with the {name} MCP server at {url}"))?;
// A notification: no id, and the server answers with no body. Sent
// because the specification requires it before any other call, and
// Exa's server does enforce it.
server.notify("notifications/initialized")?;
let listed = server
.request(2, "tools/list", json!({}))
.with_context(|| format!("asking the {name} MCP server what it offers"))?;
server.tools = listed
.get("tools")
.and_then(Value::as_array)
.map(|tools| {
tools
.iter()
.filter_map(|tool| server.describe(tool))
.collect()
})
.unwrap_or_default();
Ok(server)
}
/// Turns one entry of `tools/list` into the function definition a model is
/// given, or `None` for one this cannot name or call.
fn describe(&self, tool: &Value) -> Option<McpTool> {
let bare = tool.get("name").and_then(Value::as_str)?.to_string();
let qualified = format!("{}_{bare}", self.name);
let mut function = serde_json::Map::new();
function.insert("name".into(), json!(qualified));
if let Some(description) = tool.get("description").and_then(Value::as_str) {
function.insert("description".into(), json!(description));
}
// `inputSchema` in MCP, `parameters` in the OpenAI shape: the same
// JSON Schema under two names. A tool that declares none takes no
// arguments, which is an empty object rather than an absent key --
// some templates render the key unconditionally.
function.insert(
"parameters".into(),
tool.get("inputSchema")
.cloned()
.unwrap_or_else(|| json!({"type": "object", "properties": {}})),
);
Some(McpTool {
qualified,
bare,
definition: json!({"type": "function", "function": function}),
})
}
pub fn tools(&self) -> &[McpTool] {
&self.tools
}
/// Runs one of this server's tools, named as the model named it.
///
/// The result is the text a model is shown. A tool the server reports as
/// failing is **not** an error here: `isError` means the tool ran and went
/// wrong -- a search that found nothing, a page that would not fetch --
/// and the model is the one that has to know, so it comes back as its own
/// message. An error is reserved for not having reached the server at all.
pub fn call(&mut self, qualified: &str, arguments: &Value) -> Result<String> {
// The bare name is taken before the call, because the call needs the
// whole of `self` and the tool list is part of it.
let bare = self
.tools
.iter()
.find(|tool| tool.qualified == qualified)
.map(|tool| tool.bare.clone())
.with_context(|| format!("{} does not offer {qualified}", self.name))?;
let result = self.request(
3,
"tools/call",
json!({"name": bare, "arguments": arguments}),
)?;
Ok(rendered(&result))
}
/// One request, and its result.
///
/// `&self` rather than `&mut self` everywhere but the handshake would be
/// tidier and is wrong: the session header is assigned by the server on
/// the first reply and has to be kept.
fn request(&mut self, id: u64, method: &str, params: Value) -> Result<Value> {
let body = json!({"jsonrpc": "2.0", "id": id, "method": method, "params": params});
let answer = self.post(&body)?.with_context(|| {
format!(
"the {} MCP server answered {method} with nothing",
self.name
)
})?;
if let Some(message) = answer.pointer("/error/message").and_then(Value::as_str) {
bail!("{} refused {method}: {message}", self.name);
}
answer
.get("result")
.cloned()
.with_context(|| format!("the {} MCP server's {method} carried no result", self.name))
}
/// A message with no id, which is answered with no body.
fn notify(&mut self, method: &str) -> Result<()> {
self.post(&json!({"jsonrpc": "2.0", "method": method}))?;
Ok(())
}
/// Posts one JSON-RPC message and returns whatever came back, which for a
/// notification is nothing.
fn post(&mut self, body: &Value) -> Result<Option<Value>> {
let mut request = ureq::post(&self.url)
.config()
.timeout_global(Some(CALL_TIMEOUT))
.build()
.header("content-type", "application/json")
// Both, because which one a server replies with is its choice.
.header("accept", "application/json, text/event-stream")
.header("user-agent", USER_AGENT);
if let Some(session) = &self.session {
request = request.header("mcp-session-id", session);
}
let mut response = request
.send_json(body)
.with_context(|| format!("reaching the {} MCP server at {}", self.name, self.url))?;
if let Some(session) = response
.headers()
.get("mcp-session-id")
.and_then(|value| value.to_str().ok())
{
self.session = Some(session.to_string());
}
let streamed = response
.headers()
.get("content-type")
.and_then(|value| value.to_str().ok())
.is_some_and(|value| value.contains("text/event-stream"));
let text = response
.body_mut()
.read_to_string()
.with_context(|| format!("reading the {} MCP server's answer", self.name))?;
Ok(first_message(&text, streamed))
}
}
/// The first JSON-RPC message in a reply body.
///
/// One, not all: every call here carries a single id and the server answers it
/// once. Server-sent events are unwrapped to their payload lines; a plain JSON
/// body is itself.
fn first_message(text: &str, streamed: bool) -> Option<Value> {
if streamed {
return text
.lines()
.filter_map(|line| line.strip_prefix("data: "))
.find_map(|payload| serde_json::from_str(payload).ok());
}
serde_json::from_str(text.trim()).ok()
}
/// A `tools/call` result as the text a model is given.
///
/// MCP answers with a list of content blocks; the text ones are joined and the
/// rest are named rather than dropped, because a model told nothing came back
/// will try again. `structuredContent` is used when there is no text at all,
/// which is how some servers answer entirely.
fn rendered(result: &Value) -> String {
let blocks = result.get("content").and_then(Value::as_array);
let mut parts: Vec<String> = Vec::new();
for block in blocks.into_iter().flatten() {
match block.get("type").and_then(Value::as_str) {
Some("text") => parts.push(
block
.get("text")
.and_then(Value::as_str)
.unwrap_or_default()
.to_string(),
),
Some(kind) => parts.push(format!("[{kind} content, which this session cannot show]")),
None => {}
}
}
if parts.iter().all(|part| part.trim().is_empty())
&& let Some(structured) = result.get("structuredContent")
{
return structured.to_string();
}
parts.join("\n")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_event_stream_body_is_unwrapped_to_its_payload() {
let body =
"event: message\ndata: {\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"ok\":true}}\n\n";
assert_eq!(
first_message(body, true),
Some(json!({"jsonrpc": "2.0", "id": 1, "result": {"ok": true}})),
);
}
#[test]
fn a_plain_json_body_is_the_message() {
let body = " {\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{}}\n";
assert_eq!(
first_message(body, false),
Some(json!({"jsonrpc": "2.0", "id": 1, "result": {}})),
);
}
#[test]
/// A notification's reply, which is nothing at all.
fn an_empty_body_is_no_message() {
assert_eq!(first_message("", false), None);
assert_eq!(first_message("event: ping\n", true), None);
}
#[test]
fn text_blocks_are_joined_and_other_kinds_are_named() {
let result = json!({"content": [
{"type": "text", "text": "first"},
{"type": "image", "data": "…"},
{"type": "text", "text": "second"},
]});
assert_eq!(
rendered(&result),
"first\n[image content, which this session cannot show]\nsecond",
);
}
#[test]
/// A server that answers only in structured form. Rendering "" for it
/// would tell the model the search came back empty, which is a different
/// fact from the one that is true.
fn a_result_with_no_text_falls_back_to_its_structured_form() {
let result = json!({"content": [], "structuredContent": {"hits": 2}});
assert_eq!(rendered(&result), "{\"hits\":2}");
}
#[test]
/// The real endpoint, which is the only thing that can confirm the
/// handshake, the session header and the SSE unwrapping all agree with a
/// server nobody here wrote. Skipped without network rather than failed:
/// `./run-tests.sh` has to pass on a machine with no route out.
fn exa_answers_a_search_over_the_real_protocol() {
let Ok(mut server) = McpServer::connect("exa", super::super::EXA_MCP_URL) else {
eprintln!("skipping: could not reach Exa");
return;
};
assert!(
server
.tools()
.iter()
.any(|tool| tool.qualified == "exa_web_search_exa"),
"Exa offered {:?}",
server
.tools()
.iter()
.map(|tool| &tool.qualified)
.collect::<Vec<_>>(),
);
let answer = server
.call(
"exa_web_search_exa",
&json!({"query": "llama.cpp server", "numResults": 1}),
)
.expect("search");
assert!(!answer.trim().is_empty(), "a search returned nothing");
}
}
server/src/session/llama/mod.rs
+2052
View File
File diff suppressed because it is too large. Load diff
server/src/session/llama/tools.rs
+263
View File
@@ -0,0 +1,263 @@
//! What a llama session can do besides talk, and who runs it.
//!
//! Two sources, one list. `llama-server` started with `--tools` runs a set of
//! its own -- reading, searching, editing, a shell -- and publishes them at
//! `GET /tools` in the shape a model is given, with `POST /tools` to run one.
//! Anything else comes from an MCP server this backend is connected to (see
//! [`super::mcp`]). Both arrive here as a definition to offer and a way to
//! call, and nothing downstream of [`Tools::execute`] knows which a tool was.
//!
//! **The built-in tools run where the model does, and that is the point.** A
//! session on another machine edits that machine's files, because that is the
//! machine `llama-server` is on -- the same rule the model file already
//! follows. MCP tools run here instead, which is right for the opposite
//! reason: a web search wants the machine with a route out, not the one with
//! the GPU.
//!
//! **A tool's failure is a result, not an error.** A missing file, a command
//! that exited non-zero, a search that found nothing: all of those are things
//! the model has to read and act on, so they come back as the tool's output.
//! [`Tools::execute`] returns `Err` only when the tool could not be reached at
//! all, which is a fact about this server rather than about the work.
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use anyhow::{Context, Result};
use serde_json::{Value, json};
use super::mcp::McpServer;
/// How long one tool call may take.
///
/// This is the shell tool's budget as much as anything: a build, a test run,
/// a `find` over a large tree. Bounded because it blocks the turn, and a
/// session stuck behind a command that will never finish cannot even be told
/// to stop.
const EXECUTE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
/// The tools one session has, and how to run each of them.
pub struct Tools {
/// The `llama-server` these belong to. Replaced when the session's model
/// changes, because that is a different server on a different port.
endpoint: String,
/// What the model is given, in the order it is offered: the server's own
/// tools first, then each MCP server's.
definitions: Vec<Value>,
/// The server's tools, and whether each is run relative to a working
/// directory. Only the ones that say so are sent one -- a tool that
/// ignores it would still have its cache keyed on it.
server: HashMap<String, bool>,
/// Connected MCP servers, each of which knows its own tools by the
/// prefixed names they were offered under.
mcp: Vec<Arc<Mutex<McpServer>>>,
}
impl Tools {
/// Asks a ready `llama-server` what it offers and adds what the MCP
/// servers offered.
///
/// A server started without `--tools` answers with an empty list, and a
/// session with only MCP tools is a perfectly good session -- so nothing
/// here treats "no tools" as a failure. What *is* a failure is not being
/// able to ask, because that is the same server the conversation is about
/// to go to.
pub fn discover(endpoint: &str, mcp: Vec<Arc<Mutex<McpServer>>>) -> Result<Self> {
let catalog: Vec<Value> = ureq::get(format!("{endpoint}/tools"))
.call()
.context("asking llama-server which tools it has")?
.body_mut()
.read_json()
.context("reading llama-server's tool list")?;
let mut definitions = Vec::new();
let mut server = HashMap::new();
for entry in &catalog {
let Some(name) = entry
.pointer("/definition/function/name")
.and_then(Value::as_str)
else {
continue;
};
let Some(definition) = entry.get("definition") else {
continue;
};
server.insert(
name.to_string(),
entry
.get("uses_cwd")
.and_then(Value::as_bool)
.unwrap_or(false),
);
definitions.push(definition.clone());
}
for connected in &mcp {
for tool in connected.lock().unwrap().tools() {
definitions.push(tool.definition.clone());
}
}
Ok(Self {
endpoint: endpoint.to_string(),
definitions,
server,
mcp,
})
}
/// What goes in the request's `tools`, or `None` when there is nothing to
/// offer.
///
/// Absent rather than empty for a reason that shows on screen: a chat
/// template branches on whether tools were given, and an empty list
/// renders the whole "you may call one or more functions" preamble with no
/// functions under it.
pub fn offered(&self) -> Option<&[Value]> {
(!self.definitions.is_empty()).then_some(&self.definitions)
}
/// Whether this is a tool at all, which decides what to do about a call
/// naming something else.
pub fn knows(&self, name: &str) -> bool {
self.server.contains_key(name) || self.mcp_for(name).is_some()
}
/// The MCP server that offered `name`, if one did.
fn mcp_for(&self, name: &str) -> Option<&Arc<Mutex<McpServer>>> {
self.mcp.iter().find(|server| {
server
.lock()
.unwrap()
.tools()
.iter()
.any(|tool| tool.qualified == name)
})
}
/// Runs one call and returns what the model should read.
///
/// `cwd` is the session's working directory, sent only to the tools that
/// say they use one. A session with no working directory sends none, and
/// `llama-server` falls back to its own -- which is the honest outcome:
/// this server has no better answer for where "here" is.
pub fn execute(&self, name: &str, arguments: &Value, cwd: Option<&str>) -> Result<String> {
if let Some(server) = self.mcp_for(name) {
return server.lock().unwrap().call(name, arguments);
}
let uses_cwd = *self
.server
.get(name)
.with_context(|| format!("no tool called {name}"))?;
let mut request = ureq::post(format!("{}/tools", self.endpoint))
.config()
.timeout_global(Some(EXECUTE_TIMEOUT))
// The refusal is a sentence the model can act on, so it is read as
// one rather than discarded in favour of its status code -- the
// same reason `super::refusal` exists for generation.
.http_status_as_error(false)
.build()
.header("content-type", "application/json");
if let (true, Some(cwd)) = (uses_cwd, cwd) {
request = request.header("x-tool-cwd", cwd);
}
let mut response = request
.send_json(json!({"tool": name, "params": arguments}))
.with_context(|| format!("asking llama-server to run {name}"))?;
let body = response
.body_mut()
.read_to_string()
.with_context(|| format!("reading what {name} produced"))?;
Ok(match serde_json::from_str::<Value>(&body) {
Ok(answer) => result_text(&answer),
// Not JSON at all: hand over what was said rather than a parse
// error about it, since the model is what has to carry on.
Err(_) => body,
})
}
}
/// `POST /tools`'s answer as the text a model is given.
///
/// The server answers `plain_text_response` for a tool that ran and `error`
/// for one that did not, and both are the model's business -- see this
/// module's note on failures being results. Anything else is handed over as
/// itself rather than discarded, since a tool this build has not seen before
/// is exactly the case where guessing is worst.
fn result_text(answer: &Value) -> String {
if let Some(text) = answer.get("plain_text_response").and_then(Value::as_str) {
return text.to_string();
}
if let Some(message) = answer.get("error").and_then(Value::as_str) {
return message.to_string();
}
answer.to_string()
}
/// How much a session asks before it acts.
///
/// Two, because two is what the mechanism underneath actually has. The web UI
/// that ships with `llama-server` asks before every call and remembers the
/// tools you said "always" to, and that pair -- a prompt and a growing set of
/// exceptions -- is the whole of its permission model. A third mode sitting
/// between them would have to invent a rule about which tools are "edits",
/// and the rule would be this app's opinion rather than anything the tools
/// declare.
pub const MODES: &[&str] = &["manual", "bypassPermissions"];
/// What a new llama session asks by default.
///
/// The cautious one, matching the web UI: a model with a shell on somebody's
/// own machine is the case to be wrong about in this direction, and one tap
/// on "always allow" is what makes it bearable afterwards.
pub const DEFAULT_MODE: &str = "manual";
/// The answer that makes an allowance permanent for the session. The tool's
/// name follows it, which is what makes the transcript alone enough to
/// rebuild the set -- see `super::allowed`.
pub const ALWAYS_PREFIX: &str = "Always allow ";
pub const ALLOW_ONCE: &str = "Allow once";
pub const REFUSE: &str = "Don't allow";
/// What the model is told when a call was refused.
///
/// Addressed to the model, not to the reader: it has to understand that the
/// work did not happen and that trying the same call again is not the way
/// round it, or it retries in a loop.
pub const REFUSED: &str = "The person using this session did not allow this call, so it was not run. Do not try it \
again -- say what you were going to do and why it needed that, and let them decide.";
/// What stands in for a call that never finished, when the transcript is read
/// back into a conversation.
///
/// Every tool call in the history owes a result, because that is the shape a
/// chat template renders; a turn stopped between the call and its result
/// leaves one that has none. Saying so is better than inventing an outcome,
/// and better than dropping the call -- which would tell the model it never
/// asked.
pub const UNFINISHED: &str = "This call was interrupted before it produced anything.";
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_tool_that_ran_reads_as_its_output() {
assert_eq!(
result_text(&json!({"plain_text_response": "hello\n"})),
"hello\n",
);
}
#[test]
/// The model is told what went wrong, because the model is what has to do
/// something about it -- read a different path, fix the command.
fn a_tool_that_failed_reads_as_its_message() {
assert_eq!(
result_text(&json!({"error": "cannot stat file: /tmp/nope"})),
"cannot stat file: /tmp/nope",
);
}
#[test]
fn anything_else_is_handed_over_as_itself() {
assert_eq!(result_text(&json!({"rows": 2})), "{\"rows\":2}");
}
}
server/src/session/mod.rs
+1
View File
@@ -3980,6 +3980,7 @@ mod tests {
kind: DriverKind::ClaudeCli,
command: Some(command.to_string_lossy().into_owned()),
models: Vec::new(),
mcp_servers: Vec::new(),
},
])],
..Config::default()