iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Condense the documentation and thin the server's comments

Browse Source 
The markdown had accumulated a lot that was stale rather than wrong.
PLAN.md still described pi as the llama.cpp harness, a refcounted
LlamaServerManager, and a providers-by-hosts cross-product, all of which
were superseded or never built; it also carried a second copy of the HTTP
table that routes.rs owns. EXPLORER.md and TRANSCRIPT_CACHE.md held
implementation checklists for work that has since landed. AGENTS.md
restated most of PLAN.md's design instead of being the working-notes
layer it says it is. 3225 lines of markdown to 2180, with the stale
sections gone rather than reworded.

On the server, comments explaining what the code already says are out and
the ones recording a constraint, a measurement or an incident are kept but
cut to a few lines each: 5504 comment lines to 4586.

Four doc comments in session/mod.rs, and one each in process.rs and
usage.rs, had drifted onto the item above the one they describe --
functions were reordered without them, so `stop_session`'s doc sat on
`set_session_cwd`, `stat_of`'s on `struct Stat`, and `UsageMonitor`'s on
`type Cached`. Each is back on its own item.

routes.rs's module table also claimed later phases would add `/hosts`,
which setups replaced.

cargo test (127 passed), clippy --all-targets and fmt are clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed 2026-09-04 15:45:43 -04:00
1 parent e3e02d55f7
commit 79682f03a7
24 files changed
+4572 -6821
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

server/src/session/llama.rs
+109 -132
View File
@@ -1,29 +1,22 @@
//! The llama.cpp driver: a `llama-server` process per session, spoken to
//! over its OpenAI-compatible HTTP API and translated into the common
//! event model.
//! The llama.cpp driver: a `llama-server` process per session, spoken to over
//! its OpenAI-compatible HTTP API and translated into the common event model.
//!
//! Two things make this shaped differently from the Claude driver, and
//! both are worth knowing before changing anything here.
//! Two things make this shaped differently from the Claude driver.
//!
//! **It is spawned but not spoken to over stdio.** The process is started
//! through the same [`Transport`] as any other, and then reached over
//! HTTP on a loopback port. That is the case the transport's doc comment
//! flags: a remote llama-server would need its port forwarded as well as
//! its command wrapped, which is not built, so a session on an ssh host
//! is refused rather than silently talking to the wrong machine.
//! through the same [`Transport`] as any other and then reached over HTTP on a
//! loopback port. A remote llama-server would need its port forwarded as well
//! as its command wrapped, which is not built, so a session on an ssh host is
//! refused rather than silently talking to the wrong machine.
//!
//! **The server is stateless between requests**, so the whole
//! conversation goes with every one. It is rebuilt from the session's
//! transcript rather than kept in this struct, which is not tidiness: a
//! copy in driver memory is invisible to a second device and gone when
//! this process restarts, and the app is meant to work across devices.
//! The transcript is already the source of truth for everything else, and
//! this makes it the source of truth for the prompt too.
//! **The server is stateless between requests**, so the whole conversation goes
//! with every one. It is rebuilt from the session's transcript rather than kept
//! in this struct, which is not tidiness: a copy in driver memory is invisible
//! to a second device and gone when this process restarts.
//!
//! That leaves the Claude driver as the odd one out rather than this one:
//! the CLI's own memory of a conversation is a cache in front of the same
//! transcript, not a second truth. Anyone tempted to "fix" the
//! inconsistency should resolve it in this direction.
//! That leaves the Claude driver as the odd one out rather than this one -- the
//! CLI's own memory of a conversation is a cache in front of the same
//! transcript. Resolve any inconsistency in this direction.
use std::path::{Path, PathBuf};
use std::sync::Arc;
@@ -38,10 +31,9 @@ use super::process;
use super::transport::{Launch, Streams, Transport};
use crate::config::{ProviderConfig, SessionConfig};
/// How long to wait for a model to load before giving up on it. Loading
/// is mostly disk, and a large quantised model on a cold cache is
/// genuinely slow, so this is generous -- the failure it exists for is a
/// server that will never answer, not one that is taking its time.
/// How long to wait for a model to load before giving up. Loading is mostly
/// disk, and a large quantised model on a cold cache is genuinely slow, so this
/// is generous -- the failure it exists for is a server that will never answer.
const READY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
/// One turn in the conversation this driver keeps on the server's behalf.
@@ -62,20 +54,19 @@ pub struct LlamaDriver {
/// Set by [`Driver::interrupt`]; the streaming loop checks it between
/// chunks and stops, leaving what was generated in the transcript.
cancel: Arc<AtomicBool>,
/// Where this session's process record lives, so [`Driver::stop`] can
/// find the server it has to end.
/// Where this session's process record lives, so [`Driver::stop`] can find
/// the server it has to end.
session_dir: PathBuf,
}
impl LlamaDriver {
/// Takes charge of this session's `llama-server`: the one already
/// loaded if there is one, otherwise a new one.
/// Takes charge of this session's `llama-server`: the one already loaded if
/// there is one, otherwise a new one.
///
/// One entry point, for the reason `ClaudeDriver::launch` gives -- the
/// choice is not the caller's and a second process is the expensive
/// mistake. Here it is expensive in a different currency: two servers
/// holding the same model is twice the memory, and the second would
/// bind a different port while the phone kept talking to the first.
/// One entry point, for the reason `ClaudeDriver::launch` gives, expensive
/// in a different currency: two servers holding the same model is twice the
/// memory, and the second would bind a different port while the phone kept
/// talking to the first.
pub fn launch(
meta: &SessionConfig,
provider: &ProviderConfig,
@@ -96,10 +87,10 @@ impl LlamaDriver {
)?;
let path = model_path(models_dir, model)?;
// Already loaded and still running: keep talking to it. The
// health poll below is what confirms it is really answering, so
// adopting a pid whose server has wedged still reports as a
// failure rather than as a session that silently never replies.
// Already loaded and still running: keep talking to it. The health poll
// below confirms it is really answering, so adopting a pid whose server
// has wedged still reports as a failure rather than as a session that
// silently never replies.
if let Some(process::Record {
detail: process::Detail::Http { port },
pid,
@@ -129,9 +120,9 @@ impl LlamaDriver {
"--port".into(),
port.to_string(),
];
// Settings that belong to the server because they decide how the
// model is loaded; the sampling ones ride on each request instead,
// so changing them later needn't reload anything.
// Settings that belong to the server because they decide how the model
// is loaded; the sampling ones ride on each request instead, so changing
// them later needn't reload anything.
for (key, flag) in [
("contextSize", "-c"),
("gpuLayers", "-ngl"),
@@ -147,8 +138,8 @@ impl LlamaDriver {
let launch = Launch::new(program, args, meta.cwd.as_deref());
// Its output goes to files, not pipes. Not only so the process can
// outlive this server: nothing ever read those pipes, so a chatty
// llama-server filled the 64 KB buffer and blocked mid-load with
// no sign of why.
// llama-server filled the 64 KB buffer and blocked mid-load with no sign
// of why.
let child = transport.spawn(
&launch,
Streams::Detached {
@@ -164,10 +155,9 @@ impl LlamaDriver {
"session {} running {program} for {model} on 127.0.0.1:{port} as pid {pid}",
meta.id
);
// Reaped so it does not become a zombie while this server is still
// its parent; the health poll and the record are what actually say
// whether the session is alive, because after a restart there is no
// `Child` here to ask.
// Reaped so it does not become a zombie while this server is still its
// parent; the health poll and the record are what say whether the
// session is alive, because after a restart there is no `Child` to ask.
tokio::spawn(async move {
let mut child = child;
let _ = child.wait().await;
@@ -189,10 +179,10 @@ impl LlamaDriver {
/// The driver for a `llama-server` at `endpoint`, however it got there.
///
/// Shared by starting one and adopting one, because everything after
/// "there is a server at this address" is identical -- including
/// waiting for it to answer, which an adopted one still owes: a
/// recorded pid says a process exists, not that its model is loaded.
/// Shared by starting one and adopting one, because everything after "there
/// is a server at this address" is identical -- including waiting for it to
/// answer, which an adopted one still owes: a recorded pid says a process
/// exists, not that its model is loaded.
fn attached(
endpoint: String,
meta: &SessionConfig,
@@ -201,9 +191,9 @@ impl LlamaDriver {
session_dir: &Path,
sink: EventSink,
) -> Self {
// Loading is slow enough to be worth saying so: the session shows
// as running until the model is in memory, then goes idle, rather
// than looking ready and refusing the first message.
// Loading is slow enough to be worth saying so: the session shows as
// running until the model is in memory, rather than looking ready and
// refusing the first message.
let _ = sink.send(Event::Status {
state: SessionStatus::Running,
});
@@ -262,11 +252,9 @@ impl LlamaDriver {
/// terminal anyway.
const SERVER_LOG: &str = "llama-server.log";
/// How often a loaded server is checked for still being there.
///
/// Slower than the Claude driver's stdout poll because nothing is waiting
/// on it: this only has to notice a server that has gone, and a few
/// seconds late costs nothing.
/// How often a loaded server is checked for still being there. Slower than the
/// Claude driver's stdout poll because nothing is waiting on it: this only has
/// to notice a server that has gone.
const WATCH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(2);
/// An owner-only log opened for appending, so the two streams pointed at
@@ -281,22 +269,21 @@ fn log_file(path: &Path) -> Result<std::fs::File> {
.with_context(|| format!("opening {}", path.display()))
}
/// Reports the server going away, for as long as the session is there to
/// report it to.
/// Reports the server going away, for as long as the session is there to report
/// it to.
///
/// Polled rather than waited on, for the reason the Claude driver gives:
/// after a restart this server is not the process's parent and has nothing
/// to wait on, so liveness has to be a question asked of the record -- and
/// asking it two different ways is how the two answers come to disagree.
/// Polled rather than waited on, for the reason the Claude driver gives: after a
/// restart this server is not the process's parent, so liveness has to be a
/// question asked of the record -- and asking it two different ways is how the
/// two answers come to disagree.
fn watch(session_dir: PathBuf, sink: EventSink) {
std::thread::spawn(move || {
loop {
std::thread::sleep(WATCH_INTERVAL);
match process::recorded(&session_dir) {
Some((_, process::Liveness::Alive)) => {}
// Nothing recorded means the session was stopped or
// deleted deliberately, and whoever did that has already
// said so.
// Nothing recorded means the session was stopped or deleted
// deliberately, and whoever did that has already said so.
None => return,
Some((_, process::Liveness::Dead)) => {
let _ = sink.send(Event::Error {
@@ -335,34 +322,33 @@ impl Driver for LlamaDriver {
let cancel = Arc::clone(&self.cancel);
cancel.store(false, Ordering::Relaxed);
// Its own thread: the request blocks for as long as the model
// takes to generate, which is the whole point of streaming it.
// Its own thread: the request blocks for as long as the model takes to
// generate, which is the whole point of streaming it.
std::thread::spawn(move || {
// Nothing is ever held back here -- there is no queue to wait
// in -- so the message is taken the moment it arrives. Said
// anyway, because this is what records it: see `MessageTaken`.
// Nothing is ever held back here -- there is no queue to wait in --
// so the message is taken the moment it arrives. Said anyway,
// because this is what records it: see `MessageTaken`.
let _ = sink.send(Event::MessageTaken {
id: None,
text: text.clone(),
// Never any: this driver refuses attachments above, and
// saying so is what the refusal above is for.
// Never any: this driver refuses attachments above.
attachments: Vec::new(),
});
let _ = sink.send(Event::Status {
state: SessionStatus::Running,
});
// Everything before this message, plus this message. Read
// rather than remembered, and `text` is appended here rather
// than waited for, because the message's own transcript entry
// is still on its way when this runs.
// Everything before this message, plus this message. Read rather
// than remembered, and `text` is appended here rather than waited
// for, because the message's own transcript entry is still on its
// way when this runs.
let mut messages = conversation(&transcript);
messages.push(Message {
role: "user".into(),
content: text,
});
// The reply is not stored: the deltas below are the durable
// record, so the next turn reads back exactly what the phone
// was shown -- including a partial one that was interrupted.
// The reply is not stored: the deltas below are the durable record,
// so the next turn reads back exactly what the phone was shown --
// including a partial one that was interrupted.
if let Err(err) = generate(&endpoint, &messages, &sampling, &cancel, &sink) {
let _ = sink.send(Event::Error {
message: format!("{err:#}"),
@@ -375,17 +361,15 @@ impl Driver for LlamaDriver {
}
fn answer_question(&self, _id: &str, _answers: &[String]) {
// Nothing here asks questions: this driver has no tools, so no
// permission prompts and no AskUserQuestion.
// Nothing here asks questions: this driver has no tools.
}
fn interrupt(&self) {
self.cancel.store(true, Ordering::Relaxed);
}
// Nothing to forward: this process has no notion of what the
// conversation is called, and the rename it belongs to has already
// happened where the name lives. See `Driver::set_title`.
// Nothing to forward: this process has no notion of what the conversation
// is called, and the rename has already happened where the name lives.
fn set_title(&self, _title: &str) {}
fn set_permission_mode(&self, _mode: &str) {
@@ -420,23 +404,21 @@ impl Driver for LlamaDriver {
}
fn clear(&self) {
// All of it. `conversation` folds from the last of these, so
// recording the marker *is* the reset -- there is no driver state
// to keep in step with it, which is the same property that makes
// a second device see the same conversation this one does.
// All of it. `conversation` folds from the last of these, so recording
// the marker *is* the reset -- there is no driver state to keep in step
// with it, which is the same property that makes a second device see the
// same conversation this one does.
let _ = self.sink.send(Event::Cleared);
}
/// Stops generating and leaves the server loaded.
///
/// Worth being deliberate about, because the cost is asymmetric and
/// points the other way from the Claude driver's: a `llama-server`
/// holds its whole model in memory, so a leaked one is gigabytes
/// nobody is using. It is left anyway, because the alternative is
/// unloading and reloading that model on every backend restart --
/// minutes of disk, for a session somebody is in the middle of. The
/// record is what keeps it from being *nobody's*: the next run of this
/// server adopts it rather than starting a second one.
/// Worth being deliberate about, because the cost points the other way from
/// the Claude driver's: a `llama-server` holds its whole model in memory, so
/// a leaked one is gigabytes nobody is using. It is left anyway, because the
/// alternative is unloading and reloading that model on every backend
/// restart -- minutes of disk, for a session somebody is in the middle of.
/// The record is what keeps it from being *nobody's*.
fn detach(&self) {
self.cancel.store(true, Ordering::Relaxed);
}
@@ -452,16 +434,15 @@ impl Driver for LlamaDriver {
/// The conversation so far, folded out of the transcript.
///
/// Consecutive `AssistantText` deltas are one assistant turn, closed by
/// the next user message -- which is also what makes an interrupted reply
/// come back as the partial text the phone actually saw, rather than
/// vanishing or being invented.
/// Consecutive `AssistantText` deltas are one assistant turn, closed by the next
/// user message -- which is also what makes an interrupted reply come back as
/// the partial text the phone actually saw.
///
/// This must stay a pure function of the transcript and must never
/// re-render earlier turns. llama.cpp caches the prompt prefix, so a
/// growing conversation reprocesses almost nothing -- but only while
/// every turn is byte-identical to last time. Changing how an old turn is
/// rendered silently reprocesses the whole history on every message.
/// This must stay a pure function of the transcript and must never re-render
/// earlier turns. llama.cpp caches the prompt prefix, so a growing conversation
/// reprocesses almost nothing -- but only while every turn is byte-identical to
/// last time. Changing how an old turn is rendered silently reprocesses the
/// whole history on every message.
fn conversation(path: &Path) -> Vec<Message> {
let Ok(events) = crate::session::transcript::read_after(path, 0) else {
return Vec::new();
@@ -469,8 +450,8 @@ fn conversation(path: &Path) -> Vec<Message> {
let mut messages: Vec<Message> = Vec::new();
let mut pending = String::new();
// Everything before the last clear is still in the transcript and is
// deliberately not in the conversation. Folding from zero here would
// put it back, which is the whole of what clearing had to undo.
// deliberately not in the conversation. Folding from zero would put it back,
// which is the whole of what clearing had to undo.
let events = match events.iter().rposition(|e| e.event == Event::Cleared) {
Some(at) => &events[at + 1..],
None => &events[..],
@@ -518,12 +499,10 @@ fn model_path(models_dir: &Path, key: &str) -> Result<PathBuf> {
Ok(path)
}
/// An unused loopback port, by asking the OS for one and letting it go.
///
/// Racy in principle: something else could take it between here and
/// llama-server binding. In practice nothing on this machine is hunting
/// for ports, and the alternative -- parsing the port back out of the
/// server's log -- couples us to its output format for no real gain.
/// An unused loopback port, by asking the OS for one and letting it go. Racy in
/// principle, but nothing on this machine is hunting for ports, and the
/// alternative -- parsing the port back out of the server's log -- couples us to
/// its output format for no real gain.
fn free_port() -> Result<u16> {
let listener = std::net::TcpListener::bind("127.0.0.1:0")?;
Ok(listener.local_addr()?.port())
@@ -547,8 +526,8 @@ fn wait_until_ready(endpoint: &str) -> Result<()> {
}
/// One streamed completion: posts the conversation, emits each delta as it
/// arrives. Emits rather than returns: the transcript those events land
/// in is what the next turn reads back, so there is nothing to hand up.
/// arrives. Emits rather than returns, because the transcript those events land
/// in is what the next turn reads back.
fn generate(
endpoint: &str,
messages: &[Message],
@@ -574,16 +553,15 @@ fn generate(
let reader = std::io::BufReader::new(response.body_mut().as_reader());
let mut tokens = 0u64;
// The prompt side only, which is what the model is holding -- the same
// definition the other dialects report, so one word on the phone means
// one thing whichever kind of session it is.
// definition the other dialects report, so one word on the phone means one
// thing whichever kind of session it is.
let mut context = None;
for line in std::io::BufRead::lines(reader) {
if cancel.load(Ordering::Relaxed) {
break;
}
let line = line.context("reading the generation stream")?;
// Server-sent events: the payload lines are the ones that matter,
// and blank lines separate events.
// Server-sent events: the payload lines are the ones that matter.
let Some(payload) = line.strip_prefix("data: ") else {
continue;
};
@@ -631,8 +609,8 @@ mod tests {
use super::*;
use crate::session::transcript::Transcript;
/// Writes a transcript the way the pump does, so the fold is tested
/// against the real file format rather than a hand-built vector.
/// Writes a transcript the way the pump does, so the fold is tested against
/// the real file format rather than a hand-built vector.
fn transcript_with(events: &[Event]) -> (tempfile::TempDir, PathBuf) {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("transcript.jsonl");
@@ -685,11 +663,10 @@ mod tests {
}
#[test]
/// The interrupted case, which decides what a resumed conversation is
/// built from: whatever the phone was shown. The deltas that arrived
/// before the stop are in the transcript, so they are in the prompt --
/// the model is never told it said something the user did not see, and
/// never has a turn silently dropped from under it.
/// The interrupted case, which decides what a resumed conversation is built
/// from: whatever the phone was shown. The deltas that arrived before the
/// stop are in the transcript, so they are in the prompt -- the model is
/// never told it said something the user did not see.
fn an_interrupted_reply_stays_in_the_conversation() {
let (_dir, path) = transcript_with(&[
Event::UserMessage {
@@ -741,9 +718,9 @@ mod tests {
}
#[test]
/// Clearing decides what the *model* is given, not just what the
/// phone draws. Everything above the marker stays in the transcript
/// -- a person can still scroll back to it -- and none of it is sent.
/// Clearing decides what the *model* is given, not just what the phone
/// draws. Everything above the marker stays in the transcript and none of it
/// is sent.
fn the_conversation_starts_after_the_last_clear() {
let (_dir, path) = transcript_with(&[
Event::UserMessage {