iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'main' of git.arirex.me:iris/ai-app

Browse Source 
# Conflicts:
#	AGENTS.md
#	PLAN.md
#	app/androidApp/src/main/kotlin/com/example/aiapp/SessionUsageBar.kt
#	app/androidApp/src/main/kotlin/com/example/aiapp/SpawnScreen.kt
#	server/src/config.rs
#	server/src/main.rs
#	server/src/routes.rs
#	server/src/session/echo.rs
#	server/src/session/llama.rs
#	server/src/session/transport.rs
#	server/src/ssh.rs
#	server/src/usage.rs
This commit is contained in:
iris committed 2026-09-04 17:56:50 -04:00
2 parents 127b25e60a 1fcaa2d72d
commit 3c0214ece8
94 files changed
+8299 -9454
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

server/src/session/llama.rs
+135 -166
View File
@@ -1,43 +1,32 @@
//! The llama.cpp driver: a `llama-server` process per session, spoken to
//! over its OpenAI-compatible HTTP API and translated into the common
//! event model.
//! The llama.cpp driver: a `llama-server` process per session, spoken to over
//! its OpenAI-compatible HTTP API and translated into the common event model.
//!
//! Two things make this shaped differently from the Claude driver, and
//! both are worth knowing before changing anything here.
//! Two things make this shaped differently from the Claude driver.
//!
//! **It is spawned but not spoken to over stdio.** The process is started
//! through the same [`Transport`] as any other, and then reached over
//! HTTP on a loopback port. That is the second half of what a transport
//! is -- "run this" plus "reach this port" -- and it is what lets a
//! session run on another machine: [`Transport::reserve_port`] hands back
//! a port the server binds *there* and a port that reaches it *here*, and
//! the ssh connection carrying the command carries the tunnel between
//! them. The far `llama-server` binds loopback only, so a model is never
//! served to that machine's network.
//! through the same [`Transport`] as any other and then reached over HTTP on a
//! loopback port. That is the second half of what a transport is -- "run this"
//! plus "reach this port" -- and it is what lets a session run on another
//! machine: [`Transport::reserve_port`] hands back a port the server binds
//! *there* and one that reaches it *here*, and the ssh connection carrying the
//! command carries the tunnel between them. The far `llama-server` binds
//! loopback only, so a model is never served to that machine's network.
//!
//! **The model file is the far machine's, not this one's.** A session
//! serves a GGUF from the machine that runs `llama-server`, so a remote
//! setup names its own models directory (`SshConfig::models_dir`,
//! defaulting to the same place this backend keeps its own downloads).
//! What this backend has downloaded is on that machine only when they are
//! the same machine -- so the file is looked for *there*, and a session
//! that names a model the machine does not have says so instead of
//! starting a server that will never load one. Downloading to another
//! machine is not built; the model gets there however anything else
//! gets there.
//! **The model file is the far machine's, not this one's.** A remote setup
//! names its own models directory (`SshConfig::models_dir`, defaulting to where
//! this backend keeps its downloads), and the file is looked for *there* -- so
//! a session naming a model that machine does not have says so, instead of
//! starting a server that will never load one. Downloading to another machine
//! is not built; the model gets there however anything else does.
//!
//! **The server is stateless between requests**, so the whole
//! conversation goes with every one. It is rebuilt from the session's
//! transcript rather than kept in this struct, which is not tidiness: a
//! copy in driver memory is invisible to a second device and gone when
//! this process restarts, and the app is meant to work across devices.
//! The transcript is already the source of truth for everything else, and
//! this makes it the source of truth for the prompt too.
//! **The server is stateless between requests**, so the whole conversation goes
//! with every one. It is rebuilt from the session's transcript rather than kept
//! in this struct, which is not tidiness: a copy in driver memory is invisible
//! to a second device and gone when this process restarts.
//!
//! That leaves the Claude driver as the odd one out rather than this one:
//! the CLI's own memory of a conversation is a cache in front of the same
//! transcript, not a second truth. Anyone tempted to "fix" the
//! inconsistency should resolve it in this direction.
//! That leaves the Claude driver as the odd one out rather than this one -- the
//! CLI's own memory of a conversation is a cache in front of the same
//! transcript. Resolve any inconsistency in this direction.
use std::path::{Path, PathBuf};
use std::sync::Arc;
@@ -52,10 +41,9 @@ use super::process;
use super::transport::{Launch, Streams, Transport};
use crate::config::{ProviderConfig, SessionConfig};
/// How long to wait for a model to load before giving up on it. Loading
/// is mostly disk, and a large quantised model on a cold cache is
/// genuinely slow, so this is generous -- the failure it exists for is a
/// server that will never answer, not one that is taking its time.
/// How long to wait for a model to load before giving up. Loading is mostly
/// disk, and a large quantised model on a cold cache is genuinely slow, so this
/// is generous -- the failure it exists for is a server that will never answer.
const READY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
/// One turn in the conversation this driver keeps on the server's behalf.
@@ -76,20 +64,19 @@ pub struct LlamaDriver {
/// Set by [`Driver::interrupt`]; the streaming loop checks it between
/// chunks and stops, leaving what was generated in the transcript.
cancel: Arc<AtomicBool>,
/// Where this session's process record lives, so [`Driver::stop`] can
/// find the server it has to end.
/// Where this session's process record lives, so [`Driver::stop`] can find
/// the server it has to end.
session_dir: PathBuf,
}
impl LlamaDriver {
/// Takes charge of this session's `llama-server`: the one already
/// loaded if there is one, otherwise a new one.
/// Takes charge of this session's `llama-server`: the one already loaded if
/// there is one, otherwise a new one.
///
/// One entry point, for the reason `ClaudeDriver::launch` gives -- the
/// choice is not the caller's and a second process is the expensive
/// mistake. Here it is expensive in a different currency: two servers
/// holding the same model is twice the memory, and the second would
/// bind a different port while the phone kept talking to the first.
/// One entry point, for the reason `ClaudeDriver::launch` gives, expensive
/// in a different currency: two servers holding the same model is twice the
/// memory, and the second would bind a different port while the phone kept
/// talking to the first.
pub fn launch(
meta: &SessionConfig,
provider: &ProviderConfig,
@@ -104,10 +91,10 @@ impl LlamaDriver {
)?;
let path = model_on(transport, models_dir, model)?;
// Already loaded and still running: keep talking to it. The
// health poll below is what confirms it is really answering, so
// adopting a pid whose server has wedged still reports as a
// failure rather than as a session that silently never replies.
// Already loaded and still running: keep talking to it. The health poll
// below confirms it is really answering, so adopting a pid whose server
// has wedged still reports as a failure rather than as a session that
// silently never replies.
if let Some(process::Record {
detail: process::Detail::Http { port },
pid,
@@ -144,9 +131,9 @@ impl LlamaDriver {
"--port".into(),
forward.there.to_string(),
];
// Settings that belong to the server because they decide how the
// model is loaded; the sampling ones ride on each request instead,
// so changing them later needn't reload anything.
// Settings that belong to the server because they decide how the model
// is loaded; the sampling ones ride on each request instead, so changing
// them later needn't reload anything.
for (key, flag) in [
("contextSize", "-c"),
("gpuLayers", "-ngl"),
@@ -162,8 +149,8 @@ impl LlamaDriver {
let launch = Launch::new(program, args, meta.cwd.as_deref()).reaching(forward);
// Its output goes to files, not pipes. Not only so the process can
// outlive this server: nothing ever read those pipes, so a chatty
// llama-server filled the 64 KB buffer and blocked mid-load with
// no sign of why.
// llama-server filled the 64 KB buffer and blocked mid-load with no sign
// of why.
let child = transport.spawn(
&launch,
Streams::Detached {
@@ -183,10 +170,9 @@ impl LlamaDriver {
forward.there,
forward.here,
);
// Reaped so it does not become a zombie while this server is still
// its parent; the health poll and the record are what actually say
// whether the session is alive, because after a restart there is no
// `Child` here to ask.
// Reaped so it does not become a zombie while this server is still its
// parent; the health poll and the record are what say whether the
// session is alive, because after a restart there is no `Child` to ask.
tokio::spawn(async move {
let mut child = child;
let _ = child.wait().await;
@@ -214,10 +200,10 @@ impl LlamaDriver {
/// The driver for a `llama-server` at `endpoint`, however it got there.
///
/// Shared by starting one and adopting one, because everything after
/// "there is a server at this address" is identical -- including
/// waiting for it to answer, which an adopted one still owes: a
/// recorded pid says a process exists, not that its model is loaded.
/// Shared by starting one and adopting one, because everything after "there
/// is a server at this address" is identical -- including waiting for it to
/// answer, which an adopted one still owes: a recorded pid says a process
/// exists, not that its model is loaded.
fn attached(
endpoint: String,
meta: &SessionConfig,
@@ -226,9 +212,9 @@ impl LlamaDriver {
session_dir: &Path,
sink: EventSink,
) -> Self {
// Loading is slow enough to be worth saying so: the session shows
// as running until the model is in memory, then goes idle, rather
// than looking ready and refusing the first message.
// Loading is slow enough to be worth saying so: the session shows as
// running until the model is in memory, rather than looking ready and
// refusing the first message.
let _ = sink.send(Event::Status {
state: SessionStatus::Running,
});
@@ -287,11 +273,9 @@ impl LlamaDriver {
/// terminal anyway.
const SERVER_LOG: &str = "llama-server.log";
/// How often a loaded server is checked for still being there.
///
/// Slower than the Claude driver's stdout poll because nothing is waiting
/// on it: this only has to notice a server that has gone, and a few
/// seconds late costs nothing.
/// How often a loaded server is checked for still being there. Slower than the
/// Claude driver's stdout poll because nothing is waiting on it: this only has
/// to notice a server that has gone.
const WATCH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(2);
/// An owner-only log opened for appending, so the two streams pointed at
@@ -306,22 +290,21 @@ fn log_file(path: &Path) -> Result<std::fs::File> {
.with_context(|| format!("opening {}", path.display()))
}
/// Reports the server going away, for as long as the session is there to
/// report it to.
/// Reports the server going away, for as long as the session is there to report
/// it to.
///
/// Polled rather than waited on, for the reason the Claude driver gives:
/// after a restart this server is not the process's parent and has nothing
/// to wait on, so liveness has to be a question asked of the record -- and
/// asking it two different ways is how the two answers come to disagree.
/// Polled rather than waited on, for the reason the Claude driver gives: after a
/// restart this server is not the process's parent, so liveness has to be a
/// question asked of the record -- and asking it two different ways is how the
/// two answers come to disagree.
fn watch(session_dir: PathBuf, sink: EventSink) {
std::thread::spawn(move || {
loop {
std::thread::sleep(WATCH_INTERVAL);
match process::recorded(&session_dir) {
Some((_, process::Liveness::Alive)) => {}
// Nothing recorded means the session was stopped or
// deleted deliberately, and whoever did that has already
// said so.
// Nothing recorded means the session was stopped or deleted
// deliberately, and whoever did that has already said so.
None => return,
Some((_, process::Liveness::Dead)) => {
let _ = sink.send(Event::Error {
@@ -360,34 +343,33 @@ impl Driver for LlamaDriver {
let cancel = Arc::clone(&self.cancel);
cancel.store(false, Ordering::Relaxed);
// Its own thread: the request blocks for as long as the model
// takes to generate, which is the whole point of streaming it.
// Its own thread: the request blocks for as long as the model takes to
// generate, which is the whole point of streaming it.
std::thread::spawn(move || {
// Nothing is ever held back here -- there is no queue to wait
// in -- so the message is taken the moment it arrives. Said
// anyway, because this is what records it: see `MessageTaken`.
// Nothing is ever held back here -- there is no queue to wait in --
// so the message is taken the moment it arrives. Said anyway,
// because this is what records it: see `MessageTaken`.
let _ = sink.send(Event::MessageTaken {
id: None,
text: text.clone(),
// Never any: this driver refuses attachments above, and
// saying so is what the refusal above is for.
// Never any: this driver refuses attachments above.
attachments: Vec::new(),
});
let _ = sink.send(Event::Status {
state: SessionStatus::Running,
});
// Everything before this message, plus this message. Read
// rather than remembered, and `text` is appended here rather
// than waited for, because the message's own transcript entry
// is still on its way when this runs.
// Everything before this message, plus this message. Read rather
// than remembered, and `text` is appended here rather than waited
// for, because the message's own transcript entry is still on its
// way when this runs.
let mut messages = conversation(&transcript);
messages.push(Message {
role: "user".into(),
content: text,
});
// The reply is not stored: the deltas below are the durable
// record, so the next turn reads back exactly what the phone
// was shown -- including a partial one that was interrupted.
// The reply is not stored: the deltas below are the durable record,
// so the next turn reads back exactly what the phone was shown --
// including a partial one that was interrupted.
if let Err(err) = generate(&endpoint, &messages, &sampling, &cancel, &sink) {
let _ = sink.send(Event::Error {
message: format!("{err:#}"),
@@ -400,17 +382,15 @@ impl Driver for LlamaDriver {
}
fn answer_question(&self, _id: &str, _answers: &[String]) {
// Nothing here asks questions: this driver has no tools, so no
// permission prompts and no AskUserQuestion.
// Nothing here asks questions: this driver has no tools.
}
fn interrupt(&self) {
self.cancel.store(true, Ordering::Relaxed);
}
// Nothing to forward: this process has no notion of what the
// conversation is called, and the rename it belongs to has already
// happened where the name lives. See `Driver::set_title`.
// Nothing to forward: this process has no notion of what the conversation
// is called, and the rename has already happened where the name lives.
fn set_title(&self, _title: &str) {}
fn set_permission_mode(&self, _mode: &str) {
@@ -445,23 +425,21 @@ impl Driver for LlamaDriver {
}
fn clear(&self) {
// All of it. `conversation` folds from the last of these, so
// recording the marker *is* the reset -- there is no driver state
// to keep in step with it, which is the same property that makes
// a second device see the same conversation this one does.
// All of it. `conversation` folds from the last of these, so recording
// the marker *is* the reset -- there is no driver state to keep in step
// with it, which is the same property that makes a second device see the
// same conversation this one does.
let _ = self.sink.send(Event::Cleared);
}
/// Stops generating and leaves the server loaded.
///
/// Worth being deliberate about, because the cost is asymmetric and
/// points the other way from the Claude driver's: a `llama-server`
/// holds its whole model in memory, so a leaked one is gigabytes
/// nobody is using. It is left anyway, because the alternative is
/// unloading and reloading that model on every backend restart --
/// minutes of disk, for a session somebody is in the middle of. The
/// record is what keeps it from being *nobody's*: the next run of this
/// server adopts it rather than starting a second one.
/// Worth being deliberate about, because the cost points the other way from
/// the Claude driver's: a `llama-server` holds its whole model in memory, so
/// a leaked one is gigabytes nobody is using. It is left anyway, because the
/// alternative is unloading and reloading that model on every backend
/// restart -- minutes of disk, for a session somebody is in the middle of.
/// The record is what keeps it from being *nobody's*.
fn detach(&self) {
self.cancel.store(true, Ordering::Relaxed);
}
@@ -477,16 +455,15 @@ impl Driver for LlamaDriver {
/// The conversation so far, folded out of the transcript.
///
/// Consecutive `AssistantText` deltas are one assistant turn, closed by
/// the next user message -- which is also what makes an interrupted reply
/// come back as the partial text the phone actually saw, rather than
/// vanishing or being invented.
/// Consecutive `AssistantText` deltas are one assistant turn, closed by the next
/// user message -- which is also what makes an interrupted reply come back as
/// the partial text the phone actually saw.
///
/// This must stay a pure function of the transcript and must never
/// re-render earlier turns. llama.cpp caches the prompt prefix, so a
/// growing conversation reprocesses almost nothing -- but only while
/// every turn is byte-identical to last time. Changing how an old turn is
/// rendered silently reprocesses the whole history on every message.
/// This must stay a pure function of the transcript and must never re-render
/// earlier turns. llama.cpp caches the prompt prefix, so a growing conversation
/// reprocesses almost nothing -- but only while every turn is byte-identical to
/// last time. Changing how an old turn is rendered silently reprocesses the
/// whole history on every message.
fn conversation(path: &Path) -> Vec<Message> {
let Ok(events) = crate::session::transcript::read_after(path, 0) else {
return Vec::new();
@@ -494,8 +471,8 @@ fn conversation(path: &Path) -> Vec<Message> {
let mut messages: Vec<Message> = Vec::new();
let mut pending = String::new();
// Everything before the last clear is still in the transcript and is
// deliberately not in the conversation. Folding from zero here would
// put it back, which is the whole of what clearing had to undo.
// deliberately not in the conversation. Folding from zero would put it back,
// which is the whole of what clearing had to undo.
let events = match events.iter().rposition(|e| e.event == Event::Cleared) {
Some(at) => &events[at + 1..],
None => &events[..],
@@ -543,46 +520,40 @@ fn model_path(models_dir: &Path, key: &str) -> Result<PathBuf> {
Ok(path)
}
/// The model file's path **on the machine that will serve it**, confirmed
/// to be there.
/// The model file's path **on the machine that will serve it**, confirmed to be
/// there.
///
/// Local and remote answer the same question and it has to be asked of
/// two different filesystems, which is why this is one function rather
/// than a check beside the local path and hope for the other case. The
/// remote answer is measured for the same reason the local one is: a
/// missing file otherwise becomes a `llama-server` that starts, fails to
/// load, and reports as a session that never became ready -- which reads
/// as the machine being slow.
/// One function rather than a local check and hope for the other case: the same
/// question has to be asked of two filesystems. The remote answer is measured
/// for the reason the local one is -- a missing file otherwise becomes a
/// `llama-server` that starts, fails to load, and reports as a session that
/// never became ready, which reads as the machine being slow.
///
/// One blocking round trip on a remote spawn, which is the same cost the
/// spawn is already paying to start ssh. The alternative is a path built
/// here from a `~` this machine cannot expand.
/// One blocking round trip on a remote spawn, which is what the spawn is
/// already paying to start ssh. The alternative is a path built here from a `~`
/// this machine cannot expand.
fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result<String> {
let Transport::Ssh { name, .. } = transport else {
return Ok(model_path(models_dir, key)?.to_string_lossy().into_owned());
};
// The same directory the spawn screen listed for this machine, and
// for the same reason it is one function: a list from one place and a
// load from another is a model that appears and then fails.
// The same directory the spawn screen listed for this machine, and one
// function for the same reason: a list from one place and a load from
// another is a model that appears and then fails.
let dir = crate::models::dir_on(transport, models_dir);
// Checked here rather than in the script: `..` in a key would walk
// out of the models directory on a machine this server can start
// processes on, and the phone is where the key comes from.
// Checked here rather than in the script: `..` in a key would walk out of
// the models directory on a machine this server can start processes on,
// and the phone is where the key comes from.
for part in key.split('/') {
if part.is_empty() || part == "." || part == ".." {
bail!("\"{key}\" is not a model key this can resolve");
}
}
let path = format!("{}/{key}", dir.trim_end_matches('/'));
// `$HOME` on the far side, which is the only machine that knows what
// it is -- and the resolved path is printed back so the launch below
// hands `llama-server` something absolute.
//
// "the file is not there" is answered rather than failed, because the
// two are different things to a reader and only one of them is a
// fault: a machine that could not be asked at all has to say so in
// its own words, and it would otherwise arrive as this same sentence
// about a missing model.
// `$HOME` on the far side, which is the only machine that knows what it is,
// and the resolved path printed back so the launch hands `llama-server`
// something absolute. "Not there" is answered rather than failed, because a
// machine that could not be asked at all has to say so in its own words --
// it would otherwise arrive as this same sentence about a missing model.
let script = "p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${p#\"~/\"};; esac; \
[ -f \"$p\" ] && printf 'at\\t%s\\n' \"$p\" || printf 'missing\\n'"
.to_string();
@@ -664,8 +635,8 @@ fn log_tail(session_dir: &Path) -> String {
const LOG_TAIL_LINES: usize = 6;
/// One streamed completion: posts the conversation, emits each delta as it
/// arrives. Emits rather than returns: the transcript those events land
/// in is what the next turn reads back, so there is nothing to hand up.
/// arrives. Emits rather than returns, because the transcript those events land
/// in is what the next turn reads back.
fn generate(
endpoint: &str,
messages: &[Message],
@@ -691,16 +662,15 @@ fn generate(
let reader = std::io::BufReader::new(response.body_mut().as_reader());
let mut tokens = 0u64;
// The prompt side only, which is what the model is holding -- the same
// definition the other dialects report, so one word on the phone means
// one thing whichever kind of session it is.
// definition the other dialects report, so one word on the phone means one
// thing whichever kind of session it is.
let mut context = None;
for line in std::io::BufRead::lines(reader) {
if cancel.load(Ordering::Relaxed) {
break;
}
let line = line.context("reading the generation stream")?;
// Server-sent events: the payload lines are the ones that matter,
// and blank lines separate events.
// Server-sent events: the payload lines are the ones that matter.
let Some(payload) = line.strip_prefix("data: ") else {
continue;
};
@@ -748,8 +718,8 @@ mod tests {
use super::*;
use crate::session::transcript::Transcript;
/// Writes a transcript the way the pump does, so the fold is tested
/// against the real file format rather than a hand-built vector.
/// Writes a transcript the way the pump does, so the fold is tested against
/// the real file format rather than a hand-built vector.
fn transcript_with(events: &[Event]) -> (tempfile::TempDir, PathBuf) {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join("transcript.jsonl");
@@ -802,11 +772,10 @@ mod tests {
}
#[test]
/// The interrupted case, which decides what a resumed conversation is
/// built from: whatever the phone was shown. The deltas that arrived
/// before the stop are in the transcript, so they are in the prompt --
/// the model is never told it said something the user did not see, and
/// never has a turn silently dropped from under it.
/// The interrupted case, which decides what a resumed conversation is built
/// from: whatever the phone was shown. The deltas that arrived before the
/// stop are in the transcript, so they are in the prompt -- the model is
/// never told it said something the user did not see.
fn an_interrupted_reply_stays_in_the_conversation() {
let (_dir, path) = transcript_with(&[
Event::UserMessage {
@@ -858,9 +827,9 @@ mod tests {
}
#[test]
/// Clearing decides what the *model* is given, not just what the
/// phone draws. Everything above the marker stays in the transcript
/// -- a person can still scroll back to it -- and none of it is sent.
/// Clearing decides what the *model* is given, not just what the phone
/// draws. Everything above the marker stays in the transcript and none of it
/// is sent.
fn the_conversation_starts_after_the_last_clear() {
let (_dir, path) = transcript_with(&[
Event::UserMessage {