iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Meter a session by its provider, and let llama.cpp run over ssh

Browse Source 
The rate-limit bar answered a question about an account, and picked the
answer by machine. One machine runs echo, the Claude CLI and a local
model side by side, so every echo session on it drew the CLI's five-hour
window: a quota that session cannot spend and could never run down. A
session now names its meter (`usageProvider`, from
`DriverKind::usage_provider`, which `usage::providers_for` reads too so
the two lists cannot disagree), and the phone matches on machine *and*
provider. Nothing meters echo or llama, and nothing at all is drawn --
including while the first fetch is out, since "checking" under a session
that turns out to meter nothing is a row the screen then withdraws.

Echo gets a meter it can be *told* about instead: `/usage 42`,
`/usage 95 20`, `/usage 42 never`, `/usage notloggedin`,
`/usage unreachable`, `/usage failed`, `/usage off`. Those states cost
real quota to arrange, which is why none of them had been looked at.

And llama.cpp runs wherever a setup says, which was the last of phase 5.
`Transport::reserve_port` is the second half of what a transport is --
"run this" plus "reach this port" -- returning the port the server binds
there and the port that reaches it here, and `Launch::reaching` puts the
`-L` tunnel on the connection that already carries the command. Three
things that came out of building it:

- A forwarded launch gets a pty and every other one keeps `-T`. Killing
  the ssh client ends a CLI by closing the stdin it reads; llama-server
  never reads its stdin, so the same kill left it running on the far
  machine with the model loaded -- one orphan per stopped session.
- The model is looked for on the machine that will serve it, at that
  machine's own models directory, so `GET /setups/{id}/models` is what
  the spawn screen offers rather than the backend's own downloads.
- The readiness poll watches the process, not only the port: a model
  that will not load exits in a second and would otherwise have been
  reported as "gave up after 300s". The failure carries the log's tail.

Exercised end to end against this VM over ssh to itself: spawn, load,
answer, outlive a backend restart, be adopted, answer again, and stop --
with both the ssh client and the far llama-server gone afterwards. The
local path, the Claude bar and the spawn screen checked on the emulator.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed 2026-09-04 17:45:32 -04:00
1 parent 74110b4d72
commit 127b25e60a
20 files changed
+1212 -143
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

server/src/session/llama.rs
+147 -30
View File
@@ -7,10 +7,24 @@
//!
//! **It is spawned but not spoken to over stdio.** The process is started
//! through the same [`Transport`] as any other, and then reached over
//! HTTP on a loopback port. That is the case the transport's doc comment
//! flags: a remote llama-server would need its port forwarded as well as
//! its command wrapped, which is not built, so a session on an ssh host
//! is refused rather than silently talking to the wrong machine.
//! HTTP on a loopback port. That is the second half of what a transport
//! is -- "run this" plus "reach this port" -- and it is what lets a
//! session run on another machine: [`Transport::reserve_port`] hands back
//! a port the server binds *there* and a port that reaches it *here*, and
//! the ssh connection carrying the command carries the tunnel between
//! them. The far `llama-server` binds loopback only, so a model is never
//! served to that machine's network.
//!
//! **The model file is the far machine's, not this one's.** A session
//! serves a GGUF from the machine that runs `llama-server`, so a remote
//! setup names its own models directory (`SshConfig::models_dir`,
//! defaulting to the same place this backend keeps its own downloads).
//! What this backend has downloaded is on that machine only when they are
//! the same machine -- so the file is looked for *there*, and a session
//! that names a model the machine does not have says so instead of
//! starting a server that will never load one. Downloading to another
//! machine is not built; the model gets there however anything else
//! gets there.
//!
//! **The server is stateless between requests**, so the whole
//! conversation goes with every one. It is rebuilt from the session's
@@ -85,16 +99,10 @@ impl LlamaDriver {
session_dir: &Path,
sink: EventSink,
) -> Result<Self> {
if !matches!(transport, Transport::Here) {
bail!(
"llama.cpp sessions can only run on this machine for now: the model is served \
over HTTP, and forwarding that port to another host isn't built yet."
);
}
let model = meta.model.as_deref().context(
"a llama.cpp session needs a model -- one of the downloaded ones, by its key",
)?;
let path = model_path(models_dir, model)?;
let path = model_on(transport, models_dir, model)?;
// Already loaded and still running: keep talking to it. The
// health poll below is what confirms it is really answering, so
@@ -120,14 +128,21 @@ impl LlamaDriver {
));
}
let port = free_port().context("finding a port for llama-server")?;
// Where it listens on its own machine, and where that is reached
// from here -- the same number when that machine is this one.
let forward = transport
.reserve_port()
.context("finding a port for llama-server")?;
let mut args: Vec<String> = vec![
"-m".into(),
path.to_string_lossy().into_owned(),
path.clone(),
// Loopback there, whichever machine there is: what reaches it
// from outside that machine is the ssh tunnel and nothing
// else.
"--host".into(),
"127.0.0.1".into(),
"--port".into(),
port.to_string(),
forward.there.to_string(),
];
// Settings that belong to the server because they decide how the
// model is loaded; the sampling ones ride on each request instead,
@@ -144,7 +159,7 @@ impl LlamaDriver {
}
let program = provider.command.as_deref().unwrap_or("llama-server");
let launch = Launch::new(program, args, meta.cwd.as_deref());
let launch = Launch::new(program, args, meta.cwd.as_deref()).reaching(forward);
// Its output goes to files, not pipes. Not only so the process can
// outlive this server: nothing ever read those pipes, so a chatty
// llama-server filled the 64 KB buffer and blocked mid-load with
@@ -161,8 +176,12 @@ impl LlamaDriver {
.id()
.context("llama-server exited before it could be recorded")?;
tracing::info!(
"session {} running {program} for {model} on 127.0.0.1:{port} as pid {pid}",
meta.id
"session {} running {program} for {model} {} on 127.0.0.1:{} there, \
reached at 127.0.0.1:{} here, as pid {pid}",
meta.id,
transport.describe(),
forward.there,
forward.here,
);
// Reaped so it does not become a zombie while this server is still
// its parent; the health poll and the record are what actually say
@@ -173,12 +192,18 @@ impl LlamaDriver {
let _ = child.wait().await;
});
let record = process::Record::of(pid, process::Detail::Http { port })
// The *near* port, because that is the one anything reaching this
// server has to dial -- including a later run of this backend,
// which adopts the record without knowing which machine the server
// is on. For a remote session the recorded pid is the ssh
// client's, which is the process this machine owns and which holds
// the tunnel open for exactly as long as the far server lives.
let record = process::Record::of(pid, process::Detail::Http { port: forward.here })
.context("llama-server was gone before its start time could be read")?;
process::write(session_dir, &record);
Ok(Self::attached(
format!("http://127.0.0.1:{port}"),
format!("http://127.0.0.1:{}", forward.here),
meta,
model,
transcript,
@@ -212,7 +237,7 @@ impl LlamaDriver {
let endpoint = endpoint.clone();
let model = model.to_string();
let session_dir = session_dir.to_path_buf();
std::thread::spawn(move || match wait_until_ready(&endpoint) {
std::thread::spawn(move || match wait_until_ready(&endpoint, &session_dir) {
Ok(()) => {
tracing::info!("{model} loaded and answering at {endpoint}");
let _ = sink.send(Event::Status {
@@ -518,19 +543,76 @@ fn model_path(models_dir: &Path, key: &str) -> Result<PathBuf> {
Ok(path)
}
/// An unused loopback port, by asking the OS for one and letting it go.
/// The model file's path **on the machine that will serve it**, confirmed
/// to be there.
///
/// Racy in principle: something else could take it between here and
/// llama-server binding. In practice nothing on this machine is hunting
/// for ports, and the alternative -- parsing the port back out of the
/// server's log -- couples us to its output format for no real gain.
fn free_port() -> Result<u16> {
let listener = std::net::TcpListener::bind("127.0.0.1:0")?;
Ok(listener.local_addr()?.port())
/// Local and remote answer the same question and it has to be asked of
/// two different filesystems, which is why this is one function rather
/// than a check beside the local path and hope for the other case. The
/// remote answer is measured for the same reason the local one is: a
/// missing file otherwise becomes a `llama-server` that starts, fails to
/// load, and reports as a session that never became ready -- which reads
/// as the machine being slow.
///
/// One blocking round trip on a remote spawn, which is the same cost the
/// spawn is already paying to start ssh. The alternative is a path built
/// here from a `~` this machine cannot expand.
fn model_on(transport: &Transport, models_dir: &Path, key: &str) -> Result<String> {
let Transport::Ssh { name, .. } = transport else {
return Ok(model_path(models_dir, key)?.to_string_lossy().into_owned());
};
// The same directory the spawn screen listed for this machine, and
// for the same reason it is one function: a list from one place and a
// load from another is a model that appears and then fails.
let dir = crate::models::dir_on(transport, models_dir);
// Checked here rather than in the script: `..` in a key would walk
// out of the models directory on a machine this server can start
// processes on, and the phone is where the key comes from.
for part in key.split('/') {
if part.is_empty() || part == "." || part == ".." {
bail!("\"{key}\" is not a model key this can resolve");
}
}
let path = format!("{}/{key}", dir.trim_end_matches('/'));
// `$HOME` on the far side, which is the only machine that knows what
// it is -- and the resolved path is printed back so the launch below
// hands `llama-server` something absolute.
//
// "the file is not there" is answered rather than failed, because the
// two are different things to a reader and only one of them is a
// fault: a machine that could not be asked at all has to say so in
// its own words, and it would otherwise arrive as this same sentence
// about a missing model.
let script = "p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${p#\"~/\"};; esac; \
[ -f \"$p\" ] && printf 'at\\t%s\\n' \"$p\" || printf 'missing\\n'"
.to_string();
let launch = Launch::new(
"sh",
vec!["-c".to_string(), script, "sh".to_string(), path.clone()],
None,
);
let answer = transport
.capture_blocking(&launch)
.with_context(|| format!("couldn't ask {name} where its models are"))?;
match answer.trim().split_once('\t') {
Some(("at", resolved)) => Ok(resolved.to_string()),
_ => bail!(
"{name} has no model at {path}. A llama.cpp session serves the file from the \
machine it runs on, so the model has to be on {name} -- what this backend has \
downloaded is somewhere else."
),
}
}
/// Polls until the server says it is ready, or gives up.
fn wait_until_ready(endpoint: &str) -> Result<()> {
///
/// Watches the process as well as the port, because the two failures need
/// different words and one of them is common: a model that will not load,
/// a port already taken on the far machine, a `llama-server` too old for
/// a flag. All of those exit within a second and none of them will ever
/// answer `/health`, so waiting out the timeout turns a server that said
/// exactly what was wrong into "gave up after 300s".
fn wait_until_ready(endpoint: &str, session_dir: &Path) -> Result<()> {
let deadline = std::time::Instant::now() + READY_TIMEOUT;
let url = format!("{endpoint}/health");
loop {
@@ -539,13 +621,48 @@ fn wait_until_ready(endpoint: &str) -> Result<()> {
{
return Ok(());
}
// `None` is the session having been stopped or deleted while this
// waited, which is nobody's fault and still not worth waiting on.
match process::recorded(session_dir) {
Some((_, process::Liveness::Alive | process::Liveness::Unknown)) => {}
Some((_, process::Liveness::Dead)) | None => {
bail!("it exited before it answered.{}", log_tail(session_dir));
}
}
if std::time::Instant::now() > deadline {
bail!("gave up after {}s", READY_TIMEOUT.as_secs());
bail!(
"gave up after {}s.{}",
READY_TIMEOUT.as_secs(),
log_tail(session_dir)
);
}
std::thread::sleep(std::time::Duration::from_millis(250));
}
}
/// The end of `llama-server`'s own log, for a failure message.
///
/// Its account of what went wrong is the useful half -- "failed to load
/// model", "bind: Address already in use" -- and on a remote session it
/// is the only half, since nobody reading the phone can open a file on
/// that machine. Bounded, because this ends up in an event a phone draws.
fn log_tail(session_dir: &Path) -> String {
let Ok(text) = std::fs::read_to_string(session_dir.join(SERVER_LOG)) else {
return String::new();
};
let tail: Vec<&str> = text.lines().rev().take(LOG_TAIL_LINES).collect();
if tail.is_empty() {
return String::new();
}
format!(
" It last said: {}",
tail.into_iter().rev().collect::<Vec<_>>().join(" / ")
)
}
/// How much of that log to carry into a message somebody reads on a phone.
const LOG_TAIL_LINES: usize = 6;
/// One streamed completion: posts the conversation, emits each delta as it
/// arrives. Emits rather than returns: the transcript those events land
/// in is what the next turn reads back, so there is nothing to hand up.