Meter a session by its provider, and let llama.cpp run over ssh
The rate-limit bar answered a question about an account, and picked the
answer by machine. One machine runs echo, the Claude CLI and a local
model side by side, so every echo session on it drew the CLI's five-hour
window: a quota that session cannot spend and could never run down. A
session now names its meter (`usageProvider`, from
`DriverKind::usage_provider`, which `usage::providers_for` reads too so
the two lists cannot disagree), and the phone matches on machine *and*
provider. Nothing meters echo or llama, and nothing at all is drawn --
including while the first fetch is out, since "checking" under a session
that turns out to meter nothing is a row the screen then withdraws.
Echo gets a meter it can be *told* about instead: `/usage 42`,
`/usage 95 20`, `/usage 42 never`, `/usage notloggedin`,
`/usage unreachable`, `/usage failed`, `/usage off`. Those states cost
real quota to arrange, which is why none of them had been looked at.
And llama.cpp runs wherever a setup says, which was the last of phase 5.
`Transport::reserve_port` is the second half of what a transport is --
"run this" plus "reach this port" -- returning the port the server binds
there and the port that reaches it here, and `Launch::reaching` puts the
`-L` tunnel on the connection that already carries the command. Three
things that came out of building it:
- A forwarded launch gets a pty and every other one keeps `-T`. Killing
the ssh client ends a CLI by closing the stdin it reads; llama-server
never reads its stdin, so the same kill left it running on the far
machine with the model loaded -- one orphan per stopped session.
- The model is looked for on the machine that will serve it, at that
machine's own models directory, so `GET /setups/{id}/models` is what
the spawn screen offers rather than the backend's own downloads.
- The readiness poll watches the process, not only the port: a model
that will not load exits in a second and would otherwise have been
reported as "gave up after 300s". The failure carries the log's tail.
Exercised end to end against this VM over ssh to itself: spawn, load,
answer, outlive a backend restart, be adopted, answer again, and stop --
with both the ssh client and the far llama-server gone afterwards. The
local path, the Claude bar and the spawn screen checked on the emulator.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
74110b4d72
commit
127b25e60a
20 files changed
+1212
-143
No files matched your search
@@ -106,6 +106,19 @@ pub struct SshConfig {
|
||||
/// Extra `-o` settings, each written as `Key=value`.
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub options: Vec<String>,
|
||||
/// Where this machine keeps the GGUF models it can serve, absent for
|
||||
/// the same default this backend uses (`~/.local/share/ai-app/models`
|
||||
/// -- `$XDG_DATA_HOME` is not read on the far side, since it is this
|
||||
/// machine's environment that would answer). A `~` prefix is the
|
||||
/// remote home.
|
||||
///
|
||||
/// Here rather than on the provider because it is a fact about the
|
||||
/// machine, and because a machine reached over ssh is where the model
|
||||
/// has to be: a llama.cpp session serves the file from the machine
|
||||
/// that runs `llama-server`, and this backend's own downloads are on
|
||||
/// whichever machine that is only when they are the same one.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub models_dir: Option<PathBuf>,
|
||||
/// Where a file attached from the phone is put on this machine so the
|
||||
/// session can read it. Absent means the session's own working
|
||||
/// directory, or the login home for a session that has none. A `~`
|
||||
@@ -167,6 +180,38 @@ impl DriverKind {
|
||||
}
|
||||
}
|
||||
|
||||
/// Which paid service meters a session of this kind, and `None` for
|
||||
/// one that costs nothing.
|
||||
///
|
||||
/// The rate-limit bars answer a question about an *account*, and what
|
||||
/// decides which account -- if any -- is the provider a session runs,
|
||||
/// not the machine it runs on. Those were the same thing only for as
|
||||
/// long as a machine ran one kind of session: an echo session on a
|
||||
/// laptop that also has the Claude CLI was drawn with that CLI's
|
||||
/// five-hour window under its header, reporting a quota it cannot
|
||||
/// spend and could not run down. A llama.cpp session is the same
|
||||
/// story with the model on the far side.
|
||||
///
|
||||
/// [`DriverKind::Echo`] names a meter of its own, which exists only
|
||||
/// when a test has asked for one (`/usage` in `session::echo`). That
|
||||
/// is what makes the bar's states -- a number, a machine nobody
|
||||
/// logged into, one that could not be reached -- reachable without an
|
||||
/// account and without spending a turn on somebody else's. With no
|
||||
/// fixture set there is no snapshot for it, which the phone draws as
|
||||
/// nothing at all.
|
||||
///
|
||||
/// The string is a [`crate::usage::UsageProvider::name`], and it is
|
||||
/// what pairs a session with one of the snapshots `GET /usage`
|
||||
/// returns; the two lists have to agree, so `usage::providers_for`
|
||||
/// reads this rather than matching on kinds a second time.
|
||||
pub fn usage_provider(self) -> Option<&'static str> {
|
||||
match self {
|
||||
Self::ClaudeCli => Some(crate::usage::CLAUDE),
|
||||
Self::Echo => Some(crate::usage::ECHO),
|
||||
Self::LlamaCpp => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the conversation exists outside this app, so that deleting
|
||||
/// the session here does not end it.
|
||||
///
|
||||
@@ -442,6 +487,7 @@ mod tests {
|
||||
port: Some(2222),
|
||||
identity_file: None,
|
||||
options: Vec::new(),
|
||||
models_dir: None,
|
||||
attachments_dir: None,
|
||||
}),
|
||||
providers: vec![ProviderConfig {
|
||||
|
||||
Reference in new issue
Block a user