Meter a session by its provider, and let llama.cpp run over ssh
The rate-limit bar answered a question about an account, and picked the
answer by machine. One machine runs echo, the Claude CLI and a local
model side by side, so every echo session on it drew the CLI's five-hour
window: a quota that session cannot spend and could never run down. A
session now names its meter (`usageProvider`, from
`DriverKind::usage_provider`, which `usage::providers_for` reads too so
the two lists cannot disagree), and the phone matches on machine *and*
provider. Nothing meters echo or llama, and nothing at all is drawn --
including while the first fetch is out, since "checking" under a session
that turns out to meter nothing is a row the screen then withdraws.
Echo gets a meter it can be *told* about instead: `/usage 42`,
`/usage 95 20`, `/usage 42 never`, `/usage notloggedin`,
`/usage unreachable`, `/usage failed`, `/usage off`. Those states cost
real quota to arrange, which is why none of them had been looked at.
And llama.cpp runs wherever a setup says, which was the last of phase 5.
`Transport::reserve_port` is the second half of what a transport is --
"run this" plus "reach this port" -- returning the port the server binds
there and the port that reaches it here, and `Launch::reaching` puts the
`-L` tunnel on the connection that already carries the command. Three
things that came out of building it:
- A forwarded launch gets a pty and every other one keeps `-T`. Killing
the ssh client ends a CLI by closing the stdin it reads; llama-server
never reads its stdin, so the same kill left it running on the far
machine with the model loaded -- one orphan per stopped session.
- The model is looked for on the machine that will serve it, at that
machine's own models directory, so `GET /setups/{id}/models` is what
the spawn screen offers rather than the backend's own downloads.
- The readiness poll watches the process, not only the port: a model
that will not load exits in a second and would otherwise have been
reported as "gave up after 300s". The failure carries the log's tail.
Exercised end to end against this VM over ssh to itself: spawn, load,
answer, outlive a backend restart, be adopted, answer again, and stop --
with both the ssh client and the far llama-server gone afterwards. The
local path, the Claude bar and the spawn screen checked on the emulator.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
74110b4d72
commit
127b25e60a
20 files changed
+1212
-143
No files matched your search
@@ -35,6 +35,8 @@ use serde::Serialize;
|
||||
|
||||
use wg_app_link::private;
|
||||
|
||||
use crate::session::transport::{Launch, Transport};
|
||||
|
||||
/// Identifies this client to HuggingFace. They ask for one, and a request
|
||||
/// without it is more likely to be rate-limited.
|
||||
const USER_AGENT: &str = concat!("ai-server/", env!("CARGO_PKG_VERSION"));
|
||||
@@ -551,6 +553,83 @@ fn collect(root: &Path, dir: &Path, found: &mut Vec<LocalModel>) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Where a machine reached over ssh keeps its models, when its setup does
|
||||
/// not say.
|
||||
///
|
||||
/// The same place this backend puts its own downloads, written out rather
|
||||
/// than derived: `$XDG_DATA_HOME` here describes *this* machine's
|
||||
/// environment, and the far machine's is the far machine's business. A
|
||||
/// setup whose models are elsewhere says so (`SshConfig::models_dir`).
|
||||
const FAR_MODELS_DIR: &str = "~/.local/share/ai-app/models";
|
||||
|
||||
/// Which directory holds the models on the machine `transport` reaches.
|
||||
///
|
||||
/// One answer, because two things ask: the list a spawn screen offers,
|
||||
/// and the path a session hands `llama-server`. A machine that listed one
|
||||
/// directory and served from another would offer models that then failed
|
||||
/// to load, which reads as the model being broken.
|
||||
pub fn dir_on(transport: &Transport, local: &Path) -> String {
|
||||
match transport {
|
||||
Transport::Here => local.to_string_lossy().into_owned(),
|
||||
Transport::Ssh { ssh, .. } => ssh
|
||||
.models_dir
|
||||
.as_ref()
|
||||
.map_or(FAR_MODELS_DIR.to_string(), |dir| {
|
||||
dir.to_string_lossy().into_owned()
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/// Every GGUF on the machine a setup names, which is the machine that
|
||||
/// would have to serve it.
|
||||
///
|
||||
/// The local half of this is [`ModelStore::list`], reading the same shape
|
||||
/// off this machine's disk; a caller picks by transport, since a setup
|
||||
/// with no ssh *is* this machine and asking a shell about it would be a
|
||||
/// slower way to the same answer. What must not happen is offering this
|
||||
/// backend's downloads for a session on another machine: the file has to
|
||||
/// be where `llama-server` runs, and a list that says otherwise is a
|
||||
/// claim about the wrong filesystem.
|
||||
///
|
||||
/// `dir` is that machine's models directory, `~` included -- expanded on
|
||||
/// the far side, which is the only place that knows what it is. A
|
||||
/// directory that is not there is an empty list rather than a failure: a
|
||||
/// machine that has never had a model put on it is an ordinary state, and
|
||||
/// the same one as a machine whose directory exists and is empty.
|
||||
pub async fn on_machine(transport: &Transport, dir: &str) -> Result<Vec<LocalModel>> {
|
||||
let script = "p=$1; case $p in \"~\") p=$HOME;; \"~/\"*) p=$HOME/${p#\"~/\"};; esac; \
|
||||
[ -d \"$p\" ] || exit 0; \
|
||||
find \"$p\" -type f -name '*.gguf' -printf '%s\\t%P\\0'";
|
||||
let launch = Launch::new(
|
||||
"sh",
|
||||
vec![
|
||||
"-c".to_string(),
|
||||
script.to_string(),
|
||||
"sh".to_string(),
|
||||
dir.to_string(),
|
||||
],
|
||||
None,
|
||||
);
|
||||
let out = transport.capture(&launch).await?;
|
||||
let mut found: Vec<LocalModel> = out
|
||||
.split('\0')
|
||||
.filter(|record| !record.is_empty())
|
||||
// Two fields, and the name last, so a `\t` in a filename survives.
|
||||
.filter_map(|record| record.split_once('\t'))
|
||||
.filter_map(|(bytes, key)| {
|
||||
let (repo, file) = key.rsplit_once('/')?;
|
||||
Some(LocalModel {
|
||||
key: key.to_string(),
|
||||
repo: repo.to_string(),
|
||||
file: file.to_string(),
|
||||
bytes: bytes.trim().parse().unwrap_or(0),
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
found.sort_by(|a, b| a.key.cmp(&b.key));
|
||||
Ok(found)
|
||||
}
|
||||
|
||||
/// A model repository on HuggingFace, as the browse screen shows it.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
|
||||
Reference in new issue
Block a user