Meter a session by its provider, and let llama.cpp run over ssh

The rate-limit bar answered a question about an account, and picked the
answer by machine. One machine runs echo, the Claude CLI and a local
model side by side, so every echo session on it drew the CLI's five-hour
window: a quota that session cannot spend and could never run down. A
session now names its meter (`usageProvider`, from
`DriverKind::usage_provider`, which `usage::providers_for` reads too so
the two lists cannot disagree), and the phone matches on machine *and*
provider. Nothing meters echo or llama, and nothing at all is drawn --
including while the first fetch is out, since "checking" under a session
that turns out to meter nothing is a row the screen then withdraws.

Echo gets a meter it can be *told* about instead: `/usage 42`,
`/usage 95 20`, `/usage 42 never`, `/usage notloggedin`,
`/usage unreachable`, `/usage failed`, `/usage off`. Those states cost
real quota to arrange, which is why none of them had been looked at.

And llama.cpp runs wherever a setup says, which was the last of phase 5.
`Transport::reserve_port` is the second half of what a transport is --
"run this" plus "reach this port" -- returning the port the server binds
there and the port that reaches it here, and `Launch::reaching` puts the
`-L` tunnel on the connection that already carries the command. Three
things that came out of building it:

- A forwarded launch gets a pty and every other one keeps `-T`. Killing
  the ssh client ends a CLI by closing the stdin it reads; llama-server
  never reads its stdin, so the same kill left it running on the far
  machine with the model loaded -- one orphan per stopped session.
- The model is looked for on the machine that will serve it, at that
  machine's own models directory, so `GET /setups/{id}/models` is what
  the spawn screen offers rather than the backend's own downloads.
- The readiness poll watches the process, not only the port: a model
  that will not load exits in a second and would otherwise have been
  reported as "gave up after 300s". The failure carries the log's tail.

Exercised end to end against this VM over ssh to itself: spawn, load,
answer, outlive a backend restart, be adopted, answer again, and stop --
with both the ssh client and the far llama-server gone afterwards. The
local path, the Claude bar and the spawn screen checked on the emulator.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed 2026-09-04 17:45:32 -04:00
1 parent 74110b4d72
commit 127b25e60a
20 files changed
+1212 -143

No files matched your search

@@ -70,9 +70,10 @@ fun SpawnScreen(
// one leaves a filled-in form worth keeping, and that one leaves
// nothing to fill in.
var spawnError by remember { mutableStateOf<String?>(null) }
// Downloaded models, for a llama provider to choose between. Fetched
// beside the setups but kept separate: a Claude session needs none, so
// failing to list them must not stop the screen rendering.
// The models on the *chosen machine*, for a llama provider to choose between. Kept separate
// from the setups: a Claude session needs none, so failing to list them must not stop the
// screen rendering. Refetched when the machine changes, because a model is a file on one
// machine -- see [fetchSetupModels].
var models by remember { mutableStateOf<List<LocalModel>>(emptyList()) }
var modelKey by remember { mutableStateOf<String?>(null) }
var contextSize by remember { mutableStateOf("") }
@@ -89,9 +90,6 @@ fun SpawnScreen(
} catch (e: ApiException) {
LoadState.failed(e)
}
models =
runCatching { withContext(Dispatchers.IO) { fetchModels(settings).local } }
.getOrDefault(emptyList())
}
Column(Modifier.fillMaxSize().verticalScroll(rememberScrollState()).padding(16.dp)) {
@@ -122,6 +120,17 @@ fun SpawnScreen(
is LoadState.Loaded -> state.value
}
val setup = setups.firstOrNull { it.name == setupName }
// Whichever machine is chosen now, asked again when that changes. The old machine's list
// is dropped first rather than left on screen: a file name from another machine looks
// exactly like one from this one.
LaunchedEffect(setup?.id) {
models = emptyList()
modelKey = null
val id = setup?.id ?: return@LaunchedEffect
models =
runCatching { withContext(Dispatchers.IO) { fetchSetupModels(settings, id) } }
.getOrDefault(emptyList())
}
val current = setup?.providers?.firstOrNull { it.name == providerName }
// Only the Claude CLI has models, a working directory and
// permission modes; keying the extra fields on the kind rather
@@ -183,13 +192,14 @@ fun SpawnScreen(
)
if (isLlama) {
// A llama session names one of the models this backend has
// downloaded, so the choice is that list rather than free
// text -- there is nothing sensible to type here, and a name
// that is not on disk is a session that cannot start.
// A llama session names one of the models on the machine it will run on, so the
// choice is that list rather than free text -- there is nothing sensible to type
// here, and a name that is not on that machine's disk is a session that cannot
// start.
if (models.isEmpty()) {
Text(
"No models downloaded yet. Get one from the Models screen first.",
"No models on ${setup?.name ?: "this machine"}. The Models screen downloads " +
"to the backend; another machine needs the file put there itself.",
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)