Serve a machine's models from one shared llama-server
A llama.cpp session had its own `llama-server`: two sessions on one model held two copies of it in memory, a model change bought a load only that session benefited from, and the process was a session's to end. A machine's models are now served by one `llama-server` in **router mode** -- no `-m`, a preset file naming models and their flags, a child server per model asked for, and each request routed by its `model` field. So one server per model with that model's own settings is what a machine runs, while this backend has one process, one port and one record per machine to keep track of. The record is the mechanism every other driver already uses, so a restart adopts it; a session records the same pid in its own directory as `Detail::Shared`, and `process::signal` refuses to signal one of those -- which is what keeps stopping, deleting or cleaning up after one session from unloading a model every other session is using. Nothing stops a router on its own. That is deliberate (a loaded model is minutes of disk) and it is why the machines tab now has a card per provider that opens its own screen: how each model is loaded, how many stay in memory, Unload, and Stop. How a model is *loaded* therefore belongs to the model on its machine rather than to a session -- context size, GPU layers, threads, slots, speculative decoding -- written into the preset as llama-server's own argument names. Saving them re-reads that file, which unloads the model; that is the change taking effect, and the dialog says so before you save. What stays a session's is everything that rides on a request, including which tools it offers: the router hosts one set for the machine and the choice is a filter applied here, so it costs no reload (2,181 tokens of prompt with all seven, 698 with none). Verified end to end against the scratch backend and the emulator: two sessions sharing one loaded model with one child process, a second session joining it with a 26ms prefill, a backend restart adopting the router and answering with the prompt cache intact, the same over ssh to this VM, a model's settings reaching the running server, Unload, and Stop leaving every session `exited` with no error line.
This commit is contained in:
1 parent
74cda485e5
commit
8c323fc7a9
19 files changed
+2601
-511
No files matched your search
@@ -56,6 +56,16 @@ private sealed class Screen {
|
||||
|
||||
data object Spawn : Screen()
|
||||
|
||||
/**
|
||||
* One provider on one machine: its settings, and what its shared server is holding.
|
||||
*
|
||||
* A step down from the machines tab rather than a tab of its own, because it is about one
|
||||
* machine rather than about the backend. Addressed by ids and names rather than by the
|
||||
* [Provider] it was tapped from: what it shows is fetched, and a stale copy of a card would be
|
||||
* a second version of the same truth.
|
||||
*/
|
||||
data class ProviderSettings(val machineId: String, val provider: String) : Screen()
|
||||
|
||||
data object Settings : Screen()
|
||||
}
|
||||
|
||||
@@ -199,6 +209,9 @@ fun AppRoot(
|
||||
screen = Screen.Session(imported)
|
||||
},
|
||||
onSettings = { screen = Screen.Settings },
|
||||
onProvider = { machineId, provider ->
|
||||
screen = Screen.ProviderSettings(machineId, provider)
|
||||
},
|
||||
)
|
||||
}
|
||||
is Screen.Session ->
|
||||
@@ -268,6 +281,15 @@ fun AppRoot(
|
||||
}
|
||||
}
|
||||
}
|
||||
is Screen.ProviderSettings ->
|
||||
Box(Modifier.imePadding()) {
|
||||
ProviderScreen(
|
||||
settings = current,
|
||||
machineId = here.machineId,
|
||||
provider = here.provider,
|
||||
onBack = goToMain,
|
||||
)
|
||||
}
|
||||
is Screen.Spawn ->
|
||||
Box(Modifier.imePadding()) {
|
||||
SpawnScreen(
|
||||
|
||||
Reference in new issue
Block a user