Serve a machine's models from one shared llama-server
A llama.cpp session had its own `llama-server`: two sessions on one model held two copies of it in memory, a model change bought a load only that session benefited from, and the process was a session's to end. A machine's models are now served by one `llama-server` in **router mode** -- no `-m`, a preset file naming models and their flags, a child server per model asked for, and each request routed by its `model` field. So one server per model with that model's own settings is what a machine runs, while this backend has one process, one port and one record per machine to keep track of. The record is the mechanism every other driver already uses, so a restart adopts it; a session records the same pid in its own directory as `Detail::Shared`, and `process::signal` refuses to signal one of those -- which is what keeps stopping, deleting or cleaning up after one session from unloading a model every other session is using. Nothing stops a router on its own. That is deliberate (a loaded model is minutes of disk) and it is why the machines tab now has a card per provider that opens its own screen: how each model is loaded, how many stay in memory, Unload, and Stop. How a model is *loaded* therefore belongs to the model on its machine rather than to a session -- context size, GPU layers, threads, slots, speculative decoding -- written into the preset as llama-server's own argument names. Saving them re-reads that file, which unloads the model; that is the change taking effect, and the dialog says so before you save. What stays a session's is everything that rides on a request, including which tools it offers: the router hosts one set for the machine and the choice is a filter applied here, so it costs no reload (2,181 tokens of prompt with all seven, 698 with none). Verified end to end against the scratch backend and the emulator: two sessions sharing one loaded model with one child process, a second session joining it with a 26ms prefill, a backend restart adopting the router and answering with the prompt cache intact, the same over ssh to this VM, a model's settings reaching the running server, Unload, and Stop leaving every session `exited` with no error line.
This commit is contained in:
iris-ai committed
1 parent
74cda485e5
commit
8c323fc7a9
19 files changed
+2601
-511
No files matched your search
@@ -0,0 +1,405 @@
|
||||
package com.example.aiapp
|
||||
|
||||
import androidx.compose.foundation.clickable
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.Spacer
|
||||
import androidx.compose.foundation.layout.fillMaxSize
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.height
|
||||
import androidx.compose.foundation.layout.imePadding
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.lazy.LazyColumn
|
||||
import androidx.compose.foundation.rememberScrollState
|
||||
import androidx.compose.foundation.text.KeyboardOptions
|
||||
import androidx.compose.foundation.verticalScroll
|
||||
import androidx.compose.material3.AlertDialog
|
||||
import androidx.compose.material3.Card
|
||||
import androidx.compose.material3.CircularProgressIndicator
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.OutlinedTextField
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.material3.TextButton
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableIntStateOf
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.rememberCoroutineScope
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.text.font.FontFamily
|
||||
import androidx.compose.ui.text.input.KeyboardType
|
||||
import androidx.compose.ui.unit.dp
|
||||
import androidx.compose.ui.window.DialogProperties
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.withContext
|
||||
|
||||
/**
|
||||
* One provider on one machine: what it is, what its shared server is holding, and how each of its
|
||||
* models is loaded.
|
||||
*
|
||||
* This is where a setting that belongs to a *machine* lives, as opposed to one that belongs to a
|
||||
* session. The two were one list until llama.cpp sessions came to share one server per machine: how
|
||||
* a model is loaded stopped being anything a single session could decide, because one copy of it in
|
||||
* memory is what several sessions are talking to.
|
||||
*
|
||||
* It is also the only place a loaded model is taken out of memory. Nothing does that on its own —
|
||||
* closing a session leaves the model loaded on purpose, since the next one to want it would
|
||||
* otherwise pay the load again — so the memory is freed here, where what it costs everybody is
|
||||
* visible.
|
||||
*/
|
||||
@Composable
|
||||
fun ProviderScreen(
|
||||
settings: ServerSettings,
|
||||
machineId: String,
|
||||
provider: String,
|
||||
onBack: () -> Unit,
|
||||
) {
|
||||
val scope = rememberCoroutineScope()
|
||||
var state by remember { mutableStateOf<LoadState<ProviderView>>(LoadState.Loading) }
|
||||
var reload by remember { mutableIntStateOf(0) }
|
||||
var editing by remember { mutableStateOf<ProviderModel?>(null) }
|
||||
var confirmingStop by remember { mutableStateOf(false) }
|
||||
// What is being done to the server or to one of its models, in a word, and what went wrong
|
||||
// when it did. Both here rather than per row: these act on the whole machine.
|
||||
var busy by remember { mutableStateOf<String?>(null) }
|
||||
var actionError by remember { mutableStateOf<String?>(null) }
|
||||
|
||||
LaunchedEffect(reload) {
|
||||
state =
|
||||
try {
|
||||
withContext(Dispatchers.IO) {
|
||||
LoadState.Loaded(fetchProvider(settings, machineId, provider))
|
||||
}
|
||||
} catch (e: ApiException) {
|
||||
LoadState.failed(e)
|
||||
}
|
||||
}
|
||||
|
||||
// Say what is happening, do it, say what went wrong, refetch: every action on this screen
|
||||
// changes what it is showing.
|
||||
val act = { what: String, action: suspend () -> Unit ->
|
||||
scope.launch {
|
||||
busy = what
|
||||
actionError =
|
||||
runCatching { withContext(Dispatchers.IO) { action() } }.exceptionOrNull()?.message
|
||||
busy = null
|
||||
reload++
|
||||
}
|
||||
Unit
|
||||
}
|
||||
|
||||
Column(Modifier.fillMaxSize().padding(16.dp)) {
|
||||
Row(verticalAlignment = Alignment.CenterVertically, modifier = Modifier.fillMaxWidth()) {
|
||||
TextButton(onClick = onBack) { Text("Back") }
|
||||
}
|
||||
when (val current = state) {
|
||||
is LoadState.Loading -> CircularProgressIndicator()
|
||||
is LoadState.Error -> Text(current.message, color = MaterialTheme.colorScheme.error)
|
||||
is LoadState.Loaded -> {
|
||||
val view = current.value
|
||||
Text(view.name, style = MaterialTheme.typography.titleMedium)
|
||||
Text(
|
||||
"on ${view.machine}",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
view.command?.let {
|
||||
Text(
|
||||
it,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
fontFamily = FontFamily.Monospace,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
Spacer(Modifier.height(12.dp))
|
||||
actionError?.let {
|
||||
Text(it, color = MaterialTheme.colorScheme.error)
|
||||
Spacer(Modifier.height(8.dp))
|
||||
}
|
||||
busy?.let {
|
||||
Row(verticalAlignment = Alignment.CenterVertically) {
|
||||
CircularProgressIndicator(Modifier.height(16.dp).padding(end = 8.dp))
|
||||
Text(it, style = MaterialTheme.typography.bodySmall)
|
||||
}
|
||||
Spacer(Modifier.height(8.dp))
|
||||
}
|
||||
|
||||
LazyColumn(Modifier.fillMaxSize()) {
|
||||
view.server?.let { server ->
|
||||
item("server") {
|
||||
ServerCard(
|
||||
server = server,
|
||||
maxLoaded = view.maxLoaded,
|
||||
enabled = busy == null,
|
||||
onStop = { confirmingStop = true },
|
||||
onMaxLoaded = { chosen ->
|
||||
act("Saving…") {
|
||||
setProviderSettings(
|
||||
settings,
|
||||
machineId,
|
||||
provider,
|
||||
chosen,
|
||||
)
|
||||
}
|
||||
},
|
||||
)
|
||||
Spacer(Modifier.height(12.dp))
|
||||
}
|
||||
}
|
||||
if (view.models.isNotEmpty() && view.modelParams.isNotEmpty()) {
|
||||
item("models-heading") {
|
||||
Text("Models", style = MaterialTheme.typography.titleSmall)
|
||||
Text(
|
||||
"How a model is loaded belongs to the machine, not to a session: " +
|
||||
"one copy of it in memory answers every session using it.",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
Spacer(Modifier.height(8.dp))
|
||||
}
|
||||
}
|
||||
uniqueItems(view.models, key = { it.id }) { model ->
|
||||
ModelCard(
|
||||
model = model,
|
||||
specs = view.modelParams,
|
||||
// Tapping opens the settings; a provider whose models take none has
|
||||
// nothing to open, so the row is not a control.
|
||||
onEdit =
|
||||
if (view.modelParams.isEmpty()) null else ({ editing = model }),
|
||||
onUnload =
|
||||
if (model.status == "loaded" || model.status == "sleeping") {
|
||||
{
|
||||
act("Unloading ${model.label}…") {
|
||||
unloadProviderModel(
|
||||
settings,
|
||||
machineId,
|
||||
provider,
|
||||
model.id,
|
||||
)
|
||||
}
|
||||
}
|
||||
} else null,
|
||||
enabled = busy == null,
|
||||
)
|
||||
}
|
||||
if (view.mcpServers.isNotEmpty()) {
|
||||
item("mcp") {
|
||||
Spacer(Modifier.height(12.dp))
|
||||
Text("Tool servers", style = MaterialTheme.typography.titleSmall)
|
||||
Text(
|
||||
view.mcpServers.joinToString(", ") +
|
||||
" — configured on the backend, in its config file.",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
editing?.let { model ->
|
||||
val view = (state as? LoadState.Loaded)?.value
|
||||
ModelSettingsDialog(
|
||||
model = model,
|
||||
specs = view?.modelParams.orEmpty(),
|
||||
onDismiss = { editing = null },
|
||||
onSave = { params ->
|
||||
editing = null
|
||||
act("Saving ${model.label}…") {
|
||||
setModelSettings(settings, machineId, provider, model.id, params)
|
||||
}
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
if (confirmingStop) {
|
||||
AlertDialog(
|
||||
onDismissRequest = { confirmingStop = false },
|
||||
title = { Text("Stop this server?") },
|
||||
text = {
|
||||
// Said plainly rather than hidden: this is the only thing that frees the memory,
|
||||
// and what it costs is that every session on this machine reloads its model.
|
||||
Text(
|
||||
"Every model it is holding is unloaded. Sessions using it will show as " +
|
||||
"exited, and the next message to one loads its model again — which is " +
|
||||
"the slow part, not the sending."
|
||||
)
|
||||
},
|
||||
confirmButton = {
|
||||
TextButton(
|
||||
onClick = {
|
||||
confirmingStop = false
|
||||
act("Stopping…") { stopProviderServer(settings, machineId, provider) }
|
||||
}
|
||||
) {
|
||||
Text("Stop")
|
||||
}
|
||||
},
|
||||
dismissButton = { TextButton(onClick = { confirmingStop = false }) { Text("Cancel") } },
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ServerCard(
|
||||
server: ServerState,
|
||||
maxLoaded: Int?,
|
||||
enabled: Boolean,
|
||||
onStop: () -> Unit,
|
||||
onMaxLoaded: (Int?) -> Unit,
|
||||
) {
|
||||
// The saved value is what this starts at and what Save is compared against, so a field left
|
||||
// half-typed is visibly not saved rather than quietly either way.
|
||||
val saved = maxLoaded?.toString().orEmpty()
|
||||
var typed by remember(saved) { mutableStateOf(saved) }
|
||||
Card(Modifier.fillMaxWidth()) {
|
||||
Column(Modifier.padding(12.dp)) {
|
||||
Text("Model server", style = MaterialTheme.typography.titleSmall)
|
||||
Text(
|
||||
if (server.running) {
|
||||
"Running" + (server.port?.let { ", reached on port $it" } ?: "")
|
||||
} else {
|
||||
// Not a fault: nothing is loaded because nothing has asked. Saying it in
|
||||
// words rather than colouring the row, since "stopped" and "we could not
|
||||
// ask" would otherwise look the same.
|
||||
"Not running. A session starts it when it needs a model."
|
||||
},
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
Spacer(Modifier.height(8.dp))
|
||||
OutlinedTextField(
|
||||
value = typed,
|
||||
onValueChange = { typed = it.filter(Char::isDigit) },
|
||||
label = { Text("Models loaded at once") },
|
||||
placeholder = { Text("one -- a second model replaces the first") },
|
||||
singleLine = true,
|
||||
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Number),
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
)
|
||||
Row(verticalAlignment = Alignment.CenterVertically) {
|
||||
// Shown whether or not it is running, and disabled when there is nothing to stop:
|
||||
// a button that comes and goes makes its own absence the message.
|
||||
TextButton(enabled = enabled && server.running, onClick = onStop) { Text("Stop") }
|
||||
Spacer(Modifier.weight(1f))
|
||||
TextButton(
|
||||
enabled = enabled && typed != saved,
|
||||
onClick = { onMaxLoaded(typed.toIntOrNull()) },
|
||||
) {
|
||||
Text("Save")
|
||||
}
|
||||
}
|
||||
if (typed != saved) {
|
||||
Text(
|
||||
"Read when this server next starts.",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ModelCard(
|
||||
model: ProviderModel,
|
||||
specs: List<ParamSpec>,
|
||||
onEdit: (() -> Unit)?,
|
||||
onUnload: (() -> Unit)?,
|
||||
enabled: Boolean,
|
||||
) {
|
||||
Card(
|
||||
Modifier.fillMaxWidth()
|
||||
.padding(vertical = 4.dp)
|
||||
.then(if (onEdit != null && enabled) Modifier.clickable(onClick = onEdit) else Modifier)
|
||||
) {
|
||||
Column(Modifier.padding(12.dp)) {
|
||||
Text(model.label, style = MaterialTheme.typography.bodyMedium)
|
||||
// What the server is doing with it, in its own word. Absent means nobody could ask --
|
||||
// the server is not running -- and the line is left out rather than guessed at.
|
||||
model.status?.let {
|
||||
Text(
|
||||
it,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
if (model.settings.isNotEmpty()) {
|
||||
Text(
|
||||
// In the words the dialog uses, and in the order it draws them: a summary
|
||||
// naming `contextSize` is a summary of a different screen than the one it
|
||||
// sits under.
|
||||
specs
|
||||
.mapNotNull { spec ->
|
||||
model.settings[spec.key]?.let { "${spec.label} $it" }
|
||||
}
|
||||
.joinToString(", "),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
if (onUnload != null) {
|
||||
Row { TextButton(enabled = enabled, onClick = onUnload) { Text("Unload") } }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* How one model is loaded.
|
||||
*
|
||||
* Saved on Save rather than as it is typed, unlike the session settings dialog: writing this
|
||||
* unloads the model for everybody using it, which is not something to do once per keystroke.
|
||||
*/
|
||||
@Composable
|
||||
private fun ModelSettingsDialog(
|
||||
model: ProviderModel,
|
||||
specs: List<ParamSpec>,
|
||||
onDismiss: () -> Unit,
|
||||
onSave: (Map<String, String>) -> Unit,
|
||||
) {
|
||||
var params by remember(model.id) { mutableStateOf(model.settings) }
|
||||
AlertDialog(
|
||||
onDismissRequest = onDismiss,
|
||||
// Every control here is a number, so the keyboard is up for most of this dialog's life --
|
||||
// and a dialog that keeps its own size under the keyboard puts Save off the bottom of the
|
||||
// screen, where nothing on screen says it is there. Taking the insets ourselves is what
|
||||
// lets `imePadding` shrink it instead.
|
||||
properties = DialogProperties(decorFitsSystemWindows = false),
|
||||
modifier = Modifier.imePadding(),
|
||||
title = { Text(model.label) },
|
||||
text = {
|
||||
Column(Modifier.verticalScroll(rememberScrollState())) {
|
||||
Text(
|
||||
if (model.status == "loaded" || model.status == "sleeping") {
|
||||
"This model is loaded. Saving takes it out of memory, and the sessions " +
|
||||
"using it load it again with these settings on their next message."
|
||||
} else {
|
||||
"Read when this model is next loaded."
|
||||
},
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
Spacer(Modifier.height(12.dp))
|
||||
ProviderParamFields(
|
||||
specs = specs,
|
||||
values = params,
|
||||
onChange = { params = it },
|
||||
// Every one of these is read at load time, and the sentence above already
|
||||
// says when that is -- marking each control "on restart" would repeat it six
|
||||
// times.
|
||||
warnAboutRestart = false,
|
||||
)
|
||||
}
|
||||
},
|
||||
confirmButton = { TextButton(onClick = { onSave(params) }) { Text("Save") } },
|
||||
dismissButton = { TextButton(onClick = onDismiss) { Text("Cancel") } },
|
||||
)
|
||||
}
|
||||
Reference in new issue
Block a user