diff --git a/AGENTS.md b/AGENTS.md index 658cefd..5ea9736 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -62,6 +62,15 @@ Module-by-module intent is in PLAN.md's "Backend layout". MTP draft head is a 50% speed-up or a 33% loss; and `--spec-type draft-mtp` is conditional on the file actually having a head, because asking for one that is not there makes `llama-server` **exit**. + **Every one of those is a default rather than a constant** (2026-09-19): + `DriverKind::params` declares what a provider takes — key, label, shape, + and whether a change waits for a restart — and the phone renders whatever + arrives, on the spawn form and in the session settings dialog. Adding a + setting to a driver is one entry in that table and no app change. `tools` + is in there too, because the seven built-in definitions are ~1,300 tokens + of every prompt (2,191 against 887 with none), which on a small window is + the difference between a usable session and one that overruns; `"none"` + omits the flag, since `--tools none` is a server that exits. Codex is one persistent `codex app-server --stdio` process per session; its driver uses native turn steering and interruption, persists the protocol state and thread id, and reads subscription limits through the same CLI @@ -325,6 +334,11 @@ written, and the fold uses that same predicate to decide a reply is settled. ## Things that have bitten +- **A server started with no `--tools` answers 403 at `GET /tools`, not an + empty list.** The route is off rather than empty, so reading that as a + failure made "no tools" — the one setting whose entire purpose is to have + none — a session that never started. + - **A llama session reports `loading`, and a message sent into it waits.** Before 2026-09-19 the session showed `running` from the moment the process started, so a minute of reading a model off disk was indistinguishable from diff --git a/PLAN.md b/PLAN.md index caf67c3..6dd217c 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1435,6 +1435,26 @@ verified by running it, matching dev-updater's posture. it truncates old KV cache entries, which is silent forgetting with no summary, and it corrupts the harness's view of what the model knows. Fine as a server-side safety net; not memory management. +- **What a provider's settings are is declared by the server** (2026-09-19, + `DriverKind::params`). A spec is a key, words, a shape and whether a change + waits for a restart; the phone renders whatever arrives, so a driver that + grows a setting gets a control with no app change. The reason it is + declared rather than drawn is not tidiness: several `llama-server` flags + were hardcoded to what measured best on one machine, which is right as a + default and wrong as a constant — the next machine has a different GPU, and + nobody running this app can edit the source. `POST /sessions/{id}/params` + takes the whole map, so an absent key *is* the instruction to unset. +- **The context figure has a denominator where one can be measured** + (2026-09-19). `Event::ContextWindow` carries it, read from `llama-server`'s + `/props` once the model is up — the measurement rather than the request, + since a session that named no context size gets the model's own. Neither + coding CLI states its window, so those keep the bare figure: "2,042" and + "2,042 / 8,192" are deliberately different-looking, and a missing ceiling is + never drawn as a proportion of an assumed one. + The numerator was also wrong, by the length of the last reply: it was the + prompt alone, so a five-word answer reported 2,042 against a slot holding + 2,355. It is the turn's total now, which matches `llama-server`'s own + `n_tokens` exactly. - **MCP servers are configured in `config.ron`, not from the phone** (2026-09-19). `mcpServers` on a llama provider, with Exa preset on a newly discovered one. A phone screen for them is the obvious next step and was diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt index 5bf6e06..ee03e57 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt @@ -204,6 +204,13 @@ data class SessionSummary( * full. */ val contextTokens: Long?, + /** + * What [contextTokens] is out of, or null where this session's provider does not say. A third + * state, not a fourth reading of the same one: the occupancy is known and the ceiling is not. + */ + val contextLimit: Long?, + /** What this session's provider settings are set to; empty where it takes none. */ + val params: Map, /** * The longest edge an image should have when it reaches this session, or null where the * provider has no limit. @@ -262,6 +269,8 @@ private fun parseSession(session: JSONObject) = cwd = session.optString("cwd").ifEmpty { null }, contextTokens = if (session.has("contextTokens")) session.getLong("contextTokens") else null, + contextLimit = if (session.has("contextLimit")) session.getLong("contextLimit") else null, + params = session.optJSONObject("params").stringMap(), maxImageEdge = session.optInt("maxImageEdge", 0).takeIf { it > 0 }, usageProvider = session.optString("usageProvider").ifEmpty { null }, status = session.getString("status"), @@ -344,6 +353,27 @@ data class Provider( val models: List, val permissionModes: List, val defaultPermissionMode: String?, + /** The extra settings this provider takes, in the order to draw them. See [ParamSpec]. */ + val params: List, +) + +/** + * One setting a provider takes, as the server describes it. + * + * Declared by the server rather than drawn here, so a driver that grows a setting gets a control + * with no app change — and, more to the point, so that a value measured on one machine can ship as + * a default without becoming a constant nobody else can reach. + */ +data class ParamSpec( + val key: String, + val label: String, + /** What leaving it blank means, in words, shown as the placeholder. */ + val unset: String, + val kind: String, + /** For [kind] `"choice"`: the options, the first of which means "unset". */ + val options: List, + /** Whether a change waits for the session's process to start again. */ + val restart: Boolean, ) /** @@ -359,6 +389,11 @@ data class Machine( val providers: List, ) +/** A JSON object of strings, and the empty map for one that is absent. */ +private fun JSONObject?.stringMap(): Map = + this?.let { object_ -> object_.keys().asSequence().associateWith { object_.getString(it) } } + ?: emptyMap() + private fun parseProvider(provider: JSONObject): Provider { val kind = provider.getString("kind") return Provider( @@ -368,6 +403,17 @@ private fun parseProvider(provider: JSONObject): Provider { models = provider.optJSONArray("models")?.strings().orEmpty(), permissionModes = provider.optJSONArray("permissionModes")?.strings().orEmpty(), defaultPermissionMode = provider.optString("defaultPermissionMode").ifEmpty { null }, + params = + provider.optJSONArray("params")?.mapObjects { spec -> + ParamSpec( + key = spec.getString("key"), + label = spec.getString("label"), + unset = spec.getString("unset"), + kind = spec.getString("kind"), + options = spec.optJSONArray("options")?.strings().orEmpty(), + restart = spec.optBoolean("restart"), + ) + } ?: emptyList(), ) } @@ -481,6 +527,13 @@ data class Importable( * history from before a compaction, which the model is no longer given. */ val contextTokens: Long?, + /** + * What [contextTokens] is out of, or null where this session's provider does not say. A third + * state, not a fourth reading of the same one: the occupancy is known and the ceiling is not. + */ + val contextLimit: Long?, + /** What this session's provider settings are set to; empty where it takes none. */ + val params: Map, /** Whether [title] is a name somebody chose rather than the last thing said in the session. */ val named: Boolean, /** @@ -555,6 +608,10 @@ fun fetchImportable(settings: ServerSettings, machine: String): List contextTokens = if (session.isNull("contextTokens")) null else session.optLong("contextTokens").takeIf { it > 0L }, + contextLimit = + if (session.isNull("contextLimit")) null + else session.optLong("contextLimit").takeIf { it > 0L }, + params = session.optJSONObject("params").stringMap(), // Absent means an older backend that cannot answer, which is what "unknown" says. inUse = session.optString("inUse", "unknown"), named = session.optBoolean("named", false), @@ -1240,6 +1297,16 @@ fun setSessionEffort(settings: ServerSettings, sessionId: String, level: String? } /** Switches how much a running session asks before acting, also in place. */ +/** Replaces a session's provider settings with [params] — the whole map, not a patch. */ +fun setSessionParams(settings: ServerSettings, sessionId: String, params: Map) { + requestFromServer( + settings, + "/sessions/$sessionId/params", + method = "POST", + jsonBody = JSONObject().put("params", JSONObject(params.toMap())).toString(), + ) {} +} + fun setSessionPermissionMode(settings: ServerSettings, sessionId: String, mode: String) { requestFromServer( settings, diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Compaction.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Compaction.kt index 0b199b9..638fe0e 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/Compaction.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Compaction.kt @@ -60,3 +60,23 @@ fun compactingLabel(seconds: Long?): String = seconds < 60 -> "compacting ${seconds}s" else -> "compacting ${seconds / 60}m ${seconds % 60}s" } + +/** + * How full the session is, as the status row says it. + * + * Three states, not two, and the third is the one that needed the words: a session whose occupancy + * is known and whose ceiling is not. That one keeps the bare figure, and a session with a ceiling + * gets both — the reader can see which they are looking at. What must not happen is a missing + * ceiling drawn as a number, or as a proportion of some assumed window, which would be this screen + * inventing the very fact it does not have. + * + * A llama.cpp session always has one, since the window is a flag its own server was started with. A + * coding CLI's is the vendor's business and neither control protocol states it, so those keep the + * bare figure they have always had. + */ +fun contextLabel(held: Long?, limit: Long?): String = + when { + held == null -> "context unknown" + limit == null -> "context ${tokens(held)}" + else -> "context ${tokens(held)} / ${tokens(limit)}" + } diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt index 02ff430..e5cbfde 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt @@ -163,6 +163,9 @@ sealed class SessionEvent { */ data class UsageDelta(val tokens: Long, val context: Long?) : SessionEvent() + /** How much context this session's model has, which is what [UsageDelta.context] is out of. */ + data class ContextWindow(val tokens: Long) : SessionEvent() + /** * A compaction that finished, and how much context it recovered. * @@ -293,6 +296,7 @@ fun parseSeqEvent(json: String): SeqEvent { model = body.optString("model").ifEmpty { null }, permissionMode = body.optString("permissionMode").ifEmpty { null }, ) + "contextWindow" -> SessionEvent.ContextWindow(body.getLong("tokens")) "usageDelta" -> SessionEvent.UsageDelta( body.getLong("tokens"), @@ -363,3 +367,18 @@ fun contextAfter(current: Long?, event: SessionEvent): Long? = is SessionEvent.Cleared -> null else -> current } + +/** + * The context window after [event], mirroring the server's `context_limit_after` for the same + * reason [contextAfter] mirrors its neighbour: the screen has to keep up between page loads. + * + * A window belongs to the process, so a session whose process has exited has none — left standing, + * a session restarted on a different model would draw its occupancy against the old model's + * ceiling. + */ +fun contextLimitAfter(current: Long?, event: SessionEvent): Long? = + when (event) { + is SessionEvent.ContextWindow -> event.tokens + is SessionEvent.Status -> if (event.state == "exited") null else current + else -> current + } diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/ProviderParams.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/ProviderParams.kt new file mode 100644 index 0000000..9483533 --- /dev/null +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/ProviderParams.kt @@ -0,0 +1,117 @@ +package com.example.aiapp + +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.text.KeyboardOptions +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.OutlinedTextField +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.ui.Modifier +import androidx.compose.ui.text.input.KeyboardType +import androidx.compose.ui.unit.dp + +/** + * The controls for whatever settings a provider says it takes. + * + * One composable for both screens that offer them — the spawn form and the session settings dialog + * — and for every provider, because the server declares the list (see `DriverKind::params`) rather + * than this file knowing it. A driver that grows a setting gets a control here with no change to + * the app, which is the whole point: the values that suit one machine ship as defaults, and every + * one of them stays reachable from a phone. + * + * [values] is the whole map and [onChange] hands back the whole map. A key absent from it means the + * setting is unset, which is what every [ParamSpec.unset] describes — so clearing a field and never + * touching it are deliberately the same state. + */ +@Composable +fun ProviderParamFields( + specs: List, + values: Map, + onChange: (Map) -> Unit, + /** + * Whether to say which settings wait for a restart. False on a spawn form, where nothing is + * running yet and every setting is about to be read — saying it there would be a warning about + * a state the reader cannot be in. + */ + warnAboutRestart: Boolean, + modifier: Modifier = Modifier, +) { + if (specs.isEmpty()) return + Column(modifier.fillMaxWidth()) { + specs.forEach { spec -> + val set = { value: String -> + onChange( + // Blank clears rather than storing an empty string: the server reads an absent + // key as "use the default", and an empty one would be a value it then failed + // to parse. + if (value.isBlank()) values - spec.key else values + (spec.key to value) + ) + } + when (spec.kind) { + "choice" -> { + // The first option is what unset means, so selecting it clears the key — see + // `ParamKind::Choice`. Without that the picker could show a default it could + // not return to. + val default = spec.options.firstOrNull().orEmpty() + ChipGroup( + label = spec.label + restartSuffix(spec, warnAboutRestart), + options = spec.options, + selected = values[spec.key] ?: default, + onSelect = { chosen -> set(if (chosen == default) "" else chosen) }, + ) + } + else -> + OutlinedTextField( + value = values[spec.key].orEmpty(), + onValueChange = set, + label = { Text(spec.label + restartSuffix(spec, warnAboutRestart)) }, + placeholder = { Text(spec.unset) }, + singleLine = true, + keyboardOptions = KeyboardOptions(keyboardType = keyboardFor(spec.kind)), + modifier = Modifier.fillMaxWidth(), + ) + } + Spacer(Modifier.height(16.dp)) + } + if (warnAboutRestart && specs.any { it.restart }) { + Text( + "A setting marked “on restart” is saved now and read when this session's process " + + "next starts.", + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + } +} + +/** + * Marks a control whose value will not take effect yet. + * + * On the label rather than beside it, because the reader decides whether to change the thing before + * they touch it — a note underneath is read after the decision. + */ +private fun restartSuffix(spec: ParamSpec, warn: Boolean): String = + if (warn && spec.restart) " (on restart)" else "" + +/** + * The keyboard for a value's shape. A number field that opens the letter keyboard is one every + * entry is made harder by, and these are nearly all numbers. + */ +private fun keyboardFor(kind: String): KeyboardType = + when (kind) { + "integer" -> KeyboardType.Number + "decimal" -> KeyboardType.Decimal + else -> KeyboardType.Text + } + +/** + * How long typing has to stop before edited settings are sent. + * + * Long enough that a number is one request rather than one per digit, short enough that closing the + * dialog straight after typing still saves — the save runs on the screen behind it, which outlives + * the dialog, so this delay is not a window the value can be lost in. + */ +const val PARAM_SAVE_DELAY_MS = 700L diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt index 00c4bf7..0ab5cfa 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt @@ -233,6 +233,8 @@ fun SessionScreen( // that was measured. var contextTokens by remember(address) { mutableStateOf(if (isSubagent) null else summary.contextTokens) } + var contextLimit by + remember(address) { mutableStateOf(if (isSubagent) null else summary.contextLimit) } // When the current compaction started. The moment comes off the `compacting` status event // itself -- the server timestamps every transcript line -- rather than off this device noticing // one, which is what makes it survive leaving the session and reopening it. @@ -304,6 +306,14 @@ fun SessionScreen( // hardcoded list is a claim about a machine. var offeredModels by remember { mutableStateOf>(emptyList()) } var offeredPermissionModes by remember { mutableStateOf>(emptyList()) } + // The settings this session's provider takes, and what they are set to. The specs come from + // the provider and the values from the session, because "what can be set" and "what is set" + // are different questions with different answers. + var paramSpecs by remember { mutableStateOf>(emptyList()) } + var params by remember(summary.id) { mutableStateOf(summary.params) } + // What was last successfully saved, so the debounce below knows whether there is anything to + // send -- and so a failed save can put the controls back to what the server actually holds. + var savedParams by remember(summary.id) { mutableStateOf(summary.params) } val lifecycleOwner = LocalLifecycleOwner.current // The resume cursor, written from the stream's IO thread. val lastSeq = remember { AtomicLong(0) } @@ -489,6 +499,7 @@ fun SessionScreen( // Before the rest, and for every event rather than only the usage ones: a compaction and a // clear move this as much as a turn does. See `contextAfter`. contextTokens = contextAfter(contextTokens, entry.event) + contextLimit = contextLimitAfter(contextLimit, entry.event) when (val event = entry.event) { // Nothing further: what it carries was folded into the context above. is SessionEvent.UsageDelta -> {} @@ -1205,6 +1216,30 @@ fun SessionScreen( fun label(id: String?): String = offeredModels.firstOrNull { it.id == id }?.label ?: modelLabel(id) + /** + * Saves edited provider settings once the typing stops. + * + * Debounced rather than sent per keystroke, because a text field over the tunnel would be a + * round trip and a config write per character. Here rather than in the settings dialog for the + * reason the delay exists at all: the dialog can be dismissed mid-edit, and this screen + * outlives it, so the last value typed is still saved. + */ + LaunchedEffect(params) { + if (params == savedParams) return@LaunchedEffect + delay(PARAM_SAVE_DELAY_MS) + val wanted = params + try { + withContext(Dispatchers.IO) { setSessionParams(settings, summary.id, wanted) } + savedParams = wanted + actionError = null + } catch (e: ApiException) { + // Back to what the server holds. A control left showing a value that was refused is + // stating something untrue about the session. + params = savedParams + actionError = e.message + } + } + // Only for the model picker, which a subagent does not have. if (!isSubagent) { LaunchedEffect(summary.machine, summary.provider) { @@ -1218,6 +1253,7 @@ fun SessionScreen( } .getOrNull() offeredPermissionModes = provider?.permissionModes.orEmpty() + paramSpecs = provider?.params.orEmpty() offeredModels = provider ?.let { @@ -1916,6 +1952,7 @@ fun SessionScreen( status = status, compactingFor = compactingFor, contextTokens = contextTokens, + contextLimit = contextLimit, backgroundTasks = backgroundTasks, subagent = isSubagent, ) @@ -2197,6 +2234,9 @@ fun SessionScreen( effort = effort.takeIf { summary.takesEffort }, takesEffort = summary.takesEffort, onEffortChanged = { effort = it }, + paramSpecs = paramSpecs, + params = params, + onParamsChanged = { params = it }, cachedBytes = cachedBytes, // The purge finishes before the epoch moves, because the relaunched opening effect // reads the same directory and would otherwise draw what is about to be deleted. The @@ -2473,6 +2513,8 @@ private fun SessionStatusRow( compactingFor: Long?, /** Context the session is holding, or null where nothing has measured it. */ contextTokens: Long?, + /** What that is out of, or null where the provider does not say. */ + contextLimit: Long?, /** Provider-reported live background work; zero is deliberately not drawn. */ backgroundTasks: Int, modifier: Modifier = Modifier, @@ -2577,7 +2619,7 @@ private fun SessionStatusRow( // reports usage, and one that has not run a turn all showed nothing at all, which reads as // a conversation with room to spare. Text( - contextTokens?.let { "context ${tokens(it)}" } ?: "context unknown", + contextLabel(contextTokens, contextLimit), style = MaterialTheme.typography.labelSmall, color = MaterialTheme.colorScheme.onSurfaceVariant, ) diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/SessionSettingsDialog.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/SessionSettingsDialog.kt index ab50ca6..866df57 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/SessionSettingsDialog.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/SessionSettingsDialog.kt @@ -74,6 +74,15 @@ fun SessionSettingsDialog( onEffortChanged: (String?) -> Unit, /** Whether a level does anything here; the row is left out entirely where it does not. */ takesEffort: Boolean, + /** + * The settings this session's provider takes, and what they are set to. + * + * Declared by the server rather than listed here -- see [ProviderParamFields]. Empty for a + * provider with none, which draws no section at all. + */ + paramSpecs: List, + params: Map, + onParamsChanged: (Map) -> Unit, /** * What this phone is holding of the conversation, or null while that is being measured -- see * the Reload row below, which is what would discard it. @@ -451,6 +460,23 @@ fun SessionSettingsDialog( ) } } + if (paramSpecs.isNotEmpty()) { + Spacer(Modifier.height(16.dp)) + Text( + "Model settings", + style = MaterialTheme.typography.titleSmall, + ) + Spacer(Modifier.height(8.dp)) + // Edited here and saved by the screen behind this, which is what makes + // typing in a text field affordable: the save is debounced, and a dialog + // dismissed mid-edit would take an unsaved value with it. + ProviderParamFields( + specs = paramSpecs, + values = params, + onChange = onParamsChanged, + warnAboutRestart = true, + ) + } Spacer(Modifier.height(8.dp)) Row( verticalAlignment = Alignment.CenterVertically, diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/SpawnScreen.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/SpawnScreen.kt index b0570f5..3084775 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/SpawnScreen.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/SpawnScreen.kt @@ -73,13 +73,9 @@ fun SpawnScreen( // Only the spawn's own failure. The fetch's lives in `options`: this one leaves a filled-in // form worth keeping, and that one leaves nothing to fill in. var spawnError by remember { mutableStateOf(null) } - var contextSize by remember { mutableStateOf("") } - var temperature by remember { mutableStateOf("") } - // Whether a model that carries a multi-token-prediction head drafts with it. Left to the - // server by default, which turns it on exactly where the file has one -- see `SPECULATIVE` in - // the llama driver. Here so a machine where drafting does not pay has a way out that is not - // an edit to config.ron. - var speculative by remember { mutableStateOf(SPECULATIVE_AUTO) } + // Whatever the chosen provider says it takes, by key. Empty until something is typed: an + // absent key means the server's own default, which is what every field's placeholder says. + var params by remember { mutableStateOf>(emptyMap()) } LaunchedEffect(Unit) { // Separate from the machines fetch below and deliberately not fatal: failing to learn the @@ -145,6 +141,9 @@ fun SpawnScreen( // machine looks exactly like one from this one. LaunchedEffect(machine?.id, current?.name) { model = "" + // A key from the previous provider would be a setting this one does not have, drawn + // by no control and sent at the spawn anyway. + params = emptyMap() providerModels = emptyList() providerModelsError = null permissionMode = current?.defaultPermissionMode.orEmpty() @@ -283,35 +282,14 @@ fun SpawnScreen( Spacer(Modifier.height(16.dp)) } - if (isLlama) { - OutlinedTextField( - value = contextSize, - onValueChange = { contextSize = it }, - label = { Text("Context size (blank = the model's default)") }, - singleLine = true, - modifier = Modifier.fillMaxWidth(), - ) - Spacer(Modifier.height(16.dp)) - - OutlinedTextField( - value = temperature, - onValueChange = { temperature = it }, - label = { Text("Temperature (blank = llama.cpp's default)") }, - singleLine = true, - modifier = Modifier.fillMaxWidth(), - ) - Spacer(Modifier.height(16.dp)) - - // Said as what it is rather than as "MTP": the reader is choosing whether the session - // goes faster, and most models have nothing to turn on here at all. - ChipGroup( - label = "Speculative decoding (models that carry a draft head)", - options = listOf(SPECULATIVE_AUTO, SPECULATIVE_OFF), - selected = speculative, - onSelect = { speculative = it }, - ) - Spacer(Modifier.height(16.dp)) - } + // Nothing is running yet, so nothing here waits for a restart -- every one of these is + // read by the process this form is about to start. + ProviderParamFields( + specs = current?.params.orEmpty(), + values = params, + onChange = { params = it }, + warnAboutRestart = false, + ) // Every session whose tools act on files needs one, which is both kinds that have // tools -- a llama session's built-in tools run in it exactly as a CLI's do. @@ -387,28 +365,10 @@ fun SpawnScreen( cwd = cwd.trim().takeIf { takesCwd }, permissionMode = permissionMode.takeIf { it.isNotEmpty() }, effort = effort.takeIf { isCodingCli }, - // Sent only when set, so blank means "whatever llama.cpp does - // by default" rather than a zero. - params = - buildMap { - if (isLlama) { - contextSize - .trim() - .takeIf { it.isNotEmpty() } - ?.let { put("contextSize", it) } - temperature - .trim() - .takeIf { it.isNotEmpty() } - ?.let { put("temperature", it) } - // Only the choice that changes anything: "auto" - // is the absence of the setting, not a value of - // it, so a session spawned without an opinion - // carries none. - if (speculative == SPECULATIVE_OFF) { - put("speculative", "off") - } - } - }, + // Already only the keys somebody set: a field left blank + // removes its key rather than sending an empty value, so + // "blank" reaches the server as "your default". + params = params, ) } onSpawned(spawned) @@ -426,15 +386,6 @@ fun SpawnScreen( } } -/** - * Leave the draft head to the server, which uses one wherever the model file has one. Spelled the - * same as the absence of the `speculative` parameter, because that is what it means. - */ -private const val SPECULATIVE_AUTO = "auto" - -/** The `speculative` parameter's only other value; see the llama driver's `SPECULATIVE`. */ -private const val SPECULATIVE_OFF = "off" - /** * A labeled row of choices that wraps onto as many lines as it needs. * diff --git a/app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt b/app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt index 6c6ee2c..9abbd67 100644 --- a/app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt +++ b/app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt @@ -555,6 +555,7 @@ fun foldEvent(items: List, entry: SeqEvent): List items // Screen-level state, not transcript rows -- see SessionScreen. is SessionEvent.UsageDelta -> items + is SessionEvent.ContextWindow -> items } /** diff --git a/app/androidApp/src/test/kotlin/com/example/aiapp/SessionOrderTest.kt b/app/androidApp/src/test/kotlin/com/example/aiapp/SessionOrderTest.kt index d2aa050..3fa55d3 100644 --- a/app/androidApp/src/test/kotlin/com/example/aiapp/SessionOrderTest.kt +++ b/app/androidApp/src/test/kotlin/com/example/aiapp/SessionOrderTest.kt @@ -75,6 +75,8 @@ class SessionOrderTest { resumeAt = null, cwd = null, contextTokens = null, + contextLimit = null, + params = emptyMap(), maxImageEdge = null, usageProvider = null, status = status, diff --git a/server/src/config.rs b/server/src/config.rs index 0af947c..b66d368 100644 --- a/server/src/config.rs +++ b/server/src/config.rs @@ -312,6 +312,28 @@ impl DriverKind { } } + /// The settings this kind of session takes beyond the shared ones, for + /// the phone to offer. + /// + /// Declared rather than drawn: a spawn screen with a field per llama + /// setting is a screen that has to be edited every time a driver grows + /// one, and this app already has the `params` map to carry them. So the + /// server says what a provider takes and the phone renders it, which is + /// the same arrangement `permission_modes` uses and for the same reason + /// -- the alternative is two lists that disagree, one of them in Kotlin. + /// + /// It is also what keeps these *reachable at all*. Several were hardcoded + /// to the values measured on one machine, which is fine as a default and + /// wrong as a constant: the next machine has a different GPU and a + /// different number of cores, and nobody running this app can edit the + /// source. + pub fn params(self) -> &'static [ParamSpec] { + match self { + Self::LlamaCpp => LLAMA_PARAMS, + Self::Echo | Self::ClaudeCli | Self::CodexCli => &[], + } + } + /// The mode used when a new-session form first selects this kind. pub fn default_permission_mode(self) -> Option<&'static str> { match self { @@ -323,6 +345,134 @@ impl DriverKind { } } +/// One setting a provider takes, and enough about it to draw a control. +/// +/// Deliberately thin: a key, words for a person, and which shape the value +/// has. Anything richer -- units, validation, dependencies between settings -- +/// would be a schema language, and what the phone needs is a text field or a +/// row of chips. +#[derive(Debug, Clone, Copy, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ParamSpec { + /// The `SessionConfig::params` key this writes. + pub key: &'static str, + pub label: &'static str, + /// What happens when it is not set, in words. Shown where a control shows + /// its placeholder, so "blank" always means something specific rather than + /// leaving the reader to guess whether it means zero. + pub unset: &'static str, + /// Flattened, so a spec is one flat object: `kind` beside the rest rather + /// than an object of its own with `kind` inside it. + #[serde(flatten)] + pub kind: ParamKind, + /// Whether changing it waits for the process to start again. + /// + /// The honest half of offering these live. A sampling setting rides on the + /// next request; a server flag was decided when the model was loaded, and + /// a control that silently did nothing until some later restart would be + /// worse than one that is not there. + pub restart: bool, +} + +/// What shape a [`ParamSpec`]'s value has. +#[derive(Debug, Clone, Copy, Serialize)] +#[serde(rename_all = "camelCase", tag = "kind")] +pub enum ParamKind { + Integer, + Decimal, + Text, + /// A fixed set. **The first option is what leaving it unset means**, and + /// choosing it clears the setting rather than storing a value -- so the + /// default is a state the picker can return to, and the stored config + /// does not fill up with values nobody chose. + Choice { + options: &'static [&'static str], + }, +} + +/// What a llama.cpp session takes. +/// +/// The server flags first, in the order they matter, then the sampling ones -- +/// which is also the order of how disruptive changing one is. +const LLAMA_PARAMS: &[ParamSpec] = &[ + ParamSpec { + key: "contextSize", + label: "Context size", + unset: "the model's own trained context", + kind: ParamKind::Integer, + restart: true, + }, + ParamSpec { + key: "tools", + label: "Tools", + // Worth a control rather than a constant because of what it costs: + // the definitions of all seven are ~2,000 tokens of the context, + // every turn, before anything is said. On a small window that is the + // difference between a usable session and one that overruns. + unset: "all of them -- or a comma-separated list, or \"none\"", + kind: ParamKind::Text, + restart: true, + }, + ParamSpec { + key: "gpuLayers", + label: "Layers on the GPU", + unset: "as many as fit", + kind: ParamKind::Integer, + restart: true, + }, + ParamSpec { + key: "threads", + label: "Threads", + unset: "one per core", + kind: ParamKind::Integer, + restart: true, + }, + ParamSpec { + key: "speculative", + label: "Speculative decoding", + unset: "on, for a model whose file carries a draft head", + kind: ParamKind::Choice { + options: &["auto", "off"], + }, + restart: true, + }, + ParamSpec { + key: "specDraftNMax", + label: "Tokens drafted ahead", + unset: "llama.cpp's own default", + kind: ParamKind::Integer, + restart: true, + }, + ParamSpec { + key: "temperature", + label: "Temperature", + unset: "llama.cpp's default", + kind: ParamKind::Decimal, + restart: false, + }, + ParamSpec { + key: "topP", + label: "Top P", + unset: "llama.cpp's default", + kind: ParamKind::Decimal, + restart: false, + }, + ParamSpec { + key: "topK", + label: "Top K", + unset: "llama.cpp's default", + kind: ParamKind::Integer, + restart: false, + }, + ParamSpec { + key: "maxTokens", + label: "Reply limit", + unset: "no limit", + kind: ParamKind::Integer, + restart: false, + }, +]; + #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub struct TokenEntry { diff --git a/server/src/routes.rs b/server/src/routes.rs index 736a69c..b888ef7 100644 --- a/server/src/routes.rs +++ b/server/src/routes.rs @@ -54,6 +54,8 @@ //! which starts again in the new one //! POST /sessions/{id}/model {model} //! POST /sessions/{id}/permission-mode {permissionMode} +//! POST /sessions/{id}/params {params} -- the provider settings, whole; +//! what a provider takes is on its ProviderInfo //! POST /sessions/{id}/effort {effort} -- null for the CLI's default; //! settled at launch, so this stops the process //! POST /sessions/{id}/command {text} -- /compact, /clear, /rename x, or the dialect's own @@ -166,6 +168,7 @@ pub fn router(manager: Arc) -> Router { .route("/sessions/{id}/cwd", post(set_cwd)) .route("/sessions/{id}/model", post(set_model)) .route("/sessions/{id}/permission-mode", post(set_permission_mode)) + .route("/sessions/{id}/params", post(set_params)) .route("/sessions/{id}/effort", post(set_effort)) .route("/defaults", get(defaults).post(set_defaults)) .route("/sessions/{id}/notify", post(set_notify)) @@ -300,6 +303,11 @@ struct ProviderInfo { permission_modes: Vec<&'static str>, #[serde(skip_serializing_if = "Option::is_none")] default_permission_mode: Option<&'static str>, + /// The per-kind settings a spawn form and the session settings dialog + /// offer -- see [`crate::config::DriverKind::params`]. Empty for a + /// provider with none, which draws no section at all. + #[serde(skip_serializing_if = "<[_]>::is_empty")] + params: &'static [crate::config::ParamSpec], } async fn list_machines(State(manager): State>) -> axum::Json> { @@ -320,6 +328,7 @@ fn info_for(machine: crate::config::MachineConfig) -> MachineInfo { models: provider.models, permission_modes: provider.kind.permission_modes().to_vec(), default_permission_mode: provider.kind.default_permission_mode(), + params: provider.kind.params(), }) .collect(), } @@ -426,6 +435,7 @@ async fn probe_machine( models: provider.models, permission_modes: provider.kind.permission_modes().to_vec(), default_permission_mode: provider.kind.default_permission_mode(), + params: provider.kind.params(), }) .collect(), )) @@ -1586,6 +1596,24 @@ async fn set_permission_mode( Ok(StatusCode::NO_CONTENT) } +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct ParamsRequest { + /// The whole map, not a patch -- see `SessionManager::set_session_params`. + params: std::collections::BTreeMap, +} + +async fn set_params( + State(manager): State>, + UrlPath(id): UrlPath, + axum::Json(body): axum::Json, +) -> Result { + manager + .set_session_params(&id, body.params) + .map_err(bad_request)?; + Ok(StatusCode::NO_CONTENT) +} + #[derive(Deserialize)] #[serde(deny_unknown_fields)] struct NotifyRequest { diff --git a/server/src/session/driver.rs b/server/src/session/driver.rs index 79c9d46..bc6e96e 100644 --- a/server/src/session/driver.rs +++ b/server/src/session/driver.rs @@ -363,6 +363,26 @@ pub enum Event { #[serde(default, skip_serializing_if = "Option::is_none")] permission_mode: Option, }, + /// How much context this session's model has to hold a conversation in. + /// + /// The denominator the phone draws [`Event::UsageDelta`]'s `context` + /// against, and its own event rather than a field on that one because it + /// is not a per-turn measurement: it is fixed when the process starts and + /// changes only when a different one is started, which is what a model + /// change does. Reported the moment it is known, so the figure and what it + /// is out of arrive together rather than the first turn drawing a + /// numerator with no denominator. + /// + /// **Only ever sent by a driver that actually knows.** A window nobody has + /// measured is not an unlimited one: llama.cpp answers it exactly, because + /// the number is a flag the server was started with and `/props` reads it + /// back, while a coding CLI's context is the vendor's business and + /// nothing in either control protocol states it. Those send nothing, the + /// session has no limit, and the phone draws the figure on its own -- see + /// `SessionSummary::context_limit`. + ContextWindow { + tokens: u64, + }, /// Per-turn token counts, where the dialect reports them. UsageDelta { /// What this turn cost: the tokens it was charged for. @@ -494,6 +514,25 @@ pub fn context_tokens(input: u64, cache_creation: u64, cache_read: u64) -> u64 { input + cache_creation + cache_read } +/// The context window after `event`, given what it was before. +/// +/// Beside [`context_after`] because it has the same three readers and the same +/// hazard: a figure that outlives what made it true. A model change replaces +/// the process, so the window it reports replaces the old one -- and until the +/// new one says, there is no answer rather than the previous model's. +pub fn context_limit_after(current: Option, event: &Event) -> Option { + match event { + Event::ContextWindow { tokens } => Some(*tokens), + // The window belongs to the process, and a stopped one has none. Left + // standing, a restarted session on a different model would draw its + // occupancy against the previous model's window. + Event::Status { + state: SessionStatus::Exited, + } => None, + _ => current, + } +} + /// The context after `event`, given what it was before. /// /// The whole rule in one place, because three readers need the same answer: @@ -683,6 +722,16 @@ pub trait Driver: Send + Sync { /// `/rename` afterwards, which is what puts the same name in its own /// session picker and in what other agents see. fn set_title(&self, title: &str); + /// Takes the session's provider settings, whole. + /// + /// The whole map because it is a form's contents -- an absent key means + /// "unset", not "unchanged". A driver applies what it can apply now and + /// says so about the rest: the map is also on disk by the time this is + /// called, so a setting that only takes effect at the next start is not + /// lost, it is waiting. The default is right for a driver with no settings + /// of its own, which is every one but llama.cpp -- see + /// [`crate::config::DriverKind::params`]. + fn set_params(&self, _params: &std::collections::BTreeMap) {} /// Runs a command this session's own dialect understands, verbatim -- /// `/context`, `/usage`, anything a CLI adds next month. A driver with no /// such vocabulary says so with an [`Event::Error`] rather than sending it diff --git a/server/src/session/llama/mod.rs b/server/src/session/llama/mod.rs index 24b17b5..bfde61e 100644 --- a/server/src/session/llama/mod.rs +++ b/server/src/session/llama/mod.rs @@ -65,6 +65,32 @@ pub use tools::{DEFAULT_MODE as DEFAULT_PERMISSION_MODE, MODES as PERMISSION_MOD use mcp::McpServer; use tools::Tools; +/// The sampling half of a session's settings, in the wire's own names. +/// +/// One function because two callers need the identical mapping: the spawn, and +/// a later change through [`Driver::set_params`]. A value that will not parse +/// as a number is left out rather than passed through -- `llama-server` would +/// refuse the whole request for it, which would look like the session breaking +/// rather than like one field being wrong. +fn sampling_from( + params: &std::collections::BTreeMap, +) -> serde_json::Map { + let mut sampling = serde_json::Map::new(); + for (key, field) in [ + ("temperature", "temperature"), + ("topP", "top_p"), + ("topK", "top_k"), + ("maxTokens", "max_tokens"), + ] { + if let Some(raw) = params.get(key) + && let Ok(number) = raw.parse::() + { + sampling.insert(field.to_string(), json!(number)); + } + } + sampling +} + /// Exa's own hosted MCP server, which is what a llama session searches the web /// with. The address llama.cpp's web UI offers under "Exa" in its recommended /// servers, so a session here reaches the same thing that UI does. @@ -76,6 +102,11 @@ pub const EXA_MCP_URL: &str = "https://mcp.exa.ai/mcp"; /// is the escape for a machine where drafting turns out not to pay. const SPECULATIVE: &str = "speculative"; +/// The spawn parameter naming which built-in tools a session gets, as +/// `llama-server`'s own comma-separated list. Absent is all of them, and +/// `"none"` is the way to ask for a session that only talks. +const TOOLS: &str = "tools"; + /// How many times one message may go round the call-a-tool loop. /// /// A bound rather than a budget: a small model that has decided to read the @@ -221,8 +252,10 @@ struct Shared { /// Where this session's process record lives, so [`Driver::stop`] can find /// the server it has to end. session_dir: PathBuf, - /// Sampling settings chosen at spawn, sent with every request. - sampling: serde_json::Map, + /// Sampling settings, sent with every request. Behind a lock because they + /// are changeable while the session runs: they ride on the next request, + /// so unlike the server's own flags there is nothing to reload. + sampling: Mutex>, /// The session's working directory, which is where its tools act. `None` /// leaves that to `llama-server`, which is the honest answer rather than a /// guess at one. @@ -289,26 +322,14 @@ impl LlamaDriver { "a llama.cpp session needs a model -- one of the downloaded ones, by its key", )?; - let mut sampling = serde_json::Map::new(); - for (key, field) in [ - ("temperature", "temperature"), - ("topP", "top_p"), - ("topK", "top_k"), - ("maxTokens", "max_tokens"), - ] { - if let Some(raw) = meta.params.get(key) - && let Ok(number) = raw.parse::() - { - sampling.insert(field.to_string(), json!(number)); - } - } + let sampling = sampling_from(&meta.params); let driver = Self { shared: Arc::new(Shared { sink, transcript: transcript.to_path_buf(), session_dir: session_dir.to_path_buf(), - sampling, + sampling: Mutex::new(sampling), cwd: meta .cwd .as_ref() @@ -403,6 +424,15 @@ impl LlamaDriver { "{model} loaded and answering at {endpoint} with {} tools", tools.offered().map_or(0, |offered| offered.len()), ); + // Asked now rather than carried from the spawn flags: a + // session that named no context size gets the model's + // own, which only the loaded server knows, and one that + // named an impossible one gets whatever it settled for. + // Either way this is the measurement rather than the + // request. + if let Some(window) = context_window(&endpoint) { + shared.emit(Event::ContextWindow { tokens: window }); + } Serving::Ready { endpoint: endpoint.clone(), tools: Arc::new(tools), @@ -531,14 +561,12 @@ fn spawn_server( "127.0.0.1".into(), "--port".into(), forward.there.to_string(), - // The built-in agent tools -- read, search, edit, shell. Every one of - // them, because a session offered a subset is a session that says "I - // can't do that" about something it was installed to do, and the - // question of whether a particular call should happen is the - // permission gate's rather than a flag's. They run on the machine - // serving the model, which is the machine the files are on. - "--tools".into(), - "all".into(), + // The built-in agent tools -- read, search, edit, shell. All of them + // unless the session says otherwise, because whether a particular call + // should happen is the permission gate's question rather than a flag's. + // They run on the machine serving the model, which is the machine the + // files are on. + // // One slot, not the four `llama-server` picks on its own. A session is // one conversation making one request at a time -- the driver holds a // second message until the turn ends -- so the other three are context @@ -553,6 +581,27 @@ fn spawn_server( "-np".into(), "1".into(), ]; + // Settable because it is not free: the definitions of all seven are around + // 2,000 tokens of every prompt -- measured at 2,191 against 1,322 for two + // of them -- which on a small context window is a quarter of it spent + // before anything is said. + // + // "none" omits the flag rather than passing it on: `--tools none` is + // `tools setup failed: unknown tool "none"` and a server that exits, since + // the argument is a list of tool names and no-tools is what having no flag + // means. + match meta.params.get(TOOLS).map(|chosen| chosen.trim()) { + Some("none") => {} + chosen => { + args.push("--tools".into()); + args.push( + chosen + .filter(|c| !c.is_empty()) + .unwrap_or("all") + .to_string(), + ); + } + } // A model that carries a multi-token-prediction head drafts with it, which // is most of a 50% speed-up for free -- the tensors are in the file // whether or not they are used, and without the flag `llama-server` says @@ -573,6 +622,11 @@ fn spawn_server( ("contextSize", "-c"), ("gpuLayers", "-ngl"), ("threads", "-t"), + // How far ahead the draft head guesses. Not defaulted here: 2 measured + // 7% faster than llama.cpp's 3 on this machine's GPU, once, which is + // a reason to make the knob reachable and not a reason to move it for + // everybody. + ("specDraftNMax", "--spec-draft-n-max"), ] { if let Some(value) = meta.params.get(key) { args.push(flag.to_string()); @@ -854,7 +908,10 @@ fn converse( if shared.cancel.load(Ordering::SeqCst) { return Ok(()); } - let reply = generate(endpoint, &messages, tools, &shared.sampling, shared)?; + // Read per call rather than per turn, so a sampling change made while + // a long turn is running reaches the rest of it. + let sampling = shared.sampling.lock().unwrap().clone(); + let reply = generate(endpoint, &messages, tools, &sampling, shared)?; let calls = reply.calls; messages.push(Message { tool_calls: calls.iter().map(Call::wire).collect(), @@ -1046,6 +1103,49 @@ impl Driver for LlamaDriver { // is called, and the rename has already happened where the name lives. fn set_title(&self, _title: &str) {} + /// Takes new settings: the sampling half now, and says so about the rest. + /// + /// The split is what [`crate::config::ParamSpec::restart`] describes, and + /// it is said out loud rather than left to the screen, because the screen + /// can only say what a setting *usually* does -- this is the one place + /// that knows whether this session's server was started with the old + /// value. A session already stopped needs no such note: its next start + /// will read all of them. + fn set_params(&self, params: &std::collections::BTreeMap) { + *self.shared.sampling.lock().unwrap() = sampling_from(params); + // Only the settings that actually differ from what this session's + // server was started with. Listing every restart-only one on every + // save would be a wall of text about nothing having changed. + let waiting: Vec<&str> = crate::config::DriverKind::LlamaCpp + .params() + .iter() + .filter(|spec| { + spec.restart && params.get(spec.key) != self.respawn.meta.params.get(spec.key) + }) + .map(|spec| spec.label) + .collect(); + // Nothing to say to a session with no server: its next start reads all + // of them, which is what the note would have been asking for. + let running = matches!(&*self.shared.serving.lock().unwrap(), Serving::Ready { .. }); + if !waiting.is_empty() && running { + let one = waiting.len() == 1; + self.shared.emit(Event::Error { + message: format!( + "{} {} saved. {} when this session's server next starts -- stop and start \ + the session, or change its model, to load {} now.", + waiting.join(", "), + if one { "is" } else { "are" }, + if one { + "It takes effect" + } else { + "They take effect" + }, + if one { "it" } else { "them" }, + ), + }); + } + } + fn set_permission_mode(&self, mode: &str) { if !tools::MODES.contains(&mode) { self.shared.emit(Event::Error { @@ -1436,6 +1536,23 @@ fn log_tail(session_dir: &Path) -> String { /// How much of that log to carry into a message somebody reads on a phone. const LOG_TAIL_LINES: usize = 6; +/// How many tokens this server can hold, from the server itself. +/// +/// `/props` reports the per-slot context, which is the whole of it because +/// this driver always starts one slot -- see the `-np` argument. `None` for a +/// server that would not answer, which draws as no ceiling rather than as a +/// guessed one. +fn context_window(endpoint: &str) -> Option { + ureq::get(format!("{endpoint}/props")) + .call() + .ok()? + .body_mut() + .read_json::() + .ok()? + .pointer("/default_generation_settings/n_ctx") + .and_then(Value::as_u64) +} + /// One streamed completion: posts the conversation, emits each text delta as /// it arrives, and assembles whatever tool calls came with it. /// @@ -1491,9 +1608,16 @@ fn generate( // at once interleaves their fragments, each tagged with its index. let mut calls: Vec = Vec::new(); let mut tokens = 0u64; - // The prompt side only, which is what the model is holding -- the same - // definition the other dialects report, so one word on the phone means one - // thing whichever kind of session it is. + // What the model is holding when this call ends: the prompt it was given + // plus the reply it produced, which is exactly what the next call's prompt + // begins with. + // + // The prompt alone was wrong, and measurably: a five-word reply reported + // 2,042 against a slot holding 2,355 (2026-09-19, checked against + // `stop processing: n_tokens` in the server's own log). The gap is the + // reply, so it grows with how much the model just said -- which is the + // worst direction for a figure somebody is watching to see how much room + // is left. let mut context = None; for line in std::io::BufRead::lines(reader) { if shared.cancel.load(Ordering::SeqCst) { @@ -1510,13 +1634,13 @@ fn generate( let Ok(chunk) = serde_json::from_str::(payload) else { continue; }; - if let Some(usage) = chunk.get("usage") { - if let Some(total) = usage.get("total_tokens").and_then(Value::as_u64) { - tokens = total; - } - if let Some(prompt) = usage.get("prompt_tokens").and_then(Value::as_u64) { - context = Some(prompt); - } + // One figure answering both questions, which for this dialect it + // genuinely does: what the call was charged for and what the model is + // left holding are the same tokens, because nothing here is billed and + // the whole conversation is resent every time. + if let Some(total) = chunk.pointer("/usage/total_tokens").and_then(Value::as_u64) { + tokens = total; + context = Some(total); } let Some(delta) = chunk.pointer("/choices/0/delta") else { continue; diff --git a/server/src/session/llama/tools.rs b/server/src/session/llama/tools.rs index 61d9d6b..817fd87 100644 --- a/server/src/session/llama/tools.rs +++ b/server/src/session/llama/tools.rs @@ -57,18 +57,29 @@ impl Tools { /// Asks a ready `llama-server` what it offers and adds what the MCP /// servers offered. /// - /// A server started without `--tools` answers with an empty list, and a - /// session with only MCP tools is a perfectly good session -- so nothing - /// here treats "no tools" as a failure. What *is* a failure is not being - /// able to ask, because that is the same server the conversation is about - /// to go to. + /// A session with no built-in tools -- or none at all -- is a perfectly + /// good session, so nothing here treats "no tools" as a failure. What *is* + /// a failure is not being able to ask at all, because that is the same + /// server the conversation is about to go to. pub fn discover(endpoint: &str, mcp: Vec>>) -> Result { - let catalog: Vec = ureq::get(format!("{endpoint}/tools")) + let mut response = ureq::get(format!("{endpoint}/tools")) + // A server started with no `--tools` answers **403** here, not an + // empty list -- the route is off rather than empty. Read as a + // failure that was a session which never started, for the one + // setting whose whole purpose is to have no tools. + .config() + .http_status_as_error(false) + .build() .call() - .context("asking llama-server which tools it has")? - .body_mut() - .read_json() - .context("reading llama-server's tool list")?; + .context("asking llama-server which tools it has")?; + let catalog: Vec = if response.status().is_success() { + response + .body_mut() + .read_json() + .context("reading llama-server's tool list")? + } else { + Vec::new() + }; let mut definitions = Vec::new(); let mut server = HashMap::new(); for entry in &catalog { diff --git a/server/src/session/mod.rs b/server/src/session/mod.rs index e8a5800..7c6cf6a 100644 --- a/server/src/session/mod.rs +++ b/server/src/session/mod.rs @@ -36,7 +36,8 @@ use crate::config::{ use claude::ClaudeDriver; use codex::CodexDriver; use driver::{ - AttachmentRef, Driver, Event, EventSink, SessionCommand, SessionStatus, Unqueued, context_after, + AttachmentRef, Driver, Event, EventSink, SessionCommand, SessionStatus, Unqueued, + context_after, context_limit_after, }; use echo::EchoDriver; use llama::LlamaDriver; @@ -213,6 +214,17 @@ pub struct SessionInfo { /// are different answers and the phone draws them differently. #[serde(skip_serializing_if = "Option::is_none")] pub context_tokens: Option, + /// What that figure is out of -- see [`Event::ContextWindow`]. Absent + /// where the provider does not say, which is a third state again: not a + /// session with room to spare, and not one whose occupancy is unknown, + /// but one whose occupancy is known and whose ceiling is not. + #[serde(skip_serializing_if = "Option::is_none")] + pub context_limit: Option, + /// The provider settings this session was launched with, as the settings + /// dialog has to open on them -- what a control is *set to* is not + /// derivable from what the provider *offers*. + #[serde(skip_serializing_if = "std::collections::BTreeMap::is_empty")] + pub params: std::collections::BTreeMap, /// The longest edge an image should have by the time it gets here. /// Absent rather than a large number, because "no limit" and "a limit /// that happens to be big" are different answers. @@ -415,6 +427,7 @@ struct Shared { /// the session row so a phone opening a long conversation has the real /// figure rather than whatever its newest page mentions. context_tokens: Mutex>, + context_limit: Mutex>, /// Mirrored out of the config so the pump can read it without taking /// the manager's lock -- the pump runs underneath the manager, and /// reaching back up would invert that. @@ -579,6 +592,11 @@ impl LiveSession { effort: current.effort.clone(), takes_effort: kind.is_some_and(DriverKind::takes_effort), context_tokens: *self.shared.context_tokens.lock().unwrap(), + context_limit: *self.shared.context_limit.lock().unwrap(), + // From the config for the reason `effort` above is: the settings + // the process was started with are what a restart-only control has + // to show, and this is where they are kept. + params: current.params.clone(), notify: *self.shared.notify.lock().unwrap(), auto_resume: current.auto_resume, auto_resume_message: resume_message(current), @@ -1104,6 +1122,8 @@ impl SessionManager { takes_effort: kind_of(&inner.config, &meta.machine, &meta.provider) .is_some_and(DriverKind::takes_effort), context_tokens: None, + context_limit: None, + params: meta.params.clone(), max_image_edge: kind_of(&inner.config, &meta.machine, &meta.provider) .and_then(DriverKind::max_image_edge), usage_provider: kind_of(&inner.config, &meta.machine, &meta.provider) @@ -1319,6 +1339,43 @@ impl SessionManager { Ok(()) } + /// Changes this session's provider settings, live and persisted. + /// + /// The whole map rather than one key, because that is what a settings + /// screen has: a form is submitted as its contents, and merging one field + /// at a time would make clearing a field indistinguishable from not + /// mentioning it. An absent key *is* the instruction to unset it. + /// + /// Persisted first for the reason the model is: the config answers what to + /// launch with next time, which is the whole of what a restart-only + /// setting means. The driver is then told, and takes what it can use now. + pub fn set_session_params( + &self, + id: &str, + params: std::collections::BTreeMap, + ) -> Result<()> { + let mut inner = self.inner.write().unwrap(); + if !inner.config.sessions.iter().any(|meta| meta.id == id) { + bail!("no session {id}"); + } + let mut candidate = inner.config.clone(); + for meta in candidate.sessions.iter_mut().filter(|meta| meta.id == id) { + meta.params = params.clone(); + } + candidate.save(&self.config_path)?; + inner.config = candidate; + // Told to the driver where there is one, and simply saved where there + // is not. Deliberately not `ask`: a session with no process has not + // failed to take these, it has taken them in the only way that matters + // -- its next start reads them -- and adjusting the settings of a + // stopped session so that starting it uses them is the ordinary thing + // to do, not an error to report. + if let Some(driver) = inner.live.get(id).and_then(|session| session.driver()) { + driver.set_params(¶ms); + } + Ok(()) + } + /// Turns this session's notifications on or off, live and persisted -- /// both, or the switch moves back on its own at the next restart. /// @@ -2394,6 +2451,7 @@ fn launch( model: Mutex::new(meta.model.clone()), permission_mode: Mutex::new(meta.permission_mode.clone()), context_tokens: Mutex::new(transcript.context_tokens()), + context_limit: Mutex::new(transcript.context_limit()), notify: Mutex::new(meta.notify), written: Mutex::new(0), }); @@ -2674,6 +2732,10 @@ async fn pump( let mut context = shared.context_tokens.lock().unwrap(); *context = context_after(*context, &event); } + { + let mut limit = shared.context_limit.lock().unwrap(); + *limit = context_limit_after(*limit, &event); + } // Nothing changed, so there is nothing to record. Both of these // repeat: an imported session reads the turn state off its file's // newest record on every sync, and the CLI restates its model and diff --git a/server/src/session/transcript.rs b/server/src/session/transcript.rs index e6d698e..24388cf 100644 --- a/server/src/session/transcript.rs +++ b/server/src/session/transcript.rs @@ -15,7 +15,7 @@ use std::path::Path; use anyhow::{Context, Result}; use serde::{Deserialize, Serialize}; -use super::driver::{Event, SessionStatus, context_after}; +use super::driver::{Event, SessionStatus, context_after, context_limit_after}; /// One transcript line: an [`Event`] plus its position and time. The event /// is flattened so the wire shape stays one flat object. @@ -34,6 +34,7 @@ pub struct Transcript { last_status: Option, last_activity: Option, context_tokens: Option, + context_limit: Option, } impl Transcript { @@ -68,6 +69,12 @@ impl Transcript { context_tokens: existing .iter() .fold(None, |current, entry| context_after(current, &entry.event)), + // The same fold for the same reason: a session whose process has + // since exited has no window, and the newest `ContextWindow` line + // alone would not know that. + context_limit: existing.iter().fold(None, |current, entry| { + context_limit_after(current, &entry.event) + }), }) } @@ -107,6 +114,12 @@ impl Transcript { self.context_tokens } + /// What that figure is out of, as of opening, and `None` where this + /// session's provider does not say. + pub fn context_limit(&self) -> Option { + self.context_limit + } + /// Appends `event`, assigning it the next sequence number. Flushed per /// event: each line is tiny, and the transcript is the source of truth a /// crash must not lose the tail of.