iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Declare provider settings, and give the context figure a denominator

Browse Source 
Two things a session could not say, and one it was saying wrongly.

**Every provider setting is reachable.** `-np 1`, the MTP draft depth, the
tool set, the sampling parameters -- most were hardcoded to what measured
best on this machine, which is right as a default and wrong as a constant:
the next machine has a different GPU and a different core count, and
nobody running this app can edit the source. `DriverKind::params` now
declares what a provider takes -- key, label, shape, what blank means, and
whether a change waits for a restart -- and the phone renders whatever
arrives, on the spawn form and in the session settings dialog. Adding a
setting to a driver is one entry in that table and no app change.
`POST /sessions/{id}/params` takes the whole map, so an absent key is the
instruction to unset; the sampling half applies at once and the session is
told in words which of the rest are waiting for a restart.

`tools` is one of them, because it is the biggest lever on a tight
context: the seven built-in definitions are ~1,300 tokens of every prompt
(2,191 against 887 with none). `"none"` omits the flag rather than passing
it on, since `--tools none` is `unknown tool "none"` and a server that
exits.

**The context figure has a denominator.** `Event::ContextWindow` carries
it, read from `llama-server`'s `/props` once the model is up -- the
measurement rather than the request, since a session that named no context
size gets the model's own. Neither coding CLI states its window, so those
keep the bare figure: "2,042" and "2,042 / 8,192" are deliberately
different-looking, and a missing ceiling is never drawn as a proportion of
an assumed one.

**And the numerator was wrong**, by the length of the last reply: it was
the prompt alone, so a five-word answer reported 2,042 against a slot
holding 2,355. It is the turn's total now, which matches `llama-server`'s
own `n_tokens` to within a token.

Two defects the review found, both of which would have shipped: changing
settings on a *stopped* session reported "no process running, so it can't
take new settings", when a stopped session is exactly when you would set
them for the next start; and `GET /tools` answers **403** rather than an
empty list on a server started without `--tools`, so reading it as a
failure made the no-tools session one that never started.

Verified against real models: settings spawned and changed live, the
restart note, a session with two tools and one with none, and the counter
checked against the server's own slot occupancy each time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
iris-aiandClaude Opus 5 committed 2026-09-19 13:58:08 -04:00
1 parent ac476ab0c9
commit 81ab564a09
18 files changed
+831 -115
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+14
View File
@@ -62,6 +62,15 @@ Module-by-module intent is in PLAN.md's "Backend layout".
MTP draft head is a 50% speed-up or a 33% loss; and `--spec-type draft-mtp`
is conditional on the file actually having a head, because asking for one
that is not there makes `llama-server` **exit**.
**Every one of those is a default rather than a constant** (2026-09-19):
`DriverKind::params` declares what a provider takes — key, label, shape,
and whether a change waits for a restart — and the phone renders whatever
arrives, on the spawn form and in the session settings dialog. Adding a
setting to a driver is one entry in that table and no app change. `tools`
is in there too, because the seven built-in definitions are ~1,300 tokens
of every prompt (2,191 against 887 with none), which on a small window is
the difference between a usable session and one that overruns; `"none"`
omits the flag, since `--tools none` is a server that exits.
Codex is one persistent `codex app-server --stdio` process per session; its
driver uses native turn steering and interruption, persists the protocol
state and thread id, and reads subscription limits through the same CLI
@@ -325,6 +334,11 @@ written, and the fold uses that same predicate to decide a reply is settled.
## Things that have bitten
- **A server started with no `--tools` answers 403 at `GET /tools`, not an
empty list.** The route is off rather than empty, so reading that as a
failure made "no tools" — the one setting whose entire purpose is to have
none — a session that never started.
- **A llama session reports `loading`, and a message sent into it waits.**
Before 2026-09-19 the session showed `running` from the moment the process
started, so a minute of reading a model off disk was indistinguishable from
PLAN.md
+20
View File
@@ -1435,6 +1435,26 @@ verified by running it, matching dev-updater's posture.
it truncates old KV cache entries, which is silent forgetting with no
summary, and it corrupts the harness's view of what the model knows. Fine
as a server-side safety net; not memory management.
- **What a provider's settings are is declared by the server** (2026-09-19,
`DriverKind::params`). A spec is a key, words, a shape and whether a change
waits for a restart; the phone renders whatever arrives, so a driver that
grows a setting gets a control with no app change. The reason it is
declared rather than drawn is not tidiness: several `llama-server` flags
were hardcoded to what measured best on one machine, which is right as a
default and wrong as a constant — the next machine has a different GPU, and
nobody running this app can edit the source. `POST /sessions/{id}/params`
takes the whole map, so an absent key *is* the instruction to unset.
- **The context figure has a denominator where one can be measured**
(2026-09-19). `Event::ContextWindow` carries it, read from `llama-server`'s
`/props` once the model is up — the measurement rather than the request,
since a session that named no context size gets the model's own. Neither
coding CLI states its window, so those keep the bare figure: "2,042" and
"2,042 / 8,192" are deliberately different-looking, and a missing ceiling is
never drawn as a proportion of an assumed one.
The numerator was also wrong, by the length of the last reply: it was the
prompt alone, so a five-word answer reported 2,042 against a slot holding
2,355. It is the turn's total now, which matches `llama-server`'s own
`n_tokens` exactly.
- **MCP servers are configured in `config.ron`, not from the phone**
(2026-09-19). `mcpServers` on a llama provider, with Exa preset on a newly
discovered one. A phone screen for them is the obvious next step and was
app/androidApp/src/main/kotlin/com/example/aiapp/Api.kt
+67
View File
@@ -204,6 +204,13 @@ data class SessionSummary(
* full.
*/
val contextTokens: Long?,
/**
* What [contextTokens] is out of, or null where this session's provider does not say. A third
* state, not a fourth reading of the same one: the occupancy is known and the ceiling is not.
*/
val contextLimit: Long?,
/** What this session's provider settings are set to; empty where it takes none. */
val params: Map<String, String>,
/**
* The longest edge an image should have when it reaches this session, or null where the
* provider has no limit.
@@ -262,6 +269,8 @@ private fun parseSession(session: JSONObject) =
cwd = session.optString("cwd").ifEmpty { null },
contextTokens =
if (session.has("contextTokens")) session.getLong("contextTokens") else null,
contextLimit = if (session.has("contextLimit")) session.getLong("contextLimit") else null,
params = session.optJSONObject("params").stringMap(),
maxImageEdge = session.optInt("maxImageEdge", 0).takeIf { it > 0 },
usageProvider = session.optString("usageProvider").ifEmpty { null },
status = session.getString("status"),
@@ -344,6 +353,27 @@ data class Provider(
val models: List<String>,
val permissionModes: List<String>,
val defaultPermissionMode: String?,
/** The extra settings this provider takes, in the order to draw them. See [ParamSpec]. */
val params: List<ParamSpec>,
)
/**
* One setting a provider takes, as the server describes it.
*
* Declared by the server rather than drawn here, so a driver that grows a setting gets a control
* with no app change — and, more to the point, so that a value measured on one machine can ship as
* a default without becoming a constant nobody else can reach.
*/
data class ParamSpec(
val key: String,
val label: String,
/** What leaving it blank means, in words, shown as the placeholder. */
val unset: String,
val kind: String,
/** For [kind] `"choice"`: the options, the first of which means "unset". */
val options: List<String>,
/** Whether a change waits for the session's process to start again. */
val restart: Boolean,
)
/**
@@ -359,6 +389,11 @@ data class Machine(
val providers: List<Provider>,
)
/** A JSON object of strings, and the empty map for one that is absent. */
private fun JSONObject?.stringMap(): Map<String, String> =
this?.let { object_ -> object_.keys().asSequence().associateWith { object_.getString(it) } }
?: emptyMap()
private fun parseProvider(provider: JSONObject): Provider {
val kind = provider.getString("kind")
return Provider(
@@ -368,6 +403,17 @@ private fun parseProvider(provider: JSONObject): Provider {
models = provider.optJSONArray("models")?.strings().orEmpty(),
permissionModes = provider.optJSONArray("permissionModes")?.strings().orEmpty(),
defaultPermissionMode = provider.optString("defaultPermissionMode").ifEmpty { null },
params =
provider.optJSONArray("params")?.mapObjects { spec ->
ParamSpec(
key = spec.getString("key"),
label = spec.getString("label"),
unset = spec.getString("unset"),
kind = spec.getString("kind"),
options = spec.optJSONArray("options")?.strings().orEmpty(),
restart = spec.optBoolean("restart"),
)
} ?: emptyList(),
)
}
@@ -481,6 +527,13 @@ data class Importable(
* history from before a compaction, which the model is no longer given.
*/
val contextTokens: Long?,
/**
* What [contextTokens] is out of, or null where this session's provider does not say. A third
* state, not a fourth reading of the same one: the occupancy is known and the ceiling is not.
*/
val contextLimit: Long?,
/** What this session's provider settings are set to; empty where it takes none. */
val params: Map<String, String>,
/** Whether [title] is a name somebody chose rather than the last thing said in the session. */
val named: Boolean,
/**
@@ -555,6 +608,10 @@ fun fetchImportable(settings: ServerSettings, machine: String): List<Importable>
contextTokens =
if (session.isNull("contextTokens")) null
else session.optLong("contextTokens").takeIf { it > 0L },
contextLimit =
if (session.isNull("contextLimit")) null
else session.optLong("contextLimit").takeIf { it > 0L },
params = session.optJSONObject("params").stringMap(),
// Absent means an older backend that cannot answer, which is what "unknown" says.
inUse = session.optString("inUse", "unknown"),
named = session.optBoolean("named", false),
@@ -1240,6 +1297,16 @@ fun setSessionEffort(settings: ServerSettings, sessionId: String, level: String?
}
/** Switches how much a running session asks before acting, also in place. */
/** Replaces a session's provider settings with [params] — the whole map, not a patch. */
fun setSessionParams(settings: ServerSettings, sessionId: String, params: Map<String, String>) {
requestFromServer(
settings,
"/sessions/$sessionId/params",
method = "POST",
jsonBody = JSONObject().put("params", JSONObject(params.toMap())).toString(),
) {}
}
fun setSessionPermissionMode(settings: ServerSettings, sessionId: String, mode: String) {
requestFromServer(
settings,
app/androidApp/src/main/kotlin/com/example/aiapp/Compaction.kt
+20
View File
@@ -60,3 +60,23 @@ fun compactingLabel(seconds: Long?): String =
seconds < 60 -> "compacting ${seconds}s"
else -> "compacting ${seconds / 60}m ${seconds % 60}s"
}
/**
* How full the session is, as the status row says it.
*
* Three states, not two, and the third is the one that needed the words: a session whose occupancy
* is known and whose ceiling is not. That one keeps the bare figure, and a session with a ceiling
* gets both — the reader can see which they are looking at. What must not happen is a missing
* ceiling drawn as a number, or as a proportion of some assumed window, which would be this screen
* inventing the very fact it does not have.
*
* A llama.cpp session always has one, since the window is a flag its own server was started with. A
* coding CLI's is the vendor's business and neither control protocol states it, so those keep the
* bare figure they have always had.
*/
fun contextLabel(held: Long?, limit: Long?): String =
when {
held == null -> "context unknown"
limit == null -> "context ${tokens(held)}"
else -> "context ${tokens(held)} / ${tokens(limit)}"
}
app/androidApp/src/main/kotlin/com/example/aiapp/Events.kt
+19
View File
@@ -163,6 +163,9 @@ sealed class SessionEvent {
*/
data class UsageDelta(val tokens: Long, val context: Long?) : SessionEvent()
/** How much context this session's model has, which is what [UsageDelta.context] is out of. */
data class ContextWindow(val tokens: Long) : SessionEvent()
/**
* A compaction that finished, and how much context it recovered.
*
@@ -293,6 +296,7 @@ fun parseSeqEvent(json: String): SeqEvent {
model = body.optString("model").ifEmpty { null },
permissionMode = body.optString("permissionMode").ifEmpty { null },
)
"contextWindow" -> SessionEvent.ContextWindow(body.getLong("tokens"))
"usageDelta" ->
SessionEvent.UsageDelta(
body.getLong("tokens"),
@@ -363,3 +367,18 @@ fun contextAfter(current: Long?, event: SessionEvent): Long? =
is SessionEvent.Cleared -> null
else -> current
}
/**
* The context window after [event], mirroring the server's `context_limit_after` for the same
* reason [contextAfter] mirrors its neighbour: the screen has to keep up between page loads.
*
* A window belongs to the process, so a session whose process has exited has none — left standing,
* a session restarted on a different model would draw its occupancy against the old model's
* ceiling.
*/
fun contextLimitAfter(current: Long?, event: SessionEvent): Long? =
when (event) {
is SessionEvent.ContextWindow -> event.tokens
is SessionEvent.Status -> if (event.state == "exited") null else current
else -> current
}
app/androidApp/src/main/kotlin/com/example/aiapp/ProviderParams.kt
+117
View File
@@ -0,0 +1,117 @@
package com.example.aiapp
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.text.KeyboardOptions
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.OutlinedTextField
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.ui.Modifier
import androidx.compose.ui.text.input.KeyboardType
import androidx.compose.ui.unit.dp
/**
* The controls for whatever settings a provider says it takes.
*
* One composable for both screens that offer them — the spawn form and the session settings dialog
* — and for every provider, because the server declares the list (see `DriverKind::params`) rather
* than this file knowing it. A driver that grows a setting gets a control here with no change to
* the app, which is the whole point: the values that suit one machine ship as defaults, and every
* one of them stays reachable from a phone.
*
* [values] is the whole map and [onChange] hands back the whole map. A key absent from it means the
* setting is unset, which is what every [ParamSpec.unset] describes — so clearing a field and never
* touching it are deliberately the same state.
*/
@Composable
fun ProviderParamFields(
specs: List<ParamSpec>,
values: Map<String, String>,
onChange: (Map<String, String>) -> Unit,
/**
* Whether to say which settings wait for a restart. False on a spawn form, where nothing is
* running yet and every setting is about to be read — saying it there would be a warning about
* a state the reader cannot be in.
*/
warnAboutRestart: Boolean,
modifier: Modifier = Modifier,
) {
if (specs.isEmpty()) return
Column(modifier.fillMaxWidth()) {
specs.forEach { spec ->
val set = { value: String ->
onChange(
// Blank clears rather than storing an empty string: the server reads an absent
// key as "use the default", and an empty one would be a value it then failed
// to parse.
if (value.isBlank()) values - spec.key else values + (spec.key to value)
)
}
when (spec.kind) {
"choice" -> {
// The first option is what unset means, so selecting it clears the key — see
// `ParamKind::Choice`. Without that the picker could show a default it could
// not return to.
val default = spec.options.firstOrNull().orEmpty()
ChipGroup(
label = spec.label + restartSuffix(spec, warnAboutRestart),
options = spec.options,
selected = values[spec.key] ?: default,
onSelect = { chosen -> set(if (chosen == default) "" else chosen) },
)
}
else ->
OutlinedTextField(
value = values[spec.key].orEmpty(),
onValueChange = set,
label = { Text(spec.label + restartSuffix(spec, warnAboutRestart)) },
placeholder = { Text(spec.unset) },
singleLine = true,
keyboardOptions = KeyboardOptions(keyboardType = keyboardFor(spec.kind)),
modifier = Modifier.fillMaxWidth(),
)
}
Spacer(Modifier.height(16.dp))
}
if (warnAboutRestart && specs.any { it.restart }) {
Text(
"A setting marked “on restart” is saved now and read when this session's process " +
"next starts.",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
}
}
/**
* Marks a control whose value will not take effect yet.
*
* On the label rather than beside it, because the reader decides whether to change the thing before
* they touch it — a note underneath is read after the decision.
*/
private fun restartSuffix(spec: ParamSpec, warn: Boolean): String =
if (warn && spec.restart) " (on restart)" else ""
/**
* The keyboard for a value's shape. A number field that opens the letter keyboard is one every
* entry is made harder by, and these are nearly all numbers.
*/
private fun keyboardFor(kind: String): KeyboardType =
when (kind) {
"integer" -> KeyboardType.Number
"decimal" -> KeyboardType.Decimal
else -> KeyboardType.Text
}
/**
* How long typing has to stop before edited settings are sent.
*
* Long enough that a number is one request rather than one per digit, short enough that closing the
* dialog straight after typing still saves — the save runs on the screen behind it, which outlives
* the dialog, so this delay is not a window the value can be lost in.
*/
const val PARAM_SAVE_DELAY_MS = 700L
app/androidApp/src/main/kotlin/com/example/aiapp/SessionScreen.kt
+43 -1
View File
@@ -233,6 +233,8 @@ fun SessionScreen(
// that was measured.
var contextTokens by
remember(address) { mutableStateOf(if (isSubagent) null else summary.contextTokens) }
var contextLimit by
remember(address) { mutableStateOf(if (isSubagent) null else summary.contextLimit) }
// When the current compaction started. The moment comes off the `compacting` status event
// itself -- the server timestamps every transcript line -- rather than off this device noticing
// one, which is what makes it survive leaving the session and reopening it.
@@ -304,6 +306,14 @@ fun SessionScreen(
// hardcoded list is a claim about a machine.
var offeredModels by remember { mutableStateOf<List<OfferedModel>>(emptyList()) }
var offeredPermissionModes by remember { mutableStateOf<List<String>>(emptyList()) }
// The settings this session's provider takes, and what they are set to. The specs come from
// the provider and the values from the session, because "what can be set" and "what is set"
// are different questions with different answers.
var paramSpecs by remember { mutableStateOf<List<ParamSpec>>(emptyList()) }
var params by remember(summary.id) { mutableStateOf(summary.params) }
// What was last successfully saved, so the debounce below knows whether there is anything to
// send -- and so a failed save can put the controls back to what the server actually holds.
var savedParams by remember(summary.id) { mutableStateOf(summary.params) }
val lifecycleOwner = LocalLifecycleOwner.current
// The resume cursor, written from the stream's IO thread.
val lastSeq = remember { AtomicLong(0) }
@@ -489,6 +499,7 @@ fun SessionScreen(
// Before the rest, and for every event rather than only the usage ones: a compaction and a
// clear move this as much as a turn does. See `contextAfter`.
contextTokens = contextAfter(contextTokens, entry.event)
contextLimit = contextLimitAfter(contextLimit, entry.event)
when (val event = entry.event) {
// Nothing further: what it carries was folded into the context above.
is SessionEvent.UsageDelta -> {}
@@ -1205,6 +1216,30 @@ fun SessionScreen(
fun label(id: String?): String =
offeredModels.firstOrNull { it.id == id }?.label ?: modelLabel(id)
/**
* Saves edited provider settings once the typing stops.
*
* Debounced rather than sent per keystroke, because a text field over the tunnel would be a
* round trip and a config write per character. Here rather than in the settings dialog for the
* reason the delay exists at all: the dialog can be dismissed mid-edit, and this screen
* outlives it, so the last value typed is still saved.
*/
LaunchedEffect(params) {
if (params == savedParams) return@LaunchedEffect
delay(PARAM_SAVE_DELAY_MS)
val wanted = params
try {
withContext(Dispatchers.IO) { setSessionParams(settings, summary.id, wanted) }
savedParams = wanted
actionError = null
} catch (e: ApiException) {
// Back to what the server holds. A control left showing a value that was refused is
// stating something untrue about the session.
params = savedParams
actionError = e.message
}
}
// Only for the model picker, which a subagent does not have.
if (!isSubagent) {
LaunchedEffect(summary.machine, summary.provider) {
@@ -1218,6 +1253,7 @@ fun SessionScreen(
}
.getOrNull()
offeredPermissionModes = provider?.permissionModes.orEmpty()
paramSpecs = provider?.params.orEmpty()
offeredModels =
provider
?.let {
@@ -1916,6 +1952,7 @@ fun SessionScreen(
status = status,
compactingFor = compactingFor,
contextTokens = contextTokens,
contextLimit = contextLimit,
backgroundTasks = backgroundTasks,
subagent = isSubagent,
)
@@ -2197,6 +2234,9 @@ fun SessionScreen(
effort = effort.takeIf { summary.takesEffort },
takesEffort = summary.takesEffort,
onEffortChanged = { effort = it },
paramSpecs = paramSpecs,
params = params,
onParamsChanged = { params = it },
cachedBytes = cachedBytes,
// The purge finishes before the epoch moves, because the relaunched opening effect
// reads the same directory and would otherwise draw what is about to be deleted. The
@@ -2473,6 +2513,8 @@ private fun SessionStatusRow(
compactingFor: Long?,
/** Context the session is holding, or null where nothing has measured it. */
contextTokens: Long?,
/** What that is out of, or null where the provider does not say. */
contextLimit: Long?,
/** Provider-reported live background work; zero is deliberately not drawn. */
backgroundTasks: Int,
modifier: Modifier = Modifier,
@@ -2577,7 +2619,7 @@ private fun SessionStatusRow(
// reports usage, and one that has not run a turn all showed nothing at all, which reads as
// a conversation with room to spare.
Text(
contextTokens?.let { "context ${tokens(it)}" } ?: "context unknown",
contextLabel(contextTokens, contextLimit),
style = MaterialTheme.typography.labelSmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
app/androidApp/src/main/kotlin/com/example/aiapp/SessionSettingsDialog.kt
+26
View File
@@ -74,6 +74,15 @@ fun SessionSettingsDialog(
onEffortChanged: (String?) -> Unit,
/** Whether a level does anything here; the row is left out entirely where it does not. */
takesEffort: Boolean,
/**
* The settings this session's provider takes, and what they are set to.
*
* Declared by the server rather than listed here -- see [ProviderParamFields]. Empty for a
* provider with none, which draws no section at all.
*/
paramSpecs: List<ParamSpec>,
params: Map<String, String>,
onParamsChanged: (Map<String, String>) -> Unit,
/**
* What this phone is holding of the conversation, or null while that is being measured -- see
* the Reload row below, which is what would discard it.
@@ -451,6 +460,23 @@ fun SessionSettingsDialog(
)
}
}
if (paramSpecs.isNotEmpty()) {
Spacer(Modifier.height(16.dp))
Text(
"Model settings",
style = MaterialTheme.typography.titleSmall,
)
Spacer(Modifier.height(8.dp))
// Edited here and saved by the screen behind this, which is what makes
// typing in a text field affordable: the save is debounced, and a dialog
// dismissed mid-edit would take an unsaved value with it.
ProviderParamFields(
specs = paramSpecs,
values = params,
onChange = onParamsChanged,
warnAboutRestart = true,
)
}
Spacer(Modifier.height(8.dp))
Row(
verticalAlignment = Alignment.CenterVertically,
app/androidApp/src/main/kotlin/com/example/aiapp/SpawnScreen.kt
+18 -67
View File
@@ -73,13 +73,9 @@ fun SpawnScreen(
// Only the spawn's own failure. The fetch's lives in `options`: this one leaves a filled-in
// form worth keeping, and that one leaves nothing to fill in.
var spawnError by remember { mutableStateOf<String?>(null) }
var contextSize by remember { mutableStateOf("") }
var temperature by remember { mutableStateOf("") }
// Whether a model that carries a multi-token-prediction head drafts with it. Left to the
// server by default, which turns it on exactly where the file has one -- see `SPECULATIVE` in
// the llama driver. Here so a machine where drafting does not pay has a way out that is not
// an edit to config.ron.
var speculative by remember { mutableStateOf(SPECULATIVE_AUTO) }
// Whatever the chosen provider says it takes, by key. Empty until something is typed: an
// absent key means the server's own default, which is what every field's placeholder says.
var params by remember { mutableStateOf<Map<String, String>>(emptyMap()) }
LaunchedEffect(Unit) {
// Separate from the machines fetch below and deliberately not fatal: failing to learn the
@@ -145,6 +141,9 @@ fun SpawnScreen(
// machine looks exactly like one from this one.
LaunchedEffect(machine?.id, current?.name) {
model = ""
// A key from the previous provider would be a setting this one does not have, drawn
// by no control and sent at the spawn anyway.
params = emptyMap()
providerModels = emptyList()
providerModelsError = null
permissionMode = current?.defaultPermissionMode.orEmpty()
@@ -283,35 +282,14 @@ fun SpawnScreen(
Spacer(Modifier.height(16.dp))
}
if (isLlama) {
OutlinedTextField(
value = contextSize,
onValueChange = { contextSize = it },
label = { Text("Context size (blank = the model's default)") },
singleLine = true,
modifier = Modifier.fillMaxWidth(),
)
Spacer(Modifier.height(16.dp))
OutlinedTextField(
value = temperature,
onValueChange = { temperature = it },
label = { Text("Temperature (blank = llama.cpp's default)") },
singleLine = true,
modifier = Modifier.fillMaxWidth(),
)
Spacer(Modifier.height(16.dp))
// Said as what it is rather than as "MTP": the reader is choosing whether the session
// goes faster, and most models have nothing to turn on here at all.
ChipGroup(
label = "Speculative decoding (models that carry a draft head)",
options = listOf(SPECULATIVE_AUTO, SPECULATIVE_OFF),
selected = speculative,
onSelect = { speculative = it },
)
Spacer(Modifier.height(16.dp))
}
// Nothing is running yet, so nothing here waits for a restart -- every one of these is
// read by the process this form is about to start.
ProviderParamFields(
specs = current?.params.orEmpty(),
values = params,
onChange = { params = it },
warnAboutRestart = false,
)
// Every session whose tools act on files needs one, which is both kinds that have
// tools -- a llama session's built-in tools run in it exactly as a CLI's do.
@@ -387,28 +365,10 @@ fun SpawnScreen(
cwd = cwd.trim().takeIf { takesCwd },
permissionMode = permissionMode.takeIf { it.isNotEmpty() },
effort = effort.takeIf { isCodingCli },
// Sent only when set, so blank means "whatever llama.cpp does
// by default" rather than a zero.
params =
buildMap {
if (isLlama) {
contextSize
.trim()
.takeIf { it.isNotEmpty() }
?.let { put("contextSize", it) }
temperature
.trim()
.takeIf { it.isNotEmpty() }
?.let { put("temperature", it) }
// Only the choice that changes anything: "auto"
// is the absence of the setting, not a value of
// it, so a session spawned without an opinion
// carries none.
if (speculative == SPECULATIVE_OFF) {
put("speculative", "off")
}
}
},
// Already only the keys somebody set: a field left blank
// removes its key rather than sending an empty value, so
// "blank" reaches the server as "your default".
params = params,
)
}
onSpawned(spawned)
@@ -426,15 +386,6 @@ fun SpawnScreen(
}
}
/**
* Leave the draft head to the server, which uses one wherever the model file has one. Spelled the
* same as the absence of the `speculative` parameter, because that is what it means.
*/
private const val SPECULATIVE_AUTO = "auto"
/** The `speculative` parameter's only other value; see the llama driver's `SPECULATIVE`. */
private const val SPECULATIVE_OFF = "off"
/**
* A labeled row of choices that wraps onto as many lines as it needs.
*
app/androidApp/src/main/kotlin/com/example/aiapp/TranscriptItems.kt
+1
View File
@@ -555,6 +555,7 @@ fun foldEvent(items: List<TranscriptItem>, entry: SeqEvent): List<TranscriptItem
is SessionEvent.RetiredTaskNote -> items
// Screen-level state, not transcript rows -- see SessionScreen.
is SessionEvent.UsageDelta -> items
is SessionEvent.ContextWindow -> items
}
/**
app/androidApp/src/test/kotlin/com/example/aiapp/SessionOrderTest.kt
+2
View File
@@ -75,6 +75,8 @@ class SessionOrderTest {
resumeAt = null,
cwd = null,
contextTokens = null,
contextLimit = null,
params = emptyMap(),
maxImageEdge = null,
usageProvider = null,
status = status,
server/src/config.rs
+150
View File
@@ -312,6 +312,28 @@ impl DriverKind {
}
}
/// The settings this kind of session takes beyond the shared ones, for
/// the phone to offer.
///
/// Declared rather than drawn: a spawn screen with a field per llama
/// setting is a screen that has to be edited every time a driver grows
/// one, and this app already has the `params` map to carry them. So the
/// server says what a provider takes and the phone renders it, which is
/// the same arrangement `permission_modes` uses and for the same reason
/// -- the alternative is two lists that disagree, one of them in Kotlin.
///
/// It is also what keeps these *reachable at all*. Several were hardcoded
/// to the values measured on one machine, which is fine as a default and
/// wrong as a constant: the next machine has a different GPU and a
/// different number of cores, and nobody running this app can edit the
/// source.
pub fn params(self) -> &'static [ParamSpec] {
match self {
Self::LlamaCpp => LLAMA_PARAMS,
Self::Echo | Self::ClaudeCli | Self::CodexCli => &[],
}
}
/// The mode used when a new-session form first selects this kind.
pub fn default_permission_mode(self) -> Option<&'static str> {
match self {
@@ -323,6 +345,134 @@ impl DriverKind {
}
}
/// One setting a provider takes, and enough about it to draw a control.
///
/// Deliberately thin: a key, words for a person, and which shape the value
/// has. Anything richer -- units, validation, dependencies between settings --
/// would be a schema language, and what the phone needs is a text field or a
/// row of chips.
#[derive(Debug, Clone, Copy, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct ParamSpec {
/// The `SessionConfig::params` key this writes.
pub key: &'static str,
pub label: &'static str,
/// What happens when it is not set, in words. Shown where a control shows
/// its placeholder, so "blank" always means something specific rather than
/// leaving the reader to guess whether it means zero.
pub unset: &'static str,
/// Flattened, so a spec is one flat object: `kind` beside the rest rather
/// than an object of its own with `kind` inside it.
#[serde(flatten)]
pub kind: ParamKind,
/// Whether changing it waits for the process to start again.
///
/// The honest half of offering these live. A sampling setting rides on the
/// next request; a server flag was decided when the model was loaded, and
/// a control that silently did nothing until some later restart would be
/// worse than one that is not there.
pub restart: bool,
}
/// What shape a [`ParamSpec`]'s value has.
#[derive(Debug, Clone, Copy, Serialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum ParamKind {
Integer,
Decimal,
Text,
/// A fixed set. **The first option is what leaving it unset means**, and
/// choosing it clears the setting rather than storing a value -- so the
/// default is a state the picker can return to, and the stored config
/// does not fill up with values nobody chose.
Choice {
options: &'static [&'static str],
},
}
/// What a llama.cpp session takes.
///
/// The server flags first, in the order they matter, then the sampling ones --
/// which is also the order of how disruptive changing one is.
const LLAMA_PARAMS: &[ParamSpec] = &[
ParamSpec {
key: "contextSize",
label: "Context size",
unset: "the model's own trained context",
kind: ParamKind::Integer,
restart: true,
},
ParamSpec {
key: "tools",
label: "Tools",
// Worth a control rather than a constant because of what it costs:
// the definitions of all seven are ~2,000 tokens of the context,
// every turn, before anything is said. On a small window that is the
// difference between a usable session and one that overruns.
unset: "all of them -- or a comma-separated list, or \"none\"",
kind: ParamKind::Text,
restart: true,
},
ParamSpec {
key: "gpuLayers",
label: "Layers on the GPU",
unset: "as many as fit",
kind: ParamKind::Integer,
restart: true,
},
ParamSpec {
key: "threads",
label: "Threads",
unset: "one per core",
kind: ParamKind::Integer,
restart: true,
},
ParamSpec {
key: "speculative",
label: "Speculative decoding",
unset: "on, for a model whose file carries a draft head",
kind: ParamKind::Choice {
options: &["auto", "off"],
},
restart: true,
},
ParamSpec {
key: "specDraftNMax",
label: "Tokens drafted ahead",
unset: "llama.cpp's own default",
kind: ParamKind::Integer,
restart: true,
},
ParamSpec {
key: "temperature",
label: "Temperature",
unset: "llama.cpp's default",
kind: ParamKind::Decimal,
restart: false,
},
ParamSpec {
key: "topP",
label: "Top P",
unset: "llama.cpp's default",
kind: ParamKind::Decimal,
restart: false,
},
ParamSpec {
key: "topK",
label: "Top K",
unset: "llama.cpp's default",
kind: ParamKind::Integer,
restart: false,
},
ParamSpec {
key: "maxTokens",
label: "Reply limit",
unset: "no limit",
kind: ParamKind::Integer,
restart: false,
},
];
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct TokenEntry {
server/src/routes.rs
+28
View File
@@ -54,6 +54,8 @@
//! which starts again in the new one
//! POST /sessions/{id}/model {model}
//! POST /sessions/{id}/permission-mode {permissionMode}
//! POST /sessions/{id}/params {params} -- the provider settings, whole;
//! what a provider takes is on its ProviderInfo
//! POST /sessions/{id}/effort {effort} -- null for the CLI's default;
//! settled at launch, so this stops the process
//! POST /sessions/{id}/command {text} -- /compact, /clear, /rename x, or the dialect's own
@@ -166,6 +168,7 @@ pub fn router(manager: Arc<SessionManager>) -> Router {
.route("/sessions/{id}/cwd", post(set_cwd))
.route("/sessions/{id}/model", post(set_model))
.route("/sessions/{id}/permission-mode", post(set_permission_mode))
.route("/sessions/{id}/params", post(set_params))
.route("/sessions/{id}/effort", post(set_effort))
.route("/defaults", get(defaults).post(set_defaults))
.route("/sessions/{id}/notify", post(set_notify))
@@ -300,6 +303,11 @@ struct ProviderInfo {
permission_modes: Vec<&'static str>,
#[serde(skip_serializing_if = "Option::is_none")]
default_permission_mode: Option<&'static str>,
/// The per-kind settings a spawn form and the session settings dialog
/// offer -- see [`crate::config::DriverKind::params`]. Empty for a
/// provider with none, which draws no section at all.
#[serde(skip_serializing_if = "<[_]>::is_empty")]
params: &'static [crate::config::ParamSpec],
}
async fn list_machines(State(manager): State<Arc<SessionManager>>) -> axum::Json<Vec<MachineInfo>> {
@@ -320,6 +328,7 @@ fn info_for(machine: crate::config::MachineConfig) -> MachineInfo {
models: provider.models,
permission_modes: provider.kind.permission_modes().to_vec(),
default_permission_mode: provider.kind.default_permission_mode(),
params: provider.kind.params(),
})
.collect(),
}
@@ -426,6 +435,7 @@ async fn probe_machine(
models: provider.models,
permission_modes: provider.kind.permission_modes().to_vec(),
default_permission_mode: provider.kind.default_permission_mode(),
params: provider.kind.params(),
})
.collect(),
))
@@ -1586,6 +1596,24 @@ async fn set_permission_mode(
Ok(StatusCode::NO_CONTENT)
}
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct ParamsRequest {
/// The whole map, not a patch -- see `SessionManager::set_session_params`.
params: std::collections::BTreeMap<String, String>,
}
async fn set_params(
State(manager): State<Arc<SessionManager>>,
UrlPath(id): UrlPath<String>,
axum::Json(body): axum::Json<ParamsRequest>,
) -> Result<StatusCode, ApiError> {
manager
.set_session_params(&id, body.params)
.map_err(bad_request)?;
Ok(StatusCode::NO_CONTENT)
}
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct NotifyRequest {
server/src/session/driver.rs
+49
View File
@@ -363,6 +363,26 @@ pub enum Event {
#[serde(default, skip_serializing_if = "Option::is_none")]
permission_mode: Option<String>,
},
/// How much context this session's model has to hold a conversation in.
///
/// The denominator the phone draws [`Event::UsageDelta`]'s `context`
/// against, and its own event rather than a field on that one because it
/// is not a per-turn measurement: it is fixed when the process starts and
/// changes only when a different one is started, which is what a model
/// change does. Reported the moment it is known, so the figure and what it
/// is out of arrive together rather than the first turn drawing a
/// numerator with no denominator.
///
/// **Only ever sent by a driver that actually knows.** A window nobody has
/// measured is not an unlimited one: llama.cpp answers it exactly, because
/// the number is a flag the server was started with and `/props` reads it
/// back, while a coding CLI's context is the vendor's business and
/// nothing in either control protocol states it. Those send nothing, the
/// session has no limit, and the phone draws the figure on its own -- see
/// `SessionSummary::context_limit`.
ContextWindow {
tokens: u64,
},
/// Per-turn token counts, where the dialect reports them.
UsageDelta {
/// What this turn cost: the tokens it was charged for.
@@ -494,6 +514,25 @@ pub fn context_tokens(input: u64, cache_creation: u64, cache_read: u64) -> u64 {
input + cache_creation + cache_read
}
/// The context window after `event`, given what it was before.
///
/// Beside [`context_after`] because it has the same three readers and the same
/// hazard: a figure that outlives what made it true. A model change replaces
/// the process, so the window it reports replaces the old one -- and until the
/// new one says, there is no answer rather than the previous model's.
pub fn context_limit_after(current: Option<u64>, event: &Event) -> Option<u64> {
match event {
Event::ContextWindow { tokens } => Some(*tokens),
// The window belongs to the process, and a stopped one has none. Left
// standing, a restarted session on a different model would draw its
// occupancy against the previous model's window.
Event::Status {
state: SessionStatus::Exited,
} => None,
_ => current,
}
}
/// The context after `event`, given what it was before.
///
/// The whole rule in one place, because three readers need the same answer:
@@ -683,6 +722,16 @@ pub trait Driver: Send + Sync {
/// `/rename` afterwards, which is what puts the same name in its own
/// session picker and in what other agents see.
fn set_title(&self, title: &str);
/// Takes the session's provider settings, whole.
///
/// The whole map because it is a form's contents -- an absent key means
/// "unset", not "unchanged". A driver applies what it can apply now and
/// says so about the rest: the map is also on disk by the time this is
/// called, so a setting that only takes effect at the next start is not
/// lost, it is waiting. The default is right for a driver with no settings
/// of its own, which is every one but llama.cpp -- see
/// [`crate::config::DriverKind::params`].
fn set_params(&self, _params: &std::collections::BTreeMap<String, String>) {}
/// Runs a command this session's own dialect understands, verbatim --
/// `/context`, `/usage`, anything a CLI adds next month. A driver with no
/// such vocabulary says so with an [`Event::Error`] rather than sending it
server/src/session/llama/mod.rs
+159 -35
View File
@@ -65,6 +65,32 @@ pub use tools::{DEFAULT_MODE as DEFAULT_PERMISSION_MODE, MODES as PERMISSION_MOD
use mcp::McpServer;
use tools::Tools;
/// The sampling half of a session's settings, in the wire's own names.
///
/// One function because two callers need the identical mapping: the spawn, and
/// a later change through [`Driver::set_params`]. A value that will not parse
/// as a number is left out rather than passed through -- `llama-server` would
/// refuse the whole request for it, which would look like the session breaking
/// rather than like one field being wrong.
fn sampling_from(
params: &std::collections::BTreeMap<String, String>,
) -> serde_json::Map<String, Value> {
let mut sampling = serde_json::Map::new();
for (key, field) in [
("temperature", "temperature"),
("topP", "top_p"),
("topK", "top_k"),
("maxTokens", "max_tokens"),
] {
if let Some(raw) = params.get(key)
&& let Ok(number) = raw.parse::<f64>()
{
sampling.insert(field.to_string(), json!(number));
}
}
sampling
}
/// Exa's own hosted MCP server, which is what a llama session searches the web
/// with. The address llama.cpp's web UI offers under "Exa" in its recommended
/// servers, so a session here reaches the same thing that UI does.
@@ -76,6 +102,11 @@ pub const EXA_MCP_URL: &str = "https://mcp.exa.ai/mcp";
/// is the escape for a machine where drafting turns out not to pay.
const SPECULATIVE: &str = "speculative";
/// The spawn parameter naming which built-in tools a session gets, as
/// `llama-server`'s own comma-separated list. Absent is all of them, and
/// `"none"` is the way to ask for a session that only talks.
const TOOLS: &str = "tools";
/// How many times one message may go round the call-a-tool loop.
///
/// A bound rather than a budget: a small model that has decided to read the
@@ -221,8 +252,10 @@ struct Shared {
/// Where this session's process record lives, so [`Driver::stop`] can find
/// the server it has to end.
session_dir: PathBuf,
/// Sampling settings chosen at spawn, sent with every request.
sampling: serde_json::Map<String, Value>,
/// Sampling settings, sent with every request. Behind a lock because they
/// are changeable while the session runs: they ride on the next request,
/// so unlike the server's own flags there is nothing to reload.
sampling: Mutex<serde_json::Map<String, Value>>,
/// The session's working directory, which is where its tools act. `None`
/// leaves that to `llama-server`, which is the honest answer rather than a
/// guess at one.
@@ -289,26 +322,14 @@ impl LlamaDriver {
"a llama.cpp session needs a model -- one of the downloaded ones, by its key",
)?;
let mut sampling = serde_json::Map::new();
for (key, field) in [
("temperature", "temperature"),
("topP", "top_p"),
("topK", "top_k"),
("maxTokens", "max_tokens"),
] {
if let Some(raw) = meta.params.get(key)
&& let Ok(number) = raw.parse::<f64>()
{
sampling.insert(field.to_string(), json!(number));
}
}
let sampling = sampling_from(&meta.params);
let driver = Self {
shared: Arc::new(Shared {
sink,
transcript: transcript.to_path_buf(),
session_dir: session_dir.to_path_buf(),
sampling,
sampling: Mutex::new(sampling),
cwd: meta
.cwd
.as_ref()
@@ -403,6 +424,15 @@ impl LlamaDriver {
"{model} loaded and answering at {endpoint} with {} tools",
tools.offered().map_or(0, |offered| offered.len()),
);
// Asked now rather than carried from the spawn flags: a
// session that named no context size gets the model's
// own, which only the loaded server knows, and one that
// named an impossible one gets whatever it settled for.
// Either way this is the measurement rather than the
// request.
if let Some(window) = context_window(&endpoint) {
shared.emit(Event::ContextWindow { tokens: window });
}
Serving::Ready {
endpoint: endpoint.clone(),
tools: Arc::new(tools),
@@ -531,14 +561,12 @@ fn spawn_server(
"127.0.0.1".into(),
"--port".into(),
forward.there.to_string(),
// The built-in agent tools -- read, search, edit, shell. Every one of
// them, because a session offered a subset is a session that says "I
// can't do that" about something it was installed to do, and the
// question of whether a particular call should happen is the
// permission gate's rather than a flag's. They run on the machine
// serving the model, which is the machine the files are on.
"--tools".into(),
"all".into(),
// The built-in agent tools -- read, search, edit, shell. All of them
// unless the session says otherwise, because whether a particular call
// should happen is the permission gate's question rather than a flag's.
// They run on the machine serving the model, which is the machine the
// files are on.
//
// One slot, not the four `llama-server` picks on its own. A session is
// one conversation making one request at a time -- the driver holds a
// second message until the turn ends -- so the other three are context
@@ -553,6 +581,27 @@ fn spawn_server(
"-np".into(),
"1".into(),
];
// Settable because it is not free: the definitions of all seven are around
// 2,000 tokens of every prompt -- measured at 2,191 against 1,322 for two
// of them -- which on a small context window is a quarter of it spent
// before anything is said.
//
// "none" omits the flag rather than passing it on: `--tools none` is
// `tools setup failed: unknown tool "none"` and a server that exits, since
// the argument is a list of tool names and no-tools is what having no flag
// means.
match meta.params.get(TOOLS).map(|chosen| chosen.trim()) {
Some("none") => {}
chosen => {
args.push("--tools".into());
args.push(
chosen
.filter(|c| !c.is_empty())
.unwrap_or("all")
.to_string(),
);
}
}
// A model that carries a multi-token-prediction head drafts with it, which
// is most of a 50% speed-up for free -- the tensors are in the file
// whether or not they are used, and without the flag `llama-server` says
@@ -573,6 +622,11 @@ fn spawn_server(
("contextSize", "-c"),
("gpuLayers", "-ngl"),
("threads", "-t"),
// How far ahead the draft head guesses. Not defaulted here: 2 measured
// 7% faster than llama.cpp's 3 on this machine's GPU, once, which is
// a reason to make the knob reachable and not a reason to move it for
// everybody.
("specDraftNMax", "--spec-draft-n-max"),
] {
if let Some(value) = meta.params.get(key) {
args.push(flag.to_string());
@@ -854,7 +908,10 @@ fn converse(
if shared.cancel.load(Ordering::SeqCst) {
return Ok(());
}
let reply = generate(endpoint, &messages, tools, &shared.sampling, shared)?;
// Read per call rather than per turn, so a sampling change made while
// a long turn is running reaches the rest of it.
let sampling = shared.sampling.lock().unwrap().clone();
let reply = generate(endpoint, &messages, tools, &sampling, shared)?;
let calls = reply.calls;
messages.push(Message {
tool_calls: calls.iter().map(Call::wire).collect(),
@@ -1046,6 +1103,49 @@ impl Driver for LlamaDriver {
// is called, and the rename has already happened where the name lives.
fn set_title(&self, _title: &str) {}
/// Takes new settings: the sampling half now, and says so about the rest.
///
/// The split is what [`crate::config::ParamSpec::restart`] describes, and
/// it is said out loud rather than left to the screen, because the screen
/// can only say what a setting *usually* does -- this is the one place
/// that knows whether this session's server was started with the old
/// value. A session already stopped needs no such note: its next start
/// will read all of them.
fn set_params(&self, params: &std::collections::BTreeMap<String, String>) {
*self.shared.sampling.lock().unwrap() = sampling_from(params);
// Only the settings that actually differ from what this session's
// server was started with. Listing every restart-only one on every
// save would be a wall of text about nothing having changed.
let waiting: Vec<&str> = crate::config::DriverKind::LlamaCpp
.params()
.iter()
.filter(|spec| {
spec.restart && params.get(spec.key) != self.respawn.meta.params.get(spec.key)
})
.map(|spec| spec.label)
.collect();
// Nothing to say to a session with no server: its next start reads all
// of them, which is what the note would have been asking for.
let running = matches!(&*self.shared.serving.lock().unwrap(), Serving::Ready { .. });
if !waiting.is_empty() && running {
let one = waiting.len() == 1;
self.shared.emit(Event::Error {
message: format!(
"{} {} saved. {} when this session's server next starts -- stop and start \
the session, or change its model, to load {} now.",
waiting.join(", "),
if one { "is" } else { "are" },
if one {
"It takes effect"
} else {
"They take effect"
},
if one { "it" } else { "them" },
),
});
}
}
fn set_permission_mode(&self, mode: &str) {
if !tools::MODES.contains(&mode) {
self.shared.emit(Event::Error {
@@ -1436,6 +1536,23 @@ fn log_tail(session_dir: &Path) -> String {
/// How much of that log to carry into a message somebody reads on a phone.
const LOG_TAIL_LINES: usize = 6;
/// How many tokens this server can hold, from the server itself.
///
/// `/props` reports the per-slot context, which is the whole of it because
/// this driver always starts one slot -- see the `-np` argument. `None` for a
/// server that would not answer, which draws as no ceiling rather than as a
/// guessed one.
fn context_window(endpoint: &str) -> Option<u64> {
ureq::get(format!("{endpoint}/props"))
.call()
.ok()?
.body_mut()
.read_json::<Value>()
.ok()?
.pointer("/default_generation_settings/n_ctx")
.and_then(Value::as_u64)
}
/// One streamed completion: posts the conversation, emits each text delta as
/// it arrives, and assembles whatever tool calls came with it.
///
@@ -1491,9 +1608,16 @@ fn generate(
// at once interleaves their fragments, each tagged with its index.
let mut calls: Vec<Call> = Vec::new();
let mut tokens = 0u64;
// The prompt side only, which is what the model is holding -- the same
// definition the other dialects report, so one word on the phone means one
// thing whichever kind of session it is.
// What the model is holding when this call ends: the prompt it was given
// plus the reply it produced, which is exactly what the next call's prompt
// begins with.
//
// The prompt alone was wrong, and measurably: a five-word reply reported
// 2,042 against a slot holding 2,355 (2026-09-19, checked against
// `stop processing: n_tokens` in the server's own log). The gap is the
// reply, so it grows with how much the model just said -- which is the
// worst direction for a figure somebody is watching to see how much room
// is left.
let mut context = None;
for line in std::io::BufRead::lines(reader) {
if shared.cancel.load(Ordering::SeqCst) {
@@ -1510,13 +1634,13 @@ fn generate(
let Ok(chunk) = serde_json::from_str::<Value>(payload) else {
continue;
};
if let Some(usage) = chunk.get("usage") {
if let Some(total) = usage.get("total_tokens").and_then(Value::as_u64) {
tokens = total;
}
if let Some(prompt) = usage.get("prompt_tokens").and_then(Value::as_u64) {
context = Some(prompt);
}
// One figure answering both questions, which for this dialect it
// genuinely does: what the call was charged for and what the model is
// left holding are the same tokens, because nothing here is billed and
// the whole conversation is resent every time.
if let Some(total) = chunk.pointer("/usage/total_tokens").and_then(Value::as_u64) {
tokens = total;
context = Some(total);
}
let Some(delta) = chunk.pointer("/choices/0/delta") else {
continue;
server/src/session/llama/tools.rs
+21 -10
View File
@@ -57,18 +57,29 @@ impl Tools {
/// Asks a ready `llama-server` what it offers and adds what the MCP
/// servers offered.
///
/// A server started without `--tools` answers with an empty list, and a
/// session with only MCP tools is a perfectly good session -- so nothing
/// here treats "no tools" as a failure. What *is* a failure is not being
/// able to ask, because that is the same server the conversation is about
/// to go to.
/// A session with no built-in tools -- or none at all -- is a perfectly
/// good session, so nothing here treats "no tools" as a failure. What *is*
/// a failure is not being able to ask at all, because that is the same
/// server the conversation is about to go to.
pub fn discover(endpoint: &str, mcp: Vec<Arc<Mutex<McpServer>>>) -> Result<Self> {
let catalog: Vec<Value> = ureq::get(format!("{endpoint}/tools"))
let mut response = ureq::get(format!("{endpoint}/tools"))
// A server started with no `--tools` answers **403** here, not an
// empty list -- the route is off rather than empty. Read as a
// failure that was a session which never started, for the one
// setting whose whole purpose is to have no tools.
.config()
.http_status_as_error(false)
.build()
.call()
.context("asking llama-server which tools it has")?
.body_mut()
.read_json()
.context("reading llama-server's tool list")?;
.context("asking llama-server which tools it has")?;
let catalog: Vec<Value> = if response.status().is_success() {
response
.body_mut()
.read_json()
.context("reading llama-server's tool list")?
} else {
Vec::new()
};
let mut definitions = Vec::new();
let mut server = HashMap::new();
for entry in &catalog {
server/src/session/mod.rs
+63 -1
View File
@@ -36,7 +36,8 @@ use crate::config::{
use claude::ClaudeDriver;
use codex::CodexDriver;
use driver::{
AttachmentRef, Driver, Event, EventSink, SessionCommand, SessionStatus, Unqueued, context_after,
AttachmentRef, Driver, Event, EventSink, SessionCommand, SessionStatus, Unqueued,
context_after, context_limit_after,
};
use echo::EchoDriver;
use llama::LlamaDriver;
@@ -213,6 +214,17 @@ pub struct SessionInfo {
/// are different answers and the phone draws them differently.
#[serde(skip_serializing_if = "Option::is_none")]
pub context_tokens: Option<u64>,
/// What that figure is out of -- see [`Event::ContextWindow`]. Absent
/// where the provider does not say, which is a third state again: not a
/// session with room to spare, and not one whose occupancy is unknown,
/// but one whose occupancy is known and whose ceiling is not.
#[serde(skip_serializing_if = "Option::is_none")]
pub context_limit: Option<u64>,
/// The provider settings this session was launched with, as the settings
/// dialog has to open on them -- what a control is *set to* is not
/// derivable from what the provider *offers*.
#[serde(skip_serializing_if = "std::collections::BTreeMap::is_empty")]
pub params: std::collections::BTreeMap<String, String>,
/// The longest edge an image should have by the time it gets here.
/// Absent rather than a large number, because "no limit" and "a limit
/// that happens to be big" are different answers.
@@ -415,6 +427,7 @@ struct Shared {
/// the session row so a phone opening a long conversation has the real
/// figure rather than whatever its newest page mentions.
context_tokens: Mutex<Option<u64>>,
context_limit: Mutex<Option<u64>>,
/// Mirrored out of the config so the pump can read it without taking
/// the manager's lock -- the pump runs underneath the manager, and
/// reaching back up would invert that.
@@ -579,6 +592,11 @@ impl LiveSession {
effort: current.effort.clone(),
takes_effort: kind.is_some_and(DriverKind::takes_effort),
context_tokens: *self.shared.context_tokens.lock().unwrap(),
context_limit: *self.shared.context_limit.lock().unwrap(),
// From the config for the reason `effort` above is: the settings
// the process was started with are what a restart-only control has
// to show, and this is where they are kept.
params: current.params.clone(),
notify: *self.shared.notify.lock().unwrap(),
auto_resume: current.auto_resume,
auto_resume_message: resume_message(current),
@@ -1104,6 +1122,8 @@ impl SessionManager {
takes_effort: kind_of(&inner.config, &meta.machine, &meta.provider)
.is_some_and(DriverKind::takes_effort),
context_tokens: None,
context_limit: None,
params: meta.params.clone(),
max_image_edge: kind_of(&inner.config, &meta.machine, &meta.provider)
.and_then(DriverKind::max_image_edge),
usage_provider: kind_of(&inner.config, &meta.machine, &meta.provider)
@@ -1319,6 +1339,43 @@ impl SessionManager {
Ok(())
}
/// Changes this session's provider settings, live and persisted.
///
/// The whole map rather than one key, because that is what a settings
/// screen has: a form is submitted as its contents, and merging one field
/// at a time would make clearing a field indistinguishable from not
/// mentioning it. An absent key *is* the instruction to unset it.
///
/// Persisted first for the reason the model is: the config answers what to
/// launch with next time, which is the whole of what a restart-only
/// setting means. The driver is then told, and takes what it can use now.
pub fn set_session_params(
&self,
id: &str,
params: std::collections::BTreeMap<String, String>,
) -> Result<()> {
let mut inner = self.inner.write().unwrap();
if !inner.config.sessions.iter().any(|meta| meta.id == id) {
bail!("no session {id}");
}
let mut candidate = inner.config.clone();
for meta in candidate.sessions.iter_mut().filter(|meta| meta.id == id) {
meta.params = params.clone();
}
candidate.save(&self.config_path)?;
inner.config = candidate;
// Told to the driver where there is one, and simply saved where there
// is not. Deliberately not `ask`: a session with no process has not
// failed to take these, it has taken them in the only way that matters
// -- its next start reads them -- and adjusting the settings of a
// stopped session so that starting it uses them is the ordinary thing
// to do, not an error to report.
if let Some(driver) = inner.live.get(id).and_then(|session| session.driver()) {
driver.set_params(&params);
}
Ok(())
}
/// Turns this session's notifications on or off, live and persisted --
/// both, or the switch moves back on its own at the next restart.
///
@@ -2394,6 +2451,7 @@ fn launch(
model: Mutex::new(meta.model.clone()),
permission_mode: Mutex::new(meta.permission_mode.clone()),
context_tokens: Mutex::new(transcript.context_tokens()),
context_limit: Mutex::new(transcript.context_limit()),
notify: Mutex::new(meta.notify),
written: Mutex::new(0),
});
@@ -2674,6 +2732,10 @@ async fn pump(
let mut context = shared.context_tokens.lock().unwrap();
*context = context_after(*context, &event);
}
{
let mut limit = shared.context_limit.lock().unwrap();
*limit = context_limit_after(*limit, &event);
}
// Nothing changed, so there is nothing to record. Both of these
// repeat: an imported session reads the turn state off its file's
// newest record on every sync, and the CLI restates its model and
server/src/session/transcript.rs
+14 -1
View File
@@ -15,7 +15,7 @@ use std::path::Path;
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use super::driver::{Event, SessionStatus, context_after};
use super::driver::{Event, SessionStatus, context_after, context_limit_after};
/// One transcript line: an [`Event`] plus its position and time. The event
/// is flattened so the wire shape stays one flat object.
@@ -34,6 +34,7 @@ pub struct Transcript {
last_status: Option<SessionStatus>,
last_activity: Option<f64>,
context_tokens: Option<u64>,
context_limit: Option<u64>,
}
impl Transcript {
@@ -68,6 +69,12 @@ impl Transcript {
context_tokens: existing
.iter()
.fold(None, |current, entry| context_after(current, &entry.event)),
// The same fold for the same reason: a session whose process has
// since exited has no window, and the newest `ContextWindow` line
// alone would not know that.
context_limit: existing.iter().fold(None, |current, entry| {
context_limit_after(current, &entry.event)
}),
})
}
@@ -107,6 +114,12 @@ impl Transcript {
self.context_tokens
}
/// What that figure is out of, as of opening, and `None` where this
/// session's provider does not say.
pub fn context_limit(&self) -> Option<u64> {
self.context_limit
}
/// Appends `event`, assigning it the next sequence number. Flushed per
/// event: each line is tiny, and the transcript is the source of truth a
/// crash must not lose the tail of.