Declare provider settings, and give the context figure a denominator
Two things a session could not say, and one it was saying wrongly.
**Every provider setting is reachable.** `-np 1`, the MTP draft depth, the
tool set, the sampling parameters -- most were hardcoded to what measured
best on this machine, which is right as a default and wrong as a constant:
the next machine has a different GPU and a different core count, and
nobody running this app can edit the source. `DriverKind::params` now
declares what a provider takes -- key, label, shape, what blank means, and
whether a change waits for a restart -- and the phone renders whatever
arrives, on the spawn form and in the session settings dialog. Adding a
setting to a driver is one entry in that table and no app change.
`POST /sessions/{id}/params` takes the whole map, so an absent key is the
instruction to unset; the sampling half applies at once and the session is
told in words which of the rest are waiting for a restart.
`tools` is one of them, because it is the biggest lever on a tight
context: the seven built-in definitions are ~1,300 tokens of every prompt
(2,191 against 887 with none). `"none"` omits the flag rather than passing
it on, since `--tools none` is `unknown tool "none"` and a server that
exits.
**The context figure has a denominator.** `Event::ContextWindow` carries
it, read from `llama-server`'s `/props` once the model is up -- the
measurement rather than the request, since a session that named no context
size gets the model's own. Neither coding CLI states its window, so those
keep the bare figure: "2,042" and "2,042 / 8,192" are deliberately
different-looking, and a missing ceiling is never drawn as a proportion of
an assumed one.
**And the numerator was wrong**, by the length of the last reply: it was
the prompt alone, so a five-word answer reported 2,042 against a slot
holding 2,355. It is the turn's total now, which matches `llama-server`'s
own `n_tokens` to within a token.
Two defects the review found, both of which would have shipped: changing
settings on a *stopped* session reported "no process running, so it can't
take new settings", when a stopped session is exactly when you would set
them for the next start; and `GET /tools` answers **403** rather than an
empty list on a server started without `--tools`, so reading it as a
failure made the no-tools session one that never started.
Verified against real models: settings spawned and changed live, the
restart note, a session with two tools and one with none, and the counter
checked against the server's own slot occupancy each time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ac476ab0c9
commit
81ab564a09
18 files changed
+827
-111
No files matched your search
@@ -62,6 +62,15 @@ Module-by-module intent is in PLAN.md's "Backend layout".
|
|||||||
MTP draft head is a 50% speed-up or a 33% loss; and `--spec-type draft-mtp`
|
MTP draft head is a 50% speed-up or a 33% loss; and `--spec-type draft-mtp`
|
||||||
is conditional on the file actually having a head, because asking for one
|
is conditional on the file actually having a head, because asking for one
|
||||||
that is not there makes `llama-server` **exit**.
|
that is not there makes `llama-server` **exit**.
|
||||||
|
**Every one of those is a default rather than a constant** (2026-09-19):
|
||||||
|
`DriverKind::params` declares what a provider takes — key, label, shape,
|
||||||
|
and whether a change waits for a restart — and the phone renders whatever
|
||||||
|
arrives, on the spawn form and in the session settings dialog. Adding a
|
||||||
|
setting to a driver is one entry in that table and no app change. `tools`
|
||||||
|
is in there too, because the seven built-in definitions are ~1,300 tokens
|
||||||
|
of every prompt (2,191 against 887 with none), which on a small window is
|
||||||
|
the difference between a usable session and one that overruns; `"none"`
|
||||||
|
omits the flag, since `--tools none` is a server that exits.
|
||||||
Codex is one persistent `codex app-server --stdio` process per session; its
|
Codex is one persistent `codex app-server --stdio` process per session; its
|
||||||
driver uses native turn steering and interruption, persists the protocol
|
driver uses native turn steering and interruption, persists the protocol
|
||||||
state and thread id, and reads subscription limits through the same CLI
|
state and thread id, and reads subscription limits through the same CLI
|
||||||
@@ -325,6 +334,11 @@ written, and the fold uses that same predicate to decide a reply is settled.
|
|||||||
|
|
||||||
## Things that have bitten
|
## Things that have bitten
|
||||||
|
|
||||||
|
- **A server started with no `--tools` answers 403 at `GET /tools`, not an
|
||||||
|
empty list.** The route is off rather than empty, so reading that as a
|
||||||
|
failure made "no tools" — the one setting whose entire purpose is to have
|
||||||
|
none — a session that never started.
|
||||||
|
|
||||||
- **A llama session reports `loading`, and a message sent into it waits.**
|
- **A llama session reports `loading`, and a message sent into it waits.**
|
||||||
Before 2026-09-19 the session showed `running` from the moment the process
|
Before 2026-09-19 the session showed `running` from the moment the process
|
||||||
started, so a minute of reading a model off disk was indistinguishable from
|
started, so a minute of reading a model off disk was indistinguishable from
|
||||||
|
|||||||
@@ -1435,6 +1435,26 @@ verified by running it, matching dev-updater's posture.
|
|||||||
it truncates old KV cache entries, which is silent forgetting with no
|
it truncates old KV cache entries, which is silent forgetting with no
|
||||||
summary, and it corrupts the harness's view of what the model knows. Fine
|
summary, and it corrupts the harness's view of what the model knows. Fine
|
||||||
as a server-side safety net; not memory management.
|
as a server-side safety net; not memory management.
|
||||||
|
- **What a provider's settings are is declared by the server** (2026-09-19,
|
||||||
|
`DriverKind::params`). A spec is a key, words, a shape and whether a change
|
||||||
|
waits for a restart; the phone renders whatever arrives, so a driver that
|
||||||
|
grows a setting gets a control with no app change. The reason it is
|
||||||
|
declared rather than drawn is not tidiness: several `llama-server` flags
|
||||||
|
were hardcoded to what measured best on one machine, which is right as a
|
||||||
|
default and wrong as a constant — the next machine has a different GPU, and
|
||||||
|
nobody running this app can edit the source. `POST /sessions/{id}/params`
|
||||||
|
takes the whole map, so an absent key *is* the instruction to unset.
|
||||||
|
- **The context figure has a denominator where one can be measured**
|
||||||
|
(2026-09-19). `Event::ContextWindow` carries it, read from `llama-server`'s
|
||||||
|
`/props` once the model is up — the measurement rather than the request,
|
||||||
|
since a session that named no context size gets the model's own. Neither
|
||||||
|
coding CLI states its window, so those keep the bare figure: "2,042" and
|
||||||
|
"2,042 / 8,192" are deliberately different-looking, and a missing ceiling is
|
||||||
|
never drawn as a proportion of an assumed one.
|
||||||
|
The numerator was also wrong, by the length of the last reply: it was the
|
||||||
|
prompt alone, so a five-word answer reported 2,042 against a slot holding
|
||||||
|
2,355. It is the turn's total now, which matches `llama-server`'s own
|
||||||
|
`n_tokens` exactly.
|
||||||
- **MCP servers are configured in `config.ron`, not from the phone**
|
- **MCP servers are configured in `config.ron`, not from the phone**
|
||||||
(2026-09-19). `mcpServers` on a llama provider, with Exa preset on a newly
|
(2026-09-19). `mcpServers` on a llama provider, with Exa preset on a newly
|
||||||
discovered one. A phone screen for them is the obvious next step and was
|
discovered one. A phone screen for them is the obvious next step and was
|
||||||
|
|||||||
@@ -204,6 +204,13 @@ data class SessionSummary(
|
|||||||
* full.
|
* full.
|
||||||
*/
|
*/
|
||||||
val contextTokens: Long?,
|
val contextTokens: Long?,
|
||||||
|
/**
|
||||||
|
* What [contextTokens] is out of, or null where this session's provider does not say. A third
|
||||||
|
* state, not a fourth reading of the same one: the occupancy is known and the ceiling is not.
|
||||||
|
*/
|
||||||
|
val contextLimit: Long?,
|
||||||
|
/** What this session's provider settings are set to; empty where it takes none. */
|
||||||
|
val params: Map<String, String>,
|
||||||
/**
|
/**
|
||||||
* The longest edge an image should have when it reaches this session, or null where the
|
* The longest edge an image should have when it reaches this session, or null where the
|
||||||
* provider has no limit.
|
* provider has no limit.
|
||||||
@@ -262,6 +269,8 @@ private fun parseSession(session: JSONObject) =
|
|||||||
cwd = session.optString("cwd").ifEmpty { null },
|
cwd = session.optString("cwd").ifEmpty { null },
|
||||||
contextTokens =
|
contextTokens =
|
||||||
if (session.has("contextTokens")) session.getLong("contextTokens") else null,
|
if (session.has("contextTokens")) session.getLong("contextTokens") else null,
|
||||||
|
contextLimit = if (session.has("contextLimit")) session.getLong("contextLimit") else null,
|
||||||
|
params = session.optJSONObject("params").stringMap(),
|
||||||
maxImageEdge = session.optInt("maxImageEdge", 0).takeIf { it > 0 },
|
maxImageEdge = session.optInt("maxImageEdge", 0).takeIf { it > 0 },
|
||||||
usageProvider = session.optString("usageProvider").ifEmpty { null },
|
usageProvider = session.optString("usageProvider").ifEmpty { null },
|
||||||
status = session.getString("status"),
|
status = session.getString("status"),
|
||||||
@@ -344,6 +353,27 @@ data class Provider(
|
|||||||
val models: List<String>,
|
val models: List<String>,
|
||||||
val permissionModes: List<String>,
|
val permissionModes: List<String>,
|
||||||
val defaultPermissionMode: String?,
|
val defaultPermissionMode: String?,
|
||||||
|
/** The extra settings this provider takes, in the order to draw them. See [ParamSpec]. */
|
||||||
|
val params: List<ParamSpec>,
|
||||||
|
)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One setting a provider takes, as the server describes it.
|
||||||
|
*
|
||||||
|
* Declared by the server rather than drawn here, so a driver that grows a setting gets a control
|
||||||
|
* with no app change — and, more to the point, so that a value measured on one machine can ship as
|
||||||
|
* a default without becoming a constant nobody else can reach.
|
||||||
|
*/
|
||||||
|
data class ParamSpec(
|
||||||
|
val key: String,
|
||||||
|
val label: String,
|
||||||
|
/** What leaving it blank means, in words, shown as the placeholder. */
|
||||||
|
val unset: String,
|
||||||
|
val kind: String,
|
||||||
|
/** For [kind] `"choice"`: the options, the first of which means "unset". */
|
||||||
|
val options: List<String>,
|
||||||
|
/** Whether a change waits for the session's process to start again. */
|
||||||
|
val restart: Boolean,
|
||||||
)
|
)
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -359,6 +389,11 @@ data class Machine(
|
|||||||
val providers: List<Provider>,
|
val providers: List<Provider>,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
/** A JSON object of strings, and the empty map for one that is absent. */
|
||||||
|
private fun JSONObject?.stringMap(): Map<String, String> =
|
||||||
|
this?.let { object_ -> object_.keys().asSequence().associateWith { object_.getString(it) } }
|
||||||
|
?: emptyMap()
|
||||||
|
|
||||||
private fun parseProvider(provider: JSONObject): Provider {
|
private fun parseProvider(provider: JSONObject): Provider {
|
||||||
val kind = provider.getString("kind")
|
val kind = provider.getString("kind")
|
||||||
return Provider(
|
return Provider(
|
||||||
@@ -368,6 +403,17 @@ private fun parseProvider(provider: JSONObject): Provider {
|
|||||||
models = provider.optJSONArray("models")?.strings().orEmpty(),
|
models = provider.optJSONArray("models")?.strings().orEmpty(),
|
||||||
permissionModes = provider.optJSONArray("permissionModes")?.strings().orEmpty(),
|
permissionModes = provider.optJSONArray("permissionModes")?.strings().orEmpty(),
|
||||||
defaultPermissionMode = provider.optString("defaultPermissionMode").ifEmpty { null },
|
defaultPermissionMode = provider.optString("defaultPermissionMode").ifEmpty { null },
|
||||||
|
params =
|
||||||
|
provider.optJSONArray("params")?.mapObjects { spec ->
|
||||||
|
ParamSpec(
|
||||||
|
key = spec.getString("key"),
|
||||||
|
label = spec.getString("label"),
|
||||||
|
unset = spec.getString("unset"),
|
||||||
|
kind = spec.getString("kind"),
|
||||||
|
options = spec.optJSONArray("options")?.strings().orEmpty(),
|
||||||
|
restart = spec.optBoolean("restart"),
|
||||||
|
)
|
||||||
|
} ?: emptyList(),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -481,6 +527,13 @@ data class Importable(
|
|||||||
* history from before a compaction, which the model is no longer given.
|
* history from before a compaction, which the model is no longer given.
|
||||||
*/
|
*/
|
||||||
val contextTokens: Long?,
|
val contextTokens: Long?,
|
||||||
|
/**
|
||||||
|
* What [contextTokens] is out of, or null where this session's provider does not say. A third
|
||||||
|
* state, not a fourth reading of the same one: the occupancy is known and the ceiling is not.
|
||||||
|
*/
|
||||||
|
val contextLimit: Long?,
|
||||||
|
/** What this session's provider settings are set to; empty where it takes none. */
|
||||||
|
val params: Map<String, String>,
|
||||||
/** Whether [title] is a name somebody chose rather than the last thing said in the session. */
|
/** Whether [title] is a name somebody chose rather than the last thing said in the session. */
|
||||||
val named: Boolean,
|
val named: Boolean,
|
||||||
/**
|
/**
|
||||||
@@ -555,6 +608,10 @@ fun fetchImportable(settings: ServerSettings, machine: String): List<Importable>
|
|||||||
contextTokens =
|
contextTokens =
|
||||||
if (session.isNull("contextTokens")) null
|
if (session.isNull("contextTokens")) null
|
||||||
else session.optLong("contextTokens").takeIf { it > 0L },
|
else session.optLong("contextTokens").takeIf { it > 0L },
|
||||||
|
contextLimit =
|
||||||
|
if (session.isNull("contextLimit")) null
|
||||||
|
else session.optLong("contextLimit").takeIf { it > 0L },
|
||||||
|
params = session.optJSONObject("params").stringMap(),
|
||||||
// Absent means an older backend that cannot answer, which is what "unknown" says.
|
// Absent means an older backend that cannot answer, which is what "unknown" says.
|
||||||
inUse = session.optString("inUse", "unknown"),
|
inUse = session.optString("inUse", "unknown"),
|
||||||
named = session.optBoolean("named", false),
|
named = session.optBoolean("named", false),
|
||||||
@@ -1240,6 +1297,16 @@ fun setSessionEffort(settings: ServerSettings, sessionId: String, level: String?
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** Switches how much a running session asks before acting, also in place. */
|
/** Switches how much a running session asks before acting, also in place. */
|
||||||
|
/** Replaces a session's provider settings with [params] — the whole map, not a patch. */
|
||||||
|
fun setSessionParams(settings: ServerSettings, sessionId: String, params: Map<String, String>) {
|
||||||
|
requestFromServer(
|
||||||
|
settings,
|
||||||
|
"/sessions/$sessionId/params",
|
||||||
|
method = "POST",
|
||||||
|
jsonBody = JSONObject().put("params", JSONObject(params.toMap())).toString(),
|
||||||
|
) {}
|
||||||
|
}
|
||||||
|
|
||||||
fun setSessionPermissionMode(settings: ServerSettings, sessionId: String, mode: String) {
|
fun setSessionPermissionMode(settings: ServerSettings, sessionId: String, mode: String) {
|
||||||
requestFromServer(
|
requestFromServer(
|
||||||
settings,
|
settings,
|
||||||
|
|||||||
@@ -60,3 +60,23 @@ fun compactingLabel(seconds: Long?): String =
|
|||||||
seconds < 60 -> "compacting ${seconds}s"
|
seconds < 60 -> "compacting ${seconds}s"
|
||||||
else -> "compacting ${seconds / 60}m ${seconds % 60}s"
|
else -> "compacting ${seconds / 60}m ${seconds % 60}s"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How full the session is, as the status row says it.
|
||||||
|
*
|
||||||
|
* Three states, not two, and the third is the one that needed the words: a session whose occupancy
|
||||||
|
* is known and whose ceiling is not. That one keeps the bare figure, and a session with a ceiling
|
||||||
|
* gets both — the reader can see which they are looking at. What must not happen is a missing
|
||||||
|
* ceiling drawn as a number, or as a proportion of some assumed window, which would be this screen
|
||||||
|
* inventing the very fact it does not have.
|
||||||
|
*
|
||||||
|
* A llama.cpp session always has one, since the window is a flag its own server was started with. A
|
||||||
|
* coding CLI's is the vendor's business and neither control protocol states it, so those keep the
|
||||||
|
* bare figure they have always had.
|
||||||
|
*/
|
||||||
|
fun contextLabel(held: Long?, limit: Long?): String =
|
||||||
|
when {
|
||||||
|
held == null -> "context unknown"
|
||||||
|
limit == null -> "context ${tokens(held)}"
|
||||||
|
else -> "context ${tokens(held)} / ${tokens(limit)}"
|
||||||
|
}
|
||||||
@@ -163,6 +163,9 @@ sealed class SessionEvent {
|
|||||||
*/
|
*/
|
||||||
data class UsageDelta(val tokens: Long, val context: Long?) : SessionEvent()
|
data class UsageDelta(val tokens: Long, val context: Long?) : SessionEvent()
|
||||||
|
|
||||||
|
/** How much context this session's model has, which is what [UsageDelta.context] is out of. */
|
||||||
|
data class ContextWindow(val tokens: Long) : SessionEvent()
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A compaction that finished, and how much context it recovered.
|
* A compaction that finished, and how much context it recovered.
|
||||||
*
|
*
|
||||||
@@ -293,6 +296,7 @@ fun parseSeqEvent(json: String): SeqEvent {
|
|||||||
model = body.optString("model").ifEmpty { null },
|
model = body.optString("model").ifEmpty { null },
|
||||||
permissionMode = body.optString("permissionMode").ifEmpty { null },
|
permissionMode = body.optString("permissionMode").ifEmpty { null },
|
||||||
)
|
)
|
||||||
|
"contextWindow" -> SessionEvent.ContextWindow(body.getLong("tokens"))
|
||||||
"usageDelta" ->
|
"usageDelta" ->
|
||||||
SessionEvent.UsageDelta(
|
SessionEvent.UsageDelta(
|
||||||
body.getLong("tokens"),
|
body.getLong("tokens"),
|
||||||
@@ -363,3 +367,18 @@ fun contextAfter(current: Long?, event: SessionEvent): Long? =
|
|||||||
is SessionEvent.Cleared -> null
|
is SessionEvent.Cleared -> null
|
||||||
else -> current
|
else -> current
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The context window after [event], mirroring the server's `context_limit_after` for the same
|
||||||
|
* reason [contextAfter] mirrors its neighbour: the screen has to keep up between page loads.
|
||||||
|
*
|
||||||
|
* A window belongs to the process, so a session whose process has exited has none — left standing,
|
||||||
|
* a session restarted on a different model would draw its occupancy against the old model's
|
||||||
|
* ceiling.
|
||||||
|
*/
|
||||||
|
fun contextLimitAfter(current: Long?, event: SessionEvent): Long? =
|
||||||
|
when (event) {
|
||||||
|
is SessionEvent.ContextWindow -> event.tokens
|
||||||
|
is SessionEvent.Status -> if (event.state == "exited") null else current
|
||||||
|
else -> current
|
||||||
|
}
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
package com.example.aiapp
|
||||||
|
|
||||||
|
import androidx.compose.foundation.layout.Column
|
||||||
|
import androidx.compose.foundation.layout.Spacer
|
||||||
|
import androidx.compose.foundation.layout.fillMaxWidth
|
||||||
|
import androidx.compose.foundation.layout.height
|
||||||
|
import androidx.compose.foundation.text.KeyboardOptions
|
||||||
|
import androidx.compose.material3.MaterialTheme
|
||||||
|
import androidx.compose.material3.OutlinedTextField
|
||||||
|
import androidx.compose.material3.Text
|
||||||
|
import androidx.compose.runtime.Composable
|
||||||
|
import androidx.compose.ui.Modifier
|
||||||
|
import androidx.compose.ui.text.input.KeyboardType
|
||||||
|
import androidx.compose.ui.unit.dp
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The controls for whatever settings a provider says it takes.
|
||||||
|
*
|
||||||
|
* One composable for both screens that offer them — the spawn form and the session settings dialog
|
||||||
|
* — and for every provider, because the server declares the list (see `DriverKind::params`) rather
|
||||||
|
* than this file knowing it. A driver that grows a setting gets a control here with no change to
|
||||||
|
* the app, which is the whole point: the values that suit one machine ship as defaults, and every
|
||||||
|
* one of them stays reachable from a phone.
|
||||||
|
*
|
||||||
|
* [values] is the whole map and [onChange] hands back the whole map. A key absent from it means the
|
||||||
|
* setting is unset, which is what every [ParamSpec.unset] describes — so clearing a field and never
|
||||||
|
* touching it are deliberately the same state.
|
||||||
|
*/
|
||||||
|
@Composable
|
||||||
|
fun ProviderParamFields(
|
||||||
|
specs: List<ParamSpec>,
|
||||||
|
values: Map<String, String>,
|
||||||
|
onChange: (Map<String, String>) -> Unit,
|
||||||
|
/**
|
||||||
|
* Whether to say which settings wait for a restart. False on a spawn form, where nothing is
|
||||||
|
* running yet and every setting is about to be read — saying it there would be a warning about
|
||||||
|
* a state the reader cannot be in.
|
||||||
|
*/
|
||||||
|
warnAboutRestart: Boolean,
|
||||||
|
modifier: Modifier = Modifier,
|
||||||
|
) {
|
||||||
|
if (specs.isEmpty()) return
|
||||||
|
Column(modifier.fillMaxWidth()) {
|
||||||
|
specs.forEach { spec ->
|
||||||
|
val set = { value: String ->
|
||||||
|
onChange(
|
||||||
|
// Blank clears rather than storing an empty string: the server reads an absent
|
||||||
|
// key as "use the default", and an empty one would be a value it then failed
|
||||||
|
// to parse.
|
||||||
|
if (value.isBlank()) values - spec.key else values + (spec.key to value)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
when (spec.kind) {
|
||||||
|
"choice" -> {
|
||||||
|
// The first option is what unset means, so selecting it clears the key — see
|
||||||
|
// `ParamKind::Choice`. Without that the picker could show a default it could
|
||||||
|
// not return to.
|
||||||
|
val default = spec.options.firstOrNull().orEmpty()
|
||||||
|
ChipGroup(
|
||||||
|
label = spec.label + restartSuffix(spec, warnAboutRestart),
|
||||||
|
options = spec.options,
|
||||||
|
selected = values[spec.key] ?: default,
|
||||||
|
onSelect = { chosen -> set(if (chosen == default) "" else chosen) },
|
||||||
|
)
|
||||||
|
}
|
||||||
|
else ->
|
||||||
|
OutlinedTextField(
|
||||||
|
value = values[spec.key].orEmpty(),
|
||||||
|
onValueChange = set,
|
||||||
|
label = { Text(spec.label + restartSuffix(spec, warnAboutRestart)) },
|
||||||
|
placeholder = { Text(spec.unset) },
|
||||||
|
singleLine = true,
|
||||||
|
keyboardOptions = KeyboardOptions(keyboardType = keyboardFor(spec.kind)),
|
||||||
|
modifier = Modifier.fillMaxWidth(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
Spacer(Modifier.height(16.dp))
|
||||||
|
}
|
||||||
|
if (warnAboutRestart && specs.any { it.restart }) {
|
||||||
|
Text(
|
||||||
|
"A setting marked “on restart” is saved now and read when this session's process " +
|
||||||
|
"next starts.",
|
||||||
|
style = MaterialTheme.typography.bodySmall,
|
||||||
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Marks a control whose value will not take effect yet.
|
||||||
|
*
|
||||||
|
* On the label rather than beside it, because the reader decides whether to change the thing before
|
||||||
|
* they touch it — a note underneath is read after the decision.
|
||||||
|
*/
|
||||||
|
private fun restartSuffix(spec: ParamSpec, warn: Boolean): String =
|
||||||
|
if (warn && spec.restart) " (on restart)" else ""
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The keyboard for a value's shape. A number field that opens the letter keyboard is one every
|
||||||
|
* entry is made harder by, and these are nearly all numbers.
|
||||||
|
*/
|
||||||
|
private fun keyboardFor(kind: String): KeyboardType =
|
||||||
|
when (kind) {
|
||||||
|
"integer" -> KeyboardType.Number
|
||||||
|
"decimal" -> KeyboardType.Decimal
|
||||||
|
else -> KeyboardType.Text
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How long typing has to stop before edited settings are sent.
|
||||||
|
*
|
||||||
|
* Long enough that a number is one request rather than one per digit, short enough that closing the
|
||||||
|
* dialog straight after typing still saves — the save runs on the screen behind it, which outlives
|
||||||
|
* the dialog, so this delay is not a window the value can be lost in.
|
||||||
|
*/
|
||||||
|
const val PARAM_SAVE_DELAY_MS = 700L
|
||||||
@@ -233,6 +233,8 @@ fun SessionScreen(
|
|||||||
// that was measured.
|
// that was measured.
|
||||||
var contextTokens by
|
var contextTokens by
|
||||||
remember(address) { mutableStateOf(if (isSubagent) null else summary.contextTokens) }
|
remember(address) { mutableStateOf(if (isSubagent) null else summary.contextTokens) }
|
||||||
|
var contextLimit by
|
||||||
|
remember(address) { mutableStateOf(if (isSubagent) null else summary.contextLimit) }
|
||||||
// When the current compaction started. The moment comes off the `compacting` status event
|
// When the current compaction started. The moment comes off the `compacting` status event
|
||||||
// itself -- the server timestamps every transcript line -- rather than off this device noticing
|
// itself -- the server timestamps every transcript line -- rather than off this device noticing
|
||||||
// one, which is what makes it survive leaving the session and reopening it.
|
// one, which is what makes it survive leaving the session and reopening it.
|
||||||
@@ -304,6 +306,14 @@ fun SessionScreen(
|
|||||||
// hardcoded list is a claim about a machine.
|
// hardcoded list is a claim about a machine.
|
||||||
var offeredModels by remember { mutableStateOf<List<OfferedModel>>(emptyList()) }
|
var offeredModels by remember { mutableStateOf<List<OfferedModel>>(emptyList()) }
|
||||||
var offeredPermissionModes by remember { mutableStateOf<List<String>>(emptyList()) }
|
var offeredPermissionModes by remember { mutableStateOf<List<String>>(emptyList()) }
|
||||||
|
// The settings this session's provider takes, and what they are set to. The specs come from
|
||||||
|
// the provider and the values from the session, because "what can be set" and "what is set"
|
||||||
|
// are different questions with different answers.
|
||||||
|
var paramSpecs by remember { mutableStateOf<List<ParamSpec>>(emptyList()) }
|
||||||
|
var params by remember(summary.id) { mutableStateOf(summary.params) }
|
||||||
|
// What was last successfully saved, so the debounce below knows whether there is anything to
|
||||||
|
// send -- and so a failed save can put the controls back to what the server actually holds.
|
||||||
|
var savedParams by remember(summary.id) { mutableStateOf(summary.params) }
|
||||||
val lifecycleOwner = LocalLifecycleOwner.current
|
val lifecycleOwner = LocalLifecycleOwner.current
|
||||||
// The resume cursor, written from the stream's IO thread.
|
// The resume cursor, written from the stream's IO thread.
|
||||||
val lastSeq = remember { AtomicLong(0) }
|
val lastSeq = remember { AtomicLong(0) }
|
||||||
@@ -489,6 +499,7 @@ fun SessionScreen(
|
|||||||
// Before the rest, and for every event rather than only the usage ones: a compaction and a
|
// Before the rest, and for every event rather than only the usage ones: a compaction and a
|
||||||
// clear move this as much as a turn does. See `contextAfter`.
|
// clear move this as much as a turn does. See `contextAfter`.
|
||||||
contextTokens = contextAfter(contextTokens, entry.event)
|
contextTokens = contextAfter(contextTokens, entry.event)
|
||||||
|
contextLimit = contextLimitAfter(contextLimit, entry.event)
|
||||||
when (val event = entry.event) {
|
when (val event = entry.event) {
|
||||||
// Nothing further: what it carries was folded into the context above.
|
// Nothing further: what it carries was folded into the context above.
|
||||||
is SessionEvent.UsageDelta -> {}
|
is SessionEvent.UsageDelta -> {}
|
||||||
@@ -1205,6 +1216,30 @@ fun SessionScreen(
|
|||||||
fun label(id: String?): String =
|
fun label(id: String?): String =
|
||||||
offeredModels.firstOrNull { it.id == id }?.label ?: modelLabel(id)
|
offeredModels.firstOrNull { it.id == id }?.label ?: modelLabel(id)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Saves edited provider settings once the typing stops.
|
||||||
|
*
|
||||||
|
* Debounced rather than sent per keystroke, because a text field over the tunnel would be a
|
||||||
|
* round trip and a config write per character. Here rather than in the settings dialog for the
|
||||||
|
* reason the delay exists at all: the dialog can be dismissed mid-edit, and this screen
|
||||||
|
* outlives it, so the last value typed is still saved.
|
||||||
|
*/
|
||||||
|
LaunchedEffect(params) {
|
||||||
|
if (params == savedParams) return@LaunchedEffect
|
||||||
|
delay(PARAM_SAVE_DELAY_MS)
|
||||||
|
val wanted = params
|
||||||
|
try {
|
||||||
|
withContext(Dispatchers.IO) { setSessionParams(settings, summary.id, wanted) }
|
||||||
|
savedParams = wanted
|
||||||
|
actionError = null
|
||||||
|
} catch (e: ApiException) {
|
||||||
|
// Back to what the server holds. A control left showing a value that was refused is
|
||||||
|
// stating something untrue about the session.
|
||||||
|
params = savedParams
|
||||||
|
actionError = e.message
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Only for the model picker, which a subagent does not have.
|
// Only for the model picker, which a subagent does not have.
|
||||||
if (!isSubagent) {
|
if (!isSubagent) {
|
||||||
LaunchedEffect(summary.machine, summary.provider) {
|
LaunchedEffect(summary.machine, summary.provider) {
|
||||||
@@ -1218,6 +1253,7 @@ fun SessionScreen(
|
|||||||
}
|
}
|
||||||
.getOrNull()
|
.getOrNull()
|
||||||
offeredPermissionModes = provider?.permissionModes.orEmpty()
|
offeredPermissionModes = provider?.permissionModes.orEmpty()
|
||||||
|
paramSpecs = provider?.params.orEmpty()
|
||||||
offeredModels =
|
offeredModels =
|
||||||
provider
|
provider
|
||||||
?.let {
|
?.let {
|
||||||
@@ -1916,6 +1952,7 @@ fun SessionScreen(
|
|||||||
status = status,
|
status = status,
|
||||||
compactingFor = compactingFor,
|
compactingFor = compactingFor,
|
||||||
contextTokens = contextTokens,
|
contextTokens = contextTokens,
|
||||||
|
contextLimit = contextLimit,
|
||||||
backgroundTasks = backgroundTasks,
|
backgroundTasks = backgroundTasks,
|
||||||
subagent = isSubagent,
|
subagent = isSubagent,
|
||||||
)
|
)
|
||||||
@@ -2197,6 +2234,9 @@ fun SessionScreen(
|
|||||||
effort = effort.takeIf { summary.takesEffort },
|
effort = effort.takeIf { summary.takesEffort },
|
||||||
takesEffort = summary.takesEffort,
|
takesEffort = summary.takesEffort,
|
||||||
onEffortChanged = { effort = it },
|
onEffortChanged = { effort = it },
|
||||||
|
paramSpecs = paramSpecs,
|
||||||
|
params = params,
|
||||||
|
onParamsChanged = { params = it },
|
||||||
cachedBytes = cachedBytes,
|
cachedBytes = cachedBytes,
|
||||||
// The purge finishes before the epoch moves, because the relaunched opening effect
|
// The purge finishes before the epoch moves, because the relaunched opening effect
|
||||||
// reads the same directory and would otherwise draw what is about to be deleted. The
|
// reads the same directory and would otherwise draw what is about to be deleted. The
|
||||||
@@ -2473,6 +2513,8 @@ private fun SessionStatusRow(
|
|||||||
compactingFor: Long?,
|
compactingFor: Long?,
|
||||||
/** Context the session is holding, or null where nothing has measured it. */
|
/** Context the session is holding, or null where nothing has measured it. */
|
||||||
contextTokens: Long?,
|
contextTokens: Long?,
|
||||||
|
/** What that is out of, or null where the provider does not say. */
|
||||||
|
contextLimit: Long?,
|
||||||
/** Provider-reported live background work; zero is deliberately not drawn. */
|
/** Provider-reported live background work; zero is deliberately not drawn. */
|
||||||
backgroundTasks: Int,
|
backgroundTasks: Int,
|
||||||
modifier: Modifier = Modifier,
|
modifier: Modifier = Modifier,
|
||||||
@@ -2577,7 +2619,7 @@ private fun SessionStatusRow(
|
|||||||
// reports usage, and one that has not run a turn all showed nothing at all, which reads as
|
// reports usage, and one that has not run a turn all showed nothing at all, which reads as
|
||||||
// a conversation with room to spare.
|
// a conversation with room to spare.
|
||||||
Text(
|
Text(
|
||||||
contextTokens?.let { "context ${tokens(it)}" } ?: "context unknown",
|
contextLabel(contextTokens, contextLimit),
|
||||||
style = MaterialTheme.typography.labelSmall,
|
style = MaterialTheme.typography.labelSmall,
|
||||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -74,6 +74,15 @@ fun SessionSettingsDialog(
|
|||||||
onEffortChanged: (String?) -> Unit,
|
onEffortChanged: (String?) -> Unit,
|
||||||
/** Whether a level does anything here; the row is left out entirely where it does not. */
|
/** Whether a level does anything here; the row is left out entirely where it does not. */
|
||||||
takesEffort: Boolean,
|
takesEffort: Boolean,
|
||||||
|
/**
|
||||||
|
* The settings this session's provider takes, and what they are set to.
|
||||||
|
*
|
||||||
|
* Declared by the server rather than listed here -- see [ProviderParamFields]. Empty for a
|
||||||
|
* provider with none, which draws no section at all.
|
||||||
|
*/
|
||||||
|
paramSpecs: List<ParamSpec>,
|
||||||
|
params: Map<String, String>,
|
||||||
|
onParamsChanged: (Map<String, String>) -> Unit,
|
||||||
/**
|
/**
|
||||||
* What this phone is holding of the conversation, or null while that is being measured -- see
|
* What this phone is holding of the conversation, or null while that is being measured -- see
|
||||||
* the Reload row below, which is what would discard it.
|
* the Reload row below, which is what would discard it.
|
||||||
@@ -451,6 +460,23 @@ fun SessionSettingsDialog(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (paramSpecs.isNotEmpty()) {
|
||||||
|
Spacer(Modifier.height(16.dp))
|
||||||
|
Text(
|
||||||
|
"Model settings",
|
||||||
|
style = MaterialTheme.typography.titleSmall,
|
||||||
|
)
|
||||||
|
Spacer(Modifier.height(8.dp))
|
||||||
|
// Edited here and saved by the screen behind this, which is what makes
|
||||||
|
// typing in a text field affordable: the save is debounced, and a dialog
|
||||||
|
// dismissed mid-edit would take an unsaved value with it.
|
||||||
|
ProviderParamFields(
|
||||||
|
specs = paramSpecs,
|
||||||
|
values = params,
|
||||||
|
onChange = onParamsChanged,
|
||||||
|
warnAboutRestart = true,
|
||||||
|
)
|
||||||
|
}
|
||||||
Spacer(Modifier.height(8.dp))
|
Spacer(Modifier.height(8.dp))
|
||||||
Row(
|
Row(
|
||||||
verticalAlignment = Alignment.CenterVertically,
|
verticalAlignment = Alignment.CenterVertically,
|
||||||
|
|||||||
@@ -73,13 +73,9 @@ fun SpawnScreen(
|
|||||||
// Only the spawn's own failure. The fetch's lives in `options`: this one leaves a filled-in
|
// Only the spawn's own failure. The fetch's lives in `options`: this one leaves a filled-in
|
||||||
// form worth keeping, and that one leaves nothing to fill in.
|
// form worth keeping, and that one leaves nothing to fill in.
|
||||||
var spawnError by remember { mutableStateOf<String?>(null) }
|
var spawnError by remember { mutableStateOf<String?>(null) }
|
||||||
var contextSize by remember { mutableStateOf("") }
|
// Whatever the chosen provider says it takes, by key. Empty until something is typed: an
|
||||||
var temperature by remember { mutableStateOf("") }
|
// absent key means the server's own default, which is what every field's placeholder says.
|
||||||
// Whether a model that carries a multi-token-prediction head drafts with it. Left to the
|
var params by remember { mutableStateOf<Map<String, String>>(emptyMap()) }
|
||||||
// server by default, which turns it on exactly where the file has one -- see `SPECULATIVE` in
|
|
||||||
// the llama driver. Here so a machine where drafting does not pay has a way out that is not
|
|
||||||
// an edit to config.ron.
|
|
||||||
var speculative by remember { mutableStateOf(SPECULATIVE_AUTO) }
|
|
||||||
|
|
||||||
LaunchedEffect(Unit) {
|
LaunchedEffect(Unit) {
|
||||||
// Separate from the machines fetch below and deliberately not fatal: failing to learn the
|
// Separate from the machines fetch below and deliberately not fatal: failing to learn the
|
||||||
@@ -145,6 +141,9 @@ fun SpawnScreen(
|
|||||||
// machine looks exactly like one from this one.
|
// machine looks exactly like one from this one.
|
||||||
LaunchedEffect(machine?.id, current?.name) {
|
LaunchedEffect(machine?.id, current?.name) {
|
||||||
model = ""
|
model = ""
|
||||||
|
// A key from the previous provider would be a setting this one does not have, drawn
|
||||||
|
// by no control and sent at the spawn anyway.
|
||||||
|
params = emptyMap()
|
||||||
providerModels = emptyList()
|
providerModels = emptyList()
|
||||||
providerModelsError = null
|
providerModelsError = null
|
||||||
permissionMode = current?.defaultPermissionMode.orEmpty()
|
permissionMode = current?.defaultPermissionMode.orEmpty()
|
||||||
@@ -283,35 +282,14 @@ fun SpawnScreen(
|
|||||||
Spacer(Modifier.height(16.dp))
|
Spacer(Modifier.height(16.dp))
|
||||||
}
|
}
|
||||||
|
|
||||||
if (isLlama) {
|
// Nothing is running yet, so nothing here waits for a restart -- every one of these is
|
||||||
OutlinedTextField(
|
// read by the process this form is about to start.
|
||||||
value = contextSize,
|
ProviderParamFields(
|
||||||
onValueChange = { contextSize = it },
|
specs = current?.params.orEmpty(),
|
||||||
label = { Text("Context size (blank = the model's default)") },
|
values = params,
|
||||||
singleLine = true,
|
onChange = { params = it },
|
||||||
modifier = Modifier.fillMaxWidth(),
|
warnAboutRestart = false,
|
||||||
)
|
)
|
||||||
Spacer(Modifier.height(16.dp))
|
|
||||||
|
|
||||||
OutlinedTextField(
|
|
||||||
value = temperature,
|
|
||||||
onValueChange = { temperature = it },
|
|
||||||
label = { Text("Temperature (blank = llama.cpp's default)") },
|
|
||||||
singleLine = true,
|
|
||||||
modifier = Modifier.fillMaxWidth(),
|
|
||||||
)
|
|
||||||
Spacer(Modifier.height(16.dp))
|
|
||||||
|
|
||||||
// Said as what it is rather than as "MTP": the reader is choosing whether the session
|
|
||||||
// goes faster, and most models have nothing to turn on here at all.
|
|
||||||
ChipGroup(
|
|
||||||
label = "Speculative decoding (models that carry a draft head)",
|
|
||||||
options = listOf(SPECULATIVE_AUTO, SPECULATIVE_OFF),
|
|
||||||
selected = speculative,
|
|
||||||
onSelect = { speculative = it },
|
|
||||||
)
|
|
||||||
Spacer(Modifier.height(16.dp))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Every session whose tools act on files needs one, which is both kinds that have
|
// Every session whose tools act on files needs one, which is both kinds that have
|
||||||
// tools -- a llama session's built-in tools run in it exactly as a CLI's do.
|
// tools -- a llama session's built-in tools run in it exactly as a CLI's do.
|
||||||
@@ -387,28 +365,10 @@ fun SpawnScreen(
|
|||||||
cwd = cwd.trim().takeIf { takesCwd },
|
cwd = cwd.trim().takeIf { takesCwd },
|
||||||
permissionMode = permissionMode.takeIf { it.isNotEmpty() },
|
permissionMode = permissionMode.takeIf { it.isNotEmpty() },
|
||||||
effort = effort.takeIf { isCodingCli },
|
effort = effort.takeIf { isCodingCli },
|
||||||
// Sent only when set, so blank means "whatever llama.cpp does
|
// Already only the keys somebody set: a field left blank
|
||||||
// by default" rather than a zero.
|
// removes its key rather than sending an empty value, so
|
||||||
params =
|
// "blank" reaches the server as "your default".
|
||||||
buildMap {
|
params = params,
|
||||||
if (isLlama) {
|
|
||||||
contextSize
|
|
||||||
.trim()
|
|
||||||
.takeIf { it.isNotEmpty() }
|
|
||||||
?.let { put("contextSize", it) }
|
|
||||||
temperature
|
|
||||||
.trim()
|
|
||||||
.takeIf { it.isNotEmpty() }
|
|
||||||
?.let { put("temperature", it) }
|
|
||||||
// Only the choice that changes anything: "auto"
|
|
||||||
// is the absence of the setting, not a value of
|
|
||||||
// it, so a session spawned without an opinion
|
|
||||||
// carries none.
|
|
||||||
if (speculative == SPECULATIVE_OFF) {
|
|
||||||
put("speculative", "off")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
onSpawned(spawned)
|
onSpawned(spawned)
|
||||||
@@ -426,15 +386,6 @@ fun SpawnScreen(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Leave the draft head to the server, which uses one wherever the model file has one. Spelled the
|
|
||||||
* same as the absence of the `speculative` parameter, because that is what it means.
|
|
||||||
*/
|
|
||||||
private const val SPECULATIVE_AUTO = "auto"
|
|
||||||
|
|
||||||
/** The `speculative` parameter's only other value; see the llama driver's `SPECULATIVE`. */
|
|
||||||
private const val SPECULATIVE_OFF = "off"
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A labeled row of choices that wraps onto as many lines as it needs.
|
* A labeled row of choices that wraps onto as many lines as it needs.
|
||||||
*
|
*
|
||||||
|
|||||||
@@ -555,6 +555,7 @@ fun foldEvent(items: List<TranscriptItem>, entry: SeqEvent): List<TranscriptItem
|
|||||||
is SessionEvent.RetiredTaskNote -> items
|
is SessionEvent.RetiredTaskNote -> items
|
||||||
// Screen-level state, not transcript rows -- see SessionScreen.
|
// Screen-level state, not transcript rows -- see SessionScreen.
|
||||||
is SessionEvent.UsageDelta -> items
|
is SessionEvent.UsageDelta -> items
|
||||||
|
is SessionEvent.ContextWindow -> items
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -75,6 +75,8 @@ class SessionOrderTest {
|
|||||||
resumeAt = null,
|
resumeAt = null,
|
||||||
cwd = null,
|
cwd = null,
|
||||||
contextTokens = null,
|
contextTokens = null,
|
||||||
|
contextLimit = null,
|
||||||
|
params = emptyMap(),
|
||||||
maxImageEdge = null,
|
maxImageEdge = null,
|
||||||
usageProvider = null,
|
usageProvider = null,
|
||||||
status = status,
|
status = status,
|
||||||
|
|||||||
@@ -312,6 +312,28 @@ impl DriverKind {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The settings this kind of session takes beyond the shared ones, for
|
||||||
|
/// the phone to offer.
|
||||||
|
///
|
||||||
|
/// Declared rather than drawn: a spawn screen with a field per llama
|
||||||
|
/// setting is a screen that has to be edited every time a driver grows
|
||||||
|
/// one, and this app already has the `params` map to carry them. So the
|
||||||
|
/// server says what a provider takes and the phone renders it, which is
|
||||||
|
/// the same arrangement `permission_modes` uses and for the same reason
|
||||||
|
/// -- the alternative is two lists that disagree, one of them in Kotlin.
|
||||||
|
///
|
||||||
|
/// It is also what keeps these *reachable at all*. Several were hardcoded
|
||||||
|
/// to the values measured on one machine, which is fine as a default and
|
||||||
|
/// wrong as a constant: the next machine has a different GPU and a
|
||||||
|
/// different number of cores, and nobody running this app can edit the
|
||||||
|
/// source.
|
||||||
|
pub fn params(self) -> &'static [ParamSpec] {
|
||||||
|
match self {
|
||||||
|
Self::LlamaCpp => LLAMA_PARAMS,
|
||||||
|
Self::Echo | Self::ClaudeCli | Self::CodexCli => &[],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// The mode used when a new-session form first selects this kind.
|
/// The mode used when a new-session form first selects this kind.
|
||||||
pub fn default_permission_mode(self) -> Option<&'static str> {
|
pub fn default_permission_mode(self) -> Option<&'static str> {
|
||||||
match self {
|
match self {
|
||||||
@@ -323,6 +345,134 @@ impl DriverKind {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// One setting a provider takes, and enough about it to draw a control.
|
||||||
|
///
|
||||||
|
/// Deliberately thin: a key, words for a person, and which shape the value
|
||||||
|
/// has. Anything richer -- units, validation, dependencies between settings --
|
||||||
|
/// would be a schema language, and what the phone needs is a text field or a
|
||||||
|
/// row of chips.
|
||||||
|
#[derive(Debug, Clone, Copy, Serialize)]
|
||||||
|
#[serde(rename_all = "camelCase")]
|
||||||
|
pub struct ParamSpec {
|
||||||
|
/// The `SessionConfig::params` key this writes.
|
||||||
|
pub key: &'static str,
|
||||||
|
pub label: &'static str,
|
||||||
|
/// What happens when it is not set, in words. Shown where a control shows
|
||||||
|
/// its placeholder, so "blank" always means something specific rather than
|
||||||
|
/// leaving the reader to guess whether it means zero.
|
||||||
|
pub unset: &'static str,
|
||||||
|
/// Flattened, so a spec is one flat object: `kind` beside the rest rather
|
||||||
|
/// than an object of its own with `kind` inside it.
|
||||||
|
#[serde(flatten)]
|
||||||
|
pub kind: ParamKind,
|
||||||
|
/// Whether changing it waits for the process to start again.
|
||||||
|
///
|
||||||
|
/// The honest half of offering these live. A sampling setting rides on the
|
||||||
|
/// next request; a server flag was decided when the model was loaded, and
|
||||||
|
/// a control that silently did nothing until some later restart would be
|
||||||
|
/// worse than one that is not there.
|
||||||
|
pub restart: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What shape a [`ParamSpec`]'s value has.
|
||||||
|
#[derive(Debug, Clone, Copy, Serialize)]
|
||||||
|
#[serde(rename_all = "camelCase", tag = "kind")]
|
||||||
|
pub enum ParamKind {
|
||||||
|
Integer,
|
||||||
|
Decimal,
|
||||||
|
Text,
|
||||||
|
/// A fixed set. **The first option is what leaving it unset means**, and
|
||||||
|
/// choosing it clears the setting rather than storing a value -- so the
|
||||||
|
/// default is a state the picker can return to, and the stored config
|
||||||
|
/// does not fill up with values nobody chose.
|
||||||
|
Choice {
|
||||||
|
options: &'static [&'static str],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What a llama.cpp session takes.
|
||||||
|
///
|
||||||
|
/// The server flags first, in the order they matter, then the sampling ones --
|
||||||
|
/// which is also the order of how disruptive changing one is.
|
||||||
|
const LLAMA_PARAMS: &[ParamSpec] = &[
|
||||||
|
ParamSpec {
|
||||||
|
key: "contextSize",
|
||||||
|
label: "Context size",
|
||||||
|
unset: "the model's own trained context",
|
||||||
|
kind: ParamKind::Integer,
|
||||||
|
restart: true,
|
||||||
|
},
|
||||||
|
ParamSpec {
|
||||||
|
key: "tools",
|
||||||
|
label: "Tools",
|
||||||
|
// Worth a control rather than a constant because of what it costs:
|
||||||
|
// the definitions of all seven are ~2,000 tokens of the context,
|
||||||
|
// every turn, before anything is said. On a small window that is the
|
||||||
|
// difference between a usable session and one that overruns.
|
||||||
|
unset: "all of them -- or a comma-separated list, or \"none\"",
|
||||||
|
kind: ParamKind::Text,
|
||||||
|
restart: true,
|
||||||
|
},
|
||||||
|
ParamSpec {
|
||||||
|
key: "gpuLayers",
|
||||||
|
label: "Layers on the GPU",
|
||||||
|
unset: "as many as fit",
|
||||||
|
kind: ParamKind::Integer,
|
||||||
|
restart: true,
|
||||||
|
},
|
||||||
|
ParamSpec {
|
||||||
|
key: "threads",
|
||||||
|
label: "Threads",
|
||||||
|
unset: "one per core",
|
||||||
|
kind: ParamKind::Integer,
|
||||||
|
restart: true,
|
||||||
|
},
|
||||||
|
ParamSpec {
|
||||||
|
key: "speculative",
|
||||||
|
label: "Speculative decoding",
|
||||||
|
unset: "on, for a model whose file carries a draft head",
|
||||||
|
kind: ParamKind::Choice {
|
||||||
|
options: &["auto", "off"],
|
||||||
|
},
|
||||||
|
restart: true,
|
||||||
|
},
|
||||||
|
ParamSpec {
|
||||||
|
key: "specDraftNMax",
|
||||||
|
label: "Tokens drafted ahead",
|
||||||
|
unset: "llama.cpp's own default",
|
||||||
|
kind: ParamKind::Integer,
|
||||||
|
restart: true,
|
||||||
|
},
|
||||||
|
ParamSpec {
|
||||||
|
key: "temperature",
|
||||||
|
label: "Temperature",
|
||||||
|
unset: "llama.cpp's default",
|
||||||
|
kind: ParamKind::Decimal,
|
||||||
|
restart: false,
|
||||||
|
},
|
||||||
|
ParamSpec {
|
||||||
|
key: "topP",
|
||||||
|
label: "Top P",
|
||||||
|
unset: "llama.cpp's default",
|
||||||
|
kind: ParamKind::Decimal,
|
||||||
|
restart: false,
|
||||||
|
},
|
||||||
|
ParamSpec {
|
||||||
|
key: "topK",
|
||||||
|
label: "Top K",
|
||||||
|
unset: "llama.cpp's default",
|
||||||
|
kind: ParamKind::Integer,
|
||||||
|
restart: false,
|
||||||
|
},
|
||||||
|
ParamSpec {
|
||||||
|
key: "maxTokens",
|
||||||
|
label: "Reply limit",
|
||||||
|
unset: "no limit",
|
||||||
|
kind: ParamKind::Integer,
|
||||||
|
restart: false,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
#[serde(rename_all = "camelCase")]
|
#[serde(rename_all = "camelCase")]
|
||||||
pub struct TokenEntry {
|
pub struct TokenEntry {
|
||||||
|
|||||||
@@ -54,6 +54,8 @@
|
|||||||
//! which starts again in the new one
|
//! which starts again in the new one
|
||||||
//! POST /sessions/{id}/model {model}
|
//! POST /sessions/{id}/model {model}
|
||||||
//! POST /sessions/{id}/permission-mode {permissionMode}
|
//! POST /sessions/{id}/permission-mode {permissionMode}
|
||||||
|
//! POST /sessions/{id}/params {params} -- the provider settings, whole;
|
||||||
|
//! what a provider takes is on its ProviderInfo
|
||||||
//! POST /sessions/{id}/effort {effort} -- null for the CLI's default;
|
//! POST /sessions/{id}/effort {effort} -- null for the CLI's default;
|
||||||
//! settled at launch, so this stops the process
|
//! settled at launch, so this stops the process
|
||||||
//! POST /sessions/{id}/command {text} -- /compact, /clear, /rename x, or the dialect's own
|
//! POST /sessions/{id}/command {text} -- /compact, /clear, /rename x, or the dialect's own
|
||||||
@@ -166,6 +168,7 @@ pub fn router(manager: Arc<SessionManager>) -> Router {
|
|||||||
.route("/sessions/{id}/cwd", post(set_cwd))
|
.route("/sessions/{id}/cwd", post(set_cwd))
|
||||||
.route("/sessions/{id}/model", post(set_model))
|
.route("/sessions/{id}/model", post(set_model))
|
||||||
.route("/sessions/{id}/permission-mode", post(set_permission_mode))
|
.route("/sessions/{id}/permission-mode", post(set_permission_mode))
|
||||||
|
.route("/sessions/{id}/params", post(set_params))
|
||||||
.route("/sessions/{id}/effort", post(set_effort))
|
.route("/sessions/{id}/effort", post(set_effort))
|
||||||
.route("/defaults", get(defaults).post(set_defaults))
|
.route("/defaults", get(defaults).post(set_defaults))
|
||||||
.route("/sessions/{id}/notify", post(set_notify))
|
.route("/sessions/{id}/notify", post(set_notify))
|
||||||
@@ -300,6 +303,11 @@ struct ProviderInfo {
|
|||||||
permission_modes: Vec<&'static str>,
|
permission_modes: Vec<&'static str>,
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
default_permission_mode: Option<&'static str>,
|
default_permission_mode: Option<&'static str>,
|
||||||
|
/// The per-kind settings a spawn form and the session settings dialog
|
||||||
|
/// offer -- see [`crate::config::DriverKind::params`]. Empty for a
|
||||||
|
/// provider with none, which draws no section at all.
|
||||||
|
#[serde(skip_serializing_if = "<[_]>::is_empty")]
|
||||||
|
params: &'static [crate::config::ParamSpec],
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn list_machines(State(manager): State<Arc<SessionManager>>) -> axum::Json<Vec<MachineInfo>> {
|
async fn list_machines(State(manager): State<Arc<SessionManager>>) -> axum::Json<Vec<MachineInfo>> {
|
||||||
@@ -320,6 +328,7 @@ fn info_for(machine: crate::config::MachineConfig) -> MachineInfo {
|
|||||||
models: provider.models,
|
models: provider.models,
|
||||||
permission_modes: provider.kind.permission_modes().to_vec(),
|
permission_modes: provider.kind.permission_modes().to_vec(),
|
||||||
default_permission_mode: provider.kind.default_permission_mode(),
|
default_permission_mode: provider.kind.default_permission_mode(),
|
||||||
|
params: provider.kind.params(),
|
||||||
})
|
})
|
||||||
.collect(),
|
.collect(),
|
||||||
}
|
}
|
||||||
@@ -426,6 +435,7 @@ async fn probe_machine(
|
|||||||
models: provider.models,
|
models: provider.models,
|
||||||
permission_modes: provider.kind.permission_modes().to_vec(),
|
permission_modes: provider.kind.permission_modes().to_vec(),
|
||||||
default_permission_mode: provider.kind.default_permission_mode(),
|
default_permission_mode: provider.kind.default_permission_mode(),
|
||||||
|
params: provider.kind.params(),
|
||||||
})
|
})
|
||||||
.collect(),
|
.collect(),
|
||||||
))
|
))
|
||||||
@@ -1586,6 +1596,24 @@ async fn set_permission_mode(
|
|||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
#[serde(deny_unknown_fields)]
|
||||||
|
struct ParamsRequest {
|
||||||
|
/// The whole map, not a patch -- see `SessionManager::set_session_params`.
|
||||||
|
params: std::collections::BTreeMap<String, String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn set_params(
|
||||||
|
State(manager): State<Arc<SessionManager>>,
|
||||||
|
UrlPath(id): UrlPath<String>,
|
||||||
|
axum::Json(body): axum::Json<ParamsRequest>,
|
||||||
|
) -> Result<StatusCode, ApiError> {
|
||||||
|
manager
|
||||||
|
.set_session_params(&id, body.params)
|
||||||
|
.map_err(bad_request)?;
|
||||||
|
Ok(StatusCode::NO_CONTENT)
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
#[serde(deny_unknown_fields)]
|
#[serde(deny_unknown_fields)]
|
||||||
struct NotifyRequest {
|
struct NotifyRequest {
|
||||||
|
|||||||
@@ -363,6 +363,26 @@ pub enum Event {
|
|||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
permission_mode: Option<String>,
|
permission_mode: Option<String>,
|
||||||
},
|
},
|
||||||
|
/// How much context this session's model has to hold a conversation in.
|
||||||
|
///
|
||||||
|
/// The denominator the phone draws [`Event::UsageDelta`]'s `context`
|
||||||
|
/// against, and its own event rather than a field on that one because it
|
||||||
|
/// is not a per-turn measurement: it is fixed when the process starts and
|
||||||
|
/// changes only when a different one is started, which is what a model
|
||||||
|
/// change does. Reported the moment it is known, so the figure and what it
|
||||||
|
/// is out of arrive together rather than the first turn drawing a
|
||||||
|
/// numerator with no denominator.
|
||||||
|
///
|
||||||
|
/// **Only ever sent by a driver that actually knows.** A window nobody has
|
||||||
|
/// measured is not an unlimited one: llama.cpp answers it exactly, because
|
||||||
|
/// the number is a flag the server was started with and `/props` reads it
|
||||||
|
/// back, while a coding CLI's context is the vendor's business and
|
||||||
|
/// nothing in either control protocol states it. Those send nothing, the
|
||||||
|
/// session has no limit, and the phone draws the figure on its own -- see
|
||||||
|
/// `SessionSummary::context_limit`.
|
||||||
|
ContextWindow {
|
||||||
|
tokens: u64,
|
||||||
|
},
|
||||||
/// Per-turn token counts, where the dialect reports them.
|
/// Per-turn token counts, where the dialect reports them.
|
||||||
UsageDelta {
|
UsageDelta {
|
||||||
/// What this turn cost: the tokens it was charged for.
|
/// What this turn cost: the tokens it was charged for.
|
||||||
@@ -494,6 +514,25 @@ pub fn context_tokens(input: u64, cache_creation: u64, cache_read: u64) -> u64 {
|
|||||||
input + cache_creation + cache_read
|
input + cache_creation + cache_read
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The context window after `event`, given what it was before.
|
||||||
|
///
|
||||||
|
/// Beside [`context_after`] because it has the same three readers and the same
|
||||||
|
/// hazard: a figure that outlives what made it true. A model change replaces
|
||||||
|
/// the process, so the window it reports replaces the old one -- and until the
|
||||||
|
/// new one says, there is no answer rather than the previous model's.
|
||||||
|
pub fn context_limit_after(current: Option<u64>, event: &Event) -> Option<u64> {
|
||||||
|
match event {
|
||||||
|
Event::ContextWindow { tokens } => Some(*tokens),
|
||||||
|
// The window belongs to the process, and a stopped one has none. Left
|
||||||
|
// standing, a restarted session on a different model would draw its
|
||||||
|
// occupancy against the previous model's window.
|
||||||
|
Event::Status {
|
||||||
|
state: SessionStatus::Exited,
|
||||||
|
} => None,
|
||||||
|
_ => current,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// The context after `event`, given what it was before.
|
/// The context after `event`, given what it was before.
|
||||||
///
|
///
|
||||||
/// The whole rule in one place, because three readers need the same answer:
|
/// The whole rule in one place, because three readers need the same answer:
|
||||||
@@ -683,6 +722,16 @@ pub trait Driver: Send + Sync {
|
|||||||
/// `/rename` afterwards, which is what puts the same name in its own
|
/// `/rename` afterwards, which is what puts the same name in its own
|
||||||
/// session picker and in what other agents see.
|
/// session picker and in what other agents see.
|
||||||
fn set_title(&self, title: &str);
|
fn set_title(&self, title: &str);
|
||||||
|
/// Takes the session's provider settings, whole.
|
||||||
|
///
|
||||||
|
/// The whole map because it is a form's contents -- an absent key means
|
||||||
|
/// "unset", not "unchanged". A driver applies what it can apply now and
|
||||||
|
/// says so about the rest: the map is also on disk by the time this is
|
||||||
|
/// called, so a setting that only takes effect at the next start is not
|
||||||
|
/// lost, it is waiting. The default is right for a driver with no settings
|
||||||
|
/// of its own, which is every one but llama.cpp -- see
|
||||||
|
/// [`crate::config::DriverKind::params`].
|
||||||
|
fn set_params(&self, _params: &std::collections::BTreeMap<String, String>) {}
|
||||||
/// Runs a command this session's own dialect understands, verbatim --
|
/// Runs a command this session's own dialect understands, verbatim --
|
||||||
/// `/context`, `/usage`, anything a CLI adds next month. A driver with no
|
/// `/context`, `/usage`, anything a CLI adds next month. A driver with no
|
||||||
/// such vocabulary says so with an [`Event::Error`] rather than sending it
|
/// such vocabulary says so with an [`Event::Error`] rather than sending it
|
||||||
|
|||||||
+158
-34
@@ -65,6 +65,32 @@ pub use tools::{DEFAULT_MODE as DEFAULT_PERMISSION_MODE, MODES as PERMISSION_MOD
|
|||||||
use mcp::McpServer;
|
use mcp::McpServer;
|
||||||
use tools::Tools;
|
use tools::Tools;
|
||||||
|
|
||||||
|
/// The sampling half of a session's settings, in the wire's own names.
|
||||||
|
///
|
||||||
|
/// One function because two callers need the identical mapping: the spawn, and
|
||||||
|
/// a later change through [`Driver::set_params`]. A value that will not parse
|
||||||
|
/// as a number is left out rather than passed through -- `llama-server` would
|
||||||
|
/// refuse the whole request for it, which would look like the session breaking
|
||||||
|
/// rather than like one field being wrong.
|
||||||
|
fn sampling_from(
|
||||||
|
params: &std::collections::BTreeMap<String, String>,
|
||||||
|
) -> serde_json::Map<String, Value> {
|
||||||
|
let mut sampling = serde_json::Map::new();
|
||||||
|
for (key, field) in [
|
||||||
|
("temperature", "temperature"),
|
||||||
|
("topP", "top_p"),
|
||||||
|
("topK", "top_k"),
|
||||||
|
("maxTokens", "max_tokens"),
|
||||||
|
] {
|
||||||
|
if let Some(raw) = params.get(key)
|
||||||
|
&& let Ok(number) = raw.parse::<f64>()
|
||||||
|
{
|
||||||
|
sampling.insert(field.to_string(), json!(number));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sampling
|
||||||
|
}
|
||||||
|
|
||||||
/// Exa's own hosted MCP server, which is what a llama session searches the web
|
/// Exa's own hosted MCP server, which is what a llama session searches the web
|
||||||
/// with. The address llama.cpp's web UI offers under "Exa" in its recommended
|
/// with. The address llama.cpp's web UI offers under "Exa" in its recommended
|
||||||
/// servers, so a session here reaches the same thing that UI does.
|
/// servers, so a session here reaches the same thing that UI does.
|
||||||
@@ -76,6 +102,11 @@ pub const EXA_MCP_URL: &str = "https://mcp.exa.ai/mcp";
|
|||||||
/// is the escape for a machine where drafting turns out not to pay.
|
/// is the escape for a machine where drafting turns out not to pay.
|
||||||
const SPECULATIVE: &str = "speculative";
|
const SPECULATIVE: &str = "speculative";
|
||||||
|
|
||||||
|
/// The spawn parameter naming which built-in tools a session gets, as
|
||||||
|
/// `llama-server`'s own comma-separated list. Absent is all of them, and
|
||||||
|
/// `"none"` is the way to ask for a session that only talks.
|
||||||
|
const TOOLS: &str = "tools";
|
||||||
|
|
||||||
/// How many times one message may go round the call-a-tool loop.
|
/// How many times one message may go round the call-a-tool loop.
|
||||||
///
|
///
|
||||||
/// A bound rather than a budget: a small model that has decided to read the
|
/// A bound rather than a budget: a small model that has decided to read the
|
||||||
@@ -221,8 +252,10 @@ struct Shared {
|
|||||||
/// Where this session's process record lives, so [`Driver::stop`] can find
|
/// Where this session's process record lives, so [`Driver::stop`] can find
|
||||||
/// the server it has to end.
|
/// the server it has to end.
|
||||||
session_dir: PathBuf,
|
session_dir: PathBuf,
|
||||||
/// Sampling settings chosen at spawn, sent with every request.
|
/// Sampling settings, sent with every request. Behind a lock because they
|
||||||
sampling: serde_json::Map<String, Value>,
|
/// are changeable while the session runs: they ride on the next request,
|
||||||
|
/// so unlike the server's own flags there is nothing to reload.
|
||||||
|
sampling: Mutex<serde_json::Map<String, Value>>,
|
||||||
/// The session's working directory, which is where its tools act. `None`
|
/// The session's working directory, which is where its tools act. `None`
|
||||||
/// leaves that to `llama-server`, which is the honest answer rather than a
|
/// leaves that to `llama-server`, which is the honest answer rather than a
|
||||||
/// guess at one.
|
/// guess at one.
|
||||||
@@ -289,26 +322,14 @@ impl LlamaDriver {
|
|||||||
"a llama.cpp session needs a model -- one of the downloaded ones, by its key",
|
"a llama.cpp session needs a model -- one of the downloaded ones, by its key",
|
||||||
)?;
|
)?;
|
||||||
|
|
||||||
let mut sampling = serde_json::Map::new();
|
let sampling = sampling_from(&meta.params);
|
||||||
for (key, field) in [
|
|
||||||
("temperature", "temperature"),
|
|
||||||
("topP", "top_p"),
|
|
||||||
("topK", "top_k"),
|
|
||||||
("maxTokens", "max_tokens"),
|
|
||||||
] {
|
|
||||||
if let Some(raw) = meta.params.get(key)
|
|
||||||
&& let Ok(number) = raw.parse::<f64>()
|
|
||||||
{
|
|
||||||
sampling.insert(field.to_string(), json!(number));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let driver = Self {
|
let driver = Self {
|
||||||
shared: Arc::new(Shared {
|
shared: Arc::new(Shared {
|
||||||
sink,
|
sink,
|
||||||
transcript: transcript.to_path_buf(),
|
transcript: transcript.to_path_buf(),
|
||||||
session_dir: session_dir.to_path_buf(),
|
session_dir: session_dir.to_path_buf(),
|
||||||
sampling,
|
sampling: Mutex::new(sampling),
|
||||||
cwd: meta
|
cwd: meta
|
||||||
.cwd
|
.cwd
|
||||||
.as_ref()
|
.as_ref()
|
||||||
@@ -403,6 +424,15 @@ impl LlamaDriver {
|
|||||||
"{model} loaded and answering at {endpoint} with {} tools",
|
"{model} loaded and answering at {endpoint} with {} tools",
|
||||||
tools.offered().map_or(0, |offered| offered.len()),
|
tools.offered().map_or(0, |offered| offered.len()),
|
||||||
);
|
);
|
||||||
|
// Asked now rather than carried from the spawn flags: a
|
||||||
|
// session that named no context size gets the model's
|
||||||
|
// own, which only the loaded server knows, and one that
|
||||||
|
// named an impossible one gets whatever it settled for.
|
||||||
|
// Either way this is the measurement rather than the
|
||||||
|
// request.
|
||||||
|
if let Some(window) = context_window(&endpoint) {
|
||||||
|
shared.emit(Event::ContextWindow { tokens: window });
|
||||||
|
}
|
||||||
Serving::Ready {
|
Serving::Ready {
|
||||||
endpoint: endpoint.clone(),
|
endpoint: endpoint.clone(),
|
||||||
tools: Arc::new(tools),
|
tools: Arc::new(tools),
|
||||||
@@ -531,14 +561,12 @@ fn spawn_server(
|
|||||||
"127.0.0.1".into(),
|
"127.0.0.1".into(),
|
||||||
"--port".into(),
|
"--port".into(),
|
||||||
forward.there.to_string(),
|
forward.there.to_string(),
|
||||||
// The built-in agent tools -- read, search, edit, shell. Every one of
|
// The built-in agent tools -- read, search, edit, shell. All of them
|
||||||
// them, because a session offered a subset is a session that says "I
|
// unless the session says otherwise, because whether a particular call
|
||||||
// can't do that" about something it was installed to do, and the
|
// should happen is the permission gate's question rather than a flag's.
|
||||||
// question of whether a particular call should happen is the
|
// They run on the machine serving the model, which is the machine the
|
||||||
// permission gate's rather than a flag's. They run on the machine
|
// files are on.
|
||||||
// serving the model, which is the machine the files are on.
|
//
|
||||||
"--tools".into(),
|
|
||||||
"all".into(),
|
|
||||||
// One slot, not the four `llama-server` picks on its own. A session is
|
// One slot, not the four `llama-server` picks on its own. A session is
|
||||||
// one conversation making one request at a time -- the driver holds a
|
// one conversation making one request at a time -- the driver holds a
|
||||||
// second message until the turn ends -- so the other three are context
|
// second message until the turn ends -- so the other three are context
|
||||||
@@ -553,6 +581,27 @@ fn spawn_server(
|
|||||||
"-np".into(),
|
"-np".into(),
|
||||||
"1".into(),
|
"1".into(),
|
||||||
];
|
];
|
||||||
|
// Settable because it is not free: the definitions of all seven are around
|
||||||
|
// 2,000 tokens of every prompt -- measured at 2,191 against 1,322 for two
|
||||||
|
// of them -- which on a small context window is a quarter of it spent
|
||||||
|
// before anything is said.
|
||||||
|
//
|
||||||
|
// "none" omits the flag rather than passing it on: `--tools none` is
|
||||||
|
// `tools setup failed: unknown tool "none"` and a server that exits, since
|
||||||
|
// the argument is a list of tool names and no-tools is what having no flag
|
||||||
|
// means.
|
||||||
|
match meta.params.get(TOOLS).map(|chosen| chosen.trim()) {
|
||||||
|
Some("none") => {}
|
||||||
|
chosen => {
|
||||||
|
args.push("--tools".into());
|
||||||
|
args.push(
|
||||||
|
chosen
|
||||||
|
.filter(|c| !c.is_empty())
|
||||||
|
.unwrap_or("all")
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
// A model that carries a multi-token-prediction head drafts with it, which
|
// A model that carries a multi-token-prediction head drafts with it, which
|
||||||
// is most of a 50% speed-up for free -- the tensors are in the file
|
// is most of a 50% speed-up for free -- the tensors are in the file
|
||||||
// whether or not they are used, and without the flag `llama-server` says
|
// whether or not they are used, and without the flag `llama-server` says
|
||||||
@@ -573,6 +622,11 @@ fn spawn_server(
|
|||||||
("contextSize", "-c"),
|
("contextSize", "-c"),
|
||||||
("gpuLayers", "-ngl"),
|
("gpuLayers", "-ngl"),
|
||||||
("threads", "-t"),
|
("threads", "-t"),
|
||||||
|
// How far ahead the draft head guesses. Not defaulted here: 2 measured
|
||||||
|
// 7% faster than llama.cpp's 3 on this machine's GPU, once, which is
|
||||||
|
// a reason to make the knob reachable and not a reason to move it for
|
||||||
|
// everybody.
|
||||||
|
("specDraftNMax", "--spec-draft-n-max"),
|
||||||
] {
|
] {
|
||||||
if let Some(value) = meta.params.get(key) {
|
if let Some(value) = meta.params.get(key) {
|
||||||
args.push(flag.to_string());
|
args.push(flag.to_string());
|
||||||
@@ -854,7 +908,10 @@ fn converse(
|
|||||||
if shared.cancel.load(Ordering::SeqCst) {
|
if shared.cancel.load(Ordering::SeqCst) {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
let reply = generate(endpoint, &messages, tools, &shared.sampling, shared)?;
|
// Read per call rather than per turn, so a sampling change made while
|
||||||
|
// a long turn is running reaches the rest of it.
|
||||||
|
let sampling = shared.sampling.lock().unwrap().clone();
|
||||||
|
let reply = generate(endpoint, &messages, tools, &sampling, shared)?;
|
||||||
let calls = reply.calls;
|
let calls = reply.calls;
|
||||||
messages.push(Message {
|
messages.push(Message {
|
||||||
tool_calls: calls.iter().map(Call::wire).collect(),
|
tool_calls: calls.iter().map(Call::wire).collect(),
|
||||||
@@ -1046,6 +1103,49 @@ impl Driver for LlamaDriver {
|
|||||||
// is called, and the rename has already happened where the name lives.
|
// is called, and the rename has already happened where the name lives.
|
||||||
fn set_title(&self, _title: &str) {}
|
fn set_title(&self, _title: &str) {}
|
||||||
|
|
||||||
|
/// Takes new settings: the sampling half now, and says so about the rest.
|
||||||
|
///
|
||||||
|
/// The split is what [`crate::config::ParamSpec::restart`] describes, and
|
||||||
|
/// it is said out loud rather than left to the screen, because the screen
|
||||||
|
/// can only say what a setting *usually* does -- this is the one place
|
||||||
|
/// that knows whether this session's server was started with the old
|
||||||
|
/// value. A session already stopped needs no such note: its next start
|
||||||
|
/// will read all of them.
|
||||||
|
fn set_params(&self, params: &std::collections::BTreeMap<String, String>) {
|
||||||
|
*self.shared.sampling.lock().unwrap() = sampling_from(params);
|
||||||
|
// Only the settings that actually differ from what this session's
|
||||||
|
// server was started with. Listing every restart-only one on every
|
||||||
|
// save would be a wall of text about nothing having changed.
|
||||||
|
let waiting: Vec<&str> = crate::config::DriverKind::LlamaCpp
|
||||||
|
.params()
|
||||||
|
.iter()
|
||||||
|
.filter(|spec| {
|
||||||
|
spec.restart && params.get(spec.key) != self.respawn.meta.params.get(spec.key)
|
||||||
|
})
|
||||||
|
.map(|spec| spec.label)
|
||||||
|
.collect();
|
||||||
|
// Nothing to say to a session with no server: its next start reads all
|
||||||
|
// of them, which is what the note would have been asking for.
|
||||||
|
let running = matches!(&*self.shared.serving.lock().unwrap(), Serving::Ready { .. });
|
||||||
|
if !waiting.is_empty() && running {
|
||||||
|
let one = waiting.len() == 1;
|
||||||
|
self.shared.emit(Event::Error {
|
||||||
|
message: format!(
|
||||||
|
"{} {} saved. {} when this session's server next starts -- stop and start \
|
||||||
|
the session, or change its model, to load {} now.",
|
||||||
|
waiting.join(", "),
|
||||||
|
if one { "is" } else { "are" },
|
||||||
|
if one {
|
||||||
|
"It takes effect"
|
||||||
|
} else {
|
||||||
|
"They take effect"
|
||||||
|
},
|
||||||
|
if one { "it" } else { "them" },
|
||||||
|
),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
fn set_permission_mode(&self, mode: &str) {
|
fn set_permission_mode(&self, mode: &str) {
|
||||||
if !tools::MODES.contains(&mode) {
|
if !tools::MODES.contains(&mode) {
|
||||||
self.shared.emit(Event::Error {
|
self.shared.emit(Event::Error {
|
||||||
@@ -1436,6 +1536,23 @@ fn log_tail(session_dir: &Path) -> String {
|
|||||||
/// How much of that log to carry into a message somebody reads on a phone.
|
/// How much of that log to carry into a message somebody reads on a phone.
|
||||||
const LOG_TAIL_LINES: usize = 6;
|
const LOG_TAIL_LINES: usize = 6;
|
||||||
|
|
||||||
|
/// How many tokens this server can hold, from the server itself.
|
||||||
|
///
|
||||||
|
/// `/props` reports the per-slot context, which is the whole of it because
|
||||||
|
/// this driver always starts one slot -- see the `-np` argument. `None` for a
|
||||||
|
/// server that would not answer, which draws as no ceiling rather than as a
|
||||||
|
/// guessed one.
|
||||||
|
fn context_window(endpoint: &str) -> Option<u64> {
|
||||||
|
ureq::get(format!("{endpoint}/props"))
|
||||||
|
.call()
|
||||||
|
.ok()?
|
||||||
|
.body_mut()
|
||||||
|
.read_json::<Value>()
|
||||||
|
.ok()?
|
||||||
|
.pointer("/default_generation_settings/n_ctx")
|
||||||
|
.and_then(Value::as_u64)
|
||||||
|
}
|
||||||
|
|
||||||
/// One streamed completion: posts the conversation, emits each text delta as
|
/// One streamed completion: posts the conversation, emits each text delta as
|
||||||
/// it arrives, and assembles whatever tool calls came with it.
|
/// it arrives, and assembles whatever tool calls came with it.
|
||||||
///
|
///
|
||||||
@@ -1491,9 +1608,16 @@ fn generate(
|
|||||||
// at once interleaves their fragments, each tagged with its index.
|
// at once interleaves their fragments, each tagged with its index.
|
||||||
let mut calls: Vec<Call> = Vec::new();
|
let mut calls: Vec<Call> = Vec::new();
|
||||||
let mut tokens = 0u64;
|
let mut tokens = 0u64;
|
||||||
// The prompt side only, which is what the model is holding -- the same
|
// What the model is holding when this call ends: the prompt it was given
|
||||||
// definition the other dialects report, so one word on the phone means one
|
// plus the reply it produced, which is exactly what the next call's prompt
|
||||||
// thing whichever kind of session it is.
|
// begins with.
|
||||||
|
//
|
||||||
|
// The prompt alone was wrong, and measurably: a five-word reply reported
|
||||||
|
// 2,042 against a slot holding 2,355 (2026-09-19, checked against
|
||||||
|
// `stop processing: n_tokens` in the server's own log). The gap is the
|
||||||
|
// reply, so it grows with how much the model just said -- which is the
|
||||||
|
// worst direction for a figure somebody is watching to see how much room
|
||||||
|
// is left.
|
||||||
let mut context = None;
|
let mut context = None;
|
||||||
for line in std::io::BufRead::lines(reader) {
|
for line in std::io::BufRead::lines(reader) {
|
||||||
if shared.cancel.load(Ordering::SeqCst) {
|
if shared.cancel.load(Ordering::SeqCst) {
|
||||||
@@ -1510,13 +1634,13 @@ fn generate(
|
|||||||
let Ok(chunk) = serde_json::from_str::<Value>(payload) else {
|
let Ok(chunk) = serde_json::from_str::<Value>(payload) else {
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
if let Some(usage) = chunk.get("usage") {
|
// One figure answering both questions, which for this dialect it
|
||||||
if let Some(total) = usage.get("total_tokens").and_then(Value::as_u64) {
|
// genuinely does: what the call was charged for and what the model is
|
||||||
|
// left holding are the same tokens, because nothing here is billed and
|
||||||
|
// the whole conversation is resent every time.
|
||||||
|
if let Some(total) = chunk.pointer("/usage/total_tokens").and_then(Value::as_u64) {
|
||||||
tokens = total;
|
tokens = total;
|
||||||
}
|
context = Some(total);
|
||||||
if let Some(prompt) = usage.get("prompt_tokens").and_then(Value::as_u64) {
|
|
||||||
context = Some(prompt);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
let Some(delta) = chunk.pointer("/choices/0/delta") else {
|
let Some(delta) = chunk.pointer("/choices/0/delta") else {
|
||||||
continue;
|
continue;
|
||||||
|
|||||||
@@ -57,18 +57,29 @@ impl Tools {
|
|||||||
/// Asks a ready `llama-server` what it offers and adds what the MCP
|
/// Asks a ready `llama-server` what it offers and adds what the MCP
|
||||||
/// servers offered.
|
/// servers offered.
|
||||||
///
|
///
|
||||||
/// A server started without `--tools` answers with an empty list, and a
|
/// A session with no built-in tools -- or none at all -- is a perfectly
|
||||||
/// session with only MCP tools is a perfectly good session -- so nothing
|
/// good session, so nothing here treats "no tools" as a failure. What *is*
|
||||||
/// here treats "no tools" as a failure. What *is* a failure is not being
|
/// a failure is not being able to ask at all, because that is the same
|
||||||
/// able to ask, because that is the same server the conversation is about
|
/// server the conversation is about to go to.
|
||||||
/// to go to.
|
|
||||||
pub fn discover(endpoint: &str, mcp: Vec<Arc<Mutex<McpServer>>>) -> Result<Self> {
|
pub fn discover(endpoint: &str, mcp: Vec<Arc<Mutex<McpServer>>>) -> Result<Self> {
|
||||||
let catalog: Vec<Value> = ureq::get(format!("{endpoint}/tools"))
|
let mut response = ureq::get(format!("{endpoint}/tools"))
|
||||||
|
// A server started with no `--tools` answers **403** here, not an
|
||||||
|
// empty list -- the route is off rather than empty. Read as a
|
||||||
|
// failure that was a session which never started, for the one
|
||||||
|
// setting whose whole purpose is to have no tools.
|
||||||
|
.config()
|
||||||
|
.http_status_as_error(false)
|
||||||
|
.build()
|
||||||
.call()
|
.call()
|
||||||
.context("asking llama-server which tools it has")?
|
.context("asking llama-server which tools it has")?;
|
||||||
|
let catalog: Vec<Value> = if response.status().is_success() {
|
||||||
|
response
|
||||||
.body_mut()
|
.body_mut()
|
||||||
.read_json()
|
.read_json()
|
||||||
.context("reading llama-server's tool list")?;
|
.context("reading llama-server's tool list")?
|
||||||
|
} else {
|
||||||
|
Vec::new()
|
||||||
|
};
|
||||||
let mut definitions = Vec::new();
|
let mut definitions = Vec::new();
|
||||||
let mut server = HashMap::new();
|
let mut server = HashMap::new();
|
||||||
for entry in &catalog {
|
for entry in &catalog {
|
||||||
|
|||||||
@@ -36,7 +36,8 @@ use crate::config::{
|
|||||||
use claude::ClaudeDriver;
|
use claude::ClaudeDriver;
|
||||||
use codex::CodexDriver;
|
use codex::CodexDriver;
|
||||||
use driver::{
|
use driver::{
|
||||||
AttachmentRef, Driver, Event, EventSink, SessionCommand, SessionStatus, Unqueued, context_after,
|
AttachmentRef, Driver, Event, EventSink, SessionCommand, SessionStatus, Unqueued,
|
||||||
|
context_after, context_limit_after,
|
||||||
};
|
};
|
||||||
use echo::EchoDriver;
|
use echo::EchoDriver;
|
||||||
use llama::LlamaDriver;
|
use llama::LlamaDriver;
|
||||||
@@ -213,6 +214,17 @@ pub struct SessionInfo {
|
|||||||
/// are different answers and the phone draws them differently.
|
/// are different answers and the phone draws them differently.
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
pub context_tokens: Option<u64>,
|
pub context_tokens: Option<u64>,
|
||||||
|
/// What that figure is out of -- see [`Event::ContextWindow`]. Absent
|
||||||
|
/// where the provider does not say, which is a third state again: not a
|
||||||
|
/// session with room to spare, and not one whose occupancy is unknown,
|
||||||
|
/// but one whose occupancy is known and whose ceiling is not.
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pub context_limit: Option<u64>,
|
||||||
|
/// The provider settings this session was launched with, as the settings
|
||||||
|
/// dialog has to open on them -- what a control is *set to* is not
|
||||||
|
/// derivable from what the provider *offers*.
|
||||||
|
#[serde(skip_serializing_if = "std::collections::BTreeMap::is_empty")]
|
||||||
|
pub params: std::collections::BTreeMap<String, String>,
|
||||||
/// The longest edge an image should have by the time it gets here.
|
/// The longest edge an image should have by the time it gets here.
|
||||||
/// Absent rather than a large number, because "no limit" and "a limit
|
/// Absent rather than a large number, because "no limit" and "a limit
|
||||||
/// that happens to be big" are different answers.
|
/// that happens to be big" are different answers.
|
||||||
@@ -415,6 +427,7 @@ struct Shared {
|
|||||||
/// the session row so a phone opening a long conversation has the real
|
/// the session row so a phone opening a long conversation has the real
|
||||||
/// figure rather than whatever its newest page mentions.
|
/// figure rather than whatever its newest page mentions.
|
||||||
context_tokens: Mutex<Option<u64>>,
|
context_tokens: Mutex<Option<u64>>,
|
||||||
|
context_limit: Mutex<Option<u64>>,
|
||||||
/// Mirrored out of the config so the pump can read it without taking
|
/// Mirrored out of the config so the pump can read it without taking
|
||||||
/// the manager's lock -- the pump runs underneath the manager, and
|
/// the manager's lock -- the pump runs underneath the manager, and
|
||||||
/// reaching back up would invert that.
|
/// reaching back up would invert that.
|
||||||
@@ -579,6 +592,11 @@ impl LiveSession {
|
|||||||
effort: current.effort.clone(),
|
effort: current.effort.clone(),
|
||||||
takes_effort: kind.is_some_and(DriverKind::takes_effort),
|
takes_effort: kind.is_some_and(DriverKind::takes_effort),
|
||||||
context_tokens: *self.shared.context_tokens.lock().unwrap(),
|
context_tokens: *self.shared.context_tokens.lock().unwrap(),
|
||||||
|
context_limit: *self.shared.context_limit.lock().unwrap(),
|
||||||
|
// From the config for the reason `effort` above is: the settings
|
||||||
|
// the process was started with are what a restart-only control has
|
||||||
|
// to show, and this is where they are kept.
|
||||||
|
params: current.params.clone(),
|
||||||
notify: *self.shared.notify.lock().unwrap(),
|
notify: *self.shared.notify.lock().unwrap(),
|
||||||
auto_resume: current.auto_resume,
|
auto_resume: current.auto_resume,
|
||||||
auto_resume_message: resume_message(current),
|
auto_resume_message: resume_message(current),
|
||||||
@@ -1104,6 +1122,8 @@ impl SessionManager {
|
|||||||
takes_effort: kind_of(&inner.config, &meta.machine, &meta.provider)
|
takes_effort: kind_of(&inner.config, &meta.machine, &meta.provider)
|
||||||
.is_some_and(DriverKind::takes_effort),
|
.is_some_and(DriverKind::takes_effort),
|
||||||
context_tokens: None,
|
context_tokens: None,
|
||||||
|
context_limit: None,
|
||||||
|
params: meta.params.clone(),
|
||||||
max_image_edge: kind_of(&inner.config, &meta.machine, &meta.provider)
|
max_image_edge: kind_of(&inner.config, &meta.machine, &meta.provider)
|
||||||
.and_then(DriverKind::max_image_edge),
|
.and_then(DriverKind::max_image_edge),
|
||||||
usage_provider: kind_of(&inner.config, &meta.machine, &meta.provider)
|
usage_provider: kind_of(&inner.config, &meta.machine, &meta.provider)
|
||||||
@@ -1319,6 +1339,43 @@ impl SessionManager {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Changes this session's provider settings, live and persisted.
|
||||||
|
///
|
||||||
|
/// The whole map rather than one key, because that is what a settings
|
||||||
|
/// screen has: a form is submitted as its contents, and merging one field
|
||||||
|
/// at a time would make clearing a field indistinguishable from not
|
||||||
|
/// mentioning it. An absent key *is* the instruction to unset it.
|
||||||
|
///
|
||||||
|
/// Persisted first for the reason the model is: the config answers what to
|
||||||
|
/// launch with next time, which is the whole of what a restart-only
|
||||||
|
/// setting means. The driver is then told, and takes what it can use now.
|
||||||
|
pub fn set_session_params(
|
||||||
|
&self,
|
||||||
|
id: &str,
|
||||||
|
params: std::collections::BTreeMap<String, String>,
|
||||||
|
) -> Result<()> {
|
||||||
|
let mut inner = self.inner.write().unwrap();
|
||||||
|
if !inner.config.sessions.iter().any(|meta| meta.id == id) {
|
||||||
|
bail!("no session {id}");
|
||||||
|
}
|
||||||
|
let mut candidate = inner.config.clone();
|
||||||
|
for meta in candidate.sessions.iter_mut().filter(|meta| meta.id == id) {
|
||||||
|
meta.params = params.clone();
|
||||||
|
}
|
||||||
|
candidate.save(&self.config_path)?;
|
||||||
|
inner.config = candidate;
|
||||||
|
// Told to the driver where there is one, and simply saved where there
|
||||||
|
// is not. Deliberately not `ask`: a session with no process has not
|
||||||
|
// failed to take these, it has taken them in the only way that matters
|
||||||
|
// -- its next start reads them -- and adjusting the settings of a
|
||||||
|
// stopped session so that starting it uses them is the ordinary thing
|
||||||
|
// to do, not an error to report.
|
||||||
|
if let Some(driver) = inner.live.get(id).and_then(|session| session.driver()) {
|
||||||
|
driver.set_params(¶ms);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
/// Turns this session's notifications on or off, live and persisted --
|
/// Turns this session's notifications on or off, live and persisted --
|
||||||
/// both, or the switch moves back on its own at the next restart.
|
/// both, or the switch moves back on its own at the next restart.
|
||||||
///
|
///
|
||||||
@@ -2394,6 +2451,7 @@ fn launch(
|
|||||||
model: Mutex::new(meta.model.clone()),
|
model: Mutex::new(meta.model.clone()),
|
||||||
permission_mode: Mutex::new(meta.permission_mode.clone()),
|
permission_mode: Mutex::new(meta.permission_mode.clone()),
|
||||||
context_tokens: Mutex::new(transcript.context_tokens()),
|
context_tokens: Mutex::new(transcript.context_tokens()),
|
||||||
|
context_limit: Mutex::new(transcript.context_limit()),
|
||||||
notify: Mutex::new(meta.notify),
|
notify: Mutex::new(meta.notify),
|
||||||
written: Mutex::new(0),
|
written: Mutex::new(0),
|
||||||
});
|
});
|
||||||
@@ -2674,6 +2732,10 @@ async fn pump(
|
|||||||
let mut context = shared.context_tokens.lock().unwrap();
|
let mut context = shared.context_tokens.lock().unwrap();
|
||||||
*context = context_after(*context, &event);
|
*context = context_after(*context, &event);
|
||||||
}
|
}
|
||||||
|
{
|
||||||
|
let mut limit = shared.context_limit.lock().unwrap();
|
||||||
|
*limit = context_limit_after(*limit, &event);
|
||||||
|
}
|
||||||
// Nothing changed, so there is nothing to record. Both of these
|
// Nothing changed, so there is nothing to record. Both of these
|
||||||
// repeat: an imported session reads the turn state off its file's
|
// repeat: an imported session reads the turn state off its file's
|
||||||
// newest record on every sync, and the CLI restates its model and
|
// newest record on every sync, and the CLI restates its model and
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ use std::path::Path;
|
|||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
use super::driver::{Event, SessionStatus, context_after};
|
use super::driver::{Event, SessionStatus, context_after, context_limit_after};
|
||||||
|
|
||||||
/// One transcript line: an [`Event`] plus its position and time. The event
|
/// One transcript line: an [`Event`] plus its position and time. The event
|
||||||
/// is flattened so the wire shape stays one flat object.
|
/// is flattened so the wire shape stays one flat object.
|
||||||
@@ -34,6 +34,7 @@ pub struct Transcript {
|
|||||||
last_status: Option<SessionStatus>,
|
last_status: Option<SessionStatus>,
|
||||||
last_activity: Option<f64>,
|
last_activity: Option<f64>,
|
||||||
context_tokens: Option<u64>,
|
context_tokens: Option<u64>,
|
||||||
|
context_limit: Option<u64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Transcript {
|
impl Transcript {
|
||||||
@@ -68,6 +69,12 @@ impl Transcript {
|
|||||||
context_tokens: existing
|
context_tokens: existing
|
||||||
.iter()
|
.iter()
|
||||||
.fold(None, |current, entry| context_after(current, &entry.event)),
|
.fold(None, |current, entry| context_after(current, &entry.event)),
|
||||||
|
// The same fold for the same reason: a session whose process has
|
||||||
|
// since exited has no window, and the newest `ContextWindow` line
|
||||||
|
// alone would not know that.
|
||||||
|
context_limit: existing.iter().fold(None, |current, entry| {
|
||||||
|
context_limit_after(current, &entry.event)
|
||||||
|
}),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -107,6 +114,12 @@ impl Transcript {
|
|||||||
self.context_tokens
|
self.context_tokens
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// What that figure is out of, as of opening, and `None` where this
|
||||||
|
/// session's provider does not say.
|
||||||
|
pub fn context_limit(&self) -> Option<u64> {
|
||||||
|
self.context_limit
|
||||||
|
}
|
||||||
|
|
||||||
/// Appends `event`, assigning it the next sequence number. Flushed per
|
/// Appends `event`, assigning it the next sequence number. Flushed per
|
||||||
/// event: each line is tiny, and the transcript is the source of truth a
|
/// event: each line is tiny, and the transcript is the source of truth a
|
||||||
/// crash must not lose the tail of.
|
/// crash must not lose the tail of.
|
||||||
|
|||||||
Reference in new issue
Block a user