Declare provider settings, and give the context figure a denominator
Two things a session could not say, and one it was saying wrongly.
**Every provider setting is reachable.** `-np 1`, the MTP draft depth, the
tool set, the sampling parameters -- most were hardcoded to what measured
best on this machine, which is right as a default and wrong as a constant:
the next machine has a different GPU and a different core count, and
nobody running this app can edit the source. `DriverKind::params` now
declares what a provider takes -- key, label, shape, what blank means, and
whether a change waits for a restart -- and the phone renders whatever
arrives, on the spawn form and in the session settings dialog. Adding a
setting to a driver is one entry in that table and no app change.
`POST /sessions/{id}/params` takes the whole map, so an absent key is the
instruction to unset; the sampling half applies at once and the session is
told in words which of the rest are waiting for a restart.
`tools` is one of them, because it is the biggest lever on a tight
context: the seven built-in definitions are ~1,300 tokens of every prompt
(2,191 against 887 with none). `"none"` omits the flag rather than passing
it on, since `--tools none` is `unknown tool "none"` and a server that
exits.
**The context figure has a denominator.** `Event::ContextWindow` carries
it, read from `llama-server`'s `/props` once the model is up -- the
measurement rather than the request, since a session that named no context
size gets the model's own. Neither coding CLI states its window, so those
keep the bare figure: "2,042" and "2,042 / 8,192" are deliberately
different-looking, and a missing ceiling is never drawn as a proportion of
an assumed one.
**And the numerator was wrong**, by the length of the last reply: it was
the prompt alone, so a five-word answer reported 2,042 against a slot
holding 2,355. It is the turn's total now, which matches `llama-server`'s
own `n_tokens` to within a token.
Two defects the review found, both of which would have shipped: changing
settings on a *stopped* session reported "no process running, so it can't
take new settings", when a stopped session is exactly when you would set
them for the next start; and `GET /tools` answers **403** rather than an
empty list on a server started without `--tools`, so reading it as a
failure made the no-tools session one that never started.
Verified against real models: settings spawned and changed live, the
restart note, a session with two tools and one with none, and the counter
checked against the server's own slot occupancy each time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ac476ab0c9
commit
81ab564a09
18 files changed
+831
-115
No files matched your search
@@ -363,6 +363,26 @@ pub enum Event {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
permission_mode: Option<String>,
|
||||
},
|
||||
/// How much context this session's model has to hold a conversation in.
|
||||
///
|
||||
/// The denominator the phone draws [`Event::UsageDelta`]'s `context`
|
||||
/// against, and its own event rather than a field on that one because it
|
||||
/// is not a per-turn measurement: it is fixed when the process starts and
|
||||
/// changes only when a different one is started, which is what a model
|
||||
/// change does. Reported the moment it is known, so the figure and what it
|
||||
/// is out of arrive together rather than the first turn drawing a
|
||||
/// numerator with no denominator.
|
||||
///
|
||||
/// **Only ever sent by a driver that actually knows.** A window nobody has
|
||||
/// measured is not an unlimited one: llama.cpp answers it exactly, because
|
||||
/// the number is a flag the server was started with and `/props` reads it
|
||||
/// back, while a coding CLI's context is the vendor's business and
|
||||
/// nothing in either control protocol states it. Those send nothing, the
|
||||
/// session has no limit, and the phone draws the figure on its own -- see
|
||||
/// `SessionSummary::context_limit`.
|
||||
ContextWindow {
|
||||
tokens: u64,
|
||||
},
|
||||
/// Per-turn token counts, where the dialect reports them.
|
||||
UsageDelta {
|
||||
/// What this turn cost: the tokens it was charged for.
|
||||
@@ -494,6 +514,25 @@ pub fn context_tokens(input: u64, cache_creation: u64, cache_read: u64) -> u64 {
|
||||
input + cache_creation + cache_read
|
||||
}
|
||||
|
||||
/// The context window after `event`, given what it was before.
|
||||
///
|
||||
/// Beside [`context_after`] because it has the same three readers and the same
|
||||
/// hazard: a figure that outlives what made it true. A model change replaces
|
||||
/// the process, so the window it reports replaces the old one -- and until the
|
||||
/// new one says, there is no answer rather than the previous model's.
|
||||
pub fn context_limit_after(current: Option<u64>, event: &Event) -> Option<u64> {
|
||||
match event {
|
||||
Event::ContextWindow { tokens } => Some(*tokens),
|
||||
// The window belongs to the process, and a stopped one has none. Left
|
||||
// standing, a restarted session on a different model would draw its
|
||||
// occupancy against the previous model's window.
|
||||
Event::Status {
|
||||
state: SessionStatus::Exited,
|
||||
} => None,
|
||||
_ => current,
|
||||
}
|
||||
}
|
||||
|
||||
/// The context after `event`, given what it was before.
|
||||
///
|
||||
/// The whole rule in one place, because three readers need the same answer:
|
||||
@@ -683,6 +722,16 @@ pub trait Driver: Send + Sync {
|
||||
/// `/rename` afterwards, which is what puts the same name in its own
|
||||
/// session picker and in what other agents see.
|
||||
fn set_title(&self, title: &str);
|
||||
/// Takes the session's provider settings, whole.
|
||||
///
|
||||
/// The whole map because it is a form's contents -- an absent key means
|
||||
/// "unset", not "unchanged". A driver applies what it can apply now and
|
||||
/// says so about the rest: the map is also on disk by the time this is
|
||||
/// called, so a setting that only takes effect at the next start is not
|
||||
/// lost, it is waiting. The default is right for a driver with no settings
|
||||
/// of its own, which is every one but llama.cpp -- see
|
||||
/// [`crate::config::DriverKind::params`].
|
||||
fn set_params(&self, _params: &std::collections::BTreeMap<String, String>) {}
|
||||
/// Runs a command this session's own dialect understands, verbatim --
|
||||
/// `/context`, `/usage`, anything a CLI adds next month. A driver with no
|
||||
/// such vocabulary says so with an [`Event::Error`] rather than sending it
|
||||
|
||||
Reference in new issue
Block a user