Report the context a session holds, not what it has spent
The number on the status row was a running total of tokens spent, so it could only ever climb: a session compacted from 128k down to 10k, or cleared outright, went on reporting the larger figure, and disagreed with the divider directly above it saying what the compaction had recovered. It now reports what the model is holding -- prompt plus both cache figures -- folded through `driver::context_after`, which is the one rule the pump, the transcript and the phone all use: a turn sets it, a compaction replaces it with what the compaction measured, and a clear leaves it unmeasured. Unmeasured says so in words, because an empty context and one nobody has counted used to look identical. Taken from the turn's last assistant message rather than its `result`: measured against CLI 2.1.237, a two-message turn reported a cache read of 40,211, being 14,259 and 25,952 -- the same conversation counted twice, and no size the model ever held.
This commit is contained in:
1 parent
81c8a57181
commit
5e11b9da80
12 files changed
+436
-124
No files matched your search
@@ -181,8 +181,18 @@ it touches the transcript or the phone:
|
|||||||
not just the one that answered (added 2026-08-24, same reasoning as
|
not just the one that answered (added 2026-08-24, same reasoning as
|
||||||
`UserMessage`).
|
`UserMessage`).
|
||||||
- `Status { state }` — idle / running / awaiting-input / compacting / exited.
|
- `Status { state }` — idle / running / awaiting-input / compacting / exited.
|
||||||
- `UsageDelta { tokens }` — per-turn token counts where the dialect reports
|
- `UsageDelta { tokens, context }` — what a turn cost, and how much the
|
||||||
them (both do).
|
model was holding when it ended, where the dialect reports them (both do).
|
||||||
|
`context` is prompt plus both cache figures, taken from the **last
|
||||||
|
assistant message** rather than the turn's `result`: measured 2026-08-30
|
||||||
|
against CLI 2.1.237, the result adds a turn's messages up, so its cache
|
||||||
|
read of 40,211 was the same conversation counted twice and no size the
|
||||||
|
model ever held. It is carried rather than summed by readers because it
|
||||||
|
goes *down* — a compaction replaces it with what the compaction reports,
|
||||||
|
and a clear leaves it unmeasured. `driver::context_after` is that rule,
|
||||||
|
and the phone folds with the same one (2026-08-30: this replaced a running
|
||||||
|
spend total, which could only climb and so kept reporting a context a
|
||||||
|
compaction or a clear had already taken away).
|
||||||
- `Error { message }`.
|
- `Error { message }`.
|
||||||
|
|
||||||
Every event is appended to the session's transcript file with a sequence
|
Every event is appended to the session's transcript file with a sequence
|
||||||
@@ -611,7 +621,9 @@ dev-updater (Kotlin 2.4.x, CMP 1.11.x, JDK 21). Screens:
|
|||||||
- Input bar: text, attach (camera/gallery/file), send — **always enabled**;
|
- Input bar: text, attach (camera/gallery/file), send — **always enabled**;
|
||||||
mid-run sends become steering messages.
|
mid-run sends become steering messages.
|
||||||
- Top bar: model chip (tap to change), stop button while running, token
|
- Top bar: model chip (tap to change), stop button while running, token
|
||||||
count, compact button (llama), overflow → delete.
|
count, compact button (llama), overflow → delete. (The count settled as
|
||||||
|
context held rather than tokens spent, and sits on the status row under
|
||||||
|
the transcript — see `UsageDelta` above.)
|
||||||
4. **Usage** — window bars for the 5-hour and weekly limits with reset times.
|
4. **Usage** — window bars for the 5-hour and weekly limits with reset times.
|
||||||
5. **Settings** — server address + token, hosts editor, llama model list
|
5. **Settings** — server address + token, hosts editor, llama model list
|
||||||
editor.
|
editor.
|
||||||
|
|||||||
@@ -150,9 +150,15 @@ data class SessionSummary(
|
|||||||
*/
|
*/
|
||||||
val notify: Boolean,
|
val notify: Boolean,
|
||||||
/**
|
/**
|
||||||
* Every token this session has spent, as the server counts it -- see `SessionEvent.UsageDelta`.
|
* How much context this session is holding, as the server last measured it -- see
|
||||||
|
* `SessionEvent.UsageDelta`.
|
||||||
|
*
|
||||||
|
* Null where nothing has been measured: a session that has not run a turn, a provider that does
|
||||||
|
* not report usage, or a clear nobody has run a turn since. That is not zero, and the status
|
||||||
|
* row says so in words rather than drawing an empty context for a conversation that may be
|
||||||
|
* nearly full.
|
||||||
*/
|
*/
|
||||||
val totalTokens: Long,
|
val contextTokens: Long?,
|
||||||
/**
|
/**
|
||||||
* The longest edge an image should have when it reaches this session, or null where the
|
* The longest edge an image should have when it reaches this session, or null where the
|
||||||
* provider has no limit.
|
* provider has no limit.
|
||||||
@@ -178,7 +184,8 @@ private fun parseSession(session: JSONObject) =
|
|||||||
permissionMode = session.optString("permissionMode").ifEmpty { null },
|
permissionMode = session.optString("permissionMode").ifEmpty { null },
|
||||||
imported = session.optBoolean("imported", false),
|
imported = session.optBoolean("imported", false),
|
||||||
notify = session.optBoolean("notify", true),
|
notify = session.optBoolean("notify", true),
|
||||||
totalTokens = session.optLong("totalTokens", 0),
|
contextTokens =
|
||||||
|
if (session.has("contextTokens")) session.getLong("contextTokens") else null,
|
||||||
maxImageEdge = session.optInt("maxImageEdge", 0).takeIf { it > 0 },
|
maxImageEdge = session.optInt("maxImageEdge", 0).takeIf { it > 0 },
|
||||||
status = session.getString("status"),
|
status = session.getString("status"),
|
||||||
lastActivity = session.getDouble("lastActivity"),
|
lastActivity = session.getDouble("lastActivity"),
|
||||||
|
|||||||
@@ -37,7 +37,14 @@ fun compactionSummary(item: TranscriptItem.CompactedNote): String {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private fun tokens(count: Long): String = "%,d".format(count)
|
/**
|
||||||
|
* A token count as a reader reads one.
|
||||||
|
*
|
||||||
|
* Shared with the status row rather than formatted at each: the divider and the row report the same
|
||||||
|
* quantity about the same moment, and one of them grouping its thousands while the other did not
|
||||||
|
* read as two different measurements.
|
||||||
|
*/
|
||||||
|
fun tokens(count: Long): String = "%,d".format(count)
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* What the working indicator says while a compaction is running.
|
* What the working indicator says while a compaction is running.
|
||||||
|
|||||||
@@ -113,14 +113,16 @@ sealed class SessionEvent {
|
|||||||
data class Settings(val model: String?, val permissionMode: String?) : SessionEvent()
|
data class Settings(val model: String?, val permissionMode: String?) : SessionEvent()
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* What a turn cost, and what the session has cost in total.
|
* What a turn cost, and how much the model was holding when it ended.
|
||||||
*
|
*
|
||||||
* [total] is the server's running figure, carried on the event so a reader never adds up its
|
* [context] is prompt plus both cache figures, measured by the backend from the turn's own
|
||||||
* own: a phone opens a session on the newest page of the transcript, so a sum it computed would
|
* usage. Carried on the event rather than summed by the reader, because it is not a sum: a
|
||||||
* be that page's share of the conversation wearing the whole conversation's label. Zero on
|
* conversation's context drops at a compaction and a clear, so adding turns up would report a
|
||||||
* entries recorded before the backend sent it.
|
* figure the session stopped being true of. Null where the dialect did not say, and on entries
|
||||||
|
* recorded before the backend sent it -- which leaves the context unmeasured rather than
|
||||||
|
* unchanged.
|
||||||
*/
|
*/
|
||||||
data class UsageDelta(val tokens: Long, val total: Long) : SessionEvent()
|
data class UsageDelta(val tokens: Long, val context: Long?) : SessionEvent()
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A compaction that finished, and how much context it recovered.
|
* A compaction that finished, and how much context it recovered.
|
||||||
@@ -235,7 +237,10 @@ fun parseSeqEvent(json: String): SeqEvent {
|
|||||||
permissionMode = body.optString("permissionMode").ifEmpty { null },
|
permissionMode = body.optString("permissionMode").ifEmpty { null },
|
||||||
)
|
)
|
||||||
"usageDelta" ->
|
"usageDelta" ->
|
||||||
SessionEvent.UsageDelta(body.getLong("tokens"), body.optLong("total", 0))
|
SessionEvent.UsageDelta(
|
||||||
|
body.getLong("tokens"),
|
||||||
|
if (body.has("context")) body.getLong("context") else null,
|
||||||
|
)
|
||||||
"compacted" ->
|
"compacted" ->
|
||||||
SessionEvent.Compacted(
|
SessionEvent.Compacted(
|
||||||
preTokens = if (body.has("preTokens")) body.getLong("preTokens") else null,
|
preTokens = if (body.has("preTokens")) body.getLong("preTokens") else null,
|
||||||
@@ -248,3 +253,29 @@ fun parseSeqEvent(json: String): SeqEvent {
|
|||||||
}
|
}
|
||||||
return SeqEvent(seq = body.getLong("seq"), ts = body.getDouble("ts"), event = event)
|
return SeqEvent(seq = body.getLong("seq"), ts = body.getDouble("ts"), event = event)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The context after [event], given what it was before.
|
||||||
|
*
|
||||||
|
* The same rule the server folds with, because the screen has to keep up between page loads: the
|
||||||
|
* summary it opened with is a measurement from before this stream started, and every event that
|
||||||
|
* moves the figure arrives here.
|
||||||
|
*
|
||||||
|
* The two that lower it are the point. A clear takes the conversation away and a compaction
|
||||||
|
* replaces it with a summary, so a figure measured before either stopped being true at that moment
|
||||||
|
* -- and carrying it forward is how a session that had just been cleared went on reporting the
|
||||||
|
* context it no longer had.
|
||||||
|
*
|
||||||
|
* Null is "we don't know", which is a state each of them can reach: nothing measured yet, a
|
||||||
|
* compaction that finished without saying how much it recovered, or a clear nobody has run a turn
|
||||||
|
* since.
|
||||||
|
*/
|
||||||
|
fun contextAfter(current: Long?, event: SessionEvent): Long? =
|
||||||
|
when (event) {
|
||||||
|
// Falls back to what we had, so a turn the dialect reported no usage for is stale by a
|
||||||
|
// turn -- which every context figure is -- rather than unknown.
|
||||||
|
is SessionEvent.UsageDelta -> event.context ?: current
|
||||||
|
is SessionEvent.Compacted -> event.postTokens
|
||||||
|
is SessionEvent.Cleared -> null
|
||||||
|
else -> current
|
||||||
|
}
|
||||||
@@ -457,10 +457,10 @@ fun SessionScreen(
|
|||||||
val scope = rememberCoroutineScope()
|
val scope = rememberCoroutineScope()
|
||||||
var items by remember { mutableStateOf(listOf<TranscriptItem>()) }
|
var items by remember { mutableStateOf(listOf<TranscriptItem>()) }
|
||||||
var status by remember { mutableStateOf(summary.status) }
|
var status by remember { mutableStateOf(summary.status) }
|
||||||
// Seeded from the row this screen was opened from, so a conversation that has spent
|
// Seeded from the row this screen was opened from, so a conversation already under way says
|
||||||
// something says so before any turn happens here. Zero used to mean "nothing yet" and
|
// how much it is holding before any turn happens here. Null is "nobody has measured it",
|
||||||
// "nothing in the page I loaded" at once, and the second one is most of the long sessions.
|
// which is a different answer from an empty context and is drawn differently.
|
||||||
var totalTokens by remember(summary.id) { mutableLongStateOf(summary.totalTokens) }
|
var contextTokens by remember(summary.id) { mutableStateOf(summary.contextTokens) }
|
||||||
// When this screen saw the current compaction start, on this device's own clock, and how long
|
// When this screen saw the current compaction start, on this device's own clock, and how long
|
||||||
// ago that is. See `compactingLabel`: null is the honest answer whenever the start was not
|
// ago that is. See `compactingLabel`: null is the honest answer whenever the start was not
|
||||||
// witnessed here, which is what opening a session that is already compacting looks like.
|
// witnessed here, which is what opening a session that is already compacting looks like.
|
||||||
@@ -611,14 +611,14 @@ fun SessionScreen(
|
|||||||
*/
|
*/
|
||||||
fun apply(entry: SeqEvent) {
|
fun apply(entry: SeqEvent) {
|
||||||
lastSeq.set(entry.seq)
|
lastSeq.set(entry.seq)
|
||||||
|
// Before the rest, and for every event rather than only the usage ones: a compaction and
|
||||||
|
// a clear move this as much as a turn does, which is the whole reason it is a fold and
|
||||||
|
// not a running total. See `contextAfter`.
|
||||||
|
contextTokens = contextAfter(contextTokens, entry.event)
|
||||||
when (val event = entry.event) {
|
when (val event = entry.event) {
|
||||||
// Taken, not accumulated: the server's running total is on the event, and adding
|
// Nothing further: what it carries was folded into the context above, and what a
|
||||||
// up the deltas this screen happened to receive counted one page of a conversation
|
// turn cost is not something the transcript draws.
|
||||||
// and called it the whole. `max` because pages arrive in no guaranteed order and an
|
is SessionEvent.UsageDelta -> {}
|
||||||
// older event's total is a smaller true answer, never a correction downwards; it
|
|
||||||
// also leaves the seeded figure alone for transcripts recorded before the backend
|
|
||||||
// sent a total at all.
|
|
||||||
is SessionEvent.UsageDelta -> totalTokens = maxOf(totalTokens, event.total)
|
|
||||||
else -> {
|
else -> {
|
||||||
// What the session says it is set to now, which is the only thing that
|
// What the session says it is set to now, which is the only thing that
|
||||||
// says it: picking from either menu asks, and the answer comes back here.
|
// says it: picking from either menu asks, and the answer comes back here.
|
||||||
@@ -1214,7 +1214,11 @@ fun SessionScreen(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
SessionStatusRow(status = status, compactingFor = compactingFor, totalTokens = totalTokens)
|
SessionStatusRow(
|
||||||
|
status = status,
|
||||||
|
compactingFor = compactingFor,
|
||||||
|
contextTokens = contextTokens,
|
||||||
|
)
|
||||||
|
|
||||||
// Between the transcript and the box: above what is being typed, so the list does not
|
// Between the transcript and the box: above what is being typed, so the list does not
|
||||||
// cover the thing the command is about, and below everything that explains it.
|
// cover the thing the command is about, and below everything that explains it.
|
||||||
@@ -1468,7 +1472,8 @@ private fun SessionStatusRow(
|
|||||||
status: String,
|
status: String,
|
||||||
/** Seconds since this device saw the compaction start; null if it did not see it. */
|
/** Seconds since this device saw the compaction start; null if it did not see it. */
|
||||||
compactingFor: Long?,
|
compactingFor: Long?,
|
||||||
totalTokens: Long,
|
/** Context the session is holding, or null where nothing has measured it. */
|
||||||
|
contextTokens: Long?,
|
||||||
modifier: Modifier = Modifier,
|
modifier: Modifier = Modifier,
|
||||||
) {
|
) {
|
||||||
Row(
|
Row(
|
||||||
@@ -1538,17 +1543,21 @@ private fun SessionStatusRow(
|
|||||||
modifier = Modifier.weight(1f),
|
modifier = Modifier.weight(1f),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
// Nothing rather than "0 tok" before anything has been spent: a total of zero is a fact
|
// How full the session is, which is the number a reader is asking about -- how much room
|
||||||
// about a conversation that has not started, and it is the one reading nobody needs.
|
// is left before the next compaction -- rather than what has been spent getting here.
|
||||||
if (totalTokens > 0) {
|
//
|
||||||
|
// "unknown" in words, and always drawn. A context nobody has measured is not an empty
|
||||||
|
// one, and the two used to share an appearance: a session that had just been cleared, one
|
||||||
|
// whose provider never reports usage, and one that has not run a turn all showed nothing
|
||||||
|
// at all, which reads as a conversation with room to spare. It is the same reason the
|
||||||
|
// status word beside it names the quiet state instead of leaving the row blank.
|
||||||
Text(
|
Text(
|
||||||
"$totalTokens tok",
|
contextTokens?.let { "context ${tokens(it)}" } ?: "context unknown",
|
||||||
style = MaterialTheme.typography.labelSmall,
|
style = MaterialTheme.typography.labelSmall,
|
||||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A question (or permission request -- same shape) inline in the transcript. Option buttons until
|
* A question (or permission request -- same shape) inline in the transcript. Option buttons until
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ use std::path::{Path, PathBuf};
|
|||||||
|
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
|
|
||||||
use super::super::driver::{Event, QuestionOption, SessionStatus};
|
use super::super::driver::{Event, QuestionOption, SessionStatus, context_tokens};
|
||||||
|
|
||||||
/// Whether this line is the CLI opening a fresh model call.
|
/// Whether this line is the CLI opening a fresh model call.
|
||||||
///
|
///
|
||||||
@@ -96,6 +96,22 @@ pub(super) struct Translator {
|
|||||||
/// the next `result` whichever way it went, so a genuine failure in a
|
/// the next `result` whichever way it went, so a genuine failure in a
|
||||||
/// later turn is still reported.
|
/// later turn is still reported.
|
||||||
interrupting: bool,
|
interrupting: bool,
|
||||||
|
/// The input side of the newest assistant message, waiting for the
|
||||||
|
/// `result` that ends the turn to carry it out.
|
||||||
|
///
|
||||||
|
/// Read from the assistant message rather than from the result's own
|
||||||
|
/// usage, which is the whole turn added up: measured on 2026-08-30
|
||||||
|
/// against CLI 2.1.237, a two-message turn reported
|
||||||
|
/// `cache_read_input_tokens` of 40,211 in its result, being 14,259 and
|
||||||
|
/// 25,952 from the two messages -- the same conversation counted
|
||||||
|
/// twice. The model never held 40,211; it held 26,131, which is the
|
||||||
|
/// last message's three input figures. A turn with ten tool calls
|
||||||
|
/// would overstate it tenfold.
|
||||||
|
///
|
||||||
|
/// Its path out is that result, which takes it -- so a turn whose
|
||||||
|
/// messages carried no usage reports none rather than repeating the
|
||||||
|
/// previous turn's.
|
||||||
|
context: Option<u64>,
|
||||||
session_dir: PathBuf,
|
session_dir: PathBuf,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -106,6 +122,7 @@ impl Translator {
|
|||||||
pending: HashMap::new(),
|
pending: HashMap::new(),
|
||||||
asked: HashMap::new(),
|
asked: HashMap::new(),
|
||||||
interrupting: false,
|
interrupting: false,
|
||||||
|
context: None,
|
||||||
session_dir,
|
session_dir,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -221,8 +238,9 @@ impl Translator {
|
|||||||
.to_string(),
|
.to_string(),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
let context = self.context.take();
|
||||||
if tokens > 0 {
|
if tokens > 0 {
|
||||||
events.push(Event::UsageDelta { tokens, total: 0 });
|
events.push(Event::UsageDelta { tokens, context });
|
||||||
}
|
}
|
||||||
events.push(Event::Status {
|
events.push(Event::Status {
|
||||||
state: SessionStatus::Idle,
|
state: SessionStatus::Idle,
|
||||||
@@ -358,6 +376,14 @@ impl Translator {
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn translate_assistant(&mut self, message: &Value) -> Vec<Event> {
|
fn translate_assistant(&mut self, message: &Value) -> Vec<Event> {
|
||||||
|
if let Some(usage) = message.get("usage") {
|
||||||
|
let field = |name: &str| usage.get(name).and_then(Value::as_u64).unwrap_or(0);
|
||||||
|
self.context = Some(context_tokens(
|
||||||
|
field("input_tokens"),
|
||||||
|
field("cache_creation_input_tokens"),
|
||||||
|
field("cache_read_input_tokens"),
|
||||||
|
));
|
||||||
|
}
|
||||||
let Some(content) = message.get("content").and_then(Value::as_array) else {
|
let Some(content) = message.get("content").and_then(Value::as_array) else {
|
||||||
return Vec::new();
|
return Vec::new();
|
||||||
};
|
};
|
||||||
@@ -1054,7 +1080,7 @@ mod tests {
|
|||||||
vec![
|
vec![
|
||||||
Event::UsageDelta {
|
Event::UsageDelta {
|
||||||
tokens: 182,
|
tokens: 182,
|
||||||
total: 0
|
context: None
|
||||||
},
|
},
|
||||||
Event::Status {
|
Event::Status {
|
||||||
state: SessionStatus::Idle
|
state: SessionStatus::Idle
|
||||||
@@ -1063,6 +1089,52 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The context is the last assistant message's, not the result's.
|
||||||
|
///
|
||||||
|
/// Real figures from a two-message haiku turn on 2.1.237, captured
|
||||||
|
/// 2026-08-30. The result adds the turn up -- its
|
||||||
|
/// `cache_read_input_tokens` of 40,211 is 14,259 and 25,952, the same
|
||||||
|
/// conversation counted twice -- so reading the context off it would
|
||||||
|
/// report a size the model never held, and by more the more tool calls
|
||||||
|
/// a turn makes. The last message's three input figures are what it
|
||||||
|
/// was holding when the turn ended.
|
||||||
|
#[test]
|
||||||
|
fn the_context_is_what_the_last_message_held_not_the_turn_added_up() {
|
||||||
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
|
let mut translator = Translator::new(dir.path().to_path_buf());
|
||||||
|
let events = translate_lines(
|
||||||
|
&mut translator,
|
||||||
|
&[
|
||||||
|
r#"{"type":"assistant","message":{"content":[],"usage":{"input_tokens":9,"cache_creation_input_tokens":11693,"cache_read_input_tokens":14259,"output_tokens":3}}}"#,
|
||||||
|
r#"{"type":"assistant","message":{"content":[],"usage":{"input_tokens":8,"cache_creation_input_tokens":171,"cache_read_input_tokens":25952,"output_tokens":2}}}"#,
|
||||||
|
r#"{"type":"result","subtype":"success","is_error":false,"session_id":"s","usage":{"input_tokens":17,"cache_creation_input_tokens":11864,"cache_read_input_tokens":40211,"output_tokens":156}}"#,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
events.first(),
|
||||||
|
Some(&Event::UsageDelta {
|
||||||
|
tokens: 173,
|
||||||
|
context: Some(26_131),
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
// Taken by that result, so a following turn whose messages carry no
|
||||||
|
// usage reports none rather than repeating this one's.
|
||||||
|
let events = translate_lines(
|
||||||
|
&mut translator,
|
||||||
|
&[
|
||||||
|
r#"{"type":"result","subtype":"success","is_error":false,"session_id":"s","usage":{"input_tokens":4,"output_tokens":9}}"#,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
events.first(),
|
||||||
|
Some(&Event::UsageDelta {
|
||||||
|
tokens: 13,
|
||||||
|
context: None,
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_compaction_reports_its_start_and_what_it_recovered() {
|
fn a_compaction_reports_its_start_and_what_it_recovered() {
|
||||||
// Real lines (trimmed) from a 2.1.237 session driven through
|
// Real lines (trimmed) from a 2.1.237 session driven through
|
||||||
|
|||||||
+133
-14
@@ -247,23 +247,25 @@ pub enum Event {
|
|||||||
},
|
},
|
||||||
/// Per-turn token counts, where the dialect reports them.
|
/// Per-turn token counts, where the dialect reports them.
|
||||||
UsageDelta {
|
UsageDelta {
|
||||||
|
/// What this turn cost: the tokens it was charged for.
|
||||||
tokens: u64,
|
tokens: u64,
|
||||||
/// Every token this session has spent, this turn included.
|
/// What the model was holding when the turn ended -- see
|
||||||
|
/// [`context_tokens`] for what goes into it.
|
||||||
///
|
///
|
||||||
/// Filled in by the pump, not by drivers: a driver reports what its
|
/// Carried on the event rather than summed by whoever is reading,
|
||||||
/// turn cost, and only the pump sees all of them. Carried on the
|
/// because it is not a sum: a conversation's context goes *down*
|
||||||
/// event rather than left to be added up by whoever is reading,
|
/// at a compaction and a clear, so adding turns up would report a
|
||||||
/// because a reader has only *part* of the transcript -- a phone
|
/// figure the session stopped being true of long ago. It is also
|
||||||
/// opens a session on the newest page -- so a total it summed
|
/// the number a reader is asking about -- how much room is left
|
||||||
/// itself would be the newest page's total wearing the whole
|
/// before the next compaction -- rather than what has been spent
|
||||||
/// conversation's label. Worse when the page has no turn in it at
|
/// getting here.
|
||||||
/// all: the count then reads zero, and a zero is drawn as nothing.
|
|
||||||
///
|
///
|
||||||
/// Zero on entries written before this existed, which is why the
|
/// `None` where the dialect did not say, which every reader has to
|
||||||
/// pump seeds its running total by adding up `tokens` at startup
|
/// be able to draw: a turn whose usage the CLI omitted leaves the
|
||||||
/// rather than reading the last of these.
|
/// context unmeasured rather than unchanged, and entries written
|
||||||
#[serde(default)]
|
/// before this existed have no answer at all.
|
||||||
total: u64,
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
|
context: Option<u64>,
|
||||||
},
|
},
|
||||||
/// A compaction that finished, and how much context it recovered.
|
/// A compaction that finished, and how much context it recovered.
|
||||||
///
|
///
|
||||||
@@ -338,6 +340,51 @@ pub enum Event {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// How much the model was holding, from the three figures a turn reports.
|
||||||
|
///
|
||||||
|
/// The input side only -- prompt plus both cache figures. A cached token
|
||||||
|
/// is cheaper but it is still one the model was given, so all three count;
|
||||||
|
/// output is left out because it is what the turn produced rather than
|
||||||
|
/// what continuing from here has to carry.
|
||||||
|
///
|
||||||
|
/// One function so the definition cannot drift, because it is extracted in
|
||||||
|
/// two quite different ways: the live translators have the usage object
|
||||||
|
/// parsed, and `import::context_tokens` scans it out of a raw line without
|
||||||
|
/// parsing, since those files reach tens of megabytes.
|
||||||
|
pub fn context_tokens(input: u64, cache_creation: u64, cache_read: u64) -> u64 {
|
||||||
|
input + cache_creation + cache_read
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The context after `event`, given what it was before.
|
||||||
|
///
|
||||||
|
/// The whole rule in one place, because three readers need the same
|
||||||
|
/// answer: the pump keeping a live session's figure, the transcript
|
||||||
|
/// seeding it at startup, and the phone folding the same events into what
|
||||||
|
/// it draws. Written here beside the events it reads so a fourth reader
|
||||||
|
/// finds it.
|
||||||
|
///
|
||||||
|
/// The two that *lower* it are the point. A clear takes the conversation
|
||||||
|
/// away and a compaction replaces it with a summary, so a figure measured
|
||||||
|
/// before either stopped being true at that moment -- and carrying it
|
||||||
|
/// forward is how a session that had just been cleared went on reporting
|
||||||
|
/// the context it no longer had.
|
||||||
|
///
|
||||||
|
/// `None` is "we don't know", which is a state each of them can reach:
|
||||||
|
/// nothing has been measured yet, a compaction finished without saying
|
||||||
|
/// how much it recovered, or a clear left a conversation nobody has
|
||||||
|
/// counted since.
|
||||||
|
pub fn context_after(current: Option<u64>, event: &Event) -> Option<u64> {
|
||||||
|
match event {
|
||||||
|
// `or`, so a turn the dialect reported no usage for leaves the last
|
||||||
|
// measurement standing: it is stale by a turn, which every context
|
||||||
|
// figure is, rather than wrong.
|
||||||
|
Event::UsageDelta { context, .. } => context.or(current),
|
||||||
|
Event::Compacted { post_tokens, .. } => *post_tokens,
|
||||||
|
Event::Cleared => None,
|
||||||
|
_ => current,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Something a session can be asked to do to itself.
|
/// Something a session can be asked to do to itself.
|
||||||
///
|
///
|
||||||
/// A closed set rather than a string, because the two that are not
|
/// A closed set rather than a string, because the two that are not
|
||||||
@@ -525,4 +572,76 @@ mod tests {
|
|||||||
})
|
})
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The two events that take the context *down* are the point of the
|
||||||
|
/// fold: a figure measured before a compaction or a clear stopped being
|
||||||
|
/// true at that moment, and carrying it forward is how a session that
|
||||||
|
/// had just been cleared went on reporting the context it no longer
|
||||||
|
/// had.
|
||||||
|
#[test]
|
||||||
|
fn a_compaction_and_a_clear_move_the_context_a_turn_cannot() {
|
||||||
|
let after = |current, event| context_after(current, &event);
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
after(
|
||||||
|
Some(500),
|
||||||
|
Event::UsageDelta {
|
||||||
|
tokens: 12,
|
||||||
|
context: Some(30_100),
|
||||||
|
}
|
||||||
|
),
|
||||||
|
Some(30_100)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
after(
|
||||||
|
Some(128_402),
|
||||||
|
Event::Compacted {
|
||||||
|
pre_tokens: Some(128_402),
|
||||||
|
post_tokens: Some(9_617),
|
||||||
|
trigger: Some("auto".to_string()),
|
||||||
|
}
|
||||||
|
),
|
||||||
|
Some(9_617)
|
||||||
|
);
|
||||||
|
assert_eq!(after(Some(9_617), Event::Cleared), None);
|
||||||
|
|
||||||
|
// A compaction that did not say how much it recovered leaves the
|
||||||
|
// context unknown rather than stale: it definitely moved, and the
|
||||||
|
// one thing that is certainly wrong is the figure from before it.
|
||||||
|
assert_eq!(
|
||||||
|
after(
|
||||||
|
Some(128_402),
|
||||||
|
Event::Compacted {
|
||||||
|
pre_tokens: None,
|
||||||
|
post_tokens: None,
|
||||||
|
trigger: None,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
None
|
||||||
|
);
|
||||||
|
|
||||||
|
// A turn the dialect reported no context for is stale by a turn,
|
||||||
|
// which every context figure is, rather than unknown.
|
||||||
|
assert_eq!(
|
||||||
|
after(
|
||||||
|
Some(30_100),
|
||||||
|
Event::UsageDelta {
|
||||||
|
tokens: 12,
|
||||||
|
context: None,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
Some(30_100)
|
||||||
|
);
|
||||||
|
|
||||||
|
// Everything else leaves it alone.
|
||||||
|
assert_eq!(
|
||||||
|
after(
|
||||||
|
Some(30_100),
|
||||||
|
Event::Status {
|
||||||
|
state: SessionStatus::Idle,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
Some(30_100)
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
@@ -43,7 +43,7 @@
|
|||||||
//! is the same every run.
|
//! is the same every run.
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::sync::atomic::{AtomicBool, Ordering};
|
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||||
use std::sync::{Arc, Mutex};
|
use std::sync::{Arc, Mutex};
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
|
|
||||||
@@ -94,6 +94,12 @@ pub struct EchoDriver {
|
|||||||
/// way AskUserQuestion does, and the turn resumes when the last of
|
/// way AskUserQuestion does, and the turn resumes when the last of
|
||||||
/// them is answered rather than the first.
|
/// them is answered rather than the first.
|
||||||
pending_questions: Mutex<Vec<PendingQuestion>>,
|
pending_questions: Mutex<Vec<PendingQuestion>>,
|
||||||
|
/// A pretend context, so the status row has something that behaves the
|
||||||
|
/// way a real one does: it grows with each turn, drops to what the
|
||||||
|
/// compaction says it recovered, and a clear leaves it unmeasured. The
|
||||||
|
/// numbers are invented like everything else here; what is real is
|
||||||
|
/// which way they move.
|
||||||
|
context: Arc<AtomicU64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl EchoDriver {
|
impl EchoDriver {
|
||||||
@@ -355,6 +361,7 @@ impl EchoDriver {
|
|||||||
.map(|rest| rest.trim().to_string());
|
.map(|rest| rest.trim().to_string());
|
||||||
let busy = Arc::clone(&self.busy);
|
let busy = Arc::clone(&self.busy);
|
||||||
let queued = Arc::clone(&self.queued);
|
let queued = Arc::clone(&self.queued);
|
||||||
|
let context = Arc::clone(&self.context);
|
||||||
let dir = self.session_dir.clone();
|
let dir = self.session_dir.clone();
|
||||||
busy.store(true, Ordering::SeqCst);
|
busy.store(true, Ordering::SeqCst);
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
@@ -480,9 +487,13 @@ impl EchoDriver {
|
|||||||
});
|
});
|
||||||
tokio::time::sleep(DELTA_DELAY).await;
|
tokio::time::sleep(DELTA_DELAY).await;
|
||||||
}
|
}
|
||||||
|
// A conversation gets bigger, so the pretend context does too:
|
||||||
|
// roughly a hundred tokens a turn plus the words themselves,
|
||||||
|
// which is enough to watch it climb between compactions.
|
||||||
|
let spent = text.split_whitespace().count() as u64;
|
||||||
send(Event::UsageDelta {
|
send(Event::UsageDelta {
|
||||||
tokens: text.split_whitespace().count() as u64,
|
tokens: spent,
|
||||||
total: 0,
|
context: Some(context.fetch_add(spent + 100, Ordering::SeqCst) + spent + 100),
|
||||||
});
|
});
|
||||||
finish();
|
finish();
|
||||||
});
|
});
|
||||||
@@ -492,6 +503,7 @@ impl EchoDriver {
|
|||||||
let driver = Self {
|
let driver = Self {
|
||||||
sink,
|
sink,
|
||||||
pending_questions: Mutex::new(Vec::new()),
|
pending_questions: Mutex::new(Vec::new()),
|
||||||
|
context: Arc::new(AtomicU64::new(0)),
|
||||||
busy: Arc::new(AtomicBool::new(false)),
|
busy: Arc::new(AtomicBool::new(false)),
|
||||||
queued: Arc::new(Mutex::new(Vec::new())),
|
queued: Arc::new(Mutex::new(Vec::new())),
|
||||||
session_dir,
|
session_dir,
|
||||||
@@ -746,12 +758,18 @@ impl Driver for EchoDriver {
|
|||||||
let sink = self.sink.clone();
|
let sink = self.sink.clone();
|
||||||
let queued = Arc::clone(&self.queued);
|
let queued = Arc::clone(&self.queued);
|
||||||
let busy = Arc::clone(&self.busy);
|
let busy = Arc::clone(&self.busy);
|
||||||
|
let context = Arc::clone(&self.context);
|
||||||
busy.store(true, Ordering::SeqCst);
|
busy.store(true, Ordering::SeqCst);
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
let _ = sink.send(Event::Status {
|
let _ = sink.send(Event::Status {
|
||||||
state: SessionStatus::Compacting,
|
state: SessionStatus::Compacting,
|
||||||
});
|
});
|
||||||
tokio::time::sleep(COMPACT_TIME).await;
|
tokio::time::sleep(COMPACT_TIME).await;
|
||||||
|
// What it says it recovered is what the pretend context becomes,
|
||||||
|
// so the figure on the status row and the one on the divider
|
||||||
|
// agree -- two numbers about the same moment disagreeing is the
|
||||||
|
// thing this rig exists to catch.
|
||||||
|
context.store(9_617, Ordering::SeqCst);
|
||||||
let _ = sink.send(Event::Compacted {
|
let _ = sink.send(Event::Compacted {
|
||||||
pre_tokens: Some(128_402),
|
pre_tokens: Some(128_402),
|
||||||
post_tokens: Some(9_617),
|
post_tokens: Some(9_617),
|
||||||
@@ -766,6 +784,7 @@ impl Driver for EchoDriver {
|
|||||||
/// scroll behaviour and the transcript's shape can be exercised
|
/// scroll behaviour and the transcript's shape can be exercised
|
||||||
/// without spending a real session's context to produce one.
|
/// without spending a real session's context to produce one.
|
||||||
fn clear(&self) {
|
fn clear(&self) {
|
||||||
|
self.context.store(0, Ordering::SeqCst);
|
||||||
let _ = self.sink.send(Event::Cleared);
|
let _ = self.sink.send(Event::Cleared);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ use anyhow::{Context, Result};
|
|||||||
use serde::Serialize;
|
use serde::Serialize;
|
||||||
use serde_json::Value;
|
use serde_json::Value;
|
||||||
|
|
||||||
use super::driver::Event;
|
use super::driver::{self, Event};
|
||||||
use super::transport::{Launch, Transport};
|
use super::transport::{Launch, Transport};
|
||||||
|
|
||||||
/// How much of a transcript's tail is replayed into the phone's view.
|
/// How much of a transcript's tail is replayed into the phone's view.
|
||||||
@@ -282,9 +282,9 @@ fn parse_row(line: &str) -> Option<Importable> {
|
|||||||
/// The input tokens named in one `usage` object, added up.
|
/// The input tokens named in one `usage` object, added up.
|
||||||
///
|
///
|
||||||
/// Prompt plus cache creation plus cache read: all three are context the
|
/// Prompt plus cache creation plus cache read: all three are context the
|
||||||
/// model was given, and a cached token is cheaper but not free. Output is
|
/// model was given -- the definition is [`driver::context_tokens`]; this
|
||||||
/// deliberately left out -- it is what the turn produced, not what
|
/// is the same three figures dug out of a raw line rather than a parsed
|
||||||
/// continuing from here has to carry.
|
/// one, because these files reach tens of megabytes.
|
||||||
///
|
///
|
||||||
/// `None` for an empty blob, meaning no assistant turn has recorded usage.
|
/// `None` for an empty blob, meaning no assistant turn has recorded usage.
|
||||||
/// Missing individual fields count as zero, which is what an absent
|
/// Missing individual fields count as zero, which is what an absent
|
||||||
@@ -307,11 +307,11 @@ fn context_tokens(usage: &str) -> Option<u64> {
|
|||||||
})
|
})
|
||||||
.unwrap_or(0)
|
.unwrap_or(0)
|
||||||
};
|
};
|
||||||
Some(
|
Some(driver::context_tokens(
|
||||||
field("input_tokens")
|
field("input_tokens"),
|
||||||
+ field("cache_creation_input_tokens")
|
field("cache_creation_input_tokens"),
|
||||||
+ field("cache_read_input_tokens"),
|
field("cache_read_input_tokens"),
|
||||||
)
|
))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The first line of what a person typed, short enough for a list row.
|
/// The first line of what a person typed, short enough for a list row.
|
||||||
|
|||||||
@@ -576,6 +576,10 @@ fn generate(
|
|||||||
|
|
||||||
let reader = std::io::BufReader::new(response.body_mut().as_reader());
|
let reader = std::io::BufReader::new(response.body_mut().as_reader());
|
||||||
let mut tokens = 0u64;
|
let mut tokens = 0u64;
|
||||||
|
// The prompt side only, which is what the model is holding -- the same
|
||||||
|
// definition the other dialects report, so one word on the phone means
|
||||||
|
// one thing whichever kind of session it is.
|
||||||
|
let mut context = None;
|
||||||
for line in std::io::BufRead::lines(reader) {
|
for line in std::io::BufRead::lines(reader) {
|
||||||
if cancel.load(Ordering::Relaxed) {
|
if cancel.load(Ordering::Relaxed) {
|
||||||
break;
|
break;
|
||||||
@@ -592,11 +596,20 @@ fn generate(
|
|||||||
let Ok(chunk) = serde_json::from_str::<serde_json::Value>(payload) else {
|
let Ok(chunk) = serde_json::from_str::<serde_json::Value>(payload) else {
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
if let Some(usage) = chunk.get("usage").and_then(|u| u.get("total_tokens"))
|
if let Some(usage) = chunk.get("usage") {
|
||||||
&& let Some(total) = usage.as_u64()
|
if let Some(total) = usage
|
||||||
|
.get("total_tokens")
|
||||||
|
.and_then(serde_json::Value::as_u64)
|
||||||
{
|
{
|
||||||
tokens = total;
|
tokens = total;
|
||||||
}
|
}
|
||||||
|
if let Some(prompt) = usage
|
||||||
|
.get("prompt_tokens")
|
||||||
|
.and_then(serde_json::Value::as_u64)
|
||||||
|
{
|
||||||
|
context = Some(prompt);
|
||||||
|
}
|
||||||
|
}
|
||||||
let delta = chunk
|
let delta = chunk
|
||||||
.get("choices")
|
.get("choices")
|
||||||
.and_then(|c| c.get(0))
|
.and_then(|c| c.get(0))
|
||||||
@@ -611,7 +624,7 @@ fn generate(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if tokens > 0 {
|
if tokens > 0 {
|
||||||
let _ = sink.send(Event::UsageDelta { tokens, total: 0 });
|
let _ = sink.send(Event::UsageDelta { tokens, context });
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -721,7 +734,7 @@ mod tests {
|
|||||||
},
|
},
|
||||||
Event::UsageDelta {
|
Event::UsageDelta {
|
||||||
tokens: 12,
|
tokens: 12,
|
||||||
total: 12,
|
context: Some(12),
|
||||||
},
|
},
|
||||||
]);
|
]);
|
||||||
let messages = conversation(&path);
|
let messages = conversation(&path);
|
||||||
|
|||||||
+65
-41
@@ -31,7 +31,7 @@ use crate::config::{
|
|||||||
Config, DriverKind, ProviderConfig, SessionConfig, SetupConfig, SshConfig, TokenEntry,
|
Config, DriverKind, ProviderConfig, SessionConfig, SetupConfig, SshConfig, TokenEntry,
|
||||||
};
|
};
|
||||||
use claude::ClaudeDriver;
|
use claude::ClaudeDriver;
|
||||||
use driver::{Driver, Event, EventSink, ImageRef, SessionCommand, SessionStatus};
|
use driver::{Driver, Event, EventSink, ImageRef, SessionCommand, SessionStatus, context_after};
|
||||||
use echo::EchoDriver;
|
use echo::EchoDriver;
|
||||||
use llama::LlamaDriver;
|
use llama::LlamaDriver;
|
||||||
use transcript::{SeqEvent, Transcript};
|
use transcript::{SeqEvent, Transcript};
|
||||||
@@ -143,9 +143,16 @@ pub struct SessionInfo {
|
|||||||
pub imported: bool,
|
pub imported: bool,
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
pub cwd: Option<PathBuf>,
|
pub cwd: Option<PathBuf>,
|
||||||
/// Every token this session has spent, so a phone showing a total does
|
/// How much context this session is holding, so a phone does not have
|
||||||
/// not have to add up a transcript it only holds part of.
|
/// to fold a transcript it only holds part of.
|
||||||
pub total_tokens: u64,
|
///
|
||||||
|
/// Absent rather than zero where nothing has been measured -- a
|
||||||
|
/// session that has not run a turn, a dialect that does not report
|
||||||
|
/// usage, or a clear nobody has run a turn since. "Empty" and "we did
|
||||||
|
/// not find out" are different answers and the phone draws them
|
||||||
|
/// differently.
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pub context_tokens: Option<u64>,
|
||||||
/// The longest edge an image should have by the time it gets here, or
|
/// The longest edge an image should have by the time it gets here, or
|
||||||
/// absent where this provider has no limit -- see
|
/// absent where this provider has no limit -- see
|
||||||
/// [`DriverKind::max_image_edge`]. Absent rather than a large number,
|
/// [`DriverKind::max_image_edge`]. Absent rather than a large number,
|
||||||
@@ -280,12 +287,12 @@ struct Shared {
|
|||||||
/// session was *launched* with, so reporting from it would show the
|
/// session was *launched* with, so reporting from it would show the
|
||||||
/// mode a change had already replaced.
|
/// mode a change had already replaced.
|
||||||
permission_mode: Mutex<Option<String>>,
|
permission_mode: Mutex<Option<String>>,
|
||||||
/// Every token this session has spent.
|
/// How much context this session is holding.
|
||||||
///
|
///
|
||||||
/// Kept here because only the pump sees every turn, and reported on the
|
/// Kept here because only the pump sees every event, and reported on
|
||||||
/// session row so a phone opening a long conversation has the real
|
/// the session row so a phone opening a long conversation has the real
|
||||||
/// figure rather than the newest page's share of it.
|
/// figure rather than whatever its newest page happens to mention.
|
||||||
total_tokens: Mutex<u64>,
|
context_tokens: Mutex<Option<u64>>,
|
||||||
/// Whether this session's attention-wanting moments are announced.
|
/// Whether this session's attention-wanting moments are announced.
|
||||||
///
|
///
|
||||||
/// Mirrored out of the config so the pump can read it without taking
|
/// Mirrored out of the config so the pump can read it without taking
|
||||||
@@ -403,7 +410,7 @@ impl LiveSession {
|
|||||||
title: self.shared.title.lock().unwrap().clone(),
|
title: self.shared.title.lock().unwrap().clone(),
|
||||||
model: self.shared.model.lock().unwrap().clone(),
|
model: self.shared.model.lock().unwrap().clone(),
|
||||||
permission_mode: self.shared.permission_mode.lock().unwrap().clone(),
|
permission_mode: self.shared.permission_mode.lock().unwrap().clone(),
|
||||||
total_tokens: *self.shared.total_tokens.lock().unwrap(),
|
context_tokens: *self.shared.context_tokens.lock().unwrap(),
|
||||||
notify: *self.shared.notify.lock().unwrap(),
|
notify: *self.shared.notify.lock().unwrap(),
|
||||||
max_image_edge: kind.and_then(DriverKind::max_image_edge),
|
max_image_edge: kind.and_then(DriverKind::max_image_edge),
|
||||||
imported,
|
imported,
|
||||||
@@ -741,7 +748,7 @@ impl SessionManager {
|
|||||||
title: meta.title.clone(),
|
title: meta.title.clone(),
|
||||||
model: meta.model.clone(),
|
model: meta.model.clone(),
|
||||||
permission_mode: meta.permission_mode.clone(),
|
permission_mode: meta.permission_mode.clone(),
|
||||||
total_tokens: 0,
|
context_tokens: None,
|
||||||
max_image_edge: kind_of(&inner.config, &meta.setup, &meta.provider)
|
max_image_edge: kind_of(&inner.config, &meta.setup, &meta.provider)
|
||||||
.and_then(DriverKind::max_image_edge),
|
.and_then(DriverKind::max_image_edge),
|
||||||
notify: meta.notify,
|
notify: meta.notify,
|
||||||
@@ -1265,7 +1272,7 @@ fn launch(
|
|||||||
last_activity: Mutex::new(transcript.last_activity().unwrap_or_else(now)),
|
last_activity: Mutex::new(transcript.last_activity().unwrap_or_else(now)),
|
||||||
model: Mutex::new(meta.model.clone()),
|
model: Mutex::new(meta.model.clone()),
|
||||||
permission_mode: Mutex::new(meta.permission_mode.clone()),
|
permission_mode: Mutex::new(meta.permission_mode.clone()),
|
||||||
total_tokens: Mutex::new(transcript.total_tokens()),
|
context_tokens: Mutex::new(transcript.context_tokens()),
|
||||||
notify: Mutex::new(meta.notify),
|
notify: Mutex::new(meta.notify),
|
||||||
written: Mutex::new(0),
|
written: Mutex::new(0),
|
||||||
});
|
});
|
||||||
@@ -1398,20 +1405,15 @@ async fn pump(
|
|||||||
// for where a user's message sits: where the session read it.
|
// for where a user's message sits: where the session read it.
|
||||||
let event = match event {
|
let event = match event {
|
||||||
Event::MessageTaken { id, text, images } => Event::UserMessage { id, text, images },
|
Event::MessageTaken { id, text, images } => Event::UserMessage { id, text, images },
|
||||||
// The running total is the pump's to keep, for the reason the
|
|
||||||
// field gives: a driver knows what its own turn cost and
|
|
||||||
// nothing else does. Added here rather than at each driver so
|
|
||||||
// a new one cannot get it wrong by leaving it out.
|
|
||||||
Event::UsageDelta { tokens, .. } => {
|
|
||||||
let mut total = shared.total_tokens.lock().unwrap();
|
|
||||||
*total += tokens;
|
|
||||||
Event::UsageDelta {
|
|
||||||
tokens,
|
|
||||||
total: *total,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
other => other,
|
other => other,
|
||||||
};
|
};
|
||||||
|
// Where the session row's figure comes from. Kept here rather than
|
||||||
|
// at each driver because a clear and a compaction move it as much
|
||||||
|
// as a turn does, and only the pump sees all three.
|
||||||
|
{
|
||||||
|
let mut context = shared.context_tokens.lock().unwrap();
|
||||||
|
*context = context_after(*context, &event);
|
||||||
|
}
|
||||||
// Nothing changed, so there is nothing to record. Both of these
|
// Nothing changed, so there is nothing to record. Both of these
|
||||||
// repeat: an imported session reads the turn state off its file's
|
// repeat: an imported session reads the turn state off its file's
|
||||||
// newest record on every sync and mostly finds the answer it found
|
// newest record on every sync and mostly finds the answer it found
|
||||||
@@ -2038,16 +2040,21 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The total covers the whole conversation, not the part a reader holds.
|
/// The context figure follows the conversation down as well as up.
|
||||||
///
|
///
|
||||||
/// The bug this fixes was invisible in exactly the way that matters: a
|
/// It used to be a running total of what the session had spent, which
|
||||||
/// phone opens a session on its newest page and used to add up the
|
/// only ever climbs -- so a session that had just been compacted from
|
||||||
/// `UsageDelta`s it found there, so a long conversation reported its
|
/// 128k to 10k, or cleared outright, went on reporting the larger
|
||||||
/// last few turns as the total -- and a page with no turn in it at all
|
/// figure, and the number on the status row disagreed with the divider
|
||||||
/// reported nothing, since zero is drawn as blank. Both readings looked
|
/// directly above it. Turns raise it, a compaction replaces it with
|
||||||
/// like an answer.
|
/// what the compaction says it recovered, and a clear leaves it
|
||||||
|
/// unmeasured rather than guessing a small number.
|
||||||
|
///
|
||||||
|
/// The compaction leg is in `driver::tests` rather than here: echo
|
||||||
|
/// spends thirteen seconds on one so a person can watch the state, and
|
||||||
|
/// the rule both paths use is the same function.
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn the_token_total_covers_the_whole_conversation() {
|
async fn the_context_figure_follows_compactions_and_clears() {
|
||||||
let dir = tempfile::tempdir().expect("tempdir");
|
let dir = tempfile::tempdir().expect("tempdir");
|
||||||
let config_path = dir.path().join("config.ron");
|
let config_path = dir.path().join("config.ron");
|
||||||
let data_dir = dir.path().join("sessions");
|
let data_dir = dir.path().join("sessions");
|
||||||
@@ -2061,34 +2068,51 @@ mod tests {
|
|||||||
let info = manager.spawn_session(echo_spec()).expect("spawn");
|
let info = manager.spawn_session(echo_spec()).expect("spawn");
|
||||||
let session = manager.session(&info.id).expect("live");
|
let session = manager.session(&info.id).expect("live");
|
||||||
|
|
||||||
// Echo charges a token a word, so the arithmetic is checkable.
|
// Nothing measured yet, which is not the same as an empty context
|
||||||
|
// and is not reported as one.
|
||||||
|
assert_eq!(manager.sessions()[0].context_tokens, None);
|
||||||
|
|
||||||
|
// Echo's pretend context is a hundred a turn plus the words, so the
|
||||||
|
// arithmetic is checkable.
|
||||||
let mut rx = session.subscribe();
|
let mut rx = session.subscribe();
|
||||||
session.send_message("one two three".to_string(), Vec::new());
|
session.send_message("one two three".to_string(), Vec::new());
|
||||||
collect_turn(&mut rx).await;
|
collect_turn(&mut rx).await;
|
||||||
session.send_message("four five".to_string(), Vec::new());
|
session.send_message("four five".to_string(), Vec::new());
|
||||||
collect_turn(&mut rx).await;
|
collect_turn(&mut rx).await;
|
||||||
let running = manager.sessions()[0].total_tokens;
|
assert_eq!(
|
||||||
assert_eq!(running, 5, "two turns of three and two words");
|
manager.sessions()[0].context_tokens,
|
||||||
|
Some(205),
|
||||||
|
"two turns of three and two words"
|
||||||
|
);
|
||||||
|
|
||||||
// The event carries it too, so a phone never has to add up its own.
|
// The event carries it too, so a phone never has to fold the part of
|
||||||
let last_total = transcript::read_after(session.transcript_path(), 0)
|
// the transcript it happens to hold.
|
||||||
|
let last_context = transcript::read_after(session.transcript_path(), 0)
|
||||||
.expect("transcript")
|
.expect("transcript")
|
||||||
.iter()
|
.iter()
|
||||||
.rev()
|
.rev()
|
||||||
.find_map(|entry| match entry.event {
|
.find_map(|entry| match entry.event {
|
||||||
Event::UsageDelta { total, .. } => Some(total),
|
Event::UsageDelta { context, .. } => context,
|
||||||
_ => None,
|
_ => None,
|
||||||
})
|
})
|
||||||
.expect("a usage event");
|
.expect("a usage event");
|
||||||
assert_eq!(last_total, running);
|
assert_eq!(last_context, 205);
|
||||||
|
|
||||||
// And a restart picks it up from the file rather than starting over.
|
// A clear leaves it unmeasured: the conversation is gone, and how
|
||||||
|
// much is left is a thing nobody has counted.
|
||||||
|
session.run_command(SessionCommand::Clear);
|
||||||
|
collect_until(&mut rx, |event| matches!(event, Event::Cleared)).await;
|
||||||
|
assert_eq!(manager.sessions()[0].context_tokens, None);
|
||||||
|
|
||||||
|
// And a restart folds it back out of the file rather than starting
|
||||||
|
// over -- including the clear, which is why it is not the last
|
||||||
|
// usage event that decides.
|
||||||
drop(rx);
|
drop(rx);
|
||||||
drop(session);
|
drop(session);
|
||||||
drop(manager);
|
drop(manager);
|
||||||
let manager = SessionManager::new(config_path, data_dir.clone(), data_dir.join("models"))
|
let manager = SessionManager::new(config_path, data_dir.clone(), data_dir.join("models"))
|
||||||
.expect("manager restart");
|
.expect("manager restart");
|
||||||
assert_eq!(manager.sessions()[0].total_tokens, running);
|
assert_eq!(manager.sessions()[0].context_tokens, None);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Backdates every line in a transcript, so a restart has something to
|
/// Backdates every line in a transcript, so a restart has something to
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ use std::path::Path;
|
|||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
use super::driver::{Event, SessionStatus};
|
use super::driver::{Event, SessionStatus, context_after};
|
||||||
|
|
||||||
/// One transcript line: an [`Event`] plus its position and time. The event
|
/// One transcript line: an [`Event`] plus its position and time. The event
|
||||||
/// is flattened so the wire shape stays one flat object.
|
/// is flattened so the wire shape stays one flat object.
|
||||||
@@ -32,7 +32,7 @@ pub struct Transcript {
|
|||||||
next_seq: u64,
|
next_seq: u64,
|
||||||
last_status: Option<SessionStatus>,
|
last_status: Option<SessionStatus>,
|
||||||
last_activity: Option<f64>,
|
last_activity: Option<f64>,
|
||||||
total_tokens: u64,
|
context_tokens: Option<u64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Transcript {
|
impl Transcript {
|
||||||
@@ -62,16 +62,12 @@ impl Transcript {
|
|||||||
next_seq: last_seq + 1,
|
next_seq: last_seq + 1,
|
||||||
last_status,
|
last_status,
|
||||||
last_activity: existing.last().map(|entry| entry.ts),
|
last_activity: existing.last().map(|entry| entry.ts),
|
||||||
// Added up rather than read off the newest entry: `total` is a
|
// Folded rather than read off the newest usage entry: a clear
|
||||||
// later addition, so a transcript written before it has zero on
|
// or a compaction after it is what the answer is, and those
|
||||||
// every line while `tokens` was always there.
|
// events carry no usage of their own.
|
||||||
total_tokens: existing
|
context_tokens: existing
|
||||||
.iter()
|
.iter()
|
||||||
.map(|entry| match entry.event {
|
.fold(None, |current, entry| context_after(current, &entry.event)),
|
||||||
Event::UsageDelta { tokens, .. } => tokens,
|
|
||||||
_ => 0,
|
|
||||||
})
|
|
||||||
.sum(),
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -107,13 +103,16 @@ impl Transcript {
|
|||||||
self.last_activity
|
self.last_activity
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Everything this session has spent, as of opening.
|
/// How much context the session was holding, as of opening.
|
||||||
///
|
///
|
||||||
/// Zero for an empty transcript, which is a session that has spent
|
/// `None` for a transcript nothing has been measured in -- a new
|
||||||
/// nothing -- the one case where zero is the answer rather than the
|
/// session, one whose dialect never reported usage, or one whose last
|
||||||
/// absence of one.
|
/// word on the subject was a clear. That is not zero, and it is why
|
||||||
pub fn total_tokens(&self) -> u64 {
|
/// this is an option: a server that has just restarted has been told
|
||||||
self.total_tokens
|
/// nothing, and answering zero would draw an empty context for a
|
||||||
|
/// conversation that may be nearly full.
|
||||||
|
pub fn context_tokens(&self) -> Option<u64> {
|
||||||
|
self.context_tokens
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Appends `event`, assigning it the next sequence number. Flushed per
|
/// Appends `event`, assigning it the next sequence number. Flushed per
|
||||||
@@ -395,7 +394,7 @@ mod tests {
|
|||||||
},
|
},
|
||||||
Event::UsageDelta {
|
Event::UsageDelta {
|
||||||
tokens: 42,
|
tokens: 42,
|
||||||
total: 0,
|
context: Some(42),
|
||||||
},
|
},
|
||||||
Event::Error {
|
Event::Error {
|
||||||
message: "boom".into(),
|
message: "boom".into(),
|
||||||
|
|||||||
Reference in new issue
Block a user