Report what a reply spent reading its prompt, and pin the clock right

`UsageDelta` gains `prefillMs`, llama-server's own `timings.prompt_ms`, so the
footer under a finished reply is "read 9.5s · 50.3 tok/s · 3:00 PM". Prefill is
the half of a turn that was invisible and is often the larger: measured on the
0.6B here, 1m 4s for the first turn after a model loads against 22ms for the
next, whose prompt the server still had cached.

The clock moves to the end of the line. Everything in front of it is a
provider's own measurement, so a session on another provider has fewer of them
or none, and a reader who has learned where the time is should not have to find
it again because the model changed. The costs grow leftwards into the space
instead, and a test asserts every shape of the line ends with the same thing.

Verified on the emulator against a real llama session: three replies reading
"read 1m 4s · 193 tok/s · 3:54 PM", "read 25ms · 308 tok/s · 3:54 PM" and
"read 22ms · 194 tok/s · 3:54 PM", with the clock in one column.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
iris-aiandClaude Opus 5 committed 2026-09-19 15:57:56 -04:00
1 parent b660905098
commit 369b8f7e52
14 files changed
+135 -42

No files matched your search

+13
View File
@@ -438,6 +438,17 @@ pub enum Event {
/// times those were.
#[serde(default, skip_serializing_if = "Option::is_none")]
tokens_per_second: Option<f64>,
/// How long the provider spent reading the prompt before it began
/// answering -- see [`SessionStatus::Reading`], which is this while it
/// is happening.
///
/// The provider's own measurement or nothing, for the same reason
/// `tokens_per_second` above is: the wait this server watched also
/// contains the request and whatever else the machine was doing. It is worth reporting because it is the larger half of a
/// turn on a long conversation -- 22 seconds against 3 of generation,
/// measured on a 14,000-token prompt.
#[serde(default, skip_serializing_if = "Option::is_none")]
prefill_ms: Option<u64>,
},
/// A compaction that finished, and how much context it recovered.
///
@@ -904,6 +915,7 @@ mod tests {
tokens: 12,
context: Some(30_100),
tokens_per_second: None,
prefill_ms: None,
}
),
Some(30_100)
@@ -944,6 +956,7 @@ mod tests {
tokens: 12,
context: None,
tokens_per_second: None,
prefill_ms: None,
}
),
Some(30_100)