Report what a reply spent reading its prompt, and pin the clock right
`UsageDelta` gains `prefillMs`, llama-server's own `timings.prompt_ms`, so the footer under a finished reply is "read 9.5s · 50.3 tok/s · 3:00 PM". Prefill is the half of a turn that was invisible and is often the larger: measured on the 0.6B here, 1m 4s for the first turn after a model loads against 22ms for the next, whose prompt the server still had cached. The clock moves to the end of the line. Everything in front of it is a provider's own measurement, so a session on another provider has fewer of them or none, and a reader who has learned where the time is should not have to find it again because the model changed. The costs grow leftwards into the space instead, and a test asserts every shape of the line ends with the same thing. Verified on the emulator against a real llama session: three replies reading "read 1m 4s · 193 tok/s · 3:54 PM", "read 25ms · 308 tok/s · 3:54 PM" and "read 22ms · 194 tok/s · 3:54 PM", with the clock in one column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b660905098
commit
369b8f7e52
14 files changed
+135
-42
No files matched your search
@@ -185,6 +185,12 @@ sealed class SessionEvent {
|
||||
* over includes the network and whatever the server was doing between tokens.
|
||||
*/
|
||||
val tokensPerSecond: Double? = null,
|
||||
/**
|
||||
* How long the provider spent reading the prompt before it began answering; null where
|
||||
* nothing measured it. The same rule as [tokensPerSecond]: the provider's own figure, or
|
||||
* nothing at all.
|
||||
*/
|
||||
val prefillMs: Long? = null,
|
||||
) : SessionEvent()
|
||||
|
||||
/** How much context this session's model has, which is what [UsageDelta.context] is out of. */
|
||||
@@ -328,6 +334,7 @@ fun parseSeqEvent(json: String): SeqEvent {
|
||||
body.getLong("tokens"),
|
||||
if (body.has("context")) body.getLong("context") else null,
|
||||
if (body.has("tokensPerSecond")) body.getDouble("tokensPerSecond") else null,
|
||||
if (body.has("prefillMs")) body.getLong("prefillMs") else null,
|
||||
)
|
||||
"compacted" ->
|
||||
SessionEvent.Compacted(
|
||||
|
||||
Reference in new issue
Block a user