Draw a model's thinking, and what a reply cost to produce
A llama.cpp session's `reasoning_content` becomes `Event::Thinking` deltas closed by an `Event::ThinkingDone` carrying the span the driver measured, and the phone draws it as a card of its own: "Thinking" with the spinner a running command has, then "Thought for 12.4s". Deliberately not a tool call, so a run of calls cannot collapse the reasoning into "Called 6 tools"; the reasoning is also kept out of the next prompt, which `conversation` already ignored. `UsageDelta` gains `tokensPerSecond`, the provider's own figure or nothing -- llama.cpp reports `timings.predicted_per_second` and the coding CLIs report no such thing -- and a finished reply carries a small line under it saying when it was sent and, where there is one, how fast it came out: "3:00 PM · 149 tok/s". The compact usage bar drops the provider's name for the window and puts its length after the time left instead: "42% · 3h 20m left / 5h". Three things that had to come with it: the transcript coalesces runs of thinking deltas as it does reply deltas, so one block is one row of a page rather than a page of its own; `joinPages` welds a block cut by a page boundary (`healSplitThinking`), since the half with no ending spun for ever; and `UsageDelta` now reaches the fold, which is what carries the rate to the reply. Verified on the emulator against a real Qwen3-0.6B session and the echo rig's new `/think [seconds]`: the spinner while it runs, "Thought for 1.4s" and "2:54 PM · 149 tok/s" after, the reasoning on tapping the card, and the usage bar reading "42% · 3h 19m left / 5h". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
45f249ae91
commit
bb5ac1a242
18 files changed
+900
-101
No files matched your search
@@ -1562,9 +1562,12 @@ fn context_window(endpoint: &str) -> Option<u64> {
|
||||
/// what happens to them next -- asking, running, reporting -- is the caller's,
|
||||
/// and a call is not in the transcript until it has actually been made.
|
||||
///
|
||||
/// `reasoning_content` is dropped, which is what the Claude driver does with
|
||||
/// thinking deltas. A transcript is what was said, and this app does not draw
|
||||
/// a model's working.
|
||||
/// `reasoning_content` is a model's working, and it is emitted as
|
||||
/// [`Event::Thinking`] rather than mixed into the reply -- its own card, and
|
||||
/// deliberately not part of what the next prompt is built from (see
|
||||
/// [`conversation`], which ignores it). The block is closed with
|
||||
/// [`Event::ThinkingDone`] the moment the model says something else, which is
|
||||
/// how long it thought for.
|
||||
fn generate(
|
||||
endpoint: &str,
|
||||
messages: &[Message],
|
||||
@@ -1619,6 +1622,24 @@ fn generate(
|
||||
// worst direction for a figure somebody is watching to see how much room
|
||||
// is left.
|
||||
let mut context = None;
|
||||
// The open thinking block: when it started, so its duration is measured
|
||||
// where the deltas actually arrive rather than worked out later from two
|
||||
// timestamps. `None` between blocks -- a turn can think, speak, call a
|
||||
// tool and think again.
|
||||
let mut thinking: Option<std::time::Instant> = None;
|
||||
// What the server says its own generation ran at. Read from it rather than
|
||||
// divided out of the wall time here, which would count the request, the
|
||||
// prompt processing and this loop's own scheduling as generation.
|
||||
let mut per_second = None;
|
||||
// Closes the open block, which is anything the model says that is not more
|
||||
// working: the first word of the reply, or a tool call.
|
||||
let done_thinking = |thinking: &mut Option<std::time::Instant>| {
|
||||
if let Some(started) = thinking.take() {
|
||||
shared.emit(Event::ThinkingDone {
|
||||
ms: started.elapsed().as_millis() as u64,
|
||||
});
|
||||
}
|
||||
};
|
||||
for line in std::io::BufRead::lines(reader) {
|
||||
if shared.cancel.load(Ordering::SeqCst) {
|
||||
break;
|
||||
@@ -1642,12 +1663,27 @@ fn generate(
|
||||
tokens = total;
|
||||
context = Some(total);
|
||||
}
|
||||
if let Some(rate) = chunk
|
||||
.pointer("/timings/predicted_per_second")
|
||||
.and_then(Value::as_f64)
|
||||
{
|
||||
per_second = Some(rate);
|
||||
}
|
||||
let Some(delta) = chunk.pointer("/choices/0/delta") else {
|
||||
continue;
|
||||
};
|
||||
if let Some(fragment) = delta.get("reasoning_content").and_then(Value::as_str)
|
||||
&& !fragment.is_empty()
|
||||
{
|
||||
thinking.get_or_insert_with(std::time::Instant::now);
|
||||
shared.emit(Event::Thinking {
|
||||
delta: fragment.to_string(),
|
||||
});
|
||||
}
|
||||
if let Some(fragment) = delta.get("content").and_then(Value::as_str)
|
||||
&& !fragment.is_empty()
|
||||
{
|
||||
done_thinking(&mut thinking);
|
||||
text.push_str(fragment);
|
||||
shared.emit(Event::AssistantText {
|
||||
delta: fragment.to_string(),
|
||||
@@ -1659,11 +1695,20 @@ fn generate(
|
||||
.into_iter()
|
||||
.flatten()
|
||||
{
|
||||
done_thinking(&mut thinking);
|
||||
absorb(&mut calls, fragment);
|
||||
}
|
||||
}
|
||||
// A block left open by the end of the stream -- a model that thought and
|
||||
// then said nothing, or a turn the reader cancelled -- is still a block
|
||||
// that ended. Without this its card spins for ever.
|
||||
done_thinking(&mut thinking);
|
||||
if tokens > 0 {
|
||||
shared.emit(Event::UsageDelta { tokens, context });
|
||||
shared.emit(Event::UsageDelta {
|
||||
tokens,
|
||||
context,
|
||||
tokens_per_second: per_second,
|
||||
});
|
||||
}
|
||||
// A call whose name never arrived is not a call. It happens when a stream
|
||||
// is cut mid-fragment, and running it would mean inventing what was asked
|
||||
@@ -1815,6 +1860,30 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
/// A model's working is drawn and is deliberately not sent back to it: the
|
||||
/// prompt is what was said, and feeding reasoning back costs the whole of
|
||||
/// it in context for a model that never asked to see it again.
|
||||
fn thinking_is_not_part_of_the_conversation() {
|
||||
let (_dir, path) = transcript_with(&[
|
||||
Event::UserMessage {
|
||||
id: None,
|
||||
text: "count".into(),
|
||||
attachments: Vec::new(),
|
||||
},
|
||||
Event::Thinking {
|
||||
delta: "the user wants".into(),
|
||||
},
|
||||
Event::ThinkingDone { ms: 1200 },
|
||||
Event::AssistantText {
|
||||
delta: "one two".into(),
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
assert_eq!(messages.len(), 2);
|
||||
assert_eq!(messages[1].content, "one two");
|
||||
}
|
||||
|
||||
#[test]
|
||||
/// The interrupted case, which decides what a resumed conversation is built
|
||||
/// from: whatever the phone was shown. The deltas that arrived before the
|
||||
@@ -1862,6 +1931,7 @@ mod tests {
|
||||
Event::UsageDelta {
|
||||
tokens: 12,
|
||||
context: Some(12),
|
||||
tokens_per_second: None,
|
||||
},
|
||||
]);
|
||||
let messages = conversation(&path);
|
||||
|
||||
Reference in new issue
Block a user