The bench fixture streams a reply shaped like a real one, and keeps the run-on as stress
Iris, on the two findings from the incremental-text investigation: "let's switch to new lines for the test, and also let's keep the single line around for stress + could be something to try to optimize later." The streamed tail now takes a blank line every 4-12 deltas, so it is 53 markdown blocks with a longest of 502 characters instead of one block of 14,888 -- against a measured p50 of 147 and a largest-ever 1,580 over 7,706 blocks of real assistant messages. Layer 1's streaming frame went from p50 3.86ms / p90 8.65ms / worst 10.95ms to p50 2.20 / p90 5.90 / worst 8.78. The run-on message is kept as the first two backlog events, 14,824 characters in one block, just under text_cap's 16 KiB so it draws in full. The *streaming* pathology stays in frame_profile.rs rather than the fixture: it needs a growing block, and iterating on it there costs a second instead of a two-minute phone run. Adding it is purely additive -- the random state is saved and restored around those two events, so every other backlog event is byte-identical. That is not tidiness: the first attempt shifted the backlog and broke `a_long_press_and_drag_selects_text`, which replays a real recording at (300, 1000) and needs the content it was recorded against to still be there. BACKLOG_COUNT is 3202 now, in generate.py, fixture.rs and BenchFixture.kt, which split the file by line index. And the answer to Iris's question, which the code already had: the newest message does *not* cap. `build_row`'s `cap` is false for the live tail because a row that grew while capped would appear to stop growing, and a reply growing past the cap is never caught either since it grows through apply_delta. So a streamed block's shaping cost has no ceiling -- ~29ms per delta at 50k characters, ~58ms at 100k. Recorded but not chased: the emulator's `stream: build p50` did not move (10.4 -> 10.5ms) while layer 1's frame nearly halved, so most of a streaming frame on a GPU path is the whole-arena primitive re-upload layer 1 never performs -- 11,568 primitives rewritten per delta, with the fling phase as the control at 0.4ms for the same primitives moved through move_offsets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
43a3a345e4
commit
77cee6a8fa
7 files changed
+3816
-3619
No files matched your search
@@ -12,17 +12,37 @@ script and README, because the Compose `bench` build type points its own asset s
|
||||
at `assets/` (`app/androidApp/build.gradle.kts`'s `sourceSets { getByName("bench") }`), and a Python
|
||||
script and a markdown file have no business inside an APK:
|
||||
|
||||
- `transcript.jsonl` -- 3,601 events. The first 3,200 (`BACKLOG_COUNT`) are the scrolled-back
|
||||
- `transcript.jsonl` -- 3,603 events. The first 3,202 (`BACKLOG_COUNT`) are the scrolled-back
|
||||
history the benchmark opens with: user turns, tool calls with kilobyte-scale input/output,
|
||||
assistant replies built from headings, bold/italic/inline code, a link, fenced code blocks that
|
||||
rotate through rust/kotlin/python/sh/json/toml, a markdown table, two embedded images, and
|
||||
periodic `usageDelta`/`compacted` events. The remaining 400 (`STREAM_COUNT`) are not part of the
|
||||
opening window -- both bench harnesses replay them at a fixed rate (20/s) through the same live
|
||||
fold path a real SSE reply arrives on, which is P0's "streaming phase."
|
||||
|
||||
**The streamed reply has a blank line every few deltas** (2026-09-09), so the markdown block a
|
||||
delta lands in stays the size a real reply's blocks are -- 53 blocks, longest 502 characters,
|
||||
against a measured p50 of 147 and a largest-ever 1,580 over 7,706 blocks of real assistant
|
||||
messages. It used to be one run-on 14,888-character block, and since a row re-shapes the block a
|
||||
delta lands in, every delta re-shaped all of it: quadratic in the reply's length, and 9.5ms of
|
||||
frame time on a phone spent on a shape that does not occur. docs/RUST.md's "Incremental text"
|
||||
has the measurements.
|
||||
|
||||
**The run-on message is kept**, as the first two events of the backlog: 14,824 characters in a
|
||||
single block, just under `text_cap`'s 16 KiB `MESSAGE_BYTES` so it draws in full rather than
|
||||
behind a "Show all". It is deliberately *not* streamed -- the repeated-reshape pathology needs a
|
||||
growing block, and that lives in `app-rust/tests/frame_profile.rs` where it can be iterated on
|
||||
in a second rather than in a two-minute phone run. It is emitted with the random state saved and
|
||||
restored around it, so adding it left every other backlog event byte-identical; that is what
|
||||
keeps `phone_screen.rs`'s recorded gestures landing on the content they were recorded against.
|
||||
- `bench1.png`, `bench2.png` -- tiny (8x8) flat-colour PNGs, base64-free on disk but served the
|
||||
same way a real attachment is (`GET /sessions/{id}/files/{name}`), referenced by the two
|
||||
`"type":"image"` events in the transcript.
|
||||
|
||||
`BACKLOG_COUNT` lives in three places and moves in all of them or none: here, `app-rust/src/ui/
|
||||
fixture.rs` and `BenchFixture.kt`. The split is by line index, so a stale copy makes that app open
|
||||
a different half of the file.
|
||||
|
||||
Regenerate after changing the shape (a new event type, a different backlog/stream split) with
|
||||
`./generate.py`, and commit the result -- it is checked in rather than generated at build time so
|
||||
both apps' bench builds embed the identical bytes without needing this script at build time.
|
||||
Reference in new issue
Block a user