The bench fixture streams a reply shaped like a real one, and keeps the run-on as stress
Iris, on the two findings from the incremental-text investigation: "let's switch to new lines for the test, and also let's keep the single line around for stress + could be something to try to optimize later." The streamed tail now takes a blank line every 4-12 deltas, so it is 53 markdown blocks with a longest of 502 characters instead of one block of 14,888 -- against a measured p50 of 147 and a largest-ever 1,580 over 7,706 blocks of real assistant messages. Layer 1's streaming frame went from p50 3.86ms / p90 8.65ms / worst 10.95ms to p50 2.20 / p90 5.90 / worst 8.78. The run-on message is kept as the first two backlog events, 14,824 characters in one block, just under text_cap's 16 KiB so it draws in full. The *streaming* pathology stays in frame_profile.rs rather than the fixture: it needs a growing block, and iterating on it there costs a second instead of a two-minute phone run. Adding it is purely additive -- the random state is saved and restored around those two events, so every other backlog event is byte-identical. That is not tidiness: the first attempt shifted the backlog and broke `a_long_press_and_drag_selects_text`, which replays a real recording at (300, 1000) and needs the content it was recorded against to still be there. BACKLOG_COUNT is 3202 now, in generate.py, fixture.rs and BenchFixture.kt, which split the file by line index. And the answer to Iris's question, which the code already had: the newest message does *not* cap. `build_row`'s `cap` is false for the live tail because a row that grew while capped would appear to stop growing, and a reply growing past the cap is never caught either since it grows through apply_delta. So a streamed block's shaping cost has no ceiling -- ~29ms per delta at 50k characters, ~58ms at 100k. Recorded but not chased: the emulator's `stream: build p50` did not move (10.4 -> 10.5ms) while layer 1's frame nearly halved, so most of a streaming frame on a GPU path is the whole-arena primitive re-upload layer 1 never performs -- 11,568 primitives rewritten per delta, with the fling phase as the control at 0.4ms for the same primitives moved through move_offsets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
43a3a345e4
commit
77cee6a8fa
7 files changed
+3816
-3619
No files matched your search
@@ -26,8 +26,40 @@ import zlib
|
||||
from pathlib import Path
|
||||
|
||||
SEED = 20260905
|
||||
BACKLOG_COUNT = 3200
|
||||
# The stress message below, two events, added to the 3,200 the backlog used to be. Both apps
|
||||
# hardcode this to split the file by line index (`fixture.rs`'s `BACKLOG_COUNT`,
|
||||
# `BenchFixture.kt`'s), so it moves in three places or none.
|
||||
STRESS_EVENTS = 2
|
||||
BACKLOG_COUNT = 3200 + STRESS_EVENTS
|
||||
STREAM_COUNT = 400
|
||||
|
||||
# How many deltas fold into one markdown block of the streamed reply.
|
||||
#
|
||||
# **The blank line between them is the point** (2026-09-09). Until this run the streaming tail
|
||||
# appended `paragraph(5) + " "` STREAM_COUNT times with no blank line anywhere, so all 400 deltas
|
||||
# folded into a *single* 14,888-character block -- and a row re-shapes the block a delta lands in
|
||||
# (`RowBlocks::apply_delta`), so every delta re-shaped the whole thing. That is quadratic in the
|
||||
# reply's length, and it put 9.5ms of frame time on Iris's phone into a shape that does not occur:
|
||||
# measured over 7,706 top-level blocks from 3,675 real assistant messages, block length is p50 147
|
||||
# characters, p90 449, p99 836, largest 1,580, nothing above 4,000. `paragraph(5)` is ~35
|
||||
# characters, so 4-12 of them per block lands in that range.
|
||||
#
|
||||
# The run-on version is kept, as STRESS_CHARS below -- Iris, 2026-09-09: "let's switch to new
|
||||
# lines for the test, and also let's keep the single line around for stress".
|
||||
DELTAS_PER_BLOCK = (4, 12)
|
||||
|
||||
# One deliberately pathological message in the backlog: a single markdown block with no blank line
|
||||
# in it, the shape the streaming tail used to have. Sized just under `text_cap`'s MESSAGE_BYTES
|
||||
# (16 KiB) so it is drawn in full rather than behind a "Show all" -- which makes it a genuine
|
||||
# stress for one-shot shaping and puts a row right on the cap boundary, where nothing else is.
|
||||
#
|
||||
# It is *not* streamed: the repeated-reshape pathology needs a growing block, and that lives in
|
||||
# `app-rust/tests/frame_profile.rs`'s `what_reshaping_a_growing_message_costs`, where it can be
|
||||
# iterated on in a second rather than in a two-minute phone run. Note that the cap would not save
|
||||
# a real one anyway -- `row::build_row`'s `cap` is deliberately `false` for the live tail, because
|
||||
# a row that grew while capped would appear to stop growing, so a streamed block's shaping cost
|
||||
# has no ceiling.
|
||||
STRESS_CHARS = 14_800
|
||||
HERE = Path(__file__).resolve().parent / "assets"
|
||||
|
||||
random.seed(SEED)
|
||||
@@ -122,6 +154,26 @@ def main():
|
||||
emit("status", state="running")
|
||||
emit("settings", model="bench-model", permissionMode="auto")
|
||||
|
||||
# The stress message, emitted first and with the random state put back afterwards, so that
|
||||
# adding it is **purely additive**: every turn the loop below generates is byte-identical to
|
||||
# what it generated before this existed, and only the streaming tail changed. That matters
|
||||
# because the fixture's opening view is what `phone_screen.rs`'s recorded gestures press --
|
||||
# `a_long_press_and_drag_selects_text` drives a real recording at (300, 1000) and fails the
|
||||
# moment different content lands under it. `ts` still shifts by the two draws, which nothing
|
||||
# reads.
|
||||
rng_state = random.getstate()
|
||||
emit(
|
||||
"userMessage",
|
||||
text="And the pathological one: a reply with no paragraph break in it.",
|
||||
id=None,
|
||||
attachments=[],
|
||||
)
|
||||
run_on = ""
|
||||
while len(run_on) < STRESS_CHARS:
|
||||
run_on += paragraph(5) + " "
|
||||
emit("assistantText", delta=run_on.rstrip())
|
||||
random.setstate(rng_state)
|
||||
|
||||
image_refs = []
|
||||
turn = 0
|
||||
while seq <= BACKLOG_COUNT:
|
||||
@@ -167,11 +219,21 @@ def main():
|
||||
trigger="auto",
|
||||
)
|
||||
|
||||
# The streaming-phase tail: one long reply, built entirely from text deltas, the shape a
|
||||
# bench harness replays at a fixed events/sec through the live fold path.
|
||||
# The streaming-phase tail: one reply built entirely from text deltas, the shape a bench
|
||||
# harness replays at a fixed events/sec through the live fold path -- with a blank line every
|
||||
# few deltas, so the block a delta lands in stays the size a real reply's blocks are. See
|
||||
# DELTAS_PER_BLOCK for what the alternative measured.
|
||||
emit("userMessage", text="One more, streamed live for the benchmark's timing phase.", id=None, attachments=[])
|
||||
until_break = random.randint(*DELTAS_PER_BLOCK)
|
||||
while seq <= BACKLOG_COUNT + STREAM_COUNT:
|
||||
emit("assistantText", delta=paragraph(5) + " ")
|
||||
until_break -= 1
|
||||
if until_break <= 0:
|
||||
# The break rides on the last delta of the block rather than being an event of its
|
||||
# own, so STREAM_COUNT still counts deltas a reader sees text arrive from.
|
||||
emit("assistantText", delta=paragraph(5) + "\n\n")
|
||||
until_break = random.randint(*DELTAS_PER_BLOCK)
|
||||
else:
|
||||
emit("assistantText", delta=paragraph(5) + " ")
|
||||
emit("status", state="idle")
|
||||
|
||||
(HERE / "transcript.jsonl").write_text("\n".join(lines) + "\n")
|
||||
|
||||
Reference in new issue
Block a user