The bench fixture streams a reply shaped like a real one, and keeps the run-on as stress

Iris, on the two findings from the incremental-text investigation:
"let's switch to new lines for the test, and also let's keep the single
line around for stress + could be something to try to optimize later."

The streamed tail now takes a blank line every 4-12 deltas, so it is 53
markdown blocks with a longest of 502 characters instead of one block of
14,888 -- against a measured p50 of 147 and a largest-ever 1,580 over
7,706 blocks of real assistant messages. Layer 1's streaming frame went
from p50 3.86ms / p90 8.65ms / worst 10.95ms to p50 2.20 / p90 5.90 /
worst 8.78.

The run-on message is kept as the first two backlog events, 14,824
characters in one block, just under text_cap's 16 KiB so it draws in
full. The *streaming* pathology stays in frame_profile.rs rather than the
fixture: it needs a growing block, and iterating on it there costs a
second instead of a two-minute phone run.

Adding it is purely additive -- the random state is saved and restored
around those two events, so every other backlog event is byte-identical.
That is not tidiness: the first attempt shifted the backlog and broke
`a_long_press_and_drag_selects_text`, which replays a real recording at
(300, 1000) and needs the content it was recorded against to still be
there. BACKLOG_COUNT is 3202 now, in generate.py, fixture.rs and
BenchFixture.kt, which split the file by line index.

And the answer to Iris's question, which the code already had: the newest
message does *not* cap. `build_row`'s `cap` is false for the live tail
because a row that grew while capped would appear to stop growing, and a
reply growing past the cap is never caught either since it grows through
apply_delta. So a streamed block's shaping cost has no ceiling -- ~29ms
per delta at 50k characters, ~58ms at 100k.

Recorded but not chased: the emulator's `stream: build p50` did not move
(10.4 -> 10.5ms) while layer 1's frame nearly halved, so most of a
streaming frame on a GPU path is the whole-arena primitive re-upload
layer 1 never performs -- 11,568 primitives rewritten per delta, with the
fling phase as the control at 0.4ms for the same primitives moved
through move_offsets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed 2026-09-09 01:32:59 -04:00
1 parent 43a3a345e4
commit 77cee6a8fa
7 files changed
+3816 -3619

No files matched your search

+66 -4
View File
@@ -26,8 +26,40 @@ import zlib
from pathlib import Path
SEED = 20260905
BACKLOG_COUNT = 3200
# The stress message below, two events, added to the 3,200 the backlog used to be. Both apps
# hardcode this to split the file by line index (`fixture.rs`'s `BACKLOG_COUNT`,
# `BenchFixture.kt`'s), so it moves in three places or none.
STRESS_EVENTS = 2
BACKLOG_COUNT = 3200 + STRESS_EVENTS
STREAM_COUNT = 400
# How many deltas fold into one markdown block of the streamed reply.
#
# **The blank line between them is the point** (2026-09-09). Until this run the streaming tail
# appended `paragraph(5) + " "` STREAM_COUNT times with no blank line anywhere, so all 400 deltas
# folded into a *single* 14,888-character block -- and a row re-shapes the block a delta lands in
# (`RowBlocks::apply_delta`), so every delta re-shaped the whole thing. That is quadratic in the
# reply's length, and it put 9.5ms of frame time on Iris's phone into a shape that does not occur:
# measured over 7,706 top-level blocks from 3,675 real assistant messages, block length is p50 147
# characters, p90 449, p99 836, largest 1,580, nothing above 4,000. `paragraph(5)` is ~35
# characters, so 4-12 of them per block lands in that range.
#
# The run-on version is kept, as STRESS_CHARS below -- Iris, 2026-09-09: "let's switch to new
# lines for the test, and also let's keep the single line around for stress".
DELTAS_PER_BLOCK = (4, 12)
# One deliberately pathological message in the backlog: a single markdown block with no blank line
# in it, the shape the streaming tail used to have. Sized just under `text_cap`'s MESSAGE_BYTES
# (16 KiB) so it is drawn in full rather than behind a "Show all" -- which makes it a genuine
# stress for one-shot shaping and puts a row right on the cap boundary, where nothing else is.
#
# It is *not* streamed: the repeated-reshape pathology needs a growing block, and that lives in
# `app-rust/tests/frame_profile.rs`'s `what_reshaping_a_growing_message_costs`, where it can be
# iterated on in a second rather than in a two-minute phone run. Note that the cap would not save
# a real one anyway -- `row::build_row`'s `cap` is deliberately `false` for the live tail, because
# a row that grew while capped would appear to stop growing, so a streamed block's shaping cost
# has no ceiling.
STRESS_CHARS = 14_800
HERE = Path(__file__).resolve().parent / "assets"
random.seed(SEED)
@@ -122,6 +154,26 @@ def main():
emit("status", state="running")
emit("settings", model="bench-model", permissionMode="auto")
# The stress message, emitted first and with the random state put back afterwards, so that
# adding it is **purely additive**: every turn the loop below generates is byte-identical to
# what it generated before this existed, and only the streaming tail changed. That matters
# because the fixture's opening view is what `phone_screen.rs`'s recorded gestures press --
# `a_long_press_and_drag_selects_text` drives a real recording at (300, 1000) and fails the
# moment different content lands under it. `ts` still shifts by the two draws, which nothing
# reads.
rng_state = random.getstate()
emit(
"userMessage",
text="And the pathological one: a reply with no paragraph break in it.",
id=None,
attachments=[],
)
run_on = ""
while len(run_on) < STRESS_CHARS:
run_on += paragraph(5) + " "
emit("assistantText", delta=run_on.rstrip())
random.setstate(rng_state)
image_refs = []
turn = 0
while seq <= BACKLOG_COUNT:
@@ -167,11 +219,21 @@ def main():
trigger="auto",
)
# The streaming-phase tail: one long reply, built entirely from text deltas, the shape a
# bench harness replays at a fixed events/sec through the live fold path.
# The streaming-phase tail: one reply built entirely from text deltas, the shape a bench
# harness replays at a fixed events/sec through the live fold path -- with a blank line every
# few deltas, so the block a delta lands in stays the size a real reply's blocks are. See
# DELTAS_PER_BLOCK for what the alternative measured.
emit("userMessage", text="One more, streamed live for the benchmark's timing phase.", id=None, attachments=[])
until_break = random.randint(*DELTAS_PER_BLOCK)
while seq <= BACKLOG_COUNT + STREAM_COUNT:
emit("assistantText", delta=paragraph(5) + " ")
until_break -= 1
if until_break <= 0:
# The break rides on the last delta of the block rather than being an event of its
# own, so STREAM_COUNT still counts deltas a reader sees text arrive from.
emit("assistantText", delta=paragraph(5) + "\n\n")
until_break = random.randint(*DELTAS_PER_BLOCK)
else:
emit("assistantText", delta=paragraph(5) + " ")
emit("status", state="idle")
(HERE / "transcript.jsonl").write_text("\n".join(lines) + "\n")