45 lines
3.0 KiB
Markdown
45 lines
3.0 KiB
Markdown
# The P0 benchmark fixture
|
|
|
|
`transcript.jsonl` is a synthetic transcript in the app's event model (the JSON lines
|
|
`GET /sessions/{id}/transcript` returns; see `event-model` and
|
|
`server/src/session/driver.rs`) -- never a real one. The `bench` build opens it without a
|
|
server so renderer measurements use deterministic data.
|
|
|
|
Generated by `./generate.py` (Python stdlib only, seeded -- `SEED = 20260905` -- so re-running it
|
|
reproduces the same file byte for byte). It writes into `assets/`; only those generated assets are
|
|
embedded in the benchmark APK.
|
|
|
|
- `transcript.jsonl` -- 3,603 events. The first 3,202 (`BACKLOG_COUNT`) are the scrolled-back
|
|
history the benchmark opens with: user turns, tool calls with kilobyte-scale input/output,
|
|
assistant replies built from headings, bold/italic/inline code, a link, fenced code blocks that
|
|
rotate through rust/kotlin/python/sh/json/toml, a markdown table, two embedded images, and
|
|
periodic `usageDelta`/`compacted` events. The remaining 400 (`STREAM_COUNT`) are not part of the
|
|
opening window -- both bench harnesses replay them at a fixed rate (20/s) through the same live
|
|
fold path a real SSE reply arrives on, which is P0's "streaming phase."
|
|
|
|
**The streamed reply has a blank line every few deltas** (2026-09-09), so the markdown block a
|
|
delta lands in stays the size a real reply's blocks are -- 53 blocks, longest 502 characters,
|
|
against a measured p50 of 147 and a largest-ever 1,580 over 7,706 blocks of real assistant
|
|
messages. It used to be one run-on 14,888-character block, and since a row re-shapes the block a
|
|
delta lands in, every delta re-shaped all of it: quadratic in the reply's length, and 9.5ms of
|
|
frame time on a phone spent on a shape that does not occur. docs/RUST.md's "Incremental text"
|
|
has the measurements.
|
|
|
|
**The run-on message is kept**, as the first two events of the backlog: 14,824 characters in a
|
|
single block, just under `text_cap`'s 16 KiB `MESSAGE_BYTES` so it draws in full rather than
|
|
behind a "Show all". It is deliberately *not* streamed -- the repeated-reshape pathology needs a
|
|
growing block, and that lives in the UI profiling rig where it can be iterated on
|
|
in a second rather than in a two-minute phone run. It is emitted with the random state saved and
|
|
restored around it, so adding it left every other backlog event byte-identical; that is what
|
|
keeps `phone_screen.rs`'s recorded gestures landing on the content they were recorded against.
|
|
- `bench1.png`, `bench2.png` -- tiny (8x8) flat-colour PNGs, base64-free on disk but served the
|
|
same way a real attachment is (`GET /sessions/{id}/files/{name}`), referenced by the two
|
|
`"type":"image"` events in the transcript.
|
|
|
|
`BACKLOG_COUNT` lives here, in `generate.py`, and in `app/src/ui/fixture.rs`; change all three
|
|
together. The split is by line index.
|
|
|
|
Regenerate after changing the shape (a new event type, a different backlog/stream split) with
|
|
`./generate.py`, and commit the result -- it is checked in rather than generated at build time so
|
|
the benchmark embeds identical bytes without needing this script at build time.
|