app/bench-fixture: the synthetic transcript P0's benchmark opens in both apps
Deterministic (seeded), in the app's own event model rather than a real transcript: 3,601 events split into a 3,200-event opening backlog and a 400-event tail both bench harnesses replay as the streaming phase, with headings, inline markdown, fenced code in six languages, a table, tool calls with kilobyte-scale input/output, and two embedded PNGs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
bfe93c4188
commit
0be6a571c4
5 files changed
+3815
No files matched your search
@@ -0,0 +1,28 @@
|
||||
# The P0 benchmark fixture
|
||||
|
||||
`transcript.jsonl` is a synthetic transcript in the app's own event model (the JSON lines
|
||||
`GET /sessions/{id}/transcript` returns; see `Events.kt`'s `parseSeqEvent` and
|
||||
`server/src/session/driver.rs`) -- never a real one. It is what both the Compose `bench` build
|
||||
and iris's bench build open with no server, so the two apps draw exactly the same content and a
|
||||
frame-time comparison is measuring the renderer rather than the data.
|
||||
|
||||
Generated by `./generate.py` (Python stdlib only, seeded -- `SEED = 20260905` -- so re-running it
|
||||
reproduces the same file byte for byte). It writes into `assets/` -- a separate directory from this
|
||||
script and README, because the Compose `bench` build type points its own asset source set straight
|
||||
at `assets/` (`app/androidApp/build.gradle.kts`'s `sourceSets { getByName("bench") }`), and a Python
|
||||
script and a markdown file have no business inside an APK:
|
||||
|
||||
- `transcript.jsonl` -- 3,601 events. The first 3,200 (`BACKLOG_COUNT`) are the scrolled-back
|
||||
history the benchmark opens with: user turns, tool calls with kilobyte-scale input/output,
|
||||
assistant replies built from headings, bold/italic/inline code, a link, fenced code blocks that
|
||||
rotate through rust/kotlin/python/sh/json/toml, a markdown table, two embedded images, and
|
||||
periodic `usageDelta`/`compacted` events. The remaining 400 (`STREAM_COUNT`) are not part of the
|
||||
opening window -- both bench harnesses replay them at a fixed rate (20/s) through the same live
|
||||
fold path a real SSE reply arrives on, which is P0's "streaming phase."
|
||||
- `bench1.png`, `bench2.png` -- tiny (8x8) flat-colour PNGs, base64-free on disk but served the
|
||||
same way a real attachment is (`GET /sessions/{id}/files/{name}`), referenced by the two
|
||||
`"type":"image"` events in the transcript.
|
||||
|
||||
Regenerate after changing the shape (a new event type, a different backlog/stream split) with
|
||||
`./generate.py`, and commit the result -- it is checked in rather than generated at build time so
|
||||
both apps' bench builds embed the identical bytes without needing this script at build time.
|
||||
Reference in new issue
Block a user