docs/RUST.md: bench v2 spec and the emulator smoke run
Iris's ask (2026-09-06): the fling should travel much faster for stress-testing, plus typing and keyboard phases. Written once into the P0 box so the iris agent implements the identical four-phase spec -- constants, ordering and report shape -- rather than a second one that looks the same but isn't. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Fable 5.1 committed
1 parent
a8d24553d5
commit
5b2ca039f1
1 file changed
+180
+180
@@ -4135,6 +4135,186 @@ device.
|
|||||||
back with a diagnostic that will say the real story on the next
|
back with a diagnostic that will say the real story on the next
|
||||||
run either way.
|
run either way.
|
||||||
|
|
||||||
|
**Benchmark v2 (2026-09-06), asked for by Iris after using the
|
||||||
|
Compose build on her phone**: "it doesn't fling like I typically do
|
||||||
|
when scrolling up to find old messages. It should travel way faster
|
||||||
|
which is better for stress testing. You may also want to add typing
|
||||||
|
in the textbox as well and seeing how performant wrapping & pushing
|
||||||
|
the transcript up are, and also keyboard performance if possible."
|
||||||
|
**This is the one spec** -- written once here so both apps' "Run
|
||||||
|
benchmark" implement the identical four phases; a change to a
|
||||||
|
constant below has to be made in both `app/`'s `BenchRun.kt` and
|
||||||
|
`iris/`'s bench client, together, or the two reports stop measuring
|
||||||
|
the same thing while still looking like they do.
|
||||||
|
|
||||||
|
1. **fling.** Starting pinned at the newest end
|
||||||
|
(`listState.scrollToItem(0)` / iris's equivalent), 8 flings away
|
||||||
|
from it (toward older messages) through the list's own real fling
|
||||||
|
path -- Compose: `LazyListState.scroll { with(flingBehavior) {
|
||||||
|
performFling(velocity) } }` using the screen's actual
|
||||||
|
`FlingBehavior` (`ScrollableDefaults.flingBehavior()`, since
|
||||||
|
`TranscriptList`'s `LazyColumn` never overrides it -- **not**
|
||||||
|
`animateScrollBy`, which can only ever cover the fixed distance
|
||||||
|
and time it is given and was Iris's complaint) -- each fling's
|
||||||
|
`initialVelocity = 12,000 px/s`. That number is well above a
|
||||||
|
moderate tween-swipe's implied speed (v1's `SWIPE_PX`/`SWIPE_MS`
|
||||||
|
is roughly 4,500 px/s) and is meant to be a hard, fast flick for
|
||||||
|
stress-testing, per Iris's ask. After each fling, wait for
|
||||||
|
`isScrollInProgress` to clear (cap 3s; `performFling` already
|
||||||
|
suspends until its own decay ends, this is belt-and-suspenders)
|
||||||
|
plus 300ms between flings. Then 8 more flings back toward the
|
||||||
|
newest end (`-12,000 px/s`). Record the list's first visible
|
||||||
|
index/offset at the start, after the 8 outward flings, and at the
|
||||||
|
end, so the two apps' *travel* can be compared directly rather
|
||||||
|
than just their frame times.
|
||||||
|
2. **stream. Unchanged from v1**: 400 tail events at 20/s (20
|
||||||
|
seconds), pinned to the newest end before it starts (the same
|
||||||
|
"Jump to latest" pin `stream-bench.sh` does).
|
||||||
|
3. **type.** Pin to the newest end, focus the composer, show the IME
|
||||||
|
if the platform allows it, then insert this **exact 600-character
|
||||||
|
string** one character per 50ms through the composer's real
|
||||||
|
`TextFieldValue` state (Compose: the same `input` state
|
||||||
|
`onValueChange` writes; iris: whatever holds the composer's text
|
||||||
|
today), then delete it the same way, one character per 50ms.
|
||||||
|
Chosen for long, multisyllabic words specifically so the composer
|
||||||
|
wraps across lines and the transcript above it is pushed upward
|
||||||
|
by a growing box, which is what Iris asked to see measured:
|
||||||
|
|
||||||
|
Benchmarking this transcript screen requires unusually long, multisyllabic words so wrapping and reflow are properly exercised: internationalization, counterproductiveness, disproportionately, incomprehensibility, deinstitutionalization, uncharacteristically, overenthusiastically, misunderstanding, straightforwardness, telecommunications, and interdisciplinary collaboration all push a narrow composer field to wrap across several lines while the transcript above is pushed upward by the growing keyboard-adjacent box, which is exactly what a real reader typing a long message sees happening now!!!
|
||||||
|
|
||||||
|
Report whether the IME was actually open during typing (it should
|
||||||
|
be, from this phase's own show-IME step -- see phase 4 for what
|
||||||
|
to say if the platform refuses to show it at all).
|
||||||
|
4. **keyboard.** Show the IME (`WindowInsetsControllerCompat.show
|
||||||
|
(WindowInsetsCompat.Type.ime())` against the window/view; iris's
|
||||||
|
equivalent through its own shell), wait 1s, hide it, wait 1s;
|
||||||
|
five cycles. Confirm each show/hide with the platform's own
|
||||||
|
answer (Compose: `ViewCompat.getRootWindowInsets(view)
|
||||||
|
?.isVisible(WindowInsetsCompat.Type.ime())`, i.e. the same
|
||||||
|
`WindowInsets.isImeVisible` fact `SessionScreen`'s composer-inset
|
||||||
|
bug fix already relies on) rather than assuming the request
|
||||||
|
worked -- if it is never confirmed shown even once, the report's
|
||||||
|
`keyboard:` line has to say "**keyboard: could not be shown**"
|
||||||
|
in words (UI_RULES: never present an inferred value as a
|
||||||
|
measured one, and design the unknown/failed state before the
|
||||||
|
answer's).
|
||||||
|
|
||||||
|
**Frame accounting**: one recorder, not two. Mark each phase's start
|
||||||
|
in the existing per-frame recorder (Compose: `FrameStats.markPhase
|
||||||
|
(name)`, a list of `(name, frameIndexAtStart, wallClockAtStart)`
|
||||||
|
alongside the existing `total`/`waited`/... arrays) and slice the
|
||||||
|
same `FrameMetrics` samples by phase afterward
|
||||||
|
(`FrameStats.phaseLines`) rather than running a second listener.
|
||||||
|
|
||||||
|
**Report shape**: a `per phase:` block appears once any phase marks
|
||||||
|
exist (empty/absent on an ordinary "Copy" press, which never marks a
|
||||||
|
phase), one entry per phase: frame count, the phase's wall-clock
|
||||||
|
duration, late count/percent (against the same refresh-rate budget
|
||||||
|
the whole-run section uses), p50/p90/p99, and the worst single
|
||||||
|
frame. Then **every existing whole-run section stays, unchanged in
|
||||||
|
shape** -- `frames:`, `where the draw phase went:`, `work since this
|
||||||
|
was last copied:` -- because that is what the emulator-baseline and
|
||||||
|
phone-baseline numbers already on record in this file were read
|
||||||
|
against, and a report that dropped or renamed one of those lines
|
||||||
|
would silently stop being comparable to them. Finally `bench:` gains
|
||||||
|
new lines beside the existing CPU/RSS/battery ones: the fling
|
||||||
|
phase's total travel (start/outward/end index+offset), the typed
|
||||||
|
character count, and the keyboard phase's shown/hidden-confirmed
|
||||||
|
counts (or the "could not be shown" line).
|
||||||
|
|
||||||
|
**Compose half: done, 2026-09-06.** `FrameStats.markPhase`/
|
||||||
|
`phaseLines` (`app/androidApp/src/main/kotlin/com/example/aiapp/
|
||||||
|
FrameStats.kt`), `debugReport`'s new `phaseFrames` parameter
|
||||||
|
(`DebugStats.kt`), and `BenchRun.kt`'s four-phase `run` (fling via
|
||||||
|
`ScrollableDefaults.flingBehavior()` captured in `SessionScreen` and
|
||||||
|
passed down since it needs a `@Composable` call site; type via a new
|
||||||
|
`composerFocus: FocusRequester` attached to the composer's
|
||||||
|
`OutlinedTextField` plus a `setComposerText` callback that writes
|
||||||
|
`input` the same way a keystroke does; keyboard via
|
||||||
|
`WindowInsetsControllerCompat` against `LocalView.current`) are all
|
||||||
|
in. `BenchRun.TYPE_TEXT` is the exact 600-character constant quoted
|
||||||
|
above (verified `.length == 600`). A pre-existing, unrelated break
|
||||||
|
in `MainActivity.kt`'s `benchSessionSummary()` (missing several
|
||||||
|
`SessionSummary` constructor arguments added by a change this pass
|
||||||
|
did not otherwise touch -- confirmed pre-existing by reproducing the
|
||||||
|
same compile failure after stashing this pass's own diff) was fixed
|
||||||
|
alongside this, since it blocked `compileBenchKotlin` outright and
|
||||||
|
is in this session's own `app/` scope.
|
||||||
|
|
||||||
|
Checks all clean: `ktfmtFormat`, `compileDebugKotlin`,
|
||||||
|
`compileBenchKotlin`, `lintDebug`, `lintBench` (both "No issues
|
||||||
|
found"), `testDebugUnitTest`. `grep -n "tap [0-9]" app/*.sh` still
|
||||||
|
has its one pre-existing, unrelated hit.
|
||||||
|
|
||||||
|
**Compose bench v2, emulator smoke run, 2026-09-05** (this
|
||||||
|
checkout's AVD, cold `emu up`, `ui-trace` tap-by-label throughout --
|
||||||
|
the dialog needed a swipe to reach "Run benchmark" below the fold,
|
||||||
|
report read back over `adb logcat`):
|
||||||
|
|
||||||
|
ai-app render report
|
||||||
|
device: sdk_gphone64_x86_64 (Google), Android 16
|
||||||
|
build: release
|
||||||
|
|
||||||
|
transcript:
|
||||||
|
124 events, 26 rows, 58 units loaded
|
||||||
|
viewport 1714px, 2 units visible
|
||||||
|
on screen: the list's own 0px, AssistantMsg 18732px
|
||||||
|
0 tool calls and 0 groups open
|
||||||
|
|
||||||
|
per phase:
|
||||||
|
fling: 1620 frames over 32.3s
|
||||||
|
late: 1537 (94.9%)
|
||||||
|
total p50 20.5ms p90 29.2ms p99 45.9ms
|
||||||
|
worst 61.8ms
|
||||||
|
stream: 1079 frames over 20.6s
|
||||||
|
late: 1037 (96.1%)
|
||||||
|
total p50 21.0ms p90 33.5ms p99 39.3ms
|
||||||
|
worst 51.2ms
|
||||||
|
type: 3568 frames over 61.4s
|
||||||
|
late: 3536 (99.1%)
|
||||||
|
total p50 23.8ms p90 32.1ms p99 38.5ms
|
||||||
|
worst 50.3ms
|
||||||
|
keyboard: 215 frames over 10.0s
|
||||||
|
late: 212 (98.6%)
|
||||||
|
total p50 21.3ms p90 37.6ms p99 48.4ms
|
||||||
|
worst 50.2ms
|
||||||
|
|
||||||
|
frames:
|
||||||
|
6482 frames over 124.3s at 60Hz (16.7ms budget)
|
||||||
|
late: 6322 (97.5%)
|
||||||
|
total p50 21.7ms p90 33.1ms p99 45.3ms
|
||||||
|
gpu p50 17.4ms p90 27.0ms p99 30.6ms
|
||||||
|
|
||||||
|
where the draw phase went:
|
||||||
|
draw phase 1.27ms per frame, of which:
|
||||||
|
the transcript: 0.16ms (measure 0.09, place 0.07, record 0.00)
|
||||||
|
everything else: 1.10ms (87%)
|
||||||
|
|
||||||
|
bench:
|
||||||
|
fling: 8 flings out + 8 back at 12000px/s, travel start=idx=0/off=0px outward=idx=218/off=73px end=idx=0/off=0px
|
||||||
|
scroll: 6 cycles (24 swipes, legacy tween), streamed 400/400 fixture events
|
||||||
|
type: 600 characters inserted then deleted, one per 50ms
|
||||||
|
keyboard: shown 5/5, hidden 5/5 (confirmed via isImeVisible)
|
||||||
|
process CPU time over this run: 61192ms
|
||||||
|
peak RSS: 195716kB
|
||||||
|
battery current: mean 900000µA over 125 samples (min 900000, max 900000)
|
||||||
|
|
||||||
|
Read this the same way the v1 emulator smoke run above is read: it
|
||||||
|
proves the harness runs end to end and produces every field this
|
||||||
|
spec asked for, not a phone number -- software rasterisation, and
|
||||||
|
the fixed 900mA battery reading is the emulator's mocked charger
|
||||||
|
again. Two things worth carrying forward: the **fling phase reached
|
||||||
|
index 218** in 8 flings (against v1's `animateScrollBy` loop, which
|
||||||
|
never moved past a handful of indices in the same 8-swipe count),
|
||||||
|
which is the direct evidence the new fling travels "way faster" as
|
||||||
|
asked; and **the emulator's software keyboard toggled and was
|
||||||
|
confirmed by `isImeVisible` all 10 times**, so phase 4 is not a
|
||||||
|
guaranteed "could not be shown" on every platform, only where the
|
||||||
|
IME genuinely refuses. `frames:`'s 6,482-frame, 124.3s total matches
|
||||||
|
the sum of the four phase durations (32.3+20.6+61.4+10.0 ≈ 124.3s),
|
||||||
|
confirming the phase marks partition the whole run rather than
|
||||||
|
overlapping or dropping frames between them.
|
||||||
|
|
||||||
- [ ] **P1 — session screen parity.** History paging backward (with the
|
- [ ] **P1 — session screen parity.** History paging backward (with the
|
||||||
page-boundary healing `client-core` does not have yet, below),
|
page-boundary healing `client-core` does not have yet, below),
|
||||||
`TranscriptSource`-backed cache/server stitching, jump-to-latest,
|
`TranscriptSource`-backed cache/server stitching, jump-to-latest,
|
||||||
|
|||||||
Reference in new issue
Block a user