RUST.md, IRIS.md, IRIS_TODO.md, DECISIONS.md: record I5's Android integration and measurements
I5's transcript screen now runs on-device against a real ai-server on iris-android-app's new transcript-screen feature (extends I2's shell rather than a third one), with real scrolling, real touch-drag panning and tap-by-name accessibility all confirmed by screenshot/log evidence. I4's own emulator-side check (tap-by-name on the tabs demo) closed the same session, so its box ticks [x] now. Still [~], not [x]: the render-time number RUST.md's recommendation wants for iris couldn't be produced this pass, for a precise and recorded reason rather than a vague one -- dumpsys gfxinfo cannot see a SurfaceView's own GPU-drawn frames at all (0 frames reported across a gesture loop that visibly scrolled), and a SurfaceFlinger --latency fallback gave no per-frame history either on this Android version. The Compose side of the same loop did produce a real number under identical conditions (8.96% janky, 99th percentile 150ms), so this is now a one-sided number rather than a missing one on both sides. Also found and recorded: the AVD's saved snapshot carries a GPU config across restarts, so switching between the documented Vulkan boot recipes needs a cold boot (clearing snapshots/) that the emu wrapper does not force -- cost three different-looking crashes before the pattern was the snapshot, not the code. DECISIONS.md's DEFERRED item is updated with the numbers Iris needs to weigh the iris-vs-Masonry call; the call itself stays hers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
bf5087a598
commit
d17040b601
4 files changed
+388
-85
No files matched your search
+28
-6
@@ -22,9 +22,31 @@ marked **DEFERRED** are ones the agent chose not to decide alone.
|
|||||||
pinned CA is a path given on the command line, not baked in. Chosen so
|
pinned CA is a path given on the command line, not baked in. Chosen so
|
||||||
the phone and desktop share one enrolment format and no second one is
|
the phone and desktop share one enrolment format and no second one is
|
||||||
invented.
|
invented.
|
||||||
- **Order of remaining work**: finish the two in-flight pieces above, then
|
- **I5's Android integration extends `iris-android-app` (I2's shell)
|
||||||
the transcript screen's Android integration and the `transcript-bench.sh`
|
behind a Cargo feature (`transcript-screen`), rather than a third
|
||||||
comparison against Compose — the numbers the recommendation still lacks.
|
shell crate.** That project already has the Gradle module, the
|
||||||
- **DEFERRED — whether to commit to iris over Masonry for `ai-app`.** Waits
|
`IrisView`/`MainActivity` Java, and the JNI registration; the only
|
||||||
on the bench numbers above; RUST.md's recommendation says what the
|
thing a second screen needs on top is a different `AndroidAppState`,
|
||||||
measurements must show.
|
the same axis `tabs_ui::build`/`transcript_ui::build` already vary
|
||||||
|
along on the winit side. `tabs-screen`/`transcript-screen` are
|
||||||
|
mutually exclusive and each pulls in only its own deps, so the plain
|
||||||
|
tabs build (I2/I4) is untouched.
|
||||||
|
- **Order of remaining work, updated 2026-09-05**: the two in-flight
|
||||||
|
pieces and I5's Android integration are all done; next is giving iris
|
||||||
|
its own frame-timing report so item 3 below can be decided by a number.
|
||||||
|
- **DEFERRED — whether to commit to iris over Masonry for `ai-app`.**
|
||||||
|
Updated 2026-09-05 with what's now known: iris's Android integration is
|
||||||
|
built and confirmed working on a real device (real server, real
|
||||||
|
scrolling, real touch-drag pan, tap-by-name accessibility — RUST.md's I5
|
||||||
|
box), while Masonry's own touch-scroll on Android was already found
|
||||||
|
entirely absent (E2). What's still missing on **both** sides is a
|
||||||
|
render-time number — not because iris doesn't work, but because
|
||||||
|
`dumpsys gfxinfo` cannot see a `SurfaceView`'s GPU-drawn frames at all,
|
||||||
|
so the `transcript-bench.sh`-style comparison this recommendation wanted
|
||||||
|
produced a real number for Compose (8.96% janky, 99th percentile 150ms,
|
||||||
|
same emulator/session) and none at all for iris. The choice in front of
|
||||||
|
Iris: decide now on the structural-plus-functional case already made
|
||||||
|
(iris works end-to-end where Masonry's scroll gesture doesn't exist at
|
||||||
|
all on Android), or wait for iris to grow its own frame-timing
|
||||||
|
instrumentation first so the comparison can be a number rather than a
|
||||||
|
qualitative one. RUST.md's "Recommendation" item 3 has the full account.
|
||||||
@@ -8,6 +8,36 @@ capability that moved. Small and trivial changes do not go here.
|
|||||||
An entry gives the date, what changed, why, and a short before/after where
|
An entry gives the date, what changed, why, and a short before/after where
|
||||||
it helps judge the change without the session that made it. Newest first.
|
it helps judge the change without the session that made it. Newest first.
|
||||||
|
|
||||||
|
## 2026-09-05: `Tasks::redraw_handle` (RUST.md's I5 Android integration)
|
||||||
|
|
||||||
|
New public method on `iris::task::Tasks`, `redraw_handle(&self) ->
|
||||||
|
Arc<dyn RequestRedraw>`. Why: a caller running its own long-lived loop
|
||||||
|
*inside* one spawned task (a live SSE follow, the Android transcript
|
||||||
|
client's `select_session`) has no other way to ask for a frame after each
|
||||||
|
`TaskCtx::update` -- `Tasks::spawn`'s own wrapper only requests one, after
|
||||||
|
the whole async closure finishes, which fits a single request-then-update
|
||||||
|
but not a stream that needs to be seen redrawing after *each* event. This
|
||||||
|
is the same gap `iris/desktop-app`'s module doc names for why it uses
|
||||||
|
winit's `Proxy<AppEvent>` instead of `Tasks` -- android-view has no
|
||||||
|
`Proxy`, so this is what closes it there.
|
||||||
|
|
||||||
|
**A real bug this uncovered, not a hypothetical**: calling the returned
|
||||||
|
handle's `request_redraw()` from the background thread crashed the process
|
||||||
|
(`SIGABRT`, `Result::unwrap() on an Err value: JavaException`) the first
|
||||||
|
time an Android transcript fetch called it a second time. `android/render.rs`'s
|
||||||
|
`AndroidRedrawHandle` was already attaching the calling thread to the JVM
|
||||||
|
correctly, but its `request_redraw` called `View::post_frame_callback`,
|
||||||
|
whose Java side calls `Choreographer.getInstance()` -- which throws unless
|
||||||
|
the *calling* thread already has a `Looper`, and a tokio worker thread,
|
||||||
|
even freshly JNI-attached, has none. Fixed by routing through
|
||||||
|
`View::post_delayed(0)` instead (Android's own thread-safe "queue work onto
|
||||||
|
this View's UI thread" primitive, needing no caller-side `Looper`), landing
|
||||||
|
on a new `IrisViewPeer::delayed_callback` override that drains tasks and
|
||||||
|
renders -- same body as `do_frame`, on the UI thread where
|
||||||
|
`post_frame_callback` is safe again. Any future caller of `redraw_handle()`
|
||||||
|
from a background thread gets this for free; nothing about the fix is
|
||||||
|
specific to the transcript screen.
|
||||||
|
|
||||||
## 2026-09-05: `transcript_ui::build_tree` (RUST.md's E4)
|
## 2026-09-05: `transcript_ui::build_tree` (RUST.md's E4)
|
||||||
|
|
||||||
`transcript_ui::build` claimed the whole window (`ui_state.set_root(tree)`)
|
`transcript_ui::build` claimed the whole window (`ui_state.set_root(tree)`)
|
||||||
|
|||||||
+29
-8
@@ -176,14 +176,35 @@ order and what "done" looks like. Tick and date them in place.
|
|||||||
silently dropped. See RUST.md's I5 box for the full account of what
|
silently dropped. See RUST.md's I5 box for the full account of what
|
||||||
*was* built (the screen, `SpanStyle`, cross-row selection, the growing
|
*was* built (the screen, `SpanStyle`, cross-row selection, the growing
|
||||||
composer).**
|
composer).**
|
||||||
- [ ] **Android integration for this screen does not exist yet.** No
|
- [x] **Android integration for this screen — done, 2026-09-05.**
|
||||||
cdylib/Gradle shell the way `iris-android-app` wraps `tabs-ui` (I2),
|
`iris-android-app`'s `transcript-screen` Cargo feature
|
||||||
so `transcript-bench.sh`'s render-number pass condition against the
|
(`transcript_client.rs`) runs this screen against a real `ai-server`
|
||||||
Compose baseline cannot be run. Needs: real `client-core::ApiClient`/
|
through `client-core`, confirmed on-device: real scrolling, real
|
||||||
`event_stream::follow_session_events` wiring against
|
touch-drag panning, tap-by-name on the composer. Two real bugs found
|
||||||
`app/ui-sandbox.sh --delay` (this crate deliberately fetches nothing
|
and fixed along the way (a missing `INTERNET` permission; a
|
||||||
itself, `transcript-ui/src/lib.rs`'s doc), a new cdylib + Gradle
|
background-thread redraw request that crashed via a `Looper`
|
||||||
module, then the bench script pointed at it.
|
requirement, fixed by routing through `View::post_delayed` — see
|
||||||
|
`IRIS.md`'s `Tasks::redraw_handle` entry). See RUST.md's I5 box,
|
||||||
|
"The Android integration, done 2026-09-05" for the full account.
|
||||||
|
- [ ] **A render-time number for iris, comparable to Compose's
|
||||||
|
`transcript-bench.sh` report.** `dumpsys gfxinfo` cannot see a
|
||||||
|
`SurfaceView`'s own GPU-drawn frames at all (confirmed: 0 frames
|
||||||
|
reported across a gesture loop that visibly scrolled), and a
|
||||||
|
`dumpsys SurfaceFlinger --latency` fallback returned no per-frame
|
||||||
|
history either on this Android version's BLAST compositor. What's
|
||||||
|
needed is frame-timing instrumentation inside iris itself — a report
|
||||||
|
the way `UiRenderState::take_counters`/`AccessTree::take_rebuilds`
|
||||||
|
already expose counters, extended to timing. This is the one number
|
||||||
|
RUST.md's recommendation (item 3) is still waiting on.
|
||||||
|
- [ ] **Long-press-then-drag-to-select was not independently driven
|
||||||
|
on-device.** `ui-trace`'s two gesture primitives (`tap`, `swipe X1 Y1
|
||||||
|
X2 Y2 MS`) cannot produce "hold stationary for `LONG_PRESS`, then drag
|
||||||
|
without lifting" — `swipe` interpolates motion across its whole
|
||||||
|
duration from the start. Needs either a new `ui-trace` action (a
|
||||||
|
genuine hold-then-drag primitive) or raw multi-step `MotionEvent`
|
||||||
|
injection. `DragArbiter`'s own unit tests already cover this exact
|
||||||
|
sequence against a synthetic clock (`iris/src/sense.rs`), which is why
|
||||||
|
this is "not independently driven on-device" rather than "unverified."
|
||||||
- [x] **Touch-drag panning over a row's own rendered text — done,
|
- [x] **Touch-drag panning over a row's own rendered text — done,
|
||||||
2026-09-05.** `row.rs` used to register `CursorSense::click_or_drag()`
|
2026-09-05.** `row.rs` used to register `CursorSense::click_or_drag()`
|
||||||
on each row's `TextEdit` for cross-row selection; `TextEdit::draw`'s
|
on each row's `TextEdit` for cross-row selection; `TextEdit::draw`'s
|
||||||
|
|||||||
+301
-71
@@ -36,10 +36,26 @@ session spending an afternoon on them again.
|
|||||||
|
|
||||||
## Where things stand (2026-09-05)
|
## Where things stand (2026-09-05)
|
||||||
|
|
||||||
- **Both of the previous note's in-flight pieces are now done, 2026-09-05.**
|
- **I5's Android integration is done and measured, 2026-09-05 (still ticked
|
||||||
Design choices for both are summarised in `DECISIONS.md` at the repo
|
`[~]`, not `[x]` -- see I5's own box for exactly why).** The transcript
|
||||||
root, which is the file Iris reads for choices made without her. Next:
|
screen runs on-device against a real `ai-server`, with real scrolling,
|
||||||
I5's Android integration and the bench numbers.
|
real touch-drag panning and tap-by-name accessibility all confirmed by
|
||||||
|
screenshot/log evidence on this checkout's emulator. What the
|
||||||
|
recommendation still lacks is a render-time number for iris comparable
|
||||||
|
to Compose's `transcript-bench.sh` report -- not because the screen
|
||||||
|
doesn't work, but because `dumpsys gfxinfo` cannot see a `SurfaceView`'s
|
||||||
|
own GPU-drawn frames at all, and iris has no frame-timing
|
||||||
|
instrumentation of its own to ask instead. Two real, previously-unknown
|
||||||
|
bugs were found and fixed getting here (a missing `INTERNET` permission,
|
||||||
|
and a background-thread redraw request that crashed the process via a
|
||||||
|
`Looper` requirement neither this box nor `Tasks::redraw_handle`'s
|
||||||
|
design had anticipated) -- both in I5's own box, both in `IRIS.md`.
|
||||||
|
Next: give iris a render-time report of its own, the one piece left
|
||||||
|
before the iris-vs-Masonry recommendation can be decided by a number
|
||||||
|
rather than by structure alone.
|
||||||
|
- **Design choices for the two pieces before this are summarised in
|
||||||
|
`DECISIONS.md`** at the repo root, which is the file Iris reads for
|
||||||
|
choices made without her.
|
||||||
- **E4 done, 2026-09-05.** `iris/desktop-app`: a winit window with a
|
- **E4 done, 2026-09-05.** `iris/desktop-app`: a winit window with a
|
||||||
session list beside `transcript-ui`'s screen (`build_tree`), against a
|
session list beside `transcript-ui`'s screen (`build_tree`), against a
|
||||||
real `ai-server` through `client-core`, enrolled from the same
|
real `ai-server` through `client-core`, enrolled from the same
|
||||||
@@ -648,24 +664,47 @@ light" has a knob inside the same stack.
|
|||||||
framework and Masonry was the calibration. If it does not, the
|
framework and Masonry was the calibration. If it does not, the
|
||||||
measurement says which parts of Masonry to adopt underneath it.
|
measurement says which parts of Masonry to adopt underneath it.
|
||||||
|
|
||||||
**Not decidable yet, 2026-09-05 — what's missing, named rather than
|
**Still not decidable by a render-time number, 2026-09-05 (updated) —
|
||||||
guessed at.** Neither side of this comparison has a render number:
|
what's missing, named rather than guessed at, and now for a different
|
||||||
E2 found Masonry's own scroll gesture path absent on Android
|
reason than before.** E2 found Masonry's own scroll gesture path
|
||||||
entirely (its box, "measurable frames"), and I5 built the iris side of
|
absent on Android entirely (its box, "measurable frames") — that has
|
||||||
the screen (`iris/transcript-ui/`) but not the Android integration
|
not changed. I5's Android integration is now built and confirmed
|
||||||
around it — no cdylib/Gradle shell exists for this screen yet (unlike
|
working (real server, real scrolling, real touch-drag pan, tap-by-name
|
||||||
`tabs-ui`'s `iris-android-app`, I2), so there is nothing installed on a
|
— I5's own box, "Measurements taken"), so the earlier blocker ("no
|
||||||
device for `transcript-bench.sh` to measure against the Compose
|
cdylib/Gradle shell exists for this screen") is gone. What replaced
|
||||||
baseline. What would close this: build that integration (real
|
it: **`dumpsys gfxinfo`, the tool `transcript-bench.sh` and this
|
||||||
`client-core` networking against `app/ui-sandbox.sh --delay`, a cdylib
|
recommendation both assumed would give the comparison, cannot see a
|
||||||
+ Gradle module the way I2 did for `tabs-ui`), then run
|
`SurfaceView`'s own GPU-drawn frames at all** — it instruments
|
||||||
`transcript-bench.sh`'s gesture on both. Until then, the decision rests
|
Android's ordinary View/Skia drawing pipeline, which a `wgpu`-rendered
|
||||||
on the structural findings both sides *did* produce: Masonry cannot do
|
`SurfaceView` (iris's whole approach) bypasses entirely. Confirmed
|
||||||
cross-row selection or per-span inline rich text at all today (E2's
|
0 frames reported across a 24-swipe gesture loop that visibly scrolled
|
||||||
`grep -rln`, zero hits, cited in its own box), and iris now does both
|
the screen (screenshots differ), and a `dumpsys SurfaceFlinger
|
||||||
(I5's `SpanStyle` and `selection.rs`) as well as programmatic
|
--latency` fallback returned no per-frame history either (just the
|
||||||
touch-scroll (I3) — three structural points in iris's favour with no
|
display's refresh period) on this Android version's BLAST compositor.
|
||||||
opposing measurement yet on either side.
|
The Compose side of the same loop *did* produce a real number under
|
||||||
|
identical conditions (`EMU_GPU=software`, same emulator, same session):
|
||||||
|
**8.96% janky frames, 99th percentile 150ms.** So this is now a
|
||||||
|
one-sided number, not a missing one — the number needed to close item 3
|
||||||
|
is a render-time report from **iris itself** (the equivalent of the
|
||||||
|
Compose app's in-app copy-button report `transcript-bench.sh` already
|
||||||
|
reads), which does not exist yet and is real, scoped follow-on work
|
||||||
|
(frame timing inside `iris_core::render`, exposed the way `AccessTree`
|
||||||
|
or `UiRenderState::take_counters` already are) rather than a rerun of
|
||||||
|
anything above. Until it exists, the decision still rests on the
|
||||||
|
structural findings both sides *did* produce, now joined by a
|
||||||
|
functional one: Masonry cannot do cross-row selection or per-span
|
||||||
|
inline rich text at all today (E2's `grep -rln`, zero hits, cited in
|
||||||
|
its own box); iris does both (I5's `SpanStyle` and `selection.rs`) and
|
||||||
|
its touch-scroll now works end-to-end on a real device, not just
|
||||||
|
programmatically (I3's benchmark plus I5's on-device screenshot
|
||||||
|
evidence) — three structural points and one functional one in iris's
|
||||||
|
favour, still with no opposing *or* supporting render-time measurement
|
||||||
|
on either side. **What Iris needs to weigh this**: whether "iris works
|
||||||
|
and Masonry's Android scroll path is absent entirely" is enough to
|
||||||
|
decide without a number, or whether the frame-timing work above is
|
||||||
|
worth doing first — a product/tradeoff call, not a technical one, so it
|
||||||
|
is left to her rather than decided here (DECISIONS.md's DEFERRED
|
||||||
|
item).
|
||||||
4. Then the shell (E3), the desktop window (E4) and the packaging (E5),
|
4. Then the shell (E3), the desktop window (E4) and the packaging (E5),
|
||||||
which do not depend on the choice.
|
which do not depend on the choice.
|
||||||
|
|
||||||
@@ -2113,8 +2152,9 @@ silently on real hardware.
|
|||||||
box's scope could finish alone -- recorded here rather than left
|
box's scope could finish alone -- recorded here rather than left
|
||||||
silently undone.
|
silently undone.
|
||||||
- [x] **I4 — accessibility names via AccessKit, host half done and verified
|
- [x] **I4 — accessibility names via AccessKit, host half done and verified
|
||||||
2026-09-05; the emulator half is the one step left, named at the
|
2026-09-05; the emulator half done and verified 2026-09-05, same day
|
||||||
bottom of this box.** Built `iris_core::ui::access::AccessTree`
|
as I5's Android integration (see bottom of this box for the exact
|
||||||
|
run).** Built `iris_core::ui::access::AccessTree`
|
||||||
(`iris/core/src/ui/access.rs`) -- one flat AccessKit tree, a synthetic
|
(`iris/core/src/ui/access.rs`) -- one flat AccessKit tree, a synthetic
|
||||||
`Role::Window` root with every **named** widget as a direct child.
|
`Role::Window` root with every **named** widget as a direct child.
|
||||||
Deliberately flat rather than mirroring iris's real widget nesting:
|
Deliberately flat rather than mirroring iris's real widget nesting:
|
||||||
@@ -2226,34 +2266,60 @@ silently on real hardware.
|
|||||||
than confirmed end-to-end the way the emulator step below confirms
|
than confirmed end-to-end the way the emulator step below confirms
|
||||||
the Android path.
|
the Android path.
|
||||||
|
|
||||||
**What remains -- the one check that needs the emulator, held by
|
**Done, 2026-09-05, on this checkout's own emulator (`ai-app-2`,
|
||||||
another session during this pass.** `iris-android-app`'s tabs screen
|
`EMU_GPU=software` -- see I5's box for why plain `-gpu host` and the
|
||||||
has never been driven by `ui-trace` for real; everything above is
|
documented Vulkan-feature recipe both could not be used here).**
|
||||||
"builds, runs, produces the right data" on the host. Once the
|
`cargo ndk -t x86_64 -P 26 -o app/src/main/jniLibs/ build --release &&
|
||||||
emulator is free:
|
gradle :app:assembleDebug`, installed, launched, then each of
|
||||||
|
|
||||||
cd iris/android-app && cargo ndk -t x86_64 -P 26 -o app/src/main/jniLibs/ build --release && gradle :app:assembleDebug
|
|
||||||
adb install -r app/build/outputs/apk/debug/app-debug.apk
|
|
||||||
# launch iris-android-app on the emulator, then:
|
|
||||||
ui-trace record --do "tap 'pad'"
|
ui-trace record --do "tap 'pad'"
|
||||||
ui-trace record --do "tap 'span'"
|
ui-trace record --do "tap 'span'"
|
||||||
ui-trace record --do "tap 'image span'"
|
ui-trace record --do "tap 'image span'"
|
||||||
ui-trace record --do "tap 'text layout'"
|
ui-trace record --do "tap 'text layout'"
|
||||||
ui-trace record --do "tap 'text edit scroll'"
|
ui-trace record --do "tap 'text edit scroll'"
|
||||||
|
|
||||||
Pass condition: each tap resolves (uiautomator finds a node with
|
resolved (uiautomator found the exact label every time -- `ui-trace`
|
||||||
that exact label) and switches `main`'s visible pane the way a
|
never failed a run). **Confirming the pane actually switched needed
|
||||||
direct touch on that button already does -- i.e. `bench-lib.sh`'s
|
more than `ui-trace show`**: the tabs row is the *only* named
|
||||||
tap-by-name mechanism, unmodified, driving the iris screen instead
|
structure on this screen, its five buttons never move, so
|
||||||
of the Compose one. Also worth checking while the emulator is up,
|
`--field box` reports "nothing moved" on every run whether the pane
|
||||||
since E1 found it exactly this way: run a second `ui-trace record`
|
behind it changed or not -- a screenshot before/after is what showed
|
||||||
immediately after the first (attach, detach, attach again) and
|
it, `adb exec-out screencap -p`, hashed to confirm difference; also
|
||||||
confirm the process is still alive afterward -- the detach-abort
|
confirmed live with a temporary `log::debug!` in `switch_button`'s
|
||||||
this box's mitigation exists for.
|
click closure (reverted before committing) showing the exact index
|
||||||
|
clicked matching the tapped label. The detach-abort check also
|
||||||
|
passed: six consecutive `ui-trace record` calls against the same
|
||||||
|
process (attach, detach, attach again, five more times) left it
|
||||||
|
alive throughout -- `adb shell dumpsys window` still showed
|
||||||
|
`dev.iris.android.demo/.MainActivity` focused and rendering
|
||||||
|
afterward, no crash in `logcat`.
|
||||||
|
|
||||||
|
**One real, unrelated bug found and fixed getting here, not part of
|
||||||
|
I4's own design**: the release build was required -- a debug/dev
|
||||||
|
profile build of this same APK reliably `SIGSEGV`s inside this
|
||||||
|
emulator's Vulkan loader (`vulkan.ranchu.so`,
|
||||||
|
`vk_common_SetDebugUtilsObjectNameEXT`) the moment `wgpu` creates its
|
||||||
|
first bind group layout, because `wgpu`'s `InstanceFlags::
|
||||||
|
from_build_config()` turns on debug object-labelling in a dev build,
|
||||||
|
and labelling a `SwiftShader`-backed resource through this
|
||||||
|
emulator's loader trampoline crashes. A release build's
|
||||||
|
`InstanceFlags::empty()` never takes that path. Nothing in iris
|
||||||
|
caused this and nothing here needed to change to avoid it --
|
||||||
|
recorded because it looked exactly like a fresh regression the first
|
||||||
|
time it was hit (mid-session, after adding an unrelated temporary
|
||||||
|
log line forced a dev rebuild) and cost real time to separate from
|
||||||
|
the actual touch-dispatch question being chased at the time.
|
||||||
- [~] **I5 — the transcript screen in iris (2026-09-05). The widget-tree
|
- [~] **I5 — the transcript screen in iris (2026-09-05). The widget-tree
|
||||||
half is built, tested and screenshotted; the emulator half (real
|
half and the Android integration are both built and confirmed
|
||||||
device numbers against the Compose baseline) is not -- ticked
|
working on-device (real server, real scrolling, real touch-drag
|
||||||
partial rather than done, see "What remains" at the end of this box.**
|
pan, tap-by-name); still `[~]` rather than `[x]` because the one
|
||||||
|
thing the recommendation actually wants -- a render-time number for
|
||||||
|
iris comparable to Compose's -- could not be produced on this pass,
|
||||||
|
for a reason named precisely rather than left vague: `dumpsys
|
||||||
|
gfxinfo` cannot see a `SurfaceView`'s own GPU-drawn frames at all,
|
||||||
|
and iris has no frame-timing instrumentation of its own yet to ask
|
||||||
|
instead. See "Measurements taken" and "What remains" near the end of
|
||||||
|
this box.**
|
||||||
|
|
||||||
**Where it lives.** `iris/transcript-ui/` (new workspace member,
|
**Where it lives.** `iris/transcript-ui/` (new workspace member,
|
||||||
`[lib]`), the same shape as `iris/tabs-ui`: generic over `Rsc:
|
`[lib]`), the same shape as `iris/tabs-ui`: generic over `Rsc:
|
||||||
@@ -2464,24 +2530,186 @@ silently on real hardware.
|
|||||||
screenshots of real content leaving this repo -- these are
|
screenshots of real content leaving this repo -- these are
|
||||||
synthetic rows, but the rule is kept uniform regardless.
|
synthetic rows, but the rule is kept uniform regardless.
|
||||||
|
|
||||||
|
**The Android integration, done 2026-09-05.** Extended
|
||||||
|
`iris-android-app` (I2's shell) with a second, mutually-exclusive
|
||||||
|
`AndroidAppState` behind a new Cargo feature rather than building a
|
||||||
|
third shell -- see `iris/android-app/src/lib.rs`'s module doc for why
|
||||||
|
that was chosen over a standalone crate: the Gradle project, the
|
||||||
|
`IrisView`/`MainActivity` Java, and the `register_view_class` wiring
|
||||||
|
I2 already built are exactly what a second screen needs too, and the
|
||||||
|
only thing that differs is which `AndroidAppState` the JNI entry
|
||||||
|
point instantiates. `transcript_client.rs` (new) fetches the sandbox
|
||||||
|
server's session list, opens the first one, and follows it live --
|
||||||
|
`client_core::api`/`event_stream`/`transcript_fold` almost verbatim
|
||||||
|
from `desktop-app`'s `app.rs` (E4), down to the generation-guard
|
||||||
|
pattern; `fold_page`/`raw_seq` were hoisted into `client-core` itself
|
||||||
|
first so both callers share one copy rather than a second one being
|
||||||
|
pasted in (a separate small commit, "write the logic once").
|
||||||
|
Deliberately simplified, recorded rather than left to be
|
||||||
|
rediscovered: no session list UI and no enrollment flow exist for
|
||||||
|
this screen -- `build.rs` bakes the sandbox's host/port/token and the
|
||||||
|
pinned CA in at build time from
|
||||||
|
`AI_APP_TRANSCRIPT_HOST`/`_PORT`/`_TOKEN`/`AI_APP_CA` env vars, same
|
||||||
|
trust-boundary reasoning as the Compose app's `GeneratePinnedCert`
|
||||||
|
Gradle task (`app/androidApp/build.gradle.kts`), extended here to
|
||||||
|
also bake the enrollment since building a real one is E3/E4's scope,
|
||||||
|
not this box's. A real app needs `desktop-app`'s
|
||||||
|
`EnrolledServer`/QR-link flow or E3's Keystore-sealed
|
||||||
|
`ServerConfig.kt`.
|
||||||
|
|
||||||
|
**Two real bugs found and fixed getting an actual screen on
|
||||||
|
screen, neither anticipated by this box's earlier design:**
|
||||||
|
|
||||||
|
1. **Missing `INTERNET` permission.** `iris-android-app`'s manifest
|
||||||
|
never needed one before this screen (the tabs demo makes no
|
||||||
|
network call), so nobody had noticed it was absent. Its absence
|
||||||
|
reads nothing like a network problem: `UreqTransport::new`'s
|
||||||
|
connect failed with `EPERM` ("Operation not permitted"), not the
|
||||||
|
`ECONNREFUSED`/`ENETUNREACH` a dead server or a firewall would
|
||||||
|
give -- a seccomp-level socket denial. Added
|
||||||
|
`<uses-permission android:name="android.permission.INTERNET" />`
|
||||||
|
with a comment naming the exact symptom, so the next person
|
||||||
|
hitting `EPERM` from this codebase's own `ureq` stack finds the
|
||||||
|
answer instead of debugging the server.
|
||||||
|
2. **A background task's first redraw request past the initial one
|
||||||
|
aborted the process.** `Tasks::redraw_handle()` (new, this box,
|
||||||
|
see `IRIS.md`'s 2026-09-05 entry for the full account) exists so
|
||||||
|
`transcript_client.rs` can ask for a frame after each
|
||||||
|
`TaskCtx::update`, the way `desktop-app` uses winit's `Proxy` for
|
||||||
|
the same reason. Calling it crashed with `SIGABRT`,
|
||||||
|
`Result::unwrap() on an Err value: JavaException`, inside
|
||||||
|
`android-view`'s `View::post_frame_callback` -- its Java side
|
||||||
|
calls `Choreographer.getInstance()`, which throws unless the
|
||||||
|
*calling* thread already has a `Looper`, and a tokio worker
|
||||||
|
thread has none even once JNI-attached. Fixed by routing through
|
||||||
|
`View::post_delayed(0)` instead (thread-safe, no `Looper`
|
||||||
|
required) and a new `IrisViewPeer::delayed_callback` override
|
||||||
|
(`android/view.rs`) that drains tasks and renders on the UI
|
||||||
|
thread the callback lands on -- same body as `do_frame`. Every
|
||||||
|
future caller of `redraw_handle()` from a background thread gets
|
||||||
|
this for free.
|
||||||
|
|
||||||
|
**The emulator itself needed a boot recipe none of the three
|
||||||
|
previously-documented ones give cleanly, found the hard way.**
|
||||||
|
Plain `emu up` (`-gpu host`, no Vulkan feature) crashed instantly --
|
||||||
|
`wgpu_core::instance: Request adapter didn't find compatible
|
||||||
|
adapters` -- this AVD's default boot has no Vulkan device at all,
|
||||||
|
matching "Vulkan in the emulator" below. The documented fix for
|
||||||
|
*that* (`VK_DRIVER_FILES=... GPU_HOST_FEATURES="-feature Vulkan" emu
|
||||||
|
up`) does get a Vulkan device, but on this host it is a **second**
|
||||||
|
one alongside the real GPU's own Venus/gfxstream Vulkan adapter, and
|
||||||
|
`AndroidRenderer::new`'s `request_adapter` (no adapter-name
|
||||||
|
filtering, `PowerPreference::default()`) picked Venus -- which
|
||||||
|
crashed inside `wgpu_core::device::resource::Device::
|
||||||
|
create_bind_group_layout`, the same structural Venus incompatibility
|
||||||
|
"Vulkan in the emulator" already documents for a different call.
|
||||||
|
`EMU_GPU=software` (`-gpu swiftshader_indirect`, no host GPU
|
||||||
|
involved at all) is what actually works cleanly, because there is
|
||||||
|
then only the one Vulkan device (`SwiftShader Device (Subzero)`) for
|
||||||
|
`request_adapter` to find -- confirmed via
|
||||||
|
`wgpu_core::instance: Found 1 compatible adapters`. **One trap in
|
||||||
|
switching between these**: the AVD's saved snapshot carries over
|
||||||
|
whichever GPU config booted it last, so restarting under
|
||||||
|
`EMU_GPU=software` right after a `-feature Vulkan` boot still linked
|
||||||
|
against `vulkan.ranchu.so` and crashed (`SIGSEGV` inside
|
||||||
|
`vk_common_SetDebugUtilsObjectNameEXT`) until the AVD's
|
||||||
|
`snapshots/` directory was cleared by hand -- matches "Vulkan in the
|
||||||
|
emulator"'s own note that a GPU-config switch needs a cold boot the
|
||||||
|
`emu` wrapper does not force. Recorded here rather than only in that
|
||||||
|
section since it is what made three different crashes look like
|
||||||
|
three different bugs before the pattern was the AVD's snapshot, not
|
||||||
|
the code.
|
||||||
|
|
||||||
|
**Measurements taken, 2026-09-05, `ai-app-2`'s own emulator,
|
||||||
|
`EMU_GPU=software`, against `app/ui-sandbox.sh` (port 8519, session
|
||||||
|
`8920378e7167ebcd`, 40 real sent/echoed messages):**
|
||||||
|
|
||||||
|
(a) **Tap-by-name on a named control -- passes.**
|
||||||
|
`ui-trace record --do "tap 'Message'"` (the composer's `.label`,
|
||||||
|
I4) resolved and the field's bounds moved (`top 2329 -> 1509`,
|
||||||
|
the keyboard opening), the same shape I4's own tabs-screen taps
|
||||||
|
confirmed the same day. `iris-android-app`'s tabs screen also got the
|
||||||
|
full I4 pass-condition run this session -- see I4's own box above,
|
||||||
|
now ticked `[x]`.
|
||||||
|
|
||||||
|
(b) **The `transcript-bench.sh`-shaped scroll comparison -- a
|
||||||
|
real number for Compose, no comparable number for iris, and that
|
||||||
|
gap is itself the finding.** `transcript-bench.sh` could not be
|
||||||
|
pointed at `iris-android-app` directly -- it reads the Compose app's
|
||||||
|
in-app render-report log line, which this screen has no equivalent
|
||||||
|
of -- so the same 24-swipe gesture loop (`swipe 540 700 540 1600
|
||||||
|
200` / `swipe 540 1600 540 700 200`, alternating, matching that
|
||||||
|
script's own cycle) was driven by hand via `ui-trace record` against
|
||||||
|
both apps, each freshly opened on the same session, `dumpsys gfxinfo
|
||||||
|
<package> reset` beforehand and `dumpsys gfxinfo <package>` after.
|
||||||
|
Compose: **8.96% janky frames, 99th percentile 150ms, 212 frames
|
||||||
|
rendered** over the loop -- worse than AGENTS.md's own recorded
|
||||||
|
stock-emulator baseline (5.2-5.9%, 29-32ms), consistent with
|
||||||
|
`EMU_GPU=software`'s CPU rendering being slower than the `-gpu host`
|
||||||
|
that baseline was taken under, which is exactly why AGENTS.md's rule
|
||||||
|
against reading an absolute emulator number as the phone's applies
|
||||||
|
doubly here. **iris: `dumpsys gfxinfo` reported 0 frames rendered
|
||||||
|
for the entire gesture window, on both a run where the screen
|
||||||
|
visibly did not move and one where it visibly did** (confirmed
|
||||||
|
by `adb exec-out screencap -p`, hashed before/after -- identical
|
||||||
|
when the swipe direction was already at that end of the transcript,
|
||||||
|
different once swiped the other way). **`gfxinfo` instruments
|
||||||
|
Android's own Skia/HWUI View-drawing pipeline; it has no visibility
|
||||||
|
into a `SurfaceView` whose contents are drawn by a separately-owned
|
||||||
|
GPU context (`wgpu`/Vulkan, here) the way Compose's ordinary `View`
|
||||||
|
tree is drawn.** A `dumpsys SurfaceFlinger --latency` probe against
|
||||||
|
the transcript screen's own `SurfaceView` layer was tried as a
|
||||||
|
fallback and returned only the display's refresh period (16666666ns)
|
||||||
|
with no frame history at all -- this Android version's BLAST
|
||||||
|
compositor does not keep the per-frame timestamps that legacy API
|
||||||
|
used to report. **So there is no dumpsys-derived frame-time number
|
||||||
|
for iris on this build**, not a bad one -- the honest comparison this
|
||||||
|
pass can make is functional (both apps' lists scroll under the same
|
||||||
|
touch gesture) rather than numeric, and getting a real number for
|
||||||
|
iris needs the app's own frame-timing instrumentation (the render
|
||||||
|
report the Compose side already has, iris has none of yet) rather
|
||||||
|
than a different `dumpsys` incantation.
|
||||||
|
|
||||||
|
(c) **Touch-drag pans the list on real device touch input --
|
||||||
|
confirmed by screenshot, not by `ui-trace show`.** `ui-trace show`
|
||||||
|
cannot answer this at all here: the only named node on this screen
|
||||||
|
is the composer, which does not move when the list scrolls, so every
|
||||||
|
`--field box` query reports "nothing moved" regardless of whether
|
||||||
|
the list actually did (the same "no named structure to track"
|
||||||
|
situation I4's tabs-screen note about clipped bounds warns about,
|
||||||
|
one level further -- here there is no candidate node at all, not a
|
||||||
|
clipped one). `adb exec-out screencap -p` before and after a single
|
||||||
|
`swipe 540 700 540 1600 300` (list not already at that end) hashed
|
||||||
|
different and visibly showed different message rows on screen;
|
||||||
|
the same swipe repeated when already at that end of the transcript
|
||||||
|
correctly hashed identical -- so the mechanism responds to real
|
||||||
|
touch, in both directions, not just once by luck. **Long-press then
|
||||||
|
drag to select was not independently driven this pass**: doing it
|
||||||
|
for real needs a touch held stationary for `LONG_PRESS` (500ms)
|
||||||
|
and *then* moved without lifting, and neither of `ui-trace`'s two
|
||||||
|
gesture primitives can produce that -- `tap` has no hold, and
|
||||||
|
`swipe X1 Y1 X2 Y2 MS` interpolates motion across its whole duration
|
||||||
|
from t=0, so a long `swipe` with a short first segment is still
|
||||||
|
continuous motion throughout, not a hold followed by a drag. This
|
||||||
|
needs either a new `ui-trace` action (a `hold MS then drag X Y`
|
||||||
|
primitive) or a raw multi-step `sendevent`/`MotionEvent` injection
|
||||||
|
neither this pass's tooling nor its remaining time could build
|
||||||
|
safely. `DragArbiter`'s own unit tests (`iris/src/sense.rs`, I5's
|
||||||
|
earlier "Gap closed" section) already cover this exact sequence with
|
||||||
|
a synthetic clock, which is why the mechanism is trusted enough to
|
||||||
|
call "not independently driven on-device" rather than "unverified."
|
||||||
|
|
||||||
**What remains, named rather than silently dropped (also in
|
**What remains, named rather than silently dropped (also in
|
||||||
IRIS_TODO.md, dated 2026-09-05):**
|
IRIS_TODO.md, dated 2026-09-05):**
|
||||||
|
|
||||||
- **The emulator half of the pass condition was not attempted.**
|
- **A real per-app frame-time number for iris** -- (b) above. Needs
|
||||||
`emu list` shows `emulator-5554` (AVD `ai-app`, a different
|
`iris` to grow its own render-report/frame-timing instrumentation
|
||||||
checkout) held by another session during this pass, but even with
|
(what the Compose app's copy-button report already gives) before
|
||||||
a free emulator this needs real Android integration that does not
|
any `dumpsys`-based comparison can go further than functional.
|
||||||
exist yet for this screen: a cdylib + Gradle shell the way
|
- **Long-press-then-drag-to-select, on-device** -- (c) above. Needs
|
||||||
`iris-android-app` wraps `tabs-ui` (I2), real
|
a driving primitive this pass's tools do not have.
|
||||||
`client-core::ApiClient`/`event_stream::follow_session_events`
|
- **Row-level accessibility names** -- behaviour 6, I5's own writeup
|
||||||
wiring against `app/ui-sandbox.sh` with `--delay` (this crate
|
above.
|
||||||
deliberately does not fetch anything itself, see `lib.rs`'s doc),
|
|
||||||
and then `transcript-bench.sh`'s gesture compared against the
|
|
||||||
Compose baseline. That is real, multi-part follow-on work in its
|
|
||||||
own right -- closer in size to E2/E3 than to "run one more
|
|
||||||
script" -- not something this pass's remaining time could
|
|
||||||
responsibly rush and still report honestly.
|
|
||||||
- **Row-level accessibility names** -- behaviour 6 above.
|
|
||||||
- **A tappable link and a code-span background chip** -- behaviour 2.
|
- **A tappable link and a code-span background chip** -- behaviour 2.
|
||||||
- **`Selection`'s anchor-row shortcut** -- behaviour 1.
|
- **`Selection`'s anchor-row shortcut** -- behaviour 1.
|
||||||
- **No syntax highlighting inside a fenced code block** -- `markdown.rs`
|
- **No syntax highlighting inside a fenced code block** -- `markdown.rs`
|
||||||
@@ -2493,18 +2721,20 @@ silently on real hardware.
|
|||||||
|
|
||||||
**Net for the recommendation.** Item 3 ("decide when the transcript
|
**Net for the recommendation.** Item 3 ("decide when the transcript
|
||||||
screen exists in both, from the measurements") still cannot be
|
screen exists in both, from the measurements") still cannot be
|
||||||
decided by a number -- E2 could not produce one for Masonry, and I5
|
decided by a render-time number on either side -- E2 could not
|
||||||
has not yet produced one for iris either, for an unrelated reason
|
produce one for Masonry (no working touch-scroll to measure), and
|
||||||
(no Android harness built yet, not an absent capability). What *can*
|
this pass could not produce one for iris either, now for a different
|
||||||
be said structurally, updating E2's own conclusion: iris now also
|
and more fixable reason: `iris` itself has no frame-timing
|
||||||
demonstrates the two things E2 found Masonry structurally unable to
|
instrumentation yet, not that the screen doesn't work. What *can* be
|
||||||
do at all -- cross-row selection and true per-span inline rich text
|
said, updating E2's own conclusion further: iris now has a real,
|
||||||
inside one wrapped, selectable buffer -- neither of which exists
|
working Android integration -- a real server, a real scrolling
|
||||||
anywhere in `masonry`/`masonry_core`/`xilem` today (E2's own
|
transcript, real touch-drag panning, tap-by-name accessibility, all
|
||||||
`grep -rln` finding). That is a second structural point in iris's
|
confirmed on-device -- plus the two structural points E2 already
|
||||||
favour, alongside I2's working touch-scroll-vs-Masonry's-absent one,
|
found Masonry unable to reach at all (cross-row selection, true
|
||||||
still short of the render-number comparison the recommendation
|
per-span inline rich text). Whether iris's *smoothness* matches or
|
||||||
ultimately wants.
|
beats Compose's is the one question actually left for whoever picks
|
||||||
|
up "add frame-timing instrumentation to iris" next; DECISIONS.md's
|
||||||
|
DEFERRED item is updated with exactly this gap.
|
||||||
|
|
||||||
## For the next agent
|
## For the next agent
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user