From 6e0bd06e4da76c1ede65b0df798206eafa591963 Mon Sep 17 00:00:00 2001 From: iris <2+iris@noreply.localhost> Date: Sat, 5 Sep 2026 06:03:42 -0400 Subject: [PATCH] RUST.md: E2 -- a transcript in Masonry on android-view, and the touch-scroll gap it found Built a real transcript screen (e2-transcript, beside E1's demo in ~/src/android-view) against a live app/ui-sandbox.sh session through client-core: real fold, real ApiClient, VirtualScroll over 854 events, block-level markdown via pulldown-cmark into Prose, and a tool row that holds its top edge on expand via overwrite_anchor. The headline result is negative and load-bearing: neither VirtualScroll nor Portal reacts to a touch drag, only to wheel-style PointerEvent::Scroll (virtual_scroll.rs:504-523, portal.rs:259-267), confirmed both by reading and empirically (a real swipe and a synthetic Android scroll event both moved nothing). That blocks transcript-bench.sh's own gesture, so the render-number half of E2's pass condition has no comparison to make yet. Selection across rows and per-span rich text are also confirmed impossible on the pinned xilem commit, each cited to its source. Full writeup, repro commands and screenshots list in E2's own box. Co-Authored-By: Claude Sonnet --- RUST.md | 267 +++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 254 insertions(+), 13 deletions(-) diff --git a/RUST.md b/RUST.md index 2075ca2..9802dc0 100644 --- a/RUST.md +++ b/RUST.md @@ -34,11 +34,35 @@ the emulator, not by Mesa" and "the present mode was not the cause" are worth as much as the successes, because they are what stops the next session spending an afternoon on them again. -## Where things stand (2026-09-04) +## Where things stand (2026-09-05) - **Done**: E0 (toolchain), E1 (Masonry on android-view, which found the - keyboard gap — now explained, see below), I0a, I0b (iris builds on a - pinned nightly and runs), I1 (parley + glyph atlas). + keyboard gap — now explained, see below), E2 (a transcript in Masonry, + which found that Masonry has no touch-scroll on Android at all — see + below), I0a, I0b (iris builds on a pinned nightly and runs), I1 (parley + + glyph atlas). +- **E2 done, 2026-09-05, and its headline finding changes what "decide + from the measurements" (recommendation item 3) can mean right now.** + Built a real transcript screen (`~/src/android-view/e2-transcript`, + local, not committed — see E2's own box), fetching 854 real events from + an `app/ui-sandbox.sh` session through `client-core`. Six of the seven + "hard to get back" behaviours are answered with evidence either way; + the seventh (measurable frames) is **blocked before it can even start**: + neither of Masonry's scrolling widgets (`VirtualScroll`, `Portal`) + reacts to a touch drag, only to a wheel-style `PointerEvent::Scroll` — + confirmed by reading (`virtual_scroll.rs:504-523`, `portal.rs:259-267`) + and empirically (a real swipe and a synthetic Android scroll event both + moved nothing on screen). So `transcript-bench.sh`'s own gesture cannot + be performed against a Masonry transcript on Android today, which means + the render-number half of E2's pass condition has no comparison to make + yet — not a bad number, no number obtainable at all. Selection + spanning rows and per-span rich text (bold/italic/inline + code/links inside one paragraph) are also confirmed not possible on + the pinned commit, each for a specific, cited reason. What did work: + block-level rich text (heading size, monospace fences), real + virtualisation of 854 rows, `overwrite_anchor`-based hold-top-edge on + expand (screenshotted), and tap-by-name accessibility. Full writeup, + every citation, and the exact repro commands are in E2's own box below. - **Done, 2026-09-04: the `Widget::draw`/layout redesign (LAYOUT.md).** `desired_width`/`desired_height`/`SizeCtx`/`Cache` are gone; every widget in `iris/src/widget/` implements one `fn draw(&mut self, &mut Painter) -> @@ -98,8 +122,7 @@ session spending an afternoon on them again. now renders on the emulator on both backends (screenshotted); the GLES-only `D2`/`D2Array` warning was confirmed a red herring — still present post-fix, harmless. See I2's own entry below for the full - writeup. **E2** (a transcript in Masonry) can go in parallel in another - session. + writeup. **E2** (a transcript in Masonry) is done — see its own box. - **`client-core` built (2026-09-04)**, item 1 of the recommendation: `event-model/` (the event types, now shared with `server/`) and `client-core/` (REST and SSE clients, transcript fold, cache, highlighter, @@ -708,14 +731,232 @@ accepted. the pid captured at launch) rather than to whatever AVD is running when it wakes, and only one should be armed at a time. -- [ ] **E2 — a transcript in Masonry.** One screen: open a sandbox session, - page 800 events into `VirtualScroll` bottom-anchored, draw markdown - from `pulldown-cmark` into Parley rich text with links and code - chips, select across two rows with the platform handles, expand a - tool row holding its top edge. Pass: the render numbers land within - the Compose baseline in `transcript-bench.sh` on the GPU emulator - (same gestures, same session), and every one of the seven behaviours - above is either shown or has a written reason it cannot be. +- [x] **E2 — a transcript in Masonry (2026-09-05).** Built and run on this + emulator. It found the thing it was measuring for: a framework-wide + gap that blocks the bench comparison itself, plus a full accounting + of the seven behaviours. Ticked on E1's own precedent -- "everything + it was meant to establish is established, including the one gap." + + *Where it lives.* `~/src/android-view/e2-transcript` (new workspace + member, `crate-type = ["cdylib"]`, `lib.name = "main"`), packaged by + a new Gradle module `~/src/android-view/e2-app` copied from + `masonry-app` (`E2View`/`E2Activity`, package + `org.linebender.android.e2transcript`). Neither is committed to + `ai-app-2` or pushed anywhere -- same as E1, this is a local + experiment against the `xilem` commit + `e14ba3a5f9461b403cb30d95826187fba7f6924b` and the `android-view` + commit `bec6c62a96cef8239b0fd7fedeef9b184d02e3a1`, reproducible from + the commands below rather than from a remote. + + *Build.* Depends on `client-core`/`event-model` from this checkout by + path (`../../../repos/ai-app-2/client-core`) -- real code, not a + reimplementation: `ApiClient`/`UreqTransport` for the HTTP fetch, + `fold_event`/`group_tool_runs` for the transcript fold, exactly what + the app itself would use. The sandbox CA and a session's URL/token + are baked in at build time via `env!()`/`include_bytes!()`, the same + pattern the real APK uses to pin its CA (AGENTS.md), since this is a + throwaway screen with no enrollment flow: + + cd app && ./ui-sandbox.sh start # prints the port and token + sid=$(./ui-sandbox.sh spawn e2test) + ./ui-sandbox.sh send "$sid" @/tmp/big.md # markdown content + ./ui-sandbox.sh send "$sid" "/tools 3" # a grouped tool run + cd ~/src/android-view + E2_SANDBOX_URL=https://10.0.2.2: \ + E2_SANDBOX_TOKEN= \ + E2_SANDBOX_SESSION= \ + E2_CA_PEM_PATH=$HOME/.config/ai-app/certs/ca.pem \ + cargo ndk -t x86_64 -P 26 -o e2-app/src/main/jniLibs/ \ + build -p e2-transcript --release + ANDROID_HOME=~/Android/Sdk ./gradlew :e2-app:assembleDebug + + **`libmain.so` is 13.5 MB release** (E1's masonry-demo was 11 MB; + the difference is `client-core`'s `ureq`/`rustls` stack, which E1's + demo does not link). Release native lib, debug Gradle variant -- + the combination E1 found necessary (a debug build's + `SetDebugUtilsObjectNameEXT` segfaults this emulator's Vulkan + driver). + + *Emulator.* This checkout's own AVD (`ai-app-2`, not `ai-app`, which + another session already had up), booted with Vulkan the way E1 + established: `GPU_HOST_FEATURES="-feature Vulkan" + VK_DRIVER_FILES=$HOME/Android/Sdk/emulator/lib64/vulkan/vk_swiftshader_icd.json + emu up`. `adb shell cmd gpu vkjson` confirmed a device before + anything was installed. Torn down with `emu down` at the end of this + session (see "Where things stand" below for the exact state left). + + *What it does.* `fetch_rows()` (`e2-transcript/src/lib.rs`) makes one + blocking `fetch_transcript_page(session, None, 800, false)` call + before the widget tree exists, folds every line through + `client-core`, and groups tool runs -- 854 real events from a mixed + sandbox session (markdown paragraphs/headings/fences plus a + three-call tool run from the echo driver's `/tools 3`). Each + `TranscriptRow` becomes one `VirtualScroll` child, + built lazily from `VirtualScrollAction` the way + `masonry_winit/examples/virtual_fizzbuzz.rs` does it. **This is a + deliberate scope cut from "page 800 events" as live paging**: all + 854 rows' content is fetched once, and what `VirtualScroll` pages is + *widget construction*, not a second round of network calls per + scroll -- wiring a background-thread fetch woken across the JNI + boundary (the way I2's `ssh.rs` attach-and-call works) is real work + this experiment did not need to answer its question. `markdown.rs` + is a `pulldown-cmark` event-stream walk into a small `Block` enum + (`Text`/`Heading`/`Code`), with its own module doc explaining the + one real ceiling it hit (below). + + *Verification.* `cargo fmt -p e2-transcript -- --check` clean. + `cargo ndk -t x86_64 -P 26 clippy -p e2-transcript --all-targets`: + **zero warnings in this crate** (the only clippy output at all is + from `android-view` itself, a vendored dependency this experiment + does not own). `cargo ndk -t x86_64 -P 26 test -p e2-transcript + --lib` (run against the emulator, since the crate is + `cfg`-unconditionally Android): 1 test, `markdown::parse`'s block + split, passing. No larger test surface exists to port -- this is a + throwaway screen, not a library, matching AGENTS.md's "match the + codebase's testing posture." + + *Screenshots* (all `/tmp`, not committed -- see the standing rule + against transcripts leaving this repo, which applies equally to a + screenshot of one): `e2-screenshot2.png` first real content; + `e2-expand.png` a tool row expanded with its top edge held; + `e2-markdown.png` a heading/bold/italic/inline-code/link/fenced-code + message (the "You said: ## A Heading" line is the sandbox's echo + driver prefixing the literal input text before the `##`, which + keeps `pulldown-cmark` from recognising it as a heading -- a fixture + artifact, not a finding about Masonry). + + **The seven behaviours, each shown or given a sourced reason:** + + 1. **One selectable body of text spanning rows -- not possible, and + it is a real ceiling, not an oversight.** `Prose` wraps exactly + one `TextArea`, which wraps exactly one + `parley::PlainEditor` (`masonry/src/widgets/prose.rs`: "Note that + copying is not yet implemented"). Selection lives entirely inside + that one editor: `TextArea::on_pointer_event` + (`masonry/src/widgets/text_area.rs:414-459` in the pinned `xilem` + commit) captures the pointer on `Down` + (`ctx.capture_pointer()`) and drives `self.editor`'s own + `extend_selection_to_point` on `Move` -- there is no code path, + in `masonry_core` or `masonry`, that extends a selection into a + second widget's editor. A drag that starts in one row's `Prose` + and continues into the next is still that first row's own + `PlainEditor` being asked for a point outside its bounds; it + cannot reach the second row's text. Confirmed by reading, not + guessed at: there is no `SelectionContainer`-shaped type + anywhere in `masonry`, `masonry_core` or `xilem` (checked with + `grep -rln "SelectionContainer\|cross.widget.*selection"`, zero + hits). + 2. **Rich inline text -- block-level yes, inline no, and both for + the same reason.** `TextArea::edit_styles()` returns one + `&mut StyleSet` for the whole editor + (`masonry_core/src/core/text.rs:29-32` defines `StyleSet` as + `parley::StyleSet`, applied editor-wide); the type's + own comments say why nothing finer exists yet: + `// TODO: RichTextInput 👀` and + `// TODO: Support for links - https://github.com/linebender/xilem/issues/360` + at `masonry/src/widgets/text_area.rs:43-44`. So bold, italic, + inline code and a link *inside one paragraph* cannot each carry + their own style without leaving `TextArea` for a hand-rolled + `parley::Layout` (which loses selection, the caret and copy, + since those live inside `PlainEditor` specifically). What **is** + real: each markdown block is its own `Prose`, so a heading is a + bigger font and a fenced code block is monospace, screenshotted + in `e2-markdown.png` -- block-level style works because it is + block-level *widgets*, not a rich-text API. Tables and per-token + syntax colour inside a fence hit the identical ceiling (both are + per-range styling) and were not attempted for the same reason. + `markdown.rs`'s degraded rendering (backticks kept literally, + `[text](url)` shown as `text (url)`) is the honest fallback, + documented at the point it is produced. + 3. **Bottom-anchored virtualised list, paged, hold-top-edge on + expand -- mostly shown, with one real gap in the anchor API.** + `VirtualScroll` holds all 854 folded rows; + `overwrite_anchor` before swapping a tool row's widget for its + expanded/collapsed version is exactly the primitive + `holdTopEdge` needs, and it worked: `e2-expand.png` shows the + row growing downward from the same top edge it had collapsed, + no jump. Virtualisation is real (`ui-trace elements` only ever + lists the rows currently on screen, never all 854). **What did + not come free: hugging the bottom of the screen.** + `VirtualScroll::new`'s doc says "the item at `initial_anchor` + will have its top aligned with the top of the scroll area" -- + so anchoring on the last row puts that row's top at the + viewport's *top*, with empty space below it, not at the + viewport's bottom the way a chat transcript wants (visible in + `e2-screenshot2.png`). The complete public `WidgetMut` surface of + `VirtualScroll` is `new`, `with_valid_range`, + `will_handle_action`, `add_child`, `remove_child`, `child_mut`, + `set_valid_range`, `overwrite_anchor` + (`masonry/src/widgets/virtual_scroll.rs:257-428`) -- no + scroll-offset setter and no reverse/bottom-up layout mode exist + to ask for the other behaviour. Backward paging beyond the + initial 800 was not exercised, per the scope cut above. + 4. **The soft keyboard -- inherited gap, not re-investigated.** E2's + screen has no `TextInput`, only read-only `Prose`/`Button`, so it + does not hit `masonry/src/lib.rs:531`'s `as_input_connection` + returning `None` directly -- but it would the moment a composer + is added, per E1's finding. Nothing new to add here. + 5. **Platform integration -- out of scope by design.** Foreground + service, notifications, share sheet, deep link, Keystore, + camera, back gesture, edge-to-edge, local-network permission are + E3's list in RUST.md's own experiment order, not E2's. + 6. **Accessibility names -- shown, and the bench-script dependency + actually exercised.** `ui-trace record --do "tap '> 3 tool + calls'"` found the button by its label and pressed it (that tap + is what produced `e2-expand.png`); `Prose` rows surface their + text as their accessible name too (`ui-trace elements` lists + "You said: One more short reply..." etc. as named nodes). Tap by + name, the rule this whole project's bench scripts depend on, + works against this screen. + 7. **Measurable frames -- blocked, and this is the finding E2 was + really testing for.** Two separate problems, one of them fatal + to the render-numbers half of this box's own pass condition. + First, Masonry has no render-report/per-widget-cost + instrumentation the way Compose's `DebugStats` gives this + project -- building one was out of scope here. Second, and this + is the one that matters: **neither of Masonry's two scrolling + widgets responds to a touch drag at all.** + `VirtualScroll::on_pointer_event` + (`masonry/src/widgets/virtual_scroll.rs:504-523`) and + `Portal::on_pointer_event` + (`masonry/src/widgets/portal.rs:259-267`) both match only + `PointerEvent::Scroll` (wheel/trackpad deltas) and do nothing + with `PointerEvent::Down`/`Move`/`Up` -- there is no drag-to-scroll + gesture logic anywhere in the widget set. `android-view`'s own + Java bridge keeps the two paths separate at the source: + `RustView.java`'s `onTouchEvent` forwards raw touch straight to + Rust, and only `onGenericMotionEvent` (mouse/trackpad, not + touch) reaches the `ACTION_SCROLL` branch that becomes + `PointerEvent::Scroll` + (`android-view/src/events.rs:530`). Confirmed empirically, not + just by reading: a real swipe (`ui-trace`'s `swipe 540 1600 540 + 400 300`, twice) moved nothing (`e2-scroll.png` is pixel-identical + to the screen before it), and a synthetic Android wheel event + (`adb shell input scroll 540 1200 --axis VSCROLL,-5`) also moved + nothing. **This means `transcript-bench.sh`'s own gesture -- + a finger swipe -- cannot scroll a Masonry transcript on Android + today, at all, on this framework commit.** So the "render + numbers land within the Compose baseline" half of this box's + pass condition cannot be attempted, let alone met: there is no + way to perform the scroll the comparison asks for. This is not + a performance shortfall to close by writing faster code: it is + an absent input path upstream. The fix is a drag-to-scroll + gesture in `on_pointer_event` (the same place `TextArea`'s own + caret-drag logic already lives, so the pattern -- capture on + `Down`, accumulate delta on `Move`, release on `Up` -- exists + in this codebase already, just not wired into either scrolling + widget), and it belongs upstream in `xilem` rather than in this + project. + + **Net for RUST.md's recommendation.** Item 3 ("decide when the + transcript screen exists in both, from the measurements") cannot be + decided by a render-number comparison yet, because the comparison's + own gesture does not work on Masonry on Android. What *can* be + compared today is structural: iris already has a working scroll + gesture and a working touch model (I2, 2026-09-05) that Masonry's + upstream commit does not yet have for this exact case. That is a + point in iris's favour that a frame-time number would not have + shown any more clearly. - [ ] **E3 — the shell.** Kotlin `MainActivity` + `NotificationService` + Keystore + share intent calling into Rust over JNI, with the SSE follow loop in Rust. Pass: a notification arrives with the app