RUST.md: E2 -- a transcript in Masonry on android-view, and the touch-scroll gap it found

Built a real transcript screen (e2-transcript, beside E1's demo in
~/src/android-view) against a live app/ui-sandbox.sh session through
client-core: real fold, real ApiClient, VirtualScroll<dyn Widget> over 854
events, block-level markdown via pulldown-cmark into Prose, and a tool row
that holds its top edge on expand via overwrite_anchor.

The headline result is negative and load-bearing: neither VirtualScroll nor
Portal reacts to a touch drag, only to wheel-style PointerEvent::Scroll
(virtual_scroll.rs:504-523, portal.rs:259-267), confirmed both by reading
and empirically (a real swipe and a synthetic Android scroll event both
moved nothing). That blocks transcript-bench.sh's own gesture, so the
render-number half of E2's pass condition has no comparison to make yet.
Selection across rows and per-span rich text are also confirmed impossible
on the pinned xilem commit, each cited to its source. Full writeup, repro
commands and screenshots list in E2's own box.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
This commit is contained in:
irisandClaude Sonnet committed 2026-09-05 06:03:42 -04:00
1 parent 19c36e37f2
commit 6e0bd06e4d
1 file changed
+254 -13
+254 -13
View File
@@ -34,11 +34,35 @@ the emulator, not by Mesa" and "the present mode was not the cause" are
worth as much as the successes, because they are what stops the next
session spending an afternoon on them again.
## Where things stand (2026-09-04)
## Where things stand (2026-09-05)
- **Done**: E0 (toolchain), E1 (Masonry on android-view, which found the
keyboard gap — now explained, see below), I0a, I0b (iris builds on a
pinned nightly and runs), I1 (parley + glyph atlas).
keyboard gap — now explained, see below), E2 (a transcript in Masonry,
which found that Masonry has no touch-scroll on Android at all — see
below), I0a, I0b (iris builds on a pinned nightly and runs), I1 (parley +
glyph atlas).
- **E2 done, 2026-09-05, and its headline finding changes what "decide
from the measurements" (recommendation item 3) can mean right now.**
Built a real transcript screen (`~/src/android-view/e2-transcript`,
local, not committed — see E2's own box), fetching 854 real events from
an `app/ui-sandbox.sh` session through `client-core`. Six of the seven
"hard to get back" behaviours are answered with evidence either way;
the seventh (measurable frames) is **blocked before it can even start**:
neither of Masonry's scrolling widgets (`VirtualScroll`, `Portal`)
reacts to a touch drag, only to a wheel-style `PointerEvent::Scroll`
confirmed by reading (`virtual_scroll.rs:504-523`, `portal.rs:259-267`)
and empirically (a real swipe and a synthetic Android scroll event both
moved nothing on screen). So `transcript-bench.sh`'s own gesture cannot
be performed against a Masonry transcript on Android today, which means
the render-number half of E2's pass condition has no comparison to make
yet — not a bad number, no number obtainable at all. Selection
spanning rows and per-span rich text (bold/italic/inline
code/links inside one paragraph) are also confirmed not possible on
the pinned commit, each for a specific, cited reason. What did work:
block-level rich text (heading size, monospace fences), real
virtualisation of 854 rows, `overwrite_anchor`-based hold-top-edge on
expand (screenshotted), and tap-by-name accessibility. Full writeup,
every citation, and the exact repro commands are in E2's own box below.
- **Done, 2026-09-04: the `Widget::draw`/layout redesign (LAYOUT.md).**
`desired_width`/`desired_height`/`SizeCtx`/`Cache` are gone; every widget
in `iris/src/widget/` implements one `fn draw(&mut self, &mut Painter) ->
@@ -98,8 +122,7 @@ session spending an afternoon on them again.
now renders on the emulator on both backends (screenshotted); the
GLES-only `D2`/`D2Array` warning was confirmed a red herring — still
present post-fix, harmless. See I2's own entry below for the full
writeup. **E2** (a transcript in Masonry) can go in parallel in another
session.
writeup. **E2** (a transcript in Masonry) is done — see its own box.
- **`client-core` built (2026-09-04)**, item 1 of the recommendation:
`event-model/` (the event types, now shared with `server/`) and
`client-core/` (REST and SSE clients, transcript fold, cache, highlighter,
@@ -708,14 +731,232 @@ accepted.
the pid captured at launch) rather than to whatever AVD is running
when it wakes, and only one should be armed at a time.
- [ ] **E2 — a transcript in Masonry.** One screen: open a sandbox session,
page 800 events into `VirtualScroll` bottom-anchored, draw markdown
from `pulldown-cmark` into Parley rich text with links and code
chips, select across two rows with the platform handles, expand a
tool row holding its top edge. Pass: the render numbers land within
the Compose baseline in `transcript-bench.sh` on the GPU emulator
(same gestures, same session), and every one of the seven behaviours
above is either shown or has a written reason it cannot be.
- [x] **E2 — a transcript in Masonry (2026-09-05).** Built and run on this
emulator. It found the thing it was measuring for: a framework-wide
gap that blocks the bench comparison itself, plus a full accounting
of the seven behaviours. Ticked on E1's own precedent -- "everything
it was meant to establish is established, including the one gap."
*Where it lives.* `~/src/android-view/e2-transcript` (new workspace
member, `crate-type = ["cdylib"]`, `lib.name = "main"`), packaged by
a new Gradle module `~/src/android-view/e2-app` copied from
`masonry-app` (`E2View`/`E2Activity`, package
`org.linebender.android.e2transcript`). Neither is committed to
`ai-app-2` or pushed anywhere -- same as E1, this is a local
experiment against the `xilem` commit
`e14ba3a5f9461b403cb30d95826187fba7f6924b` and the `android-view`
commit `bec6c62a96cef8239b0fd7fedeef9b184d02e3a1`, reproducible from
the commands below rather than from a remote.
*Build.* Depends on `client-core`/`event-model` from this checkout by
path (`../../../repos/ai-app-2/client-core`) -- real code, not a
reimplementation: `ApiClient`/`UreqTransport` for the HTTP fetch,
`fold_event`/`group_tool_runs` for the transcript fold, exactly what
the app itself would use. The sandbox CA and a session's URL/token
are baked in at build time via `env!()`/`include_bytes!()`, the same
pattern the real APK uses to pin its CA (AGENTS.md), since this is a
throwaway screen with no enrollment flow:
cd app && ./ui-sandbox.sh start # prints the port and token
sid=$(./ui-sandbox.sh spawn e2test)
./ui-sandbox.sh send "$sid" @/tmp/big.md # markdown content
./ui-sandbox.sh send "$sid" "/tools 3" # a grouped tool run
cd ~/src/android-view
E2_SANDBOX_URL=https://10.0.2.2:<port> \
E2_SANDBOX_TOKEN=<token> \
E2_SANDBOX_SESSION=<sid> \
E2_CA_PEM_PATH=$HOME/.config/ai-app/certs/ca.pem \
cargo ndk -t x86_64 -P 26 -o e2-app/src/main/jniLibs/ \
build -p e2-transcript --release
ANDROID_HOME=~/Android/Sdk ./gradlew :e2-app:assembleDebug
**`libmain.so` is 13.5 MB release** (E1's masonry-demo was 11 MB;
the difference is `client-core`'s `ureq`/`rustls` stack, which E1's
demo does not link). Release native lib, debug Gradle variant --
the combination E1 found necessary (a debug build's
`SetDebugUtilsObjectNameEXT` segfaults this emulator's Vulkan
driver).
*Emulator.* This checkout's own AVD (`ai-app-2`, not `ai-app`, which
another session already had up), booted with Vulkan the way E1
established: `GPU_HOST_FEATURES="-feature Vulkan"
VK_DRIVER_FILES=$HOME/Android/Sdk/emulator/lib64/vulkan/vk_swiftshader_icd.json
emu up`. `adb shell cmd gpu vkjson` confirmed a device before
anything was installed. Torn down with `emu down` at the end of this
session (see "Where things stand" below for the exact state left).
*What it does.* `fetch_rows()` (`e2-transcript/src/lib.rs`) makes one
blocking `fetch_transcript_page(session, None, 800, false)` call
before the widget tree exists, folds every line through
`client-core`, and groups tool runs -- 854 real events from a mixed
sandbox session (markdown paragraphs/headings/fences plus a
three-call tool run from the echo driver's `/tools 3`). Each
`TranscriptRow` becomes one `VirtualScroll<dyn Widget>` child,
built lazily from `VirtualScrollAction` the way
`masonry_winit/examples/virtual_fizzbuzz.rs` does it. **This is a
deliberate scope cut from "page 800 events" as live paging**: all
854 rows' content is fetched once, and what `VirtualScroll` pages is
*widget construction*, not a second round of network calls per
scroll -- wiring a background-thread fetch woken across the JNI
boundary (the way I2's `ssh.rs` attach-and-call works) is real work
this experiment did not need to answer its question. `markdown.rs`
is a `pulldown-cmark` event-stream walk into a small `Block` enum
(`Text`/`Heading`/`Code`), with its own module doc explaining the
one real ceiling it hit (below).
*Verification.* `cargo fmt -p e2-transcript -- --check` clean.
`cargo ndk -t x86_64 -P 26 clippy -p e2-transcript --all-targets`:
**zero warnings in this crate** (the only clippy output at all is
from `android-view` itself, a vendored dependency this experiment
does not own). `cargo ndk -t x86_64 -P 26 test -p e2-transcript
--lib` (run against the emulator, since the crate is
`cfg`-unconditionally Android): 1 test, `markdown::parse`'s block
split, passing. No larger test surface exists to port -- this is a
throwaway screen, not a library, matching AGENTS.md's "match the
codebase's testing posture."
*Screenshots* (all `/tmp`, not committed -- see the standing rule
against transcripts leaving this repo, which applies equally to a
screenshot of one): `e2-screenshot2.png` first real content;
`e2-expand.png` a tool row expanded with its top edge held;
`e2-markdown.png` a heading/bold/italic/inline-code/link/fenced-code
message (the "You said: ## A Heading" line is the sandbox's echo
driver prefixing the literal input text before the `##`, which
keeps `pulldown-cmark` from recognising it as a heading -- a fixture
artifact, not a finding about Masonry).
**The seven behaviours, each shown or given a sourced reason:**
1. **One selectable body of text spanning rows -- not possible, and
it is a real ceiling, not an oversight.** `Prose` wraps exactly
one `TextArea<false>`, which wraps exactly one
`parley::PlainEditor` (`masonry/src/widgets/prose.rs`: "Note that
copying is not yet implemented"). Selection lives entirely inside
that one editor: `TextArea::on_pointer_event`
(`masonry/src/widgets/text_area.rs:414-459` in the pinned `xilem`
commit) captures the pointer on `Down`
(`ctx.capture_pointer()`) and drives `self.editor`'s own
`extend_selection_to_point` on `Move` -- there is no code path,
in `masonry_core` or `masonry`, that extends a selection into a
second widget's editor. A drag that starts in one row's `Prose`
and continues into the next is still that first row's own
`PlainEditor` being asked for a point outside its bounds; it
cannot reach the second row's text. Confirmed by reading, not
guessed at: there is no `SelectionContainer`-shaped type
anywhere in `masonry`, `masonry_core` or `xilem` (checked with
`grep -rln "SelectionContainer\|cross.widget.*selection"`, zero
hits).
2. **Rich inline text -- block-level yes, inline no, and both for
the same reason.** `TextArea::edit_styles()` returns one
`&mut StyleSet<T>` for the whole editor
(`masonry_core/src/core/text.rs:29-32` defines `StyleSet` as
`parley::StyleSet<BrushIndex>`, applied editor-wide); the type's
own comments say why nothing finer exists yet:
`// TODO: RichTextInput 👀` and
`// TODO: Support for links - https://github.com/linebender/xilem/issues/360`
at `masonry/src/widgets/text_area.rs:43-44`. So bold, italic,
inline code and a link *inside one paragraph* cannot each carry
their own style without leaving `TextArea` for a hand-rolled
`parley::Layout` (which loses selection, the caret and copy,
since those live inside `PlainEditor` specifically). What **is**
real: each markdown block is its own `Prose`, so a heading is a
bigger font and a fenced code block is monospace, screenshotted
in `e2-markdown.png` -- block-level style works because it is
block-level *widgets*, not a rich-text API. Tables and per-token
syntax colour inside a fence hit the identical ceiling (both are
per-range styling) and were not attempted for the same reason.
`markdown.rs`'s degraded rendering (backticks kept literally,
`[text](url)` shown as `text (url)`) is the honest fallback,
documented at the point it is produced.
3. **Bottom-anchored virtualised list, paged, hold-top-edge on
expand -- mostly shown, with one real gap in the anchor API.**
`VirtualScroll<dyn Widget>` holds all 854 folded rows;
`overwrite_anchor` before swapping a tool row's widget for its
expanded/collapsed version is exactly the primitive
`holdTopEdge` needs, and it worked: `e2-expand.png` shows the
row growing downward from the same top edge it had collapsed,
no jump. Virtualisation is real (`ui-trace elements` only ever
lists the rows currently on screen, never all 854). **What did
not come free: hugging the bottom of the screen.**
`VirtualScroll::new`'s doc says "the item at `initial_anchor`
will have its top aligned with the top of the scroll area" --
so anchoring on the last row puts that row's top at the
viewport's *top*, with empty space below it, not at the
viewport's bottom the way a chat transcript wants (visible in
`e2-screenshot2.png`). The complete public `WidgetMut` surface of
`VirtualScroll` is `new`, `with_valid_range`,
`will_handle_action`, `add_child`, `remove_child`, `child_mut`,
`set_valid_range`, `overwrite_anchor`
(`masonry/src/widgets/virtual_scroll.rs:257-428`) -- no
scroll-offset setter and no reverse/bottom-up layout mode exist
to ask for the other behaviour. Backward paging beyond the
initial 800 was not exercised, per the scope cut above.
4. **The soft keyboard -- inherited gap, not re-investigated.** E2's
screen has no `TextInput`, only read-only `Prose`/`Button`, so it
does not hit `masonry/src/lib.rs:531`'s `as_input_connection`
returning `None` directly -- but it would the moment a composer
is added, per E1's finding. Nothing new to add here.
5. **Platform integration -- out of scope by design.** Foreground
service, notifications, share sheet, deep link, Keystore,
camera, back gesture, edge-to-edge, local-network permission are
E3's list in RUST.md's own experiment order, not E2's.
6. **Accessibility names -- shown, and the bench-script dependency
actually exercised.** `ui-trace record --do "tap '> 3 tool
calls'"` found the button by its label and pressed it (that tap
is what produced `e2-expand.png`); `Prose` rows surface their
text as their accessible name too (`ui-trace elements` lists
"You said: One more short reply..." etc. as named nodes). Tap by
name, the rule this whole project's bench scripts depend on,
works against this screen.
7. **Measurable frames -- blocked, and this is the finding E2 was
really testing for.** Two separate problems, one of them fatal
to the render-numbers half of this box's own pass condition.
First, Masonry has no render-report/per-widget-cost
instrumentation the way Compose's `DebugStats` gives this
project -- building one was out of scope here. Second, and this
is the one that matters: **neither of Masonry's two scrolling
widgets responds to a touch drag at all.**
`VirtualScroll::on_pointer_event`
(`masonry/src/widgets/virtual_scroll.rs:504-523`) and
`Portal::on_pointer_event`
(`masonry/src/widgets/portal.rs:259-267`) both match only
`PointerEvent::Scroll` (wheel/trackpad deltas) and do nothing
with `PointerEvent::Down`/`Move`/`Up` -- there is no drag-to-scroll
gesture logic anywhere in the widget set. `android-view`'s own
Java bridge keeps the two paths separate at the source:
`RustView.java`'s `onTouchEvent` forwards raw touch straight to
Rust, and only `onGenericMotionEvent` (mouse/trackpad, not
touch) reaches the `ACTION_SCROLL` branch that becomes
`PointerEvent::Scroll`
(`android-view/src/events.rs:530`). Confirmed empirically, not
just by reading: a real swipe (`ui-trace`'s `swipe 540 1600 540
400 300`, twice) moved nothing (`e2-scroll.png` is pixel-identical
to the screen before it), and a synthetic Android wheel event
(`adb shell input scroll 540 1200 --axis VSCROLL,-5`) also moved
nothing. **This means `transcript-bench.sh`'s own gesture --
a finger swipe -- cannot scroll a Masonry transcript on Android
today, at all, on this framework commit.** So the "render
numbers land within the Compose baseline" half of this box's
pass condition cannot be attempted, let alone met: there is no
way to perform the scroll the comparison asks for. This is not
a performance shortfall to close by writing faster code: it is
an absent input path upstream. The fix is a drag-to-scroll
gesture in `on_pointer_event` (the same place `TextArea`'s own
caret-drag logic already lives, so the pattern -- capture on
`Down`, accumulate delta on `Move`, release on `Up` -- exists
in this codebase already, just not wired into either scrolling
widget), and it belongs upstream in `xilem` rather than in this
project.
**Net for RUST.md's recommendation.** Item 3 ("decide when the
transcript screen exists in both, from the measurements") cannot be
decided by a render-number comparison yet, because the comparison's
own gesture does not work on Masonry on Android. What *can* be
compared today is structural: iris already has a working scroll
gesture and a working touch model (I2, 2026-09-05) that Masonry's
upstream commit does not yet have for this exact case. That is a
point in iris's favour that a frame-time number would not have
shown any more clearly.
- [ ] **E3 — the shell.** Kotlin `MainActivity` + `NotificationService`
+ Keystore + share intent calling into Rust over JNI, with the SSE
follow loop in Rust. Pass: a notification arrives with the app