RUST.md: E2 -- a transcript in Masonry on android-view, and the touch-scroll gap it found
Built a real transcript screen (e2-transcript, beside E1's demo in ~/src/android-view) against a live app/ui-sandbox.sh session through client-core: real fold, real ApiClient, VirtualScroll<dyn Widget> over 854 events, block-level markdown via pulldown-cmark into Prose, and a tool row that holds its top edge on expand via overwrite_anchor. The headline result is negative and load-bearing: neither VirtualScroll nor Portal reacts to a touch drag, only to wheel-style PointerEvent::Scroll (virtual_scroll.rs:504-523, portal.rs:259-267), confirmed both by reading and empirically (a real swipe and a synthetic Android scroll event both moved nothing). That blocks transcript-bench.sh's own gesture, so the render-number half of E2's pass condition has no comparison to make yet. Selection across rows and per-span rich text are also confirmed impossible on the pinned xilem commit, each cited to its source. Full writeup, repro commands and screenshots list in E2's own box. Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
This commit is contained in:
1 parent
19c36e37f2
commit
6e0bd06e4d
1 file changed
+254
-13
@@ -34,11 +34,35 @@ the emulator, not by Mesa" and "the present mode was not the cause" are
|
|||||||
worth as much as the successes, because they are what stops the next
|
worth as much as the successes, because they are what stops the next
|
||||||
session spending an afternoon on them again.
|
session spending an afternoon on them again.
|
||||||
|
|
||||||
## Where things stand (2026-09-04)
|
## Where things stand (2026-09-05)
|
||||||
|
|
||||||
- **Done**: E0 (toolchain), E1 (Masonry on android-view, which found the
|
- **Done**: E0 (toolchain), E1 (Masonry on android-view, which found the
|
||||||
keyboard gap — now explained, see below), I0a, I0b (iris builds on a
|
keyboard gap — now explained, see below), E2 (a transcript in Masonry,
|
||||||
pinned nightly and runs), I1 (parley + glyph atlas).
|
which found that Masonry has no touch-scroll on Android at all — see
|
||||||
|
below), I0a, I0b (iris builds on a pinned nightly and runs), I1 (parley +
|
||||||
|
glyph atlas).
|
||||||
|
- **E2 done, 2026-09-05, and its headline finding changes what "decide
|
||||||
|
from the measurements" (recommendation item 3) can mean right now.**
|
||||||
|
Built a real transcript screen (`~/src/android-view/e2-transcript`,
|
||||||
|
local, not committed — see E2's own box), fetching 854 real events from
|
||||||
|
an `app/ui-sandbox.sh` session through `client-core`. Six of the seven
|
||||||
|
"hard to get back" behaviours are answered with evidence either way;
|
||||||
|
the seventh (measurable frames) is **blocked before it can even start**:
|
||||||
|
neither of Masonry's scrolling widgets (`VirtualScroll`, `Portal`)
|
||||||
|
reacts to a touch drag, only to a wheel-style `PointerEvent::Scroll` —
|
||||||
|
confirmed by reading (`virtual_scroll.rs:504-523`, `portal.rs:259-267`)
|
||||||
|
and empirically (a real swipe and a synthetic Android scroll event both
|
||||||
|
moved nothing on screen). So `transcript-bench.sh`'s own gesture cannot
|
||||||
|
be performed against a Masonry transcript on Android today, which means
|
||||||
|
the render-number half of E2's pass condition has no comparison to make
|
||||||
|
yet — not a bad number, no number obtainable at all. Selection
|
||||||
|
spanning rows and per-span rich text (bold/italic/inline
|
||||||
|
code/links inside one paragraph) are also confirmed not possible on
|
||||||
|
the pinned commit, each for a specific, cited reason. What did work:
|
||||||
|
block-level rich text (heading size, monospace fences), real
|
||||||
|
virtualisation of 854 rows, `overwrite_anchor`-based hold-top-edge on
|
||||||
|
expand (screenshotted), and tap-by-name accessibility. Full writeup,
|
||||||
|
every citation, and the exact repro commands are in E2's own box below.
|
||||||
- **Done, 2026-09-04: the `Widget::draw`/layout redesign (LAYOUT.md).**
|
- **Done, 2026-09-04: the `Widget::draw`/layout redesign (LAYOUT.md).**
|
||||||
`desired_width`/`desired_height`/`SizeCtx`/`Cache` are gone; every widget
|
`desired_width`/`desired_height`/`SizeCtx`/`Cache` are gone; every widget
|
||||||
in `iris/src/widget/` implements one `fn draw(&mut self, &mut Painter) ->
|
in `iris/src/widget/` implements one `fn draw(&mut self, &mut Painter) ->
|
||||||
@@ -98,8 +122,7 @@ session spending an afternoon on them again.
|
|||||||
now renders on the emulator on both backends (screenshotted); the
|
now renders on the emulator on both backends (screenshotted); the
|
||||||
GLES-only `D2`/`D2Array` warning was confirmed a red herring — still
|
GLES-only `D2`/`D2Array` warning was confirmed a red herring — still
|
||||||
present post-fix, harmless. See I2's own entry below for the full
|
present post-fix, harmless. See I2's own entry below for the full
|
||||||
writeup. **E2** (a transcript in Masonry) can go in parallel in another
|
writeup. **E2** (a transcript in Masonry) is done — see its own box.
|
||||||
session.
|
|
||||||
- **`client-core` built (2026-09-04)**, item 1 of the recommendation:
|
- **`client-core` built (2026-09-04)**, item 1 of the recommendation:
|
||||||
`event-model/` (the event types, now shared with `server/`) and
|
`event-model/` (the event types, now shared with `server/`) and
|
||||||
`client-core/` (REST and SSE clients, transcript fold, cache, highlighter,
|
`client-core/` (REST and SSE clients, transcript fold, cache, highlighter,
|
||||||
@@ -708,14 +731,232 @@ accepted.
|
|||||||
the pid captured at launch) rather than to whatever AVD is running
|
the pid captured at launch) rather than to whatever AVD is running
|
||||||
when it wakes, and only one should be armed at a time.
|
when it wakes, and only one should be armed at a time.
|
||||||
|
|
||||||
- [ ] **E2 — a transcript in Masonry.** One screen: open a sandbox session,
|
- [x] **E2 — a transcript in Masonry (2026-09-05).** Built and run on this
|
||||||
page 800 events into `VirtualScroll` bottom-anchored, draw markdown
|
emulator. It found the thing it was measuring for: a framework-wide
|
||||||
from `pulldown-cmark` into Parley rich text with links and code
|
gap that blocks the bench comparison itself, plus a full accounting
|
||||||
chips, select across two rows with the platform handles, expand a
|
of the seven behaviours. Ticked on E1's own precedent -- "everything
|
||||||
tool row holding its top edge. Pass: the render numbers land within
|
it was meant to establish is established, including the one gap."
|
||||||
the Compose baseline in `transcript-bench.sh` on the GPU emulator
|
|
||||||
(same gestures, same session), and every one of the seven behaviours
|
*Where it lives.* `~/src/android-view/e2-transcript` (new workspace
|
||||||
above is either shown or has a written reason it cannot be.
|
member, `crate-type = ["cdylib"]`, `lib.name = "main"`), packaged by
|
||||||
|
a new Gradle module `~/src/android-view/e2-app` copied from
|
||||||
|
`masonry-app` (`E2View`/`E2Activity`, package
|
||||||
|
`org.linebender.android.e2transcript`). Neither is committed to
|
||||||
|
`ai-app-2` or pushed anywhere -- same as E1, this is a local
|
||||||
|
experiment against the `xilem` commit
|
||||||
|
`e14ba3a5f9461b403cb30d95826187fba7f6924b` and the `android-view`
|
||||||
|
commit `bec6c62a96cef8239b0fd7fedeef9b184d02e3a1`, reproducible from
|
||||||
|
the commands below rather than from a remote.
|
||||||
|
|
||||||
|
*Build.* Depends on `client-core`/`event-model` from this checkout by
|
||||||
|
path (`../../../repos/ai-app-2/client-core`) -- real code, not a
|
||||||
|
reimplementation: `ApiClient`/`UreqTransport` for the HTTP fetch,
|
||||||
|
`fold_event`/`group_tool_runs` for the transcript fold, exactly what
|
||||||
|
the app itself would use. The sandbox CA and a session's URL/token
|
||||||
|
are baked in at build time via `env!()`/`include_bytes!()`, the same
|
||||||
|
pattern the real APK uses to pin its CA (AGENTS.md), since this is a
|
||||||
|
throwaway screen with no enrollment flow:
|
||||||
|
|
||||||
|
cd app && ./ui-sandbox.sh start # prints the port and token
|
||||||
|
sid=$(./ui-sandbox.sh spawn e2test)
|
||||||
|
./ui-sandbox.sh send "$sid" @/tmp/big.md # markdown content
|
||||||
|
./ui-sandbox.sh send "$sid" "/tools 3" # a grouped tool run
|
||||||
|
cd ~/src/android-view
|
||||||
|
E2_SANDBOX_URL=https://10.0.2.2:<port> \
|
||||||
|
E2_SANDBOX_TOKEN=<token> \
|
||||||
|
E2_SANDBOX_SESSION=<sid> \
|
||||||
|
E2_CA_PEM_PATH=$HOME/.config/ai-app/certs/ca.pem \
|
||||||
|
cargo ndk -t x86_64 -P 26 -o e2-app/src/main/jniLibs/ \
|
||||||
|
build -p e2-transcript --release
|
||||||
|
ANDROID_HOME=~/Android/Sdk ./gradlew :e2-app:assembleDebug
|
||||||
|
|
||||||
|
**`libmain.so` is 13.5 MB release** (E1's masonry-demo was 11 MB;
|
||||||
|
the difference is `client-core`'s `ureq`/`rustls` stack, which E1's
|
||||||
|
demo does not link). Release native lib, debug Gradle variant --
|
||||||
|
the combination E1 found necessary (a debug build's
|
||||||
|
`SetDebugUtilsObjectNameEXT` segfaults this emulator's Vulkan
|
||||||
|
driver).
|
||||||
|
|
||||||
|
*Emulator.* This checkout's own AVD (`ai-app-2`, not `ai-app`, which
|
||||||
|
another session already had up), booted with Vulkan the way E1
|
||||||
|
established: `GPU_HOST_FEATURES="-feature Vulkan"
|
||||||
|
VK_DRIVER_FILES=$HOME/Android/Sdk/emulator/lib64/vulkan/vk_swiftshader_icd.json
|
||||||
|
emu up`. `adb shell cmd gpu vkjson` confirmed a device before
|
||||||
|
anything was installed. Torn down with `emu down` at the end of this
|
||||||
|
session (see "Where things stand" below for the exact state left).
|
||||||
|
|
||||||
|
*What it does.* `fetch_rows()` (`e2-transcript/src/lib.rs`) makes one
|
||||||
|
blocking `fetch_transcript_page(session, None, 800, false)` call
|
||||||
|
before the widget tree exists, folds every line through
|
||||||
|
`client-core`, and groups tool runs -- 854 real events from a mixed
|
||||||
|
sandbox session (markdown paragraphs/headings/fences plus a
|
||||||
|
three-call tool run from the echo driver's `/tools 3`). Each
|
||||||
|
`TranscriptRow` becomes one `VirtualScroll<dyn Widget>` child,
|
||||||
|
built lazily from `VirtualScrollAction` the way
|
||||||
|
`masonry_winit/examples/virtual_fizzbuzz.rs` does it. **This is a
|
||||||
|
deliberate scope cut from "page 800 events" as live paging**: all
|
||||||
|
854 rows' content is fetched once, and what `VirtualScroll` pages is
|
||||||
|
*widget construction*, not a second round of network calls per
|
||||||
|
scroll -- wiring a background-thread fetch woken across the JNI
|
||||||
|
boundary (the way I2's `ssh.rs` attach-and-call works) is real work
|
||||||
|
this experiment did not need to answer its question. `markdown.rs`
|
||||||
|
is a `pulldown-cmark` event-stream walk into a small `Block` enum
|
||||||
|
(`Text`/`Heading`/`Code`), with its own module doc explaining the
|
||||||
|
one real ceiling it hit (below).
|
||||||
|
|
||||||
|
*Verification.* `cargo fmt -p e2-transcript -- --check` clean.
|
||||||
|
`cargo ndk -t x86_64 -P 26 clippy -p e2-transcript --all-targets`:
|
||||||
|
**zero warnings in this crate** (the only clippy output at all is
|
||||||
|
from `android-view` itself, a vendored dependency this experiment
|
||||||
|
does not own). `cargo ndk -t x86_64 -P 26 test -p e2-transcript
|
||||||
|
--lib` (run against the emulator, since the crate is
|
||||||
|
`cfg`-unconditionally Android): 1 test, `markdown::parse`'s block
|
||||||
|
split, passing. No larger test surface exists to port -- this is a
|
||||||
|
throwaway screen, not a library, matching AGENTS.md's "match the
|
||||||
|
codebase's testing posture."
|
||||||
|
|
||||||
|
*Screenshots* (all `/tmp`, not committed -- see the standing rule
|
||||||
|
against transcripts leaving this repo, which applies equally to a
|
||||||
|
screenshot of one): `e2-screenshot2.png` first real content;
|
||||||
|
`e2-expand.png` a tool row expanded with its top edge held;
|
||||||
|
`e2-markdown.png` a heading/bold/italic/inline-code/link/fenced-code
|
||||||
|
message (the "You said: ## A Heading" line is the sandbox's echo
|
||||||
|
driver prefixing the literal input text before the `##`, which
|
||||||
|
keeps `pulldown-cmark` from recognising it as a heading -- a fixture
|
||||||
|
artifact, not a finding about Masonry).
|
||||||
|
|
||||||
|
**The seven behaviours, each shown or given a sourced reason:**
|
||||||
|
|
||||||
|
1. **One selectable body of text spanning rows -- not possible, and
|
||||||
|
it is a real ceiling, not an oversight.** `Prose` wraps exactly
|
||||||
|
one `TextArea<false>`, which wraps exactly one
|
||||||
|
`parley::PlainEditor` (`masonry/src/widgets/prose.rs`: "Note that
|
||||||
|
copying is not yet implemented"). Selection lives entirely inside
|
||||||
|
that one editor: `TextArea::on_pointer_event`
|
||||||
|
(`masonry/src/widgets/text_area.rs:414-459` in the pinned `xilem`
|
||||||
|
commit) captures the pointer on `Down`
|
||||||
|
(`ctx.capture_pointer()`) and drives `self.editor`'s own
|
||||||
|
`extend_selection_to_point` on `Move` -- there is no code path,
|
||||||
|
in `masonry_core` or `masonry`, that extends a selection into a
|
||||||
|
second widget's editor. A drag that starts in one row's `Prose`
|
||||||
|
and continues into the next is still that first row's own
|
||||||
|
`PlainEditor` being asked for a point outside its bounds; it
|
||||||
|
cannot reach the second row's text. Confirmed by reading, not
|
||||||
|
guessed at: there is no `SelectionContainer`-shaped type
|
||||||
|
anywhere in `masonry`, `masonry_core` or `xilem` (checked with
|
||||||
|
`grep -rln "SelectionContainer\|cross.widget.*selection"`, zero
|
||||||
|
hits).
|
||||||
|
2. **Rich inline text -- block-level yes, inline no, and both for
|
||||||
|
the same reason.** `TextArea::edit_styles()` returns one
|
||||||
|
`&mut StyleSet<T>` for the whole editor
|
||||||
|
(`masonry_core/src/core/text.rs:29-32` defines `StyleSet` as
|
||||||
|
`parley::StyleSet<BrushIndex>`, applied editor-wide); the type's
|
||||||
|
own comments say why nothing finer exists yet:
|
||||||
|
`// TODO: RichTextInput 👀` and
|
||||||
|
`// TODO: Support for links - https://github.com/linebender/xilem/issues/360`
|
||||||
|
at `masonry/src/widgets/text_area.rs:43-44`. So bold, italic,
|
||||||
|
inline code and a link *inside one paragraph* cannot each carry
|
||||||
|
their own style without leaving `TextArea` for a hand-rolled
|
||||||
|
`parley::Layout` (which loses selection, the caret and copy,
|
||||||
|
since those live inside `PlainEditor` specifically). What **is**
|
||||||
|
real: each markdown block is its own `Prose`, so a heading is a
|
||||||
|
bigger font and a fenced code block is monospace, screenshotted
|
||||||
|
in `e2-markdown.png` -- block-level style works because it is
|
||||||
|
block-level *widgets*, not a rich-text API. Tables and per-token
|
||||||
|
syntax colour inside a fence hit the identical ceiling (both are
|
||||||
|
per-range styling) and were not attempted for the same reason.
|
||||||
|
`markdown.rs`'s degraded rendering (backticks kept literally,
|
||||||
|
`[text](url)` shown as `text (url)`) is the honest fallback,
|
||||||
|
documented at the point it is produced.
|
||||||
|
3. **Bottom-anchored virtualised list, paged, hold-top-edge on
|
||||||
|
expand -- mostly shown, with one real gap in the anchor API.**
|
||||||
|
`VirtualScroll<dyn Widget>` holds all 854 folded rows;
|
||||||
|
`overwrite_anchor` before swapping a tool row's widget for its
|
||||||
|
expanded/collapsed version is exactly the primitive
|
||||||
|
`holdTopEdge` needs, and it worked: `e2-expand.png` shows the
|
||||||
|
row growing downward from the same top edge it had collapsed,
|
||||||
|
no jump. Virtualisation is real (`ui-trace elements` only ever
|
||||||
|
lists the rows currently on screen, never all 854). **What did
|
||||||
|
not come free: hugging the bottom of the screen.**
|
||||||
|
`VirtualScroll::new`'s doc says "the item at `initial_anchor`
|
||||||
|
will have its top aligned with the top of the scroll area" --
|
||||||
|
so anchoring on the last row puts that row's top at the
|
||||||
|
viewport's *top*, with empty space below it, not at the
|
||||||
|
viewport's bottom the way a chat transcript wants (visible in
|
||||||
|
`e2-screenshot2.png`). The complete public `WidgetMut` surface of
|
||||||
|
`VirtualScroll` is `new`, `with_valid_range`,
|
||||||
|
`will_handle_action`, `add_child`, `remove_child`, `child_mut`,
|
||||||
|
`set_valid_range`, `overwrite_anchor`
|
||||||
|
(`masonry/src/widgets/virtual_scroll.rs:257-428`) -- no
|
||||||
|
scroll-offset setter and no reverse/bottom-up layout mode exist
|
||||||
|
to ask for the other behaviour. Backward paging beyond the
|
||||||
|
initial 800 was not exercised, per the scope cut above.
|
||||||
|
4. **The soft keyboard -- inherited gap, not re-investigated.** E2's
|
||||||
|
screen has no `TextInput`, only read-only `Prose`/`Button`, so it
|
||||||
|
does not hit `masonry/src/lib.rs:531`'s `as_input_connection`
|
||||||
|
returning `None` directly -- but it would the moment a composer
|
||||||
|
is added, per E1's finding. Nothing new to add here.
|
||||||
|
5. **Platform integration -- out of scope by design.** Foreground
|
||||||
|
service, notifications, share sheet, deep link, Keystore,
|
||||||
|
camera, back gesture, edge-to-edge, local-network permission are
|
||||||
|
E3's list in RUST.md's own experiment order, not E2's.
|
||||||
|
6. **Accessibility names -- shown, and the bench-script dependency
|
||||||
|
actually exercised.** `ui-trace record --do "tap '> 3 tool
|
||||||
|
calls'"` found the button by its label and pressed it (that tap
|
||||||
|
is what produced `e2-expand.png`); `Prose` rows surface their
|
||||||
|
text as their accessible name too (`ui-trace elements` lists
|
||||||
|
"You said: One more short reply..." etc. as named nodes). Tap by
|
||||||
|
name, the rule this whole project's bench scripts depend on,
|
||||||
|
works against this screen.
|
||||||
|
7. **Measurable frames -- blocked, and this is the finding E2 was
|
||||||
|
really testing for.** Two separate problems, one of them fatal
|
||||||
|
to the render-numbers half of this box's own pass condition.
|
||||||
|
First, Masonry has no render-report/per-widget-cost
|
||||||
|
instrumentation the way Compose's `DebugStats` gives this
|
||||||
|
project -- building one was out of scope here. Second, and this
|
||||||
|
is the one that matters: **neither of Masonry's two scrolling
|
||||||
|
widgets responds to a touch drag at all.**
|
||||||
|
`VirtualScroll::on_pointer_event`
|
||||||
|
(`masonry/src/widgets/virtual_scroll.rs:504-523`) and
|
||||||
|
`Portal::on_pointer_event`
|
||||||
|
(`masonry/src/widgets/portal.rs:259-267`) both match only
|
||||||
|
`PointerEvent::Scroll` (wheel/trackpad deltas) and do nothing
|
||||||
|
with `PointerEvent::Down`/`Move`/`Up` -- there is no drag-to-scroll
|
||||||
|
gesture logic anywhere in the widget set. `android-view`'s own
|
||||||
|
Java bridge keeps the two paths separate at the source:
|
||||||
|
`RustView.java`'s `onTouchEvent` forwards raw touch straight to
|
||||||
|
Rust, and only `onGenericMotionEvent` (mouse/trackpad, not
|
||||||
|
touch) reaches the `ACTION_SCROLL` branch that becomes
|
||||||
|
`PointerEvent::Scroll`
|
||||||
|
(`android-view/src/events.rs:530`). Confirmed empirically, not
|
||||||
|
just by reading: a real swipe (`ui-trace`'s `swipe 540 1600 540
|
||||||
|
400 300`, twice) moved nothing (`e2-scroll.png` is pixel-identical
|
||||||
|
to the screen before it), and a synthetic Android wheel event
|
||||||
|
(`adb shell input scroll 540 1200 --axis VSCROLL,-5`) also moved
|
||||||
|
nothing. **This means `transcript-bench.sh`'s own gesture --
|
||||||
|
a finger swipe -- cannot scroll a Masonry transcript on Android
|
||||||
|
today, at all, on this framework commit.** So the "render
|
||||||
|
numbers land within the Compose baseline" half of this box's
|
||||||
|
pass condition cannot be attempted, let alone met: there is no
|
||||||
|
way to perform the scroll the comparison asks for. This is not
|
||||||
|
a performance shortfall to close by writing faster code: it is
|
||||||
|
an absent input path upstream. The fix is a drag-to-scroll
|
||||||
|
gesture in `on_pointer_event` (the same place `TextArea`'s own
|
||||||
|
caret-drag logic already lives, so the pattern -- capture on
|
||||||
|
`Down`, accumulate delta on `Move`, release on `Up` -- exists
|
||||||
|
in this codebase already, just not wired into either scrolling
|
||||||
|
widget), and it belongs upstream in `xilem` rather than in this
|
||||||
|
project.
|
||||||
|
|
||||||
|
**Net for RUST.md's recommendation.** Item 3 ("decide when the
|
||||||
|
transcript screen exists in both, from the measurements") cannot be
|
||||||
|
decided by a render-number comparison yet, because the comparison's
|
||||||
|
own gesture does not work on Masonry on Android. What *can* be
|
||||||
|
compared today is structural: iris already has a working scroll
|
||||||
|
gesture and a working touch model (I2, 2026-09-05) that Masonry's
|
||||||
|
upstream commit does not yet have for this exact case. That is a
|
||||||
|
point in iris's favour that a frame-time number would not have
|
||||||
|
shown any more clearly.
|
||||||
- [ ] **E3 — the shell.** Kotlin `MainActivity` + `NotificationService`
|
- [ ] **E3 — the shell.** Kotlin `MainActivity` + `NotificationService`
|
||||||
+ Keystore + share intent calling into Rust over JNI, with the SSE
|
+ Keystore + share intent calling into Rust over JNI, with the SSE
|
||||||
follow loop in Rust. Pass: a notification arrives with the app
|
follow loop in Rust. Pass: a notification arrives with the app
|
||||||
|
|||||||
Reference in new issue
Block a user