iris/ai-app
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b133d8594347cfe33603841f46c8697a6355b809
ai-app/RUST.md
T
irisandClaude Fable 5.1 b133d85943 RUST.md, IRIS.md, CLIENT_CORE.md: record E4 done 
RUST.md: E4 ticked with the screenshot path, the exact commands against
app/ui-sandbox.sh, and the streaming-duplication bug the screenshot found;
"Where things stand" moved E4 out of "in flight" into its own done bullet.
IRIS.md: transcript_ui::build_tree, the public API change transcript-ui
gained for this. CLIENT_CORE.md: client_core::config's table row and its
correspondence note.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 12:55:59 -04:00

2588 lines
160 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Moving the app to Rust
Working document for the question Iris asked on 2026-09-04: what are the
options for switching the phone app to Rust, ideally pure Rust with one UI
framework shared with a future winit-based desktop application, at full
feature parity and without giving up anything native, performance
especially. Constraints she set: no Dioxus and nothing that draws through a
WebView; **no UI DSL** (which rules out Makepad and Slint); the result
should stay lightweight; platform-specific pieces are fine to maintain;
reimplementing a framework piece from scratch where it does not fit is
fine; effort and elapsed time do not matter, long-term robustness does;
this clone is where things get tried before anything is committed to
`ai-app`. Her own library, [iris](https://github.com/cat16/iris), is the
**in-house framework to be built up** for this, with Masonry as the
yardstick it is measured against.
Decisions get a date and a reason here, the way `PLAN.md` does. A section
describes something that has been tried only where it says so, with a date.
## Keep this file current as you work
**This file is the handoff, and it is meant to let a session be cleared.**
Write each result into it *as you get it*, not at the end: the box ticked
or the reason it could not be, the measurement with its number, the
decision with its date and what it rejected, and anything that cost time to
find out. Then a session that has filled its context can be cleared and the
next one can pick up from this file alone, which is much cheaper than
carrying a long conversation or re-deriving what was already measured.
Two things that follow. Write for somebody who was not here — name the
command, the file and the number rather than "the fix" or "the earlier
run". And write the failures and the dead ends too: "Venus is blocked by
the emulator, not by Mesa" and "the present mode was not the cause" are
worth as much as the successes, because they are what stops the next
session spending an afternoon on them again.
## Where things stand (2026-09-05)
- **In flight, 2026-09-05 (session cleared mid-work, picked up again):**
the I5 touch-drag pan-vs-select gap, as a `DragArbiter` in
`iris/src/sense.rs` wired into `transcript-ui`'s selection. Design
choices are summarised in `DECISIONS.md` at the repo root, which is the
file Iris reads for choices made without her. Next after it: I5's
Android integration and the bench numbers.
- **E4 done, 2026-09-05.** `iris/desktop-app`: a winit window with a
session list beside `transcript-ui`'s screen (`build_tree`), against a
real `ai-server` through `client-core`, enrolled from the same
`aiapp://enroll?...` link a phone scans. Both pass conditions held on
`app/ui-sandbox.sh` -- see E4's own box for the commands, the
screenshot, and a real streaming-duplication bug the screenshot found
and a regression test now covers.
- **Done**: E0 (toolchain), E1 (Masonry on android-view, which found the
keyboard gap — now explained, see below), E2 (a transcript in Masonry,
which found that Masonry has no touch-scroll on Android at all — see
below), E3 (the Kotlin/Java shell over a JNI bridge into Rust, both
pass conditions proved on the emulator — see its own box), E5 (the
Gradle-free packaging xtask, both pass conditions proved — see its own
box), I0a, I0b (iris builds on a pinned nightly and runs), I1 (parley +
glyph atlas), I2 (iris on android-view), I3 (`iris::widget::List`), I4
(host half).
- **E5 done, 2026-09-05.** `cargo xtask apk` (new `xtask/` crate at the
repo root, zero dependencies) replaces Gradle for packaging
`app/shellApp`: `cargo ndk` → `javac`/`d8` → `aapt2` → `zipalign` →
`apksigner`, signed with the same key `app/build-apk.sh` uses. Both pass
conditions held on this checkout's emulator: `adb install -r` over the
Gradle-built `shellApp` succeeded (same key, so the signatures matched),
and the notification service reached its follow-loop and posted a real
notification while the app was backgrounded. E3's open `kotlinc`
question resolved itself as a side effect of the one Gradle call still
needed for AndroidX dependency resolution — see E5's own box for the
full account, including the one disclosed place Gradle still runs and
what was deliberately left undone (a real-device `arm64-v8a` install,
dex shrinking).
- **I5 — the transcript screen in iris: partial, 2026-09-05 (ticked `[~]`
in its own box, not `[x]`).** `iris/transcript-ui/` builds a real
transcript screen — markdown-folded rows in `iris::widget::List`,
cross-row selection, a growing composer, tool-row expand-hold — on top
of a new, genuinely useful iris capability this box added:
**`SpanStyle`**, per-range text styling (`core/src/primitive/text.rs`),
which is what lets one wrapped, selectable `TextEdit` carry a heading,
bold, italic, inline code and a link all inside the same paragraph —
exactly the inline-rich-text ceiling E2 found Masonry structurally
unable to cross. Screenshotted via `run-headless.sh` (real inline
styling visible, not just block-level). 9 new tests, all passing;
`cargo build/clippy/fmt/test --workspace` and `cargo ndk` (both `iris`
and `transcript-ui`) all clean. **What did not happen this pass**: any
Android integration for this specific screen (no cdylib/Gradle shell
exists for it yet, unlike `tabs-ui`'s `iris-android-app`), and therefore
the emulator-side pass condition (`transcript-bench.sh` against the
Compose baseline, `ui-trace` tap-by-name on a row) — `emu list` showed
the one emulator here held by another session, but the real blocker is
that the integration work itself is unbuilt, not the emulator being
busy. Touch-drag panning over a row's own rendered text is also not yet
reachable, for a specific, diagnosed reason (it competes with this box's
own row-level drag-select for the same gesture) rather than an absent
primitive. Full accounting, every citation, and the dated
IRIS_TODO.md items are in I5's own box below.
- **E3 done, 2026-09-05, and unlike E1/E2 it is committed to this repo**
(`android-shell/` — a JNI-bridge crate on `client-core` — plus a new
Gradle module `app/shellApp/`, left deliberately separate from
`app/androidApp` so its ~13,000 lines of working Compose UI are
untouched). Both pass conditions held: a notification arrived in
Android's drawer while the app was closed, and a shared text share
landed as a real `userMessage` in a sandbox session's transcript. Found
and fixed three real bugs along the way — a generic `JObject` native
parameter silently exporting the wrong JNI signature
(`UnsatisfiedLinkError`), a class-by-name lookup failing from this
crate's own background thread because a Rust-attached thread has no app
`ClassLoader` (`Error::NoClassDefFound`, invisible without a logger
installed), and `onStartCommand` opening two `/notifications`
connections per enrollment — the last one a latent bug in
`Notifications.kt` itself, found here rather than there. See E3's own
box for the full account, the exact commands, and what was deliberately
cut (attachment uploads, a session picker, the on-screen/banner
suppression — all pending E4's screen).
- **I4 — accessibility names via AccessKit: host half done and verified
2026-09-05, ticked in the box below.** `iris_core::ui::access::AccessTree`
builds one flat AccessKit tree from `Widgets::named()` (a side set only
`.label()` populates, so an unnamed widget costs this nothing), pushed
through `accesskit_winit` on the desktop and `accesskit_android` on
Android, updated only when a name/role/bounds actually changes (a
counter confirms it: 1 rebuild on first draw, 0 across an unchanged
frame, 1 more after a real move). E1's detach-abort mitigation is
carried (`android/access.rs`'s `raise_if_enabled`). Every check that
doesn't need the emulator is clean — see I4's own box for the exact
numbers. **What's left**: the emulator itself is held by another session
this pass, so `ui-trace record --do "tap 'pad'"` against
`iris-android-app`'s tabs screen (which now has five named buttons) has
not been run for real yet — exact commands at the bottom of I4's box.
- **E2 done, 2026-09-05, and its headline finding changes what "decide
from the measurements" (recommendation item 3) can mean right now.**
Built a real transcript screen (`~/src/android-view/e2-transcript`,
local, not committed — see E2's own box), fetching 854 real events from
an `app/ui-sandbox.sh` session through `client-core`. Six of the seven
"hard to get back" behaviours are answered with evidence either way;
the seventh (measurable frames) is **blocked before it can even start**:
neither of Masonry's scrolling widgets (`VirtualScroll`, `Portal`)
reacts to a touch drag, only to a wheel-style `PointerEvent::Scroll` —
confirmed by reading (`virtual_scroll.rs:504-523`, `portal.rs:259-267`)
and empirically (a real swipe and a synthetic Android scroll event both
moved nothing on screen). So `transcript-bench.sh`'s own gesture cannot
be performed against a Masonry transcript on Android today, which means
the render-number half of E2's pass condition has no comparison to make
yet — not a bad number, no number obtainable at all. Selection
spanning rows and per-span rich text (bold/italic/inline
code/links inside one paragraph) are also confirmed not possible on
the pinned commit, each for a specific, cited reason. What did work:
block-level rich text (heading size, monospace fences), real
virtualisation of 854 rows, `overwrite_anchor`-based hold-top-edge on
expand (screenshotted), and tap-by-name accessibility. Full writeup,
every citation, and the exact repro commands are in E2's own box below.
- **Done, 2026-09-04: the `Widget::draw`/layout redesign (LAYOUT.md).**
`desired_width`/`desired_height`/`SizeCtx`/`Cache` are gone; every widget
in `iris/src/widget/` implements one `fn draw(&mut self, &mut Painter) ->
Size`. A moved widget (`Scroll`, `Offset`) now costs one
`move_offsets` write resolved by a shared `resolve_move` WGSL function in
both shader stages, independent of how many primitives are in its
subtree — measured at 500 in `iris/src/layout_tests.rs`, which also
covers the unchanged-frame, hit-test-after-move and mask-follows-move
pass conditions as plain unit tests (no GPU or window needed, since
`UiRenderState` touches neither). All four examples render
pixel-identically to before the change. See LAYOUT.md's "Deviations
found during implementation" for five real bugs the design's first draft
did not anticipate — worth reading before touching `Aligned`, `Sized`,
`MaxSize`, `Scroll`, or the move-slot lifecycle again. `GpuTextures::grow_array`
(a second atlas layer opening) has now been exercised too, on `tabs` with
`PAGE` temporarily lowered — see TEXTURES.md's "Exercised, 2026-09-04".
Not done: a pixel-level screenshot check of a `Masked`-wrapped `Scroll`
(no example builds one yet — the numeric check in `layout_tests.rs`
stands in).
- **E1's keyboard gap is Masonry's `as_input_connection` returning `None`
(a TODO), not android-view or `EditorInfo`.** android-view's own demo
implements the `InputConnection` trait over a parley editor and gets
real Gboard suggestions on this emulator — screenshotted 2026-09-04.
android-view's `accesskit_android` adapter also has a reproducible abort
(a client detaching, not attaching, is the trigger) — see E1 below for
both, with the mitigation iris/I4 needs to carry.
- **Resolved, 2026-09-04: iris's binding array does not survive real
Android hardware.** iris's texture pipeline used to ask every device,
unconditionally, for `VK_EXT_descriptor_indexing` ("bindless" binding
arrays), which a real share of Android hardware lacks. It has been
rebuilt per TEXTURES.md's "Recommended shape": the glyph atlas is one
`texture_2d_array` (a layer per page), a standalone image is its own
ordinary `Texture`/`BindGroup`, and `request_device` now asks for no
features and no binding-array limits at all. `rigs/gpu-probe`, rewritten
to match, confirms `request_device` now succeeds on the emulator's
software Vulkan (`EMU_GPU=software`, SwiftShader) — see TEXTURES.md's
"Implemented, 2026-09-04" for the exact command and output, and for what
was verified (rendering, via `run-headless.sh`) versus what was reasoned
through but not separately stress-tested (a real second-atlas-page
grow under load). Nothing here has been run on real Android hardware
yet, only the emulator; the Android Vulkan Profile 2025 sourcing in
"iris's binding array does not survive real Android hardware" below is
what stands in for that until I2 gets a device.
- **I2 — iris on android-view: done 2026-09-05.** The android-view backend
(`iris/src/android/`), the `iris-android-app` cdylib and Gradle shell,
insets, the back gesture, and the full `InputConnection` bridge are all in
and measured working — Gboard's suggestion strip reads real buffer content
through it, the same bar E1 set. **The render gap (nothing drew but the
clear colour) is fixed**: `UiRenderNode::new` seeded the GPU's window
uniform from `WindowUniform::default()` (0, 0) rather than the surface's
real size, so the vertex shader's `/ window.dim` produced `NaN`/`Inf` clip
positions on every primitive, on both Vulkan and GLES — winit's backend
never hit this because winit fires an initial `WindowEvent::Resized` that
corrects it before the first frame, and android-view has no equivalent
event. Fixed by seeding the uniform from `config.width`/`height` at
construction instead of depending on a later resize call. The tabs example
now renders on the emulator on both backends (screenshotted); the
GLES-only `D2`/`D2Array` warning was confirmed a red herring — still
present post-fix, harmless. See I2's own entry below for the full
writeup. **E2** (a transcript in Masonry) is done — see its own box.
- **I3 — `iris::widget::List` built and benchmarked 2026-09-05, ticked in
the box below.** Variable-height rows, virtualised, moved not
relaid-out on scroll, insert-above-anchor and expand-hold both measured
flat across N = 100/1,000/10,000. What is left is wiring it into an
actual transcript screen and comparing against `transcript-bench.sh`'s
Compose baseline on the GPU emulator, which needs a session/scroll model
around it (closer to I5's scope) — see I3's own box for the exact
command once that screen exists. Read `list.rs`'s module doc and
`IRIS.md`'s 2026-09-05 entry before touching it: a widget that fills
whatever region it's offered (a `Rect` background) cannot be measured at
a throwaway region and merely repositioned, a lesson that generalises
beyond this one widget.
- **`client-core` built (2026-09-04)**, item 1 of the recommendation:
`event-model/` (the event types, now shared with `server/`) and
`client-core/` (REST and SSE clients, transcript fold, cache, highlighter,
ANSI parser, 85 ported tests). `CLIENT_CORE.md` maps Kotlin file to Rust
module and lists what is not yet covered. `./run-tests.sh` runs all three
crates.
- **The app itself is untouched.** Everything so far is in `iris/`, in
`rigs/gpu-probe` (a headless wgpu/Vulkan feature probe, pushable to a
device with no APK — see the binding-array section), and in the other
rigs; nothing under `app/` or `server/` has changed.
- **Changed outside this repo**, both in `emulator-tools` and both pushed:
`avd_serial` now validates its cache by asking the device its AVD name
rather than by checking the serial is still attached (a recycled port
silently pointed this checkout at another session's emulator), and
`EMU_GPU=software` was added as an opt-in that keeps a run off the host
GPU and gives the guest a software Vulkan device. The default is
unchanged, because `-gpu host` was measured and the Compose benchmarks
depend on it.
## What has to be reproduced
The app is ~19,000 lines of Kotlin. It splits three ways, and the split is
what decides how much of a port is mechanical.
**Pure logic with no Compose or Android in it, ~4,500 lines.** `Api.kt`
(1,142), `Events.kt`, `EventStream.kt`, `Sse.kt`, `TranscriptCache.kt`
(589, touches `java.io.File` only), `TranscriptSource.kt`,
`MarkdownSyntax.kt`, `Languages.kt`, `Highlighter.kt`, `Ansi.kt`,
`ResetCountdown.kt`, `Durations.kt`, `Sizes.kt`, `ModelName.kt`,
`LoadState.kt`, `ImportableStream.kt`. `TranscriptUnits.kt` and
`TranscriptItems.kt` (the event fold into rows, ~940 lines) are logic with
a handful of Compose annotations. This is also exactly the code that has
JVM unit tests today. All of it ports directly, and most of it already has a
Rust twin in `server/`: `Events.kt` is a hand-kept mirror of
`session/driver.rs`'s enum, the highlighter and the syntax scanner exist on
the server for the explorer, and the cache compares the server's own JSON
lines. **Sharing these types between server and app is the single largest
"keep things in sync" win available, and it does not depend on which UI
framework wins.**
**Compose UI, ~13,000 lines.** Screens, dialogs, the transcript list, the
markdown renderer's customisations, tool cards, the file explorer viewer and
editor. This is the part a UI framework choice is about.
**Android platform code, ~1,500 lines**, spread over 20 files. Every one of
these is a Java-side object that no Rust framework can replace, because
Android only offers them as Java classes:
- `NotificationService` — a **foreground service** holding the
`/notifications` SSE stream while the app is closed, with its ongoing
notification, `specialUse` type and the `POST_NOTIFICATIONS` request.
- `MainActivity` — edge-to-edge, the `ACCESS_LOCAL_NETWORK` runtime
permission (Android 17), `singleTop` intent routing for `aiapp://enroll`,
notification taps, and the **share sheet** (`ACTION_SEND`, any MIME type).
- `ServerConfig` — the bearer token sealed under an **Android Keystore**
AES-GCM key, shared with Dev Updater through `wg-app-link`'s `:link`.
- `EnrollmentScanActivity` — the in-app **QR scanner** (zxing, camera).
- `Attachments` — `ContentResolver` reads of shared URIs, `BitmapFactory`
decode and downscale, **EXIF** orientation.
- `SessionImage` — bitmap decode for produced images.
- `ScrollAnchor`, `Drafts` — `SharedPreferences`; `CrashLog` — `filesDir`.
- `TranscriptCache` — `cacheDir`.
- `DebugStats`/`FrameStats` — `Choreographer` frame timing and the render
report; `runtime-tracing` names composables in a system trace.
So **"pure Rust" on Android means Rust owns every line of logic and
drawing, behind a thin shell of Java stubs**, and a packaging step that
produces a signed APK. How thin, and whether Gradle is inevitable, are
answered below.
### How much Java is unavoidable, and why
Rust can *call* any Android API through JNI (`jni` crate, with
`ndk-context` handing over the `JavaVM` and the Activity): posting a
notification, `startForegroundService`, the Keystore, `ContentResolver`
reads, permission requests, `WindowInsets`, the clipboard. None of that
needs a line of Kotlin. What JNI cannot do is *define* a class that the
system instantiates **by name from the manifest** — an `Activity`, a
`Service`, an `Application`, a `BroadcastReceiver`. Those must exist as dex
bytecode inside the APK before any Rust runs, because the framework
constructs them and only then calls into native code. `NativeActivity` is
the platform's own stub for the Activity case; there is no
`NativeService`, and android-view ships its own `View` subclass for the
same reason.
So the floor is roughly **two Java classes of ten lines each**: an
`Activity` and a `Service` whose lifecycle methods are declared `native`
and registered from `JNI_OnLoad`, plus whatever android-view already
provides. Everything they would have done in Kotlin — insets, intent
routing, the SSE follow loop, the notification builder — is Rust reached
through those stubs. Writing the stubs in Java rather than Kotlin drops
`kotlinc` from the toolchain; `javac` comes with the JDK Gradle already
needs. Generating the dex from Rust is not worth it: there is no mature
Rust dex writer, and the stubs never change.
### Can the APK be built without Gradle?
Yes. An APK is a zip containing a binary-XML `AndroidManifest.xml`,
`resources.arsc`, `classes.dex`, `lib/<abi>/*.so` and assets, aligned and
signed with the v2 scheme. The tools are `aapt2` (manifest and resources),
`d8` (Java bytecode to dex), `zipalign` and `apksigner`, all in the SDK's
`build-tools`, none of them Gradle. Three ways to drive them:
- **A `cargo xtask`** (or `build.rs`-adjacent script) that runs `cargo ndk`
for each ABI, `javac` + `d8` for the stubs, `aapt2 link`, `zipalign`,
`apksigner`. About 150 lines, every step visible, no AGP, no Gradle
daemon holding 2.8 GB between builds. The pinned-CA constant becomes a
`build.rs` reading the same `certs/ca.pem` path.
- **[cargo-apk2](https://github.com/mzdk100/cargo-apk2)**: the maintained
successor to cargo-apk, and unlike it compiles `java_sources` /
`kotlin_sources` into the dex and declares multiple activities **and
services** with intent filters from `[package.metadata.android]`, with
per-profile keystores and optional `aapt2`. Exactly the shape needed;
the question is whether a third-party tool with one maintainer beats
150 lines we own.
- **cargo-apk / xbuild**: unmaintained and `NativeActivity`-only. No.
What Gradle would take with it: Android Lint (which found two real bugs
here, but in Kotlin that would no longer exist — with forty lines of Java
stubs there is little left for it to find), manifest merging, R8, and the
generated-source plumbing. What it gives back: one toolchain, `cargo`
end to end, and Dev Updater keeps calling `build-apk.sh` exactly as now.
**Recommendation: the xtask**, with cargo-apk2 read for the details it
already got right (v2 signing, `uses-feature`, ABI splits).
### The behaviours that are hard to get back
Reading the Compose code for what a replacement must be able to express,
rather than what it happens to look like:
1. **The transcript is one selectable body of text.** One
`SelectionContainer` around the whole lazy list, so a selection runs from
a reply into the tool output beneath it. The framework needs selectable
read-only rich text across many rows, with the platform's selection
handles and clipboard on the phone.
2. **Rich inline text**: markdown with links (one tap detector per text,
not a node per link), inline code chips drawn behind the text, tables
with wrapping cells and a sideways scroll, syntax-highlighted fences,
ANSI colour in tool output, Nerd Font icon glyphs. Needs a text layout
engine with spans, not just styled labels.
3. **A bottom-anchored virtualised list of variable-height rows**, paged in
both directions (800-event pages, `HISTORY_SCREENS` measured in
viewports), with a saved scroll anchor per session, "hold the edge
nearest the tap" when a row expands (`holdTopEdge`, done in the layout
pass so the wrong frame is never drawn), and rows keyed so that a run of
tool calls stays one row while it grows.
4. **The soft keyboard**: the composer resizes with the IME, the guard
against a stuck inset animation, drafts per session, autocorrect and
suggestions from the phone's own keyboard. This is where most Rust
frameworks fail on Android today; see below.
5. **Platform integration through the app model**: foreground service,
notifications, share sheet, deep link, Keystore, camera, back gesture,
edge-to-edge insets, local-network permission.
6. **Accessibility names on icon buttons**, which the bench scripts depend
on (`ui-trace` taps by label). A framework with no accessibility tree
also breaks the measuring rig.
7. **Measurable frames**: the debug render report, and a way to attribute
a frame's cost to a widget on the real phone.
## The two constraints that decide it
**1. Android text input.** Every framework built on `winit` inherits
winit's Android backend, and that backend cannot drive the soft keyboard
properly: the IME tracking issues
([#1823](https://github.com/rust-windowing/winit/issues/1823),
[#2766](https://github.com/rust-windowing/winit/issues/2766)) are open,
`ReceivedCharacter` is unimplemented on Android
([#2305](https://github.com/rust-windowing/winit/issues/2305)), and the
`android-activity` groundwork for editor actions only merged in February
2026 ([PR #214](https://github.com/rust-mobile/android-activity/pull/214))
with the winit half still to come. Composition, autocorrect and suggestions
need an `InputConnection` implemented on the Java side, which winit's
`NativeActivity`/`GameActivity` model does not offer. The frameworks that
type on Android today each wrote their own Java glue (Slint, Makepad), and
the one designed to do it the way Android intends is
[`android-view`](https://github.com/rust-mobile/android-view): a Rust
implementation of an Android `View`, with text input through
`InputConnection`, accessibility, touch, callbacks on the UI thread, usable
either as a whole app or embedded beside ordinary Android components. It is
marked WIP. Both Linebender (its Masonry demo lives in that repo) and
Robius/Makepad ([Robrix's release notes](https://github.com/project-robius/robrix/releases)
say Android lacks a "full" keyboard and they are integrating android-view
for it) are converging on it. **That makes android-view the phone-side
foundation whichever widget set sits on top**, and the first thing to
build and measure here.
**2. Rich, selectable text and a virtualised list.** Frameworks group by
their text stack:
- **Parley + Fontique + Vello** (Linebender): rich spans, selection and
editing utilities, IME support driven through `ui-events`, AccessKit text
properties ([Linebender 2026 Q1](https://linebender.org/blog/tmil-25/),
[parley](https://github.com/linebender/parley)). Used by Masonry/Xilem,
and by Blitz. Vello proper needs compute shaders; `vello_hybrid` (CPU
path processing, GPU compositing) is "roughly beta" and runs on GLES too,
and Vello CPU exists as a no-GPU fallback.
- **cosmic-text** (iced, egui optionally): good layout, but the widgets on
top decide selection. iced's `markdown` widget is not selectable
([discourse](https://discourse.iced.rs/t/markdown-widgets-text-should-be-selectable/1107)).
- **Slint's own**: `TextInput` with `read-only` is the selectable-text
trick; there is **no inline rich text at all** (issue
[#1325](https://github.com/slint-ui/slint/issues/1325), markdown request
[#6684](https://github.com/slint-ui/slint/issues/6684) both open). A
markdown transcript with links and code chips cannot be drawn.
- **Makepad's own**: GPU/SDF text, a `Markdown` widget and a virtualised
`PortalList` in `makepad-widgets`.
## Options
### A. Keep Compose, move the logic into a Rust core (uniffi)
A `client-core` crate (events shared with the server, API client, SSE,
transcript fold, cache, markdown model, highlighter, ANSI) exposed to
Kotlin through [uniffi](https://github.com/mozilla/uniffi-rs). Compose keeps
drawing. Desktop would be a second UI (iced or Compose Desktop) over the
same core.
- **For**: the logic and the wire types stop drifting from the server
today, with tests in one language. Incremental and always shippable.
- **Against**: it is not what was asked for. The 13,000 lines of UI stay
Kotlin, the desktop app shares no UI code, and the `:link` Kotlin module
stays. uniffi's Kotlin Multiplatform bindings are a
[community fork](https://github.com/UbiqueInnovation/uniffi-kotlin-multiplatform-bindings);
the Android-only bindings are Mozilla's and solid.
- **Verdict**: not the destination, but **step one of every other option**
is building this crate, so it costs nothing to keep it as the fallback.
### B. Slint
Rust on Android is officially supported (minSdk 26, `android-activity`
backend, own Java IME glue, safe areas and keyboard insets since 1.15,
Skia renderer needs `clang`). Royalty-free licence requires disclosing
Slint use; GPLv3 otherwise. UI is a separate `.slint` DSL, not Rust.
- **Against**: no rich inline text (see above), so the transcript cannot be
drawn as it is today; the UI language is not Rust, which forfeits the
"compiler catches it" motivation for the half of the code that is UI.
- **Verdict**: rejected on rich text alone.
### C. iced
Elm-style, Rust-only widgets, desktop-first, `winit` + `wgpu`. Has a
`markdown` widget and `rich_text` with links. The maintainer states mobile
is a non-goal ([iced](https://github.com/iced-rs/iced)); a community
Android example exists and its author could not get the soft keyboard
working, patched widgets for touch, and notes no accessibility
([HN thread](https://news.ycombinator.com/item?id=46350641)). Markdown is
not selectable; `scrollable` is not virtualised.
- **Verdict**: a fine desktop toolkit and the one Iris named, but every
phone-side gap (IME, touch, accessibility, selection, virtualisation)
would be ours to build and maintain against a project that does not want
them. Not the shared framework.
### D. egui
Immediate mode, `winit`-based on Android, AccessKit integration,
selectable labels across a `Ui`. Repaints only on input by default, so
battery is not the immediate-mode worry. Android IME is blocked on winit
([discussion](https://github.com/emilk/egui/discussions/2053)); the
workaround is an in-app virtual keyboard, which is exactly the
non-native keyboard to avoid. Variable-height virtualised lists are manual
(`show_rows` assumes uniform heights). Looks like egui, not Material.
- **Verdict**: workable on desktop, wrong on the phone for the same reason
as iced, plus a look that would need a full custom style.
### E. Makepad
GPU-rendered, hybrid retained/immediate, `live_design!` DSL with hot
reload, MIT, 1.0 in 2025 ([makepad](https://github.com/makepad/makepad)).
Ships Android apps today with its own Java glue; Robrix (a Matrix chat
client, the closest analogue to this app) is its reference application on
Android, iOS and desktop. Has `Markdown`, `PortalList` (virtualised),
`TextInput`. Robrix reports the Android keyboard is not "full" and is moving
to android-view for it; the README says non-standard targets "may require
minor fixes".
- **For**: the only option that already ships a chat-shaped app on Android
and desktop from one codebase, with the widgets this app needs.
- **Against**: the DSL is its own language with its own shader-based
styling, so a large part of the UI would not be checked by rustc; the
rendering model (SDF everything) is a different world from Compose's,
and selection across a `Markdown` widget is unverified.
- **Verdict**: **rejected 2026-09-04** — Iris does not want a DSL. Kept
here so its Android keyboard status stays a data point about
android-view, not as an option.
### F. Masonry / Xilem on android-view (Linebender)
Retained widget tree (Masonry) with a reactive view layer (Xilem) that
reads like Compose; Rust all the way down; Vello, Parley, Fontique,
AccessKit, `ui-events`. Widgets include `Prose` (selectable read-only rich
text), `TextArea`, `VirtualScroll`, and this year `Svg`, `Split`,
`CollapsePanel`, a new layout system, and IME through `ui-events`
independent of winit. `masonry_android_view` exists in the android-view
repo and is "not yet generally usable"; Xilem calls itself experimental.
Desktop runs on winit. Vello needs a compute-capable GPU or falls back to
`vello_hybrid`/CPU.
- **For**: the only stack where every hard behaviour above maps onto a
component designed for it: selection and rich text (Parley/Prose),
virtualised variable heights (`VirtualScroll`), native IME
(android-view's `InputConnection`), accessibility (AccessKit, now with an
Android crate), one Rust widget language on both platforms. The team is
the one writing the Android integration everyone else is adopting.
- **Against**: pre-1.0 with API churn each release; a small team; no
Material widget set, so every control's look is ours; some of the pieces
(`masonry_android_view`, `vello_hybrid`) are explicitly unfinished. Being
early means fixing things upstream ourselves, which Iris said is
acceptable.
- **Verdict**: **the option to try first**, because it is the only one
whose gaps are "not finished yet" rather than "not designed for this".
### G. iris — the in-house library, and what "from scratch" means here
[cat16/iris](https://github.com/cat16/iris), read 2026-09-04 from the one
public commit (2026-01-31, "portfolio copy"; ~8,700 lines in `core`,
`macro` and the crate itself). Retained-mode widgets stored outside the
render tree, `wgpu` 28 directly, `winit` 0.30, `cosmic-text` 0.16 (parley
since I1), a
relative-anchor-plus-offset layout with `rest()` and `rel()` lengths, a
postfix builder API (`rect(..).radius(30).on(CursorSense::click(), ..)
.sized(..).align(..)`), events handled where the widget is declared, and
a single-threaded context passed explicitly — all of which reads like this
codebase's own rules. There is text editing (`widget/text/edit.rs`),
images, masks, spans and stacks; the TODO names text resizing as
per-frame slow and scaling as unsolved. It requires **nightly** (fourteen
`#![feature]` gates as vendored, among them `const_trait_impl`,
`unboxed_closures`, `portable_simd`, `associated_type_defaults`; eleven
after I0b and I1 — see those steps for the current list). Desktop only; no
Android surface, no IME, no accessibility tree, no virtualised list, no
rich-text selection.
**That list is the work, and some of it is done.** As of 2026-09-04 it
builds on a pinned nightly, runs on this machine's GPU, has parley and a
glyph atlas, and `iris-core` cross-compiles to Android. What it still
lacks from the list above is the Android surface, the IME bridge, the
accessibility tree and the virtualised list — I2, I3 and I4.
**iris is not a candidate to be tested as it stands. It is the in-house
library** (Iris, 2026-09-04): "essentially a good start to a rewrite from
scratch", to be maintained and extended by the sessions working here.
So the list above of what it lacks is a **work list, not a score**. When
the app needs something iris does not have, the answer is to build it
into iris. The layer iris has is the widget and layout layer; the layers
it needs are the same ones Masonry gets from android-view, Parley and
AccessKit, and there is no reason iris cannot sit on those same
foundations rather than reinvent them — the surface, the keyboard bridge
and the accessibility tree are platform plumbing, not a framework's
identity. The text stack was the first real design decision in that work,
and it is settled: **Parley, with a glyph atlas** (I1, 2026-09-04).
Two things to carry into that work honestly. Nightly is the opposite of
"holds up long term": a build that breaks on a toolchain update, on the
machine Dev Updater builds on, unattended. **Done in I0b** —
`iris/rust-toolchain.toml` pins `nightly-2026-09-03` — and the gate list
lives with I0b and I1, to be retired as they stabilise or are designed
around; it is down from fourteen to eleven. And a one-person framework
carries every gap itself, which is what Iris said she is willing to do.
"From scratch" therefore means iris, not a fourth thing. Masonry stays in
the plan as the **yardstick and the fallback**: building its demo and its
version of the transcript screen first says what a finished stack costs
on this hardware, proves android-view before iris depends on it, and
gives a comparison that is measured rather than remembered.
Not considered further: **GPUI** (Zed) mobile is a community fork that
depends on unpublished crates; **Dioxus/Blitz** is excluded by Iris (its
native renderer is Parley/Vello under HTML semantics, and the earlier
`tdep-survey/app-dioxus` spike parked it on a `vello_hybrid` stroke bug and
shipped the WebView); **Compose Multiplatform Desktop** would give a desktop
app for nothing but in Kotlin, which is the opposite direction.
### Weight and debug builds
Iris remembers the Linebender stack being slow in debug. What is behind
that is the dependency graph — Vello, wgpu, Parley, Fontique, Skrifa —
running unoptimised on the CPU side (path encoding, shaping), not the
widget layer. Xilem's own advice is only `split-debuginfo = "unpacked"` to
keep `target/` small; the fix everyone with this shape of dependency tree
uses is to optimise dependencies while leaving the app crate at `opt-level
= 0`:
[profile.dev.package."*"]
opt-level = 2
**Measured 2026-09-04, and it is not the widget layer.** iris's own
graph (wgpu + winit + cosmic-text at the time) built cold in **43s** with
a 2.1 GB `target/`, and **1m46s** with a 1.5 GB `target/` under the
profile above — so the knob costs build time and saves disk here, and
plain debug was never the problem. Masonry's graph is the one with Vello,
Parley, Fontique and Skrifa in it, and E1 gives the number that matters
for it: **`libmain.so` is 181 MB in debug and 11 MB in release.** That
size is also a correctness issue rather than only a weight one — a debug
build labels its Vulkan objects, and the emulator's driver segfaults in
`SetDebugUtilsObjectNameEXT` when it does. Runtime cost of the profile
knob is still unmeasured; resident memory and the frame cost of an
800-event page want E2. Vello proper needs compute shaders and carries a
large shader set; `vello_hybrid` is lighter and Masonry can now render
through either (or Vello CPU) via its `imaging` abstraction, so "keep it
light" has a knob inside the same stack.
## Recommendation
1. **Build `client-core` now, whatever the framework** (done 2026-09-04, see `CLIENT_CORE.md`). A Rust crate holding
the event model (shared with `server/` as one crate, ending the
`Events.kt` mirror), the API and SSE clients, the transcript fold, the
cache, the markdown block model, the highlighter and the ANSI parser,
with the existing JVM tests ported. It is the part of the app that is
already tested, already logic, and already duplicated on the server.
2. **One foundation, two widget layers.** The platform plumbing is shared
whichever way the decision goes: android-view for the Android surface,
keyboard and accessibility bridge; `wgpu` for the GPU; AccessKit for
names; winit on the desktop. On top of it, **Masonry as the yardstick**
(E1, E2) and **iris as the thing being built** (I0–I5), both aimed at
the same transcript screen with the same pass conditions.
3. **Decide when the transcript screen exists in both**, from the
measurements, and record the decision here with the numbers. If iris
carries the screen within the Compose baseline, it is the app's
framework and Masonry was the calibration. If it does not, the
measurement says which parts of Masonry to adopt underneath it.
**Not decidable yet, 2026-09-05 — what's missing, named rather than
guessed at.** Neither side of this comparison has a render number:
E2 found Masonry's own scroll gesture path absent on Android
entirely (its box, "measurable frames"), and I5 built the iris side of
the screen (`iris/transcript-ui/`) but not the Android integration
around it — no cdylib/Gradle shell exists for this screen yet (unlike
`tabs-ui`'s `iris-android-app`, I2), so there is nothing installed on a
device for `transcript-bench.sh` to measure against the Compose
baseline. What would close this: build that integration (real
`client-core` networking against `app/ui-sandbox.sh --delay`, a cdylib
+ Gradle module the way I2 did for `tabs-ui`), then run
`transcript-bench.sh`'s gesture on both. Until then, the decision rests
on the structural findings both sides *did* produce: Masonry cannot do
cross-row selection or per-span inline rich text at all today (E2's
`grep -rln`, zero hits, cited in its own box), and iris now does both
(I5's `SpanStyle` and `selection.rs`) as well as programmatic
touch-scroll (I3) — three structural points in iris's favour with no
opposing measurement yet on either side.
4. Then the shell (E3), the desktop window (E4) and the packaging (E5),
which do not depend on the choice.
## Experiments, in order
Each has a pass condition that is a measurement in this clone. The rig
matters: this emulator runs `-gpu host` with **host Vulkan switched off**
(`GPU_HOST_FEATURES` in `emulator-tools`, a gfxstream/Venus gap), so inside
the guest a `wgpu` app gets GLES, not Vulkan; the earlier Dioxus spike also
needed `WGPU_GLES_MINOR_VERSION=1` for compute shaders and found `wgpu`'s
Android backend wants API 26 (a libc symbol). The real phone has Vulkan.
Per the standing rule, a rig limit is something to fix before it is
accepted.
- [x] **E0 — toolchain (done 2026-09-04).** Installed under the
user-owned SDK: **NDK r29 (`29.0.14206865`)**, 2.4 GB at
`~/Android/Sdk/ndk/29.0.14206865`, the newest stable — r30 is still
at rc.3. **cargo-ndk 4.1.2**. Verified by cross-compiling a scratch
`cdylib` to both ABIs: `file` reports "for Android 26, built by NDK
r29 (14206865)" for `aarch64-linux-android` and
`x86_64-linux-android`. Two things to know at the call site.
**cargo-ndk 4's API-level flag is `-P`, not `-p`** — `-p` is now
passed through to cargo as `--package`, so the old
`cargo ndk -t arm64-v8a -p 26` panics with `unknown package: 26`
*and dumps the whole environment to stdout* as a bug report, which is
worth not doing in a log somebody might paste. And the Android
targets were installed for **stable** only; the pinned nightly needs
its own, which `iris/rust-toolchain.toml` now declares.
- [x] **E1 — android-view's Masonry demo on this emulator (2026-09-04).**
It builds, renders on the GPU through Vulkan, exposes its
accessibility tree, and **the phone's own keyboard types into its
editor** — but with **no autocorrect and no suggestions**. Ticked
because everything it was meant to establish is established,
including the one gap; that gap is now E2's problem and I2's.
*Build.* `~/src/android-view` at `bec6c62`, x86_64 rather than the
README's arm64 because that is what this emulator is:
`cargo ndk -t x86_64 -P 26 -o masonry-app/src/main/jniLibs/ build -p
android-view-masonry-demo --release`, then
`./gradlew :masonry-app:assembleDebug`. **`libmain.so` is 181 MB in
debug and 11 MB in release** — the loudest single number about
Vello's dependency graph, and the reason the release build matters
for more than speed.
*Renderer.* wgpu takes **Vulkan**, and the emulator log confirms it
from the other side: `Created VkDevice ... for application:'wgpu'`.
Two things were needed. The emulator must be given Vulkan at all —
`-feature Vulkan` with `VK_DRIVER_FILES` pointing at the SDK's
`vk_swiftshader_icd.json`, **plus `-no-snapshot-load`**, which is the
piece this file had flagged as untested: without a cold boot the
guest keeps the snapshot's old GPU config and `cmd gpu vkjson`
reports zero devices however the host is configured. And the native
library must be **release**: a debug build calls
`SetDebugUtilsObjectNameEXT` to label its image views, and the
emulator's own guest driver (`vulkan.ranchu.so`) segfaults inside it.
On GLES, with no Vulkan available, it instead fails
`Surface::configure` with "Invalid surface" — untriaged, since the
Vulkan path works and Vello wants compute shaders anyway.
*Accessibility works*, which E2's condition 6 and every bench script
depend on. `ui-trace` reads Masonry's AccessKit tree: "Add task"
arrives as a named `Button`, the editor as an `EditText` node. So
tap-by-name works against a Masonry screen for any control carrying
a name; the demo's editor carries none, which is the demo's omission
rather than the framework's.
*The keyboard: real input yes, suggestions no.* Tapping the editor
opens the actual soft keyboard (`mInputShown=true`, Gboard), and
tapping its keys types into Masonry — "teh" typed key by key, with a
caret. What does **not** appear is Gboard's suggestion strip. The
control is what makes that a finding rather than an impression: the
**same three key taps in the Settings app's search field, on the same
device in the same session, produce "teh | the | yeh"**. So the strip
works here and android-view's editor is not asking for it — most
likely the `EditorInfo` its `InputConnection` reports. That matches
Robrix's report that the Android keyboard is not yet "full", and it
is the single most important thing to fix or fund upstream, because
composition, autocorrect and suggestions are exactly what the
composer in this app needs and exactly what `winit` cannot do at all.
**Cause found 2026-09-04, and it is Masonry's, not android-view's.**
`~/src/android-view/masonry/src/lib.rs:531` is
fn as_input_connection(&mut self) -> Option<&mut dyn InputConnection> {
// TODO
None
}
so the Masonry demo has **no `InputConnection` at all**; `RustView`
returns null from `onCreateInputConnection` and the IME falls back to
dispatching raw key events, which is exactly the behaviour observed —
keys arrive, composition does not exist, so there is nothing for
Gboard to suggest against. It is not a wrong `EditorInfo`, and the
guess above that it was is withdrawn.
android-view's **own** demo (`demo/src/lib.rs`, packaged by `app/`)
implements the whole trait against a parley editor and asks for
`INPUT_TYPE_CLASS_TEXT | CAP_SENTENCES | AUTO_CORRECT | MULTI_LINE`
with `IME_FLAG_NO_FULLSCREEN | NO_EXTRACT_UI | NO_ENTER_ACTION`
(`demo/src/lib.rs:588`). So the capability is present in the layer
iris would sit on, and the 30-odd method `InputConnection` trait in
`src/ime.rs` — `set_composing_text`, `set_composing_region`,
`finish_composing_text`, `text_before_cursor`, `cursor_caps_mode`,
`request_cursor_updates`, and `InputMethodManager::update_selection`
to push the selection back — is the full surface an IME needs.
**This changes what I2 costs**: the IME bridge is a trait to
implement over iris's parley editor, not a gap to fund upstream. It
also means E2 inherits Masonry's TODO, so a Masonry transcript will
have the same dead composer until somebody fills that in.
**Measured on the emulator, same session, same device.** Built
android-view's own demo — `cargo ndk -t x86_64 -P 26 -o
app/src/main/jniLibs/ build -p android-view-demo --release`, then
`./gradlew :app:assembleDebug`, installed with `ANDROID_SERIAL=$(emu
serial)` — and tapped into its editor. `dumpsys input_method` reports
`mInputShown=true` with `mServedView=…viewdemo.DemoView`, and the
screenshot shows **Gboard's suggestion strip populated with "dolor |
Dolores | door"**: the caret had landed inside the word *dolor* in
the demo's lorem ipsum, and Gboard read that word out of the Rust
editor through `text_before_cursor`. So on this emulator, through
android-view, a parley editor gets a real IME with real suggestions
drawn from its own buffer. That is the bar E1 could not reach and the
bar I2 is written against, and it is now known to be reachable.
*One crash seen once — reproduced and diagnosed 2026-09-04.* With an
accessibility client attached the app aborted, stack:
`android_view::view::do_frame` → `CallbackCtx::finish` →
`accesskit_android::event::QueuedEvents::raise` →
`send_completed_event` → `unwrap()` on `Err(JavaException)`.
android-view builds `panic = "abort"`, so a JNI call that throws
takes the process. Two later `ui-trace record` runs left the app
alive, so the trigger looked narrower than "a client is attached".
**It is the opposite of "a client is attached": it is a client
having *detached*.** `accesskit_android`'s `State` enum
(`adapter.rs:161` in 0.4.0, `:192` in 0.8.0) is
`Inactive | Placeholder | Active`, and **nothing ever moves it back
to `Inactive`**. A client — `ui-trace`, which is uiautomator — calls
into the node provider once, `get_or_init_tree` promotes the adapter
to `Active`, and it stays there for the life of the process. Every
later change then returns `Some(QueuedEvents)`, `raise` calls
`ViewParent.requestSendAccessibilityEvent`, and that reaches
`AccessibilityManager.sendAccessibilityEvent`, which on the main
looper **throws `IllegalStateException("Accessibility off. Did you
forget to check that?")` when accessibility is disabled**. jni-rs
returns `Err(JavaException)`, `send_completed_event` unwraps it, and
`panic = "abort"` ends the process.
*The controlled run*, one process (pid 4085), `settings get secure
accessibility_enabled` = 0 throughout:
- tapped the editor and typed three keys with `adb shell input tap`,
no client ever attached — **alive**;
- one `ui-trace record -d 800` with no gesture at all, then two
seconds' wait — **still alive** (the queue was raised while the
client was still there);
- the very next three keystrokes, same process — **aborted**, same
stack.
So the failure is not the recording; it is the **first thing that
changes the accessibility tree after a recording ends**. That makes
it a standing hazard for this project rather than an oddity:
`transcript-bench.sh`, `stream-bench.sh` and `bench-lib.sh`'s
tap-by-name all attach and detach uiautomator, so on a Rust app the
typing or scrolling *after* a bench run is what dies, several
seconds away from anything that looks like a cause.
**Still present at head**: 0.8.0 is the newest `accesskit_android`
(the demo resolves 0.4.0) and both the unconditional `unwrap` in
`send_completed_event` and the one-way `State` are unchanged there,
so upgrading is not the fix. **Our mitigation for I2/I4 is a gate we
own**: ask `AccessibilityManager.isEnabled()` before calling
`raise`, and drop the events when it says no. Worth reporting
upstream as well — the honest fix is for `raise` to clear a pending
exception rather than unwrap it, since a view can be detached or
accessibility switched off between queueing and raising no matter
who is calling.
*A rig trap that cost a wrong conclusion.* Several bounded runs were
given `sleep N; emu down` watchdogs, and one armed for an earlier
experiment fired in the middle of a later one — the app vanished, adb
hung, and it read exactly like the Vulkan path crashing. It was not.
A watchdog must be scoped to the process it guards (`kill $pid`, with
the pid captured at launch) rather than to whatever AVD is running
when it wakes, and only one should be armed at a time.
- [x] **E2 — a transcript in Masonry (2026-09-05).** Built and run on this
emulator. It found the thing it was measuring for: a framework-wide
gap that blocks the bench comparison itself, plus a full accounting
of the seven behaviours. Ticked on E1's own precedent -- "everything
it was meant to establish is established, including the one gap."
*Where it lives.* `~/src/android-view/e2-transcript` (new workspace
member, `crate-type = ["cdylib"]`, `lib.name = "main"`), packaged by
a new Gradle module `~/src/android-view/e2-app` copied from
`masonry-app` (`E2View`/`E2Activity`, package
`org.linebender.android.e2transcript`). Neither is committed to
`ai-app-2` or pushed anywhere -- same as E1, this is a local
experiment against the `xilem` commit
`e14ba3a5f9461b403cb30d95826187fba7f6924b` and the `android-view`
commit `bec6c62a96cef8239b0fd7fedeef9b184d02e3a1`, reproducible from
the commands below rather than from a remote.
*Build.* Depends on `client-core`/`event-model` from this checkout by
path (`../../../repos/ai-app-2/client-core`) -- real code, not a
reimplementation: `ApiClient`/`UreqTransport` for the HTTP fetch,
`fold_event`/`group_tool_runs` for the transcript fold, exactly what
the app itself would use. The sandbox CA and a session's URL/token
are baked in at build time via `env!()`/`include_bytes!()`, the same
pattern the real APK uses to pin its CA (AGENTS.md), since this is a
throwaway screen with no enrollment flow:
cd app && ./ui-sandbox.sh start # prints the port and token
sid=$(./ui-sandbox.sh spawn e2test)
./ui-sandbox.sh send "$sid" @/tmp/big.md # markdown content
./ui-sandbox.sh send "$sid" "/tools 3" # a grouped tool run
cd ~/src/android-view
E2_SANDBOX_URL=https://10.0.2.2:<port> \
E2_SANDBOX_TOKEN=<token> \
E2_SANDBOX_SESSION=<sid> \
E2_CA_PEM_PATH=$HOME/.config/ai-app/certs/ca.pem \
cargo ndk -t x86_64 -P 26 -o e2-app/src/main/jniLibs/ \
build -p e2-transcript --release
ANDROID_HOME=~/Android/Sdk ./gradlew :e2-app:assembleDebug
**`libmain.so` is 13.5 MB release** (E1's masonry-demo was 11 MB;
the difference is `client-core`'s `ureq`/`rustls` stack, which E1's
demo does not link). Release native lib, debug Gradle variant --
the combination E1 found necessary (a debug build's
`SetDebugUtilsObjectNameEXT` segfaults this emulator's Vulkan
driver).
*Emulator.* This checkout's own AVD (`ai-app-2`, not `ai-app`, which
another session already had up), booted with Vulkan the way E1
established: `GPU_HOST_FEATURES="-feature Vulkan"
VK_DRIVER_FILES=$HOME/Android/Sdk/emulator/lib64/vulkan/vk_swiftshader_icd.json
emu up`. `adb shell cmd gpu vkjson` confirmed a device before
anything was installed. Torn down with `emu down` at the end of this
session (see "Where things stand" below for the exact state left).
*What it does.* `fetch_rows()` (`e2-transcript/src/lib.rs`) makes one
blocking `fetch_transcript_page(session, None, 800, false)` call
before the widget tree exists, folds every line through
`client-core`, and groups tool runs -- 854 real events from a mixed
sandbox session (markdown paragraphs/headings/fences plus a
three-call tool run from the echo driver's `/tools 3`). Each
`TranscriptRow` becomes one `VirtualScroll<dyn Widget>` child,
built lazily from `VirtualScrollAction` the way
`masonry_winit/examples/virtual_fizzbuzz.rs` does it. **This is a
deliberate scope cut from "page 800 events" as live paging**: all
854 rows' content is fetched once, and what `VirtualScroll` pages is
*widget construction*, not a second round of network calls per
scroll -- wiring a background-thread fetch woken across the JNI
boundary (the way I2's `ssh.rs` attach-and-call works) is real work
this experiment did not need to answer its question. `markdown.rs`
is a `pulldown-cmark` event-stream walk into a small `Block` enum
(`Text`/`Heading`/`Code`), with its own module doc explaining the
one real ceiling it hit (below).
*Verification.* `cargo fmt -p e2-transcript -- --check` clean.
`cargo ndk -t x86_64 -P 26 clippy -p e2-transcript --all-targets`:
**zero warnings in this crate** (the only clippy output at all is
from `android-view` itself, a vendored dependency this experiment
does not own). `cargo ndk -t x86_64 -P 26 test -p e2-transcript
--lib` (run against the emulator, since the crate is
`cfg`-unconditionally Android): 1 test, `markdown::parse`'s block
split, passing. No larger test surface exists to port -- this is a
throwaway screen, not a library, matching AGENTS.md's "match the
codebase's testing posture."
*Screenshots* (all `/tmp`, not committed -- see the standing rule
against transcripts leaving this repo, which applies equally to a
screenshot of one): `e2-screenshot2.png` first real content;
`e2-expand.png` a tool row expanded with its top edge held;
`e2-markdown.png` a heading/bold/italic/inline-code/link/fenced-code
message (the "You said: ## A Heading" line is the sandbox's echo
driver prefixing the literal input text before the `##`, which
keeps `pulldown-cmark` from recognising it as a heading -- a fixture
artifact, not a finding about Masonry).
**The seven behaviours, each shown or given a sourced reason:**
1. **One selectable body of text spanning rows -- not possible, and
it is a real ceiling, not an oversight.** `Prose` wraps exactly
one `TextArea<false>`, which wraps exactly one
`parley::PlainEditor` (`masonry/src/widgets/prose.rs`: "Note that
copying is not yet implemented"). Selection lives entirely inside
that one editor: `TextArea::on_pointer_event`
(`masonry/src/widgets/text_area.rs:414-459` in the pinned `xilem`
commit) captures the pointer on `Down`
(`ctx.capture_pointer()`) and drives `self.editor`'s own
`extend_selection_to_point` on `Move` -- there is no code path,
in `masonry_core` or `masonry`, that extends a selection into a
second widget's editor. A drag that starts in one row's `Prose`
and continues into the next is still that first row's own
`PlainEditor` being asked for a point outside its bounds; it
cannot reach the second row's text. Confirmed by reading, not
guessed at: there is no `SelectionContainer`-shaped type
anywhere in `masonry`, `masonry_core` or `xilem` (checked with
`grep -rln "SelectionContainer\|cross.widget.*selection"`, zero
hits).
2. **Rich inline text -- block-level yes, inline no, and both for
the same reason.** `TextArea::edit_styles()` returns one
`&mut StyleSet<T>` for the whole editor
(`masonry_core/src/core/text.rs:29-32` defines `StyleSet` as
`parley::StyleSet<BrushIndex>`, applied editor-wide); the type's
own comments say why nothing finer exists yet:
`// TODO: RichTextInput 👀` and
`// TODO: Support for links - https://github.com/linebender/xilem/issues/360`
at `masonry/src/widgets/text_area.rs:43-44`. So bold, italic,
inline code and a link *inside one paragraph* cannot each carry
their own style without leaving `TextArea` for a hand-rolled
`parley::Layout` (which loses selection, the caret and copy,
since those live inside `PlainEditor` specifically). What **is**
real: each markdown block is its own `Prose`, so a heading is a
bigger font and a fenced code block is monospace, screenshotted
in `e2-markdown.png` -- block-level style works because it is
block-level *widgets*, not a rich-text API. Tables and per-token
syntax colour inside a fence hit the identical ceiling (both are
per-range styling) and were not attempted for the same reason.
`markdown.rs`'s degraded rendering (backticks kept literally,
`[text](url)` shown as `text (url)`) is the honest fallback,
documented at the point it is produced.
3. **Bottom-anchored virtualised list, paged, hold-top-edge on
expand -- mostly shown, with one real gap in the anchor API.**
`VirtualScroll<dyn Widget>` holds all 854 folded rows;
`overwrite_anchor` before swapping a tool row's widget for its
expanded/collapsed version is exactly the primitive
`holdTopEdge` needs, and it worked: `e2-expand.png` shows the
row growing downward from the same top edge it had collapsed,
no jump. Virtualisation is real (`ui-trace elements` only ever
lists the rows currently on screen, never all 854). **What did
not come free: hugging the bottom of the screen.**
`VirtualScroll::new`'s doc says "the item at `initial_anchor`
will have its top aligned with the top of the scroll area" --
so anchoring on the last row puts that row's top at the
viewport's *top*, with empty space below it, not at the
viewport's bottom the way a chat transcript wants (visible in
`e2-screenshot2.png`). The complete public `WidgetMut` surface of
`VirtualScroll` is `new`, `with_valid_range`,
`will_handle_action`, `add_child`, `remove_child`, `child_mut`,
`set_valid_range`, `overwrite_anchor`
(`masonry/src/widgets/virtual_scroll.rs:257-428`) -- no
scroll-offset setter and no reverse/bottom-up layout mode exist
to ask for the other behaviour. Backward paging beyond the
initial 800 was not exercised, per the scope cut above.
4. **The soft keyboard -- inherited gap, not re-investigated.** E2's
screen has no `TextInput`, only read-only `Prose`/`Button`, so it
does not hit `masonry/src/lib.rs:531`'s `as_input_connection`
returning `None` directly -- but it would the moment a composer
is added, per E1's finding. Nothing new to add here.
5. **Platform integration -- out of scope by design.** Foreground
service, notifications, share sheet, deep link, Keystore,
camera, back gesture, edge-to-edge, local-network permission are
E3's list in RUST.md's own experiment order, not E2's.
6. **Accessibility names -- shown, and the bench-script dependency
actually exercised.** `ui-trace record --do "tap '> 3 tool
calls'"` found the button by its label and pressed it (that tap
is what produced `e2-expand.png`); `Prose` rows surface their
text as their accessible name too (`ui-trace elements` lists
"You said: One more short reply..." etc. as named nodes). Tap by
name, the rule this whole project's bench scripts depend on,
works against this screen.
7. **Measurable frames -- blocked, and this is the finding E2 was
really testing for.** Two separate problems, one of them fatal
to the render-numbers half of this box's own pass condition.
First, Masonry has no render-report/per-widget-cost
instrumentation the way Compose's `DebugStats` gives this
project -- building one was out of scope here. Second, and this
is the one that matters: **neither of Masonry's two scrolling
widgets responds to a touch drag at all.**
`VirtualScroll::on_pointer_event`
(`masonry/src/widgets/virtual_scroll.rs:504-523`) and
`Portal::on_pointer_event`
(`masonry/src/widgets/portal.rs:259-267`) both match only
`PointerEvent::Scroll` (wheel/trackpad deltas) and do nothing
with `PointerEvent::Down`/`Move`/`Up` -- there is no drag-to-scroll
gesture logic anywhere in the widget set. `android-view`'s own
Java bridge keeps the two paths separate at the source:
`RustView.java`'s `onTouchEvent` forwards raw touch straight to
Rust, and only `onGenericMotionEvent` (mouse/trackpad, not
touch) reaches the `ACTION_SCROLL` branch that becomes
`PointerEvent::Scroll`
(`android-view/src/events.rs:530`). Confirmed empirically, not
just by reading: a real swipe (`ui-trace`'s `swipe 540 1600 540
400 300`, twice) moved nothing (`e2-scroll.png` is pixel-identical
to the screen before it), and a synthetic Android wheel event
(`adb shell input scroll 540 1200 --axis VSCROLL,-5`) also moved
nothing. **This means `transcript-bench.sh`'s own gesture --
a finger swipe -- cannot scroll a Masonry transcript on Android
today, at all, on this framework commit.** So the "render
numbers land within the Compose baseline" half of this box's
pass condition cannot be attempted, let alone met: there is no
way to perform the scroll the comparison asks for. This is not
a performance shortfall to close by writing faster code: it is
an absent input path upstream. The fix is a drag-to-scroll
gesture in `on_pointer_event` (the same place `TextArea`'s own
caret-drag logic already lives, so the pattern -- capture on
`Down`, accumulate delta on `Move`, release on `Up` -- exists
in this codebase already, just not wired into either scrolling
widget), and it belongs upstream in `xilem` rather than in this
project.
**Net for RUST.md's recommendation.** Item 3 ("decide when the
transcript screen exists in both, from the measurements") cannot be
decided by a render-number comparison yet, because the comparison's
own gesture does not work on Masonry on Android. What *can* be
compared today is structural: iris already has a working scroll
gesture and a working touch model (I2, 2026-09-05) that Masonry's
upstream commit does not yet have for this exact case. That is a
point in iris's favour that a frame-time number would not have
shown any more clearly.
- [x] **E3 — the shell (2026-09-05).** Both pass-condition proofs held on
the emulator: a notification arrived while the app was closed, and a
shared text share landed as a real message in a session's transcript.
Committed to this repo (unlike E1/E2's external, uncommitted trees),
since this is lightweight glue rather than a multi-gigabyte native
build.
*Where it lives.* `android-shell/` (new crate, `client-core` as its
only real dependency) is the JNI bridge; `app/shellApp/` is a **new
Gradle module**, not a rewrite of `app/androidApp` in place --
that module is ~13,000 lines of working Compose UI this experiment
does not touch or risk, and the two install side by side on one
development device. `app/shellApp`'s manifest, channel names,
notification wording and share intent-filter are copied from
`androidApp`'s (`Notifications.kt`, `Share.kt`, the manifest) per
AGENTS.md's "reuse rather than re-derive" -- see each file's own doc
comment for exactly what was carried over. Two deliberate
differences, both practical rather than behavioural: application id
`com.example.aiapp.shell` and deep-link scheme `aiappshell` (not
`aiapp`), so this experiment's install cannot collide with the real
app's enrollment or Keystore alias on the same phone -- see
`android-shell/src/settings.rs`'s `SCHEME` doc.
*The Java floor, and one line more than planned.* Two classes,
matching "How much Java is unavoidable" almost exactly:
`MainActivity.java` (`onCreate`/`onNewIntent` forward to
`nativeHandleIntent`) and `NotificationService.java`
(`onStartCommand`/`onDestroy`/a `sync()` companion, three natives).
Both ~30 lines including the license-free boilerplate Java itself
demands (imports, `System.loadLibrary`). **One addition the analysis
did not anticipate**: `MainActivity.toast(Context, String)`, a
plain (non-native) static method Rust *calls* rather than
implements, because posting a `Toast` from `share.rs`'s background
thread needs a hop back to the main looper
(`new Handler(Looper.getMainLooper()).post(...)`), and JNI can call
an existing Java method on any thread but cannot construct a Java
`Runnable` to hand to `Handler.post`/`runOnUiThread` without a
reflection proxy uglier than three lines of Java. Recorded here
because "the floor is two classes of ten lines" undersold this by
exactly one small, call-only method -- the pattern (Rust calls
Java, never Rust implements a Java interface) is worth keeping the
next time this floor is estimated.
*What client-core gained.* `notifications.rs`: `SessionNotification`,
`NotificationKind` (mirroring `server/src/session/mod.rs`'s wire
shape field-for-field) and `follow_notifications`, the SSE parse
over `/notifications` built on the same `sse::SseReader` and
`Transport` trait `event_stream.rs` already uses. `attention_line`
is ported verbatim from `Notifications.kt`. 3 new tests (88 total in
the crate); `android-shell` itself has none, since every function in
it needs a live `Env` and there is no pure logic left to test in
isolation once client-core owns the parsing -- matches E2's
precedent ("a throwaway screen, not a library").
*Scope cuts, each recorded at its own point in the code rather than
only here:*
- **Text-only share.** `Intent.EXTRA_TEXT` becomes a session message;
a shared file/photo URI is not uploaded, because `client-core`'s
`ApiClient` has no `/sessions/{id}/attachments` route yet either
(`CLIENT_CORE.md`'s own "not covered" list) -- porting
`Attachments.kt`'s `ContentResolver` reads and bitmap downscaling
is real work belonging to whichever caller needs it next, not a
detour inside this box.
- **No session picker.** With no screen drawn yet (E4's job), a
share attaches to whichever session has the latest
`last_activity` -- documented as a placeholder in `share.rs`,
not a designed behaviour.
- **No banner/on-screen suppression.** `notify::show` skips
`Notifications.kt`'s "nothing if this session is on screen" /
"hand to the app as a banner" branches entirely: both read
process-wide state that only means something once a screen
exists to register against it, so every notification here takes
the platform-drawer branch -- which is also exactly what the pass
condition asks for. Revisit once E4 draws something.
- **Keystore is not reimplemented in Rust.** `settings.rs` calls
`wg-app-link`'s existing `ServerStore`/`ServerSettings` Kotlin
classes over JNI rather than re-deriving the AES-GCM sealing:
that code is shared with Dev Updater, already tested, and tied to
a Keystore alias an existing enrolled phone depends on. This does
mean `kotlinc` stays in the toolchain regardless of what E5 does
with `javac`/`d8` for this module's own two classes -- a
correction to "Can the APK be built without Gradle?"'s assumption
that dropping Kotlin drops `kotlinc` outright; it drops it for
*this app's own code*, not for a shared submodule pulled in as a
dependency.
*`jni` 0.22, not the older API most examples assume.* This is a
real API split (`Env` for real work, `EnvUnowned` as the FFI-safe
type a native fn receives, joined by `EnvUnowned::with_env`), and
the `native_method!` macro (used for all four natives here, via
`const _: NativeMethod = native_method! { ... }`) generates both the
mangled `Java_...` export and the panic/error-handling wrapper from
one Rust function signature -- chosen over hand-written
`#[unsafe(no_mangle)] extern "system" fn Java_com_..._method` because
a hand-typed export name and a hand-typed JNI signature string
routinely drift from the Java they claim to match, silently (see
the next two findings, both of which were exactly that drift).
`error_policy = LogErrorAndDefault` reports a failure to logcat
rather than throwing it back into Java as an exception that would
crash the app over something recoverable -- matching
`Notifications.kt`'s own "log, don't crash" posture, but it is a
no-op without a logger backend (`android_logger`, Android-only
dependency, `lib.rs`'s `ensure_logger`) installed; the class of bug
this exists to report was found once with no logger and read as
nothing having gone wrong at all.
**Three real findings, each cost a failed run before being
diagnosed, each written where the fix lives so a reader who touches
that file again does not lose an afternoon to it:**
1. **A generic `JObject` parameter type silently exports the wrong
JNI signature.** `native_method!`'s shorthand
`fn native_sync(context: JObject) -> ()` encodes the export as
`(Ljava/lang/Object;)V`, because it has no way to know the
intended Java type is `android.content.Context` from a bare
`JObject`. The real Java method is declared
`(Landroid/content/Context;)V`; the two mangled names never
resolve to each other, and the failure is
`UnsatisfiedLinkError: No implementation found`, thrown the
moment Java calls it -- not a build error on either side. Fixed
by spelling each parameter as its actual Java type in the macro
invocation (`context: android.content.Context`, `activity:
android.app.Activity`, ...), which the macro accepts directly
per its "Java Object Types" syntax, while the Rust implementation
function keeps the parameter as plain `JObject` (the "Built-in
Types" fallback for a Java class with no dedicated Rust
wrapper). `lib.rs`'s comment beside the first `native_method!`
call is the citation.
2. **A class looked up by name from this crate's own background
thread fails, and only for app classes.** `android-shell`'s
follow-loop and share threads are Rust-spawned and attached via
`JavaVM::attach_current_thread`, which the platform never handed
an app `ClassLoader` -- so `FindClass`'s default fallback (used
internally by `find_class`/`new_object`/`call_static_method`/
`get_static_field`, anything that resolves a class *by name*
rather than from an object it already holds) only reaches the
bootstrap loader's framework classes. `androidx.core.app.
NotificationManagerCompat`, packaged inside this app's own APK,
is invisible from there: `Error::NoClassDefFound`, logged by
`notify::show`'s `LogErrorAndDefault` as "failed to resolve Java
class ... (class not found or linkage error)" -- which on a real
device is indistinguishable from "the notification silently
never arrives," since the *ongoing* foreground notification
(built on the main thread, before this thread exists) posts
fine regardless, so nothing else looks wrong. Fixed in
`jcall.rs`: `remember_class_loader` caches the app's own
`ClassLoader` (`context.getClass().getClassLoader()`) the first
time any entry point with a `Context` runs, and every
class-by-name lookup goes through `LoaderContext::Loader`
explicitly rather than the thread-dependent default -- correct
on the main thread and this crate's background threads alike.
`jcall.rs`'s module doc has the full account.
3. **`onStartCommand` spawning a thread unconditionally opens a
second connection, and `Notifications.kt` has the same bug.**
Enrolling calls `sync()` twice in one launch (once
unconditionally in `MainActivity.onCreate`, again inside
`handle_enrollment` after saving the token), each of which starts
the service, and Android runs `onStartCommand` once per start
request -- so the follow-loop thread was spawned twice, caught on
`adb logcat` as two `jni::vm::java_vm: Attached thread
ai-app-notifications` lines for one enrollment. Kotlin's
`onStartCommand` has the identical shape (`thread(isDaemon =
true) { follow(settings) }`, no guard), so this is a latent bug
in the reference implementation this port found by testing
rather than something E3 introduced -- worth carrying the same
guard back to `Notifications.kt` separately, not done here.
Fixed in `notify.rs` with a `RUNNING` `AtomicBool`, `swap`ped
true before spawning and reset in `on_destroy`; see its doc
comment for the accepted race this shares with the pre-existing
`STOPPING` gap below.
**Known gap, not fixed, written where it will be found.**
`notify.rs`'s `STOPPING` flag (checked between reconnects) cannot
interrupt a `ureq` read already blocked inside one connection --
unlike `HttpURLConnection.disconnect()`, `client_core::Transport`
exposes no cancellation handle. `/notifications` is idle between
events (a keep-alive), so in practice a stop is a bounded wait
rather than a hang; closing this for real means adding a
cancellation point to the `Transport` trait itself, a decision
affecting every caller, not an `android-shell`-only fix.
*Verification, exact commands.* `cargo fmt -- --check`,
`cargo clippy --all-targets` (zero warnings) and `cargo build`
clean for both `client-core` and `android-shell` on the host
target; `cargo ndk -t x86_64 -P 26 clippy --all-targets` clean for
`android-shell` on the Android target too (the `android_logger`
dependency is Android-only, so this is the only way to compile-check
it). `./run-tests.sh` from the repo root: 127 `server` tests, 88
`client-core` tests (85 + the 3 new to `notifications.rs`), all
passing -- the port added no regression to what already worked.
`./gradlew :shellApp:lintDebug`: `No issues found` (the report at
`app/shellApp/build/reports/lint-results-debug.txt`).
*The two pass-condition proofs*, both on this checkout's own AVD
(`ai-app-2`, GPU host per the default, torn down with `emu down`
when this session finished) against `app/ui-sandbox.sh`:
- **Notification with the app closed.** Enrolled via
`adb shell "am start -a android.intent.action.VIEW -d
'aiappshell://enroll?host=10.0.2.2&port=<sandbox port>&token=<token>'"`
(per the sandbox's own banner, substituting the scheme), granted
`POST_NOTIFICATIONS`, pressed home, then
`./ui-sandbox.sh spawn e3notif2` and
`./ui-sandbox.sh send <sid> "/question Should I proceed with the deploy?"`.
`adb shell dumpsys notification --noredact` shows a
`channel=sessions` record, `android.title=e3notif2`,
`android.text=Waiting for you` (matching `attention_line` and the
session's own title, exactly what `Notifications.kt` would have
shown) -- posted while the app held no visible activity. Tapping
it (`ui-trace record --do "tap 'e3notif2'"`, found in the
expanded shade after `adb shell cmd statusbar
expand-notifications`) launched
`com.example.aiapp.shell/.MainActivity` with
`dat=aiappshell://session/...`, confirmed in `adb logcat`'s
`ActivityTaskManager: START` line -- the `PendingIntent` names
the right session.
- **A share lands in a session.** With the app enrolled and a
session already active,
`adb shell "am start -a android.intent.action.SEND -t text/plain
--es android.intent.extra.TEXT 'Please check the deploy logs for
errors.' -n com.example.aiapp.shell/.MainActivity"` (the classic
`adb shell` quoting trap from `this-machine-android` applies here
too: the whole `am start` invocation has to be one single-quoted
string handed to the *remote* shell, or the extra's spaces get
re-split away). `./ui-sandbox.sh api
'/sessions/<sid>/transcript?limit=20'` shows
`{"type":"userMessage","text":"Please check the deploy logs for
errors."}` followed by the echo driver's reply -- the share
reached the most-recently-active session as a real message, not
a mock.
- [x] **E4 — the same screen on the desktop (2026-09-05).** A new
`iris/desktop-app` crate (added to the `iris` workspace's members, not
excluded the way `android-app` is -- nothing here needs the NDK):
a real winit window showing a session list (`iris::widget::Span`,
rebuilt on selection) beside `transcript-ui`'s screen
(`transcript_ui::build_tree`, new this box -- see IRIS.md's
2026-09-05 entry), talking to a real `ai-server` through
`client-core`'s `ApiClient`/`UreqTransport`/`follow_session_events`.
Enrolment is `client_core::config::EnrolledServer::parse_link`
against the same `aiapp://enroll?host=H&port=P&token=T` link a phone
scans, pasted via `--link` and persisted at
`$XDG_CONFIG_HOME/ai-app-desktop/enrollment.json` (0600 --
`iris/desktop-app/src/config.rs`); the pinned CA is a `--ca PATH`
argument, never baked in (DECISIONS.md, 2026-09-05).
*Both pass-condition proofs held, against `app/ui-sandbox.sh`'s real
server.* (1) The list showed the sandbox's spawned session
("Demo session", its live status); selecting it loaded the real
transcript and the composer's `Submit` posted a message whose reply
streamed in live over SSE, both proved by two `run-headless.sh`
screenshots taken seconds apart around a real `./ui-sandbox.sh send`
-- the second showed the new turn appended under the first with
nothing duplicated or lost. (2) Screenshotted headless:
`/tmp/iris_e4_desktop.png` (1920x1200, 15.9 KB, the real first-run
state -- list populated, "Select a session." on the right, nothing
selected yet). `run-headless.sh` gained a `--bin` flag for this
(`cargo build --bin NAME` + `target/debug/NAME` instead of the
`--example` path, since `desktop-app` is a real binary a person
runs, not a demo) and `$RUN_HEADLESS_ARGS`, word-split into the
launched binary's own argv (a real CLI's flags, which no example
needed a way to pass before). Exact commands, from `iris/`:
TOKEN=$(cat "${XDG_CONFIG_HOME:-$HOME/.config}/ai-app/sandbox-token")
LINK="aiapp://enroll?host=127.0.0.1&port=<PORT>&token=$(python3 -c \
'import sys,urllib.parse;print(urllib.parse.quote(sys.argv[1],safe=""))' "$TOKEN")"
CA="${XDG_CONFIG_HOME:-$HOME/.config}/ai-app/certs/ca.pem"
RUN_HEADLESS_ARGS="--ca $CA --link $LINK" \
./run-headless.sh desktop-app --bin --shot /tmp/iris_e4_desktop.png -- -p desktop-app
**A real bug this screenshot found, not a synthetic one**: the first
attempt resumed the live SSE stream from
`items.iter().map(TranscriptItem::seq).max()` -- the *folded* item's
seq, which for a still-open `AssistantMsg` is the seq of its
*first* delta by design (`fold_event`'s own doc comment: "a row
whose identity changed with every delta would be a new row every
frame"). Resuming from there re-delivered every delta already
folded into that message, and the screenshot showed the assistant's
reply with its own tail duplicated ("You said: ... testsaid: ...
test"). Fixed by computing the resume cursor from the raw wire
`seq` of the last fetched line (`app.rs`'s `raw_seq`) instead of
from any folded item -- regression test
`the_resume_cursor_is_the_last_wire_seq_not_the_last_items_seq` in
`iris/desktop-app/src/app.rs`. Exactly the class of bug CODE_RULES
warns about under "a fix tried only on what it was meant to fix":
the bare REST fetch (no live stream yet) looked perfect on its own,
and only *resuming* a stream after it exposed the seam.
**Deliberately left simple, and why** (`app.rs`'s module doc has the
full account): every incoming SSE event refolds the session's whole
item list and rebuilds the entire right-hand widget tree from
scratch, rather than reaching for `TranscriptScreen::push_row`'s
incremental append -- `push_row` can only add a new row, and a
streaming reply is exactly a row whose text keeps changing after it
first appears. Fine at the size a desktop session's conversation
is; wrong for a long, fast-streaming one, and the real fix needs
`transcript-ui` to expose updating a row already on screen, which it
does not yet. The composer's in-progress text is saved and restored
across a rebuild so a reply streaming in while the reader is typing
a followup doesn't erase it. No history paging (I3's job, reused
as-is if this becomes permanent) and no scroll-position preservation
across a rebuild -- both named rather than silently missing.
Background network I/O runs on plain `std::thread`s reporting back
through winit's `EventLoopProxy<AppEvent>` rather than iris's own
`Tasks`/`task_on`, because `Tasks` only requests a redraw once after
its whole async closure finishes, which fits "one request, one
update" and not a live stream that needs a redraw after *each*
event it relays.
Verification: `cargo fmt --all`, `cargo clippy --workspace
--all-targets` (zero warnings), `cargo test --workspace` from
`iris/` (7 new tests in `desktop-app` -- 4 for
`config.rs`'s save/load/permissions/corruption, 3 for `app.rs`'s
transcript folding and the resume-cursor regression above -- plus
the existing 37 unchanged), and `./run-tests.sh` at the repo root
(127 passing, `client-core` alone 93 -- the `EnrolledServer` parsing
tests already existed before this box). Android is untouched by
this step, as asked.
- [x] **E5 — the packaging xtask (2026-09-05).** Both pass-condition
proofs held on this checkout's own emulator: `adb install -r` of the
xtask-built APK over the Gradle-built one succeeded, and the
notification service reached its follow-loop and posted a real
notification while the app was backgrounded. `cargo xtask apk` at
the repo root (`.cargo/config.toml`'s alias for `cargo run
--manifest-path xtask/Cargo.toml --`) runs `cargo ndk` →
`javac`/`d8` → `aapt2` → `zipalign` → `apksigner` with no Gradle
driving the packaging itself -- one disclosed exception, below.
*Where it lives.* `xtask/` (new, independent crate at the repo
root -- **no Cargo workspace**, matching every other crate here;
`run-tests.sh` already `cd`s into each rather than assuming one).
**Zero dependencies**: every step is "run this SDK/JDK tool with
these arguments and check its exit status," which needs nothing a
crate would add (AGENTS.md's "new dependencies need a reason").
`src/sdk.rs` finds the SDK root/build-tools/`android.jar` the same
way `app/android-env.sh` does ($ANDROID_HOME, then
$ANDROID_SDK_ROOT, then `~/Android/Sdk`); `src/keystore.rs`
finds-or-generates the release key with the exact recipe
`app/build-apk.sh` uses (same env vars, same path, same `keytool`
invocation) so the two tools sign with the *same* key, plus a
`--debug` path using the conventional `~/.android/debug.keystore`;
`src/apk.rs` is the pipeline itself, `src/main.rs` the ~40-line CLI.
About 420 lines total against RUST.md's earlier "about 150" guess --
the difference is almost entirely dependency handling (below), which
the earlier estimate didn't anticipate.
*`:link`'s `kotlinc` question, resolved.* Checked first, since E3
left it open: no standalone `kotlinc` exists on this machine (not on
PATH, not under any SDK -- only `kotlin-compiler-embeddable` jars
inside Gradle's own distributions). So the choice was never
"invoke kotlinc" versus "port `ServerStore`/`ServerSettings` to
Java" as originally framed -- a third route fell out of solving the
*other* open dependency problem (androidx, next paragraph): the one
Gradle call already needed for that also compiles `:link`'s Kotlin
as a side effect, via Gradle's own embedded compiler, and hands back
the resulting `classes.jar` in the same resolved-jars list. That is
RUST.md's own "prebuild it once into a jar/aar E5 consumes as a
binary input" option, arrived at for free rather than built
specially -- no Java port of `ServerStore` was written, and
`android-shell/src/settings.rs`'s JNI class-by-name lookup
(`com/example/wgapplink/ServerStore`) needed no change.
*One disclosed exception to "no Gradle in the loop": dependency
resolution.* `app/shellApp` depends on `:link` (Kotlin, above) and
on `androidx.core:core-ktx` -- not a compile-time dependency of the
two Java stub classes (`MainActivity`/`NotificationService` import
only `android.*`), but a **runtime** one: `android-shell/src/notify.rs`
reaches `NotificationCompat`/`NotificationChannelCompat`/
`NotificationManagerCompat`/`ServiceCompat`/`ContextCompat` by class
name over JNI, so their bytecode has to be in the final dex even
though nothing in this pipeline's own Java source mentions them.
Reimplementing a Maven/AAR dependency resolver to avoid one Gradle
call was not a good trade against "smallest honest route" (the
standard this file already applied to `kotlinc`) -- so
`app/shellApp/build.gradle.kts` gained one task,
`printRuntimeClasspathJars`, which asks the `releaseRuntimeClasspath`
configuration for its artifacts through an `ArtifactView` requesting
the `android-classes-jar` attribute (the same post-AAR-transform
view AGP's own dexing task consumes, so an AAR is already unpacked
to a plain `.jar` by the time the xtask sees it) and writes their
absolute paths, one per line, to
`app/shellApp/build/xtask/runtime-classpath.txt`. `cargo xtask apk`
runs `./gradlew :shellApp:printRuntimeClasspathJars` once (a few
seconds, mostly UP-TO-DATE on a warm Gradle daemon), reads that file,
and hands every jar in it to `d8` as an ordinary program input --
`:link`'s `classes.jar` among them, per the paragraph above. Nothing
past that one call touches Gradle. **What this trades away**: the
pipeline is not Gradle-free end to end, only Gradle-free for the
part that was actually expensive (assembling and dexing the app's
own code, which the earlier options -- kotlinc, or a hand-rolled
resolver -- were the two ways to avoid entirely). Recorded here
rather than left implicit, matching how the `kotlinc` compromise
above is recorded.
*The rest of the pipeline, in order (`apk.rs`):* `cargo ndk -t
arm64-v8a -t x86_64 -P 26 -o app/shellApp/src/main/jniLibs/ build
--release -p android-shell` (both ABIs by default -- real phone and
this machine's emulator -- `--abi` overrides; always `--release`
for the native library regardless of the APK's signing variant, for
the reason E1 already established: a debug build's Vulkan
object-labelling segfaults this emulator's driver, and there is no
reason for a signing choice to make this crate's `.so` bigger).
`javac -cp android.jar` compiles `MainActivity.java`,
`NotificationService.java` and a freshly generated `PinnedCa.java`
(same template as the Gradle `generatePinnedCa` task, same
opening-quotes-adjacent-to-`"""` rule from AGENTS.md's "Things that
have bitten") into one `classes.jar` (`jar cf` -- `d8` rejects a
bare directory of `.class` files outright, "Unsupported source file
type", discovered by trying it). `d8 --release --min-api 24 --lib
android.jar` dexes that jar plus every classpath jar from the
paragraph above into one `classes.dex` (no multidex needed at this
size). `aapt2 link` compiles `app/shellApp/src/main/AndroidManifest.xml`
into the base APK's `resources.arsc` -- the checked-in manifest has
no `package` attribute (Gradle injects one from `android.namespace`
during a manifest merge this pipeline doesn't run), so `apk.rs`
writes a copy with `package="com.example.aiapp.shell"` spliced in
rather than editing the source manifest, and refuses to run at all
if the source ever gains one of its own (a version-drift guard
cheaper than a real merge). `--min-sdk-version`/`--target-sdk-version`/
`--version-code`/`--version-name` are passed on the command line for
the same reason -- the raw manifest carries none of them, Gradle's
`defaultConfig` normally does. `jar uf` (not a hand-rolled zip
writer -- `jar` ships with the JDK this pipeline already needs)
merges `classes.dex` and a staged `lib/<abi>/libandroid_shell.so`
tree into the base APK (cargo-ndk's `-o` writes
`jniLibs/<abi>/*.so`, matching the Gradle source-set layout it was
pointed at; Android's own zip convention wants `lib/<abi>/*.so` at
the archive root, hence the staging copy rather than an in-place
rename). `zipalign -f -p 4` then `apksigner sign` finish it, signed
with `~/.config/ai-app/release.jks` by default or
`~/.android/debug.keystore` under `--debug`. The signed APK is
copied to `xtask/build/outputs/apk/<mode>/ai-app-shell-<mode>.apk`
as a final step -- a Gradle-shaped path (`*/build/outputs/apk/*/*.apk`)
chosen so Dev Updater's fixed-pattern APK discovery
(`~/repos/dev-updater/server/src/discover.rs`'s `APK_PATTERNS`,
which has no per-component path override) finds it without any
change on that side; the working files above it stay under
`target/xtask/apk/`, an ordinary build-cache location (gitignored,
along with `xtask/target/`).
*Wired into `.dev-updater.ron`*: a second `Apk` component, `shell`,
`build: "cargo xtask apk"`, `modes: ["release", "debug"]`, no `cwd`
(defaults to the checkout root, which both the `cargo xtask` alias
and the publish path above need -- `.cargo/config.toml`'s alias
resolves its `--manifest-path` relative to the *invoking* working
directory, not to where the config file lives, which is what ruled
out giving this component its own `cwd`). Dev Updater's `ByMode`
appends the chosen mode word as the command's last argument
(`build-apk.sh`'s own interface, per that component's comment), so
`main.rs` accepts bare `release`/`debug` as well as `--release`/
`--debug` for typing by hand. The existing `app` component
(`build-apk.sh`, Gradle) is untouched.
*Verification.* `cargo fmt -- --check` and `cargo clippy
--all-targets` clean, zero warnings, for `xtask` (host target --
nothing in it is Android-specific; it *runs* `cargo ndk`, it isn't
cross-compiled itself). `./run-tests.sh`: 127 `server` + 88
`client-core` tests, unaffected, still passing. `apksigner verify
--print-certs` on the xtask's release output confirms a V3 signer
with `CN=ai-app` -- the same key `build-apk.sh` generates.
*The two pass-condition proofs*, both on this checkout's own AVD
(`ai-app-2`, GPU host, brought up and torn down within this
session):
- **Installs over the Gradle-built one.** Built the Gradle release
variant first (`AI_APP_KEYSTORE=~/.config/ai-app/release.jks
AI_APP_KEYSTORE_PASSWORD=$(cat
~/.config/ai-app/release.jks.password) ./gradlew
:shellApp:assembleRelease` -- needed its own signing block added
to `app/shellApp/build.gradle.kts`, copied from `androidApp`'s,
since `shellApp` had none before this), installed it fresh
(`adb uninstall com.example.aiapp.shell` first -- an older debug
install from E3 testing was signed with a different key and
`install -r` over it fails loudly with
`INSTALL_FAILED_UPDATE_INCOMPATIBLE`, which is the correct,
expected failure for a mismatched key rather than a bug), then
`adb install -r xtask/build/outputs/apk/release/ai-app-shell-release.apk`:
**`Success`**.
- **The notification service starts.** Enrolled via
`adb shell "am start -a android.intent.action.VIEW -d
'aiappshell://enroll?host=10.0.2.2&port=<sandbox port>&token=<token>'"`,
force-stopped the app, then re-launched it once (enrollment calls
`sync()` from `MainActivity.onCreate`). `adb logcat` shows
`ActivityManager: Background started FGS: Allowed ... intent:
... cmp=com.example.aiapp.shell/.NotificationService`, immediately
followed by `android-shell: jni::vm::java_vm: Attached thread
ai-app-notifications`, a real TLS handshake to the sandbox's
`10.0.2.2:<port>`, and `Response { status: 200 ... }` on
`/notifications`. Pressed home, spawned a sandbox session and sent
it `/question Should E5 proceed?`; `adb shell dumpsys notification
--noredact` then shows a live `NotificationRecord` for
`com.example.aiapp.shell`, `channel=sessions`, `tag=<session id>`
-- posted while the app held no visible activity, the same bar
E3's own proof cleared.
*Left undone, honestly.* No attempt to shrink the dex (R8/minify is
off, matching `shellApp`'s existing `isMinifyEnabled = false`, so
the APK carries the full unshrunk `androidx`/Kotlin-stdlib/coroutines
graph -- about 5.2 MB signed with both ABIs, most of it native
libraries and that dependency graph rather than this project's own
code). No `--abi arm64-v8a`-only real-device install was attempted
this session (no physical phone reachable from here); the emulator
proof above is `x86_64` plus a cross-compiled but unexercised
`arm64-v8a` `.so` in the same APK. Multidex is unneeded at today's
size but nothing in `dex()` checks for the 64k-method ceiling should
the dependency graph grow.
### The iris track
These build iris up to carry the app. Each is a feature added to iris
with a pass condition, in dependency order. Work in `iris/` in this
repository on the `rustify` branch, and record in this file what each
step measured.
- [x] **I0a — where iris lives (decided 2026-09-04).** For now it is
**vendored at `iris/` in this repository**, history not carried,
and consumed by path. Iris's decision: keep it close while it is
being reshaped for this app, and give it back its own repository —
`iris/iris` on the gitea remote, which already holds the full
244-commit history, on a branch of its own — once it has proved
itself. The vendored tree is that repository's `main` at
`7b54aaf` ("readme", 2026-01-29), byte-identical to the public
GitHub copy, so a later reconciliation has a known base. A crate
that uses it says `iris = { path = "../iris" }`.
- [x] **I0b — make it build here (done 2026-09-04).** iris now builds,
clippy-clean and rustfmt-clean at the defaults, on a pinned dated
nightly, and the `tabs` example draws on this VM's GPU.
**The pin** is `nightly-2026-09-03` (rustc 1.100.0-nightly,
`2e2b193f8`), declared in `iris/rust-toolchain.toml` along with the
`clippy`/`rustfmt` components and the two Android targets, so a
fresh clone provisions itself. It is dated rather than `nightly`
because the whole failure below was a rolling channel moving under
an unattended build. Installed with `--profile minimal`: 912 MB.
**The 36 errors were one syntax change, and the earlier diagnosis in
this file was wrong.** It is not that a trait must now be declared
`const trait` — the vendored tree already declares them that way,
which is how it was written in January. What changed is the *impl*
keyword order: `impl const Trait for T` is now
`const impl Trait for T`, and generics go on the `impl`
(`const impl<T: [const] Foo> Bar for T`). Bounds are unaffected;
`T: const Foo`, `T: [const] Foo` and `impl const Foo` in argument
position all still compile. Everything else — the unresolved
`UiVec2`/`Vec2`/`impl_op` imports, and a `Color<u8>` that resolved
to `wgpu_types::Color` — cascaded from the seven files that failed
to parse. The rewrite was mechanical across 20 sites and took the
workspace from 36 errors to 0.
**`#![feature]` gates, 12 after this step** (two were declared and
unused, and were removed: `map_try_insert`, `const_cmp`).
Load-bearing and worth watching: `const_trait_impl`, `const_ops`,
`const_convert`, `const_destruct` are the const-traits family and
the one that has already broken once — they move together, so
advancing the pin means re-reading this section. `unboxed_closures`
+ `fn_traits` (postfix builder API) and `unsize` +
`coerce_unsized` (widget handles) are pairs. The rest are
individually small: `macro_metavar_expr_concat`, `portable_simd`,
`associated_type_defaults`, `option_into_flat_iter`, and `gen_blocks`
in the top crate.
**Running it headless.** `iris/run-headless.sh EXAMPLE [--shot PNG]`
with `iris/headless.conf`, the same trick `emu` uses: a headless
sway, and `grim` for the picture. It deliberately starts its *own*
compositor rather than joining `emu`'s — sway tiles, so adding a
window to the one an emulator sits in resizes that emulator.
Unlike `emu`'s it disables Xwayland, since winit speaks Wayland.
**This VM has a real GPU for this**: Vulkan 1.4 through Venus onto
the host's RX 7900 XT, and GL 4.6 through virgl — so desktop wgpu
work here is not software-rasterised, unlike inside the emulator.
**iris has no tests at all** (`cargo test --workspace`: 0 passed
across 6 targets). Nothing to keep passing, and nothing to catch a
regression — worth knowing before I1 changes the text stack.
**`iris-core` no longer depends on winit, and now cross-compiles to
Android.** It wanted exactly one thing from it — `PhysicalSize<u32>`
in `UiRenderNode::resize`'s signature, for two numbers it immediately
turned into floats — and that pulled a whole windowing backend into
the layer below it, the wrong direction. `resize` takes
`impl Into<Vec2>` now, like `UiRenderState::resize` beside it already
did. The consequence is the point: with winit in the graph an Android
build of the core failed in `android-activity` (which needs a backend
feature nothing here selects), and without it
`cargo ndk -t arm64-v8a -P 26 build -p iris-core` finishes in 30s and
produces an rlib, wgpu's Android backend included. So **iris's
widget, layout and render core already builds for the phone**, and
what I2 has to supply is the surface, the input and the IME — not a
port of the library.
**Build weight, cold, on this VM's 8 cores** (`rm -rf target`, then
`cargo build --example tabs`), since "the Linebender stack is slow in
debug" was the worry behind this question: plain debug **43s** and a
2.1 GB `target/`; with the `[profile.dev.package."*"] opt-level = 2`
knob, **1m46s** and 1.5 GB. So iris's own wgpu + winit + cosmic-text
graph is not the slow thing — which makes it a calibration for E1
rather than an answer about Masonry, whose graph adds Vello, Parley,
Fontique and Skrifa. Runtime cost of the knob was not measured here.
**Fixed: iris never called `pre_present_notify`.** The symptom was
that about one start in five kept the window's 800x600 startup layout
on a 1920x1200 surface for good. What settled it was tracing iris's
own decisions into memory and dumping them from another thread —
`eprintln!` in the draw path makes the defect vanish, which is why
earlier attempts kept losing it. The traces from a good and a bad run
are **byte-identical**: both lay out and draw `redraw_all at
(1920, 1200)` into a 1920x1200 texture with `suboptimal=false`. iris
was drawing the right frame every time; the compositor was still
showing the first one, and forcing a full repaint did not shift it.
What was missing is winit's `Window::pre_present_notify`, called
immediately before `present`, which on Wayland is what ties the
commit to the surface's frame callback. Without it a frame drawn with
nothing following it can sit unpresented with nothing left to flush
it — which is exactly a window that has just settled after its
opening resize. Measured: **0 bad in 40** with the fix, against 4 in
20 before it, and — the stronger evidence — 0 in 20 in the
instrumented configuration that had been 15 in 20. Runtime resizing
still round-trips to a byte-identical layout.
Two things ruled out on the way, both worth not re-trying: the
present mode (the fault survived the move from `AutoNoVsync` to
`AutoVsync` at the same rate) and the size cache (`redraw_all` clears
it). A `desired_maximum_frame_latency` of 1 moved the rate without
fixing it, and was reverted. Iris's own note that she had never seen
the library fail to resize was the useful steer: it pointed away from
the layout code, where two hours had already gone.
One thing was fixed on the way, and it is not that bug: `update`
redrew everything when `resized` was set, but `needs_redraw` — which
is what decides whether to *ask* for a frame — did not know about
`resized` at all. The two now share one `needs_redraw_all`, since a
condition in one and not the other is a frame nobody requests. It is
latent on Wayland only because winit asks for a redraw after a resize
by itself; on Android, where the surface work of I2 will not have
winit underneath it, nothing else here would have asked.
- [x] **I1 — parley, and a glyph atlas (done 2026-09-04).** No bake-off:
Iris decided for parley directly ("I wanted to switch it to parley
anyways"), and then asked for the atlas as well ("just do the atlas,
commit to it, we do want it"). Both are in.
**What parley bought, beyond shaping.** Its editing model addresses
text by byte offset into one string, where cosmic-text used
`(line, index)` — so `select_content`, `delete_between`,
`insert_inner` and `newline` collapse into ordinary string
operations. Bigger: `Selection::geometry` and `Cursor::geometry`
replace `iter_layout_lines`, `index_x` and `cursor_pos`, which walked
runs by hand to place the caret and the selection boxes and were not
bidi- or wrap-correct. `edit.rs` lost about 130 lines and gained
Home/End. Its cursor motions map onto parley's `next_visual`,
`previous_visual_word`, `next_line` and so on, in one function.
**The atlas is what "text resizing (per frame) is really slow"
was.** Every string used to be rasterised into its own `RgbaImage`
and uploaded as a whole texture whenever anything about it changed,
so a window resize re-rasterised and re-uploaded every visible
string. Now a glyph is rasterised once per font, size and subpixel
phase, shared by every string that contains it, and a resize
re-emits quads without touching the GPU's copy. **The tabs example
reports it: `views`, the number of texture views bound, went from 6
to 1** — six per-string textures became one shared page. Supporting
pieces: a `GLYPH` primitive that samples a sub-rectangle and tints
it (the existing texture primitive samples a whole texture), a
`Patch` texture update so a new glyph costs its own bytes rather
than a 4 MB page, and `GpuTextures` keeping its `Texture`s, since a
view cannot be written through.
**Not yet measured**, and the honest gap in this step: the TODO's
"really slow" was never given a number, so neither is the
improvement. What is evidence rather than argument is the view count
and the shape of the work — a resize no longer rasterises. A
before/after timing wants the transcript screen of I5 to be worth
taking.
**Two bugs found on the way**, both pre-existing: `primitives!`'s
`@count` rule recursed comma-separated while matching
space-separated, so it terminated only for exactly two primitives
and adding a third hit the recursion limit; and `Color` had no
`Default`, which parley's `Brush` requires.
**Fourteen tests**, iris's first. The editor is the one part that is
pure logic rather than something needing a GPU and a window, and it
was rewritten wholesale with no way to exercise it — synthetic input
does not reach a client under the headless compositor, which has no
seat devices. Two of the tests are aimed at what the rewrite could
plausibly have broken: the IME preedit path, and editing multi-byte
text now that offsets are bytes.
Dropping cosmic-text and unicode-segmentation also retired two
nightly gates — `portable_simd` (the old glyph compositing) and
`gen_blocks` (the deleted line iterator). **Eleven left.**
### iris's binding array does not survive real Android hardware (found 2026-09-04, resolved 2026-09-04)
**Resolved the same day**: see "Where things stand" above and
TEXTURES.md's "Implemented, 2026-09-04". The measurement and sourcing
below are unchanged and are why the fix looks the way it does; nothing
here needs re-checking on its own account.
Iris asked, of the "unknown number of images" case — a transcript with an
unbounded number of attached screenshots — whether iris's approach even
works on a phone, since her recollection was that mobile does not support
it. Checked rather than assumed, and the recollection is right, with
sources rather than a guess.
**What iris does today.** Every texture — every `Image` widget
(`src/widget/image.rs`) and every glyph atlas page — gets its own
permanent slot in one array via `Textures::add`
(`core/src/primitive/texture.rs:65`), and both the `TEXTURE` and `GLYPH`
primitives sample it by `view_idx` into `binding_array<texture_2d<f32>>`
at `core/src/render/shader.wgsl:56`, sized by `UiLimits::default` — 100,000
textures, 1,000 samplers (`core/src/render/mod.rs:347`). That needs three
wgpu features: `TEXTURE_BINDING_ARRAY`,
`SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING`,
`PARTIALLY_BOUND_BINDING_ARRAY` — Vulkan's `VK_EXT_descriptor_indexing`
("bindless"), promoted to core in 1.2. So a transcript with an unbounded
number of images is exactly the case that grows this array without bound,
one permanent slot per image.
**Measured first on the emulator, and it fails outright.** A rig
(`rigs/gpu-probe`, a plain executable with no window, pushed with `adb
push` and run from `/data/local/tmp` — no APK needed to ask a device what
it supports) asks `wgpu::Adapter::request_device` for exactly iris's
features and limits. Against the emulator's guest Vulkan — both
SwiftShader (`vk_swiftshader_icd.json`) and lavapipe (`lvp_icd.json`,
cold-booted) — `request_device` **fails**: `Unsupported features were
requested: TEXTURE_BINDING_ARRAY |
SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING |
PARTIALLY_BOUND_BINDING_ARRAY`. A second, raw query through `ash`
(`rigs/gpu-probe/src/vk.rs`, bypassing wgpu) shows lavapipe's
`vkGetPhysicalDeviceFeatures2` actually reporting all seven descriptor-
indexing sub-features as `true` at device api version 1.3 — so on this
software renderer wgpu-hal's own feature detection is being more
conservative than the driver, for a reason not chased further (a likely
instance-version negotiation gap, since `VK_EXT_descriptor_indexing` was
only promoted to core at 1.2 and wgpu-hal's own `Instance::init` may be
requesting less). That part is an emulator/wgpu-hal question and not the
finding that matters.
**The finding that matters is about real phones, not the emulator, and it
is sourced rather than recalled.** The **Android Vulkan Profile 2025** —
Google and Khronos's current baseline, covering **80.1% of active
Vulkan-capable Android devices** as of October 2025
([developer.android.com/ndk/guides/graphics/android-vulkan-profile](https://developer.android.com/ndk/guides/graphics/android-vulkan-profile))
— does **not** require `VK_EXT_descriptor_indexing` or any descriptor-
indexing feature. It requires `shaderSampledImageArrayDynamicIndexing`
(indexing an array of samplers by a value uniform across the invocation —
Vulkan 1.0 baseline, unrelated to bindless) and stops there; the same is
true of the 2021 and 2022 profiles. On the hardware side, Arm's own
developer documentation states **"`VK_EXT_descriptor_indexing` is
supported on all Valhall and 5th Gen GPUs"**
([developer.arm.com/mobile-graphics-and-gaming/vulkan-api-best-practices-on-arm-gpus](https://developer.arm.com/mobile-graphics-and-gaming/vulkan-api-best-practices-on-arm-gpus)) —
Mali generations from roughly 2019 (Mali-G77) onward, named affirmatively
with no claim made for Bifrost, Midgard or Utgard, which are still common
in budget and older Android phones still in use. So this is not a
software-renderer artifact: a real, currently-shipping share of the
Android fleet lacks the feature iris's texture pipeline asks for
unconditionally, and the newest official baseline does not promise it
either. (A crates.io/search-engine claim of "1% support on Android" for
this extension was checked against its cited source, an Arm blog post,
and was not actually there — that number does not appear anywhere primary
and should not be repeated; the 80.1%-baseline-excludes-it finding above
is the one with an attributable source.)
**Recommendation, not yet implemented.** iris already solved the
identical problem for text in I1: the glyph atlas
(`core/src/render/atlas.rs`) packs many small rasters into a handful of
shared 1024×1024 pages and samples them by UV offset, so **text needs
none of the three features above** — only ordinary single-texture
sampling. The same technique generalizes to images: route an `Image`
widget through a shared atlas when it is small enough to pack (thumbnails,
downscaled attachment previews, avatars, icons), and fall back to one
ordinary, non-array texture bind group — selected per batched draw call
the way every immediate-mode 2D renderer already does — for anything too
large to atlas well (a photo opened at full resolution). Either path is
plain Vulkan 1.0 / GLES texture sampling, so it removes the descriptor-
indexing requirement from iris's device request entirely, which is also
what would make the emulator work regardless of the wgpu-hal question
above: a device that never asks for the feature cannot be refused for
lacking it. This is a change to iris's rendering core — the shader's
binding group layout, `Textures`, the texture and glyph primitives, and
`ui/painter.rs` — so it is written here as a recommendation rather than
started, per the project's rule to confirm a load-bearing design change
before making it. **It should be resolved before I2 is called done**,
since I2's pass condition is the phone, not just the emulator, and this
is exactly the kind of thing that passes on a desktop GPU and fails
silently on real hardware.
- [x] **I2 — iris on android-view (2026-09-05).** The android-view backend,
the `iris-android-app` cdylib and Gradle shell, insets, the back
gesture and the full `InputConnection` bridge are in and measured
working; the tabs example now renders on the emulator (Vulkan/
SwiftShader and GLES/virgl both), and the composer's keyboard shows
real Gboard suggestions through the IME bridge. See below for the
render-gap root cause and fix.
**Layout.** `iris/src/android/` mirrors `default/`'s module split
(`view.rs` is `app.rs`+`state.rs` combined, since android-view has one
harness type where winit splits `ApplicationHandler` from per-window
state; `render.rs`, `input.rs`, `attr.rs` correspond directly;
`ime.rs` and `insets.rs` have no winit counterpart). What used to live
only in `default/` and had no winit dependency — `WidgetState`,
`CursorState`/the sense machinery, `Tasks`, `Selector`/`Selectable`'s
focus handling — moved to crate-root modules (`state.rs`, `sense.rs`,
`task.rs`, `attr.rs`) so both backends use one copy; `Tasks`' redraw
nudge is now behind a `RequestRedraw` trait (`Window` for winit, a
`JavaVM`+`GlobalRef` attach-and-call for android-view) rather than a
concrete `winit::window::Window`. `winit`/`arboard` and
`android-view`/`send_wrapper` are now `[target.'cfg(...)']`
dependencies, and `default`/`android` are target-gated modules,
because winit's own Android support needs `android-activity` with a
backend feature selected — exactly what `iris-core` was kept free of.
Confirmed by trying it before the split (`cargo ndk -t x86_64 -P 26
build -p iris` failed inside `android-activity` itself) and after
(clean). `iris/tabs-ui` is the tabs example's widget tree factored out
of `examples/tabs/main.rs` into a crate generic over `Rsc: HasEvents`
+ `Rsc::State: FocusHost`, so the winit example and
`iris/android-app` (the new cdylib, excluded from the `iris` workspace
because android-view needs the NDK sysroot to link — see that
`Cargo.toml`'s comment) call the same `build()`.
android-view pinned to `bec6c62a96cef8239b0fd7fedeef9b184d02e3a1`, the
commit E1 measured against. `RustView.java`/`RustInputConnection.java`
are vendored (no published AAR to depend on) into
`iris/android-app/app/src/.../org/linebender/android/rustview/`, with
one deliberate diff from upstream noted in a comment: `mViewPeer` is
`protected` rather than package-private, so `IrisView` (a different
package) can pass it to the window-insets native call android-view
has no hook for.
**Insets and the back gesture**, both without touching android-view.
The back gesture takes no new plumbing at all: with no
`OnBackPressedCallback` registered, Android still delivers it as an
ordinary `KEYCODE_BACK` `KeyEvent` through the existing key path (the
legacy behaviour every view-based app gets by default), handled in
`view.rs`'s `on_key_down`. Insets have no such stand-in, so
`android/insets.rs` registers one more native method
(`applyWindowInsetsNative`) directly on `IrisView`, writing into an
`Rc<RefCell<Shared>>` a second copy of which lives in
`AndroidUiState` — the peer id android-view hands back from
`register_view_peer` is opaque outside that crate, so this is a
side table keyed on the same id rather than a way to reach the peer
itself. `MainActivity` wires `setOnApplyWindowInsetsListener`,
including the API 30+ `ime()` inset specifically (falls back to 0
below that). Not yet consumed by any widget's layout — `insets()` is
exposed on `AndroidUiState` but nothing reads it yet, since the tabs
example has no chrome that needs to avoid the keyboard.
**The IME bridge is implemented and its pass condition holds.**
`android/ime.rs` implements the full `InputConnection` trait
(`text_before_cursor`/`after_cursor`/`selected_text`,
`cursor_caps_mode`, `delete_surrounding_text[_in_code_points]`,
`set_composing_text`/`_region`, `finish_composing_text`,
`set_selection`, `begin`/`end_batch_edit`, `send_key_event`,
`request_cursor_updates`) directly against `TextEdit` — the same
preedit-replace bookkeeping `default`'s `Ime::Preedit` handling uses
(`compose_len`, in chars), with new byte<->UTF-16 conversion helpers
since parley (since I1) is byte-indexed and Java strings are not.
Two approximations, both commented in place rather than silently
dropped: `set_composing_region` declines (no separate composing range
exists to move) and `set_selection`/`delete_surrounding_text_in_code_points`
collapse to an approximation rather than a real span/code-point
count. `TextEdit` gained `text()`/`selection_range()`/`caret()`
getters and `TextEditCtx::delete_byte_range`/`set_cursor_byte`, all
unconditional (no winit dependency added); `apply_event`/
`TextInputResult`, which do take a `winit::event::KeyEvent`, are now
`#[cfg(not(target_os = "android"))]` instead of being ported, since
android's own `input.rs` calls `TextEdit`'s primitives
(`backspace`/`delete`/`motion`/`insert`) directly from
`ndk::event::Keycode` and never needed a winit `KeyEvent` shape.
**Measured on the emulator, 2026-09-05, x86_64 API 26,
`-feature Vulkan` + SwiftShader per the Vulkan section below.**
`adb shell dumpsys input_method` after tapping the composer field:
`mInputShown=true`, `mServedInputConnection` is
`org.linebender.android.rustview.RustInputConnection` attached to
`IrisView`. `adb shell input text "hi"` followed by a screenshot
shows **Gboard's suggestion strip populated with "hi | Hi | HI"** —
capitalization variants read back out of the real buffer through
`text_before_cursor`, the same kind of evidence E1 recorded (there:
"dolor | Dolores | door"). That is the bar this box asks for, met.
**Resolved 2026-09-05: the render gap was the window uniform, never
the atlas.** `UiRenderNode::new` (`core/src/render/mod.rs`) seeded the
GPU's `window_buffer` from `WindowUniform::default()` — width=0,
height=0 — and the only thing that ever corrected it was a later call
to `UiRenderNode::resize`, renamed `AndroidRenderer::resize` on the
android side. winit's backend gets away with the same default because
winit fires an initial `WindowEvent::Resized` before the first frame,
which `default/mod.rs`'s event loop turns straight into that resize
call — a real event this project never had to add on purpose, so
nothing here noticed the node depended on it. android-view has no such
automatic event: `surface_changed` (`src/android/view.rs:363-388`)
only calls `self.render.resize(...)`, which is
`UiRenderState::resize` — the CPU-side *layout* width the widget tree
lays out against — not `AndroidRenderer::resize`, which is the one
that writes the GPU uniform. `AndroidRenderer::new` builds a fresh
`UiRenderNode` with the correct `SurfaceConfiguration` (so the surface
itself was always the right size, and the clear colour reached it) but
that node's window buffer was never subsequently written, so it sat at
`(0, 0)` for the node's entire life. `shader.wgsl`'s `vs_main` divides
by `window.dim` to reach clip space
(`let pos = (top_left + uv * size) / window.dim * 2.0 - 1.0;`), so
every primitive's clip position came out `NaN`/`Inf` and was dropped
before rasterization on **both** backends — Vulkan and GLES alike,
exactly the cross-backend symmetry that should have pointed away from
a GL-specific cause sooner. The layout engine reporting the correct
widget count and pixel region the whole time is consistent with this:
that path never touches `window.dim` at all, since it is a separate
copy of the window size (`UiRenderState`'s own, fed by
`self.render.resize`) that the CPU-side layout and hit-testing use.
**The GLES `D2`/`D2Array` warning was confirmed a red herring.**
Reproduced again after the fix, unchanged, on a build forced to
`Backends::GL` — it fires on every frame regardless, and primitives
draw correctly on that backend anyway (screenshot below), so it is a
cosmetic wgpu-hal heuristic notice, not a correctness bug in the atlas
path. Left as-is; chasing it further is not warranted.
**Fix** (`core/src/render/mod.rs`, `UiRenderNode::new`): seed
`WindowUniform` from `config.width`/`config.height` — already the
surface's real size at construction time on both backends — instead
of `WindowUniform::default()`. This removes the dependency on an
external resize call entirely (winit's initial `Resized` event still
fires and still calls `resize()`, now idempotently) rather than
papering over android-view's missing event with one more call in the
android-specific path; a future third backend gets a correct window
buffer from its first frame with no equivalent event of its own to
remember.
**Verified on the emulator, 2026-09-05, `ai-app-2`'s own AVD, x86_64
API 26, `-feature Vulkan` + SwiftShader per the Vulkan section.**
`logcat` after launch: `render(): after update active=39
root_px=Some(PixelRegion { top_left: (0, 0), bot_right: (1080,
2219) })`, no wgpu validation warnings on the Vulkan build. Screenshot
(`/tmp/iris_i2_render.png`) shows the tabs example's coloured spans,
the red rounded rect and the tab bar all drawn — the milestone this
section asked for. Rebuilt with `Backends::GL` forced (reverted
afterwards; the shipped code still requests `Backends::PRIMARY`) and
reinstalled: same screenshot, same widgets, `AdapterInfo` logged as
`Android Emulator OpenGL ES Translator (virgl (AMD Radeon RX 7900
XT...` confirming the real GLES/virgl path, with the `D2`/`D2Array`
warning present and harmless as above. Text glyphs render with visible
artifacting on the GLES path specifically (not investigated further —
out of scope for this box, which is about primitives appearing at
all, and it does not affect the Vulkan path this app ships behind).
**Not built yet**: anything consuming `insets()`, a real phone
measurement (only the emulator so far — matches every other Android
finding in this file), and AccessKit (I4's job, so `ui-trace`
couldn't be used here; a raw `adb shell input tap`/`input text` stood
in for driving the UI, which is why this section says "the same bar
as E1" rather than citing a `ui-trace` transcript).
**Verification.** Host: `cargo fmt --all -- --check`,
`cargo build --workspace --all-targets`, `cargo clippy --all-targets`,
`cargo test --workspace` (19 tests) all clean in `iris/`; `iris/run-headless.sh
tabs --shot` still renders pixel-identically (27266 bytes, byte-for-byte
unchanged). Android cross-compile: `cargo ndk -t x86_64 -P 26 build`
and `... clippy` clean for both `iris` (with the android module) and
`iris/android-app`. Emulator: `emu up` with
`VK_DRIVER_FILES=.../vk_swiftshader_icd.json` and
`GPU_HOST_FEATURES="-feature Vulkan -no-snapshot-load -no-snapshot-save"`
per the Vulkan section; `cd android-app && cargo ndk -t x86_64 -P 26
-o app/src/main/jniLibs/ build --release && gradle :app:assembleDebug`
(release native lib per E1's segfault finding, debug Gradle variant --
the jniLibs contents are what matters, not the Gradle build type);
`adb install -r app/build/outputs/apk/debug/app-debug.apk`. Emulator
torn down after verification (`emu down`) per the machine's memory
rule.
- [x] **I3 — a virtualised, bottom-anchored list (2026-09-05).** Variable-height
rows, keyed, composed only while visible, paged in both directions
with a "more" sentinel at each end, a scroll anchor that survives
rows being inserted above, and "hold the edge nearest the tap" done
in the layout pass. Built as `iris::widget::List`
(`iris/src/widget/list.rs`, its module doc is the design writeup) --
see `IRIS.md`'s 2026-09-05 entry for the public API and the one
correctness lesson worth carrying elsewhere (a fill-shaped background
cannot be measured at a throwaway oversized region and merely
`reposition`ed into place; it has to be placed at its cached real
size, or measured-then-redrawn via `draw_twice` on first appearance).
**Done**: the widget, 6 unit tests (`cargo test -p iris`, anchor and
edge-hold logic, all pure -- no GPU/window needed, same harness as
`layout_tests.rs`), `iris/benches/message_list.rs` rewritten to
measure the real widget instead of a hand-built `Span`+`Scroll`, two
new benchmark scenarios ((d) insert-above-anchor, (e)
expand-a-row-holding-its-edge), and `iris/examples/message_list.rs`
(800 rows, varied wrapped-text length, one in twelve with an image,
mouse-wheel scrollable) rendered via `run-headless.sh` and visually
verified (cropped with a throwaway PNG decoder, since this VM has no
image tooling -- see the commit for the crop script's shape).
**Numbers (2026-09-05, release, this VM), all flat across N =
100/1,000/10,000 as required:**
cd iris && ./run-bench.sh list
(a) first frame: ~12.3-12.9ms draws=80 rewrites=3 moves=0
(b) scroll, 200 ticks: 4.8-6.5ms draws=328 rewrites=12 moves=10131 (~0.025-0.033ms/tick)
(c) input grows, 40 lines: 8.9ms draws=1846 rewrites=102 moves=1195 (~0.22ms/line)
(d) insert-above-anchor, 200 pushes: 0.4ms draws=200 rewrites=0 moves=0 (~0.002ms/push)
(e) expand-hold, 40 growths: 0.10-0.11ms draws=119 rewrites=40 moves=15 (~0.003ms/growth)
(d) is the cleanest confirmation: 200 rows prepended one at a time
while scrolled to the loaded window's start cost 200 draws total (the
list widget's own redraw each push) and **zero** row draws or moves
-- none of the prepended rows ever entered the viewport, exactly as
the anchor-by-slot-index design predicts. (e) similarly stays tiny
and flat: growing one row 40 times, each preceded by `note_tap` at
its own edge, costs a total of 15 moves (the rows on the far side of
the held edge) regardless of how many thousand rows exist elsewhere
in the list.
**Verification.** `cargo fmt --all -- --check`,
`cargo build --workspace --all-targets`,
`cargo clippy --all-targets` (and `--benches --release` separately,
since benches aren't always covered), `cargo test --workspace` (25
passed) all clean in `iris/`.
**What remains — the emulator half of the pass condition, blocked on
the emulator being held by another session during this pass.** The
condition as written ("800 rows of real transcript text from the
sandbox scroll without a frame over the Compose baseline in
`transcript-bench.sh`, measured on the GPU emulator") needs the
transcript screen actually rebuilt on top of `List` (this box only
built and measured the widget in isolation, per the task scope) and
then driven through the real emulator rig. Once that screen exists,
the exact command is:
cd app && ./transcript-bench.sh -k # or without -k for a fresh session
# compare its render report against the iris build's equivalent
This is a genuinely separate step (wiring `List` into an actual
session screen, i.e. most of I5's work) rather than something this
box's scope could finish alone -- recorded here rather than left
silently undone.
- [x] **I4 — accessibility names via AccessKit, host half done and verified
2026-09-05; the emulator half is the one step left, named at the
bottom of this box.** Built `iris_core::ui::access::AccessTree`
(`iris/core/src/ui/access.rs`) -- one flat AccessKit tree, a synthetic
`Role::Window` root with every **named** widget as a direct child.
Deliberately flat rather than mirroring iris's real widget nesting:
nothing upstream of a named leaf needs a node, since a screen
reader's traversal (and uiautomator's tap-by-name, this box's own
pass condition) works from each node's on-screen bounds, not from
tree structure -- and mirroring the real tree would rebuild
intermediate nodes on every resize of any container above a named
widget, which is most frames.
**Modular the way input's sense registry is.** `Widgets` gained one
`HashSet<WidgetId>` (`named`), populated only by `.label()`/
`set_label` and drained by `free_next` (the same removal path a
freed id already went through -- no second bookkeeping call added
anywhere). `AccessTree::update` walks `widgets.named()` directly,
never the full widget arena, so a widget nobody named costs this
subsystem nothing -- not a visit, not a branch. Roles come from a
new `Widget::access_role(&self) -> accesskit::Role` trait method,
default `Unknown`; the one override so far is `TextEdit` ->
`TextInput`/`MultilineTextInput` by `EditMode`. Bounds come from
`UiRenderState::window_region`, which sits on `resolved_region`'s
move-chain walk -- so a widget moved via `Offset`/`Scroll` (never
redrawn from scratch) still reports where it actually ended up; see
`bounds_follow_a_moved_widget_and_updates_stay_incremental` below.
**Incremental, not per-frame.** `AccessTree` keeps the last
`HashMap<WidgetId, Entry>` (name, role, bounds) it sent and only
returns a new `TreeUpdate` -- and only then bumps its `rebuilds`
counter, `take_rebuilds()`'s the AccessKit twin of
`UiRenderState::take_counters` -- when that set actually differs.
Confirmed by `bounds_follow_a_moved_widget_and_updates_stay_incremental`
(`iris/src/access_tests.rs`): 1 rebuild on the first draw, 0 across an
unchanged frame, 1 more after a real move, regardless of how many
other widgets are on screen.
**`SlotId::as_u64`** (`core/src/util/slot.rs`) encodes a `WidgetId`
into accesskit's flat `NodeId(u64)`, offset by one so a real widget
never collides with the reserved window node (`NodeId(0)`).
**Pushed through two backends, each behind an inert action/activation
handler** -- see below for why inert is correct, not incomplete.
`default/access.rs` (winit): `accesskit_winit::Adapter`, built in
`DefaultApp::new` with the window created hidden
(`with_visible(false)`) and shown only after the adapter exists,
which is what that constructor requires. `process_event` runs on
every `WindowEvent`; `update_if_active` runs once per
`RedrawRequested`, after `render.update()` so bounds reflect the
frame just drawn. `android/access.rs` (android-view):
`accesskit_android::Adapter` on `AndroidUiState`, `IrisViewPeer` now
implements `AccessibilityNodeProvider`
(`create_accessibility_node_info`/`find_focus`/`perform_action`), and
`render()` (now taking `&mut CallbackCtx`, needed for the JNI handle
any `raise` requires) pushes the same `AccessTree::update` after
every draw.
**Why the `ActionHandler`s are empty, not a placeholder for later
work**: AGENTS.md's own "Driving the UI" section says it plainly --
`ui-trace record --do "tap 'Save'"` resolves the label against the
screen and performs a **real touch at that node's bounds**, the same
as a person's finger. It does not call into AccessKit's action
system at all. So once `AccessTree` reports correct bounds, the
ordinary pointer path (already built, already tested) is what
answers the tap -- there is nothing for `do_action` to do for this
pass condition specifically. A future real screen reader's own
double-tap-to-activate gesture works the same way, for the same
reason. If iris ever needs to answer an AccessKit `Action::Click`
injected without a matching touch (e.g. a switch-access scanner),
that is new scope, not a gap in this box.
**E1's abort mitigation, carried.** `android/access.rs`'s
`raise_if_enabled` is the one place `QueuedEvents::raise` may be
called: it asks `AccessibilityManager.isEnabled()` (a `getSystemService`
JNI call, since android-view has no ready-made wrapper) immediately
before every `raise` and drops the events instead when the answer is
no. Every call site (`render`'s per-frame push, `perform_action`)
goes through it, and each pushes it as a *deferred* callback exactly
like android-view's own demo, so it runs after the current JNI
callback has released whatever it's holding -- `raise`'s own
documented requirement. Not independently re-triggered on this
pass (that needs the emulator, see below); the mitigation is coded
to the exact mechanism E1 diagnosed (`sendAccessibilityEvent`
throwing when accessibility is off) rather than to the symptom, so
there is no reason to expect it behaves differently here than it did
there.
**Verified, 2026-09-05, host only.**
`cargo fmt --all -- --check`, `cargo build --workspace --all-targets`,
`cargo clippy --all-targets` (both plain and `--all-targets`) clean;
`cargo test --workspace` -- 28 tests in `iris/`, three of them new
(`access_tests::a_named_widget_reaches_the_tree_with_its_role_and_bounds`,
`::a_widget_with_no_label_never_reaches_the_tree`,
`::bounds_follow_a_moved_widget_and_updates_stay_incremental`).
`cd iris/android-app && cargo ndk -t x86_64 -P 26 build` and
`... clippy` clean for both `iris` (with the android module) and
`iris-android-app`, same shape as I2/I3's checks.
`iris/run-headless.sh tabs --shot /tmp/iris_i4_tabs.png --seconds 4`
still renders -- **27266 bytes, byte-for-byte identical to I2's own
post-fix screenshot** -- confirming the hidden-window-then-adapter
change to `DefaultApp::new` cost nothing visible. `tabs-ui`'s five
switch buttons (`tabs-ui/src/lib.rs`) now carry `.label()`s matching
their on-screen text ("pad", "span", "image span", "text layout",
"text edit scroll") -- both so the desktop run above exercises a
non-empty tree and so the emulator step below has real names to tap.
Not independently checked on this pass: whether `accesskit_winit`'s
Linux path (AT-SPI, via `accesskit_unix`) actually reaches a real
assistive-technology client on this VM's headless sway -- there is
no AT-SPI registry running here, so `default/access.rs`'s handlers
are exercised as inert code paths (built, called, no panic) rather
than confirmed end-to-end the way the emulator step below confirms
the Android path.
**What remains -- the one check that needs the emulator, held by
another session during this pass.** `iris-android-app`'s tabs screen
has never been driven by `ui-trace` for real; everything above is
"builds, runs, produces the right data" on the host. Once the
emulator is free:
cd iris/android-app && cargo ndk -t x86_64 -P 26 -o app/src/main/jniLibs/ build --release && gradle :app:assembleDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk
# launch iris-android-app on the emulator, then:
ui-trace record --do "tap 'pad'"
ui-trace record --do "tap 'span'"
ui-trace record --do "tap 'image span'"
ui-trace record --do "tap 'text layout'"
ui-trace record --do "tap 'text edit scroll'"
Pass condition: each tap resolves (uiautomator finds a node with
that exact label) and switches `main`'s visible pane the way a
direct touch on that button already does -- i.e. `bench-lib.sh`'s
tap-by-name mechanism, unmodified, driving the iris screen instead
of the Compose one. Also worth checking while the emulator is up,
since E1 found it exactly this way: run a second `ui-trace record`
immediately after the first (attach, detach, attach again) and
confirm the process is still alive afterward -- the detach-abort
this box's mitigation exists for.
- [~] **I5 — the transcript screen in iris (2026-09-05). The widget-tree
half is built, tested and screenshotted; the emulator half (real
device numbers against the Compose baseline) is not -- ticked
partial rather than done, see "What remains" at the end of this box.**
**Where it lives.** `iris/transcript-ui/` (new workspace member,
`[lib]`), the same shape as `iris/tabs-ui`: generic over `Rsc:
HasEvents` + `Rsc::State: FocusHost` so the same `build()` can run
under winit (`transcript-ui/examples/transcript.rs`) or an
android-view cdylib later. Depends on `client-core`/`event-model` by
path (real code, matching E2's precedent) and `pulldown-cmark`
(0.13.4, current stable). Four modules: `markdown.rs` (CommonMark ->
plain text + `Vec<SpanStyle>`), `row.rs` (one `iris::widget::List`
row per folded `TranscriptRow`), `selection.rs` (cross-row
selection), `composer.rs` (the growing input field). `lib.rs`'s own
module doc has the screen's shape and the one gap it documents up
front (below).
**New iris API, added in this box and recorded in `IRIS.md`:
`SpanStyle`, per-range text styling.** This is the actual answer to
RUST.md's E2 finding against Masonry ("rich inline text -- block-level
yes, inline no, and both for the same reason":
`masonry/src/widgets/text_area.rs:43-44`'s `TextArea::edit_styles()`
returns one `StyleSet` for the whole editor, with `// TODO:
RichTextInput` beside it). `core/src/primitive/text.rs`'s
`TextBuffer` gained `spans: Vec<SpanStyle>` and `set_spans`;
`SpanStyle{range, color, family, font_size, bold, italic,
underline}` pushes into parley's `RangedBuilder` via `.push(property,
range)` instead of only `.push_default(...)`, so one `TextEdit` can
carry a heading's bigger bold font, an inline-code span's monospace
colour, a link's colour+underline and an ordinary paragraph's base
style all in the *same* wrapped, selectable buffer.
`core/src/render/atlas.rs`'s `PlacedGlyph` gained a `color: UiColor`
field (read from parley's own per-run `Style::brush`,
`core/src/primitive/text.rs`'s `TextData::place`) and
`core/src/ui/painter.rs`'s `glyphs()` now colours each glyph from
that field instead of one colour for the whole `RenderedText` --
the change that actually makes a span's colour reach the screen.
**Real bug found and fixed while wiring this in**: `TextBuilder`'s
`.spans(...)` was only threaded through `TextOutput::run` (the
read-only `Text` widget), not the sibling `TextEditOutput::run` (the
`TextEdit` every transcript row actually uses) -- a "rule that
governs a set belongs to the set, not one member" miss, per
CODE_RULES.md; found because `run-headless.sh`'s screenshot showed
*no* styling at all despite `markdown.rs`'s own unit tests passing
(they only check the string/range logic, not the render path -- see
`iris/src/widget/text/build.rs`'s `TextEditOutput::run`, now fixed).
**The seven behaviours, each shown or given a sourced reason, same
structure as E2's own accounting:**
1. **Selection spanning rows -- shown, with a scoped shortcut
recorded rather than hidden.** `selection.rs`'s `Selection`
coordinates each visible row's own `TextEditCtx::select`/
`select_all`/`deselect` (already built for one field, I2) from a
single drag that crosses row boundaries: rows between the anchor
and the pointer get `select_all()`, the row under the pointer gets
a true partial selection from whichever edge faces the anchor,
and `selected_text()` concatenates the result in row order. The
one shortcut: the *anchor* row is selected in full once the drag
leaves it, rather than "from the click point to its far edge",
because that needs the row's own laid-out size and
`TextEditCtx`'s `layout()` helper is private
(`iris/src/widget/text/edit.rs`) -- see `selection.rs`'s module
doc. Pure range-membership logic (`in_range`, mirroring
`begin`/`extend`'s row-selection arithmetic) is unit-tested
without any render harness; the widget-level wiring is not
independently screenshotted this pass (would need a synthetic
drag injected into the winit example -- not attempted, time).
2. **Rich inline text -- shown, genuinely inline this time.**
`markdown::render_markdown` folds one row's whole markdown (not
one block at a time) into one string plus spans, so a heading, a
**bold** word, *italic* text, `inline code`, and a
[link](url) inside the same paragraph render in one `TextEdit`
that still wraps and selects as a single buffer --
screenshotted, see below. Deliberately not attempted, each
recorded at the point it would have gone in `markdown.rs`'s own
doc: a background chip behind inline code (needs glyph-run
geometry `TextEdit`-internal and not exposed, the same primitive
`TextEdit::draw`'s selection highlight uses,
`iris/src/widget/text/edit.rs:99`), a tappable link (same missing
primitive), a real table layout, and per-token syntax colour
inside a fence.
3. **Bottom-anchored virtualised list, hold-the-edge on expand --
shown**, reusing I3's `List` unmodified. A `TranscriptRow::Tools`
row collapses to "N tool calls" and expands to every call's own
tool/input/output on tap; `row.rs`'s click handler calls
`List::extent(key)` to convert the tap's row-local position into
the viewport-relative position `List::note_tap` wants, exactly
the two-step contract `list.rs`'s module doc describes for
`holdTopEdge`. Not independently screenshotted mid-expand this
pass (no input-injection into the desktop example was built) --
the mechanism is the same one I3 already benchmarked
(`expand-hold`, flat at 0.10-0.11ms across N), applied to real
content instead of a synthetic row.
4. **The soft keyboard -- inherited from I2, not re-investigated.**
The composer (`composer.rs`) is an ordinary `TextEdit` with the
same `InputConnection` bridge I2 built and measured (Gboard
suggestions over real buffer content); nothing new to add here,
and no Android shell exists yet for this screen specifically to
re-verify it against (see "What remains").
5. **Platform integration -- out of scope by design**, same as E2:
E3's list, not this box's.
6. **Accessibility names -- shown for the composer, not yet for
rows.** The composer field carries `.label("Message")` (I4). Rows
do not yet carry per-row labels (a row's own text *is* its
accessible content via `TextEdit`'s `access_role`, I4, but
nothing calls `.label()` on it, so `Widgets::named()` does not
include it) -- a small, real gap, recorded as an IRIS_TODO.md
item rather than silently left, since AGENTS.md's bench scripts
depend on exactly this for driving a screen by name.
7. **Measurable frames / the render-number pass condition -- not
attempted, and unlike E2 the reason is not an absent gesture
path.** `List` demonstrably scrolls (I3's flat draws/moves,
programmatic `scroll()`) and mouse-wheel scrolling is wired here
(`lib.rs`'s `CursorSense::Scroll` on `list`). What is *not*
reachable yet is a **touch-drag pan starting on a row's own
text**: `row.rs` registers `CursorSense::click_or_drag()` on each
row's `TextEdit` for selection, and `TextEdit::draw` calls
`painter.child_layer()` (`iris/src/widget/text/edit.rs:87`), so
`core/src/sense.rs`'s `run_sensors` (which stops at the first
layer, checked innermost-first, that consumed the gesture) gives
that row first refusal on *every* frame it is pressed, not just
the frame the press started -- a row's drag-select wins the same
gesture a list-level pan would want. `lib.rs`'s own module doc
states this precisely, with the fix named (a press distance/time
arbiter deciding pan vs. select before either commits, or gating
text-drag-selection behind a long-press). This is a genuine,
diagnosed architecture gap this box's *own* two features created
by both wanting the same gesture -- not a missing primitive the
way Masonry's absent `on_pointer_event` drag handling was.
**Verification, exact commands and results (2026-09-05, this VM):**
- `cargo fmt --all -- --check`: clean.
- `cargo build --workspace --all-targets`: clean, all six workspace
members (`iris`, `iris-core`, `iris-macro`, `tabs-ui`,
`transcript-ui`, plus the excluded `android-app`).
- `cargo clippy --all-targets` and `cargo clippy -p transcript-ui
--all-targets`: zero warnings.
- `cargo test --workspace`: 28 tests in `iris`/`iris-core` (all
pre-existing, unaffected) + **9 new in `transcript-ui`** -- 5 pure
markdown tests (`bold_and_italic_produce_spans_over_the_right_range`,
`heading_gets_a_bigger_font_size_span`,
`link_is_styled_and_keeps_its_visible_text`,
`fenced_code_block_is_monospaced`, a plain-text baseline) and 4
selection tests (forward/backward/single-row range arithmetic,
plus `unregister_forgets_the_row_and_clears_a_matching_anchor`
against a real minimal `TextEdit` in the arena, no window needed --
same harness style as `list.rs`'s own tests).
- `cargo ndk -t x86_64 -P 26 build -p transcript-ui` and `... clippy
-p transcript-ui --lib`: clean (`--lib` only -- the example uses
`iris::default`, winit-only by design, same as `iris/examples/
tabs`'s own example never having an Android build of itself; the
Android-facing entry point is a separate cdylib, not built this
pass, see below). `cargo ndk ... build -p iris` / `clippy -p iris`
also re-checked clean, since this box touched `iris-core`'s text
pipeline.
- `run-headless.sh transcript --shot ... -- -p transcript-ui`:
renders. Cropped for legibility (this VM has no image viewer --
see I3's own note on the same limitation and the throwaway crop
tool used here, not committed): a full conversation with a
**bold** word, *italic* text, `inline code` in its own colour, a
`# Sure` heading rendered visibly larger and bold, a coloured link,
a monospaced fenced code block, a collapsed "▸ 3 tool calls" row,
and the composer bar at the bottom -- every one of E2's markdown
screenshot's features, now inline within single paragraphs rather
than block-per-widget. Screenshots at `/tmp/iris_i5_transcript2.png`
(full) and crops there, not committed per the standing rule against
screenshots of real content leaving this repo -- these are
synthetic rows, but the rule is kept uniform regardless.
**What remains, named rather than silently dropped (also in
IRIS_TODO.md, dated 2026-09-05):**
- **The emulator half of the pass condition was not attempted.**
`emu list` shows `emulator-5554` (AVD `ai-app`, a different
checkout) held by another session during this pass, but even with
a free emulator this needs real Android integration that does not
exist yet for this screen: a cdylib + Gradle shell the way
`iris-android-app` wraps `tabs-ui` (I2), real
`client-core::ApiClient`/`event_stream::follow_session_events`
wiring against `app/ui-sandbox.sh` with `--delay` (this crate
deliberately does not fetch anything itself, see `lib.rs`'s doc),
and then `transcript-bench.sh`'s gesture compared against the
Compose baseline. That is real, multi-part follow-on work in its
own right -- closer in size to E2/E3 than to "run one more
script" -- not something this pass's remaining time could
responsibly rush and still report honestly.
- **Touch-drag panning over a row's own text** -- behaviour 7 above.
- **Row-level accessibility names** -- behaviour 6 above.
- **A tappable link and a code-span background chip** -- behaviour 2.
- **`Selection`'s anchor-row shortcut** -- behaviour 1.
- **No syntax highlighting inside a fenced code block** -- `markdown.rs`
notes `client_core::highlight` exists and could feed this.
- **`row.rs`'s tool-row expand and `selection.rs`'s cross-row drag
are not independently screenshotted/driven** -- covered by reading
and by the primitives they reuse (I3's `List` tests, this box's
own unit tests), not by a dedicated repro this pass.
**Net for the recommendation.** Item 3 ("decide when the transcript
screen exists in both, from the measurements") still cannot be
decided by a number -- E2 could not produce one for Masonry, and I5
has not yet produced one for iris either, for an unrelated reason
(no Android harness built yet, not an absent capability). What *can*
be said structurally, updating E2's own conclusion: iris now also
demonstrates the two things E2 found Masonry structurally unable to
do at all -- cross-row selection and true per-span inline rich text
inside one wrapped, selectable buffer -- neither of which exists
anywhere in `masonry`/`masonry_core`/`xilem` today (E2's own
`grep -rln` finding). That is a second structural point in iris's
favour, alongside I2's working touch-scroll-vs-Masonry's-absent one,
still short of the render-number comparison the recommendation
ultimately wants.
## For the next agent
What to do when you pick this up, in order, so nothing here has to be
re-derived:
1. Read this file, then `AGENTS.md` and `PLAN.md`. The rules there
(measure, do not read; fix the rig before accepting its limits; the
emulator is this checkout's own) all apply.
2. Work on the **`rustify`** branch of this clone (`ai-app-2`), not on
`main` and not in `ai-app`. Nothing on this branch is production until
Iris says so. Commit and push as you go.
3. Take the next unchecked box above, in order. E1 has proved android-view
on this emulator, so the E-steps and the I-steps can now proceed in
parallel in separate sessions; see "Where things stand" at the top for
which is next.
4. Every step ends with its measurement written into this file beside the
box, and the box ticked or the reason it could not be written in its
place. A step that is blocked says by what, not "later". Write it as you
go rather than at the end — see "Keep this file current as you work".
5. Run the existing rigs rather than inventing new ones: `ui-sandbox.sh`
for a server with fixtures, `transcript-bench.sh` for the scroll
baseline, `ui-trace` for anything positional, `emu up` for the
emulator, `iris/run-headless.sh EXAMPLE --shot PNG` for an iris
example on this displayless machine, and `rigs/gpu-probe` to ask a
device (this VM, the emulator, or a real phone over `adb push`) what
`wgpu` features and limits it actually has before building anything on
the assumption it does. The Vulkan section below says how to get a
Vulkan path in the emulator when a `wgpu` backend needs one.
6. **Bound anything heavy at the moment you start it.** An emulator or a
long build gets a deadline — `timeout`, or a watchdog scoped to the pid
you just started — rather than a plan to stop it later. Scope it to
that pid: a watchdog written as `sleep N; emu down` fired into a later
experiment here and made a working Vulkan build look like a crash. And
stop the emulator when the work needing it is done rather than between
tasks.
7. Decisions belong here with a date and what was rejected, the way
`PLAN.md` does it. Do not put design into commit messages alone.
### Vulkan in the emulator (measured 2026-09-04)
**Settled 2026-09-04: the guest gets Vulkan from SwiftShader, and the
missing step was a cold boot.** `-feature Vulkan` plus
`VK_DRIVER_FILES=$HOME/Android/Sdk/emulator/lib64/vulkan/vk_swiftshader_icd.json`
gets the *host* side to select SwiftShader, but the guest keeps reporting
zero devices until `-no-snapshot-load` is added, because it boots from a
snapshot saved under the previous GPU config — `-no-snapshot-save` is
worth adding too, so the Vulkan-configured snapshot does not then break
the next ordinary boot. With that, `cmd gpu vkjson` reports SwiftShader
Subzero and wgpu takes its Vulkan path (E1). `EMU_GPU=software` in
`emulator-tools` gets the same guest Vulkan with no GPU use at all, for
work where the emulator's frame rate is not what is being measured.
The rest of this section stands as the record of why host Vulkan is not
available. A `wgpu` app in this emulator was going to get GLES only,
because host Vulkan is switched off in `emulator-tools`. Retried on Mesa
26.1.7: **Venus still fails the same way** — gfxstream picks
`externalMemoryMode: OpaqueFd`, probes `VK_FORMAT_R8G8B8A8_UNORM` for an
exportable colour buffer, and Venus says the format is unsupported
(`Failed to find memory type for ColorBuffers`, fatal before adb sees the
device). Venus does advertise `VK_KHR_external_memory_fd` and
`VK_EXT_external_memory_dma_buf`, so the gap is specifically opaque-fd
image export. gfxstream has a string-valued `VulkanExternalMemoryMode`
setting ("overrides what would otherwise be determined automatically"),
but `-feature Name=Value` is rejected as a bad feature name, and the only
mode words compiled into this emulator's `libgfxstream_backend.so`
(37.1.11) are `OpaqueFd`, `Metal` and `none` — there is no dma-buf mode
in this build to switch to. So Venus is blocked by the emulator, not by
Mesa; retry when the emulator package updates, since upstream gfxstream
does have dma-buf external memory.
What **does** work: pointing the emulator's Vulkan loader at the software
ICDs the emulator ships itself, with the feature enabled:
VK_DRIVER_FILES=$HOME/Android/Sdk/emulator/lib64/vulkan/vk_swiftshader_icd.json \
GPU_HOST_FEATURES="-feature Vulkan" emu up
The guest then reports Vulkan 1.3 (`cmd gpu vkjson`, SwiftShader
Subzero) while GLES still runs on the real GPU through virgl — so a
Vello/wgpu app can take its real Vulkan path here, with compute shaders,
CPU-rasterised. That is enough to test *correctness* of the Vulkan path
in the emulator; GPU *performance* of it is a phone measurement either
way, exactly as `MACHINE.md` already says about frame times. **lavapipe**
(`lvp_icd.json`, the other ICD the emulator ships) selected llvmpipe and
booted, then the emulator died right after loading the `default_boot`
snapshot with nothing in the log; a snapshot saved under a different
Vulkan device is the suspect, and `-no-snapshot-load` is the untested
next step. SwiftShader is the one that works today.
`EMU_GPU=software` is now in `emulator-tools` (agreed with the ai-app
session and with Iris, default unchanged, since `-gpu host` was measured
and the Compose scroll benchmarks depend on it). The cold-boot flags are
not a knob there: that wants snapshot invalidation as well, which is a
bigger design question in shared tooling.
## Things a Rust app changes elsewhere
- **`wg-app-link`'s `:link`** (pinned TLS, enrollment store, QR activity)
is Kotlin shared with Dev Updater. The certificate code already exists on
the Rust side of the submodule; the pinned-CA build step
(`generatePinnedCert`) becomes a `build.rs` reading the same path. The QR
scanner stays a Kotlin activity, since the camera is a platform feature.
- **Tooling** becomes `cargo` for everything but packaging: `cargo test`,
`clippy`, `fmt` cover the whole client, which is the motivation. Gradle
remains for the APK, signing (`~/.config/ai-app/release.jks`) and Dev
Updater's build modes; `build-apk.sh` would call `cargo ndk` first.
- **The bench scripts** (`ui-trace` by accessibility label) keep working
only if the framework exposes names through AccessKit on Android; that is
part of E2's pass condition, not a nicety.
- **Icons** stay Nerd Font glyphs from the committed subset; Parley/Fontique
loads a font file directly, so `build-icon-font.sh` is unchanged.
## Sources
- iced: [repo](https://github.com/iced-rs/iced), [0.14 release](https://github.com/iced-rs/iced/releases/tag/0.14.0), [Android thread](https://news.ycombinator.com/item?id=46350641), [markdown selection request](https://discourse.iced.rs/t/markdown-widgets-text-should-be-selectable/1107)
- Linebender: [2026 Q1 report](https://linebender.org/blog/tmil-25/), [xilem](https://github.com/linebender/xilem), [parley](https://github.com/linebender/parley), [vello](https://github.com/linebender/vello), [vello_hybrid](https://docs.rs/vello_hybrid/latest/vello_hybrid/)
- android-view: [repo](https://github.com/rust-mobile/android-view); android-activity [PR #214](https://github.com/rust-mobile/android-activity/pull/214)
- winit Android IME: [#1823](https://github.com/rust-windowing/winit/issues/1823), [#2766](https://github.com/rust-windowing/winit/issues/2766), [#2305](https://github.com/rust-windowing/winit/issues/2305)
- egui on Android: [discussion #2053](https://github.com/emilk/egui/discussions/2053)
- Slint: [Android guide](https://docs.slint.dev/latest/docs/slint/guide/platforms/mobile/android/), [1.15 release](https://slint.dev/blog/slint-1.15-released), [licensing](https://slint.dev/faqs), rich text [#1325](https://github.com/slint-ui/slint/issues/1325), markdown [#6684](https://github.com/slint-ui/slint/issues/6684)
- Makepad: [repo](https://github.com/makepad/makepad), [makepad-widgets](https://docs.rs/makepad-widgets), [Robrix](https://github.com/project-robius/robrix), [Robrix releases](https://github.com/project-robius/robrix/releases)
- AccessKit: [releases](https://github.com/AccessKit/accesskit/releases)
- Build tools: [cargo-ndk](https://github.com/bbqsrc/cargo-ndk), [cargo-apk](https://github.com/rust-mobile/cargo-apk), [rust-mobile](https://github.com/rust-mobile)
- uniffi: [repo](https://github.com/mozilla/uniffi-rs), [KMP bindings fork](https://github.com/UbiqueInnovation/uniffi-kotlin-multiplatform-bindings)
- GPUI mobile: [gpui-mobile](https://github.com/itsbalamurali/gpui-mobile)
- The earlier Dioxus spike's findings on `wgpu`/Vulkan in this emulator: `~/repos/tdep-survey/app-dioxus/README.md`
Reference in new issue View Git Blame Copy Permalink