Add RUST.md: the plan for moving the app to Rust
Research and measurements from 2026-09-04: what the Compose app has to reproduce, why Android text input and rich selectable text decide the framework, the options considered (Masonry as the yardstick, iris as the in-house library to build up; Slint, iced, egui, Makepad rejected with reasons), how thin the Java shell can be, building the APK without Gradle, and the ordered experiments with pass conditions. Includes the emulator Vulkan findings: Venus is blocked by this emulator's gfxstream, SwiftShader over the emulator's own ICD works. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Fable 5.1 committed
1 parent
1fcaa2d72d
commit
0f8ba49f4a
2 files changed
+594
No files matched your search
@@ -46,6 +46,10 @@ Module-by-module intent is in PLAN.md's "Backend layout".
|
|||||||
before touching `TranscriptCache.kt`, `TranscriptSource.kt`, or the opening
|
before touching `TranscriptCache.kt`, `TranscriptSource.kt`, or the opening
|
||||||
and stream effects in `SessionScreen.kt`.
|
and stream effects in `SessionScreen.kt`.
|
||||||
- `TODO.md` — the working list.
|
- `TODO.md` — the working list.
|
||||||
|
- `RUST.md` — the plan for moving the app to Rust (on the `rustify`
|
||||||
|
branch of the `ai-app-2` clone): what has to be reproduced, the
|
||||||
|
framework decision, and the ordered experiments with their pass
|
||||||
|
conditions. Read it before touching anything under that branch.
|
||||||
- `.dev-updater.ron` — what Dev Updater builds here: the server (run as
|
- `.dev-updater.ron` — what Dev Updater builds here: the server (run as
|
||||||
`service: Managed(…)`, supervised by Dev Updater's own implementation
|
`service: Managed(…)`, supervised by Dev Updater's own implementation
|
||||||
rather than a script kept here) and the APK, in parallel. It points at
|
rather than a script kept here) and the APK, in parallel. It points at
|
||||||
|
|||||||
@@ -0,0 +1,590 @@
|
|||||||
|
# Moving the app to Rust
|
||||||
|
|
||||||
|
Working document for the question Iris asked on 2026-09-04: what are the
|
||||||
|
options for switching the phone app to Rust, ideally pure Rust with one UI
|
||||||
|
framework shared with a future winit-based desktop application, at full
|
||||||
|
feature parity and without giving up anything native, performance
|
||||||
|
especially. Constraints she set: no Dioxus and nothing that draws through a
|
||||||
|
WebView; **no UI DSL** (which rules out Makepad and Slint); the result
|
||||||
|
should stay lightweight; platform-specific pieces are fine to maintain;
|
||||||
|
reimplementing a framework piece from scratch where it does not fit is
|
||||||
|
fine; effort and elapsed time do not matter, long-term robustness does;
|
||||||
|
this clone is where things get tried before anything is committed to
|
||||||
|
`ai-app`. Her own library, [iris](https://github.com/cat16/iris), is the
|
||||||
|
**in-house framework to be built up** for this, with Masonry as the
|
||||||
|
yardstick it is measured against.
|
||||||
|
|
||||||
|
Decisions get a date and a reason here, the way `PLAN.md` does. Nothing in
|
||||||
|
this file has been tried yet unless a section says it has.
|
||||||
|
|
||||||
|
## What has to be reproduced
|
||||||
|
|
||||||
|
The app is ~19,000 lines of Kotlin. It splits three ways, and the split is
|
||||||
|
what decides how much of a port is mechanical.
|
||||||
|
|
||||||
|
**Pure logic with no Compose or Android in it, ~4,500 lines.** `Api.kt`
|
||||||
|
(1,142), `Events.kt`, `EventStream.kt`, `Sse.kt`, `TranscriptCache.kt`
|
||||||
|
(589, touches `java.io.File` only), `TranscriptSource.kt`,
|
||||||
|
`MarkdownSyntax.kt`, `Languages.kt`, `Highlighter.kt`, `Ansi.kt`,
|
||||||
|
`ResetCountdown.kt`, `Durations.kt`, `Sizes.kt`, `ModelName.kt`,
|
||||||
|
`LoadState.kt`, `ImportableStream.kt`. `TranscriptUnits.kt` and
|
||||||
|
`TranscriptItems.kt` (the event fold into rows, ~940 lines) are logic with
|
||||||
|
a handful of Compose annotations. This is also exactly the code that has
|
||||||
|
JVM unit tests today. All of it ports directly, and most of it already has a
|
||||||
|
Rust twin in `server/`: `Events.kt` is a hand-kept mirror of
|
||||||
|
`session/driver.rs`'s enum, the highlighter and the syntax scanner exist on
|
||||||
|
the server for the explorer, and the cache compares the server's own JSON
|
||||||
|
lines. **Sharing these types between server and app is the single largest
|
||||||
|
"keep things in sync" win available, and it does not depend on which UI
|
||||||
|
framework wins.**
|
||||||
|
|
||||||
|
**Compose UI, ~13,000 lines.** Screens, dialogs, the transcript list, the
|
||||||
|
markdown renderer's customisations, tool cards, the file explorer viewer and
|
||||||
|
editor. This is the part a UI framework choice is about.
|
||||||
|
|
||||||
|
**Android platform code, ~1,500 lines**, spread over 20 files. Every one of
|
||||||
|
these is a Java-side object that no Rust framework can replace, because
|
||||||
|
Android only offers them as Java classes:
|
||||||
|
|
||||||
|
- `NotificationService` — a **foreground service** holding the
|
||||||
|
`/notifications` SSE stream while the app is closed, with its ongoing
|
||||||
|
notification, `specialUse` type and the `POST_NOTIFICATIONS` request.
|
||||||
|
- `MainActivity` — edge-to-edge, the `ACCESS_LOCAL_NETWORK` runtime
|
||||||
|
permission (Android 17), `singleTop` intent routing for `aiapp://enroll`,
|
||||||
|
notification taps, and the **share sheet** (`ACTION_SEND`, any MIME type).
|
||||||
|
- `ServerConfig` — the bearer token sealed under an **Android Keystore**
|
||||||
|
AES-GCM key, shared with Dev Updater through `wg-app-link`'s `:link`.
|
||||||
|
- `EnrollmentScanActivity` — the in-app **QR scanner** (zxing, camera).
|
||||||
|
- `Attachments` — `ContentResolver` reads of shared URIs, `BitmapFactory`
|
||||||
|
decode and downscale, **EXIF** orientation.
|
||||||
|
- `SessionImage` — bitmap decode for produced images.
|
||||||
|
- `ScrollAnchor`, `Drafts` — `SharedPreferences`; `CrashLog` — `filesDir`.
|
||||||
|
- `TranscriptCache` — `cacheDir`.
|
||||||
|
- `DebugStats`/`FrameStats` — `Choreographer` frame timing and the render
|
||||||
|
report; `runtime-tracing` names composables in a system trace.
|
||||||
|
|
||||||
|
So **"pure Rust" on Android means Rust owns every line of logic and
|
||||||
|
drawing, behind a thin shell of Java stubs**, and a packaging step that
|
||||||
|
produces a signed APK. How thin, and whether Gradle is inevitable, are
|
||||||
|
answered below.
|
||||||
|
|
||||||
|
### How much Java is unavoidable, and why
|
||||||
|
|
||||||
|
Rust can *call* any Android API through JNI (`jni` crate, with
|
||||||
|
`ndk-context` handing over the `JavaVM` and the Activity): posting a
|
||||||
|
notification, `startForegroundService`, the Keystore, `ContentResolver`
|
||||||
|
reads, permission requests, `WindowInsets`, the clipboard. None of that
|
||||||
|
needs a line of Kotlin. What JNI cannot do is *define* a class that the
|
||||||
|
system instantiates **by name from the manifest** — an `Activity`, a
|
||||||
|
`Service`, an `Application`, a `BroadcastReceiver`. Those must exist as dex
|
||||||
|
bytecode inside the APK before any Rust runs, because the framework
|
||||||
|
constructs them and only then calls into native code. `NativeActivity` is
|
||||||
|
the platform's own stub for the Activity case; there is no
|
||||||
|
`NativeService`, and android-view ships its own `View` subclass for the
|
||||||
|
same reason.
|
||||||
|
|
||||||
|
So the floor is roughly **two Java classes of ten lines each**: an
|
||||||
|
`Activity` and a `Service` whose lifecycle methods are declared `native`
|
||||||
|
and registered from `JNI_OnLoad`, plus whatever android-view already
|
||||||
|
provides. Everything they would have done in Kotlin — insets, intent
|
||||||
|
routing, the SSE follow loop, the notification builder — is Rust reached
|
||||||
|
through those stubs. Writing the stubs in Java rather than Kotlin drops
|
||||||
|
`kotlinc` from the toolchain; `javac` comes with the JDK Gradle already
|
||||||
|
needs. Generating the dex from Rust is not worth it: there is no mature
|
||||||
|
Rust dex writer, and the stubs never change.
|
||||||
|
|
||||||
|
### Can the APK be built without Gradle?
|
||||||
|
|
||||||
|
Yes. An APK is a zip containing a binary-XML `AndroidManifest.xml`,
|
||||||
|
`resources.arsc`, `classes.dex`, `lib/<abi>/*.so` and assets, aligned and
|
||||||
|
signed with the v2 scheme. The tools are `aapt2` (manifest and resources),
|
||||||
|
`d8` (Java bytecode to dex), `zipalign` and `apksigner`, all in the SDK's
|
||||||
|
`build-tools`, none of them Gradle. Three ways to drive them:
|
||||||
|
|
||||||
|
- **A `cargo xtask`** (or `build.rs`-adjacent script) that runs `cargo ndk`
|
||||||
|
for each ABI, `javac` + `d8` for the stubs, `aapt2 link`, `zipalign`,
|
||||||
|
`apksigner`. About 150 lines, every step visible, no AGP, no Gradle
|
||||||
|
daemon holding 2.8 GB between builds. The pinned-CA constant becomes a
|
||||||
|
`build.rs` reading the same `certs/ca.pem` path.
|
||||||
|
- **[cargo-apk2](https://github.com/mzdk100/cargo-apk2)**: the maintained
|
||||||
|
successor to cargo-apk, and unlike it compiles `java_sources` /
|
||||||
|
`kotlin_sources` into the dex and declares multiple activities **and
|
||||||
|
services** with intent filters from `[package.metadata.android]`, with
|
||||||
|
per-profile keystores and optional `aapt2`. Exactly the shape needed;
|
||||||
|
the question is whether a third-party tool with one maintainer beats
|
||||||
|
150 lines we own.
|
||||||
|
- **cargo-apk / xbuild**: unmaintained and `NativeActivity`-only. No.
|
||||||
|
|
||||||
|
What Gradle would take with it: Android Lint (which found two real bugs
|
||||||
|
here, but in Kotlin that would no longer exist — with forty lines of Java
|
||||||
|
stubs there is little left for it to find), manifest merging, R8, and the
|
||||||
|
generated-source plumbing. What it gives back: one toolchain, `cargo`
|
||||||
|
end to end, and Dev Updater keeps calling `build-apk.sh` exactly as now.
|
||||||
|
**Recommendation: the xtask**, with cargo-apk2 read for the details it
|
||||||
|
already got right (v2 signing, `uses-feature`, ABI splits).
|
||||||
|
|
||||||
|
### The behaviours that are hard to get back
|
||||||
|
|
||||||
|
Reading the Compose code for what a replacement must be able to express,
|
||||||
|
rather than what it happens to look like:
|
||||||
|
|
||||||
|
1. **The transcript is one selectable body of text.** One
|
||||||
|
`SelectionContainer` around the whole lazy list, so a selection runs from
|
||||||
|
a reply into the tool output beneath it. The framework needs selectable
|
||||||
|
read-only rich text across many rows, with the platform's selection
|
||||||
|
handles and clipboard on the phone.
|
||||||
|
2. **Rich inline text**: markdown with links (one tap detector per text,
|
||||||
|
not a node per link), inline code chips drawn behind the text, tables
|
||||||
|
with wrapping cells and a sideways scroll, syntax-highlighted fences,
|
||||||
|
ANSI colour in tool output, Nerd Font icon glyphs. Needs a text layout
|
||||||
|
engine with spans, not just styled labels.
|
||||||
|
3. **A bottom-anchored virtualised list of variable-height rows**, paged in
|
||||||
|
both directions (800-event pages, `HISTORY_SCREENS` measured in
|
||||||
|
viewports), with a saved scroll anchor per session, "hold the edge
|
||||||
|
nearest the tap" when a row expands (`holdTopEdge`, done in the layout
|
||||||
|
pass so the wrong frame is never drawn), and rows keyed so that a run of
|
||||||
|
tool calls stays one row while it grows.
|
||||||
|
4. **The soft keyboard**: the composer resizes with the IME, the guard
|
||||||
|
against a stuck inset animation, drafts per session, autocorrect and
|
||||||
|
suggestions from the phone's own keyboard. This is where most Rust
|
||||||
|
frameworks fail on Android today; see below.
|
||||||
|
5. **Platform integration through the app model**: foreground service,
|
||||||
|
notifications, share sheet, deep link, Keystore, camera, back gesture,
|
||||||
|
edge-to-edge insets, local-network permission.
|
||||||
|
6. **Accessibility names on icon buttons**, which the bench scripts depend
|
||||||
|
on (`ui-trace` taps by label). A framework with no accessibility tree
|
||||||
|
also breaks the measuring rig.
|
||||||
|
7. **Measurable frames**: the debug render report, and a way to attribute
|
||||||
|
a frame's cost to a widget on the real phone.
|
||||||
|
|
||||||
|
## The two constraints that decide it
|
||||||
|
|
||||||
|
**1. Android text input.** Every framework built on `winit` inherits
|
||||||
|
winit's Android backend, and that backend cannot drive the soft keyboard
|
||||||
|
properly: the IME tracking issues
|
||||||
|
([#1823](https://github.com/rust-windowing/winit/issues/1823),
|
||||||
|
[#2766](https://github.com/rust-windowing/winit/issues/2766)) are open,
|
||||||
|
`ReceivedCharacter` is unimplemented on Android
|
||||||
|
([#2305](https://github.com/rust-windowing/winit/issues/2305)), and the
|
||||||
|
`android-activity` groundwork for editor actions only merged in February
|
||||||
|
2026 ([PR #214](https://github.com/rust-mobile/android-activity/pull/214))
|
||||||
|
with the winit half still to come. Composition, autocorrect and suggestions
|
||||||
|
need an `InputConnection` implemented on the Java side, which winit's
|
||||||
|
`NativeActivity`/`GameActivity` model does not offer. The frameworks that
|
||||||
|
type on Android today each wrote their own Java glue (Slint, Makepad), and
|
||||||
|
the one designed to do it the way Android intends is
|
||||||
|
[`android-view`](https://github.com/rust-mobile/android-view): a Rust
|
||||||
|
implementation of an Android `View`, with text input through
|
||||||
|
`InputConnection`, accessibility, touch, callbacks on the UI thread, usable
|
||||||
|
either as a whole app or embedded beside ordinary Android components. It is
|
||||||
|
marked WIP. Both Linebender (its Masonry demo lives in that repo) and
|
||||||
|
Robius/Makepad ([Robrix's release notes](https://github.com/project-robius/robrix/releases)
|
||||||
|
say Android lacks a "full" keyboard and they are integrating android-view
|
||||||
|
for it) are converging on it. **That makes android-view the phone-side
|
||||||
|
foundation whichever widget set sits on top**, and the first thing to
|
||||||
|
build and measure here.
|
||||||
|
|
||||||
|
**2. Rich, selectable text and a virtualised list.** Frameworks group by
|
||||||
|
their text stack:
|
||||||
|
|
||||||
|
- **Parley + Fontique + Vello** (Linebender): rich spans, selection and
|
||||||
|
editing utilities, IME support driven through `ui-events`, AccessKit text
|
||||||
|
properties ([Linebender 2026 Q1](https://linebender.org/blog/tmil-25/),
|
||||||
|
[parley](https://github.com/linebender/parley)). Used by Masonry/Xilem,
|
||||||
|
and by Blitz. Vello proper needs compute shaders; `vello_hybrid` (CPU
|
||||||
|
path processing, GPU compositing) is "roughly beta" and runs on GLES too,
|
||||||
|
and Vello CPU exists as a no-GPU fallback.
|
||||||
|
- **cosmic-text** (iced, egui optionally): good layout, but the widgets on
|
||||||
|
top decide selection. iced's `markdown` widget is not selectable
|
||||||
|
([discourse](https://discourse.iced.rs/t/markdown-widgets-text-should-be-selectable/1107)).
|
||||||
|
- **Slint's own**: `TextInput` with `read-only` is the selectable-text
|
||||||
|
trick; there is **no inline rich text at all** (issue
|
||||||
|
[#1325](https://github.com/slint-ui/slint/issues/1325), markdown request
|
||||||
|
[#6684](https://github.com/slint-ui/slint/issues/6684) both open). A
|
||||||
|
markdown transcript with links and code chips cannot be drawn.
|
||||||
|
- **Makepad's own**: GPU/SDF text, a `Markdown` widget and a virtualised
|
||||||
|
`PortalList` in `makepad-widgets`.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
### A. Keep Compose, move the logic into a Rust core (uniffi)
|
||||||
|
|
||||||
|
A `client-core` crate (events shared with the server, API client, SSE,
|
||||||
|
transcript fold, cache, markdown model, highlighter, ANSI) exposed to
|
||||||
|
Kotlin through [uniffi](https://github.com/mozilla/uniffi-rs). Compose keeps
|
||||||
|
drawing. Desktop would be a second UI (iced or Compose Desktop) over the
|
||||||
|
same core.
|
||||||
|
|
||||||
|
- **For**: the logic and the wire types stop drifting from the server
|
||||||
|
today, with tests in one language. Incremental and always shippable.
|
||||||
|
- **Against**: it is not what was asked for. The 13,000 lines of UI stay
|
||||||
|
Kotlin, the desktop app shares no UI code, and the `:link` Kotlin module
|
||||||
|
stays. uniffi's Kotlin Multiplatform bindings are a
|
||||||
|
[community fork](https://github.com/UbiqueInnovation/uniffi-kotlin-multiplatform-bindings);
|
||||||
|
the Android-only bindings are Mozilla's and solid.
|
||||||
|
- **Verdict**: not the destination, but **step one of every other option**
|
||||||
|
is building this crate, so it costs nothing to keep it as the fallback.
|
||||||
|
|
||||||
|
### B. Slint
|
||||||
|
|
||||||
|
Rust on Android is officially supported (minSdk 26, `android-activity`
|
||||||
|
backend, own Java IME glue, safe areas and keyboard insets since 1.15,
|
||||||
|
Skia renderer needs `clang`). Royalty-free licence requires disclosing
|
||||||
|
Slint use; GPLv3 otherwise. UI is a separate `.slint` DSL, not Rust.
|
||||||
|
|
||||||
|
- **Against**: no rich inline text (see above), so the transcript cannot be
|
||||||
|
drawn as it is today; the UI language is not Rust, which forfeits the
|
||||||
|
"compiler catches it" motivation for the half of the code that is UI.
|
||||||
|
- **Verdict**: rejected on rich text alone.
|
||||||
|
|
||||||
|
### C. iced
|
||||||
|
|
||||||
|
Elm-style, Rust-only widgets, desktop-first, `winit` + `wgpu`. Has a
|
||||||
|
`markdown` widget and `rich_text` with links. The maintainer states mobile
|
||||||
|
is a non-goal ([iced](https://github.com/iced-rs/iced)); a community
|
||||||
|
Android example exists and its author could not get the soft keyboard
|
||||||
|
working, patched widgets for touch, and notes no accessibility
|
||||||
|
([HN thread](https://news.ycombinator.com/item?id=46350641)). Markdown is
|
||||||
|
not selectable; `scrollable` is not virtualised.
|
||||||
|
|
||||||
|
- **Verdict**: a fine desktop toolkit and the one Iris named, but every
|
||||||
|
phone-side gap (IME, touch, accessibility, selection, virtualisation)
|
||||||
|
would be ours to build and maintain against a project that does not want
|
||||||
|
them. Not the shared framework.
|
||||||
|
|
||||||
|
### D. egui
|
||||||
|
|
||||||
|
Immediate mode, `winit`-based on Android, AccessKit integration,
|
||||||
|
selectable labels across a `Ui`. Repaints only on input by default, so
|
||||||
|
battery is not the immediate-mode worry. Android IME is blocked on winit
|
||||||
|
([discussion](https://github.com/emilk/egui/discussions/2053)); the
|
||||||
|
workaround is an in-app virtual keyboard, which is exactly the
|
||||||
|
non-native keyboard to avoid. Variable-height virtualised lists are manual
|
||||||
|
(`show_rows` assumes uniform heights). Looks like egui, not Material.
|
||||||
|
|
||||||
|
- **Verdict**: workable on desktop, wrong on the phone for the same reason
|
||||||
|
as iced, plus a look that would need a full custom style.
|
||||||
|
|
||||||
|
### E. Makepad
|
||||||
|
|
||||||
|
GPU-rendered, hybrid retained/immediate, `live_design!` DSL with hot
|
||||||
|
reload, MIT, 1.0 in 2025 ([makepad](https://github.com/makepad/makepad)).
|
||||||
|
Ships Android apps today with its own Java glue; Robrix (a Matrix chat
|
||||||
|
client, the closest analogue to this app) is its reference application on
|
||||||
|
Android, iOS and desktop. Has `Markdown`, `PortalList` (virtualised),
|
||||||
|
`TextInput`. Robrix reports the Android keyboard is not "full" and is moving
|
||||||
|
to android-view for it; the README says non-standard targets "may require
|
||||||
|
minor fixes".
|
||||||
|
|
||||||
|
- **For**: the only option that already ships a chat-shaped app on Android
|
||||||
|
and desktop from one codebase, with the widgets this app needs.
|
||||||
|
- **Against**: the DSL is its own language with its own shader-based
|
||||||
|
styling, so a large part of the UI would not be checked by rustc; the
|
||||||
|
rendering model (SDF everything) is a different world from Compose's,
|
||||||
|
and selection across a `Markdown` widget is unverified.
|
||||||
|
- **Verdict**: **rejected 2026-09-04** — Iris does not want a DSL. Kept
|
||||||
|
here so its Android keyboard status stays a data point about
|
||||||
|
android-view, not as an option.
|
||||||
|
|
||||||
|
### F. Masonry / Xilem on android-view (Linebender)
|
||||||
|
|
||||||
|
Retained widget tree (Masonry) with a reactive view layer (Xilem) that
|
||||||
|
reads like Compose; Rust all the way down; Vello, Parley, Fontique,
|
||||||
|
AccessKit, `ui-events`. Widgets include `Prose` (selectable read-only rich
|
||||||
|
text), `TextArea`, `VirtualScroll`, and this year `Svg`, `Split`,
|
||||||
|
`CollapsePanel`, a new layout system, and IME through `ui-events`
|
||||||
|
independent of winit. `masonry_android_view` exists in the android-view
|
||||||
|
repo and is "not yet generally usable"; Xilem calls itself experimental.
|
||||||
|
Desktop runs on winit. Vello needs a compute-capable GPU or falls back to
|
||||||
|
`vello_hybrid`/CPU.
|
||||||
|
|
||||||
|
- **For**: the only stack where every hard behaviour above maps onto a
|
||||||
|
component designed for it: selection and rich text (Parley/Prose),
|
||||||
|
virtualised variable heights (`VirtualScroll`), native IME
|
||||||
|
(android-view's `InputConnection`), accessibility (AccessKit, now with an
|
||||||
|
Android crate), one Rust widget language on both platforms. The team is
|
||||||
|
the one writing the Android integration everyone else is adopting.
|
||||||
|
- **Against**: pre-1.0 with API churn each release; a small team; no
|
||||||
|
Material widget set, so every control's look is ours; some of the pieces
|
||||||
|
(`masonry_android_view`, `vello_hybrid`) are explicitly unfinished. Being
|
||||||
|
early means fixing things upstream ourselves, which Iris said is
|
||||||
|
acceptable.
|
||||||
|
- **Verdict**: **the option to try first**, because it is the only one
|
||||||
|
whose gaps are "not finished yet" rather than "not designed for this".
|
||||||
|
|
||||||
|
### G. iris — the in-house library, and what "from scratch" means here
|
||||||
|
|
||||||
|
[cat16/iris](https://github.com/cat16/iris), read 2026-09-04 from the one
|
||||||
|
public commit (2026-01-31, "portfolio copy"; ~8,700 lines in `core`,
|
||||||
|
`macro` and the crate itself). Retained-mode widgets stored outside the
|
||||||
|
render tree, `wgpu` 28 directly, `winit` 0.30, `cosmic-text` 0.16, a
|
||||||
|
relative-anchor-plus-offset layout with `rest()` and `rel()` lengths, a
|
||||||
|
postfix builder API (`rect(..).radius(30).on(CursorSense::click(), ..)
|
||||||
|
.sized(..).align(..)`), events handled where the widget is declared, and
|
||||||
|
a single-threaded context passed explicitly — all of which reads like this
|
||||||
|
codebase's own rules. There is text editing (`widget/text/edit.rs`),
|
||||||
|
images, masks, spans and stacks; the TODO names text resizing as
|
||||||
|
per-frame slow and scaling as unsolved. It requires **nightly** (fourteen
|
||||||
|
`#![feature]` gates, among them `const_trait_impl`, `unboxed_closures`,
|
||||||
|
`portable_simd`, `associated_type_defaults`). Desktop only; no Android
|
||||||
|
surface, no IME, no accessibility tree, no virtualised list, no rich-text
|
||||||
|
selection.
|
||||||
|
|
||||||
|
**iris is not a candidate to be tested as it stands. It is the in-house
|
||||||
|
library** (Iris, 2026-09-04): "essentially a good start to a rewrite from
|
||||||
|
scratch", to be maintained and extended by the sessions working here.
|
||||||
|
So the list above of what it lacks is a **work list, not a score**. When
|
||||||
|
the app needs something iris does not have, the answer is to build it
|
||||||
|
into iris. The layer iris has is the widget and layout layer; the layers
|
||||||
|
it needs are the same ones Masonry gets from android-view, Parley and
|
||||||
|
AccessKit, and there is no reason iris cannot sit on those same
|
||||||
|
foundations rather than reinvent them — the surface, the keyboard bridge
|
||||||
|
and the accessibility tree are platform plumbing, not a framework's
|
||||||
|
identity. Whether the text stack stays cosmic-text or moves to Parley is
|
||||||
|
the first real design decision in that work (see I1 below).
|
||||||
|
|
||||||
|
Two things to carry into that work honestly. Nightly is the opposite of
|
||||||
|
"holds up long term": a build that breaks on a toolchain update, on the
|
||||||
|
machine Dev Updater builds on, unattended. Pin a dated nightly in
|
||||||
|
`rust-toolchain.toml` immediately, and keep a list of which `#![feature]`
|
||||||
|
gates are load-bearing so they can be retired as they stabilise or are
|
||||||
|
designed around. And a one-person framework carries every gap itself,
|
||||||
|
which is what Iris said she is willing to do.
|
||||||
|
|
||||||
|
"From scratch" therefore means iris, not a fourth thing. Masonry stays in
|
||||||
|
the plan as the **yardstick and the fallback**: building its demo and its
|
||||||
|
version of the transcript screen first says what a finished stack costs
|
||||||
|
on this hardware, proves android-view before iris depends on it, and
|
||||||
|
gives a comparison that is measured rather than remembered.
|
||||||
|
|
||||||
|
Not considered further: **GPUI** (Zed) mobile is a community fork that
|
||||||
|
depends on unpublished crates; **Dioxus/Blitz** is excluded by Iris (its
|
||||||
|
native renderer is Parley/Vello under HTML semantics, and the earlier
|
||||||
|
`tdep-survey/app-dioxus` spike parked it on a `vello_hybrid` stroke bug and
|
||||||
|
shipped the WebView); **Compose Multiplatform Desktop** would give a desktop
|
||||||
|
app for nothing but in Kotlin, which is the opposite direction.
|
||||||
|
|
||||||
|
### Weight and debug builds
|
||||||
|
|
||||||
|
Iris remembers the Linebender stack being slow in debug. What is behind
|
||||||
|
that is the dependency graph — Vello, wgpu, Parley, Fontique, Skrifa —
|
||||||
|
running unoptimised on the CPU side (path encoding, shaping), not the
|
||||||
|
widget layer. Xilem's own advice is only `split-debuginfo = "unpacked"` to
|
||||||
|
keep `target/` small; the fix everyone with this shape of dependency tree
|
||||||
|
uses is to optimise dependencies while leaving the app crate at `opt-level
|
||||||
|
= 0`:
|
||||||
|
|
||||||
|
[profile.dev.package."*"]
|
||||||
|
opt-level = 2
|
||||||
|
|
||||||
|
E1 measures this rather than remembering it: cold and incremental build
|
||||||
|
time, APK size, resident memory at rest and while streaming, and the
|
||||||
|
frame cost of one 800-event page — for the Masonry demo as shipped, then
|
||||||
|
with the profile above. Vello proper needs compute shaders and carries a
|
||||||
|
large shader set; `vello_hybrid` is lighter and Masonry can now render
|
||||||
|
through either (or Vello CPU) via its `imaging` abstraction, so "keep it
|
||||||
|
light" has a knob inside the same stack.
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
1. **Build `client-core` now, whatever the framework.** A Rust crate holding
|
||||||
|
the event model (shared with `server/` as one crate, ending the
|
||||||
|
`Events.kt` mirror), the API and SSE clients, the transcript fold, the
|
||||||
|
cache, the markdown block model, the highlighter and the ANSI parser,
|
||||||
|
with the existing JVM tests ported. It is the part of the app that is
|
||||||
|
already tested, already logic, and already duplicated on the server.
|
||||||
|
2. **One foundation, two widget layers.** The platform plumbing is shared
|
||||||
|
whichever way the decision goes: android-view for the Android surface,
|
||||||
|
keyboard and accessibility bridge; `wgpu` for the GPU; AccessKit for
|
||||||
|
names; winit on the desktop. On top of it, **Masonry as the yardstick**
|
||||||
|
(E1, E2) and **iris as the thing being built** (I0–I5), both aimed at
|
||||||
|
the same transcript screen with the same pass conditions.
|
||||||
|
3. **Decide when the transcript screen exists in both**, from the
|
||||||
|
measurements, and record the decision here with the numbers. If iris
|
||||||
|
carries the screen within the Compose baseline, it is the app's
|
||||||
|
framework and Masonry was the calibration. If it does not, the
|
||||||
|
measurement says which parts of Masonry to adopt underneath it.
|
||||||
|
4. Then the shell (E3), the desktop window (E4) and the packaging (E5),
|
||||||
|
which do not depend on the choice.
|
||||||
|
|
||||||
|
## Experiments, in order
|
||||||
|
|
||||||
|
Each has a pass condition that is a measurement in this clone. The rig
|
||||||
|
matters: this emulator runs `-gpu host` with **host Vulkan switched off**
|
||||||
|
(`GPU_HOST_FEATURES` in `emulator-tools`, a gfxstream/Venus gap), so inside
|
||||||
|
the guest a `wgpu` app gets GLES, not Vulkan; the earlier Dioxus spike also
|
||||||
|
needed `WGPU_GLES_MINOR_VERSION=1` for compute shaders and found `wgpu`'s
|
||||||
|
Android backend wants API 26 (a libc symbol). The real phone has Vulkan.
|
||||||
|
Per the standing rule, a rig limit is something to fix before it is
|
||||||
|
accepted.
|
||||||
|
|
||||||
|
- [ ] **E0 — toolchain.** No NDK is installed (`~/Android/Sdk/ndk` is
|
||||||
|
empty; the Android Rust targets are). Install the NDK under the
|
||||||
|
user-owned SDK, `cargo-ndk`, and record the versions here.
|
||||||
|
- [ ] **E1 — android-view's Masonry demo on this emulator.** Pass: it
|
||||||
|
builds with `cargo-ndk` + Gradle, renders on the GPU, and the phone's
|
||||||
|
own keyboard types into its editor with autocorrect and suggestions.
|
||||||
|
Note which `wgpu` backend it took and any environment flags needed.
|
||||||
|
- [ ] **E2 — a transcript in Masonry.** One screen: open a sandbox session,
|
||||||
|
page 800 events into `VirtualScroll` bottom-anchored, draw markdown
|
||||||
|
from `pulldown-cmark` into Parley rich text with links and code
|
||||||
|
chips, select across two rows with the platform handles, expand a
|
||||||
|
tool row holding its top edge. Pass: the render numbers land within
|
||||||
|
the Compose baseline in `transcript-bench.sh` on the GPU emulator
|
||||||
|
(same gestures, same session), and every one of the seven behaviours
|
||||||
|
above is either shown or has a written reason it cannot be.
|
||||||
|
- [ ] **E3 — the shell.** Kotlin `MainActivity` + `NotificationService`
|
||||||
|
+ Keystore + share intent calling into Rust over JNI, with the SSE
|
||||||
|
follow loop in Rust. Pass: a notification arrives with the app
|
||||||
|
closed, and a share lands in a session.
|
||||||
|
- [ ] **E4 — the same screen on the desktop** in a winit window, from the
|
||||||
|
same crate, with only the layout differing.
|
||||||
|
- [ ] **E5 — the packaging xtask**: `cargo ndk` → `javac`/`d8` → `aapt2`
|
||||||
|
→ `zipalign` → `apksigner`, signed with the existing release key,
|
||||||
|
installed through Dev Updater. Pass: the APK installs over the
|
||||||
|
Gradle-built one and the notification service starts.
|
||||||
|
|
||||||
|
### The iris track
|
||||||
|
|
||||||
|
These build iris up to carry the app. Each is a feature added to iris
|
||||||
|
with a pass condition, in dependency order. Work in iris's own repository,
|
||||||
|
pinned here by commit, and record in this file what each step measured.
|
||||||
|
|
||||||
|
- [ ] **I0 — bring iris in-house.** Decide where it lives — the
|
||||||
|
recommendation is a repository of its own on the gitea remote,
|
||||||
|
consumed here the way `wg-app-link` is (a submodule pinned to a
|
||||||
|
commit), so the app and the library version-lock and the library
|
||||||
|
stays usable by the desktop app on its own. Pin a dated nightly in
|
||||||
|
`rust-toolchain.toml`; list every `#![feature]` gate with what it is
|
||||||
|
for, so each can be retired when it stabilises. Get `cargo build`,
|
||||||
|
`clippy` and `fmt` clean at the defaults and the `tabs` example
|
||||||
|
running in a window. Pass: a fresh clone builds unattended from the
|
||||||
|
pinned toolchain.
|
||||||
|
- [ ] **I1 — the text stack decision.** iris uses cosmic-text; the
|
||||||
|
transcript needs rich inline spans (links, code chips, colour),
|
||||||
|
selection across many widgets with the platform's handles on the
|
||||||
|
phone, and an editor the IME can drive (composition regions, not
|
||||||
|
just committed characters). Compare cosmic-text and Parley against
|
||||||
|
exactly those three, in iris, on a real transcript's text. Parley
|
||||||
|
is the expectation because android-view's IME bridge and AccessKit's
|
||||||
|
text properties are written against it; measure rather than assume.
|
||||||
|
Pass: a written decision here with what each was tried on, and the
|
||||||
|
TODO's "text resizing per frame is really slow" measured and either
|
||||||
|
fixed or explained.
|
||||||
|
- [ ] **I2 — iris on android-view.** An `android-view` surface as a second
|
||||||
|
backend beside winit: `wgpu` on the view's surface (GLES here, see
|
||||||
|
the Vulkan section; Vulkan on the phone), touch as pointer events,
|
||||||
|
window insets and the keyboard inset as layout inputs, the back
|
||||||
|
gesture as an event, the IME bridge feeding the editor from I1.
|
||||||
|
Pass: the `tabs` example and a text field run on the emulator, and
|
||||||
|
the phone's own keyboard types into the field with autocorrect and
|
||||||
|
suggestions — the same bar as E1.
|
||||||
|
- [ ] **I3 — a virtualised, bottom-anchored list.** Variable-height rows,
|
||||||
|
keyed, composed only while visible, paged in both directions with a
|
||||||
|
"more" sentinel at each end, a scroll anchor that survives rows
|
||||||
|
being inserted above, and "hold the edge nearest the tap" done in
|
||||||
|
the layout pass. Pass: 800 rows of real transcript text from the
|
||||||
|
sandbox scroll without a frame over the Compose baseline in
|
||||||
|
`transcript-bench.sh`, measured on the GPU emulator.
|
||||||
|
- [ ] **I4 — accessibility names via AccessKit.** Every control carries a
|
||||||
|
name; `ui-trace` can find and tap it by label. Pass: `bench-lib.sh`'s
|
||||||
|
tap-by-name works against the iris screen unchanged.
|
||||||
|
- [ ] **I5 — the transcript screen in iris.** E2's pass conditions, all
|
||||||
|
seven behaviours, against the sandbox with `--delay`. This is the
|
||||||
|
point the decision in the recommendation is made at.
|
||||||
|
|
||||||
|
## For the next agent
|
||||||
|
|
||||||
|
What to do when you pick this up, in order, so nothing here has to be
|
||||||
|
re-derived:
|
||||||
|
|
||||||
|
1. Read this file, then `AGENTS.md` and `PLAN.md`. The rules there
|
||||||
|
(measure, do not read; fix the rig before accepting its limits; the
|
||||||
|
emulator is this checkout's own) all apply.
|
||||||
|
2. Work on the **`rustify`** branch of this clone (`ai-app-2`), not on
|
||||||
|
`main` and not in `ai-app`. Nothing on this branch is production until
|
||||||
|
Iris says so. Commit and push as you go.
|
||||||
|
3. Take the next unchecked box above, in order: E0 first, then E1, then
|
||||||
|
the iris track from I0. E-steps and I-steps can proceed in parallel in
|
||||||
|
separate sessions once E1 has proved android-view on this emulator.
|
||||||
|
4. Every step ends with its measurement written into this file beside the
|
||||||
|
box, and the box ticked or the reason it could not be written in its
|
||||||
|
place. A step that is blocked says by what, not "later".
|
||||||
|
5. Run the existing rigs rather than inventing new ones: `ui-sandbox.sh`
|
||||||
|
for a server with fixtures, `transcript-bench.sh` for the scroll
|
||||||
|
baseline, `ui-trace` for anything positional, `emu up` for the
|
||||||
|
emulator. The Vulkan section below says how to get a Vulkan path in the
|
||||||
|
emulator when a `wgpu` backend needs one.
|
||||||
|
6. Decisions belong here with a date and what was rejected, the way
|
||||||
|
`PLAN.md` does it. Do not put design into commit messages alone.
|
||||||
|
|
||||||
|
### Vulkan in the emulator (measured 2026-09-04)
|
||||||
|
|
||||||
|
A `wgpu` app in this emulator was going to get GLES only, because host
|
||||||
|
Vulkan is switched off in `emulator-tools`. Retried today on Mesa
|
||||||
|
26.1.7: **Venus still fails the same way** — gfxstream picks
|
||||||
|
`externalMemoryMode: OpaqueFd`, probes `VK_FORMAT_R8G8B8A8_UNORM` for an
|
||||||
|
exportable colour buffer, and Venus says the format is unsupported
|
||||||
|
(`Failed to find memory type for ColorBuffers`, fatal before adb sees the
|
||||||
|
device). Venus does advertise `VK_KHR_external_memory_fd` and
|
||||||
|
`VK_EXT_external_memory_dma_buf`, so the gap is specifically opaque-fd
|
||||||
|
image export. gfxstream has a string-valued `VulkanExternalMemoryMode`
|
||||||
|
setting ("overrides what would otherwise be determined automatically"),
|
||||||
|
but `-feature Name=Value` is rejected as a bad feature name, and the only
|
||||||
|
mode words compiled into this emulator's `libgfxstream_backend.so`
|
||||||
|
(37.1.11) are `OpaqueFd`, `Metal` and `none` — there is no dma-buf mode
|
||||||
|
in this build to switch to. So Venus is blocked by the emulator, not by
|
||||||
|
Mesa; retry when the emulator package updates, since upstream gfxstream
|
||||||
|
does have dma-buf external memory.
|
||||||
|
|
||||||
|
What **does** work: pointing the emulator's Vulkan loader at the software
|
||||||
|
ICDs the emulator ships itself, with the feature enabled:
|
||||||
|
|
||||||
|
VK_DRIVER_FILES=$HOME/Android/Sdk/emulator/lib64/vulkan/vk_swiftshader_icd.json \
|
||||||
|
GPU_HOST_FEATURES="-feature Vulkan" emu up
|
||||||
|
|
||||||
|
The guest then reports Vulkan 1.3 (`cmd gpu vkjson`, SwiftShader
|
||||||
|
Subzero) while GLES still runs on the real GPU through virgl — so a
|
||||||
|
Vello/wgpu app can take its real Vulkan path here, with compute shaders,
|
||||||
|
CPU-rasterised. That is enough to test *correctness* of the Vulkan path
|
||||||
|
in the emulator; GPU *performance* of it is a phone measurement either
|
||||||
|
way, exactly as `MACHINE.md` already says about frame times. **lavapipe**
|
||||||
|
(`lvp_icd.json`, the other ICD the emulator ships) selected llvmpipe and
|
||||||
|
booted, then the emulator died right after loading the `default_boot`
|
||||||
|
snapshot with nothing in the log; a snapshot saved under a different
|
||||||
|
Vulkan device is the suspect, and `-no-snapshot-load` is the untested
|
||||||
|
next step. SwiftShader is the one that works today.
|
||||||
|
Making this an `emu` option belongs in `emulator-tools` and is a shared
|
||||||
|
tooling change, so it goes through the other sessions first.
|
||||||
|
|
||||||
|
## Things a Rust app changes elsewhere
|
||||||
|
|
||||||
|
- **`wg-app-link`'s `:link`** (pinned TLS, enrollment store, QR activity)
|
||||||
|
is Kotlin shared with Dev Updater. The certificate code already exists on
|
||||||
|
the Rust side of the submodule; the pinned-CA build step
|
||||||
|
(`generatePinnedCert`) becomes a `build.rs` reading the same path. The QR
|
||||||
|
scanner stays a Kotlin activity, since the camera is a platform feature.
|
||||||
|
- **Tooling** becomes `cargo` for everything but packaging: `cargo test`,
|
||||||
|
`clippy`, `fmt` cover the whole client, which is the motivation. Gradle
|
||||||
|
remains for the APK, signing (`~/.config/ai-app/release.jks`) and Dev
|
||||||
|
Updater's build modes; `build-apk.sh` would call `cargo ndk` first.
|
||||||
|
- **The bench scripts** (`ui-trace` by accessibility label) keep working
|
||||||
|
only if the framework exposes names through AccessKit on Android; that is
|
||||||
|
part of E2's pass condition, not a nicety.
|
||||||
|
- **Icons** stay Nerd Font glyphs from the committed subset; Parley/Fontique
|
||||||
|
loads a font file directly, so `build-icon-font.sh` is unchanged.
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- iced: [repo](https://github.com/iced-rs/iced), [0.14 release](https://github.com/iced-rs/iced/releases/tag/0.14.0), [Android thread](https://news.ycombinator.com/item?id=46350641), [markdown selection request](https://discourse.iced.rs/t/markdown-widgets-text-should-be-selectable/1107)
|
||||||
|
- Linebender: [2026 Q1 report](https://linebender.org/blog/tmil-25/), [xilem](https://github.com/linebender/xilem), [parley](https://github.com/linebender/parley), [vello](https://github.com/linebender/vello), [vello_hybrid](https://docs.rs/vello_hybrid/latest/vello_hybrid/)
|
||||||
|
- android-view: [repo](https://github.com/rust-mobile/android-view); android-activity [PR #214](https://github.com/rust-mobile/android-activity/pull/214)
|
||||||
|
- winit Android IME: [#1823](https://github.com/rust-windowing/winit/issues/1823), [#2766](https://github.com/rust-windowing/winit/issues/2766), [#2305](https://github.com/rust-windowing/winit/issues/2305)
|
||||||
|
- egui on Android: [discussion #2053](https://github.com/emilk/egui/discussions/2053)
|
||||||
|
- Slint: [Android guide](https://docs.slint.dev/latest/docs/slint/guide/platforms/mobile/android/), [1.15 release](https://slint.dev/blog/slint-1.15-released), [licensing](https://slint.dev/faqs), rich text [#1325](https://github.com/slint-ui/slint/issues/1325), markdown [#6684](https://github.com/slint-ui/slint/issues/6684)
|
||||||
|
- Makepad: [repo](https://github.com/makepad/makepad), [makepad-widgets](https://docs.rs/makepad-widgets), [Robrix](https://github.com/project-robius/robrix), [Robrix releases](https://github.com/project-robius/robrix/releases)
|
||||||
|
- AccessKit: [releases](https://github.com/AccessKit/accesskit/releases)
|
||||||
|
- Build tools: [cargo-ndk](https://github.com/bbqsrc/cargo-ndk), [cargo-apk](https://github.com/rust-mobile/cargo-apk), [rust-mobile](https://github.com/rust-mobile)
|
||||||
|
- uniffi: [repo](https://github.com/mozilla/uniffi-rs), [KMP bindings fork](https://github.com/UbiqueInnovation/uniffi-kotlin-multiplatform-bindings)
|
||||||
|
- GPUI mobile: [gpui-mobile](https://github.com/itsbalamurali/gpui-mobile)
|
||||||
|
- The earlier Dioxus spike's findings on `wgpu`/Vulkan in this emulator: `~/repos/tdep-survey/app-dioxus/README.md`
|
||||||
Reference in new issue
Block a user