Merge remote-tracking branch 'origin/rustify' into worktree-agent-a27094a7db775552a
# Conflicts: # AGENTS.md # server/src/session/driver.rs
This commit is contained in:
iris committed
commit
88631f5e8b
216 files changed
+52342
-461
No files matched your search
@@ -0,0 +1,146 @@
|
||||
# client-core
|
||||
|
||||
`client-core/` is the app's pure logic held once instead of twice, per
|
||||
RUST.md's recommendation item 1. It is a plain Rust library crate with no UI
|
||||
framework dependency of any kind, so it can outlive whichever one the app
|
||||
ends up drawing with (Masonry, iris, or something else -- see RUST.md).
|
||||
`event-model/` is its sibling: the wire shape both this crate and `server/`
|
||||
share, extracted from `server/src/session/driver.rs` and
|
||||
`session/transcript.rs` on 2026-09-04.
|
||||
|
||||
Neither crate is wired into anything yet. `server/` re-exports `event-model`
|
||||
so its own behaviour is unchanged (`./run-tests.sh` covers it); `client-core`
|
||||
has no caller -- it exists for whichever experiment in RUST.md picks it up
|
||||
next (a Masonry or iris transcript screen, most likely).
|
||||
|
||||
## What's here, and what Kotlin file it replaces
|
||||
|
||||
| `client-core/src/…` | Kotlin original | Status |
|
||||
|------------------------------------------|-------------------------------------------|--------|
|
||||
| `event-model/src/lib.rs` (shared crate) | `Events.kt` (the enum mirror) | Done |
|
||||
| `ansi.rs` | `Ansi.kt` | Done, ported test-for-test |
|
||||
| `highlight/mod.rs`, `languages.rs` | `Highlighter.kt`, `Languages.kt` | Done, ported test-for-test |
|
||||
| `highlight/markdown.rs` | `MarkdownSyntax.kt` | Done, ported test-for-test |
|
||||
| `transcript_cache.rs` | `TranscriptCache.kt` | Done, ported test-for-test |
|
||||
| `sse.rs` | `Sse.kt` (the framing half) | Done, new tests (Kotlin had none of its own beyond integration) |
|
||||
| `api.rs` | `Api.kt` | Partial -- see below |
|
||||
| `event_stream.rs` | `EventStream.kt` | Done |
|
||||
| `transcript_fold.rs` | `TranscriptItems.kt`, `ToolRows.kt` | Partial -- see below |
|
||||
| `config.rs` | `ServerConfig.kt`'s `handleEnrollment` | New, desktop-only so far -- see below |
|
||||
| *(not started)* | `TranscriptSource.kt` | Not started |
|
||||
| *(not ported, and may never be)* | `TranscriptUnits.kt` | Out of scope -- see below |
|
||||
|
||||
Every file above whose Kotlin counterpart had a JVM unit test (`AnsiTest`,
|
||||
`HighlighterTest`, `TranscriptCacheTest`) has had every one of those test
|
||||
cases ported alongside it, plus new tests for the pieces that had none
|
||||
(`sse.rs`, `api.rs`, `event_stream.rs`, `transcript_fold.rs`). Test count by
|
||||
crate as of this writing: **85 in `client-core`**, 0 in `event-model` (its
|
||||
types carry no logic of their own to test -- `server/`'s own tests exercise
|
||||
them via `session::transcript`'s round-trip coverage).
|
||||
|
||||
## Correspondence notes worth knowing before touching either side
|
||||
|
||||
- **`ansi.rs`'s `StyledText`/`Style`/`Rgb`** stand in for Compose's
|
||||
`AnnotatedString`/`SpanStyle`/`Color`, since this crate has no Compose.
|
||||
`StyledText` is plain text plus a `Vec<(Range<usize>, Style)>` of
|
||||
non-overlapping spans. Whatever UI framework ends up consuming this
|
||||
crate maps `Style` onto its own text-styling type; nothing here should
|
||||
change to accommodate a particular one.
|
||||
- **`highlight`'s `Span`/`Kind`** use **char indices, not byte offsets**
|
||||
(`Vec<char>` internally), mirroring the Kotlin original's `Char`-indexed
|
||||
strings. `highlight::span_text` turns a `Span` back into text for a
|
||||
caller working the same way; a caller that wants byte offsets into a
|
||||
`&str` has to convert.
|
||||
- **`transcript_cache.rs`'s `SessionCache::guard`** found a real
|
||||
translation bug while it was being written: an early draft let a
|
||||
*damaged* chunk (one file unreadable, discard just this session) and a
|
||||
genuine I/O failure (disk gone, disable the whole cache) both surface as
|
||||
the same `Err` from one closure, which would have disabled every
|
||||
session's cache over a single corrupt chunk. Fixed by checking a
|
||||
thread-local "was this damage" flag before deciding which failure mode
|
||||
it was -- see the comment on `guard` and the commit message for
|
||||
`transcript_cache.rs`.
|
||||
|
||||
## What `api.rs` covers, and what it does not yet
|
||||
|
||||
`ApiClient` wraps a `Transport` trait (network I/O kept out from behind, so
|
||||
`ApiClient` and `event_stream::follow_session_events` are tested with a
|
||||
fake transport and no server). `UreqTransport` is the only real
|
||||
implementation, backed by `ureq` -- see its Cargo.toml comment for why
|
||||
(blocking, already a project dependency, no extra TLS crate needed since
|
||||
`ureq::tls::Certificate::from_pem` reads the pinned CA directly).
|
||||
|
||||
Covered: session list/read, message send, unqueue, answer, interrupt,
|
||||
stop, start, rename, cwd, model, permission-mode, notify, command,
|
||||
compact, delete, and one transcript page.
|
||||
|
||||
**Not covered, and each is real work rather than a stub to fill in:**
|
||||
setups (`/setups*`, machine and provider discovery), the file explorer
|
||||
(`/setups/{id}/dir|file`), usage (`/usage`), models
|
||||
(`/models*`, HuggingFace browsing and downloads), attachments
|
||||
(`/sessions/{id}/attachments`), importing (`/setups/{id}/importable*`),
|
||||
and the `/notifications` stream. `server/src/routes.rs`'s module doc is
|
||||
the full table to work from when one of these is next.
|
||||
|
||||
## What `transcript_fold.rs` covers, and what it does not yet
|
||||
|
||||
`fold_event` covers every `Event` variant server/ can produce today,
|
||||
including tool-call/question/image attachment and peer-message placement.
|
||||
`group_tool_runs` groups adjacent calls into `TranscriptRow::Tools`.
|
||||
|
||||
**Not ported:** `TranscriptItems.kt`'s `joinPages` (and its
|
||||
`healSplitMessage`/`adoptRun` helpers) -- the page-boundary healing that
|
||||
merges a tool call split across two fetched pages and re-merges a run a
|
||||
boundary cut through. This matters the moment paging backward through
|
||||
history is exercised; it is deliberately left rather than rushed, since
|
||||
it is exactly the kind of boundary logic this project's own "things that
|
||||
have bitten" section warns reads fine and is wrong at the edges.
|
||||
|
||||
**Known gap, and a decision for whoever closes it:** `event_model::Event`
|
||||
has no `Unknown`/catch-all variant, unlike `Events.kt`'s hand-kept mirror.
|
||||
A server newer than this build that adds an event type will fail to parse
|
||||
that line rather than degrading to a placeholder row. Closing this means
|
||||
deciding how `event_model` itself represents "a shape I don't recognise"
|
||||
-- a shared-model decision affecting `server/` too, not a `client-core`-only
|
||||
fix, so it is recorded here rather than silently worked around.
|
||||
|
||||
## `config.rs`: `EnrolledServer`
|
||||
|
||||
`EnrolledServer` (host, port, bearer token) plus `parse_link`, which reads
|
||||
the exact `aiapp://enroll?host=H&port=P&token=T` deep link
|
||||
`wg-app-link`'s `enroll` mints and `ServerConfig.kt`'s `handleEnrollment`
|
||||
parses on the phone -- so any Rust client enrols from the same text a
|
||||
phone would scan as a QR, with no second format invented for it (RUST.md's
|
||||
E4, DECISIONS.md 2026-09-05). Deliberately does not decide where it is
|
||||
persisted or under what file permissions -- a phone seals its token in the
|
||||
Android Keystore, `iris/desktop-app/src/config.rs` writes it to
|
||||
`$XDG_CONFIG_HOME/ai-app-desktop/enrollment.json` at 0600 -- since that is
|
||||
caller-specific (the code rules' "ask for the least you need"). Its only
|
||||
caller today is `desktop-app`; a future Android build of this crate would
|
||||
be a second one, not a reason to move the type.
|
||||
|
||||
## What is not started at all
|
||||
|
||||
- **`TranscriptSource.kt`** -- the layer that decides whether a page comes
|
||||
from the transcript cache or the server, and stitches the two. Needs
|
||||
`transcript_cache.rs` and `api.rs`'s transcript-page method, both of
|
||||
which exist now, so this is unblocked whenever picked up.
|
||||
- **The markdown *block* model beyond syntax spans** -- `highlight/markdown.rs`
|
||||
colours a `.md` file or fence for the highlighter, but does not build the
|
||||
block tree (headings, lists, tables, fences as distinct nodes) that a
|
||||
renderer walks to lay out prose versus code versus a table.
|
||||
`CodeFence.kt`'s use of `org.intellij.markdown` for that full CommonMark
|
||||
AST is Compose rendering plumbing, not something to port as-is; a Rust
|
||||
UI layer will want its own block parser or a crate for it, decided
|
||||
alongside the framework choice in RUST.md.
|
||||
- **`TranscriptUnits.kt`** (see above) -- deliberately out of scope, since
|
||||
it flattens a row into bounded units for a *specific* lazy-list
|
||||
framework's composition cost, which is a fact about that framework
|
||||
rather than about the transcript.
|
||||
|
||||
## Verifying
|
||||
|
||||
`./run-tests.sh` from the repo root now runs `event-model`, `client-core`
|
||||
and `server` in that order (each `cargo test`, forwarding arguments the
|
||||
same way it always has). From `client-core/` directly: `cargo test`,
|
||||
`cargo clippy --all-targets`, `cargo fmt` -- all clean as of this writing.
|
||||
@@ -0,0 +1,226 @@
|
||||
# Decisions taken for Iris to review
|
||||
|
||||
Short list of design choices made by the design agent without asking, so
|
||||
they can be judged and reversed later. Detail lives in RUST.md (and IRIS.md
|
||||
for iris API changes); this file is only the summary. Newest first. Items
|
||||
marked **DEFERRED** are ones the agent chose not to decide alone.
|
||||
|
||||
## 2026-09-05
|
||||
|
||||
- **P0's Compose half is built and smoke-tested on the emulator** — the
|
||||
`bench` build type, the shared `app/bench-fixture/` transcript, and an
|
||||
in-process fake backend (`BenchFixture.kt`/`BenchNetwork.kt`) that
|
||||
answers `TranscriptSource`/`EventStream` from an in-memory event log
|
||||
instead of a real server, so the fold and paging under test are the real
|
||||
ones. Full account, the smoke run's report, and what is deliberately
|
||||
left (the iris half, the real on-phone runs) are in RUST.md's P0 box.
|
||||
Not a decision to review so much as the gate itself now being runnable —
|
||||
flagged here because it is the first half of something Iris explicitly
|
||||
asked to see before P1.
|
||||
|
||||
- **The intermittent touch-scroll dropout is root-caused and fixed: a
|
||||
missed `ACTION_DOWN` hit-test, not the previously-suspected coalesced
|
||||
first `ACTION_MOVE`.** Diagnosed by temporary logcat tracing of every
|
||||
touch event, `DragArbiter` state transition and `Selection::drag`
|
||||
dispatch (removed once confirmed), reproduced on this checkout's own
|
||||
emulator against a real sandbox session. The trace showed the actual
|
||||
mechanism: a gesture's `ACTION_DOWN` lands wherever the finger actually
|
||||
is, which is not guaranteed to fall inside the same row-local sensor
|
||||
region a later `ACTION_MOVE` in the same gesture lands in (a row's own
|
||||
padding/gap, or its non-selectable sender-name header, is
|
||||
pointer-transparent to `iris::sense::CursorSense`). When that happens,
|
||||
the widget that ends up handling the gesture never saw `PressStart`, so
|
||||
`DragArbiter` sits in `Idle` — which answers every subsequent frame with
|
||||
`Undecided` and has no way to tell "no press is happening" from "a press
|
||||
is happening but I missed its start," so it never recovers on its own
|
||||
for the rest of that gesture. One real trace showed exactly this: touch
|
||||
`Down`/`Move`/`Up` all delivered correctly, but zero `PressStart`
|
||||
reaching the arbiter, `state=Idle` unchanged from first frame to last.
|
||||
Fixed at the call site that has the context to recover
|
||||
(`iris::transcript_ui::selection::Selection::drag`,
|
||||
`iris/transcript-ui/src/selection.rs`): a new `DragArbiter::is_idle()`
|
||||
(`iris/src/sense.rs`) lets it notice a `Pressing` frame arriving with the
|
||||
arbiter still `Idle` — which can only mean a missed `PressStart`, since a
|
||||
`Pressing` sense requires the button to genuinely be down — and start the
|
||||
press there instead of where it was missed. Three new unit tests in
|
||||
`sense.rs`'s `drag_arbiter_tests` and one in `transcript-ui`'s
|
||||
`selection::tests` (the latter fails on the code before this fix).
|
||||
Commit follows. Not the same failure the earlier pass's `DECISIONS.md`
|
||||
DEFERRED item speculated about (a coalesced first `ACTION_MOVE` skipping
|
||||
slop detection) — that hypothesis is now ruled out; the arbiter's own
|
||||
slop/long-press logic was never wrong. RUST.md's I5 box,
|
||||
"Touch-scroll dropout root-caused, 2026-09-05" has the full trace.
|
||||
- **P0, a phone benchmark gate before any porting, asked for by Iris
|
||||
2026-09-05**: "before P1 I'd like to see benchmarks & also maybe stress
|
||||
test on my own phone ... If it doesn't match compose reasonably well then
|
||||
I don't think I'd wanna continue." Design (RUST.md's P0 box has the
|
||||
detail): the same embedded synthetic fixture in both apps with no server
|
||||
needed; the same scripted scroll loop then a streaming phase, run
|
||||
programmatically since the phone has no usable system tracing and no
|
||||
agent can drive it; the same report from both (frames, janky %, p50/p90/
|
||||
p99, process CPU time, peak RSS, battery current where readable) with a
|
||||
copy button; the iris app under its own id and the Compose one as a new
|
||||
`bench` build type with an id suffix, so neither replaces her production
|
||||
install; two arm64 APKs plus instructions delivered under `~/host/bench/`.
|
||||
The gate is hers: iris within a reasonable margin of Compose release on
|
||||
p50, p99 and CPU time, no crashes, no visible stutter. If it fails, the
|
||||
port stops.
|
||||
- **The rest of the port is one UI crate, `iris/app-ui`, grown out of
|
||||
`iris/transcript-ui` rather than started beside it.** It holds a
|
||||
`Screen` enum plus a back stack — the Rust equivalent of `AppRoot.kt`'s
|
||||
`when` — and `iris/desktop-app`/`iris/android-app` become thin entry
|
||||
points over it. Chosen over a fresh crate because `transcript-ui`
|
||||
already has the right generic shape (`Rsc: HasEvents` +
|
||||
`Rsc::State: FocusHost`) and the `client-core`/`event-model` path
|
||||
dependencies every later screen needs, so growing it in place is the
|
||||
smaller diff. Platform-only code (notification service, share target,
|
||||
QR scanner, Keystore token, deep-link enrolment) stays in the E3/E5
|
||||
Java shell (`android-shell/` + `app/shellApp`) rather than moving into
|
||||
this crate, since none of it is a screen. The Android APK is built by
|
||||
`cargo xtask apk` (E5), merging the app-ui cdylib into the E3 shell so
|
||||
there is one app rather than a demo shell plus a service shell.
|
||||
`app/androidApp` (the Compose app) stays untouched and is the baseline
|
||||
every step is measured against, until parity is reached (P7 decides
|
||||
the switch, and is itself a load-bearing decision left to Iris). Order
|
||||
is by risk to the daily-use path: session screen first (P1, where
|
||||
every hard behaviour already lives), then the shell merge and a real
|
||||
phone install (P2), then root tabs (P3), the explorer (P4),
|
||||
settings/enrolment (P5), desktop parity (P6), and the cutover itself
|
||||
(P7). Full plan: RUST.md's "The port, in order (decided 2026-09-05)".
|
||||
- **iris gets its own measured frame report, rather than waiting on a
|
||||
`dumpsys`/`gfxinfo` answer that cannot see a `SurfaceView`'s GPU-drawn
|
||||
frames.** `iris_core::FrameReport` (`iris/core/src/render/frame_report.rs`)
|
||||
times each frame's wall clock from the same point `render()`'s redraw
|
||||
starts to just after `queue.submit` + `present()` — the span Compose's
|
||||
own render report and `gfxinfo` both count — into a fixed 4096-entry
|
||||
ring (no allocation per frame; `report()` is the only place that
|
||||
allocates, and only on a button tap). The report gives total frames,
|
||||
janky % over the same 16.7ms budget `gfxinfo` uses, P50/P90/P99 and the
|
||||
worst, plus a reset. Exposed the way the Compose app's copy-button
|
||||
report already is: two named controls ("Frame report", "Reset frame
|
||||
report") on the transcript screen, tappable by accessibility name via
|
||||
`ui-trace`, logging under this crate's fixed `android_logger` tag
|
||||
(`iris-android-app`) so a script can grep `"iris frame report"` the way
|
||||
`transcript-bench.sh` greps `"ai-app render report"`. The report's own
|
||||
`Display` line says plainly that it measures up to the `present()` call
|
||||
returning, not GPU/compositor completion — wgpu's `present()` is not
|
||||
fenced against either, so presenting that span as "time to reach the
|
||||
screen" would be a measured-looking number that is actually inferred,
|
||||
which the standing UI rule forbids.
|
||||
- **`ui-trace` gains a hold-then-drag gesture, additive, in
|
||||
`emulator-tools`.** Neither of its two existing actions can produce
|
||||
"hold stationary for `LONG_PRESS`, then move without lifting" — `tap`
|
||||
has no hold and `swipe X1 Y1 X2 Y2 MS` interpolates motion across its
|
||||
whole duration from t=0. A new action presses, waits, then moves to a
|
||||
second point and releases as one continuous touch (raw
|
||||
`sendevent`/`MotionEvent` injection, extending whatever mechanism the
|
||||
existing `swipe` already uses), so `DragArbiter`'s pan-vs-select rule
|
||||
(`iris/src/sense.rs`, already covered by 8 unit tests against a
|
||||
synthetic clock) can finally be driven on a real device instead of only
|
||||
in a test harness.
|
||||
- **Touch drag on a transcript row follows Android's own rule**: a vertical
|
||||
drag pans the list immediately; a stationary press held 500 ms starts a
|
||||
text selection which further dragging extends; a horizontal drag while
|
||||
something is already selected extends that selection without the wait.
|
||||
One `DragArbiter` per list decides it (`iris/src/sense.rs`). Chosen over a
|
||||
"text layer always wins" or "list always wins" rule because either loses
|
||||
one of the two gestures a reader expects.
|
||||
- **E4's desktop shape is a new `iris/desktop-app` crate**: a winit window
|
||||
holding `transcript-ui`'s screen beside a session list, talking to a real
|
||||
`ai-server` through `client-core`. It enrols by pasting the same
|
||||
`aiapp://enroll?…` link a phone scans (`client-core::config::EnrolledServer`)
|
||||
and keeps it owner-only under `$XDG_CONFIG_HOME/ai-app-desktop/`. The
|
||||
pinned CA is a path given on the command line, not baked in. Chosen so
|
||||
the phone and desktop share one enrolment format and no second one is
|
||||
invented.
|
||||
- **I5's Android integration extends `iris-android-app` (I2's shell)
|
||||
behind a Cargo feature (`transcript-screen`), rather than a third
|
||||
shell crate.** That project already has the Gradle module, the
|
||||
`IrisView`/`MainActivity` Java, and the JNI registration; the only
|
||||
thing a second screen needs on top is a different `AndroidAppState`,
|
||||
the same axis `tabs_ui::build`/`transcript_ui::build` already vary
|
||||
along on the winit side. `tabs-screen`/`transcript-screen` are
|
||||
mutually exclusive and each pulls in only its own deps, so the plain
|
||||
tabs build (I2/I4) is untouched.
|
||||
- **Order of remaining work, updated 2026-09-05**: the two in-flight
|
||||
pieces and I5's Android integration are all done; next is giving iris
|
||||
its own frame-timing report so item 3 below can be decided by a number.
|
||||
- **DECIDED by Iris, 2026-09-05: iris is the app's framework; Masonry was
|
||||
the calibration.** Her words: "I think iris definitely makes more sense
|
||||
based on the limitations we've found." The limitations: Masonry has no
|
||||
touch scroll on Android (E2), no per-span rich text and no cross-row
|
||||
selection on the pinned commit (E2), and its keyboard bridge is a TODO
|
||||
(E1); iris carries the same screen under the Compose baseline on the
|
||||
host GPU (p50 15.0 ms against Compose's 20.0 ms, RUST.md's I5 box). What
|
||||
follows: the E-steps are closed as calibration, and the port proceeds
|
||||
on iris — screens, the shell (E3/E5), and `client-core` underneath.
|
||||
The item below is kept as the record of what she decided from.
|
||||
- **Was DEFERRED — whether to commit to iris over Masonry for `ai-app`.**
|
||||
Updated 2026-09-05 with the clean comparison the recommendation wanted:
|
||||
same sandbox session content, same emulator, `EMU_GPU=software`, one
|
||||
session. Headline numbers (RUST.md's I5 box, "Clean scroll comparison,
|
||||
2026-09-05," has the full table and every caveat):
|
||||
|
||||
| app | build | frames | janky % | p50 | p90 | p99 | worst |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| Compose (in-app report) | debug | 1102 | 99.0% late | 33.8ms | 50.6ms | 79.5ms | -- |
|
||||
| Compose (`dumpsys gfxinfo`) | debug | 1499 | 21.15% (95.66% legacy) | 32ms | 48ms | 150ms (p99) | -- |
|
||||
| iris (`FrameReport`) | **release** | 299 | 94.65% | 79.1ms | 98.6ms | 117.8ms | 212.6ms |
|
||||
| iris (`FrameReport`, repeat) | **release** | 233 | 94.42% | 109.3ms | 130.8ms | 147.1ms | 150.5ms |
|
||||
|
||||
**Not a clean apples-to-apples reading, stated plainly rather than
|
||||
smoothed over**: iris had to be built **release** (debug `SIGSEGV`s on
|
||||
this emulator's Vulkan loader, I4's finding) against Compose's mandated
|
||||
**debug** build, so this asymmetry likely *understates* iris's gap
|
||||
rather than the reverse; the three frame-time sources measure different
|
||||
things (Compose's own phase accounting vs. Android's HWUI deadline-miss
|
||||
definition vs. iris's redraw-start-to-present window, the last of which
|
||||
`dumpsys gfxinfo` cannot see at all for iris's `SurfaceView`); and both
|
||||
figures are emulator numbers under software rasterisation, which
|
||||
Compose's *own* in-app report shows already costs 20-34ms/frame in
|
||||
`swap`+`gpu` alone under this GPU mode, so a same-mode iris number well
|
||||
above 16.7ms was expected going in for either app. A second pair under
|
||||
`-gpu host` was not taken this pass. The earlier session's suspected
|
||||
intermittent touch-delivery dropout was **not reproduced** this pass —
|
||||
the zero-frame results this time traced to this pass's own script bug
|
||||
(a `cd` that changed which emulator `ui-trace` targeted), not the
|
||||
emulator; a CPU-load rise during the gesture was observed by a sampler
|
||||
running throughout, but did not correlate with any failure, so the
|
||||
original candidate is neither confirmed nor ruled out.
|
||||
The choice in front of Iris, updated: decide now on the
|
||||
structural-plus-functional case already made (iris works end-to-end
|
||||
where Masonry's scroll gesture doesn't exist at all on Android) plus
|
||||
this table — reading the two build profiles and three jank definitions
|
||||
with the caveats above rather than as a single number — or ask for a
|
||||
same-profile, same-GPU-mode rerun first. RUST.md's I5 box has the full
|
||||
account.
|
||||
|
||||
**Updated 2026-09-05, the `-gpu host` pair taken.** Real GPU rendering
|
||||
(`force-gles` -- the default Vulkan backend has no adapter at all under
|
||||
plain host-GPU boot, confirmed by the exact `wgpu` error) reverses the
|
||||
software-mode shape:
|
||||
|
||||
| app | build | GPU mode | frames | janky % | p50 | p90 | p99 | worst | cpu p50 | gpu-wait p50 |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| Compose (in-app report) | debug | host (virgl) | 1268 | 96.4% late | 20.0ms | 28.4ms | 37.7ms | -- | -- | -- |
|
||||
| iris (`FrameReport`) | **release**, `force-gles` | host (virgl) | 62 | 41.94% | 15.0ms | 21.8ms | 37.1ms | 37.1ms | 0.2ms | 12.9ms |
|
||||
|
||||
Under real GPU rendering iris's median frame is *faster* than
|
||||
Compose's, not the 2-3x-slower shape the software-mode table shows. A
|
||||
new split inside `FrameReport` (redraw-to-submit vs. submit-to-present,
|
||||
commit `e2a1fad`) says why: iris's own CPU work per frame is a median
|
||||
0.2ms -- almost the entire frame is time spent handing the frame to the
|
||||
driver, not in iris's layout/text/primitive code. This is consistent
|
||||
with the earlier software-mode gap being mostly SwiftShader's CPU
|
||||
rasterisation cost rather than an iris-specific slowness, but is not
|
||||
proof of it: a same-mode software `force-gles` run to isolate the
|
||||
backend crashed for an unrelated reason (SwiftShader's GL path reports
|
||||
itself as OpenGL ES 3.0, which has no compute shaders, and iris's device
|
||||
request assumes them unconditionally) — real scope to fix, not done
|
||||
here — and the two apps' frame populations still differ in kind the same
|
||||
way the software-mode caveats describe. A real intermittent touch-
|
||||
scroll dropout was also reproduced this pass (six consecutive swipes
|
||||
produced zero redraws while taps kept working; an identical retry then
|
||||
succeeded) and is not explained. RUST.md's I5 box, "Where iris's frame
|
||||
time goes, 2026-09-05, the `-gpu host` pass," has the full account. The
|
||||
iris-vs-Masonry choice itself is still Iris's to make.
|
||||
@@ -0,0 +1,351 @@
|
||||
# The file explorer
|
||||
|
||||
Asked for by Bryan on 2026-09-03 and built the same day: browse a machine's
|
||||
directories, open files with the existing syntax highlighting and line
|
||||
numbers, edit behind a pencil, create through a modal, work over ssh, and
|
||||
open at the session's working directory.
|
||||
|
||||
This is the design, decision by decision with the reason and what was
|
||||
rejected, so that when one changes it is changed here rather than re-argued.
|
||||
The operational half — how to run it and what to produce on purpose — is in
|
||||
AGENTS.md. `server/src/files.rs` is the backend and `FilesScreen.kt` /
|
||||
`FileViewer.kt` / `FileEditor.kt` / `FileLines.kt` are the app.
|
||||
|
||||
## What it is, in one paragraph
|
||||
|
||||
A machine's filesystem, seen from the phone through the backend. The explorer
|
||||
belongs to a **setup** (a machine), not to a session: a session only says
|
||||
where to start. Every operation — list, read, write, create — is one shell
|
||||
script run through `Transport`, exactly the way the import listing and the
|
||||
usage fetch already work, so the local and the ssh case are one
|
||||
implementation and a machine the backend cannot reach fails with ssh's own
|
||||
message. The phone draws what came back.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Keyed on the machine, opened from the session
|
||||
|
||||
Routes live under `/setups/{id}/…`, beside `importable`, because a filesystem
|
||||
is a property of a machine. The session screen's folder button opens the
|
||||
explorer with the session's setup and its `cwd`; a session with no `cwd`
|
||||
opens at the machine's home, which the **machine** resolves (`cd` with no
|
||||
argument and `pwd -P`), never a path the phone guessed. Nothing in the
|
||||
explorer knows what a session is, so a later entry point from the setups tab
|
||||
is one more caller and no new code.
|
||||
|
||||
Rejected: routes under `/sessions/{id}/`. The session would be a detour to
|
||||
find the setup, and "browse this machine" from anywhere else would need a
|
||||
session to exist first.
|
||||
|
||||
### 2. One shell script per operation, over `Transport`, on both transports
|
||||
|
||||
Each operation is a small POSIX script handed to `sh -c script sh "$path" …`
|
||||
through `Transport::capture` (or `capture_with_input`). The path and every
|
||||
other value cross as **positional arguments**, never interpolated into the
|
||||
script — the same rule `import::find` follows and the same reason
|
||||
`ssh::quote` exists: a path is attacker-adjacent input in a server whose job
|
||||
is running commands. `PATH_PRELUDE` is the one line that gives a leading `~`
|
||||
its meaning, since a shell expands a tilde in text and not in an argument.
|
||||
|
||||
The scripts assume GNU coreutils and findutils (`find -printf`, `stat -c`,
|
||||
`sha256sum`, `chmod --reference`) — already what `import.rs` assumes, and
|
||||
both machines that exist are Linux. A machine without them fails with that
|
||||
tool's own message, which names what is missing.
|
||||
|
||||
Rejected: `std::fs` for the local transport and scripts for ssh. Two
|
||||
implementations of "list a directory" drift — the ordering of entries, what a
|
||||
symlink reports, how a permission error reads — and the local one is the one
|
||||
that gets tested, so the remote one ships broken. The cost is an `sh` process
|
||||
per operation locally, which is under a millisecond.
|
||||
|
||||
Rejected: a Rust SSH or SFTP library. The system `ssh` inherits
|
||||
`~/.ssh/config`, agents and jump hosts, and there is one place to configure a
|
||||
connection; SFTP would need a second.
|
||||
|
||||
### 3. The token can now name a path, and that is written down
|
||||
|
||||
Elsewhere the phone picks an **id** and the server resolves which file it
|
||||
names, so an enrolled token cannot become "read me an arbitrary file". The
|
||||
explorer's whole purpose is the path, so it takes one. Recorded in PLAN.md's
|
||||
Security section in these terms: the token already gates spawning a
|
||||
bypass-permissions agent in any directory on any machine a setup names, and
|
||||
that agent can already read and write every file its user can. The explorer
|
||||
is a shorter path to authority the token already holds, not new authority.
|
||||
The import rule stands where it is, because there a path was unnecessary and
|
||||
refusing it cost nothing.
|
||||
|
||||
What is *not* changed: **no route accepts a command.** Listing, reading and
|
||||
writing are fixed scripts; the phone chooses only the path and the bytes.
|
||||
|
||||
### 4. Paths are absolute or `~`-prefixed, and the machine answers with the real one
|
||||
|
||||
Same rule as `POST /sessions/{id}/cwd`, with the same wording, because where
|
||||
a relative path would be depends on something the reader cannot see. Every
|
||||
listing answers with `pwd -P` of the directory it listed, so the phone
|
||||
navigates on a resolved absolute path — the parent is a string operation on
|
||||
that, and a `~` the session was spawned with is shown as what it turned out
|
||||
to be. The phone never resolves `..` itself.
|
||||
|
||||
### 5. A read is capped and typed, and every state it can be in has a word
|
||||
|
||||
`GET /setups/{id}/file` answers with one of `text` (content, size, mtime,
|
||||
sha256), `binary` (not UTF-8; size reported, nothing shown), `tooBig` (over
|
||||
`FILE_LIMIT`, 1 MiB; size reported so the reader knows what they are looking
|
||||
at), or the machine's own error.
|
||||
|
||||
Four outcomes rather than content-or-error, because a binary file drawn as
|
||||
text and a big file cut off silently are both wrong in ways the reader cannot
|
||||
see, and "couldn't read it" must not look like "it is empty". An empty file
|
||||
is `text` with empty content, drawn as one empty line numbered 1, which is
|
||||
what it is.
|
||||
|
||||
### 6. A write is conditional on what the reader saw
|
||||
|
||||
`PUT /setups/{id}/file` carries the sha256 the read reported. The script
|
||||
compares it against the file as it is now and exits distinctly if it differs;
|
||||
the server answers **409**. Agents edit files while people read them; this is
|
||||
the common case, not the exotic one, and silently overwriting an agent's edit
|
||||
with a stale copy is the worst available outcome. The phone offers three ways
|
||||
out and says what each costs: **Overwrite** (theirs is lost), **Reload**
|
||||
(yours is lost), **Cancel** (keep editing).
|
||||
|
||||
The write is `cat > "$1.ai-app-tmp" && chmod --reference="$1" … && mv -f`,
|
||||
with the bytes on stdin: a temp file and a rename, so a connection dropped
|
||||
mid-write leaves the old file whole rather than truncated, and
|
||||
`chmod --reference` keeps the mode a fresh file would lose (an executable
|
||||
script would stop being one). What this trades away is the inode, so a hard
|
||||
link elsewhere stops being the same file — accepted; editors do the same. The
|
||||
check-then-write is not atomic against a writer landing between the two, a
|
||||
window of microseconds on the same machine; accepted, and noted at the
|
||||
script. The response carries the new size, mtime and sha256, so the editor's
|
||||
precondition is fresh without a second read.
|
||||
|
||||
### 7. Create refuses to overwrite
|
||||
|
||||
`POST /setups/{id}/file` runs under `set -C` (noclobber) and `: > "$1"`, so a
|
||||
name that exists fails with the shell's own message rather than truncating
|
||||
somebody's file; `POST /setups/{id}/dir` is `mkdir --` with the same
|
||||
property. The modal names one thing in the current directory and has a switch
|
||||
for "directory"; a created file opens straight into edit mode, because an
|
||||
empty file is not something to look at.
|
||||
|
||||
Rejected: create-with-content in one request. The editor is where content is
|
||||
typed, and a modal with a text area is a second editor.
|
||||
|
||||
### 8. The viewer is a list of lines, coloured once
|
||||
|
||||
The file is scanned once, **off the main thread**, by `scan` in
|
||||
`Highlighter.kt`; the spans are bucketed per line in one pass and each line's
|
||||
`AnnotatedString` is built when that line is composed. A `LazyColumn` of
|
||||
lines, not one `Text`: text layout is linear in the text, so a 20,000-line
|
||||
file in one `Text` measures all of it to draw a screenful.
|
||||
|
||||
**Every row is given the same width**, and that is what makes the shared
|
||||
horizontal scroll work. `horizontalScroll` is a node per row, and each one
|
||||
coerces the shared offset into *its own* range — content width less viewport
|
||||
— so with rows at their natural widths a short line's range is zero and it
|
||||
does not move at all while the long line beside it does. Each row also writes
|
||||
`maxValue` as it measures, so how far the file could be dragged was decided
|
||||
by whichever row measured last and changed as the list scrolled. The width is
|
||||
the longest line in columns times one character's advance, which is
|
||||
arithmetic rather than twenty thousand measurements because the face is
|
||||
monospace. A tab counts as eight columns and deliberately upwards —
|
||||
over-estimating leaves a little empty space past the longest line,
|
||||
under-estimating puts the end of that line out of reach — and the width is
|
||||
capped well under what `Constraints` can carry, so a minified file is a
|
||||
scroll that stops early rather than a crash. Reported by Iris on 2026-09-04
|
||||
as "it seems to affect different rows differently", which is precisely what a
|
||||
per-row range looks like.
|
||||
|
||||
**The stretch at the ends is one effect too**, shared by every row and
|
||||
rendered once on the box around the list — `horizontalScroll` makes its own
|
||||
per node otherwise, so only the line under the finger bent while the rest of
|
||||
the file sat still. It cannot be seen from this VM: the emulator's
|
||||
screenshots come back with no stretch in them at all, for any scrollable, so
|
||||
that one is checked on the phone.
|
||||
|
||||
**The numbers sit outside that box**, so they neither travel with the text
|
||||
nor bend with it. The rows leave a spacer where the numbers go and a
|
||||
`SubcomposeLayout` beside the list draws them. That is the one arrangement
|
||||
that keeps them level: which numbers exist *and* where each goes both come
|
||||
from the list's own `layoutInfo`, read in the measure block, and
|
||||
subcomposition happens during measurement — so it composes from the answer
|
||||
the list has just produced rather than one it read a frame ago. A column
|
||||
translated by the scroll position could not, since the translation would be
|
||||
current while the set of numbers was a composition behind, and during a fling
|
||||
the numbers would slide against their lines. Checked at about 1kHz through a
|
||||
fling: 23,520 row observations over 552 frames, every one with its number at
|
||||
exactly its own top. A consequence worth having: the numbers are outside the
|
||||
`SelectionContainer`, so copying part of a file gives the code rather than
|
||||
the code with a number in front of every line.
|
||||
|
||||
The gutter is right-aligned, its width taken from the digit count of the line
|
||||
count in the same monospace style, so a 9-line file and a 12,000-line file
|
||||
each get exactly the width they need and nothing is measured by hand. Because
|
||||
nothing wraps, a logical line is one visual line and the gutter cannot drift
|
||||
from the text it numbers. Numbers take `onSurfaceVariant`; the text takes the
|
||||
scanner's palette on `rawSurface`, the surface every verbatim thing in the
|
||||
app already sits on.
|
||||
|
||||
The language comes from the file's extension through the same table
|
||||
`fenceLanguage` reads — one table, not two, so a language added for fences is
|
||||
added for files. A file with no entry is drawn plain.
|
||||
|
||||
### 9. The editor is the legacy text field with a highlighting transformation
|
||||
|
||||
Edit mode swaps the viewer for a `BasicTextField(TextFieldValue)` in the same
|
||||
monospace style, inside the same horizontal scroll so it does not wrap, with
|
||||
a `VisualTransformation` that returns the text unchanged and the scanner's
|
||||
spans as styles (`OffsetMapping.Identity`, since no character moves). This is
|
||||
the one Compose API that colours a field's text without replacing the field;
|
||||
the newer `TextFieldState` API has no hook for styles. The gutter is one
|
||||
`Text` of `1\n2\n…` beside the field, aligned for the same reason as the
|
||||
viewer.
|
||||
|
||||
Save is a glyph in the header, **disabled** until the text differs from what
|
||||
was loaded — never hidden, since a control that comes and goes makes its own
|
||||
absence the signal. Back with unsaved changes asks, and says the edits will
|
||||
be lost. The explorer draws over the session, which deliberately has no
|
||||
`imePadding`, so the explorer's own box adds it.
|
||||
|
||||
### 10. The explorer draws over the session, and back closes it first
|
||||
|
||||
`Screen.Session` in `AppRoot` gains a `files: FilesTarget?`. When set, the
|
||||
`FilesScreen` is composed **on top of** the session in the same `Box`, and
|
||||
the session stays composed under it: its event stream keeps flowing, its
|
||||
scroll position and draft stay where they were, and returning from a file
|
||||
costs nothing. Back — the button and the platform gesture — clears `files`
|
||||
when set and goes to the list otherwise. Inside the explorer the same back
|
||||
steps one level: editor → viewer (with the unsaved question) → listing →
|
||||
parent directory, and only from the starting directory does it close. "Back
|
||||
returns; it does not exit."
|
||||
|
||||
Rejected: a `Screen.Files` beside `Screen.Session`. Every route back from a
|
||||
leaf screen goes to Main today, and a session disposed and re-created on each
|
||||
return refetches its transcript over the tunnel — exactly the flip between
|
||||
"what did it change" and "what is it saying" this feature is for. The image
|
||||
viewer already made the same choice for the same reason.
|
||||
|
||||
### 11. The listing is drawn as it came, sorted at display time
|
||||
|
||||
Entries carry name, kind (`directory`, `file`, `other`), size, mtime, and
|
||||
whether the entry is a symlink — with the kind being the *target's*, from
|
||||
`find -printf '%Y'`, so a link to a directory navigates. Sorted on the phone,
|
||||
stably: directories first, then case-insensitive name. Dotfiles are shown; in
|
||||
a repository they are half of what matters. Each directory's entries are kept
|
||||
for as long as the explorer is open, keyed by path, so returning to one does
|
||||
not refetch it; the header's refresh glyph refetches the current one on
|
||||
purpose, and a create refetches the directory it created into, since that is
|
||||
what the operation changed.
|
||||
|
||||
An empty directory says "Nothing here". A listing that failed says why, in
|
||||
the machine's words, where the rows would be — never an empty list.
|
||||
|
||||
Entries are separated by `\0` in the script's output and by `\t` within a
|
||||
line, so a filename with a newline or a tab in it survives; `parse_entries`
|
||||
is a unit test with exactly those names in it.
|
||||
|
||||
### 12. Icons
|
||||
|
||||
Added to `NerdIcons.kt` **and** `build-icon-font.sh`, then the script rerun
|
||||
and its output committed: `md-folder` U+F024B (the header button and
|
||||
directory rows), `md-plus` U+F0415, `md-pencil` U+F03EB,
|
||||
`md-content_save` U+F0193, `md-file_outline` U+F0224. The folder and the plus
|
||||
are the same codepoints dev-updater uses and must not drift from it, as the
|
||||
cog and the refresh arrow already must not. All five were looked up in Nerd
|
||||
Fonts' own `glyphnames.json` rather than copied from memory, which is the
|
||||
check that a codepoint means the glyph its comment names.
|
||||
|
||||
**The folder button sits between the usage chart and the cog**, so the header
|
||||
reads widest scope to narrowest and the cog stays at the end where every
|
||||
other screen keeps it. Asked for in that order by Iris on 2026-09-03.
|
||||
|
||||
### 13. The render report moved, and the benches moved with it
|
||||
|
||||
The speedometer went; the report is a "Copy render timings" row in
|
||||
`SessionSettingsDialog`, where the session's other about-the-session controls
|
||||
already are. **Moving it is where the no-coordinate-taps rule got enforced**
|
||||
(Bryan, 2026-09-03) — see AGENTS.md's "Driving the UI".
|
||||
|
||||
## HTTP surface
|
||||
|
||||
In `routes.rs`'s module doc with the rest. Bodies use `deny_unknown_fields`
|
||||
like every other body here; paths in the query string are URL-encoded by
|
||||
`Api.kt`'s existing helper.
|
||||
|
||||
```json
|
||||
GET dir -> {"path":"/home/bob/repos/ai-app",
|
||||
"entries":[{"name":"app","kind":"directory","size":4096,"modified":1756900000,"link":false}]}
|
||||
GET file -> {"path":"/…/x.rs","kind":"text","size":1234,"modified":…,"sha256":"…","content":"…"}
|
||||
| {"path":"/…/a.png","kind":"binary","size":45678,"modified":…}
|
||||
| {"path":"/…/big.log","kind":"tooBig","size":12345678,"modified":…}
|
||||
PUT file -> {"size":1240,"modified":…,"sha256":"…"}
|
||||
```
|
||||
|
||||
Errors: `BadRequest` with the machine's message for a path that is not there,
|
||||
not allowed or not absolute; 409 for the precondition; `Internal` only for
|
||||
the server's own faults. The message is what the phone shows, in place, so it
|
||||
is written to be read there.
|
||||
|
||||
## What the measurements said (2026-09-04)
|
||||
|
||||
Taken on the emulator in a **debug** build, which runs Compose at a fraction
|
||||
of release speed and renders in software — so these rank correctly against
|
||||
each other and are pessimistic in absolute terms. Generated Rust, through the
|
||||
app's own render report.
|
||||
|
||||
| file | lines | scan + cut | scan per keystroke | worst frame record |
|
||||
|--------|--------|------------|--------------------|--------------------|
|
||||
| 32 kB | 917 | 11ms | 10ms | 183ms |
|
||||
| 128 kB | 3,633 | -- | 40ms | 2,027ms |
|
||||
| 1 MB | 28,660 | 460ms | -- | -- |
|
||||
|
||||
Three things followed.
|
||||
|
||||
**The viewer's scan had to leave the main thread.** Decision 8 said "off the
|
||||
main thread" and the first version did it in a `remember` inside the
|
||||
composition, which is not that: 460ms of frozen screen at the size the server
|
||||
is willing to send, long enough that the accessibility tree cannot be read —
|
||||
which is exactly what "the app has stopped" looks like from outside.
|
||||
|
||||
**`FILE_LIMIT` at 1 MiB is right for reading.** Time to first line for a
|
||||
1 MiB file, tap to text on screen, was **2.4s** against the sandbox — 1.2s of
|
||||
which is that server's deliberate `--delay`, and 460ms the scan. The transfer
|
||||
is not what dominates, so the route gains nothing from streaming.
|
||||
|
||||
**Edit mode needed a cap, and not the one that was expected.** The plan
|
||||
expected to be deciding a size below which highlighting stays on. That is not
|
||||
the cost that matters: highlighting 128 kB costs 40ms a keystroke, which is
|
||||
survivable, while laying the same text out in one `BasicTextField` costs two
|
||||
seconds — characters typed into it were dropped, and a 1 MiB file stopped the
|
||||
app responding altogether. Since every arrangement of a single text field
|
||||
pays that, switching highlighting off would have saved nothing. So
|
||||
`EDIT_LIMIT` is **32 kB**, the largest size measured as usable, and above it
|
||||
the pencil is disabled with the reason said in words beside it — a disabled
|
||||
control teaches what the thing can do but cannot say why it is off, and a
|
||||
reader who cannot edit a file they can plainly read would otherwise conclude
|
||||
the app is broken.
|
||||
|
||||
Reading is unaffected: the viewer opens and scrolls the 1 MiB file fine,
|
||||
because it is a `LazyColumn` of lines rather than one text object. That
|
||||
difference is the whole of decision 8.
|
||||
|
||||
## Later, deliberately not now
|
||||
|
||||
- Delete, rename and move. Destructive controls belong here eventually, shown
|
||||
and confirmed rather than hidden, but none is needed to read or change a
|
||||
file.
|
||||
- Images in the viewer, through the existing `SessionImageViewer`.
|
||||
- Following an agent's edits live: a file open in the viewer refreshing when
|
||||
a `Write`/`Edit` tool call on the same path lands in the transcript. The
|
||||
transcript already knows the path.
|
||||
- Remembering the last directory per session.
|
||||
- Uploading from the phone into a directory. Attachments already do the
|
||||
upload half.
|
||||
- Search within a file, and find-in-files.
|
||||
- **A line-by-line editor**, which is the way past `EDIT_LIMIT`. The viewer
|
||||
already draws a file as rows and stays fast on a megabyte; an editor built
|
||||
the same way — a field per line, or a field over the lines on screen —
|
||||
would not pay Compose's cost of laying out one enormous text. It is a good
|
||||
deal more than this feature needed, and 32 kB covers the config files,
|
||||
notes and ordinary source files anybody edits from a phone.
|
||||
+391
@@ -0,0 +1,391 @@
|
||||
# iris: notable public API changes
|
||||
|
||||
For Iris to read on her own time. Each entry is a change to iris's public
|
||||
surface that a widget author or app author would notice: a trait method
|
||||
added, removed or re-shaped; a type that callers construct differently; a
|
||||
capability that moved. Small and trivial changes do not go here.
|
||||
|
||||
An entry gives the date, what changed, why, and a short before/after where
|
||||
it helps judge the change without the session that made it. Newest first.
|
||||
|
||||
## 2026-09-05 (later the same day): `iris_core::FrameReport` (RUST.md's I5 box)
|
||||
|
||||
New public type, `iris_core::FrameReport` (re-exported from `iris_core`'s
|
||||
`render` module alongside `FrameStats` and `JANK_THRESHOLD`). Why: `dumpsys
|
||||
gfxinfo` cannot see a `SurfaceView`'s own GPU-drawn frames at all, so a
|
||||
`wgpu`-rendered iris screen had no way to ask "was this smooth" the way
|
||||
Compose's own in-app render report already can -- item 3 of RUST.md's
|
||||
recommendation was stuck on a one-sided number for exactly this reason.
|
||||
|
||||
`FrameReport::record(elapsed: Duration)` is called once per frame (wired
|
||||
into `android/view.rs`'s `render()`, wrapping the same span from redraw
|
||||
start to after `queue.submit`+`present()` that Compose's report and
|
||||
`gfxinfo` both count) and writes into a fixed 4096-entry ring -- no
|
||||
allocation on the hot path. `FrameReport::report() -> Option<FrameStats>`
|
||||
gives total frames, janky % (over `JANK_THRESHOLD`, the same 16.7ms 60Hz
|
||||
budget `gfxinfo` uses), P50/P90/P99 and the worst; `None` if nothing has
|
||||
been recorded since the last `reset()`, not a zeroed report that would
|
||||
read as a real measurement. `FrameStats`'s `Display` line says plainly
|
||||
that it measures up to `present()` being called, not GPU/compositor
|
||||
completion, since wgpu's `present()` isn't fenced against either.
|
||||
|
||||
`AndroidUiState` gained a `pub frame_report: FrameReport` field --
|
||||
anything with `HasAndroidUiState` can now read or reset it. Before this,
|
||||
there was no way to ask iris's own render path how long a frame took at
|
||||
all, on any backend.
|
||||
|
||||
Before/after, for a caller that already has `ui_state: &AndroidUiState`:
|
||||
|
||||
```rust
|
||||
// before: no such question could be asked
|
||||
// after:
|
||||
match ui_state.frame_report.report() {
|
||||
Some(stats) => log::info!("iris frame report: {stats}"),
|
||||
None => log::info!("iris frame report: no frames recorded yet"),
|
||||
}
|
||||
ui_state.frame_report.reset(); // via android_state_mut()
|
||||
```
|
||||
|
||||
`iris-android-app`'s transcript screen exposes this as two named,
|
||||
tappable controls ("Frame report", "Reset frame report") rather than
|
||||
requiring a caller to wire its own UI -- see `transcript_client.rs`'s
|
||||
`frame_report_controls`.
|
||||
|
||||
## 2026-09-05: `Tasks::redraw_handle` (RUST.md's I5 Android integration)
|
||||
|
||||
New public method on `iris::task::Tasks`, `redraw_handle(&self) ->
|
||||
Arc<dyn RequestRedraw>`. Why: a caller running its own long-lived loop
|
||||
*inside* one spawned task (a live SSE follow, the Android transcript
|
||||
client's `select_session`) has no other way to ask for a frame after each
|
||||
`TaskCtx::update` -- `Tasks::spawn`'s own wrapper only requests one, after
|
||||
the whole async closure finishes, which fits a single request-then-update
|
||||
but not a stream that needs to be seen redrawing after *each* event. This
|
||||
is the same gap `iris/desktop-app`'s module doc names for why it uses
|
||||
winit's `Proxy<AppEvent>` instead of `Tasks` -- android-view has no
|
||||
`Proxy`, so this is what closes it there.
|
||||
|
||||
**A real bug this uncovered, not a hypothetical**: calling the returned
|
||||
handle's `request_redraw()` from the background thread crashed the process
|
||||
(`SIGABRT`, `Result::unwrap() on an Err value: JavaException`) the first
|
||||
time an Android transcript fetch called it a second time. `android/render.rs`'s
|
||||
`AndroidRedrawHandle` was already attaching the calling thread to the JVM
|
||||
correctly, but its `request_redraw` called `View::post_frame_callback`,
|
||||
whose Java side calls `Choreographer.getInstance()` -- which throws unless
|
||||
the *calling* thread already has a `Looper`, and a tokio worker thread,
|
||||
even freshly JNI-attached, has none. Fixed by routing through
|
||||
`View::post_delayed(0)` instead (Android's own thread-safe "queue work onto
|
||||
this View's UI thread" primitive, needing no caller-side `Looper`), landing
|
||||
on a new `IrisViewPeer::delayed_callback` override that drains tasks and
|
||||
renders -- same body as `do_frame`, on the UI thread where
|
||||
`post_frame_callback` is safe again. Any future caller of `redraw_handle()`
|
||||
from a background thread gets this for free; nothing about the fix is
|
||||
specific to the transcript screen.
|
||||
|
||||
## 2026-09-05: `transcript_ui::build_tree` (RUST.md's E4)
|
||||
|
||||
`transcript_ui::build` claimed the whole window (`ui_state.set_root(tree)`)
|
||||
as its last step, which is right for a window that *is* the transcript
|
||||
screen (the winit example, an eventual Android cdylib) and wrong for the
|
||||
desktop app, which puts a session list beside it. `build_tree` is `build`
|
||||
minus that last step: it returns `(TranscriptScreen, StrongWidget)` instead
|
||||
of just `TranscriptScreen`, and the caller decides where the tree goes —
|
||||
into `ui_state.set_root`, or into a `WidgetPtr` alongside something else
|
||||
(`iris/desktop-app`'s `rebuild_transcript`). `build` is now one line calling
|
||||
`build_tree` and doing the `set_root` itself, so existing callers are
|
||||
unaffected.
|
||||
|
||||
```rust
|
||||
// before, and still available, for a caller that wants to *be* the window:
|
||||
let screen = transcript_ui::build(rsc, &mut ui_state, rows);
|
||||
|
||||
// new, for a caller embedding the screen beside something else:
|
||||
let (screen, tree) = transcript_ui::build_tree(rsc, rows);
|
||||
some_widget_ptr(rsc).set(tree);
|
||||
```
|
||||
|
||||
|
||||
## 2026-09-05: `DragArbiter`, pan-vs-select for one shared touch gesture (RUST.md's I5)
|
||||
|
||||
New public type, `iris::sense::DragArbiter`. Why: a widget author who
|
||||
registers both a list-level pan and a row-level drag-to-select on the same
|
||||
touch gesture has no way to arbitrate between them — `core/src/sense.rs`'s
|
||||
`run_sensors` always gives the innermost layer first refusal, so the inner
|
||||
one wins every frame it is pressed, not just the frame the press started
|
||||
(this is exactly what left transcript-ui's touch-drag panning unreachable
|
||||
until now). `DragArbiter` is one small state machine, one instance per
|
||||
gesture surface (a whole list, not per row), that a caller drives with its
|
||||
own `press_start`/`update`/`release` calls and a caller-supplied `Instant`
|
||||
(so it is unit-testable without a real clock or a render harness). It
|
||||
decides the way Android itself does: an ordinary vertical drag pans
|
||||
immediately; a stationary press held `LONG_PRESS` (500ms) starts a
|
||||
selection, which any further drag then extends; a horizontal drag while
|
||||
something is already selected extends it immediately, skipping the wait.
|
||||
|
||||
```rust
|
||||
// One per list, held alongside whatever state coordinates the rows:
|
||||
let mut arbiter = DragArbiter::new();
|
||||
|
||||
// On press-down:
|
||||
arbiter.press_start(pos, Instant::now(), already_selected);
|
||||
// Every frame the button/finger stays down:
|
||||
match arbiter.update(pos, Instant::now()) {
|
||||
DragOutcome::Pan(dy) => list.scroll(-dy),
|
||||
DragOutcome::SelectStart => selection.begin(...),
|
||||
DragOutcome::SelectExtend => selection.extend(...),
|
||||
DragOutcome::Undecided => {}
|
||||
}
|
||||
// On release:
|
||||
arbiter.release();
|
||||
```
|
||||
|
||||
`transcript-ui`'s `Selection::drag` (`transcript-ui/src/selection.rs`) is
|
||||
the reference caller: every row's `CursorSense::click_or_drag() |
|
||||
CursorSense::unclick()` handler routes through one `Selection`-owned
|
||||
arbiter instead of calling `begin`/`extend` directly, so a drag that starts
|
||||
on a row's own rendered text now pans the list correctly instead of
|
||||
always starting a selection. 8 new unit tests in `iris/src/sense.rs`'s
|
||||
`drag_arbiter_tests` module.
|
||||
|
||||
### 2026-09-05, later: `DragArbiter::is_idle()`, recovering a missed `press_start`
|
||||
|
||||
Follow-up to the above, from a real touch-scroll dropout: a gesture's
|
||||
`ACTION_DOWN` can land on a caller's own dead space (a row's padding, a
|
||||
gap, a header with no handler) that never calls `press_start`, so the
|
||||
first frame the arbiter actually sees is a `Pressing`-shaped `update`
|
||||
with no matching start. Before this, `update`'s `Idle` arm had no way to
|
||||
tell that apart from "nothing is happening" and answered `Undecided`
|
||||
forever for the rest of that gesture. `is_idle(&self) -> bool` lets a
|
||||
caller notice the gap and recover: if `is_idle()` is true on a frame the
|
||||
caller knows a press is genuinely down (its own `Pressing`/equivalent
|
||||
sense fired), call `press_start` right there instead of assuming one
|
||||
already happened. `transcript-ui`'s `Selection::drag` is the reference
|
||||
caller — one new match arm, checked before the ordinary `update`-only
|
||||
case. Any other `DragArbiter` caller with the same "one sensor per
|
||||
sub-region, no fallback for dead space" shape has the same gap and wants
|
||||
the same recovery.
|
||||
|
||||
## 2026-09-05: `SpanStyle`, per-range text styling (RUST.md's I5)
|
||||
|
||||
A `TextBuffer` used to have exactly one style (`TextAttrs`: colour, size,
|
||||
family, ...) for its whole string, applied via `push_default` into parley's
|
||||
ranged builder. `SpanStyle` is a second, optional layer: a byte range plus
|
||||
whichever of colour/family/font size/bold/italic/underline it overrides,
|
||||
pushed with parley's own `push(property, range)` instead. Why: a transcript
|
||||
row's markdown (a heading, **bold**, `inline code`, a link) all inside one
|
||||
wrapped paragraph needs each to carry its own look while the paragraph
|
||||
still wraps and selects as a single buffer — the thing `masonry`'s
|
||||
`TextArea` cannot do (`StyleSet` is one style for the whole editor,
|
||||
`text_area.rs:43-44`'s `// TODO: RichTextInput`), and the reason this
|
||||
existed at all.
|
||||
|
||||
```rust
|
||||
let (text, spans) = transcript_ui::markdown::render_markdown(src, 16.0);
|
||||
wtext(text)
|
||||
.spans(spans) // new: TextBuilder::spans, on both Text and TextEdit
|
||||
.editable(EditMode::MultiLine)
|
||||
.add(rsc);
|
||||
```
|
||||
|
||||
Two things a widget author should know before reaching for it:
|
||||
|
||||
- **Call `.spans()` before or after `.editable()`, both work** — the field
|
||||
lives on `TextBuilder` itself, not either output type, and both
|
||||
`TextOutput::run` and `TextEditOutput::run` apply it to the buffer via
|
||||
`TextBuffer::set_spans`. **These two call sites are a pair**: adding a
|
||||
third `TextBuilderOutput` impl without also calling `set_spans` there
|
||||
reproduces the exact bug this box shipped once already (spans silently
|
||||
dropped for `TextEdit`, found only by screenshotting, not by any test —
|
||||
`markdown.rs`'s own unit tests check string/range logic, which is
|
||||
correct in isolation and proves nothing about whether the render path
|
||||
ever sees it).
|
||||
- **Colour is now per-glyph, not per-buffer.** `PlacedGlyph` gained a
|
||||
`color: UiColor` field (from parley's own per-run `Style::brush`), and
|
||||
`Painter::glyphs` draws each glyph in its own colour instead of
|
||||
`RenderedText::color` uniformly. `RenderedText::color` still exists (the
|
||||
buffer's *base* colour, for a caller that wants it as a whole, e.g. to
|
||||
tint a cursor) but no longer drives what a glyph actually renders as.
|
||||
|
||||
## 2026-09-05: accessibility names via AccessKit (RUST.md's I4)
|
||||
|
||||
`.label()` (already in `trait_fns.rs`, previously unused anywhere in-tree)
|
||||
is now load-bearing: it's the one thing that puts a widget in the AccessKit
|
||||
tree `iris_core::ui::access::AccessTree` builds and both backends push
|
||||
out. A widget author who wants a control to be findable by name (and
|
||||
tappable by name, through `ui-trace`/a real screen reader) calls `.label()`
|
||||
on it; nothing else is required, and a widget nobody labels is invisible
|
||||
to this system at zero cost, not just zero UI.
|
||||
|
||||
```rust
|
||||
let button = rect(Color::LIME)
|
||||
.on(CursorSense::click(), move |_, rsc| { ... })
|
||||
.label("Add task"); // now findable by uiautomator/AccessKit as "Add task"
|
||||
```
|
||||
|
||||
Two new things a widget author might touch directly:
|
||||
|
||||
- **`Widget::access_role(&self) -> accesskit::Role`**, default `Unknown`.
|
||||
Override it if your widget has a real platform equivalent —
|
||||
`TextEdit` now returns `TextInput`/`MultilineTextInput` by `EditMode`.
|
||||
Only consulted for a widget that also has a `.label()`; an unlabelled
|
||||
widget's `access_role` is never called.
|
||||
- **`Widgets::named() -> impl Iterator<Item = WidgetId>`** — every widget
|
||||
with an explicit label, for anything else that wants to walk the same
|
||||
set `AccessTree` does.
|
||||
|
||||
Nothing about `Painter`, `draw`, or the layout/move machinery changed —
|
||||
this sits entirely beside them, reading `resolved_region`'s output rather
|
||||
than participating in producing it.
|
||||
|
||||
## 2026-09-05: `List`, a virtualised bottom-anchored list (RUST.md's I3)
|
||||
|
||||
A new widget, `iris::widget::List` (`iris/src/widget/list.rs` -- read its
|
||||
module doc first), for the transcript's kind of screen: variable-height
|
||||
rows, keyed by a `u64`, composed only while visible, moved rather than
|
||||
re-laid-out on scroll, a scroll anchor that survives a row inserted above
|
||||
it, "more" sentinels at each end, and "hold the edge nearest the tap" when
|
||||
a row's height changes (`note_tap`, resolved in the layout pass).
|
||||
|
||||
```rust
|
||||
let mut list = List::new(Axis::Y);
|
||||
list.push_back(ListRow::new(key, row_widget)); // O(1)
|
||||
list.push_front(ListRow::new(older_key, row)); // O(1), anchor unaffected
|
||||
list.set_more_before(Some(spinner_widget)); // sentinel, drawn at the edge
|
||||
list.note_tap(viewport_y); // before mutating a row's height
|
||||
let (top, bottom) = list.extent(key).unwrap(); // last frame's on-screen box, if visible
|
||||
```
|
||||
|
||||
Built entirely out of existing primitives (`Painter::widget`/`widget_within`/
|
||||
`reposition`/`draw_twice`, and `draw_inner`'s own old-children diffing) --
|
||||
no new mechanism was added to the render core for it. One correctness
|
||||
lesson worth reading even for other widgets: a row that fills whatever
|
||||
region it is offered (`Rect`, `is_size_independent`) cannot be measured at
|
||||
a throwaway oversized region and then merely `reposition`ed into place --
|
||||
`reposition` only ever writes an offset, never a size, so the oversized
|
||||
primitive stays oversized. `List` fixes this by caching each row's real
|
||||
height once measured and placing an already-known row directly at its
|
||||
exact box; see `list.rs`'s `place` for the full reasoning and
|
||||
`a_fill_shaped_background_is_not_left_oversized` for the regression test.
|
||||
|
||||
## 2026-09-05: a second backend (android-view), and what moved to make room for it
|
||||
|
||||
RUST.md's I2. Three changes a widget or app author would notice, all in
|
||||
service of the same thing: `default` (winit) and the new `android`
|
||||
(android-view) backends sharing what does not depend on windowing.
|
||||
|
||||
- **`Selector`/`Selectable`'s bound changed from `Rsc::State:
|
||||
HasDefaultUiState` to `Rsc::State: FocusHost`** (new trait, `attr.rs`).
|
||||
`HasDefaultUiState` still exists and still works — `default/attr.rs` now
|
||||
implements `FocusHost` for anything that has it — so a winit app's
|
||||
existing code is unaffected. An Android app implements `FocusHost` via
|
||||
`HasAndroidUiState` instead. Affects only an app that referenced
|
||||
`HasDefaultUiState` directly at a `Selectable`/`Selector` call site
|
||||
rather than through `.attr::<Selectable>(())`, which nothing in-tree
|
||||
does.
|
||||
- **`Tasks::init` takes `Arc<dyn RequestRedraw>` instead of
|
||||
`Arc<winit::window::Window>`.** `RequestRedraw` (`task.rs`) is one method,
|
||||
`fn request_redraw(&self)`; `winit::window::Window` implements it
|
||||
(`default/render.rs`), so `Tasks::init(window)` at a call site is
|
||||
unchanged by inference. Only matters if something constructed a `Tasks`
|
||||
directly rather than through `DefaultRsc`/`AndroidRsc`.
|
||||
- **`TextEdit::apply_event`/`TextInputResult` are `#[cfg(not(target_os =
|
||||
"android"))]`** — they take a `winit::event::KeyEvent`, which does not
|
||||
exist on Android; `android/input.rs` drives the same primitives
|
||||
(`backspace`/`delete`/`motion`/`insert`, all still unconditional) from
|
||||
`ndk::event::Keycode` directly instead. New unconditional getters on the
|
||||
way: `TextEdit::text()`/`selection_range()`/`caret()`, and
|
||||
`TextEditCtx::delete_byte_range`/`set_cursor_byte` — the primitives
|
||||
`android/ime.rs`'s `InputConnection` bridge needed and that were not
|
||||
previously exposed publicly.
|
||||
|
||||
## 2026-09-04: `Widget::draw` reports the size it used; `desired_width`/`desired_height` are gone
|
||||
|
||||
A widget used to implement three methods (`draw`, `desired_width`,
|
||||
`desired_height`); it now implements one, `fn draw(&mut self, painter: &mut
|
||||
Painter) -> Size`, which draws into `painter.region()` and returns how much
|
||||
of it was used. Why: the two extra methods routinely re-simulated what
|
||||
`draw` was about to do anyway (`Span::desired_ortho` copied its own draw
|
||||
loop to get cross-axis sizing right) — one visit per widget per frame
|
||||
instead of up to three. A container that needs a child's size before
|
||||
placing it (alignment, centering) draws the child once at a provisional
|
||||
region, reads the returned `Size`, and calls the new `Painter::reposition`
|
||||
to move it into its final spot — an O(1) offset write, not a second draw. A
|
||||
widget whose drawn output never depends on the size it's given (a
|
||||
fixed-size `Rect`, a decoded `Image`) overrides the new `fn
|
||||
is_size_independent(&self) -> bool { false }` to `true`, which skips
|
||||
redrawing it when only its offered region changes shape.
|
||||
|
||||
```rust
|
||||
// before
|
||||
fn draw(&mut self, painter: &mut Painter) { /* ... */ }
|
||||
fn desired_width(&mut self, ctx: &mut SizeCtx) -> Len { /* ... */ }
|
||||
fn desired_height(&mut self, ctx: &mut SizeCtx) -> Len { /* ... */ }
|
||||
|
||||
// after
|
||||
fn draw(&mut self, painter: &mut Painter) -> Size { /* ... */ }
|
||||
```
|
||||
|
||||
`SizeCtx` and `Cache` are gone with it — see `LAYOUT.md` for the full
|
||||
design, the move-offset mechanism this shipped alongside, and the file
|
||||
list.
|
||||
|
||||
## 2026-09-04: texture pipeline rebuilt off the binding array
|
||||
|
||||
`Textures`/`TextureHandle`, `GlyphPrimitive`, and `UiRenderNode::new` all
|
||||
changed shape. Why: the old pipeline bound every texture ever drawn in one
|
||||
`binding_array<texture_2d<f32>>` and asked every device, unconditionally,
|
||||
for `VK_EXT_descriptor_indexing` — a real share of Android GPUs lack it,
|
||||
and it failed outright on the Android emulator's software Vulkan. See
|
||||
TEXTURES.md's "Recommended shape" and "Implemented, 2026-09-04".
|
||||
|
||||
- **`UiRenderNode::new` drops its `limits: UiLimits` parameter, and
|
||||
`UiLimits` is gone.** Before: `UiRenderNode::new(&device, &queue,
|
||||
&config, UiLimits::default())`. After: `UiRenderNode::new(&device,
|
||||
&queue, &config)`. Nothing replaces it — there are no more
|
||||
binding-array limits to size.
|
||||
- **`src/default/render.rs`'s device request asks for no features and no
|
||||
binding-array limits.** Before: `required_features:
|
||||
Features::TEXTURE_BINDING_ARRAY | Features::PARTIALLY_BOUND_BINDING_ARRAY
|
||||
| Features::SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING`
|
||||
plus two `max_binding_array_*` limits. After: `Features::empty()` (the
|
||||
`DeviceDescriptor` default) and only `max_buffer_size` set, which was
|
||||
never about the binding array.
|
||||
- **`TextureHandle` has no `primitive()` method any more**; a caller
|
||||
outside `iris` shouldn't have been calling it (it fed the old renderer's
|
||||
internals), but if something did: use `image_index()` for a standalone
|
||||
image's bind-group index. There is no equivalent for a page — a page has
|
||||
no bind group of its own now, see below.
|
||||
- **`GlyphPrimitive` has no public constructor from a struct literal.**
|
||||
Before: `GlyphPrimitive { uv_min, uv_max, view_idx, sampler_idx, color,
|
||||
flags }`. After: `GlyphPrimitive::new(uv_min, uv_max, layer, color,
|
||||
flags)` — one `layer` (the shared atlas array's layer) instead of a
|
||||
`view_idx`/`sampler_idx` pair, since a page is now a layer of one array
|
||||
texture rather than its own bound texture.
|
||||
- **A widget author drawing images is unaffected**: `Painter::texture`/
|
||||
`texture_at`/`texture_within` and `Textures::add` keep their signatures.
|
||||
What changed underneath is that each standalone image now gets its own
|
||||
`wgpu::BindGroup` and draw call instead of a slot in the shared array —
|
||||
invisible from the widget API, visible only in `UiRenderNode`'s internals
|
||||
and in `iris`'s device requirements.
|
||||
|
||||
## 2026-09-05: `FrameReport` splits each frame at `queue.submit`
|
||||
|
||||
`FrameStats` gains two fields, and `FrameReport` gains a second recording
|
||||
method, to answer "is a slow frame iris's own CPU work or the driver/GPU"
|
||||
with a number instead of a guess (RUST.md's I5 box).
|
||||
|
||||
- **`FrameReport::record_split(total, submit_to_present)`** is a second way
|
||||
to record a frame, alongside the existing `record(total)` (unchanged,
|
||||
and still what a caller with no split should use — it now reads as
|
||||
`cpu_p50 == total`, `gpu_wait_p50 == 0`, rather than fabricating a
|
||||
number for a half it never measured).
|
||||
- **`FrameStats` gains `cpu_p50` and `gpu_wait_p50`**: medians of
|
||||
redraw-start-to-submit and submit-to-after-`present()` respectively,
|
||||
independent of each other and of the existing `p50`/`p90`/`p99`/`worst`
|
||||
(which are unchanged, and still over the whole frame). The Android
|
||||
renderer's `draw()` now returns the `submit_to_present` `Duration` it
|
||||
measured, which `android::view::render()` passes to `record_split`.
|
||||
- **Caveat carried in both doc comments**: `submit_to_present` is not
|
||||
fenced against the GPU actually finishing — it is "how long the CPU was
|
||||
blocked handing the frame to the driver," not a confirmed GPU-completion
|
||||
time. Enough to separate "iris is slow building the frame" from "iris is
|
||||
slow handing it off," not enough to claim an exact GPU budget.
|
||||
@@ -0,0 +1,352 @@
|
||||
# iris: known problems and things still to build
|
||||
|
||||
Iris's own list for the library, recorded 2026-09-04 in her words where it
|
||||
matters, so the agents working through RUST.md pick these up in a sensible
|
||||
order rather than rediscovering them. Each item says where it sits in the
|
||||
order and what "done" looks like. Tick and date them in place.
|
||||
|
||||
## Fix
|
||||
|
||||
- [x] **Input does not fall through by input type (2026-09-04).**
|
||||
`SensorUi::run_sensors` (`src/default/sense.rs`) used to set "consumed,
|
||||
stop checking lower layers" from mere hover — a widget registered for
|
||||
nothing but `click()` blocked a `Scroll` meant for whatever was behind
|
||||
it, since "the cursor is over this widget" and "this widget handled the
|
||||
event" were the same check. Fixed by judging consumption per input
|
||||
kind: with no button transition and no scroll happening this frame
|
||||
("momentary" activity), the topmost hovered widget still wins, same as
|
||||
before; when something momentary *is* happening, only a widget whose
|
||||
registered senses actually include a matching non-hover one (checked
|
||||
via a new `TypeEventManager::registered`, which lists what a widget
|
||||
registered without running anything) consumes it, so a widget with only
|
||||
`Hovering`/click handlers can no longer block a scroll from reaching a
|
||||
list underneath. `iris/src/sense_tests.rs` builds a button-over-a-list
|
||||
`Stack` with a plain `HasEvents` impl (no GPU or window) and checks both
|
||||
directions: a scroll over the button reaches the list, and a real click
|
||||
still reaches the button — confirmed to fail on the pre-fix code and
|
||||
pass after.
|
||||
|
||||
- [x] **Appending one image to an already-loaded list rebuilds every other
|
||||
image's bind group (2026-09-05, fixed 2026-09-05).** Found by the
|
||||
benchmark below: `GpuTextures::update` (`core/src/render/texture.rs`)
|
||||
triggered `rebuild_image_bind_groups` — a loop over *every live
|
||||
standalone image*, rebuilding its `BindGroup` — whenever the shared
|
||||
`masks` or `move_offsets` GPU buffer was resized (`masks_resized ||
|
||||
moves_resized` in `UiRenderNode::update`, `core/src/render/mod.rs`), and
|
||||
a widget getting its *first* move-offset slot (LAYOUT.md section 2 —
|
||||
every widget gets one on first draw) could be exactly what grows that
|
||||
buffer. So one new message with one new image, appended to a transcript
|
||||
that already has N images loaded, did not cost O(1): it cost one
|
||||
`create_image` for the new image plus one `make_image_bind_group` per
|
||||
*existing* image, because the new widget's own move slot pushed the
|
||||
arena past its capacity. Measured directly in
|
||||
`iris/examples/bench_images.rs`: appending a 1,001st image to 1,000
|
||||
already-settled ones reported **1,001** bind-group creates for that one
|
||||
frame, not 1 (`./run-bench.sh images`, frame 5 in the transcript below).
|
||||
|
||||
**Fix**: `masks`/`move_offsets` never belonged in a standalone image's own
|
||||
bind group (group 2) in the first place — the group also holds that
|
||||
image's own texture view, which is the only thing that is genuinely
|
||||
per-image, so a buffer shared by *everything* forced a rebuild of
|
||||
*every* group the moment it moved. Gave masks/move_offsets their own
|
||||
bind group (group 3 in `shader.wgsl` and `UiRenderNode`: `masks_layout`/
|
||||
`masks_group`), bound once per frame in `UiRenderNode::draw` rather than
|
||||
once per draw call, instead of duplicating them into every per-image
|
||||
group. `GpuTextures` and its image bind groups now know nothing about
|
||||
either buffer — `rebuild_image_bind_groups` is called only from
|
||||
`grow_array` (the atlas array texture growing, which genuinely does
|
||||
change what every image's own bind group must reference) — so a
|
||||
masks/move_offsets resize now touches exactly one bind group, ever,
|
||||
regardless of how many images are live. Numbers after the fix, same
|
||||
benchmark and command:
|
||||
|
||||
./run-bench.sh images
|
||||
frame=1 bind_group_creates=1000 (cold load, unchanged)
|
||||
frame=2 bind_group_creates=0 (was 1000 -- see the item below)
|
||||
frame=3 bind_group_creates=0
|
||||
frame=4 bind_group_creates=0
|
||||
(append one image here)
|
||||
frame=5 bind_group_creates=1 (was 1001)
|
||||
frame=6 bind_group_creates=0
|
||||
|
||||
`run-headless.sh tabs --shot` still 27266 bytes, byte-for-byte unchanged,
|
||||
confirming the bind-group restructuring changed nothing about what is
|
||||
drawn.
|
||||
- [x] **Bind-group creation takes two frames to reach the steady state, not
|
||||
one (2026-09-05, closed by the fix above, 2026-09-05).** Same benchmark:
|
||||
loading 1,000 images cold used to report 1,000 creates on frame 1
|
||||
(expected — `create_image`, one per new image) *and again* 1,000 on
|
||||
frame 2, before settling to 0 from frame 3. This was `rebuild_image_bind_groups`
|
||||
firing a second time for the same masks/move-offsets buffer-growth
|
||||
reason as the item above, confirming the guess recorded here — the two
|
||||
were exactly the same root cause measured two different ways. Frame 2
|
||||
now reports 0 (see the numbers above); not a separate fix.
|
||||
|
||||
## Build
|
||||
|
||||
- [x] **Benchmarks**, not unit tests, run on demand (2026-09-05; a
|
||||
`benches/` or a script under `iris/`, never in `cargo test`). The
|
||||
scenario that matters most is a **message list** — chat apps and this
|
||||
app's transcript alike — stressed with many messages and many images.
|
||||
One case in particular: **resizing an input box** (typing enough text to
|
||||
grow it) that pushes a long list of messages above it must stay very
|
||||
fast and recalculate almost nothing — a move of everything above, not a
|
||||
re-layout. That is exactly the O(1) move chain in LAYOUT.md; the
|
||||
benchmark is what proves it. Done when the numbers are in this file with
|
||||
the command, and the input-box case reports draws re-run, not just frame
|
||||
time.
|
||||
|
||||
**Built as two rigs**, chosen per scenario by whether a real `wgpu`
|
||||
device is needed (`UiRenderState`/`Widgets` touch no GPU or window, so
|
||||
most of this runs as an ordinary binary — the same property
|
||||
`layout_tests.rs` relies on):
|
||||
|
||||
- `iris/benches/message_list.rs` — a plain `Instant`-timed binary
|
||||
(`[[bench]] harness = false` in `iris/Cargo.toml`), not criterion: see
|
||||
the file's own header for why (short version — every scenario here
|
||||
reduces to a *count* `UiRenderState::take_counters` already produces,
|
||||
which criterion's statistical machinery adds nothing to and which a
|
||||
new dependency is not worth pulling in for). Covers (a) first-frame
|
||||
cost of a message list of N wrapped-text rows (one in 20 also carrying
|
||||
a small in-memory image) for N = 100/1,000/10,000; (b) per-frame cost
|
||||
of scrolling that list, 200 ticks; (c) the input-box case — a
|
||||
fixed-height field at the bottom of the screen growing by a line 40
|
||||
times, with the message list above it filling the rest of the screen.
|
||||
Run: `cd iris && cargo bench --bench message_list` (always release —
|
||||
`cargo bench` builds the `bench` profile, which is optimized).
|
||||
- `iris/examples/bench_images.rs` — needs a real device, so it runs
|
||||
through `iris/run-headless.sh bench_images`, printing
|
||||
`UiRenderNode::take_image_bind_group_creates()` (a new counter, added
|
||||
in `core/src/render/texture.rs` and `core/src/render/mod.rs`,
|
||||
mirroring `UiRenderState::take_counters`) each frame. Covers (d): 1,000
|
||||
image rows, checked both cold (does bind-group creation reach zero
|
||||
once loaded) and after appending one more image once settled (does
|
||||
*that* stay cheap) — the second question is what actually matters for
|
||||
a live transcript and is what turned up the two Fix items above.
|
||||
- `iris/run-bench.sh [list|images]` runs either or both and is what to
|
||||
run before/after touching `Scroll`, `Span`, `Sized`, the move-offset
|
||||
chain, or `GpuTextures`.
|
||||
|
||||
**Numbers (2026-09-05, release, `cargo bench`/`run-headless.sh`, this
|
||||
VM: AMD Ryzen 7 3800X, 8 cores, rustc 1.98.0 nightly-2026-09-03):**
|
||||
|
||||
cd iris && cargo bench --bench message_list
|
||||
(a) first frame, N=100: 30.30ms draws=227 rewrites=15 moves=0
|
||||
(a) first frame, N=1000: 186.04ms draws=2252 rewrites=150 moves=0
|
||||
(a) first frame, N=10000:1770.36ms draws=22502 rewrites=1500 moves=0
|
||||
(b) scroll, N=100/1000/10000, 200 ticks each:
|
||||
draws=200 rewrites=0 moves=200 (identical at every N)
|
||||
per-tick average: 0.0002ms (identical at every N)
|
||||
(c) input grows 40 lines, N=100/1000/10000 rows above it:
|
||||
draws=320 rewrites=40 moves=160 (identical at every N)
|
||||
per-line average: 0.0012-0.0013ms (identical at every N)
|
||||
|
||||
cd iris && ./run-bench.sh images (2026-09-05, before the fix)
|
||||
frame=1 bind_group_creates=1000 (cold load)
|
||||
frame=2 bind_group_creates=1000 (see Fix item above)
|
||||
frame=3 bind_group_creates=0
|
||||
frame=4 bind_group_creates=0
|
||||
(append one image here)
|
||||
frame=5 bind_group_creates=1001 (see Fix item above)
|
||||
frame=6 bind_group_creates=0
|
||||
|
||||
cd iris && ./run-bench.sh images (2026-09-05, after the fix)
|
||||
frame=1 bind_group_creates=1000 (cold load, unchanged -- genuine work)
|
||||
frame=2 bind_group_creates=0
|
||||
frame=3 bind_group_creates=0
|
||||
frame=4 bind_group_creates=0
|
||||
(append one image here)
|
||||
frame=5 bind_group_creates=1 (one image's own create_image, O(1))
|
||||
frame=6 bind_group_creates=0
|
||||
|
||||
**Reading it**: (a) is real, necessary work — shaping and laying out N
|
||||
never-before-seen text rows — and scales with N as it must, ~10x cost
|
||||
per 10x N. (b) and (c) are the pass conditions that matter: both are
|
||||
**exactly flat across N = 100 to 10,000**, confirming LAYOUT.md's O(1)
|
||||
move chain holds for both scrolling and for a growing input box pushing
|
||||
the message list — draws/moves per tick or per line do not grow with
|
||||
list size, and the per-operation cost (a fraction of a microsecond) is
|
||||
nowhere near a frame budget. (d)'s cold-load and steady-state halves
|
||||
behave as designed; its *append* half did not, until the fix above moved
|
||||
masks/move_offsets out of the per-image bind group — now flat at O(1)
|
||||
the same way (b) and (c) are.
|
||||
|
||||
- **I5's transcript screen (`iris/transcript-ui/`, 2026-09-05) — what it
|
||||
left, each recorded at the point in the code it would go rather than
|
||||
silently dropped. See RUST.md's I5 box for the full account of what
|
||||
*was* built (the screen, `SpanStyle`, cross-row selection, the growing
|
||||
composer).**
|
||||
- [x] **Android integration for this screen — done, 2026-09-05.**
|
||||
`iris-android-app`'s `transcript-screen` Cargo feature
|
||||
(`transcript_client.rs`) runs this screen against a real `ai-server`
|
||||
through `client-core`, confirmed on-device: real scrolling, real
|
||||
touch-drag panning, tap-by-name on the composer. Two real bugs found
|
||||
and fixed along the way (a missing `INTERNET` permission; a
|
||||
background-thread redraw request that crashed via a `Looper`
|
||||
requirement, fixed by routing through `View::post_delayed` — see
|
||||
`IRIS.md`'s `Tasks::redraw_handle` entry). See RUST.md's I5 box,
|
||||
"The Android integration, done 2026-09-05" for the full account.
|
||||
- [x] **A render-time number for iris, comparable to Compose's
|
||||
`transcript-bench.sh` report — instrumentation done and a real number
|
||||
obtained, 2026-09-05 (later the same day); the clean comparable loop
|
||||
is not.** `iris_core::FrameReport` (`iris/core/src/render/
|
||||
frame_report.rs`, `IRIS.md`'s new entry) times every frame from
|
||||
`render()`'s redraw start to after `queue.submit`+`present()`, exposed
|
||||
as two named on-screen controls ("Frame report", "Reset frame
|
||||
report"). Driven against a real on-device touch-drag it read
|
||||
`frames=34 janky%=61.76 p50=26.5ms p90=48.0ms p99=98.1ms
|
||||
worst=98.1ms` — real, not inferred, but accumulated across several
|
||||
gestures rather than one clean 24-swipe loop, because of the new
|
||||
finding below. See RUST.md's I5 box, "Update, 2026-09-05, later the
|
||||
same day" for the full account.
|
||||
- [ ] **New, 2026-09-05: intermittent touch delivery to iris's
|
||||
`SurfaceView` under this checkout's `EMU_GPU=software` emulator.**
|
||||
The same swipe coordinates, confirmed (by scanning a screenshot
|
||||
column for the first non-black pixel) to sit over real row text,
|
||||
sometimes produced 30+ real frames and a screenshot diff and
|
||||
sometimes produced zero of either, across otherwise-identical
|
||||
`ui-trace` invocations. Not the already-understood "already at that
|
||||
scroll edge" case (reproduced with content confirmed taller than the
|
||||
viewport, in both directions). Leading candidate, not yet confirmed:
|
||||
this checkout's emulator was independently observed at ~78% of one
|
||||
CPU core, continuously, while idle on-screen — SwiftShader's software
|
||||
rasterisation is CPU-bound by design, and a synthetic touch competing
|
||||
with that load for delivery is plausible but unmeasured *during* a
|
||||
failing gesture (the standing rule against diagnosing from
|
||||
after-the-fact measurements applies here). Needs a sampler (load,
|
||||
`dumpsys input`, a `-i 0` `ui-trace` capture) running while a failing
|
||||
gesture is driven, and ideally a comparison under `-gpu host` (real
|
||||
Vulkan) to see whether it is specific to software rendering. This is
|
||||
what blocks the clean, comparable 24-swipe loop above.
|
||||
- [x] **Long-press-then-drag-to-select — confirmed on-device, 2026-09-05
|
||||
(later the same day).** `ui-trace` gained a `holddrag X1 Y1 X2 Y2
|
||||
HOLD_MS MOVE_MS` action (`emulator-tools`, additive, extends the same
|
||||
`MotionEvent`/`injectInputEvent` mechanism `swipe` already used):
|
||||
press, hold past `LONG_PRESS`, move, release, as one continuous touch.
|
||||
Driven against a real row (`holddrag 300 1850 300 2050 600 300`) it
|
||||
produced `iris selection: begin at row ...` then a sequence of
|
||||
`iris selection: extend to row ...` log lines
|
||||
(`transcript-ui/src/selection.rs`, a new small `log` dependency since
|
||||
selection has no accessibility label of its own yet — see the next
|
||||
item), and a screenshot taken right after shows the expected
|
||||
highlighted selection spanning multiple rows. `DragArbiter`'s own
|
||||
unit tests already covered this sequence against a synthetic clock;
|
||||
this is the first time it has been driven by a real device touch.
|
||||
- [x] **Touch-drag panning over a row's own rendered text — done,
|
||||
2026-09-05.** `row.rs` used to register `CursorSense::click_or_drag()`
|
||||
on each row's `TextEdit` for cross-row selection; `TextEdit::draw`'s
|
||||
`painter.child_layer()` (`iris/src/widget/text/edit.rs:87`) meant that
|
||||
registration won `core/src/sense.rs::run_sensors`'s per-layer
|
||||
arbitration on every frame it was pressed, not just the frame the
|
||||
press started, so a list pan gesture registered on `List` itself never
|
||||
got a turn while a row was under the finger. Fixed with
|
||||
`iris::sense::DragArbiter` (recorded in `IRIS.md`), one small state
|
||||
machine per list deciding pan vs. select the way Android does (a
|
||||
vertical drag pans immediately; a stationary press held `LONG_PRESS`
|
||||
(500ms) starts a selection which further drag extends; a horizontal
|
||||
drag while something is already selected extends immediately) —
|
||||
`transcript-ui/src/selection.rs`'s `Selection::drag` is the one place
|
||||
every row's drag now routes through. 8 new unit tests
|
||||
(`iris/src/sense.rs`'s `drag_arbiter_tests`); `cargo fmt/clippy/test
|
||||
--workspace` and `cargo ndk` (both `iris` and `transcript-ui`) all
|
||||
clean; `run-headless.sh` screenshot byte-identical to before the
|
||||
change (38578 bytes). See RUST.md's I5 box, "Gap closed, 2026-09-05".
|
||||
- [x] **Intermittent touch-scroll dropout — root-caused and fixed,
|
||||
2026-09-05.** Not the coalesced-`ACTION_MOVE` hypothesis the earlier
|
||||
pass suspected (ruled out): a gesture's `ACTION_DOWN` can land on a
|
||||
row's own padding/gap or its header, which no `CursorSense` covers,
|
||||
so `DragArbiter` never gets `press_start` and sits in `Idle`
|
||||
(answers `Undecided` forever) for that whole gesture. Fixed via a new
|
||||
`DragArbiter::is_idle()` that `Selection::drag`
|
||||
(`transcript-ui/src/selection.rs`) checks to recover a missed press
|
||||
on the next `Pressing` frame. Four new unit tests. See RUST.md's I5
|
||||
box, "Touch-scroll dropout root-caused, 2026-09-05", for the trace and
|
||||
what a peer session sharing this checkout's emulator mid-pass
|
||||
prevented from being re-verified end-to-end (the aggregate
|
||||
`iris-scroll.sh` three-run confirmation and a re-taken FrameReport
|
||||
row) — a future pass should finish that once the emulator is free.
|
||||
- [ ] **Row-level accessibility names.** The composer carries
|
||||
`.label("Message")`; transcript rows do not carry a `.label()` of
|
||||
their own yet, so `Widgets::named()` (I4) does not include them —
|
||||
`row.rs`'s `build_text_row` is where one would go, keyed to something
|
||||
stable per row (its sender + a short excerpt, matching what a screen
|
||||
reader announcing a chat message would say).
|
||||
- [ ] **A tappable link and a background chip behind inline code.**
|
||||
Both need per-range glyph geometry that `TextEditCtx` does not expose
|
||||
outside `iris::widget::text` (`edit.rs`'s `layout()` helper is
|
||||
private) — see `markdown.rs`'s module doc for the exact shape the fix
|
||||
would take (the same primitive `TextEdit::draw`'s own selection
|
||||
highlight already uses internally,
|
||||
`iris/src/widget/text/edit.rs:99`).
|
||||
- [ ] **`Selection`'s anchor-row shortcut.** The row a drag started in
|
||||
is selected in full (`select_all`) the moment the drag leaves it,
|
||||
rather than "from the click point to whichever edge points away from
|
||||
the drag" — needs the same private `layout()` access as the item
|
||||
above. `selection.rs`'s module doc has the exact reasoning.
|
||||
- [ ] **No syntax highlighting inside a fenced code block.**
|
||||
`client_core::highlight` exists (built for the file explorer) and
|
||||
could feed per-token `SpanStyle`s into a code block's span; wiring it
|
||||
in was not attempted this pass.
|
||||
|
||||
- [ ] **Masks defined relative to each other.** Wanted: mask A multiplies
|
||||
by something *and also* applies mask B — a mask can reference a parent
|
||||
mask, the way the move chain references a parent offset. Today masks
|
||||
are independent regions. Design it beside the move chain (same shape:
|
||||
a parent index and a bounded walk in the shader); do it when a real
|
||||
widget needs it, not before.
|
||||
- [ ] **Positions as a single float per scroll.** Iris raised, and half
|
||||
rejected, letting a scroll update one float rather than positions:
|
||||
input handling cares about most elements in a list, so absolute
|
||||
positions must be computed on the CPU anyway. LAYOUT.md's design
|
||||
already lands here (GPU walks the chain, CPU resolves on demand for
|
||||
hit tests). Keep the CPU resolution lazy and per query; do not
|
||||
materialise every row's absolute position per frame.
|
||||
- [ ] **Animations, last.** Cosmetic, so after everything above. Must be
|
||||
**modular — a piece of the library rather than a core part forced into
|
||||
everything, the same way input is**. Whatever the mechanism, a widget
|
||||
that does not animate must pay nothing and import nothing for it.
|
||||
|
||||
## Build (for the port)
|
||||
|
||||
Widgets `RUST.md`'s "The port, in order (decided 2026-09-05)" needs and
|
||||
iris does not have yet, one entry per gap, named against the P-step that
|
||||
first needs it. Move an entry up to "Fix" or tick it in place once built;
|
||||
do not duplicate it there.
|
||||
|
||||
- [ ] **A history-paging cushion measured in on-screen viewports, not a
|
||||
row count.** (**P1**.) `iris::widget::List` has no equivalent of the
|
||||
Compose app's `HISTORY_SCREENS` — AGENTS.md's "Things that have
|
||||
bitten" is explicit that a fixed row count under-fills a screen on a
|
||||
tool-heavy transcript and over-fills one on a text-heavy one, so
|
||||
whatever loads the next page has to ask the list how many viewports
|
||||
are actually on screen, not assume a constant.
|
||||
- [ ] **A scaled thumbnail/image widget for an in-transcript image.**
|
||||
(**P1**.) `SessionImage.kt`'s bitmap decode-and-downscale has no iris
|
||||
counterpart; iris's own image widget (used by `bench_images.rs`) draws
|
||||
a loaded texture but does nothing about sourcing or scaling one from a
|
||||
server-produced attachment.
|
||||
- [ ] **A modal/dialog primitive.** (**P1**, reused by **P3** and
|
||||
**P5**.) Needed for the session settings dialog, `UsageDialog`'s
|
||||
equivalent, and the delete-with-`deleteForeign` confirmation with its
|
||||
toggle switch. Build once, wherever it is first needed, rather than
|
||||
once per screen that wants one.
|
||||
- [ ] **A horizontal gauge/bar widget.** (**P1**.) For
|
||||
`SessionUsageBar`'s equivalent — a bounded fill reflecting a fraction,
|
||||
nothing fancier.
|
||||
- [ ] **A `BusyItem` equivalent: a dimmed row carrying an operation
|
||||
label that does not block its list's own scroll/drag.** (**P3**.) The
|
||||
Compose version tried an overlay first and it swallowed the drag along
|
||||
with the tap (AGENTS.md's "Shared appearance") — worth not repeating
|
||||
that attempt in iris before building the row-level version directly.
|
||||
- [ ] **A toggle switch.** (**P3**.) For the delete dialog's
|
||||
`deleteForeign` control; iris has no switch/checkbox widget yet as far
|
||||
as this pass found.
|
||||
|
||||
## Reconsider
|
||||
|
||||
- [ ] **`WidgetView`.** Iris is unsure of it: what she wants is an easy way
|
||||
to compose a widget from others (a button is the main case). With
|
||||
sizing folded into `draw`, composing may be easy enough that `View` is
|
||||
redundant. Decide after the layout change lands, by writing a button
|
||||
both ways and keeping the one that is shorter to explain; delete the
|
||||
other rather than keeping two ways.
|
||||
+897
@@ -0,0 +1,897 @@
|
||||
# iris: one `draw` that reports a size
|
||||
|
||||
Preference stated by Iris, 2026-09-04, on the `rustify` branch. Recorded before
|
||||
any design or code so that it survives a cleared session. **Status: implemented
|
||||
2026-09-04, against every pass condition in §8** (measured, not assumed — see
|
||||
that section). Every widget listed in §7 was migrated in one change; none
|
||||
kept `desired_width`/`desired_height`. Five points needed correction or
|
||||
refinement beyond what this file originally specified — see "Deviations
|
||||
found during implementation" below, added right before "For IRIS.md" — read
|
||||
that section before touching `Aligned`, `Sized`, `MaxSize`, `Scroll`, or the
|
||||
move-slot lifecycle in `render_state.rs`, since each of those five is a real
|
||||
bug this file's first draft would have reproduced if implemented literally.
|
||||
|
||||
## What Iris asked for
|
||||
|
||||
> I don't like that widgets need both a draw and size functions. I'd much
|
||||
> rather them have a single draw that reports a size, and if it needs to be
|
||||
> moved then that can be done after the fact efficiently, or resized just
|
||||
> done after as well. This should be done efficiently like everything else
|
||||
> tries to do right now.
|
||||
|
||||
She added, a few minutes later: "single draw is not a requirement. It
|
||||
just seems more efficient from what I've heard. Feel free to override any
|
||||
decision I've made if you can find a genuinely better & still clean
|
||||
alternative." So the single-draw model is the default to design against,
|
||||
and the design below may reject it, but only with a written comparison
|
||||
showing the alternative does less work per frame and is no harder to use.
|
||||
|
||||
Standing constraints from RUST.md still apply: no DSL, plain Rust, do as
|
||||
little processing as possible per frame, but the model must cover every
|
||||
layout need a real app has (the transcript's virtualised list, wrapped
|
||||
text whose height depends on width, rows and columns that size to their
|
||||
children, overlays, masks).
|
||||
|
||||
## What exists today
|
||||
|
||||
`Widget` (`iris/core/src/widget/mod.rs`) has three methods: `draw(&mut
|
||||
self, &mut Painter)`, `desired_width(&mut self, &mut SizeCtx) -> Len` and
|
||||
`desired_height`. A parent asks `SizeCtx::width/height` for a child, which
|
||||
is memoised per widget id and axis in `Cache.size` keyed on the outer
|
||||
size, then places the child with `Painter::widget_within(region)`. So a
|
||||
child is visited twice (sized, then drawn), every widget implements sizing
|
||||
twice (one per axis), and a widget whose size depends on what it draws
|
||||
(wrapped text, a laid-out paragraph) does the layout in the size pass and
|
||||
again in the draw pass unless it caches by hand.
|
||||
|
||||
Primitives are already positioned by `UiRegion` values whose scalars have
|
||||
a `rel` and an `abs` part, resolved against the window in the vertex
|
||||
shader (`core/src/render/shader.wgsl`), and `Primitives::region_mut`
|
||||
exists to rewrite one instance's region in place. That is the mechanism a
|
||||
"move after the fact" can build on.
|
||||
|
||||
## What the design must answer
|
||||
|
||||
1. **Parent-before-child ordering.** A row has to know each child's width
|
||||
to place the next one, but under "one draw" the child's size only
|
||||
exists after it has drawn. The answer is meant to be: the child draws
|
||||
at a provisional origin, reports its size, and the parent *moves* it.
|
||||
The move must be O(1) per moved subtree, not O(primitives in the
|
||||
subtree). One way: every instance carries an index into a small
|
||||
per-widget offset buffer, so moving a widget writes one entry and the
|
||||
vertex shader adds it. Other ways may be better; the design should say
|
||||
what was considered.
|
||||
2. **Move vs resize are different costs and must be kept apart.** A move
|
||||
never re-runs `draw`. A resize re-runs `draw` for exactly the widgets
|
||||
whose size input changed, and a widget whose output does not depend on
|
||||
its size (an icon, a fixed rect) must be able to say so and be skipped.
|
||||
3. **Size-dependent content.** Wrapped text is the hard case: its height
|
||||
is a function of its width. A single `draw` receives the available
|
||||
size (what `SizeCtx.outer` is today) and reports what it used, so the
|
||||
two-pass "measure then draw" collapses into one for the common case.
|
||||
The design must say what happens when a parent wants the child's
|
||||
height *before* deciding the width it will offer (rare; say whether it
|
||||
is supported, or is done by drawing twice as an explicit, opt-in cost).
|
||||
4. **Caching.** Today's `Cache.size` memoises by (id, axis, outer). The
|
||||
replacement should memoise the whole draw result by (id, available
|
||||
size) so that an unchanged subtree costs nothing on the next frame,
|
||||
which is what makes a virtualised list cheap.
|
||||
5. **Everything currently written against `desired_width`/`desired_height`
|
||||
moves over in one change**, per the code rules: two names for one
|
||||
concept is not an intermediate state to leave behind. The widgets are
|
||||
in `iris/src/widget/` (`ptr`, `mask`, `image`, `rect`, `trait_fns`, and
|
||||
whatever else is there when the change is made).
|
||||
|
||||
## Order relative to the texture work
|
||||
|
||||
TEXTURES.md's redesign touches the render core (shader, `GpuTextures`,
|
||||
`Primitives`, `Painter`'s texture calls). This change touches the widget
|
||||
trait, `SizeCtx`, `Cache`, `Painter`'s widget calls, and any offset
|
||||
mechanism the vertex shader needs. They overlap in `Painter` and the
|
||||
shader, so they are done **in sequence, textures first**, and the layout
|
||||
design here is written (not implemented) while the texture work is in
|
||||
progress, then implemented on top of it.
|
||||
|
||||
## Design
|
||||
|
||||
### 1. The new `Widget` trait
|
||||
|
||||
```rust
|
||||
pub trait Widget: Any {
|
||||
/// Draw within `painter.region()` (the space the parent offered) and
|
||||
/// report how much of it was actually used, per axis.
|
||||
fn draw(&mut self, painter: &mut Painter) -> Size;
|
||||
|
||||
/// True if `draw`'s output (both the primitives it writes and the
|
||||
/// `Size` it returns) is the same for any `painter.region()` of the
|
||||
/// same *content* -- an icon, a fixed-size rect, an already-decoded
|
||||
/// image at its natural size. Default `false` (redraw on any change to
|
||||
/// the offered region) because assuming independence wrongly produces
|
||||
/// a stale draw; a widget must opt in.
|
||||
fn is_size_independent(&self) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
No `available` parameter: `Painter` already carries the region the parent
|
||||
handed down (`Painter::region()`, `core/src/ui/painter.rs:137`) and already
|
||||
exposes the pixel-resolved form (`px_size()`, `:156`) and the output surface
|
||||
size (`output_size()`, `:152`). Passing it again would be the same value
|
||||
under a second name. `desired_width`/`desired_height` (`core/src/widget/mod.rs:20-21`)
|
||||
and `WidgetAxisFns::desired_len` (`:24-35`) are deleted outright — not
|
||||
deprecated, not kept as a fallback — because a widget that implements both
|
||||
`draw` and `desired_*` for the same thing is exactly the "two names for one
|
||||
concept" the code rules call out, and it is what today's `Span::desired_ortho`
|
||||
(`iris/src/widget/position/span.rs:98-152`) already complains about in its
|
||||
own comment: "this literally copies draw so that the lengths are correctly
|
||||
set in the context, which makes this slow and not cool." Folding sizing into
|
||||
`draw` deletes that duplicate simulation, not just moves it.
|
||||
|
||||
**No single-draw alternative was found that does less work per frame.** The
|
||||
two-method trait was checked against three properties a real screen needs —
|
||||
a row placing children in sequence, a widget centering on its own content,
|
||||
and wrapped text — and in every one, `draw` already has to visit the child
|
||||
to get a size that is *this specific one's* answer, which today's
|
||||
`desired_width`/`desired_height` re-derive by re-running (a shrunk copy of)
|
||||
the same layout the draw pass will do again. So the two-method trait is not
|
||||
"measure once, draw once" in the general case; it is "measure once per axis,
|
||||
then draw once," i.e. up to three visits per widget per frame, against one
|
||||
under the design here. The single-draw model is therefore adopted as
|
||||
proposed, not merely accepted as a preference.
|
||||
|
||||
### 2. Move: O(1) per moved subtree, via a per-widget offset chain
|
||||
|
||||
**What exists today, and why it is not O(1).** `UiRenderState::mov`
|
||||
(`core/src/ui/render_state.rs:156-168`) fires when a widget's region keeps
|
||||
its *size* but changes *position* (`draw_inner`, `:85-100`:
|
||||
`active.region.size() == region.size()` after excluding the exact-match
|
||||
case). It rewrites every primitive's `region` field via
|
||||
`Primitives::region_mut` (`core/src/render/primitive.rs:176-179`) for the
|
||||
widget's own primitives, then recurses into every child — O(primitives in
|
||||
the subtree). Both call sites that trigger it today, `Scroll::draw`
|
||||
(`iris/src/widget/position/scroll.rs:29-31`) and `Offset::draw`
|
||||
(`iris/src/widget/position/offset.rs:9-11`), are "translate this subtree by
|
||||
an abs pixel amount, `rel` framing unchanged" — a transcript scroll
|
||||
re-touches every glyph in every visible row, every frame of the drag, and
|
||||
I3's target is 800 rows on screen.
|
||||
|
||||
**Recommendation: a per-widget offset slot forming a parent-linked chain,
|
||||
resolved in the vertex shader.**
|
||||
|
||||
- `UiData` (`core/src/ui/mod.rs:14-20`) gains
|
||||
`pub move_offsets: TrackedArena<MoveOffset, u32>`, the same arena shape
|
||||
already used for `masks: TrackedArena<Mask, u32>` on the line above it.
|
||||
- `render/data.rs` gains `pub struct MoveOffset { pub delta: [f32; 2], pub
|
||||
parent: u32 }` (`Pod`/`Zeroable`, `parent = u32::MAX` = "no ancestor,
|
||||
add nothing more"). A pure abs-pixel translation, not a general
|
||||
`UiRegion` remap — sufficient for every existing call site (above).
|
||||
- `PrimitiveInstance` (`render/data.rs:11-18`) gains `pub move_idx: u32`,
|
||||
a vertex attribute at `@location(7)` beside `mask_idx` at `6` — the same
|
||||
kind of per-instance handle.
|
||||
- `ActiveData` (`core/src/ui/active.rs`) gains `pub move_slot: MoveIdx`,
|
||||
assigned **when the widget is first drawn** (`draw_inner`, beside
|
||||
`active.insert`), with `parent` = the drawing widget's parent's slot.
|
||||
`Painter` threads a `move_slot` field down exactly as it already threads
|
||||
`mask` and `layer` (`painter.rs:9-20`), so a freshly-drawn descendant is
|
||||
correct from its first frame — nothing is ever retrofitted onto an
|
||||
already-active primitive. An unmoved widget's slot just stays `[0, 0]`.
|
||||
- `Painter::primitive_at` (`painter.rs:23-38`) writes `move_idx:
|
||||
self.move_slot`, matching how it already writes `mask_idx: self.mask`.
|
||||
- `mov(id, delta)` becomes: look up `id`'s slot, write
|
||||
`move_offsets[slot].delta += delta`. One write — no primitive touched, no
|
||||
recursion, since descendants already reference this slot transitively.
|
||||
- `shader.wgsl`'s vertex stage, after computing `top_left`/`bot_right` in
|
||||
pixels (after `:106`, before the clip-space divide at `:113`), walks
|
||||
`move_idx → move_offsets[i].parent` for a bounded number of steps (a
|
||||
small constant, e.g. 16, with a CPU-side debug assertion that no chain
|
||||
exceeds it), summing `delta` into both corners. Cost is O(chain depth),
|
||||
paid every frame regardless of whether anything moved — negligible next
|
||||
to the per-fragment texture sampling TEXTURES.md already measures this
|
||||
GPU as not bound by.
|
||||
|
||||
**Why the chain, not the flatter thing first proposed.** Iris's own
|
||||
phrasing — "every instance carries an index into a small per-widget offset
|
||||
buffer" — describes a flat table: one slot per subtree *declared* movable,
|
||||
no parent link. It breaks the moment two such subtrees nest — a row inside
|
||||
a scrolling list, itself later given its own animated offset (a
|
||||
swipe-to-delete mid-scroll) — because the row's primitives would have to
|
||||
pick one slot and lose the other's contribution. The chain costs one extra
|
||||
field and a bounded shader loop in exchange for no such gap, and since
|
||||
every `ActiveData` gets a slot unconditionally rather than lazily, it costs
|
||||
no more at the common depth of one than the flat version would.
|
||||
|
||||
**Against `region_mut` as the steady-state mechanism**: rejected for being
|
||||
O(primitives in the subtree) — the cost this section removes — but kept
|
||||
for a resize that changes a region's `rel` component (a genuine reflow,
|
||||
§3) and for a size-independent widget's resize (§3), where the content's
|
||||
shape doesn't change and one field write already suffices.
|
||||
|
||||
### 2b. Two more readers of "where is this widget," and masks
|
||||
|
||||
Moving the offset into the vertex shader means `ActiveData.region` is no
|
||||
longer the on-screen truth once a widget has been moved — it is where the
|
||||
widget was *drawn*, before any `move_offsets` delta. Two things read it as
|
||||
if it still were, and both must move to a resolved query or they silently
|
||||
answer with the pre-move position: a click landing on a scrolled row would
|
||||
be routed to whatever used to be there, with nothing on screen to say so —
|
||||
exactly the "wrong answer that looks like a right one" case the code rules
|
||||
single out.
|
||||
|
||||
**Hit-testing.** `SensorUi::run_sensors` (`src/default/sense.rs:154-200`)
|
||||
does the actual pointer routing, and line 170 is the read in question:
|
||||
`let shape = self.active.get(id).unwrap().region;` (`self: &UiRenderState`),
|
||||
immediately turned into pixels and tested against the cursor at `:171-172`.
|
||||
Under this design that region must be resolved through the same chain the
|
||||
GPU walks before it means anything. Add to `UiRenderState`:
|
||||
|
||||
```rust
|
||||
/// `active[id].region`, corrected by every `move_offsets` delta between
|
||||
/// `id` and the root — the CPU-side twin of the vertex shader's chain
|
||||
/// walk, over the same arena, so the two cannot disagree about where a
|
||||
/// widget is. O(chain depth), not O(primitives): a plain Rust loop over
|
||||
/// `move_offsets`, bounded by the same constant the shader loop uses
|
||||
/// (name it once, e.g. `render::MOVE_CHAIN_LIMIT`, and reference it from
|
||||
/// the WGSL loop bound in a comment, since WGSL cannot `include!` a Rust
|
||||
/// const across the language boundary).
|
||||
pub fn resolved_region(&self, id: WidgetId) -> UiRegion;
|
||||
```
|
||||
|
||||
`window_region` (`core/src/ui/render_state.rs:264-267`), the public
|
||||
coordinate query already used outside hit-testing
|
||||
(`src/default/attr.rs:15,17,70`, e.g. positioning one widget relative to
|
||||
another's on-screen box), is reimplemented to call `resolved_region(id)`
|
||||
before `.to_px(...)` instead of reading `.region` directly — one change
|
||||
covers both call sites listed there. `sense.rs:170` changes to
|
||||
`let shape = self.resolved_region(*id);`. Both are required the moment §2
|
||||
lands, not an optional follow-up: an unmoved widget's chain is empty and
|
||||
`resolved_region` costs one arena read to find that out, so there is no
|
||||
version of this design where skipping the fix is a legitimate
|
||||
optimization — it is a correctness gap, not a performance one.
|
||||
|
||||
**Masks.** `Painter::set_mask` (`core/src/ui/painter.rs:49-52`) bakes the
|
||||
painter's *current* region into a `Mask` pushed onto
|
||||
`masks: TrackedArena<Mask, u32>` (`core/src/ui/mod.rs:19`), and the
|
||||
fragment shader clips every primitive against `masks[in.mask_idx]`'s raw
|
||||
`rel`/`abs` fields, unaffected by any move (`shader.wgsl:147-157`). If the
|
||||
widget that called `set_mask` — `Masked::draw`,
|
||||
`iris/src/widget/mask.rs:7-11`, `painter.set_mask(painter.region()); ...` —
|
||||
is itself later moved, its clip rectangle stays where it was drawn while
|
||||
its content moves out from under it: a visibly wrong clip, immediately on
|
||||
screen, not a latency question.
|
||||
|
||||
Fix: `Mask` (`core/src/render/data.rs:46-49`) gains `pub move_idx: u32`,
|
||||
written from `Painter::set_mask` as `self.move_slot` — the identical slot
|
||||
the mask-owning widget's own primitives already get (§2), not a second
|
||||
mechanism. Resolution happens in the **fragment** shader, not the CPU, and
|
||||
not the vertex shader either: `shader.wgsl`'s mask check (`:147-157`)
|
||||
currently computes the mask's `top_left`/`bot_right` inline from
|
||||
`masks[in.mask_idx]`; that computation is extended to walk the same
|
||||
move-offset chain §2 added, via one shared function —
|
||||
|
||||
```wgsl
|
||||
fn resolve_move(idx: u32) -> vec2<f32> { /* the bounded parent walk, used by both stages */ }
|
||||
```
|
||||
|
||||
— called from `vs_main` for a primitive's own corners and from `fs_main`
|
||||
for its mask's corners, so the walk is written once and the two stages
|
||||
cannot drift apart (the sibling-rule from the code rules: one loop, not a
|
||||
hand-copied second one in the other shader stage).
|
||||
|
||||
**Why the fragment shader, not a CPU-side mask rewrite at move time.** A
|
||||
primitive's mask is frequently owned by a *different* widget than the
|
||||
primitive itself — often several levels up a subtree, with its own,
|
||||
independent move slot — so a primitive's resolved offset and its mask's
|
||||
resolved offset are two different chain sums, both needed, and only the
|
||||
fragment shader has both `in.move_idx` (this fragment's own chain) and
|
||||
`in.mask_idx` (indirecting to a second, possibly unrelated chain) already
|
||||
in hand per-fragment. Resolving mask regions on the CPU at move time would
|
||||
mean, for every `mov()` call, walking forward to every mask instance the
|
||||
moved widget's slot could affect and rewriting its raw region — exactly
|
||||
the O(subtree) cost §2 exists to remove, just moved from primitives to
|
||||
masks. The fragment shader already re-reads `masks[in.mask_idx]` every
|
||||
frame (`:148`); one more arena read to resolve its chain costs nothing
|
||||
extra in kind.
|
||||
|
||||
**The scroll-container case, checked rather than assumed.** A masked,
|
||||
scrollable region is built as a `Masked` wrapping a `Scroll`
|
||||
(`iris/src/widget/position/scroll.rs`, `iris/src/widget/mask.rs`) — the
|
||||
viewport border is drawn (and `set_mask` called) by `Masked`, which is
|
||||
never itself the target of `mov()`; only `Scroll`'s inner content is,
|
||||
every frame the user drags. Because each widget's move slot is its own
|
||||
(§2: assigned per `ActiveData`, not shared), `Masked`'s mask references
|
||||
its own, stationary slot, while the scrolled content underneath references
|
||||
a separate, deeper slot whose `parent` chain passes through — but does not
|
||||
write to — the viewport's slot. Moving the content therefore never touches
|
||||
the mask's resolved position, and the mask staying still while its content
|
||||
slides past it is what this design already produces with no special case,
|
||||
not an extra rule that had to be added for it.
|
||||
|
||||
### 3. Resize scope
|
||||
|
||||
A resize is "the region a widget's parent offers it changes such that the
|
||||
widget's draw might produce different output" — as opposed to a move, which
|
||||
by construction cannot (§2 is scoped to pure translation). Two independent
|
||||
narrowings apply, and both are real, measured properties of the code as it
|
||||
stands rather than new machinery:
|
||||
|
||||
**(a) A window resize does not, by itself, require touching most widgets.**
|
||||
`shader.wgsl:105-106` recomputes every primitive's pixel position from
|
||||
`window.dim` and the primitive's stored `rel`/`abs` pair *every frame,
|
||||
already, on the GPU*. A widget laid out purely in `rel`/`abs` terms (no
|
||||
call to `px_size()`, `output_size()`, or anything else that reads a
|
||||
concrete pixel count) is therefore already correct after a resize with zero
|
||||
CPU work — the shader did it. `UiRenderState::needs_redraw_all`
|
||||
(`render_state.rs:229-231`) currently ignores this and redraws the entire
|
||||
tree on every `resized`, which was the safe default while sizing and
|
||||
drawing were two passes; it should be narrowed to only the widgets that
|
||||
*do* read a concrete pixel value. Track this the same way `needs_redraw`
|
||||
already tracks per-widget dirtiness (`Widgets::needs_redraw`,
|
||||
`core/src/widget/widgets.rs:9`): a widget's `draw` call marks itself
|
||||
pixel-dependent by calling through `Painter` methods that read
|
||||
`output_size`/`px_size` (both already funnel through `Painter`, so the
|
||||
marking is one line at each), and `resize()` (`render_state.rs:32-35`)
|
||||
walks only that set instead of unconditionally setting `resized = true`
|
||||
for a full `redraw_all`. This turns "every resize redraws everything" into
|
||||
"every resize redraws what depends on pixels" — a real behavior change
|
||||
beyond what was asked, so verify it against the I0b `pre_present_notify`
|
||||
resize regression (that fix depended on `redraw_all`'s completeness)
|
||||
before narrowing this.
|
||||
|
||||
**(b) A widget's `available` (its parent's offered region) can change
|
||||
without the widget's *content* changing — this is what
|
||||
`is_size_independent` (§1) answers.** When a container's own layout shifts
|
||||
(a sibling grew or shrank, changing this widget's offered box), a widget
|
||||
that returns `true` from `is_size_independent` is not redrawn: its
|
||||
primitives are unaffected by size, only by placement, so the parent
|
||||
either (i) issues a move (§2) if only position changed, or (ii) rewrites
|
||||
the primitive's `region` fields directly via `region_mut` if the box
|
||||
changed shape too (still O(primitives owned directly by this widget, not
|
||||
its subtree, since a size-independent widget by definition has no
|
||||
size-dependent descendants worth distinguishing — in practice this is
|
||||
always a leaf: `Rect`, `Image`, a fixed glyph). A widget that returns
|
||||
`false` (the default) is redrawn in full whenever `available` changes,
|
||||
which is correct always, just not free.
|
||||
|
||||
**Ancestor propagation** (a resized child changing its own reported size,
|
||||
requiring its parent to re-lay-out) is unchanged in spirit from today's
|
||||
`redraw` (`render_state.rs:270-305`), which already walks up exactly the
|
||||
ancestors whose cached size differs from the new one and stops as soon as
|
||||
a size is unchanged (`:274-286`). That loop moves from consulting
|
||||
`Cache.size` to consulting `ActiveData.size` (§5) but keeps its shape.
|
||||
|
||||
### 4. Wrapped text, and "needs child height before choosing width"
|
||||
|
||||
**Wrapped text is not a special case any more; it already reads as one
|
||||
draw.** `TextView::render` (`iris/src/widget/text/mod.rs:57-76`) already
|
||||
does exactly what single-draw asks for: it reads `ctx.px_size().x` as the
|
||||
wrap width, shapes once, and memoizes the shaped layout keyed on that width
|
||||
plus a changed-flag on the buffer and attrs (`:63-69`) — a second call with
|
||||
the same width is a hash-map-style cache hit, not a re-shape. Under the new
|
||||
trait this collapses `Text::draw`/`desired_width`/`desired_height`
|
||||
(`text/mod.rs:133-147`, three functions) into one `Text::draw` that calls
|
||||
`self.view.draw(painter)` once, which internally still calls `render`
|
||||
once, hits its own cache, and returns the size it already computed. No
|
||||
new caching is needed here; the two now-redundant call sites
|
||||
(`desired_width`/`desired_height` each separately calling `render`) simply
|
||||
disappear, which is a second `render` avoided per frame per text widget
|
||||
that is being measured by a parent.
|
||||
|
||||
**"Parent wants the child's height before deciding the width it will
|
||||
offer"** — the genuinely circular case named in the brief, e.g. a column
|
||||
that sizes its own width to its widest child, where that child is wrapped
|
||||
text whose height (which the column's *own* height depends on) depends on
|
||||
the width the column has not yet decided. This is not solvable in one pass
|
||||
for the same reason it is not solvable in CSS shrink-to-fit with wrapped
|
||||
content: the two axes' answers are mutually dependent. `Span::desired_ortho`
|
||||
(`span.rs:98-136`) already hits exactly this today and already resolves it
|
||||
by an explicit second, throwaway pass (its own comment: "this literally
|
||||
copies draw ... which makes this slow and not cool"). The design keeps that
|
||||
resolution, made explicit rather than accidental: `Painter` gets
|
||||
|
||||
```rust
|
||||
/// Draw `child` at a provisional region to learn its size under one
|
||||
/// axis's worth of assumption, discard everything it wrote, then draw it
|
||||
/// again at the region that assumption produced. For the rare parent that
|
||||
/// cannot pick an offered size without already knowing the answer.
|
||||
/// Twice the cost of one `draw`; every other case in this file avoids it.
|
||||
pub fn draw_twice(&mut self, child: &StrongWidget, first: UiRegion, second: impl FnOnce(Size) -> UiRegion) -> Size;
|
||||
```
|
||||
|
||||
implemented as: draw at `first`, record `Size`, remove the widget and its
|
||||
subtree the same way a resize-triggered redraw already does (`draw_inner`'s
|
||||
"if not \[same region\], maintain resize and track old children," `:97-100`,
|
||||
which already frees the old primitives before redrawing) — reusing that
|
||||
path rather than adding a second one — draw again at `second(size)`, return
|
||||
the final `Size`. It is opt-in and named for its cost, so a widget only
|
||||
pays it if it is the one that needs it; `Span`'s cross-axis case is the one
|
||||
call site converted to it, replacing the hand-rolled duplicate loop.
|
||||
|
||||
### 5. Caching and invalidation
|
||||
|
||||
`Cache.size` (`core/src/ui/cache.rs`) is **deleted, not replaced with an
|
||||
equivalent** — the thing it memoized (a `desired_width`/`desired_height`
|
||||
answer, independent of drawing) no longer exists as a separate query, so
|
||||
there is nothing left to cache at that layer. What already provides "an
|
||||
unchanged subtree costs nothing" is the check `draw_inner` performs before
|
||||
touching a widget at all (`render_state.rs:85-90`): if the widget is active,
|
||||
its region is unchanged, and it is not marked dirty, `draw_inner` returns
|
||||
immediately — no `Painter` constructed, no primitive touched, no shader
|
||||
work beyond what the GPU already redraws from the unchanged instance
|
||||
buffer. That check is kept exactly as it is; it is the caching mechanism,
|
||||
and it already operates at (id, region) granularity, which subsumes "(id,
|
||||
available size)" once size *is* what a region change means.
|
||||
|
||||
What is added: `ActiveData` gains `pub size: Size` — the value `draw`
|
||||
returned, stored the moment it is (`draw_inner`, alongside building the
|
||||
`ActiveData` struct at `:134-143`). This is what a parent placing this
|
||||
widget for a second frame without redrawing it (because nothing changed)
|
||||
reads instead of recomputing — it replaces `Cache.size`'s role of "answer a
|
||||
size question without a full draw" with "read the size of the last actual
|
||||
draw," which is always available because `draw_inner`'s skip path is only
|
||||
reachable once the widget has been drawn at least once. `Cache::remove`/
|
||||
`Cache::clear` (`cache.rs:9-17`) are deleted with the type; `ActiveData`
|
||||
already has an equivalent lifecycle (removed in `remove`/`remove_rec`,
|
||||
`render_state.rs:171-198`, freed with the widget).
|
||||
|
||||
### 6. Before / after
|
||||
|
||||
**A leaf, `iris/src/widget/rect.rs`** — the size-independent case:
|
||||
|
||||
```rust
|
||||
// before
|
||||
impl Widget for Rect {
|
||||
fn draw(&mut self, painter: &mut Painter) {
|
||||
painter.primitive(RectPrimitive { color: self.color, radius: self.radius,
|
||||
thickness: self.thickness, inner_radius: self.inner_radius });
|
||||
}
|
||||
fn desired_width(&mut self, _: &mut SizeCtx) -> Len { Len::rest(1) }
|
||||
fn desired_height(&mut self, _: &mut SizeCtx) -> Len { Len::rest(1) }
|
||||
}
|
||||
```
|
||||
|
||||
```rust
|
||||
// after
|
||||
impl Widget for Rect {
|
||||
fn draw(&mut self, painter: &mut Painter) -> Size {
|
||||
painter.primitive(RectPrimitive { color: self.color, radius: self.radius,
|
||||
thickness: self.thickness, inner_radius: self.inner_radius });
|
||||
Size::REST // fills whatever it was given -- used == available
|
||||
}
|
||||
fn is_size_independent(&self) -> bool { true } // content never depends on region size
|
||||
}
|
||||
```
|
||||
|
||||
**A container that needs the child's size before placing it,
|
||||
`iris/src/widget/position/align.rs`**:
|
||||
|
||||
```rust
|
||||
// before
|
||||
impl Widget for Aligned {
|
||||
fn draw(&mut self, painter: &mut Painter) {
|
||||
let region = match self.align.tuple() {
|
||||
(Some(x), Some(y)) => painter.size(&self.inner).to_uivec2().align(RegionAlign { x, y }),
|
||||
(Some(x), None) => { let x = painter.size_ctx().width(&self.inner).apply_rest().align(x);
|
||||
UiRegion::new(x, UiSpan::FULL) }
|
||||
(None, Some(y)) => { let y = painter.size_ctx().height(&self.inner).apply_rest().align(y);
|
||||
UiRegion::new(UiSpan::FULL, y) }
|
||||
(None, None) => UiRegion::FULL,
|
||||
};
|
||||
painter.widget_within(&self.inner, region);
|
||||
}
|
||||
fn desired_width(&mut self, ctx: &mut SizeCtx) -> Len { ctx.width(&self.inner) }
|
||||
fn desired_height(&mut self, ctx: &mut SizeCtx) -> Len { ctx.height(&self.inner) }
|
||||
}
|
||||
```
|
||||
|
||||
```rust
|
||||
// after
|
||||
impl Widget for Aligned {
|
||||
fn draw(&mut self, painter: &mut Painter) -> Size {
|
||||
let full = painter.region();
|
||||
// Draw once at the full region to learn the child's real size --
|
||||
// this placement is provisional and corrected below without a
|
||||
// second draw.
|
||||
let used = painter.widget_within(&self.inner, full);
|
||||
let region = match self.align.tuple() {
|
||||
(Some(x), Some(y)) => used.to_uivec2().align(RegionAlign { x, y }).within(&full),
|
||||
(Some(x), None) => used.x.apply_rest().align(x).within(&full),
|
||||
(None, Some(y)) => used.y.apply_rest().align(y).within(&full),
|
||||
(None, None) => full,
|
||||
};
|
||||
painter.reposition(&self.inner, region); // O(1): one offset write, no second draw
|
||||
used
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`Painter::widget_within`/`widget`/`widget_at` (`painter.rs:55-76`) change
|
||||
return type from `()` to `Size`, carrying the child's `draw` result back —
|
||||
the only signature change needed to let a parent see what its child used.
|
||||
`Painter::reposition` is new, computing the delta between where a child
|
||||
was actually drawn and where it belongs and calling the O(1) `mov` from
|
||||
§2. `SizeCtx` and `Painter::size_ctx`/`size`/`len_axis` (`painter.rs:141-150,
|
||||
180-182`) are deleted — nothing calls `desired_len` any more, so there is
|
||||
nothing left for `SizeCtx` to answer; `draw_text`/`label`/`px_size`/
|
||||
`output_size` already exist redundantly on both `SizeCtx` and `Painter`
|
||||
today (compare `size.rs:71-90` against `painter.rs:152-174`) and this
|
||||
deletes the `SizeCtx` copies, keeping the `Painter` ones.
|
||||
|
||||
### 7. Migration — every file and widget that changes
|
||||
|
||||
One change, in dependency order (rename-and-move-together, per the code
|
||||
rules — no intermediate state with both trait shapes):
|
||||
|
||||
- `core/src/widget/mod.rs` — the `Widget` trait (§1), delete
|
||||
`WidgetAxisFns`, update `impl Widget for ()`.
|
||||
- `core/src/ui/size.rs` — delete `SizeCtx` (the type and all its methods).
|
||||
- `core/src/ui/cache.rs` — delete `Cache` (§5).
|
||||
- `core/src/ui/painter.rs` — `widget`/`widget_within`/`widget_at` return
|
||||
`Size`; add `reposition`, `draw_twice`; delete `size_ctx`, `size`,
|
||||
`len_axis`; `primitive_at` writes `move_idx`.
|
||||
- `core/src/ui/render_state.rs` — `draw_inner` captures and stores
|
||||
`ActiveData.size`; `mov` becomes the O(1) offset write (§2); resize
|
||||
narrowing (§3a); `redraw`'s per-axis loop reads `ActiveData.size`
|
||||
instead of `Cache.size`.
|
||||
- `core/src/ui/active.rs` — `ActiveData` gains `size: Size`,
|
||||
`move_slot: MoveIdx`.
|
||||
- `core/src/ui/mod.rs` — `UiData` gains `move_offsets`.
|
||||
- `core/src/render/data.rs` — `PrimitiveInstance` gains `move_idx`;
|
||||
new `MoveOffset` struct.
|
||||
- `core/src/render/primitive.rs` — thread `move_idx` through `PrimitiveInst`
|
||||
and `Primitives::write`, matching `mask_idx`.
|
||||
- `core/src/render/mod.rs` — bind the new `move_offsets` storage buffer
|
||||
(group 2, beside `masks`) and its update path.
|
||||
- `core/src/render/shader.wgsl` — `InstanceInput` gains `move_idx`;
|
||||
`MoveOffset`/`UiScalar`-shaped storage binding; a shared `resolve_move`
|
||||
function (§2b) called from both `vs_main` (a primitive's own corners)
|
||||
and `fs_main` (its mask's corners, once `Mask` carries `move_idx`).
|
||||
- `core/src/ui/render_state.rs` — additionally, `resolved_region` (§2b)
|
||||
and `window_region` (`:264-267`) reimplemented on top of it.
|
||||
- `src/default/sense.rs` — `run_sensors`'s hit-test read (`:170`) switches
|
||||
from `self.active.get(id).unwrap().region` to `self.resolved_region(*id)`
|
||||
(§2b) — the pointer-routing fix this design requires, not an optional
|
||||
follow-up.
|
||||
- `core/src/render/data.rs` — additionally, `Mask` (`:46-49`) gains
|
||||
`move_idx: u32` (§2b).
|
||||
- `core/src/ui/painter.rs` — additionally, `set_mask` (`:49-52`) writes
|
||||
`move_idx: self.move_slot` into the `Mask` it pushes (§2b).
|
||||
- Every widget with a two-method `impl Widget`, collapsed to one `draw`
|
||||
(§1, §6), `is_size_independent` added where true: `core/src/widget/mod.rs`
|
||||
(`impl Widget for ()`), `iris/src/widget/rect.rs` (`Rect`, → true),
|
||||
`iris/src/widget/image.rs` (`Image`, → true — a decoded image's primitive
|
||||
never depends on the region it is offered, same as `Rect`),
|
||||
`iris/src/widget/mask.rs` (`Masked`), `iris/src/widget/ptr.rs`
|
||||
(`WidgetPtr`), `iris/src/widget/text/mod.rs` (`Text`, §4),
|
||||
`iris/src/widget/text/edit.rs` (`TextEdit`),
|
||||
`iris/src/widget/position/scroll.rs` (`Scroll`, keeps its `mov`-shaped
|
||||
offset, now O(1) automatically via §2), `iris/src/widget/position/align.rs`
|
||||
(`Aligned`, §6), `iris/src/widget/position/max_size.rs` (`MaxSize`),
|
||||
`iris/src/widget/position/layer.rs` (`LayerOffset`),
|
||||
`iris/src/widget/position/pad.rs` (`Pad`),
|
||||
`iris/src/widget/position/stack.rs` (`Stack`),
|
||||
`iris/src/widget/position/offset.rs` (`Offset`),
|
||||
`iris/src/widget/position/span.rs` (`Span`, §4's `draw_twice` for the
|
||||
cross-axis case, deleting `desired_ortho`'s duplicate loop),
|
||||
`iris/src/widget/position/sized.rs` (`Sized`).
|
||||
This list was produced by `grep -rn "impl Widget for\|fn desired_width\|fn desired_height"`
|
||||
across `core/` and `src/`; re-run it before starting, since it is the
|
||||
authoritative check that nothing was missed, not this paragraph.
|
||||
- `iris/examples/{minimal.rs,task.rs,view.rs,tabs/main.rs}` — no direct
|
||||
`impl Widget` found in any example (verified by the same grep); they use
|
||||
the builder DSL in `core/src/widget/trait_fns.rs` and should need no
|
||||
source change, which is itself part of the pass condition below.
|
||||
|
||||
### 8. Pass conditions
|
||||
|
||||
1. **Every example under `iris/examples` renders identically.** Run
|
||||
`iris/run-headless.sh EXAMPLE --shot PNG` for each of `minimal`, `task`,
|
||||
`view`, `tabs` before and after, and diff the PNGs pixel-for-pixel — not
|
||||
"looks right," since a subtle wrap or alignment regression is exactly
|
||||
what a diff catches and a glance does not.
|
||||
|
||||
**Result (2026-09-04): pass, all four, 0 differing bytes.** No PNG
|
||||
library is installed in this VM (no PIL, no ImageMagick, no pip), so the
|
||||
diff is a from-scratch PNG decoder (`zlib` + the five filter types) at
|
||||
`/tmp/layout-shots/pngdiff.py`, comparing decoded pixel bytes rather than
|
||||
file bytes (`cmp` alone is not conclusive across two separately-encoded
|
||||
PNGs, though it happened to agree here for `minimal`). Before-shots were
|
||||
taken with `git stash` at the pre-change commit; `tabs` needed two real
|
||||
fixes (deviations 1 and 2 below) before it stopped differing — the other
|
||||
three matched on the first try.
|
||||
2. **Unchanged-frame cost, measured, not assumed.** Add a counter beside
|
||||
the existing `debug_layers`/`active_widgets` instrumentation
|
||||
(`render_state.rs:241-262`) for (a) `Widget::draw` invocations and (b)
|
||||
`Primitives::write`/`region_mut` calls, both per `update()` call. Drive
|
||||
one example (`tabs`, since it already has multiple widgets and an
|
||||
interactive element) through one frame with nothing changed and report
|
||||
both counts — the pass condition is **0 draws and 0 primitive rewrites**
|
||||
for a frame in which nothing was marked dirty, resized, or moved.
|
||||
|
||||
**Result (2026-09-04): pass, 0 and 0.** Implemented as
|
||||
`UiRenderState::take_counters() -> (u64, u64, u64)` (draws, `region_mut`
|
||||
rewrites, `move_offsets` writes — a third counter, for condition 3
|
||||
below), reset on read. Measured in
|
||||
`iris/src/layout_tests.rs::an_unchanged_frame_draws_and_rewrites_nothing`
|
||||
against a `Scroll` over 500 fixed-height rects (not the `tabs` example —
|
||||
see the note on condition 3 for why this runs as a plain unit test
|
||||
instead).
|
||||
3. **Single-moved-child cost, measured.** Same counters, one frame in
|
||||
which exactly one widget is moved (not resized) with N primitives in its
|
||||
subtree — the pass condition is **1 write to `move_offsets`, 0 calls to
|
||||
`Widget::draw`, 0 calls to `region_mut`**, independent of N. Construct
|
||||
the case with a `tabs`-style example holding a deliberately large text
|
||||
block (hundreds of glyphs) inside a `Scroll`, so N is large enough that
|
||||
an O(N) regression would show up as a non-trivial write count rather
|
||||
than being lost in noise.
|
||||
|
||||
**Result (2026-09-04): pass — 0 draws, 0 rewrites, 1 move_offsets
|
||||
write, N = 500.** Built with rects rather than glyphs
|
||||
(`iris/src/layout_tests.rs::scrolling_moves_in_o1_without_a_redraw`):
|
||||
`iris-core`/`iris` touch no GPU or window to lay out and move a tree, so
|
||||
this runs as a plain `cargo test`, not through `run-headless.sh` — a
|
||||
`Widgets`/`UiData` pair and a bare `UiRsc` impl are enough, and it is
|
||||
faster and more precise than reading counters out of a real example's
|
||||
stderr. Getting a clean single move took two follow-up fixes beyond the
|
||||
design as written (deviation 3, the `parent_move_slot` threading; and
|
||||
the `Scroll` design decision below about offering last frame's content
|
||||
length) — without either, the count was in the thousands (every rect in
|
||||
the subtree redrawing) rather than 1.
|
||||
4. **Hit-testing follows the move, not just the render.** In the same
|
||||
scrolled-`tabs` construction as condition 3, scroll the content, then
|
||||
send a synthetic cursor position over a widget that moved and assert
|
||||
`run_sensors` (`src/default/sense.rs:154-200`) routes to that widget's
|
||||
id, not to whatever is now at its pre-scroll coordinates or to nothing.
|
||||
This is a correctness check, not a timing one — §2b's fix is required
|
||||
before §2 can ship at all, and this is what would fail silently
|
||||
(nothing on screen indicates a missed or misrouted hit) if it were
|
||||
skipped.
|
||||
|
||||
**Result (2026-09-04): pass**, but checked one level below
|
||||
`run_sensors`: `iris/src/layout_tests.rs::hit_testing_follows_a_scrolled_widget`
|
||||
scrolls a widget and asserts `UiRenderState::resolved_region` (the
|
||||
query `run_sensors`'s hit-test and `window_region` both now go through,
|
||||
per §2b) reports the moved, not the pre-scroll, position — within
|
||||
0.01px of the exact expected delta. `run_sensors` itself needs a
|
||||
`HasEvents`/window/cursor-state harness this pass did not build; the
|
||||
coverage that matters (does the position query the router uses reflect
|
||||
the move) is exercised directly instead.
|
||||
5. **A mask moves with its subtree.** Render a `Masked`-wrapped `Scroll`
|
||||
both before and after scrolling it (`iris/run-headless.sh` against a
|
||||
small purpose-built example, or an addition to `tabs`), and diff the
|
||||
two frames: the clipped edge of the content must have moved with the
|
||||
scroll while the viewport's own border (drawn by `Masked`, not moved)
|
||||
stays put — the specific case worked through in §2b. A mask rectangle
|
||||
that stayed at its pre-scroll position while its content slid past it
|
||||
is the regression this checks for, and it is visible in a single
|
||||
screenshot, not just in a counter.
|
||||
|
||||
**Result (2026-09-04): pass, checked numerically rather than by
|
||||
screenshot.** No example in this repository builds a `Masked`-wrapped
|
||||
`Scroll` (`tabs`'s "text edit scroll" tab uses `TextEdit`'s own internal
|
||||
scrolling, not this widget), so there was nothing to screenshot without
|
||||
first authoring a new example. Checked instead in
|
||||
`iris/src/layout_tests.rs::a_mask_stays_put_while_its_scrolled_content_moves`,
|
||||
on the exact data the fragment shader's `resolve_move` reads: the
|
||||
masked widget's own `move_offsets` slot delta is `[0, 0]` both before
|
||||
and after scrolling its content, because `Masked` is never itself the
|
||||
target of a move — only its child is, on a separate, deeper slot in the
|
||||
chain (§2b's "scroll-container case, checked rather than assumed"). A
|
||||
pixel-level screenshot check of this remains open; see RUST.md's next
|
||||
step.
|
||||
6. **`cargo test --workspace`, `cargo clippy --all-targets`, `cargo fmt`**
|
||||
stay clean at the defaults (iris has no tests today per I0b, so this is
|
||||
presently only clippy/fmt; add the first real widget-layer tests here if
|
||||
the move-offset chain or `draw_twice` are non-trivial enough to want
|
||||
one, per "match the codebase's testing posture" — judge that once the
|
||||
code exists rather than pre-committing to a number of tests here).
|
||||
|
||||
**Result (2026-09-04): pass.** `cargo fmt --all -- --check`,
|
||||
`cargo build --workspace --all-targets`, and `cargo clippy --all-targets`
|
||||
are all clean (one pre-existing, unrelated warning about `naga`/`wgpu`/
|
||||
`winit` future-incompatibility, from dependencies, not this change).
|
||||
`cargo test --workspace`: the 14 pre-existing `TextEdit` tests plus 4 new
|
||||
ones in `iris/src/layout_tests.rs` (conditions 2–5 above), 18 passed, 0
|
||||
failed — the move-offset chain turned out non-trivial enough (three real
|
||||
bugs found only by writing it) to clearly clear the "match the testing
|
||||
posture" bar this section left open.
|
||||
|
||||
### 9. Rejected, and why
|
||||
|
||||
- **A flat (non-chained) per-subtree offset table**, Iris's literal
|
||||
phrasing — rejected in §2 for breaking under nested independent moves
|
||||
(a swiped row inside a scrolling list). Costs nothing extra to avoid: the
|
||||
chain is the same mechanism with one more field.
|
||||
- **Keeping `region_mut` recursion as the only move mechanism** — rejected
|
||||
as the steady-state path (O(primitives in subtree), exactly what a
|
||||
transcript scroll must not pay every frame) but kept for resize-shaped
|
||||
changes (§3) where the content's own region field, not an ancestor
|
||||
chain, is what has to change.
|
||||
- **A second, size-only trait method kept alongside `draw`** (e.g.
|
||||
`fn size_hint(&self) -> Option<Size>` as a fast path some widgets could
|
||||
implement to skip a draw when a cheap answer exists) — considered and
|
||||
rejected: it reintroduces exactly the "two names for one concept" split
|
||||
this change removes, for a saving `is_size_independent` (§1, §3b)
|
||||
already covers for the cases where it would actually help (fixed-size
|
||||
leaves). A widget whose size is cheap to compute but whose *drawing* is
|
||||
not (unlikely in this codebase's widget set, but conceivable) is better
|
||||
served by that widget caching its own draw output internally — exactly
|
||||
the pattern `TextView::render` already uses (§4) — than by a second
|
||||
trait method every implementor has to reason about.
|
||||
- **Passing `available` as an explicit parameter to `draw`** (mirroring
|
||||
Masonry's `layout(&mut self, ctx, bc: &BoxConstraints) -> Size`, the
|
||||
yardstick per AGENTS.md) — rejected as redundant with `Painter::region()`,
|
||||
which already carries the same information into every widget that needs
|
||||
it; adding a parameter would just be a second route to a value already
|
||||
reachable, and would invite the two drifting apart.
|
||||
- **Eagerly propagating a moved widget's delta into every descendant's own
|
||||
offset value** (rather than chaining and resolving in the shader) —
|
||||
rejected as O(descendant widgets), which is smaller than O(primitives)
|
||||
but still not O(1), and the shader-side chain costs nothing extra to get
|
||||
the better bound.
|
||||
|
||||
## Deviations found during implementation (2026-09-04)
|
||||
|
||||
Five corrections this file's first draft did not anticipate, each found by
|
||||
`iris/run-headless.sh tabs --shot` disagreeing with a pixel-identical
|
||||
pre-change screenshot (pass condition 1) and traced with `eprintln!` in
|
||||
`draw_inner`/`reposition` — not by reasoning about the design in the
|
||||
abstract. Recorded here rather than silently fixed in place, per the code
|
||||
rules' escape-hatch requirement.
|
||||
|
||||
1. **`Aligned`'s provisional draw must call `painter.widget`, not
|
||||
`widget_within(&self.inner, painter.region())`.** §6's original text drew
|
||||
the sample as the latter. `widget_within` composes its `region` argument
|
||||
as *local*, `UiRegion::FULL`-relative coordinates against
|
||||
`painter.region()` (exactly what `UiRegion::FULL.within(&self.region) ==
|
||||
self.region` relies on); handing it `painter.region()` itself —
|
||||
already-resolved, window-relative coordinates — composes that frame a
|
||||
second time. For the root widget this is silently the identity (its
|
||||
region already is `[0,1]`), which is why it can look correct in a
|
||||
trivial case and only breaks once something is nested — i.e. always, in
|
||||
practice. Symptom: a centered child rendered at a wildly wrong offset
|
||||
nested more than one level deep. Fixed by using `painter.widget`, which
|
||||
hands the child `self.region` unmodified, with no second composition.
|
||||
|
||||
2. **A widget that reports a size smaller than its offered region must
|
||||
actually paint at that size, anchored top-left of what it was given —
|
||||
not fill the full offered region while merely *reporting* a smaller
|
||||
number.** `Sized` and `MaxSize` both had exactly this bug: their
|
||||
`desired_width`/`desired_height` predecessors capped the *reported*
|
||||
value but their `draw` bodies called `painter.widget(&self.inner)`
|
||||
unconstrained, which was harmless under the old two-pass model (a parent
|
||||
always queried the size *before* drawing, so by the time `draw` ran the
|
||||
offered region already matched) but wrong under `Aligned`'s new
|
||||
provisional-draw-then-reposition pattern, which offers the *whole*
|
||||
region on the first, learning pass. Symptom: a `.sized((100, 100))` rect
|
||||
rendered stretched to fill its whole row instead of a 100×100 square.
|
||||
Fixed by having both widgets carve the declared sub-region (`UiSpan`
|
||||
sized to the axis's `Len`, anchored at `AxisAlign::Neg`) out of whatever
|
||||
they were offered before drawing the child in it. `Image` needed the
|
||||
same treatment from the start (`texture_within` at its own natural size,
|
||||
not `texture()` at the full offered region) and was written that way in
|
||||
the first pass, once this was understood; `Rect`'s "fill whatever I'm
|
||||
given" is the one case where painting the *whole* offered region really
|
||||
is the declared behavior, so it needed no change.
|
||||
|
||||
3. **The move-offset chain's `parent` link cannot be found by looking up
|
||||
the parent's `ActiveData` in `draw_inner`, because the parent's
|
||||
`ActiveData` does not exist yet while its own `Widget::draw` is still
|
||||
running.** `ActiveData` is inserted only after `draw` returns
|
||||
(`render_state.rs`, end of `draw_inner`), so a child drawn partway
|
||||
through its parent's `draw` body — the ordinary case, since every
|
||||
composite widget draws its children from inside its own `draw` — would
|
||||
always read "no parent" from `self.active`, silently orphaning it at the
|
||||
root of the chain. Fixed by threading the parent's `move_slot` down
|
||||
through `Painter` (it already carries `mask`/`layer` the same way) and
|
||||
passing it explicitly into `draw_inner` as `parent_move_slot`, rather
|
||||
than deriving it from `self.active.get(parent_id)`. `move_parent_of`
|
||||
(the `self.active`-based lookup) is kept, but only for `redraw()`, whose
|
||||
target's parent genuinely is already active at that call site — the
|
||||
doc comment on it says which is which. Symptom: `reposition` computed
|
||||
the right delta and wrote it to the right slot, but the shader never
|
||||
saw it, because the primitive doing the actual painting chained to
|
||||
`u32::MAX` one level too early.
|
||||
|
||||
4. **`Painter::reposition` cannot reuse `active.region` as "where the
|
||||
widget currently is," because for a widget offered more room than it
|
||||
used, `active.region` is the *offered* box, not the *painted* one.**
|
||||
This only matters for `reposition` (used by `Aligned`); `mov` (used by
|
||||
`draw_inner`'s own same-size-different-position dispatch, for `Scroll`
|
||||
and `Offset`) has no such gap, because there the offered region *is*
|
||||
the visual footprint — content is sized to fill exactly what it is
|
||||
given. `reposition` instead reconstructs "from" as `active.size`
|
||||
(already tracked, per §5) anchored at `AxisAlign::Neg` within
|
||||
`active.region` — i.e. it assumes the child painted itself top-left of
|
||||
whatever it was offered, per point 2's convention — and **overwrites**
|
||||
the slot's delta rather than accumulating it the way `mov` does, since
|
||||
"from" is recomputed fresh from stable inputs every call and repeating
|
||||
the same `reposition` (an unrelated redraw elsewhere re-running this
|
||||
widget's parent) must not drift further each time. The one shape this
|
||||
does not cover: `Aligned` wrapping `Aligned`, where the inner one's own
|
||||
`reposition` may have moved its content away from top-left already. No
|
||||
widget or example in this codebase builds that today; if one needs to,
|
||||
`reposition` would need the child to report *where* it painted, not
|
||||
just how big, which is a larger change than this pass's scope.
|
||||
|
||||
5. **A widget's `move_offsets` slot is allocated once, on its first-ever
|
||||
draw, and reused in place — never reallocated — for every later redraw
|
||||
of the same id, with its delta reset to `[0, 0]` on each reuse.** Not
|
||||
spelled out in §2's original text, which only said slots are assigned
|
||||
"when the widget is first drawn." Reallocating a fresh slot on every
|
||||
redraw would leave any *retained* (not-redrawn) descendant's `parent`
|
||||
link pointing at a now-orphaned old slot — a permanent leak, and worse,
|
||||
a descendant that silently stops tracking its ancestor's future moves.
|
||||
Resetting the delta on reuse (rather than carrying it forward) is
|
||||
required because a full redraw bakes the widget's correct absolute
|
||||
position into the fresh `region` argument directly; a stale delta left
|
||||
over from before the redraw would double-offset it.
|
||||
|
||||
Two further points worth recording because they were *design decisions*
|
||||
made while implementing, not bugs — `LAYOUT.md`'s own text left them
|
||||
unspecified rather than getting them wrong:
|
||||
|
||||
- **`Scroll` offers its content a region sized by the *previous* frame's
|
||||
measured content length, not a fresh one.** A fresh measurement would
|
||||
require drawing the content once to learn its size and — since that
|
||||
provisional size essentially never matches the previously active one —
|
||||
redrawing it a second time at the real size, on every single scroll
|
||||
tick, which is exactly the cost §2 exists to remove. Using the stale
|
||||
length means an ordinary scroll (position changes, content does not)
|
||||
offers the same *size* as last frame, only shifted, which is what makes
|
||||
`draw_inner` dispatch it as the O(1) move. The cost: a real content-size
|
||||
change lags one frame before the container's scroll range reflects it,
|
||||
self-correcting the frame after (the content length itself, read from
|
||||
what was actually drawn, is never stale — only the offered *region* used
|
||||
for placement is). No example in this repository builds a `Scroll` yet,
|
||||
so this could not be checked against a pixel diff; it is covered instead
|
||||
by `iris/src/layout_tests.rs`'s three `Scroll`-based unit tests, which
|
||||
build a tree and drive `UiRenderState` directly with no GPU or window
|
||||
needed.
|
||||
- **`redraw()`'s parent-relayout check draws the widget first, then
|
||||
compares the fresh `ActiveData.size` the draw produced against the size
|
||||
from before removal** — the mirror image of the old code's "query size,
|
||||
compare, decide whether to draw," which no longer has a size query to
|
||||
do the comparison with before drawing (§5 deleted `Cache`/`SizeCtx`
|
||||
along with `desired_width`/`desired_height`). This can occasionally draw
|
||||
a widget once more than the old code would have (if the parent it
|
||||
bubbles up to ends up redrawing the same widget again as part of its own
|
||||
relayout) — `draw_inner`'s own skip/move dispatch absorbs most of that
|
||||
redundancy for free, and this path is not one of §8's measured
|
||||
conditions, so the remaining slack was accepted rather than chased
|
||||
further.
|
||||
|
||||
## For IRIS.md
|
||||
|
||||
When this lands, copy this entry into `IRIS.md` (newest first):
|
||||
|
||||
> **2026-09-04 — `Widget::draw` reports the size it used; `desired_width`/
|
||||
> `desired_height` are gone.** A widget used to implement three methods
|
||||
> (`draw`, `desired_width`, `desired_height`); it now implements one,
|
||||
> `fn draw(&mut self, painter: &mut Painter) -> Size`, which draws into
|
||||
> `painter.region()` and returns how much of it was used. Why: the two
|
||||
> extra methods routinely re-simulated what `draw` was about to do anyway
|
||||
> (`Span::desired_ortho` copied its own draw loop to get cross-axis sizing
|
||||
> right) — one visit per widget per frame instead of up to three. A
|
||||
> container that needs a child's size before placing it (alignment,
|
||||
> centering) draws the child once at a provisional region, reads the
|
||||
> returned `Size`, and calls the new `Painter::reposition` to move it into
|
||||
> its final spot — an O(1) offset write, not a second draw. A widget whose
|
||||
> drawn output never depends on the size it's given (a fixed-size `Rect`,
|
||||
> a decoded `Image`) overrides the new `fn is_size_independent(&self) ->
|
||||
> bool { false }` to `true`, which skips redrawing it when only its
|
||||
> offered region changes shape.
|
||||
>
|
||||
> ```rust
|
||||
> // before
|
||||
> fn draw(&mut self, painter: &mut Painter) { /* ... */ }
|
||||
> fn desired_width(&mut self, ctx: &mut SizeCtx) -> Len { /* ... */ }
|
||||
> fn desired_height(&mut self, ctx: &mut SizeCtx) -> Len { /* ... */ }
|
||||
>
|
||||
> // after
|
||||
> fn draw(&mut self, painter: &mut Painter) -> Size { /* ... */ }
|
||||
> ```
|
||||
>
|
||||
> `SizeCtx` and `Cache` are gone with it — see `LAYOUT.md` for the full
|
||||
> design, the move-offset mechanism this shipped alongside, and the file
|
||||
> list.
|
||||
+983
@@ -0,0 +1,983 @@
|
||||
# ai-app — plan
|
||||
|
||||
A phone interface to AI coding sessions — Claude Code and llama.cpp — built
|
||||
to replace the Claude app for day-to-day use. Two motivations: local models
|
||||
need a front end at all, and owning the client means fixing what the official
|
||||
app gets wrong (it won't deliver a typed message until the turn fully
|
||||
finishes, where the TUI injects it at the next tool boundary).
|
||||
|
||||
Same shape as `../dev-updater`: a Rust (Axum) backend on the desktop, a
|
||||
Kotlin/Compose Android app, pinned self-signed TLS between them.
|
||||
|
||||
This file records decisions with their date, their rationale, and what was
|
||||
rejected. Update it in place when one changes; `AGENTS.md` is the working
|
||||
notes layer and must not become a second version of it.
|
||||
|
||||
## The one idea everything hangs off
|
||||
|
||||
**A session is a child process, translated into one common event model.**
|
||||
The backend spawns it, translates its dialect into a common event stream,
|
||||
and keeps an append-only transcript. A new session type is a new driver —
|
||||
never a session-type branch in shared code (routes, transcript, app
|
||||
screens). SSH falls out of the same shape: a remote session is the identical
|
||||
command wrapped in `ssh host …`, and the driver never learns which it got.
|
||||
|
||||
```
|
||||
Android app (Compose)
|
||||
│ HTTPS (pinned CA) — REST for actions, SSE for live events
|
||||
▼
|
||||
backend (Rust/Axum, desktop)
|
||||
├─ SessionManager ── Session ── Driver (trait)
|
||||
│ ├─ ClaudeDriver (claude stream-json over stdio)
|
||||
│ ├─ LlamaDriver (llama-server over HTTP)
|
||||
│ └─ EchoDriver (the test rig)
|
||||
│ each driver's process is spawned through a Transport,
|
||||
│ locally or as `ssh host …`, decided by the setup it names
|
||||
├─ usage.rs (Anthropic OAuth usage endpoint, per machine)
|
||||
├─ models.rs (HuggingFace browsing and GGUF downloads)
|
||||
├─ files.rs (the file explorer's half of the backend)
|
||||
└─ config.ron + per-session transcript files
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Setups and providers (2026-08-28)
|
||||
|
||||
**A setup is a machine, and it carries the providers that machine has.**
|
||||
Optional ssh details, plus the list of what can be run there. Spawning is
|
||||
two choices in order: pick a setup, then pick one of its providers.
|
||||
|
||||
This replaced an independent providers × hosts cross-product, because the
|
||||
two axes are not independent: a provider is only real on a machine where
|
||||
that CLI is installed, so the cross-product offered combinations that cannot
|
||||
work — `claude-cli` on a machine with no `claude`, and every provider paired
|
||||
with a host the driver ignores (`EchoDriver` takes no host, so "Run on" was
|
||||
a control that silently did nothing).
|
||||
|
||||
- **Echo is seeded, not implicit.** It lives in the setup with no ssh,
|
||||
because it runs in-process and has no transport to cross. It is written
|
||||
into `config.ron` on first run rather than conjured at read time — a
|
||||
provider nobody can see in the file is one nobody can edit from the phone.
|
||||
- **Providers are discovered by asking the machine**, never typed, so an
|
||||
enrolled token cannot introduce a command. The escape hatch for a binary
|
||||
somewhere unusual is editing `config.ron`, deliberately the one authority
|
||||
the phone does not have.
|
||||
- **Migration code is deleted once the update carrying it is received.** The
|
||||
providers/hosts migration ran on the one host there is and is gone. A file
|
||||
in the old shape now fails to parse, which is correct because no such file
|
||||
exists.
|
||||
|
||||
### Backend layout (`server/`)
|
||||
|
||||
axum 0.8, axum-server + rustls, tokio, serde, clap, tracing. Rust edition
|
||||
2024, warning-clean, clippy clean.
|
||||
|
||||
- `main.rs` — bootstrap, TLS listener, auth layer, enrollment, wg0 binding.
|
||||
- `routes.rs` — the whole HTTP table in its module doc comment. **That
|
||||
comment is the surface's source of truth**; this file does not repeat it.
|
||||
- `auth.rs` — the bearer-token middleware.
|
||||
- `config.rs` — the persisted schema.
|
||||
- `setups.rs` — machines and provider discovery.
|
||||
- `files.rs` — the file explorer (`EXPLORER.md`).
|
||||
- `usage.rs` — Anthropic usage polling, per machine.
|
||||
- `models.rs` — HuggingFace browsing and GGUF downloads.
|
||||
- `media.rs` — the image media-type/extension table, shared by the four
|
||||
places that must agree: storing an upload, serving it back, handing one to
|
||||
a driver, and saving one a tool produced.
|
||||
- `session/mod.rs` — `SessionManager`, the live registry; every mutation
|
||||
funnels through it so in-memory and on-disk state cannot come apart.
|
||||
- `session/driver.rs` — the `Driver` trait and the common event model.
|
||||
- `session/claude.rs`, `session/llama.rs`, `session/echo.rs` — the drivers.
|
||||
- `session/transcript.rs` — the append-only JSONL event log per session,
|
||||
with monotonically increasing sequence numbers (the phone's resume cursor).
|
||||
- `session/transport.rs`, `ssh.rs` — running a driver's command locally or
|
||||
over ssh.
|
||||
- `session/process.rs` — the pid + start-time record that lets a process
|
||||
outlive the backend.
|
||||
- `session/import.rs` — continuing a Claude Code session the machine has.
|
||||
- `session/pending.rs` — operations in flight on importable sessions.
|
||||
|
||||
The certificates, enrollment, wg0 binding, owner-only file modes and RON
|
||||
house rules live in the `wg-app-link` submodule, shared with dev-updater.
|
||||
|
||||
### The common event model
|
||||
|
||||
Driver output, whatever the dialect, is normalized into one enum before it
|
||||
touches the transcript or the phone. Every event is appended to the session's
|
||||
transcript with a sequence number, then fanned out to SSE subscribers. The
|
||||
phone renders purely from this stream: reconnecting is "give me events after
|
||||
seq N", so there is no separate history path to drift from the live one.
|
||||
|
||||
- `UserMessage { text }` — echoed into the transcript **by the manager, not
|
||||
by drivers**, so every device renders the conversation from one stream.
|
||||
- `AssistantText { delta }` — streaming text, rendered as markdown.
|
||||
- `ToolStart / ToolUpdate / ToolEnd { tool, input, output }`.
|
||||
- `Image { ref }` — saved under the session dir, fetched by URL.
|
||||
- `Question { id, prompt, options }` — anything needing a human. Claude's
|
||||
AskUserQuestion and permission requests (canUseTool) are the same shape;
|
||||
a permission is a question with two bare options, not a different kind.
|
||||
- `Answered { id, answer }` — so a question card resolves on every connected
|
||||
device, not just the one that answered.
|
||||
- `Status { state }` — idle / running / awaiting-input / compacting / exited.
|
||||
- `UsageDelta { tokens, context }` — what a turn cost and how much the model
|
||||
was holding when it ended. `context` is prompt plus both cache figures,
|
||||
taken from the **last assistant message** rather than the turn's `result`:
|
||||
measured 2026-08-30 against CLI 2.1.237, the result adds a turn's messages
|
||||
up, so its cache read of 40,211 was the same conversation counted twice.
|
||||
It is carried rather than summed, because it goes *down* — a compaction
|
||||
replaces it and a clear leaves it unmeasured. `driver::context_after` is
|
||||
that rule and the phone folds with the same one. A session the server has
|
||||
no measurement of asks the CLI's own file instead of waiting for a turn
|
||||
(`import::context_of`).
|
||||
- `MessageQueued` / `MessageDropped` — see "Taking a queued message back".
|
||||
- `PeerMessage` — see "A message from another agent".
|
||||
- `Error { message }`.
|
||||
|
||||
Inbound, the `Driver` trait is small: send a message, answer a question,
|
||||
interrupt, set the model, compact, unqueue, and two ways out — `detach` (the
|
||||
server is going away and means to come back) and `stop` (the session is being
|
||||
deleted, so the process must not survive). Every driver owes exactly one of
|
||||
the two.
|
||||
|
||||
`send_user_message` during a run is the point of the whole app: the dialect
|
||||
queues it for injection at the next tool boundary rather than the end of the
|
||||
turn.
|
||||
|
||||
### Claude driver specifics
|
||||
|
||||
Spawn: `claude -p --verbose --input-format stream-json --output-format
|
||||
stream-json --permission-mode <mode>` in the chosen working directory, plus
|
||||
`--model` and, where one has been chosen, `--effort`. Wire-format notes are
|
||||
pinned against CLI 2.1.237 in `session/claude.rs`'s module doc: permissions
|
||||
need the hidden `--permission-prompt-tool stdio` flag, AskUserQuestion answers
|
||||
ride `updatedInput.answers` keyed by question text, and `set_model`/`interrupt`
|
||||
are control requests.
|
||||
|
||||
**The thinking level is settled at launch** (added 2026-09-04, because it is
|
||||
the largest saving available on a long session: output is about an eighth of
|
||||
what a session costs and thinking is the bulk of output, against the ~1.5% that
|
||||
is prose). The CLI's only two setting control requests are `set_model` and
|
||||
`set_permission_mode` -- checked against the 2.1.258 binary -- so there is no
|
||||
way to ask a running process to think differently. `set_session_effort` is
|
||||
therefore shaped like `set_session_cwd` rather than like `set_session_model`:
|
||||
it records the level and **stops the process**, and the next message or Start
|
||||
launches one that has it. It lives in the session settings dialog beside the
|
||||
working directory for that reason, not on the session bar beside the model and
|
||||
the mode, which do take effect mid-turn. `None` is a level in its own right --
|
||||
the CLI's own default -- so the picker can return to it; a level this app named
|
||||
as the default instead would be this app choosing one.
|
||||
|
||||
**What a new session starts at is `Config::default_effort`**, applied in
|
||||
`spawn_session` rather than filled in by the spawn screen, so it holds for an
|
||||
import and a bare API call as well. It is set by the spawn screen's own
|
||||
picker, whose label says so: one control, where new sessions are made, rather
|
||||
than a settings page for a single value. It is not on a provider, because
|
||||
providers are discovered and the next rediscovery would erase it, and not on
|
||||
the phone, because a second device would then spawn at a level nobody there
|
||||
chose. `GET`/`POST /defaults` carry it, as a struct rather than a bare value
|
||||
so the permission mode -- still hardcoded to `auto` on the spawn screen -- can
|
||||
move there without a second route.
|
||||
|
||||
**`--resume` only ever runs when nothing else has that session open.** That
|
||||
is the rule behind the import refusal, the single `ClaudeDriver::launch`
|
||||
entry point, and the `Exited` correction below; two CLIs on one session file
|
||||
duplicate the conversation into it and bill the second for re-reading it all.
|
||||
|
||||
### The llama driver
|
||||
|
||||
One `llama-server` per session, started through the same `Transport` as any
|
||||
other process and then reached over HTTP on a loopback port. Two things are
|
||||
deliberate and easy to undo by accident:
|
||||
|
||||
- **The conversation is rebuilt from the transcript**, not kept in the
|
||||
driver. A copy in driver memory is invisible to a second device and gone
|
||||
when the process restarts. That leaves the Claude driver as the odd one
|
||||
out rather than this one — the CLI's memory is a cache in front of the same
|
||||
transcript. Resolve any inconsistency in this direction.
|
||||
- **A llama session runs on whatever machine its setup names** (2026-09-04,
|
||||
the last of phase 5). A transport is "run this" plus "reach this port", and
|
||||
the second half is `Transport::reserve_port` — the port the server binds
|
||||
*there* and the port that reaches it *here*, the same number locally —
|
||||
carried by `Launch::reaching` onto the connection that already runs the
|
||||
command. `llama-server` binds loopback on the far machine, so nothing is
|
||||
served to its network. The far port is a guess from a range below the
|
||||
ephemeral one, because no portable way to ask a machine for a free port
|
||||
avoids racing the bind anyway; a collision is not silent, since the server
|
||||
fails to bind and the readiness poll reports what its log said.
|
||||
- **The model file lives on the machine that serves it** (2026-09-04). Each
|
||||
setup names its own models directory (`SshConfig::models_dir`, default
|
||||
`~/.local/share/ai-app/models` expanded *there*), and a spawn resolves the
|
||||
key on that machine — one round trip answering "at /abs/path" or "missing",
|
||||
so a model that is not there is refused at the spawn rather than becoming a
|
||||
server that never becomes ready. The spawn screen offers
|
||||
`GET /setups/{id}/models`, that machine's list, rather than `GET /models`,
|
||||
which is this backend's downloads. Downloading *to* another machine is
|
||||
deliberately not built: a multi-gigabyte transfer with no progress
|
||||
anywhere, and the file gets there however anything else on that machine
|
||||
did.
|
||||
- **The readiness poll watches the process, not only the port.** A model that
|
||||
will not load, a port already taken, a flag an older build does not know:
|
||||
all exit within a second and none will ever answer `/health`, so waiting
|
||||
out the 300s timeout turned the server's own account of the problem into
|
||||
"gave up". The failure carries the tail of `llama-server.log`, which on a
|
||||
remote session is the only copy anybody reading the phone can see.
|
||||
|
||||
### Models (2026-08-28)
|
||||
|
||||
- **A download belongs to the model, not to the request.** Keyed by
|
||||
`owner/repo/file.gguf` and owned by the server, so a second device can
|
||||
watch one it did not start and an hour-long fetch survives a locked screen.
|
||||
Every run has an id and its outcome outlives it, because "not downloading"
|
||||
otherwise means finished, never started, or someone else's run ended while
|
||||
you were away.
|
||||
- **Progress is measured**, never estimated: `total` is Content-Length, or
|
||||
Content-Range's last field on a resume, and absent when the server says
|
||||
nothing.
|
||||
- **Resume is guarded by identity, not by hope.** A partial carries the ETag
|
||||
it was written against and a mismatch discards it. `If-Range` would be the
|
||||
tidy mechanism but HuggingFace's CDN ignores it (probed 2026-08-28). The
|
||||
published sha256 is checked before the file is renamed.
|
||||
- Sampling parameters reach a driver as an untyped `params` map, so the
|
||||
shared schema does not grow llama.cpp's vocabulary.
|
||||
|
||||
### Transport (ssh)
|
||||
|
||||
- A remote session is a local one with the command wrapped in `ssh -T host …`,
|
||||
every argument shell-quoted, run with `exec` so dropping the connection
|
||||
takes the CLI down rather than orphaning it. Key-based auth only, through
|
||||
the system `ssh` client, which inherits `~/.ssh/config`, agents and jump
|
||||
hosts for free.
|
||||
- **The transport wraps the driver, not the other way round** (2026-08-28).
|
||||
A driver says what to run; something above it turns that into a process.
|
||||
Otherwise transport knowledge sits inside a translator whose job is a wire
|
||||
format, and every future driver has to remember to do the same.
|
||||
- **A forwarded launch gets a pty and every other one does not** (measured
|
||||
2026-09-04). Killing the ssh client ends a CLI because it closes the stdin
|
||||
that CLI is reading; `llama-server` never reads its stdin, so the same kill
|
||||
left it running on the far machine with the model loaded — one orphan per
|
||||
stopped session. With `-tt` the far side takes SIGHUP when the connection
|
||||
goes. Its log then arrives through a line discipline, which nothing parses.
|
||||
`-T` stays everywhere else, where a pty would rewrite the JSONL.
|
||||
- **`command -v` follows ssh's non-login PATH**, which is narrower than an
|
||||
interactive shell's, so a binary somewhere unusual is invisible to
|
||||
discovery. Point `command` at an absolute path.
|
||||
- **Images need no file transfer.** `attachment_block` base64s an upload into
|
||||
the stream-json message, and produced images come back the same way.
|
||||
- **Any other file is told to the session by path** (2026-09-03): a trace, a
|
||||
log, a zip — things a model cannot be shown and the CLI can read. The
|
||||
upload is streamed to disk under the session's attachments on this machine,
|
||||
and the message ends with `Attached file: /abs/path`. For a session on
|
||||
another machine the upload also copies the file there in the same request,
|
||||
over one `ssh` invocation, landing in the setup's `attachmentsDir` if set,
|
||||
else the session's cwd, else the login home. The resolved remote path is
|
||||
recorded beside the file (`<name>.remote`) and is what the driver names. A
|
||||
copy that fails fails the upload, so no message ever names a file that is
|
||||
not there.
|
||||
|
||||
### Moving a session to another directory (2026-08-31)
|
||||
|
||||
`POST /sessions/{id}/cwd`, from the session settings dialog. A working
|
||||
directory is settled when the process is spawned, so this records the new one
|
||||
and **ends** the process in the old one. It does not start a replacement: a
|
||||
session with no process starts on the next thing said to it or on Start,
|
||||
which is this app's rule everywhere else.
|
||||
|
||||
The path is checked against the session's own machine and **refused** if it
|
||||
is not there, rather than corrected. The spawn path corrects instead, because
|
||||
it is resuming a directory the *machine* recorded, which can be gone through
|
||||
nobody's fault; a path somebody has just typed is different, and a mistyped
|
||||
one accepted here surfaces much later as a session that will not start.
|
||||
|
||||
**Nothing of Claude Code's own is moved.** Measured against CLI 2.1.237:
|
||||
`claude --resume <id>` finds a session from any working directory. Relocating
|
||||
the file would mean reproducing a rule this app cannot see the whole of — the
|
||||
project directory is the path with every non-alphanumeric character replaced
|
||||
by `-`, truncated at 200 characters with a hash appended, and overridable.
|
||||
|
||||
### A message from another agent (measured 2026-08-31)
|
||||
|
||||
Measured by sending a real cross-session message to a real stream-json
|
||||
session on CLI 2.1.237: the CLI emits **no `user` record** for it, and
|
||||
nothing in the partial-message stream mentions it. The whole of it arrives as
|
||||
an `origin` object on the turn's `result`, in the same shape the session file
|
||||
records — so `import::peer_message` reads both and there is one function for
|
||||
one wire format. Only peer-caused turns carry it.
|
||||
|
||||
**The cost is the position, and it is paid on the wire rather than on
|
||||
screen.** The event cannot be recorded in place: at no earlier point does the
|
||||
CLI say why the turn started, and the transcript is append-only, so by the
|
||||
time anyone knows, everything the message caused is already written above it.
|
||||
Tailing the CLI's own session file instead was rejected and stays rejected —
|
||||
two sources of truth for one conversation and a poll per live session.
|
||||
|
||||
So `PeerMessage` carries a `turnStart`: the seq of the `Status` that opened
|
||||
the turn, stamped by the pump, which is the only thing that knows a seq and
|
||||
sees every driver's turns. The phone draws the note at that seq. A status
|
||||
draws no row, so there is nothing to collide with and the list stays sorted,
|
||||
which is what the scroll anchor and paging depend on. `turnStart` is absent
|
||||
where there is nothing to correct — a message replayed by `import` is already
|
||||
in the right place. The echo driver models both shapes: `/peer` and
|
||||
`/peer-turn`.
|
||||
|
||||
### Taking a queued message back (2026-08-31)
|
||||
|
||||
`POST /sessions/{id}/unqueue`, answered by `Driver::unqueue` and recorded as
|
||||
`Event::MessageDropped` so every device loses the bubble and a reconnect does
|
||||
not replay it.
|
||||
|
||||
The answer has **three** states rather than a yes/no, and that is the whole
|
||||
design: `Dropped`, `AlreadySent`, and `Unknown`. The Claude driver can only
|
||||
ever give the middle one — it writes a steer into stdin the instant it
|
||||
arrives, which is what makes a steer reach the model at the next tool
|
||||
boundary instead of the end of the turn. What waits in `awaiting` is the
|
||||
*announcement*, not the message. Holding the write until a boundary would
|
||||
make the drop real everywhere but costs a steer one model call, which is the
|
||||
latency the immediate write removed. So the refusal is the honest answer, and
|
||||
it is reported on the bubble the reader pressed rather than in the screen's
|
||||
error row a screen away.
|
||||
|
||||
`Unknown` is not "we could not find out": a driver that is gone reported
|
||||
everything it was holding when it closed.
|
||||
|
||||
### Session processes outlive the backend (2026-08-29)
|
||||
|
||||
A session's process is **left running when the backend stops and adopted
|
||||
again when it starts.** A rebuild, a service restart or a crash must not end
|
||||
a turn somebody is waiting on, and a turn can be minutes long. What this
|
||||
replaced leaked processes either way: `shutdown_all` asked every driver to
|
||||
stop and then exited immediately, with the SIGKILL escape hatch on a timer
|
||||
inside the dying runtime, and whatever survived was orphaned with nothing
|
||||
written down to find it by.
|
||||
|
||||
Inside the session directory, beside the transcript:
|
||||
|
||||
- `process.json` — the pid, the kernel's **start time** for that pid, and how
|
||||
much of the output log has been read. The start time is what makes the pid
|
||||
an identity: pids are reused, and adopting a stranger's would mean never
|
||||
resuming the real conversation and signalling something unrelated.
|
||||
- `stdin.fifo` — opened **read-write** and inherited by the process, so it is
|
||||
its own last writer and never reads EOF when the server goes away. Closing
|
||||
stdin therefore stops being the graceful-exit signal; ending a process is a
|
||||
signal, and only `Driver::stop` sends one.
|
||||
- `stdout.log` / `stderr.log` — plain appended files, read from a byte
|
||||
offset. A fifo would fill its 64 KB buffer and block the process while
|
||||
nothing drained it, stalling the very turn this exists to protect.
|
||||
|
||||
**Remote sessions are adopted too, and the recorded pid is the `ssh`
|
||||
client's** — the process the backend owns, which lives exactly as long as the
|
||||
remote command does. The far `claude` always has an sshd pipe on stdin
|
||||
whichever version started it, since the fifo is on the backend's side, so a
|
||||
remote session's stdin says nothing about which server started it.
|
||||
|
||||
**A zombie is dead.** `/proc/<pid>/stat` keeps the entry, with the same pid
|
||||
and start time, until the exit status is collected — so a finished process
|
||||
answered "still there" for as long as nothing reaped it, and `Alive` is the
|
||||
word that makes `Exited` unsayable. `process::stat_of` reads the state field
|
||||
alongside the start time.
|
||||
|
||||
### Stopping and starting a session's process (2026-08-30)
|
||||
|
||||
`POST /sessions/{id}/stop` and `/start`: end the process without ending the
|
||||
session, and start it again on the same conversation. Three decisions worth
|
||||
not undoing:
|
||||
|
||||
- **Stop signals the recorded process and says nothing else.** It does not go
|
||||
through the driver and does not announce `Exited`. The record is the
|
||||
session's rather than any dialect's, so this works for a session whose
|
||||
driver is in no state to be asked, and the driver's own reader already
|
||||
reports the death correctly. Announcing it here would be a guess arriving
|
||||
ahead of the measurement, and wrong for the grace period.
|
||||
- **Start replaces the driver and nothing else.** The transcript, the event
|
||||
pump and every open SSE stream stay where they were, so starting again is
|
||||
not a reconnect for anybody watching, and there is still exactly one writer
|
||||
of the transcript. `LiveSession` and `Commands` share one
|
||||
`Mutex<Arc<dyn Driver>>` rather than each holding a copy.
|
||||
- **Start is refused unless the session is *known* to have exited.**
|
||||
`Unknown` means nobody could find out, and starting on that is exactly the
|
||||
two-CLIs-on-one-conversation fault `session::process` exists to prevent.
|
||||
|
||||
**`Exited` is a claim about a process, and the record is what settles it.**
|
||||
It is the one status that draws the phone's Start button and lets
|
||||
`start_session` build a driver, so it is checked against `session::process`
|
||||
before it is believed (`corrected`, called in `launch` and `start_session`).
|
||||
A record not known to be dead makes it false and the session reports
|
||||
`Unknown` instead. Every other status is left alone — those are the pump's,
|
||||
written from what the process itself said. Without this, a session adopted at
|
||||
a backend start kept the transcript's `Exited` while its CLI ran, Start was
|
||||
accepted every press, and each press attached *another* reader to one
|
||||
process: one reply drawn interleaved several times over
|
||||
(`GotGotGot it — it — it —`). **A driver that `start_session` replaces gets
|
||||
`Driver::detach`**, because swapping the `Arc` does not end the tasks the old
|
||||
one is running.
|
||||
|
||||
**Who says so matters as much as what is said.** A status written into the
|
||||
manager's view alone is two screens disagreeing — the list reads the
|
||||
manager's status and the session screen replays the transcript, which showed
|
||||
up as a stop button turning into a play button a moment after the screen
|
||||
opened. So **a driver announces the state it starts in, through the event
|
||||
sink.** It says `Idle` only when it *started* a process; adopting says
|
||||
nothing, because a process already running may be mid-turn and the
|
||||
transcript's last word is the better answer until its output says otherwise.
|
||||
|
||||
**A message or a command starts the process if there isn't one.** Refusing
|
||||
was work handed back: read the status word, find the other button, press it,
|
||||
type the thing again. `--resume` puts the new process on the same
|
||||
conversation, so nothing about what was typed changes. A rename is included
|
||||
for a sharper reason: Claude Code keeps its own copy of the name, that copy
|
||||
is what its session picker and other agents' session lists show, and a
|
||||
session is only ever *given* a name at birth since every later start is a
|
||||
`--resume` — so a rename reaching no process would leave the two lists
|
||||
disagreeing permanently. Its save happens before the telling, so a failure
|
||||
there says the telling failed rather than the rename.
|
||||
|
||||
`start_if_exited` is one function under one write lock, which is what stops
|
||||
two requests arriving together from starting two CLIs. Its callers want
|
||||
opposite answers: "there is already a process" is a refusal worth showing to
|
||||
somebody who pressed Start, and nothing at all to a message. Only `Exited`
|
||||
starts anything — `Unknown` has a process that may well be reading its fifo.
|
||||
`run_command` judges against what `start_if_exited` returned rather than
|
||||
re-reading a status the pump may not have caught up with.
|
||||
|
||||
On the phone this is one button in the composer, left of Send, whose mark and
|
||||
colour say what pressing it would do now: an orange pause while a turn runs
|
||||
(interrupt — the process stays), a red stop when it is not (end the process),
|
||||
a green play when it has exited. One button rather than three that come and
|
||||
go, so its presence is never the signal. It is disabled while its own request
|
||||
is in flight, as a courtesy; the server refuses the second request either way.
|
||||
|
||||
### A backend start adopts, and starts nothing (2026-08-30)
|
||||
|
||||
`SessionManager::new` takes charge of the processes still running and
|
||||
**leaves every other session exactly as it found it** — listed, with its
|
||||
transcript, its pump and its SSE stream, and no driver until somebody asks
|
||||
for one. It used to launch a driver for every session in the config, and
|
||||
`ClaudeDriver::launch` starts a process when there is none to adopt, so a
|
||||
session somebody had deliberately stopped came back at the next rebuild, and
|
||||
the `Idle` the new driver announced stamped it as active at the moment of the
|
||||
restart. On the phone that read as *every* session idle and "just now", with
|
||||
the list sorted by that time in an order that meant nothing.
|
||||
|
||||
- **`Launching` is the parameter that says which it is**, and an import's
|
||||
seed rides on the asked-for variant, because a restart re-seeding a
|
||||
transcript would write the imported conversation into it twice.
|
||||
- **A session with no process has no driver.** `DriverCell` is an option
|
||||
rather than a driver whose requests go nowhere, so "nothing is running
|
||||
this" is a state the code can be asked about instead of one it discovers by
|
||||
sending into a dead fifo. `LiveSession::ask` answers it with an
|
||||
`Event::Error` naming what could not happen — a request nobody can carry
|
||||
out is reported, never swallowed.
|
||||
- **A launch never moves a session's clock.** A status a launch has to
|
||||
correct is written at the time of the last thing the session actually did,
|
||||
not at `now()`. Taking charge of nothing, every word but `Exited` is
|
||||
disproved at once — a backend killed mid-turn leaves a transcript saying
|
||||
`Running`, which draws a stop button for a turn that ended hours ago — but
|
||||
stamping the correction with `now` is the same lie in the same field that
|
||||
`Transcript::last_activity` exists to prevent.
|
||||
- **A session that has never done anything reports when it was created.** Its
|
||||
transcript is empty, since a driver announcing the state it starts in is
|
||||
not news, so it is the one session with no line to read a time off. Not the
|
||||
file's mtime, which is a worse answer for a checkout that can be copied;
|
||||
`SessionConfig::created` is recorded rather than inferred.
|
||||
|
||||
### Sessions spawned while testing clean themselves up (2026-08-30)
|
||||
|
||||
`--throwaway-sessions`, **on by default in a debug build**. Every session
|
||||
such a server spawns is marked `throwaway` in the config, and a marked
|
||||
session's process is stopped when the server exits or is signalled.
|
||||
|
||||
Leaving processes running is right for the sessions somebody is using and
|
||||
exactly wrong for the ones a test made: those leave a `claude` behind that
|
||||
every later server adopts, they cost tokens if anything speaks to them, and
|
||||
nothing says they are there — twelve accumulated on this machine in a day.
|
||||
|
||||
- **The flag marks; the mark decides.** What a server was told at startup
|
||||
governs only the sessions it spawns, and the mark is written into the
|
||||
session, so it outlives that server. A session spawned deliberately keeps
|
||||
running whichever server is up when one exits, and a throwaway one is
|
||||
cleaned away even by a server started without the flag. The alternative —
|
||||
the exiting server stopping whatever it has marked in memory — makes
|
||||
cleanup depend on which process is up.
|
||||
- **Stopping is not asking.** `process::stop` leaves its SIGKILL on a tokio
|
||||
timer, which a shutting-down runtime never runs; that is precisely how the
|
||||
original `shutdown_all` leaked. The exit path waits with
|
||||
`process::wait_gone` — one deadline for all of them, since they were
|
||||
signalled together — and kills whatever is left.
|
||||
|
||||
### Importing refuses a session that is already open (2026-08-29)
|
||||
|
||||
Claude Code keeps a descriptor per live session at
|
||||
`~/.claude/sessions/<pid>.json` carrying the `sessionId` and a `procStart` —
|
||||
the same pid-plus-start-time identity used above. So "is this session open
|
||||
right now" is a **measurement**, and the import list reports it as `no` /
|
||||
`yes` / `unknown`. Three answers because a machine that keeps no such record
|
||||
cannot answer, and "could not check" is not "nobody is using it".
|
||||
|
||||
`yes` is refused. On 2026-08-29 an agent imported the session it was itself
|
||||
running in: two `claude --resume` processes on one file, the whole 65 MB
|
||||
conversation with 154 embedded screenshots duplicated into it under a new
|
||||
prompt id, and the adopted copy billed for re-reading all of it. It ended at
|
||||
the account's session limit.
|
||||
|
||||
**Importing and deleting run on the server, and a batch is handed over in one
|
||||
call.** `POST /setups/{id}/importable/{delete,import}` each take a list of
|
||||
ids, answer 202, and do the work in spawned tasks — the phone that asked is
|
||||
free to leave, and used to cancel its own batch by doing so. A list rather
|
||||
than a route per session because one request per row made a handover only as
|
||||
atomic as the network, and a row nobody asked for looks exactly like a row
|
||||
nobody picked. Only the *registering* is atomic; the work settles per row,
|
||||
since six deletes that all roll back together is not something a filesystem
|
||||
offers.
|
||||
|
||||
What replaces the reply is `session::pending`: every row carries `pending`
|
||||
and `error`, and `/importable/events` streams the changes. **Both, not
|
||||
either** — the stream is a broadcast with no memory, so an operation that
|
||||
starts and finishes while it is still connecting is one nothing will ever be
|
||||
said about, which left a row marked "waiting" for ever. A single tap still
|
||||
waits, because "continue this and take me to it" needs the session it made
|
||||
and 202 does not carry one; the batch and the tap share `spawn` so the two
|
||||
cannot drift about what importing means.
|
||||
|
||||
An imported session **keeps itself level with the CLI's file**, so work done
|
||||
at a terminal appears without anyone pressing anything. Which lines came from
|
||||
*here* is answered by counting the events this session has recorded, **not**
|
||||
by looking at its status — a turn that starts and finishes between two polls
|
||||
reads as idle at both, and its own output gets replayed on top of itself.
|
||||
That bug was visible on screen as `donedone`.
|
||||
|
||||
### Usage limits (Claude)
|
||||
|
||||
Poll `https://api.anthropic.com/api/oauth/usage` — the endpoint behind Claude
|
||||
Code's `/usage` — with the OAuth token from `~/.claude/.credentials.json`,
|
||||
headers `anthropic-beta: oauth-2025-04-20` and `User-Agent:
|
||||
claude-code/<version>` (without the User-Agent it lands in an aggressively
|
||||
rate-limited bucket). Poll at ≥180 s, only while a Claude session exists or
|
||||
the usage screen is open, and cache the last answer. It is undocumented, so
|
||||
`usage.rs` treats every field as optional and degrades rather than erroring.
|
||||
|
||||
**Per provider, not per machine (2026-09-04).** A machine is not what is
|
||||
metered; the provider a session runs is. One machine offers echo, the Claude
|
||||
CLI and a local model side by side, and only the second spends anything — so
|
||||
pairing a session with a snapshot by machine alone drew the CLI's five-hour
|
||||
window under every echo session on it, a quota that session cannot spend. A
|
||||
session now names its meter (`usageProvider`, from
|
||||
`DriverKind::usage_provider`, which `usage::providers_for` reads too, so the
|
||||
two lists cannot disagree) and `GET /usage` is matched on machine *and*
|
||||
provider. `None` is a session that meters nothing, and the phone draws
|
||||
nothing at all for it — not a zero, and not "unknown".
|
||||
|
||||
`DriverKind::Echo` names a meter of its own that exists only when a test has
|
||||
asked for one: `/usage` in an echo session sets an invented answer
|
||||
(`usage::Fixture`), and with none set there is no snapshot and no bar. That
|
||||
is what makes those screens' states reachable — a number near the top, a
|
||||
window between blocks with no reset time, a machine nobody logged into, one
|
||||
that could not be reached — without spending real quota to arrange them,
|
||||
which is why none of them had ever been looked at.
|
||||
|
||||
**Per machine, not per backend (2026-08-29).** The credential store that
|
||||
matters is the one on the machine the session runs on, because that is the
|
||||
account being billed — and in the layout this aims at, `ai-server` is on the
|
||||
host, the host has no `claude`, and the CLI machine is a remote. So
|
||||
credentials are read through the session `Transport` (`$HOME` expanded by the
|
||||
far shell, because a path built locally is the wrong home), one snapshot per
|
||||
setup that offers Claude. The HTTP call stays on the backend, so the far end
|
||||
needs nothing but a shell.
|
||||
|
||||
The snapshot says which of four things happened rather than carrying a flag
|
||||
and a message: `ok`, `notLoggedIn`, `unreachable`, `failed`. `notLoggedIn` is
|
||||
the one that matters — a machine nobody put an account on is working as
|
||||
configured, and collapsing it into an error string made a healthy setup read
|
||||
as broken. A machine with no Claude provider is not asked at all.
|
||||
|
||||
**The five-hour window has no reset time between blocks, and that is not a
|
||||
missing value.** Measured 2026-08-31: the API anchors the window to the block
|
||||
it started in, and when no block is running there is nothing to reset, so
|
||||
`resets_at` is `null`. The weekly windows always have one because a week is
|
||||
always running. So absent means **not running**, and only a timestamp that
|
||||
arrives and cannot be parsed is unknown. `WindowEnd` in `ResetCountdown.kt`
|
||||
is the one rule both readers go through.
|
||||
|
||||
### Auto-resume (2026-09-05)
|
||||
|
||||
**A session may pick itself back up when the account's usage limit lifts.**
|
||||
Off unless somebody switched that session to it, because it spends quota the
|
||||
moment quota exists and does so with nobody looking — that is not a thing a
|
||||
default may decide. It sends one message, `continue` unless another was
|
||||
typed, and then it is done; there is no retry loop around the conversation
|
||||
itself.
|
||||
|
||||
**Running out of quota is a state, not an error.** `Event::LimitReached`
|
||||
carries the dialect's reset time where it gave one, and recognising it
|
||||
belongs to the driver — the Claude CLI ends the turn with `is_error` and
|
||||
`Claude AI usage limit reached|1788546972`, and nothing above the driver
|
||||
matches on a string. The transcript draws it as a divider, like a clear or a
|
||||
compaction: what a reader scrolling back wants from it is why the
|
||||
conversation stops at that line.
|
||||
|
||||
**The schedule is a plan to ask, never a plan to send.** Every reset time
|
||||
available here is untrustworthy in the direction that matters: the dialect's
|
||||
is written when the turn fails, and the endpoint's moves when the window
|
||||
does. So the wait ends in a question to `usage.rs`, and only `ok` with no
|
||||
window at 100% sends anything. A window still spent reschedules to *its own*
|
||||
reset time — which is what makes a limit that lifts later than promised wait
|
||||
longer, and one that lifts sooner resume sooner. A meter that cannot be
|
||||
asked at all is a longer wait too, never a send: "we could not find out"
|
||||
must not be able to produce the same action as "there is room".
|
||||
|
||||
Bounded, because something has to be: a day after the limit was hit the wait
|
||||
stops and says so in the session's own transcript. A machine that can never
|
||||
be asked would otherwise be retried for ever with nothing on screen saying
|
||||
so.
|
||||
|
||||
The schedule is persisted on the session (`resume: Some(ScheduledResume)`),
|
||||
not held in memory: a five-hour window routinely outlasts a backend restart,
|
||||
and a wait forgotten across one is a session that silently never comes back.
|
||||
`resume.rs` is the top layer — it holds the manager and the monitor and
|
||||
neither holds it — which is what lets the decision be a pure function of a
|
||||
snapshot and a clock. The pump reports limits downward on a broadcast, for
|
||||
the reason `Shared` exists: the pump runs underneath the manager.
|
||||
|
||||
**Exercised with echo, never with a real account.** `/limit [minutes]` in an
|
||||
echo session reports the same event a real driver does, and `/usage` sets
|
||||
what the meter answers — deliberately two commands, because the two
|
||||
disagreeing is the state the whole design is about. The loop was driven end
|
||||
to end that way on 2026-09-05: the wait moved from the dialect's two minutes
|
||||
to the meter's seven when the meter changed its mind, and the message went
|
||||
out on the first check after the meter came back under the limit.
|
||||
|
||||
### Subagents (2026-09-05)
|
||||
|
||||
**A subagent is a second transcript owned by a session, in the same event
|
||||
model, with no process and no controls of its own.** Full design and wire
|
||||
shape in `SUBAGENTS.md`, kept separate because the app half is being built
|
||||
against it in parallel and it is the shared contract between the two. The
|
||||
one-paragraph reason: a session's Task-tool helpers already speak the common
|
||||
event model on the parent's own stdout (each line carrying
|
||||
`parent_tool_use_id`), so giving each one its own small transcript — same
|
||||
file format, same paging routes, same SSE stream, reused by addressing rather
|
||||
than by copying — costs a routing step in the translator and a registry
|
||||
(`session/subagent.rs`) rather than a second session type with a driver, a
|
||||
process and a config entry it does not need.
|
||||
|
||||
### HTTP surface
|
||||
|
||||
**`routes.rs`'s module doc comment is the table.** REST for actions, one SSE
|
||||
stream per open session screen for events, all over the pinned TLS listener.
|
||||
SSE rather than WebSocket because resume-by-cursor (`Last-Event-ID` =
|
||||
transcript seq) is native to it and the inbound direction is plain POSTs.
|
||||
|
||||
Sessions live in `config.ron` (`$XDG_CONFIG_HOME/ai-app/`) plus a per-session
|
||||
directory under `$XDG_DATA_HOME/ai-app/sessions/` (transcript, attachments,
|
||||
produced images, process record), owner-only. Deleting a session is the
|
||||
complete path out of everything spawning one created.
|
||||
|
||||
**Every request body refuses fields it does not know**
|
||||
(`serde(deny_unknown_fields)`). A caller that misspells `permissionMode` got
|
||||
a 200 and a session in the default mode, which is indistinguishable from
|
||||
success at the place they are looking. Query strings are deliberately
|
||||
permissive.
|
||||
|
||||
**A phone that falls behind is answered with `reset`.** Past
|
||||
`CATCH_UP_LIMIT` the stream sends a `reset` frame and the newest window, and
|
||||
the client rebuilds from it exactly as it does when the screen opens. Not
|
||||
optional: without it the window is spliced onto rows no longer adjacent to
|
||||
it, which reads as ordinary output. The stream used to replay everything
|
||||
after the client's cursor, unbounded, while *opening* a session was bounded
|
||||
to a page — so a long disconnect delivered thousands of events one frame at a
|
||||
time.
|
||||
|
||||
### The file explorer (2026-09-03)
|
||||
|
||||
**`EXPLORER.md` holds this design.** The one-line version: a machine's
|
||||
filesystem, seen from the phone through the backend, keyed on the **setup**
|
||||
rather than on a session (a session only says where to start), with every
|
||||
operation one fixed shell script run through `Transport` so the local and the
|
||||
ssh case are one implementation.
|
||||
|
||||
### Security
|
||||
|
||||
- **TLS with a self-signed CA, pinned in the app.** Generated in process on
|
||||
first start into `$XDG_CONFIG_HOME/ai-app/certs`, so one place decides the
|
||||
extensions, the file modes and which addresses the leaf covers — every
|
||||
local IPv4 plus loopback and the emulator's host alias, so nobody maintains
|
||||
a hardcoded IP. The CA is created once and left alone; the leaf is reissued
|
||||
every start, so covering a new address is a restart. **Regenerating the CA
|
||||
strands the installed app** — the one-way door.
|
||||
- Unlike dev-updater, the pinned CA is **not a constant in the source**:
|
||||
the build reads `$XDG_CONFIG_HOME/ai-app/certs/ca.pem` from the machine
|
||||
doing the build and generates the constant (`generatePinnedCert` in
|
||||
`app/androidApp/build.gradle.kts`; `AI_APP_CA` overrides). That does
|
||||
three things at once — the trust anchor follows the build machine, so an
|
||||
APK built in the dev VM is only good for its emulator; there is no second
|
||||
anchor to add for development and forget to remove; and regenerating a CA
|
||||
needs a rebuild rather than a paste, so a stale constant cannot quietly
|
||||
disagree with the server.
|
||||
- **The dev VM is untrusted** (2026-08-25): not malicious, but it could
|
||||
become so. The repo is a read-write mount shared between the VM and the
|
||||
backend host, so everything in it — source, binaries, and the shell scripts
|
||||
the host runs — is attacker-writable.
|
||||
- **Nothing secret lives in the repo.** A CA private key the VM could read
|
||||
would let it mint a leaf the pinned app accepts, which is precisely the
|
||||
attack pinning exists to stop. Transcripts move for a plainer reason:
|
||||
they are whole conversations.
|
||||
- **The host should not execute what the VM can write** — build and run the
|
||||
backend from a host-only checkout rather than the shared mount. Moving
|
||||
the keys closes the smaller door; this is the larger one.
|
||||
- Accepted: a compromised VM can return anything it likes from the sessions
|
||||
it runs, since running an agent there is the point. The blast radius is
|
||||
that session's content, not the backend.
|
||||
- **This server's API *is* remote code execution** (spawn a
|
||||
bypass-permissions Claude on any ssh host). Pinning authenticates the
|
||||
server to the phone but not the phone to the server, so a bearer token adds
|
||||
the other direction. The token gates LAN-reachable RCE; it cannot defend a
|
||||
compromised backend host or phone — those are inside the trust boundary,
|
||||
and a compromised phone is handled by rotation.
|
||||
- **No route accepts a command.** Listing, reading and writing files are
|
||||
fixed scripts in `files.rs`; the phone chooses only the path and the
|
||||
bytes. Provider discovery asks the machine rather than taking a command.
|
||||
- **The explorer's routes take a path, and that is deliberate**
|
||||
(EXPLORER.md's decision 3). Elsewhere the phone picks an **id** and the
|
||||
server resolves which file it names, so an enrolled token cannot become
|
||||
"read me an arbitrary file" — the import listing is written that way. The
|
||||
explorer is different because the path is the whole feature, and it
|
||||
grants nothing new: the same token already spawns a bypass-permissions
|
||||
agent in any directory on any machine a setup names. The import rule
|
||||
stands where it is, because there a path was unnecessary.
|
||||
- **Generation**: 256 bits from the OS CSPRNG, base64url. A machine
|
||||
credential, never typed twice, so at this entropy no stretching is needed.
|
||||
- **Enrollment**: printed once as a terminal QR code encoding
|
||||
`aiapp://enroll?host=…&port=…&token=…`. The CA is embedded in the APK, so
|
||||
the QR carries no trust material — photographing the terminal leaks only
|
||||
the token, never a way to weaken pinning. The app registers an intent
|
||||
filter for the scheme, and the Settings screen also scans in-app via
|
||||
`zxing-android-embedded`, because not every phone's stock camera
|
||||
redirects a scanned URI to an app reliably.
|
||||
- **Storage**: the server keeps only the SHA-256 in `config.ron`; a plain
|
||||
hash is enough for high-entropy random input. No "show token again" —
|
||||
lost means rotate. The phone seals it with an Android Keystore AES-GCM
|
||||
key (`ServerConfig.kt`; Jetpack's EncryptedSharedPreferences is deprecated
|
||||
with no drop-in successor and Google's guidance is now "use Keystore
|
||||
directly").
|
||||
- **Transport**: `Authorization: Bearer` on every request including the SSE
|
||||
GET, never a query parameter, since URLs leak into logs. The tracing layer
|
||||
must not log the header — covered by a test, so a logging change cannot
|
||||
silently start leaking it.
|
||||
- **Verification**: one middleware wrapping the entire router, never
|
||||
per-route, so a new route cannot forget auth. Zero unauthenticated
|
||||
endpoints, `/health` included. Hash-then-constant-time-compare
|
||||
(`subtle`); failures logged with peer address plus a small fixed delay —
|
||||
not against brute force, but so scanners show up in the log.
|
||||
- **Rotation (the path out)**: `--rotate-token` regenerates, invalidates
|
||||
the old hash, reprints the QR. Config stores a *list* of `{name, hash}`,
|
||||
so per-device revocation is a config entry later, not a migration.
|
||||
- **Why not mTLS**: stronger in theory, but given pinning the delta is only
|
||||
"someone reads the token off a device already inside the trust boundary",
|
||||
and it costs Android client-cert provisioning and a worse new-phone
|
||||
story. Revisit if this outgrows single-user-on-LAN.
|
||||
- **Off-network access: plain WireGuard** (2026-08-24, no third party). The
|
||||
backend binds `wg0` only; the phone runs the official WireGuard app,
|
||||
enrolled by scanning its config as a terminal QR. The only internet-visible
|
||||
thing is one forwarded UDP port silent to unauthenticated packets, so the
|
||||
pre-auth surface is reachable only from enrolled peers and the token becomes
|
||||
defence in depth rather than the sole gate. Addressing stays single-path:
|
||||
the phone reaches the backend at its WireGuard address from everywhere.
|
||||
- Accepted operationally: the endpoint is a DDNS name, since the home IP is
|
||||
not guaranteed static. The WireGuard app resolves it when the tunnel comes
|
||||
up and does not re-resolve, so a rare IP change is fixed by toggling the
|
||||
tunnel once DDNS catches up. The symptom is obvious and lossless — the SSE
|
||||
cursor design replays whatever was missed.
|
||||
- Rejected: **Tailscale** and **Headscale**, which add a coordination
|
||||
service this setup does not need at two or three devices; **forwarding
|
||||
the HTTPS port directly**, which puts every internet scanner one pre-auth
|
||||
bug away from RCE on a machine holding SSH keys.
|
||||
- The server refuses to start without TLS, so the token cannot travel
|
||||
unencrypted by misconfiguration, and binding fails closed — refusing to
|
||||
start if `wg0` is absent rather than falling back to 0.0.0.0.
|
||||
`--bind <ip>` is an *explicit, logged* override for development, a
|
||||
deliberate flag and never a fallback.
|
||||
|
||||
## App (`app/`)
|
||||
|
||||
Kotlin + Compose Multiplatform, single `:androidApp` module, same versions as
|
||||
dev-updater (Kotlin 2.4.x, CMP 1.11.x, JDK 21).
|
||||
|
||||
1. **Session list** — kind icon, title, setup, model, status, last activity.
|
||||
Sessions awaiting an answer sort to the top: the "your turn" inbox.
|
||||
2. **Import** — Claude Code sessions the machine already has, selected in
|
||||
batches (hold to enter, tap to add), with Delete and Import along the
|
||||
bottom. Submitting clears the selection immediately and marks every chosen
|
||||
row, so the bar goes away and the affected set is what says the work is
|
||||
happening. Rows are taken out as each one lands rather than all at the
|
||||
end: a finished row still sitting there looks exactly like one that has
|
||||
not been imported, and tapping it starts a second CLI on the same
|
||||
transcript. That makes rows below slide up under the reader's finger, so a
|
||||
row that has just moved ignores taps for `SETTLE_MS`.
|
||||
3. **Models** and **Setups** — browsing and downloading GGUFs; adding,
|
||||
renaming, re-probing and removing machines.
|
||||
4. **Session screen** — the core:
|
||||
- The transcript rendered from the event stream: markdown, inline images,
|
||||
tool cards, question cards.
|
||||
- **Anything that is a note *about* the conversation rather than a turn in
|
||||
it is closed by default** — a tool call, a peer message, a memory note.
|
||||
Open-ness is the screen's, never the card's: a card that remembered for
|
||||
itself forgets the moment the lazy list stops composing it, so a note
|
||||
opened and scrolled past would shut behind the reader.
|
||||
- **An answered question keeps its options and marks the one taken**, in
|
||||
the same purple that says "picked" while it is open — it does not
|
||||
collapse into a line repeating the answer. The options are what the
|
||||
question *was*, and "Deny" alone does not say Allow was the alternative.
|
||||
One rule in two places (`AskedQuestion` and `PermissionAsk`). An answer
|
||||
typed into **Other** matches no option, so that one is still written out.
|
||||
- **Expanding a row keeps still the end nearest the tap**: touch a row's
|
||||
upper half and its top edge holds, so it opens downwards; touch its
|
||||
lower half and the bottom edge holds, as the list does by default. Which
|
||||
half, rather than which control, so everything that opens behaves alike
|
||||
whether or not it has a control at each end. The transcript is laid out
|
||||
from the bottom, so a bottom edge is anchored for free and the top one
|
||||
has to be arranged: `Modifier.holdTopEdge` asks the list to shift during
|
||||
the *layout* phase, before anything is drawn. From an effect instead,
|
||||
the wrong position is drawn once first, which reads as a flick.
|
||||
- **The full-screen image lives on the screen, not in the row that drew
|
||||
the thumbnail** (`SessionImageViewer`). A `Read` whose result is an image
|
||||
is a row of one call until the next call arrives and makes it a group — a
|
||||
different composable in a different part of the tree, so the old subtree
|
||||
and its open dialog go. Somebody looking at a screenshot was thrown back
|
||||
to the transcript because the session made another tool call.
|
||||
- **All transcript text is selectable, from one `SelectionContainer`
|
||||
around the whole list.** Not per row: a transcript is one body of text,
|
||||
so a selection has to run from a reply into the tool output under it —
|
||||
and a container per row leaves whatever was drawn without one silently
|
||||
unselectable. An inline code chip is drawn *behind* the text rather than
|
||||
as the renderer's span background, because a span background is part of
|
||||
the text's own drawing and hid the selection under it.
|
||||
- Input bar: text, attach, send — **always enabled**; mid-run sends become
|
||||
steering messages. A queued message can be tapped to take it back.
|
||||
- The composer's process button (interrupt / stop / start) as above.
|
||||
|
||||
### Markdown
|
||||
|
||||
**A reply is drawn as pieces of one parse, never as re-parsed substrings.**
|
||||
A `Piece` addresses a top-level block of the message's tree, or one item of a
|
||||
top-level list, and every piece is drawn from the same cached parse. That is
|
||||
what bounds a lazy-list item without parsing a message more than once, and it
|
||||
is why a forty-item list of sources is forty units rather than one. Links are
|
||||
spans with one tap detector per text, not a layout node per link — the cost
|
||||
that made a list of sources bumpy.
|
||||
|
||||
**A table wraps its cells and never cuts one off.** The renderer's defaults
|
||||
draw every cell at one line with an ellipsis, which on a phone loses most of
|
||||
a table — and an elided cell looks exactly like a short one. `LinkedTableRow`
|
||||
gives a cell as many lines as it needs, aligned to the top of the row so a
|
||||
two-line cell does not re-centre its neighbours. A column narrows to 136dp
|
||||
and no further, past which the whole table scrolls sideways; 136 because it
|
||||
is the widest floor that still fits three columns across a phone. Exercise it
|
||||
with the echo driver's `/table N`, which writes long cells on purpose — a
|
||||
fixture of tidy one-word values renders fine either way.
|
||||
|
||||
### The transcript cache
|
||||
|
||||
The backend's transcript is the source of truth, and the app keeps a copy of
|
||||
what it has already been sent — see **`TRANSCRIPT_CACHE.md`** (2026-09-04),
|
||||
because reopening a session over the tunnel was re-downloading a conversation
|
||||
the phone had just read. It is the server's own event lines, per session,
|
||||
under `cacheDir`; it is checked against the server before a stream is resumed
|
||||
from it, thrown away rather than patched when that check fails, and **never
|
||||
load-bearing** — every path that reads it has a network path beside it giving
|
||||
the same answer. What the app does not keep is anything *derived*: the folded
|
||||
rows are rebuilt from events every time.
|
||||
|
||||
### Notifications: two places, never both (2026-08-30)
|
||||
|
||||
`GET /notifications` is one SSE stream of attention-wanting moments, and the
|
||||
app decides where each one is said. Three outcomes, in one place
|
||||
(`NotificationService.show`):
|
||||
|
||||
- **Nothing at all** if the session is the one on screen. The transcript in
|
||||
front of the reader is already saying it.
|
||||
- **A banner over the app** if the app is up — `SessionAlerts`, queued, one
|
||||
per session replacing that session's own, dismissable by a push in either
|
||||
direction and otherwise retiring itself when the bar across its foot runs
|
||||
out.
|
||||
- **A row in Android's drawer** otherwise, which is what the foreground
|
||||
service exists for.
|
||||
|
||||
Never two of them for one moment. A drawer that fills up behind an app that
|
||||
showed you each one is a drawer nobody reads. Which of the three applies is
|
||||
answered without a flag anybody has to keep level: the session on screen is
|
||||
registered by the one composable that draws one, and "the app is up" *is* the
|
||||
banner queue being collected, since it collects only while it is on screen.
|
||||
|
||||
**What counts as finished** is decided in `notification_for`, and since
|
||||
2026-08-31 it takes the number of messages the session has been given and not
|
||||
started reading. With one waiting, a turn ending is not the work ending: a
|
||||
message written into the tail of a turn is read the moment that turn's
|
||||
`result` lands, so the session goes idle and immediately runs again — and the
|
||||
phone that sent it was told its work had finished seconds before any of it
|
||||
was done. The count is kept in `pump`, the one place that sees every event in
|
||||
transcript order. It does not suppress *awaiting input*: a question is worth
|
||||
saying whatever is queued behind it.
|
||||
|
||||
Rejected: giving the app its own connection to `/notifications` while it is
|
||||
in front. That is a second stream per device saying the same thing, and it
|
||||
puts the "which of these two shows it" decision in two processes' worth of
|
||||
code instead of one function.
|
||||
|
||||
### Deferred polish
|
||||
|
||||
Noticed and deliberately not fixed, so they are not re-found from scratch.
|
||||
|
||||
- **The session screen's header is lopsided.** The row is
|
||||
`padding(horizontal = 8.dp)`, so the status on the right sits exactly 8dp
|
||||
from the edge while "Back" on the left is a `TextButton` whose touch target
|
||||
is wider than its text. It is the "align the mark, not the box" case:
|
||||
either align the button's content or size the button to what it draws,
|
||||
rather than nudging with a hardcoded offset.
|
||||
|
||||
## Status
|
||||
|
||||
Phases 1–3 (the skeleton pipe, the full Claude driver, the usage screen) done
|
||||
2026-08-24. Phase 4 (llama.cpp: model browsing, downloads, and `llama-server`
|
||||
through its OpenAI-compatible endpoint) and phase 5 (ssh) done 2026-08-28,
|
||||
except for the remote `llama-server` and its port forward, which landed
|
||||
2026-09-04. The file explorer and the transcript cache followed in September. What is
|
||||
left is real-phone/WireGuard bring-up, which is operational rather than code.
|
||||
|
||||
Each phase ended runnable and verified against the real thing. The backend
|
||||
gets tests where logic is pure — event normalization, transcript cursors,
|
||||
config persistence, the syntax scanner; the app is UI over the API and is
|
||||
verified by running it, matching dev-updater's posture.
|
||||
|
||||
## Open questions and risks
|
||||
|
||||
- **The Claude stream-json control protocol** is the least-documented
|
||||
dependency and is version-coupled to the installed CLI. What works is
|
||||
pinned in `session/claude.rs`'s module doc against the version it was
|
||||
measured on.
|
||||
- **The usage endpoint is undocumented** and has changed rate-limit
|
||||
behaviour before; treat as best-effort.
|
||||
- **Compaction for llama sessions is not built.** `LlamaDriver::compact`
|
||||
refuses. The design when it is built: every response reports prompt and
|
||||
completion token counts, so track them against `n_ctx` (from `/props`), and
|
||||
at ~75% summarize all but the last few turns and replace them, keeping the
|
||||
full pre-compaction transcript on disk so the phone's view never loses
|
||||
history. llama-server's own `--context-shift` is rejected as the strategy:
|
||||
it truncates old KV cache entries, which is silent forgetting with no
|
||||
summary, and it corrupts the harness's view of what the model knows. Fine
|
||||
as a server-side safety net; not memory management.
|
||||
- **Remote llama-server** needs its port forwarded (`ssh -L`) and is not
|
||||
built; such a session is refused rather than misdirected.
|
||||
- **Claude sessions over ssh need the remote machine logged in to Claude.**
|
||||
Usage reporting reads each machine's own credentials, so this is visible
|
||||
rather than silent.
|
||||
|
||||
## References
|
||||
|
||||
- llama-server API (`/health`, `/props`, OpenAI-compatible endpoints):
|
||||
https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
|
||||
- Usage endpoint (`GET https://api.anthropic.com/api/oauth/usage`, bearer
|
||||
token from `~/.claude/.credentials.json`, headers
|
||||
`anthropic-beta: oauth-2025-04-20` + `User-Agent: claude-code/<version>`,
|
||||
≥180 s polling; a wrong User-Agent lands in an aggressive 429 bucket):
|
||||
https://github.com/anthropics/claude-code/issues/31637
|
||||
- The sibling project this repo's conventions mirror: `../dev-updater`
|
||||
(README.md + AGENTS.md — server/registry/routes layout, cert scheme,
|
||||
testing posture).
|
||||
+3806
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,496 @@
|
||||
# How iris should render an unbounded number of images
|
||||
|
||||
## Status (2026-09-04)
|
||||
|
||||
**Implemented**, on the `rustify` branch of `ai-app-2`, in `iris/core` and
|
||||
`iris/src/default/render.rs`. See "Implemented, 2026-09-04" at the bottom for
|
||||
what landed, what differs from the proposal below and why, and what was
|
||||
verified versus merely reasoned about. The short version: the binding array
|
||||
is gone, `request_device` asks for no features and no binding-array limits,
|
||||
and that is now proven on the emulator's software Vulkan
|
||||
(`rigs/gpu-probe`), not just read from the code. `RUST.md`'s blocking item
|
||||
is resolved.
|
||||
|
||||
Iris (the person) asked whether iris's (the library's) approach to
|
||||
"draw however many images happen to be on screen" — relevant here because a
|
||||
transcript can hold an unbounded number of attached screenshots — actually
|
||||
works on mobile, her recollection being that it does not. Checked rather
|
||||
than assumed, on 2026-09-04, on the `rustify` branch of `ai-app-2`. This
|
||||
file is that investigation and the resulting recommendation, written for a
|
||||
second agent to review before anything in iris's render core changes — no
|
||||
code has been written against this yet.
|
||||
|
||||
## The problem
|
||||
|
||||
Every texture iris ever creates — every `Image` widget
|
||||
(`iris/src/widget/image.rs`) and every glyph atlas page — gets a permanent
|
||||
slot in one array via `Textures::add` (`iris/core/src/primitive/texture.rs:65`).
|
||||
Both of iris's texture-sampling primitives (`TEXTURE` and `GLYPH`) read that
|
||||
array by index: `core/src/render/shader.wgsl:56` declares
|
||||
`var views: binding_array<texture_2d<f32>>`, sized by
|
||||
`UiLimits::default()` (`core/src/render/mod.rs:347`) at **100,000 textures,
|
||||
1,000 samplers**. Getting a device to accept that layout needs three wgpu
|
||||
features — `TEXTURE_BINDING_ARRAY`,
|
||||
`SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING`,
|
||||
`PARTIALLY_BOUND_BINDING_ARRAY` — which correspond to Vulkan's
|
||||
`VK_EXT_descriptor_indexing` ("bindless"), promoted to Vulkan core at 1.2.
|
||||
|
||||
A transcript with an unbounded number of image attachments is exactly the
|
||||
case that grows this array without bound: each attachment becomes its own
|
||||
`Image` widget, which takes its own permanent array slot until dropped.
|
||||
|
||||
## What was measured
|
||||
|
||||
**A new rig, `rigs/gpu-probe`**, asks a device for exactly iris's features
|
||||
and limits with no window and no APK — a plain executable pushed with
|
||||
`adb push` and run from `/data/local/tmp`. It has two parts:
|
||||
`wgpu::Adapter::request_device` with iris's exact `Features`/`Limits`
|
||||
(`src/main.rs`), and a raw Vulkan query bypassing wgpu entirely via `ash`
|
||||
(`src/vk.rs`), to tell "the driver doesn't have it" apart from "wgpu didn't
|
||||
detect it."
|
||||
|
||||
- **On this VM's own GPU** (Vulkan via Venus onto an RX 7900 XT):
|
||||
`IRIS DEVICE: ok`. Not the case that matters — nobody's phone is a
|
||||
discrete desktop GPU — but it is why the design was never checked before
|
||||
now: it always worked in the one place it was tried.
|
||||
- **On the Android emulator's guest Vulkan**, both ICDs it ships
|
||||
(`vk_swiftshader_icd.json` and, cold-booted, `lvp_icd.json`/lavapipe):
|
||||
`request_device` **fails** —
|
||||
`Unsupported features were requested: TEXTURE_BINDING_ARRAY |
|
||||
SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING |
|
||||
PARTIALLY_BOUND_BINDING_ARRAY`. The raw `ash` query on lavapipe shows the
|
||||
driver itself reporting all seven descriptor-indexing sub-features as
|
||||
`true` at device API version 1.3 — so wgpu-hal's own feature detection is
|
||||
being more conservative than the driver here, for a reason not chased
|
||||
further (a likely instance-version negotiation gap, since the extension
|
||||
only promoted to core at 1.2). That part is a wgpu-hal/emulator question,
|
||||
not the finding that matters, and is **not** why this design is rejected.
|
||||
|
||||
**The finding that matters is about real phones, sourced rather than
|
||||
recalled:**
|
||||
|
||||
- The **Android Vulkan Profile 2025** — Google and Khronos's current
|
||||
baseline, covering **80.1% of active Vulkan-capable Android devices** as
|
||||
of October 2025
|
||||
([developer.android.com/ndk/guides/graphics/android-vulkan-profile](https://developer.android.com/ndk/guides/graphics/android-vulkan-profile)) —
|
||||
does **not** require `VK_EXT_descriptor_indexing` or any descriptor-
|
||||
indexing feature. It requires `shaderSampledImageArrayDynamicIndexing`
|
||||
(indexing by a value uniform across the invocation — Vulkan 1.0 baseline,
|
||||
unrelated to bindless) and stops there; true of the 2021 and 2022
|
||||
profiles as well.
|
||||
- Arm's own developer documentation states **"`VK_EXT_descriptor_indexing`
|
||||
is supported on all Valhall and 5th Gen GPUs"**
|
||||
([developer.arm.com/mobile-graphics-and-gaming/vulkan-api-best-practices-on-arm-gpus](https://developer.arm.com/mobile-graphics-and-gaming/vulkan-api-best-practices-on-arm-gpus)) —
|
||||
Mali generations from roughly 2019 (Mali-G77) onward, with no claim made
|
||||
for Bifrost, Midgard or Utgard, which are still common in budget and
|
||||
older Android phones that are still in daily use.
|
||||
- A search engine's summarized claim of "1% support on Android" for this
|
||||
extension was checked against its cited source (an Arm blog post from
|
||||
2021) and **was not actually there** — that number does not appear in
|
||||
any primary source found and should not be repeated. The baseline-
|
||||
profile finding above is the one with an attributable source; use it
|
||||
instead.
|
||||
|
||||
So this is not a software-renderer artifact. A real, currently-shipping
|
||||
share of the Android fleet lacks the feature iris's texture pipeline asks
|
||||
for unconditionally, and neither the emulator's failure nor the current
|
||||
official hardware baseline gives any reason to expect that to change soon.
|
||||
|
||||
## What growth already costs today, before any redesign
|
||||
|
||||
Checked directly in `core/src/render/mod.rs` and `core/src/render/texture.rs`,
|
||||
because "does this redesign make things worse" needs the current baseline
|
||||
first:
|
||||
|
||||
- The `RenderPipeline` (`UiRenderNode::new`) is created **once** and never
|
||||
rebuilt for any reason related to texture count — its bind group
|
||||
*layouts* declare fixed slot counts (`limits.max_textures`,
|
||||
`limits.max_samplers`) up front and that never changes at runtime. Growth
|
||||
was never at risk of recreating the pipeline, in the current design or
|
||||
any redesign discussed below.
|
||||
- What **does** get rebuilt: `UiRenderNode::update` calls
|
||||
`self.textures.update(&mut ui.textures)`, and if that reports any change,
|
||||
rebuilds `self.rsc_group` — one `BindGroup` whose entries are
|
||||
`BindingResource::TextureViewArray(&tex_manager.views())`, collected
|
||||
fresh over **every currently-live texture**, plus the sampler array and
|
||||
the mask buffer. This happens on every texture `Push`, `Set`, or `Free`
|
||||
— an image added anywhere in the whole app rebuilds one shared structure
|
||||
referencing every other image too.
|
||||
- The one path already excluded from this, on purpose, is a `Patch` —
|
||||
writing into an existing texture's pixels without changing which
|
||||
textures exist. The code says why directly
|
||||
(`core/src/render/texture.rs`, in `GpuTextures::update`): *"A patch
|
||||
changes texture contents, not the binding array, so it must not report
|
||||
`changed` — rebuilding the bind group per glyph is the cost this exists
|
||||
to avoid."* This is exactly the mechanism I1 built for the glyph atlas:
|
||||
growing an existing atlas page costs a `write_texture` into a sub-rect,
|
||||
nothing else.
|
||||
|
||||
So today, growth that stays inside an existing texture (glyphs added to an
|
||||
atlas page) is already free. Growth that adds a *new* texture — a new atlas
|
||||
page, or any standalone image — already rebuilds the one shared array
|
||||
regardless of how the array is populated, before any change discussed
|
||||
below. That existing cost is O(live texture count) in CPU work to collect
|
||||
the view list and in however expensive the driver finds a
|
||||
descriptor-set-sized-for-N-descriptors to be.
|
||||
|
||||
## Prior art, checked rather than assumed
|
||||
|
||||
Two independent projects were checked to see whether "atlas for images"
|
||||
is actually how this is normally done, rather than a guess:
|
||||
|
||||
- **egui_wgpu** (`crates/egui-wgpu/src/renderer.rs` in emilk/egui), the
|
||||
closest prior art to iris — an immediate-mode wgpu-backed UI library that
|
||||
ships on Android. It keeps a `HashMap<TextureId, Texture>` and gives
|
||||
**each texture its own ordinary `BindGroup`** — one texture, one sampler,
|
||||
no array, no descriptor indexing of any kind. Draw calls are batched by
|
||||
texture id and the bind group is switched between batches within the
|
||||
render pass.
|
||||
- **Vello** — the renderer Masonry (E1/E2's Linebender stack) draws
|
||||
through — hit the identical problem and wrote down why in their own
|
||||
roadmap document
|
||||
([github.com/linebender/vello/blob/main/doc/roadmap_2023.md](https://github.com/linebender/vello/blob/main/doc/roadmap_2023.md)):
|
||||
*"The number of images that may appear in a scene is not bounded, which
|
||||
is not a good fit for the basic descriptor binding model... Until then,
|
||||
we'll do a workaround of having a single atlas image containing all the
|
||||
images in the scene."* Their reason is broader than Android — WebGPU 1.0
|
||||
has no descriptor indexing at all — but it reaches the same conclusion
|
||||
for the same shape of problem: atlas, not a bigger bindless array.
|
||||
|
||||
**This is also a live hazard, not a solved one.** Vello's own changelog
|
||||
(Sparse Strips v0.2.0) lists a fix titled *"WebGL image-atlas allocation
|
||||
and growth on Mali-G52 GPUs, avoiding application-not-responding errors"*
|
||||
— an actual ANR, from atlas growth, on an actual mid-range Android GPU,
|
||||
in the renderer Masonry is built on. The same release added
|
||||
`AtlasSpaceDiagnostics`/`AtlasLayerDiagnostics` (per-layer free-space,
|
||||
utilization, fragmentation) because growth needed instrumenting in
|
||||
production, not because it turned out to be free.
|
||||
|
||||
## Recommendation (not yet implemented)
|
||||
|
||||
1. **Small, plentiful textures** — glyphs (already done, I1), thumbnails,
|
||||
downscaled attachment previews, icons — go through a shared atlas, the
|
||||
same technique as `core/src/render/atlas.rs` generalized beyond glyphs.
|
||||
Adding one to an existing page is a `Patch`, already free per the
|
||||
section above.
|
||||
2. **Large or one-off images** — a photo attachment opened at full
|
||||
resolution, anything that would fragment a shared page — get their
|
||||
**own ordinary, non-array bind group**, the egui_wgpu way. Creating one
|
||||
is O(1): it references only itself, and does not touch any other
|
||||
texture's binding, unlike today's shared array where every push
|
||||
rebuilds a structure listing everything.
|
||||
3. **Opening a new atlas page** is the one case that still resembles
|
||||
today's rebuild — infrequent (bounded by how many *pages* are needed,
|
||||
not by how many images have ever been attached) but not free, and
|
||||
Vello's Mali-G52 fix says this specifically deserves care: it should
|
||||
never be allowed to block a frame, and it is worth having the
|
||||
equivalent of Vello's atlas diagnostics before trusting it under load.
|
||||
4. **Net effect**: dropping `TEXTURE_BINDING_ARRAY`,
|
||||
`SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING`, and
|
||||
`PARTIALLY_BOUND_BINDING_ARRAY` from iris's device request entirely.
|
||||
Every path above is plain Vulkan 1.0 / GLES-level texture sampling.
|
||||
This is also what fixes the emulator failure measured above, regardless
|
||||
of the unresolved wgpu-hal question: a device that never asks for the
|
||||
feature cannot be refused for lacking it.
|
||||
|
||||
## What this touches, and what is still open
|
||||
|
||||
Implementing this reworks iris's rendering core: the shader's binding
|
||||
group layout (`shader.wgsl`), `Textures` and `GpuTextures`
|
||||
(`core/src/primitive/texture.rs`, `core/src/render/texture.rs`), both
|
||||
texture-sampling primitives, and `core/src/ui/painter.rs`'s draw-call
|
||||
batching (today one draw call can reference any texture by index; the
|
||||
per-texture-bind-group path needs draws grouped by which bind group they
|
||||
use). Nothing has been started.
|
||||
|
||||
Open questions a reviewer should weigh in on:
|
||||
|
||||
- **The size threshold** between "goes in an atlas page" and "gets its own
|
||||
bind group." Too low and ordinary attachment thumbnails end up as
|
||||
one-off bind groups, losing the batching benefit the atlas exists for;
|
||||
too high and a page fragments on a handful of medium images.
|
||||
- **Eviction policy** for atlas pages once the working set does not fit —
|
||||
today's `GlyphAtlas` never evicts, because a font's glyph set is small
|
||||
and bounded; images are not. An LRU at the page level, or at the
|
||||
individual-image level within a page, has not been designed.
|
||||
- **Whether iris should keep any binding array at all**, even a small
|
||||
fixed one (say, capped at a few dozen slots) for atlas pages themselves,
|
||||
or whether every atlas page should also be its own ordinary bind group
|
||||
like standalone images — the array's only remaining justification would
|
||||
be avoiding a bind-group-per-draw-call switch cost that has not been
|
||||
measured on this project's actual target hardware.
|
||||
- **How this interacts with I2/E2's virtualised list** (I3): a
|
||||
bottom-anchored transcript composes only visible rows, so the live
|
||||
texture set should already be bounded by what is on screen rather than
|
||||
by the whole conversation — worth confirming that invariant holds before
|
||||
relying on it to keep atlas/bind-group churn small.
|
||||
|
||||
## Review, 2026-09-04
|
||||
|
||||
A second pass over the file above against the code, done before anything
|
||||
is implemented. Iris's worry going in: a bind group per texture means a
|
||||
draw call per image, and she wants this as efficient as it can be.
|
||||
|
||||
### What checked out
|
||||
|
||||
Every code reference above is accurate as of this commit: the 100,000 /
|
||||
1,000 limits, the one-time pipeline, the `rsc_group` rebuild on every
|
||||
`Push`/`Set`/`Free`, and the `Patch` exclusion. The device request that
|
||||
asks for the three features is `iris/src/default/render.rs:96`, which the
|
||||
text above does not name. egui-wgpu and Vello are described correctly.
|
||||
|
||||
### The emulator refusal is a wgpu-hal gap, now located
|
||||
|
||||
The file guessed "a likely instance-version negotiation gap." It is
|
||||
narrower than that and it is in wgpu-hal, not the emulator. wgpu-hal
|
||||
28.0.0 (`src/vulkan/adapter.rs:1618`) only queries
|
||||
`PhysicalDeviceDescriptorIndexingFeaturesEXT` **when the device advertises
|
||||
the `VK_EXT_descriptor_indexing` extension string**. A Vulkan 1.2+ driver
|
||||
that has descriptor indexing as core need not list the extension, and
|
||||
lavapipe at 1.3 evidently does not, so wgpu never asks and reports the
|
||||
features absent, which is why `ash` sees seven `true`s and wgpu sees none.
|
||||
The properties query beside it (line 1486) correctly accepts
|
||||
`device_api_version >= 1.2 || extension`; the features query does not.
|
||||
wgpu-hal 30.0.1 in the local registry has the same asymmetry (lines
|
||||
1872 and 2036). Worth an upstream issue, but not a reason to keep the
|
||||
design: on real phones the gate that matters is stricter still.
|
||||
|
||||
**wgpu's `TEXTURE_BINDING_ARRAY` needs six sub-features, not one**
|
||||
(`adapter.rs:160-177`): non-uniform indexing *and* update-after-bind for
|
||||
sampled images, storage images and storage buffers, all together, because
|
||||
wgpu marks every array-bearing descriptor set update-after-bind. So Arm's
|
||||
"the extension is supported on Valhall" is necessary but not sufficient;
|
||||
a driver with sampled-image indexing and without storage-buffer
|
||||
update-after-bind is refused too. That widens the excluded set beyond
|
||||
what the Arm quote suggests and strengthens the conclusion.
|
||||
|
||||
### A live bug in the current code, found on the way
|
||||
|
||||
`GpuTextures::update` (`core/src/render/texture.rs:33`) implements
|
||||
"a patch must not report changed" as `changed = false`, unconditionally,
|
||||
which also **cancels a `Push` earlier in the same batch**. That ordering is
|
||||
exactly what opening a new atlas page produces: `GlyphAtlas::allocate`
|
||||
pushes the page and `insert` patches it in the same frame, so the bind
|
||||
group is not rebuilt and the new page's view is not bound until some
|
||||
unrelated texture change happens to rebuild it. It is hidden today only
|
||||
because the masks path also sets `changed`. The fix is one line
|
||||
(`changed |= !matches!(update, Patch)` in spirit); it should go in with
|
||||
the redesign since that code is being replaced, and it is recorded here
|
||||
so it is not rediscovered.
|
||||
|
||||
### In-layer draw order is already undefined
|
||||
|
||||
Relevant to any batching redesign: `Primitives::apply_free`
|
||||
(`core/src/render/primitive.rs:147`) uses `swap_remove`, so the instance
|
||||
order within a layer is permuted whenever anything is freed. Overlap order
|
||||
inside one layer is therefore not something the renderer promises today;
|
||||
ordering is done with layers. That means grouping a layer's draws by
|
||||
texture, or drawing a layer's images after its rects and glyphs, loses
|
||||
nothing that currently exists. It should be written down as an invariant
|
||||
when the redesign lands, because the new code will depend on it.
|
||||
|
||||
### On "a draw call per image"
|
||||
|
||||
Two corrections to the worry. First, it is a draw per *distinct texture per
|
||||
layer*, not per image primitive: every glyph quad in a layer shares the
|
||||
atlas and stays one instanced draw, and a thumbnail atlas would do the same
|
||||
for previews. Second, the count is bounded by what is on screen, which I3's
|
||||
virtualised transcript already bounds, and a mobile GPU is not draw-call
|
||||
bound at tens of draws per frame; egui ships exactly this on Android. What
|
||||
does cost is per-frame *bind group creation* and per-frame *sorting*, and
|
||||
the current code already creates a `primitive_group` bind group every time
|
||||
a layer updates (`render/mod.rs:103`), so one more per new image is not a
|
||||
regression in kind.
|
||||
|
||||
### Recommended shape (proposal, for Iris to accept or change)
|
||||
|
||||
Aimed at the fewest moving parts that need no feature beyond Vulkan 1.0:
|
||||
|
||||
1. **Atlas pages become layers of one `texture_2d_array`**, not separate
|
||||
textures. Every page is already `PAGE`x`PAGE` RGBA8, which is the one
|
||||
constraint an array texture imposes. A layer index is an ordinary
|
||||
sampling operand in WGSL and needs no indexing feature, so `GLYPH`
|
||||
(and any future atlased-image primitive) carries a layer instead of a
|
||||
`view_idx` and all of a layer's text stays **one draw**. This answers
|
||||
the open question above about keeping a small binding array: no. Cost
|
||||
of opening a page: recreate the array with one more layer and
|
||||
`copy_texture_to_texture` the old ones, GPU-side, no readback; grow
|
||||
with headroom (double) so it is rare. wgpu's default
|
||||
`max_texture_array_layers` is 256, at 4 MB each, so the cap is memory
|
||||
rather than the API.
|
||||
2. **Every standalone image is its own texture with its own bind group**,
|
||||
and its instances live in a **separate per-layer instance list**, not
|
||||
the main one. Then the main instance buffer never contains an image,
|
||||
there is nothing to sort, no handle remapping beyond what
|
||||
`apply_free` already does, and each image is `draw(0..4, k..k+1)` with
|
||||
its bind group set first. Group 2's layout becomes `{atlas array,
|
||||
one image texture, sampler, masks}`; the main draw binds a 1x1 null
|
||||
image in the image slot, each image draw binds its own. One pipeline,
|
||||
one shader, one layout.
|
||||
3. **No thumbnail atlas in the first version.** With images on their own
|
||||
textures, the threshold and eviction questions above disappear: an
|
||||
image is freed when the row that owns its `TextureHandle` scrolls out.
|
||||
Add an image atlas only if a measured screen shows enough small images
|
||||
to matter, which a transcript rarely does.
|
||||
4. **Drop the three features and the two `max_binding_array_*` limits from
|
||||
`src/default/render.rs`**, and the `UiLimits` counts with them.
|
||||
5. **Sampling is `NonFiltering` today** (`render/mod.rs:290,299`), so a
|
||||
downscaled attachment will alias. Either request a filtering sampler
|
||||
for the image slot or downscale on the CPU before upload; decide when
|
||||
the image widget is touched, not as part of this.
|
||||
|
||||
What this costs against the file's original recommendation: `Textures`
|
||||
needs to know an image from a page (two kinds of handle, or a kind on
|
||||
`TextureHandle`), and `Primitives` gets a second instance list per layer.
|
||||
What it saves: the sort, the size threshold, the eviction policy, and any
|
||||
per-page bind group switch.
|
||||
|
||||
## Implemented, 2026-09-04
|
||||
|
||||
The shape above, built as proposed with one structural addition the proposal
|
||||
didn't need to spell out and one bug it predicted made moot rather than
|
||||
literally fixed. Files: `core/src/primitive/texture.rs` (`Textures`,
|
||||
`TextureHandle`), `core/src/render/texture.rs` (`GpuTextures`),
|
||||
`core/src/render/primitive.rs` (`Primitives`, `GlyphPrimitive`),
|
||||
`core/src/render/atlas.rs`, `core/src/ui/painter.rs`,
|
||||
`core/src/render/mod.rs` (`UiRenderNode`, `UiLimits` removed),
|
||||
`core/src/render/shader.wgsl`, `src/default/render.rs`, and
|
||||
`rigs/gpu-probe/src/main.rs`.
|
||||
|
||||
**1. Atlas pages as array layers.** `GpuTextures` owns one
|
||||
`texture_2d_array` (`array_texture`/`array_view`), grown by doubling
|
||||
(`grow_array`): a new texture is created at twice the layer capacity, the
|
||||
old layers are copied across with `copy_texture_to_texture` (GPU-side, no
|
||||
readback), and every bind group that referenced the old view — the main
|
||||
one and every live standalone image's — is rebuilt, since the view's
|
||||
identity changed. `GlyphPrimitive` carries `layer: u32` instead of
|
||||
`view_idx`/`sampler_idx`; the layer number is assigned synchronously in
|
||||
`Textures::add_page` (a plain counter, `next_page_layer`), not by the
|
||||
renderer, because `GlyphAtlas::insert` needs it in the same call, before
|
||||
any GPU sync happens — the renderer only finds out later, when it
|
||||
processes the queued `Push`.
|
||||
|
||||
**2. Standalone images, one bind group each.** `TextureKind` on
|
||||
`TextureHandle`/`Textures` distinguishes `Image` (a plain bind-group index,
|
||||
`slot`) from `Page { layer }`. `Primitives` gained a second per-layer list
|
||||
— `images: Vec<PrimitiveInstance>`, tagged `IMAGE_BINDING` — separate from
|
||||
`instances` (rects and glyphs), written by `Painter::write_image` rather
|
||||
than through the generic `Primitive` trait, since an image has nowhere in
|
||||
`PrimitiveData` to put a per-instance entry once the bind group already
|
||||
picks the texture. `UiRenderNode::draw` draws a layer's `instance` buffer
|
||||
once as before, then walks `image_instance` one entry at a time, binding
|
||||
that texture's `BindGroup` (`GpuTextures::image_bind_group`) and issuing
|
||||
`draw(0..4, k..k+1)` per image. Group 2's layout is exactly the proposed
|
||||
`{atlas array, one image texture, sampler, masks}`; the main draw binds a
|
||||
1x1 null view in the image slot.
|
||||
|
||||
**The one addition beyond the proposal**: the masks storage buffer lives
|
||||
in every per-image bind group (group 2, binding 3), and `ArrBuf<Mask>`
|
||||
recreates its buffer whenever the mask count changes size
|
||||
(`render/util/mod.rs`'s `ArrBuf::update` now returns whether it resized).
|
||||
A resize invalidates every bind group holding the old buffer, not just the
|
||||
main one, so `GpuTextures::update` takes a `masks_resized: bool` and calls
|
||||
`rebuild_image_bind_groups` when it's set, alongside the same rebuild the
|
||||
array-growth path already needed. This wasn't a design question the
|
||||
proposal had to answer (it treated bind-group construction as a given),
|
||||
but it's exactly the shape of trap layer growth already had, so it uses
|
||||
the same fix.
|
||||
|
||||
**3. No thumbnail atlas.** Not built, as proposed.
|
||||
|
||||
**4. Removed**: `TEXTURE_BINDING_ARRAY`, `PARTIALLY_BOUND_BINDING_ARRAY`,
|
||||
`SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING` from
|
||||
`src/default/render.rs`'s `request_device`, and `UiLimits` (the type
|
||||
itself, not just its binding-array methods — once its two fields were
|
||||
gone there was nothing left in it, and `UiRenderNode::new` no longer takes
|
||||
a limits parameter). `binding_array` no longer appears anywhere in
|
||||
`shader.wgsl`.
|
||||
|
||||
**5. Sampling** is still `NonFiltering`, unchanged, per the proposal's own
|
||||
note that this is a separate decision for whenever the image widget itself
|
||||
is touched.
|
||||
|
||||
**The `changed = false` bug is structurally gone, not patched.** The old
|
||||
`GpuTextures::update` held one `changed: bool` that a `Patch` reset
|
||||
unconditionally, which could erase an earlier `Push` in the same batch (a
|
||||
new atlas page's `Push` immediately followed by `GlyphAtlas::insert`'s
|
||||
`Patch`, both queued before the renderer ever runs). The new `update`
|
||||
computes the rebuild signal by OR-ing each event's own answer
|
||||
(`rebuild_main |= self.push(...)`), and `Patch`'s arm simply never
|
||||
contributes to it — there is no shared mutable flag left for a `Patch` to
|
||||
stomp on. Documented at the call site
|
||||
(`core/src/render/texture.rs`, `GpuTextures::update`'s doc comment and the
|
||||
`Patch` match arm's comment) rather than fixed as a one-line diff, since
|
||||
the mechanism that could go wrong no longer exists.
|
||||
|
||||
**In-layer draw order is an explicit invariant now, not just a fact about
|
||||
`swap_remove`.** `UiRenderNode::draw` draws every layer's images after its
|
||||
rects and glyphs, and `Primitives::apply_free`'s doc comment states
|
||||
directly that both of a layer's lists (`instances` and `images`) free with
|
||||
`swap_remove` and that nothing may assume adjacency survives a free —
|
||||
recorded there because `apply_free` is the one place a change to either
|
||||
list's ordering would have to be reconciled.
|
||||
|
||||
**Verified:**
|
||||
|
||||
- `cargo fmt --all -- --check`, `cargo build --workspace --all-targets`,
|
||||
`cargo clippy --all-targets`, `cargo test --workspace` all clean in
|
||||
`iris/`, on the pinned `nightly-2026-09-03` toolchain. 14 tests pass
|
||||
(unchanged from I1; nothing here is pure-logic enough to add a unit
|
||||
test to — it's all GPU resource wiring).
|
||||
- `iris/run-headless.sh minimal --shot /tmp/minimal.png` and
|
||||
`iris/run-headless.sh tabs --shot /tmp/tabs.png`: both render correctly
|
||||
on this VM's GPU (Venus) — `tabs`'s glyph-atlas text renders in every
|
||||
panel, confirming `GlyphPrimitive.layer` addresses the array correctly.
|
||||
- The standalone-image path specifically: a throwaway example (not
|
||||
committed) with an `image(...)` widget as part of the root, run the same
|
||||
way, rendered the image next to glyph-atlas text in one frame —
|
||||
confirming a live `BindGroup` built by `GpuTextures::create_image` and
|
||||
bound per-`draw()` call actually samples the right texture. `tabs`'s own
|
||||
"image span" tab exercises the same widget but needs a click to reach,
|
||||
which the headless compositor can't deliver (no seat devices, per I1's
|
||||
own note on this file) — the throwaway example is what stood in for it.
|
||||
- **Exercised, 2026-09-04: `grow_array` under real load, on `tabs`.**
|
||||
Rather than building a purpose-made glyph flood, `PAGE`
|
||||
(`core/src/render/atlas.rs`) was temporarily dropped from 1024 to 64 —
|
||||
small enough that `tabs`'s ordinary mix of sizes and families (nothing
|
||||
exotic: a handful of `Text` widgets at a few sizes, one at
|
||||
`Family::Monospace`) already exceeds one page's worth of distinct
|
||||
glyphs. A one-line `eprintln!` in `grow_array` confirmed two real grows
|
||||
in a single run (`GROW_ARRAY: 1 -> 2` then `GROW_ARRAY: 2 -> 4`, i.e.
|
||||
glyphs landed on at least a third layer), and
|
||||
`iris/run-headless.sh tabs --shot` showed every tab's text rendering
|
||||
correctly with no corruption or missing glyphs — confirming the
|
||||
`copy_texture_to_texture` grow-and-relocate path and cross-layer
|
||||
sampling (`GlyphPrimitive.layer` addressing a layer beyond the first)
|
||||
both work. Command:
|
||||
`sed -i 's/PAGE: u32 = 1024/PAGE: u32 = 64/' core/src/render/atlas.rs`,
|
||||
rebuild, `./run-headless.sh tabs --shot /tmp/x.png`, then
|
||||
`git checkout -- core/src/render/atlas.rs` to revert — this is a
|
||||
throwaway diagnostic value, never a committed change, since a real
|
||||
1024px page holding only a handful of glyphs at a time would be mostly
|
||||
wasted space in normal use. Confirmed the revert left `tabs` and
|
||||
`minimal` byte-identical to the pre-check screenshots afterward.
|
||||
- **The decisive check**, `rigs/gpu-probe` rewritten to request iris's new
|
||||
(empty) feature/limit set and run on this checkout's own emulator
|
||||
(`ai-app-2`, via `emu`), booted with `EMU_GPU=software` so the guest gets
|
||||
a real Vulkan device (SwiftShader) rather than the `-gpu host` default,
|
||||
which disables Vulkan in this VM entirely (`-feature -Vulkan`, because
|
||||
gfxstream can't pair Venus with the real GPU here — worth remembering,
|
||||
since the *default* `emu up` gives a device with **no** Vulkan adapter
|
||||
at all, which reads exactly like the old bindless failure if you don't
|
||||
know to ask for `EMU_GPU=software`):
|
||||
|
||||
cd rigs/gpu-probe
|
||||
ANDROID_NDK_HOME=$HOME/Android/Sdk/ndk/29.0.14206865 \
|
||||
cargo ndk -t arm64-v8a -P 26 build --release
|
||||
EMU_GPU=software emu up # from ~/repos/emulator-tools
|
||||
adb push target/aarch64-linux-android/release/gpu-probe /data/local/tmp/
|
||||
adb shell chmod 755 /data/local/tmp/gpu-probe
|
||||
adb shell /data/local/tmp/gpu-probe
|
||||
|
||||
Output: `adapters: 1 — Vulkan SwiftShader Device (Subzero) (Cpu)`,
|
||||
`features iris requires:` (none listed — the set is empty),
|
||||
`max_buffer_size … ok`, and **`IRIS DEVICE: ok`**. This is the fix
|
||||
measured working, on the exact rig that first measured it failing.
|
||||
Emulator stopped afterward (`emu down`); nothing was left running.
|
||||
@@ -0,0 +1,35 @@
|
||||
# TODO
|
||||
|
||||
Working list from Iris, 2026-09-03. Remove an entry when it lands; annotate
|
||||
one in place when it turns out to need a decision.
|
||||
|
||||
## App — transcript
|
||||
|
||||
- [ ] Messages received from other agents are inconsistent — sometimes they
|
||||
appear, sometimes they don't. **Needs a rig.** Read the code rather than
|
||||
measured: a live Claude session only learns of a peer message from the
|
||||
`origin` object on a turn's `result`
|
||||
(`session/claude/translate.rs`), which the CLI attaches to a turn the
|
||||
message *started*. So a message that arrives mid-turn, or a second one
|
||||
within one turn, has nowhere to be reported — while an imported session,
|
||||
which syncs from the CLI's own file, picks up every one of them. That
|
||||
would show exactly as "sometimes". Confirming it means driving a real
|
||||
stream-json session and sending it messages in both states.
|
||||
|
||||
## Session settings
|
||||
|
||||
- [ ] Autocompact belongs in session settings; empty disables it, which is the
|
||||
default. Iris chose "hand it to the driver" — only where a driver has
|
||||
auto-compaction of its own. **That option was offered on a false premise
|
||||
and is not buildable yet.** It named pi's `set_auto_compaction`, but pi
|
||||
was never built as a driver here: `session/llama.rs` talks to
|
||||
`llama-server`'s OpenAI-compatible endpoint directly, and its `compact()`
|
||||
refuses outright. Claude Code's auto-compaction is the CLI's own and
|
||||
nothing in the stream-json control protocol this app uses configures it.
|
||||
So the setting would be stored, passed to a driver, refused by every one
|
||||
of them, and the field would never appear on any session. What is needed
|
||||
first is either a driver that can take it, or a different rule — the
|
||||
server watching `contextTokens` and running `/compact` itself is the one
|
||||
that would work today, for Claude sessions, and it is the option that was
|
||||
not chosen.
|
||||
|
||||
@@ -0,0 +1,427 @@
|
||||
# The transcript cache
|
||||
|
||||
Asked for by Iris on 2026-09-04 and built the same day: keep the transcripts
|
||||
of recently visited sessions on the phone, so reopening one does not download
|
||||
it again. It has to save data over the tunnel, must not disturb a reply that
|
||||
is streaming when the screen is reopened, must never skip an event, and needs
|
||||
a manual reload for when the file on the machine has changed under it.
|
||||
|
||||
Like EXPLORER.md this records each decision with its reason and what was
|
||||
rejected, so that when one changes it is changed here rather than re-argued.
|
||||
"What building it changed" at the foot says which of them moved while it was
|
||||
being built. How to exercise it, and what has bitten, are in AGENTS.md.
|
||||
|
||||
## What it is, in one paragraph
|
||||
|
||||
A per-session file on the phone holding the exact JSON lines the server has
|
||||
already sent, in transcript order, with a record of which sequence numbers
|
||||
each run of lines covers. Everything the session screen fetches — the opening
|
||||
window, the pages it scrolls back through, the span an anchor restore reaches
|
||||
for — is asked of the cache first and of the server only for what the cache
|
||||
does not hold, and everything that arrives from the server is written into
|
||||
it. The live stream then resumes from the newest cached event, exactly as it
|
||||
resumes from the newest event on screen, so the server sends only what
|
||||
happened since. One tiny request checks that the cached tail is still what
|
||||
the server has before the stream is opened from it, and a button in session
|
||||
settings throws the cache away and rebuilds the screen as a cold open for the
|
||||
cases that check cannot see.
|
||||
|
||||
## The invariants
|
||||
|
||||
When a decision below looks arbitrary, it is one of these forcing it.
|
||||
|
||||
1. **What is on screen is what the server's transcript says, in order, with
|
||||
nothing missing, for every sequence number the screen claims to show.**
|
||||
The cache is a copy of server output and is never inferred, folded, or
|
||||
edited on the phone. Where the copy cannot be shown to be current, it is
|
||||
thrown away, not patched.
|
||||
2. **A cached line is never ahead of the live cursor, and the live cursor is
|
||||
never ahead of the cache.** The stream resumes from the newest cached
|
||||
event, so a reply that was mid-stream when the screen closed picks up at
|
||||
its next delta and folds into the same row.
|
||||
3. **The cache is never load-bearing.** A missing, evicted, corrupt or
|
||||
unwritable cache degrades to a cold open, never to a blank or wrong
|
||||
screen. Every path that reads it has a network path beside it producing
|
||||
the same result.
|
||||
4. **Data crosses the tunnel once.** A line already on the phone is not
|
||||
fetched again unless the reader asks (the reload button) or the check in
|
||||
decision 3 says it must be.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Raw server lines, on the phone, keyed by server and session
|
||||
|
||||
The cache stores the server's own JSON, one event per line, byte-for-byte as
|
||||
it arrived: the elements of the `/transcript` array and the `data:` payload
|
||||
of each SSE frame. Reading the cache runs the same `parseSeqEvent` the
|
||||
network path runs, so a cached transcript and a fetched one cannot draw
|
||||
differently, and an event type this build does not know
|
||||
(`SessionEvent.Unknown`) survives on disk for the build that will.
|
||||
|
||||
It lives under `context.cacheDir`, which is exactly what that directory is
|
||||
for: bytes the phone can regenerate from the server, which Android may delete
|
||||
under storage pressure without asking. Keyed by the server's host and port,
|
||||
because two servers can hold a session with the same id (the sandbox and the
|
||||
real server, or a re-enrolment) and a line from one shown against the other
|
||||
is invariant 1 broken. The `v1` segment is the format version: any change to
|
||||
the layout below bumps it, and a directory of another version is deleted on
|
||||
first use.
|
||||
|
||||
Rejected: a database (Room, SQLite). The access pattern is "the newest N
|
||||
lines" and "the lines before seq X", on files of tens of megabytes at most,
|
||||
and a JSONL file per contiguous run answers both by reading from its end. A
|
||||
database would be a new dependency for an index the file layout provides.
|
||||
|
||||
Rejected: caching folded `TranscriptItem` rows instead of events. Rows are a
|
||||
*rendering* of events, and their shape changes when the fold changes; the
|
||||
cache would need invalidating on every app update that touched `foldEvent`,
|
||||
and would still have to keep raw seqs for the stream cursor. Events are the
|
||||
server's contract and the only thing that is stable.
|
||||
|
||||
### 2. Chunks with explicit coverage; one contiguous run behind the cursor
|
||||
|
||||
A page from the server is a set of lines *and a claim about what they cover*,
|
||||
and the two are not the same thing. A coalesced page joins each run of
|
||||
`assistantText` deltas into one event carrying the seq of its *oldest* delta,
|
||||
so a page whose newest event has seq 1,200 may in fact cover every line up to
|
||||
the `before` it was asked with, say 1,650. Nothing in the lines themselves
|
||||
says so. So each stored chunk records its coverage as a half-open range
|
||||
`[first, end)`, where `end` is the `before` the request was made with — or,
|
||||
for a raw chunk, its newest seq plus one.
|
||||
|
||||
Chunks are files named by their coverage:
|
||||
|
||||
<first>-<end>.rows.jsonl a coalesced page; end is the `before` it was fetched with
|
||||
<first>-<end>.raw.jsonl an uncoalesced page or a closed live run
|
||||
<first>-open.raw.jsonl the live run: appended to by the stream
|
||||
|
||||
Two chunks are **adjacent** when one's `end` equals the other's `first`. The
|
||||
cache serves only the contiguous run of adjacent chunks that ends at the
|
||||
newest raw chunk (the **suffix**); chunks behind a gap are kept on disk,
|
||||
because the gap is usually filled (decision 4), but are never served across
|
||||
it.
|
||||
|
||||
**The newest chunk is always raw.** That is what makes the stream cursor and
|
||||
the probe well defined: a raw chunk's last line is a real event at a real
|
||||
seq, and the server never coalesces the newest window. It holds by
|
||||
construction — the opening window is fetched with no `before`, stream frames
|
||||
are raw, and a `reset` window is raw — and is *checked* on read: a `.rows`
|
||||
chunk found newest (which can only happen if the app died between closing one
|
||||
live run and appending to the next) purges the session's cache.
|
||||
|
||||
There is at most one open chunk. A stream event whose seq is not the open
|
||||
chunk's `end` — which is what a `reset` looks like from here — closes it by
|
||||
renaming it with its real end and starts a new one. An event whose seq is
|
||||
below the open chunk's `end` is already covered and is not written; the SSE
|
||||
contract is `seq > after`, so that is a guard rather than a path.
|
||||
|
||||
Rejected: one file per session, rewritten to prepend older pages. A 20 MB
|
||||
transcript would be rewritten on every page scrolled back to. The chunk
|
||||
directory costs a directory listing per open instead.
|
||||
|
||||
Rejected: trimming chunks to resolve overlaps. A coalesced event cannot be
|
||||
split at a seq inside its run, so an overlap between a coalesced page and an
|
||||
existing chunk has no clean cut. The cache therefore **never stores a page
|
||||
that overlaps an existing chunk**; decision 4 makes sure such a page is never
|
||||
fetched, and one that arrives anyway is used for display and not stored.
|
||||
|
||||
### 3. The cached tail is checked against the server before the stream opens from it
|
||||
|
||||
The transcript file is append-only in ordinary use, but it can be replaced or
|
||||
truncated — a sandbox re-seeded with the same ids, a backup restored, a
|
||||
session deleted and re-imported — and `catch_up` on such a file would hand
|
||||
the phone a continuation of a *different* conversation, spliced onto the
|
||||
cached one with no seam. That is the worst thing this feature can do, and it
|
||||
is caught with one request.
|
||||
|
||||
**The probe** is `GET /sessions/{id}/transcript?before=<cursor+1>&limit=1`,
|
||||
where `cursor` is the seq of the cache's newest line. `read_window` with that
|
||||
`before` returns the single newest event with seq ≤ cursor, which is the
|
||||
event *at* the cursor when it exists. It passes when that response, parsed
|
||||
with `parseSeqEvent`, is `==` to the cached line parsed the same way — over
|
||||
seq, ts, and the whole event. It fails when the response is empty, is a
|
||||
different seq, or differs in any field.
|
||||
|
||||
That equality rested on an assumption this plan stated and did not check:
|
||||
that the two ways the server hands out a line agree bit for bit. **They did
|
||||
not**, and the server was fixed — see AGENTS.md's entry on `float_roundtrip`.
|
||||
Comparing everything *except* `ts` was the other option and was rejected: a
|
||||
re-seeded fixture is identical in content and differs only in when it
|
||||
happened, which is exactly the case the probe exists for.
|
||||
|
||||
A failed probe **purges the session's cache and proceeds as a cold open**. A
|
||||
probe that cannot be made leaves the cached transcript on screen, shows the
|
||||
error on the stream banner where a connection failure shows today, and is
|
||||
retried on the stream loop's schedule; the stream is never opened until a
|
||||
probe has passed once for this screen instance.
|
||||
|
||||
What the probe does *not* catch: a line changed in the middle of the file
|
||||
with the tail intact, or a file rewritten so that the event at the cursor
|
||||
happens to be identical. Those are what the reload button is for, and the
|
||||
button's caption says so.
|
||||
|
||||
Cost: one request of a few hundred bytes, in the slot where the opening
|
||||
page's request would be — so the round trips before the stream is live are
|
||||
unchanged at two, and the bytes fall from a page to a line. The cached rows
|
||||
are drawn *before* the probe returns, which is the whole point; a failed
|
||||
probe replaces them, with the same appearance as a `reset`.
|
||||
|
||||
Rejected: a server-side check on the stream, answered with a distinct frame
|
||||
when the event at N is not what the phone thinks. Strictly better coverage —
|
||||
it would run on every reconnect — and no extra round trip. Not chosen because
|
||||
it puts a cache's validation into a protocol that otherwise knows nothing
|
||||
about caching, and because the reset frame already has to keep meaning "you
|
||||
are behind, your history is fine". Worth revisiting if the probe's round trip
|
||||
is ever measured as the thing making reopen slow.
|
||||
|
||||
Rejected: trusting the cache and relying on the reload button. Invariant 1 is
|
||||
not something a button restores after the fact.
|
||||
|
||||
Rejected: fetching the newest page as before and using it to validate the
|
||||
overlap. Zero saving on the opening page, which is the request paid on every
|
||||
open.
|
||||
|
||||
### 4. Pages ask the server only for the gap: `after` on `/transcript`
|
||||
|
||||
After a reader has been away, the cache holds `[a, b)` and the screen holds
|
||||
the newest window `[W, …)` with a gap between `b` and `W`. Paging back from
|
||||
`W` asks for a coalesced page before `W`, and that page may reach back past
|
||||
`b` — a single reply is hundreds of lines, so forty rows can be thousands of
|
||||
seqs — producing exactly the overlap decision 2 refuses to store. Left like
|
||||
that, every cached chunk would be dropped in turn as the reader paged back
|
||||
through the gap, and the cache would save nothing for the sessions it exists
|
||||
for.
|
||||
|
||||
So the transcript route takes a lower bound, `after`, named to match the SSE
|
||||
route's (exclusive, `seq > after`). `read_window` starts the walk at
|
||||
`first_at_or_after(after + 1)` instead of at `end - limit`. A delta run cut
|
||||
at the start is emitted as the partial it is, exactly as one cut by `limit`
|
||||
already is, and `healSplitMessage` welds it on the phone — no new mechanism.
|
||||
|
||||
The phone passes `after = b - 1` where `b` is the `end` of the nearest chunk
|
||||
whose `end ≤ before`, and nothing when there is none. A page that comes back
|
||||
with `first == b` is adjacent, and the suffix now runs through the old
|
||||
chunks: the gap is closed with exactly the bytes it was wide, and the history
|
||||
behind it is served locally from then on.
|
||||
|
||||
Rejected: fetching the gap raw in one request, which is what the anchor
|
||||
restore does. Exact, but a gap of ten thousand lines is several megabytes
|
||||
downloaded to save re-downloading history the reader may never scroll to.
|
||||
|
||||
Rejected: dropping the cached run whenever a gap opens. Being more than
|
||||
`CATCH_UP_LIMIT` (200) events behind is the *ordinary* state of an active
|
||||
session revisited — 200 raw events is one reply — so this would empty the
|
||||
cache for exactly the sessions that are opened most.
|
||||
|
||||
### 5. A page is served locally in rows, mirroring the server's count
|
||||
|
||||
`loadOlderPage` asks for `HISTORY_PAGE` (40) **rows** when coalescing and for
|
||||
a number of **events** otherwise (the anchor restore). Served from the cache,
|
||||
the events branch is the `limit` lines before `before`. The rows branch walks
|
||||
back counting rows the way `parse_coalesced` does — every event that is not
|
||||
an `assistantText` is a row, and each maximal run of `assistantText` lines is
|
||||
one row — stopping only between rows. It does not join the deltas; the fold
|
||||
does that, and the joined row keeps the seq of its first delta either way, so
|
||||
anchors and the next `before` land where they do on the network path.
|
||||
|
||||
A cached page is allowed to be **short**: a walk that reaches the suffix's
|
||||
oldest chunk returns what it found. The caller already treats a short page as
|
||||
a page; only an *empty* page means "start of the conversation", and the cache
|
||||
never returns one — it returns `null` (a miss) and the network is asked.
|
||||
|
||||
A miss is `before` **outside what the suffix covers continuously** — above
|
||||
its newest `end`, or at or below its oldest `first`. This plan first said a
|
||||
miss was "no chunk of the suffix ends at `before`", which is wrong in the
|
||||
commonest case there is: a warm open draws the newest eighty lines of the
|
||||
live run, so the cursor the reader then scrolls back from is in the *middle*
|
||||
of a chunk. Under the narrower rule every warm open sent its first backwards
|
||||
page to the server, and that page overlapped what the phone already held and
|
||||
could not be stored, so the same history was fetched again on every visit.
|
||||
The feature would have saved the opening window and nothing else.
|
||||
|
||||
The row rule is a copy of the server's, and copies drift. It is short, it is
|
||||
pure, and it is under a JVM unit test with the same fixture as the server's
|
||||
`coalescing_counts_rows_and_joins_delta_runs` — a run cut by the limit, a
|
||||
`usageDelta` inside a run (the server flushes the run there, so it is two
|
||||
rows), and a page that is all one run.
|
||||
|
||||
### 6. What a `reset` means for the cache: behind, not wrong
|
||||
|
||||
The server sends `reset` when the cursor is more than `CATCH_UP_LIMIT` events
|
||||
behind, then the newest 200 raw events. For the cache that means **the
|
||||
history is intact and there is a gap**: the probe passed, the file is
|
||||
append-only, and the window's first seq is above the open chunk's end. The
|
||||
store learns this from the first window event's seq and needs no signal from
|
||||
the screen; the gap is filled by paging.
|
||||
|
||||
The reset handler also clears `queued` and `waitingCommands`, which it did
|
||||
not originally. Both are folded from events, and a `messageQueued` whose
|
||||
resolving `userMessage` fell in the gap would otherwise draw a waiting bubble
|
||||
for a message the session has long since read. That was a latent bug made
|
||||
likely by the cache, because a cached tail is older than a fetched one.
|
||||
`contextTokens` needs no clearing: `UsageDelta.context` is absolute, so the
|
||||
window's first one corrects it.
|
||||
|
||||
### 7. Session state that is not the transcript comes from the list, not the cache
|
||||
|
||||
`apply` derives `status`, `model`, `permissionMode` and `compactingSince`
|
||||
from `Status` and `Settings` events. Replayed from a fetched page those are
|
||||
current; replayed from the cache they are as old as the last visit, while the
|
||||
list row the reader just tapped was fetched moments ago. So the cache replay
|
||||
runs through `apply` for the transcript's sake and then **reassigns those
|
||||
four from `summary`**, which is the newer of the two measurements; the
|
||||
stream's catch-up then makes them current. Without this a session that
|
||||
finished an hour ago would open saying "working" until the stream connected,
|
||||
which is a status row lying for a round trip.
|
||||
|
||||
### 8. Reload, in session settings
|
||||
|
||||
A row under the working directory showing what the button discards:
|
||||
|
||||
[ Transcript ] 2.3 MB cached [ Reload ]
|
||||
|
||||
The size is the unknown state made visible — `null` while the directory is
|
||||
being measured (spinner, as the notifications switch does), "nothing cached"
|
||||
when the directory is absent or empty, else the size. The caption is in the
|
||||
style of Move's, because the button costs something the reader cannot see:
|
||||
*"Reload throws away this phone's copy and fetches the transcript from the
|
||||
server again. Use it when what is shown here disagrees with the file on the
|
||||
machine."*
|
||||
|
||||
Pressing it purges the session's cache directory, closes the dialog, and
|
||||
rebuilds the screen as a cold open, with the reader put back where they were.
|
||||
The mechanism is an `epoch` counter in the key of the opening effect and the
|
||||
stream effect; incrementing it cancels both and relaunches them. `savedAnchor`
|
||||
is keyed on the epoch too, so the restore reads the anchor saved at the
|
||||
reader's *current* position. The button is enabled whether or not anything is
|
||||
cached: "what I see disagrees with the machine" is a state an empty cache can
|
||||
also be in, and a control that comes and goes makes its own presence the
|
||||
signal.
|
||||
|
||||
Nothing is announced on success — the transcript shows the opening spinner
|
||||
and then the rows, which is what the screen already says about a reload. A
|
||||
failure is the opening fetch's, and lands on the stream banner.
|
||||
|
||||
Rejected: a global "clear transcript cache" in the app's settings. Not asked
|
||||
for; eviction bounds the total, and the per-session button is where the
|
||||
reader is when they notice a problem. Easy to add as one more caller of
|
||||
`purgeAll`.
|
||||
|
||||
### 9. Budget, eviction, pruning
|
||||
|
||||
Bounded three ways, each with its path out written beside the path in:
|
||||
|
||||
- **Budget.** `CACHE_BUDGET_BYTES` is 256 MB across all sessions of one
|
||||
server. Each open touches the session directory's mtime; after the opening
|
||||
replay, on `Dispatchers.IO`, the store sums the server's directories and
|
||||
deletes least-recently-touched ones (never the one on screen) until under
|
||||
budget. 256 MB is a dozen of the largest transcripts seen in this VM
|
||||
(21 MB for 24,000 events) and a small fraction of a phone; it is a number
|
||||
to revisit against real use, not a measurement.
|
||||
- **Deleted sessions.** The list screen's delete purges after `deleteSession`
|
||||
succeeds, and every successful list fetch calls `retainOnly(ids)`, so a
|
||||
session deleted from another device is pruned on the next visit to the
|
||||
list. `Drafts.kt` chose not to prune because its residue is bytes; here it
|
||||
is megabytes.
|
||||
- **Android.** `cacheDir` may be emptied at any moment, including while a
|
||||
screen is open. Every read tolerates a missing directory and every write
|
||||
failure is swallowed once.
|
||||
|
||||
### 10. The cache never breaks the screen
|
||||
|
||||
Every store operation that touches the disk catches `IOException` and answers
|
||||
as if the cache were empty: `null` from a read, no-op from a write, logged
|
||||
once. After a write failure the instance stops writing, so a full disk costs
|
||||
one log line rather than one per delta. A line at the end of an open chunk
|
||||
that does not parse — the app died mid-write — is dropped and the file
|
||||
truncated to the last good line before anything is served from it; a line
|
||||
that does not parse anywhere else purges the session's cache, since that file
|
||||
was not written by this code. None of this is reported on screen: none of it
|
||||
changes what the screen shows, and the reader has nothing to do about it.
|
||||
|
||||
## Layout on disk
|
||||
|
||||
<cacheDir>/transcripts/
|
||||
v1/
|
||||
10.0.2.2_8443/ one directory per server (host_port)
|
||||
3f2c…/ one per session id
|
||||
1-1650.rows.jsonl coalesced page: covers seqs 1..1649
|
||||
1650-2001.rows.jsonl
|
||||
2001-2400.raw.jsonl a closed live run
|
||||
2600-open.raw.jsonl the live run
|
||||
|
||||
Here 2400..2599 is a gap: the reader was away for two hundred events and the
|
||||
stream reset. The suffix is the single chunk `2600-open`; the first backwards
|
||||
page asks the server for `before=2600&after=2399&coalesce=true`, and once a
|
||||
page comes back with `first == 2400` the suffix runs to seq 1.
|
||||
|
||||
Each `.jsonl` is one JSON object per line, oldest first, exactly as the
|
||||
server sent it. No header, no index: coverage is in the name, order is the
|
||||
file's, and the seq is in every line.
|
||||
|
||||
## What building it changed
|
||||
|
||||
Each of these contradicted the plan, and each was found by running it rather
|
||||
than by reading it. The decisions above are amended in place; this is what
|
||||
moved, so a reader who remembers the first version knows what to re-read.
|
||||
|
||||
- **The probe's equality had a false premise** (decision 3). The server did
|
||||
not hand out the same line twice the same way. Fixed on the server.
|
||||
- **A cached page starts anywhere inside the run** (decision 5). Requiring a
|
||||
chunk boundary would have made the cache save the opening window and
|
||||
nothing else.
|
||||
- **The opening window is stored by `append`, not by `storePage`.** The
|
||||
sketch had `storePage` grow a special case for "this page is the new open
|
||||
chunk", decided by an implicit condition a raw history page also satisfies.
|
||||
Appending each line instead is the mechanism that already exists, and the
|
||||
open chunk stays the one thing that grows.
|
||||
- **Chunks are read backwards, in blocks, and never whole.** Every question
|
||||
the cache is asked is about the newest end, and a live run reaches the size
|
||||
of the conversation — so reading a chunk to answer with eighty lines of it
|
||||
is the cost the server's own reader was rewritten to stop paying, arriving
|
||||
on the phone. Damage is therefore noticed when a read reaches it rather
|
||||
than up front, which is the better time: what is not read cannot be wrong.
|
||||
- **The stream waits for the opening effect's probe.** The screen lifts
|
||||
`ready` before the probe returns — that is the point of the cache — so
|
||||
`ready` stopped being the whole gate, and the stream loop asked the same
|
||||
question a second time and raced its own answer. Two probes per warm open,
|
||||
visible in the server's log.
|
||||
- **`SessionCache` is synchronized.** The stream appends live events from one
|
||||
IO thread while a reader scrolling back reads pages from another; the open
|
||||
chunk's name, its end and its writer must never be seen half-rotated.
|
||||
|
||||
## What it cost, measured
|
||||
|
||||
On the emulator against `app/ui-sandbox.sh`, 2026-09-04, on a session of 505
|
||||
events (three short exchanges and two 300-delta replies):
|
||||
|
||||
- **Reopening it: one request, for one event.** The probe, and nothing else —
|
||||
including scrolling the whole conversation back to its first line. A cold
|
||||
open of the same session is two requests and 100 events.
|
||||
- **A reset after falling 300 events behind costs the gap and no more.** The
|
||||
window arrived at seq 306, the phone held up to 202, and the first
|
||||
backwards page asked `before=306&after=201` and came back with **four
|
||||
coalesced rows** covering 202..305 — against the 104 raw events an
|
||||
unbounded page would have re-fetched and thrown away.
|
||||
- **Every chunk is exactly what the server says for the range its name
|
||||
claims**, checked line by line against `/transcript` for each chunk's own
|
||||
`before`/`after`/`coalesce`, across a reset and a gap-fill.
|
||||
- **Nothing about drawing changed**, which is what a cache must not do:
|
||||
`transcript-bench.sh` before and after, same viewport content and gestures,
|
||||
p50 16.9ms both times and the transcript's own draw accounting at 0.33ms
|
||||
against 0.32ms.
|
||||
|
||||
Still to measure, in real use rather than here: the size the cache reaches
|
||||
against `CACHE_BUDGET_BYTES`, and whether the probe's round trip is ever what
|
||||
a reader waits on.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **The probe on every reconnect, not only on open?** A file replaced *while*
|
||||
the screen is open is not made worse than it was, but the server-side check
|
||||
decision 3 rejects would close it. Decide after measuring how often the
|
||||
probe's round trip is what the reader waits on.
|
||||
- **Images.** `SessionImage` fetches bytes from the files route on draw; they
|
||||
are not part of this cache and are re-downloaded per view. A separate,
|
||||
simpler cache (a directory of refs, no ordering) if the measurement above
|
||||
says the images are where the data goes.
|
||||
Reference in new issue
Block a user