IRIS_TODO.md's "the composer has no touch-drag scroll". `Scroll::drag`
takes its pan from the same `sense::DragGesture` `List` is driven by --
arbitration, DRAG_SLOP, velocity and pointer capture all stay in sense.rs
and only what a committed pan *means* is decided per caller -- and
`WidgetLike::scrollable()` registers it beside the wheel handler it already
registered, so every scroll area pans on a finger with nothing added at the
call site. No fling: `Scroll` has no per-frame tick to animate one and the
areas it wraps are at most a screenful. `Scroll::amt()` exposes the pan
position.
`attr.rs`'s `on_press` treated an already-focused field as the plain
click_or_drag case, so every Pressing frame extended a selection. It now
applies the same DRAG_SLOP rule its unfocused branch already did: a press
past the slop vertically abandons its pending selection for the rest of the
gesture, so the scroll area around the field wins it. That is Android
EditText's own behaviour and it is what lets a swipe up over the composer
scroll instead of dragging a highlight through what you typed.
Also fixed, found doing it: `ActiveData::mask` stored the mask a widget
*set* rather than the one it was drawn *under*, and `redraw` feeds that
field back in as the inherited mask -- so a targeted redraw of any `Masked`
handed it its own mask and aborted on `set_mask`'s nested-mask assert. A
real abort on the emulator, `assertion failed: self.mask == MaskIdx::NONE`.
And the per-frame orphan guard from 76b1f99 is now a count comparison
(O(active widgets)); the O(primitives) walk only runs to build the failure
message, because running it per frame made a debug build on the emulator too
slow to finish a bench run at all.
Tests: four in scroll.rs (pan past the slop, a tap inside it, a horizontal
drag, the end clamp), `a_finger_drag_over_a_scroll_area_pans_it` in
sense_tests.rs driving the whole registration/dispatch/capture path (fails
with "got 0" without the new registration), and
`redrawing_a_masked_widget_does_not_nest_its_own_mask` in layout_tests.rs
(aborts on the pre-fix code).
The composer itself is deliberately still not `.scrollable()`: `Scroll`
measures against the window rather than its own offered box, so inside the
`MaxSize` capping it at six lines it pans the field out of the bar --
measured, reverted and written down in RUST.md and DECISIONS.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
648 lines
40 KiB
Markdown
648 lines
40 KiB
Markdown
# iris: known problems and things still to build
|
|
|
|
Iris's own list for the library, recorded 2026-09-04 in her words where it
|
|
matters, so the agents working through RUST.md pick these up in a sensible
|
|
order rather than rediscovering them. Each item says where it sits in the
|
|
order and what "done" looks like. Tick and date them in place.
|
|
|
|
## Fix
|
|
|
|
- [x] **`request_device` asked for compute-shader limits it never uses
|
|
(2026-09-05).** `Limits::default()` (both `iris/src/android/render.rs`
|
|
and `iris/src/default/render.rs`) requests desktop-tier compute limits
|
|
unconditionally, even though nothing in `iris`/`iris-core` creates a
|
|
`ComputePipeline` or writes a `@compute` shader stage — confirmed by
|
|
grepping the whole tree, not assumed. That crashed device creation
|
|
outright on the Android emulator's software GL path (`EMU_GPU=software`,
|
|
`--features force-gles`): SwiftShader's GL reports itself as OpenGL ES
|
|
3.0, which has no compute shaders, so the adapter's real limit is 0
|
|
against the unconditional request for 65535 — the same would happen on
|
|
any real GLES-3.0-only Android device. Fixed by a new, shared
|
|
`iris_core::device_limits()` (`iris/core/src/render/mod.rs`) that zeros
|
|
exactly the six `max_compute_*` fields rather than switching to a
|
|
downlevel `Limits` preset — `downlevel_webgl2_defaults()` also zeros
|
|
`max_storage_buffers_per_shader_stage`, which `shader.wgsl`'s vertex
|
|
stage needs (four `var<storage>` buffers), so that preset would trade
|
|
this crash for a bind-group-layout one on the same hardware.
|
|
`rigs/gpu-probe`'s own hand-mirrored `Limits` (it is deliberately its
|
|
own crate, not able to call `device_limits()` directly) was updated to
|
|
match. See `DECISIONS.md` and RUST.md's I5 box for the account,
|
|
including what could not be re-verified on-device this pass (the
|
|
emulator was in concurrent use by another session).
|
|
|
|
- [x] **Input does not fall through by input type (2026-09-04).**
|
|
`SensorUi::run_sensors` (`src/default/sense.rs`) used to set "consumed,
|
|
stop checking lower layers" from mere hover — a widget registered for
|
|
nothing but `click()` blocked a `Scroll` meant for whatever was behind
|
|
it, since "the cursor is over this widget" and "this widget handled the
|
|
event" were the same check. Fixed by judging consumption per input
|
|
kind: with no button transition and no scroll happening this frame
|
|
("momentary" activity), the topmost hovered widget still wins, same as
|
|
before; when something momentary *is* happening, only a widget whose
|
|
registered senses actually include a matching non-hover one (checked
|
|
via a new `TypeEventManager::registered`, which lists what a widget
|
|
registered without running anything) consumes it, so a widget with only
|
|
`Hovering`/click handlers can no longer block a scroll from reaching a
|
|
list underneath. `iris/src/sense_tests.rs` builds a button-over-a-list
|
|
`Stack` with a plain `HasEvents` impl (no GPU or window) and checks both
|
|
directions: a scroll over the button reaches the list, and a real click
|
|
still reaches the button — confirmed to fail on the pre-fix code and
|
|
pass after.
|
|
|
|
- [x] **Appending one image to an already-loaded list rebuilds every other
|
|
image's bind group (2026-09-05, fixed 2026-09-05).** Found by the
|
|
benchmark below: `GpuTextures::update` (`core/src/render/texture.rs`)
|
|
triggered `rebuild_image_bind_groups` — a loop over *every live
|
|
standalone image*, rebuilding its `BindGroup` — whenever the shared
|
|
`masks` or `move_offsets` GPU buffer was resized (`masks_resized ||
|
|
moves_resized` in `UiRenderNode::update`, `core/src/render/mod.rs`), and
|
|
a widget getting its *first* move-offset slot (LAYOUT.md section 2 —
|
|
every widget gets one on first draw) could be exactly what grows that
|
|
buffer. So one new message with one new image, appended to a transcript
|
|
that already has N images loaded, did not cost O(1): it cost one
|
|
`create_image` for the new image plus one `make_image_bind_group` per
|
|
*existing* image, because the new widget's own move slot pushed the
|
|
arena past its capacity. Measured directly in
|
|
`iris/examples/bench_images.rs`: appending a 1,001st image to 1,000
|
|
already-settled ones reported **1,001** bind-group creates for that one
|
|
frame, not 1 (`./run-bench.sh images`, frame 5 in the transcript below).
|
|
|
|
**Fix**: `masks`/`move_offsets` never belonged in a standalone image's own
|
|
bind group (group 2) in the first place — the group also holds that
|
|
image's own texture view, which is the only thing that is genuinely
|
|
per-image, so a buffer shared by *everything* forced a rebuild of
|
|
*every* group the moment it moved. Gave masks/move_offsets their own
|
|
bind group (group 3 in `shader.wgsl` and `UiRenderNode`: `masks_layout`/
|
|
`masks_group`), bound once per frame in `UiRenderNode::draw` rather than
|
|
once per draw call, instead of duplicating them into every per-image
|
|
group. `GpuTextures` and its image bind groups now know nothing about
|
|
either buffer — `rebuild_image_bind_groups` is called only from
|
|
`grow_array` (the atlas array texture growing, which genuinely does
|
|
change what every image's own bind group must reference) — so a
|
|
masks/move_offsets resize now touches exactly one bind group, ever,
|
|
regardless of how many images are live. Numbers after the fix, same
|
|
benchmark and command:
|
|
|
|
./run-bench.sh images
|
|
frame=1 bind_group_creates=1000 (cold load, unchanged)
|
|
frame=2 bind_group_creates=0 (was 1000 -- see the item below)
|
|
frame=3 bind_group_creates=0
|
|
frame=4 bind_group_creates=0
|
|
(append one image here)
|
|
frame=5 bind_group_creates=1 (was 1001)
|
|
frame=6 bind_group_creates=0
|
|
|
|
`run-headless.sh tabs --shot` still 27266 bytes, byte-for-byte unchanged,
|
|
confirming the bind-group restructuring changed nothing about what is
|
|
drawn.
|
|
- [x] **Bind-group creation takes two frames to reach the steady state, not
|
|
one (2026-09-05, closed by the fix above, 2026-09-05).** Same benchmark:
|
|
loading 1,000 images cold used to report 1,000 creates on frame 1
|
|
(expected — `create_image`, one per new image) *and again* 1,000 on
|
|
frame 2, before settling to 0 from frame 3. This was `rebuild_image_bind_groups`
|
|
firing a second time for the same masks/move-offsets buffer-growth
|
|
reason as the item above, confirming the guess recorded here — the two
|
|
were exactly the same root cause measured two different ways. Frame 2
|
|
now reports 0 (see the numbers above); not a separate fix.
|
|
|
|
- [ ] **A read-only text display has no widget of its own — P0's bench
|
|
report area is a `TextEdit` standing in for one (2026-09-05).** The only
|
|
way to get selectable text on screen today is `.editable(...)` plus
|
|
`.attr::<Selectable>(())` (`Selectable` is only implemented for
|
|
`TextEdit`, `iris/src/attr.rs`), which also makes the field focusable —
|
|
tapping the bench report opens the soft keyboard over text nothing lets
|
|
you type into. Harmless for a bench-only debug screen (not fixed this
|
|
pass), but a real "selectable, not editable" text primitive would
|
|
remove the keyboard side effect and is worth having before another
|
|
screen wants the same thing (P1's own transcript rows already read
|
|
their content from a `TextEdit` for the same reason).
|
|
|
|
## From the phone, 2026-09-06
|
|
|
|
Found on Iris's own phone while working RUST.md's P0 box's phone-report
|
|
follow-ups. Recorded here rather than fixed in that pass, so a follow-up
|
|
agent takes them without colliding with that pass's `bench_client.rs`/
|
|
`android/view.rs`/`android/sense.rs` changes.
|
|
|
|
- [x] **Swiping has no momentum, fixed 2026-09-06.** `List::fling`/
|
|
`VelocityTracker`/`FlingCalculator` (`iris/src/widget/list.rs`,
|
|
`iris/src/sense.rs`) -- IRIS.md's 2026-09-06 entry has the full account.
|
|
Wired through `Selection::drag`'s release path, cancelled by the next
|
|
touch-down, clamped at the loaded content's start/end. Verified by unit
|
|
test (fling distance against the closed-form spline result, cancel-on-
|
|
touch, the clamp), not yet by an on-device or emulator feel-check --
|
|
that is still open.
|
|
- [x] **Scrolling down sometimes jitters the text, fixed 2026-09-06.**
|
|
Root-caused by reading `DragArbiter::update`'s `Undecided`-to-`Panning`
|
|
transition rather than by an on-device trace (no emulator was used this
|
|
pass): it was the first named suspect, not the second. `self.last` stays
|
|
at the press origin for every `Undecided` frame (nothing pans while the
|
|
gesture might still be a selection), so the frame that finally crosses
|
|
`DRAG_SLOP` returned `Pan(dy)` with `dy` measured from `press_start` --
|
|
the *whole* pre-threshold drag, applied to the list in one step, however
|
|
many frames it had taken to get there. Fixed by applying only the
|
|
excess past `DRAG_SLOP` on that one frame (`dy - DRAG_SLOP.copysign
|
|
(dy)`), the same "consume the slop, don't replay it" rule Android's own
|
|
touch handling follows. New regression test,
|
|
`crossing_the_slop_by_a_little_pans_by_a_little` (`iris/src/sense.rs`).
|
|
**Not yet done**: an emulator trace of the real per-frame offset
|
|
confirming this was the whole story on real touch input rather than
|
|
only the arbiter's own unit tests -- worth a follow-up pass before
|
|
calling it fully closed.
|
|
- [x] **Composing text held back until a space, caret not moving, fixed
|
|
2026-09-06.** `InputMethodManager.updateSelection` was never called --
|
|
see IRIS.md's 2026-09-06 entry and RUST.md's P0 box, item 1, for the
|
|
full account and the emulator evidence.
|
|
- [x] **Swipe over the composer summons the keyboard, fixed 2026-09-06.**
|
|
`Selector`/`Selectable` now wait for a completed tap -- see IRIS.md's
|
|
2026-09-06 entry and RUST.md's P0 box, item 5. Verified via `dumpsys
|
|
input_method`'s `mInputShown` on the emulator, not yet on the phone.
|
|
- [x] **Text disappears again after leaving and returning to the app,
|
|
fixed 2026-09-06.** `GlyphAtlas::clear`/`Textures::reset` on a
|
|
genuinely new renderer -- see IRIS.md's 2026-09-06 entry and RUST.md's
|
|
P0 box, item 4. Verified on the emulator (home, reopen, screenshot);
|
|
not yet on the phone.
|
|
- [x] **Composed/typed text never becomes visible at all -- root-caused
|
|
and fixed 2026-09-06.** Not the renderer at all: **the composer's buffer
|
|
was empty the whole time.** `TextEditCtx::select` (`iris/src/widget/
|
|
text/edit.rs`) compared the tap against the *laid-out text's* box and
|
|
set `selection = None` for anything outside it -- and an empty field's
|
|
layout is a zero-width box, so tapping an empty composer granted focus
|
|
and opened the keyboard while leaving no caret; `insert_str` returns
|
|
early with no caret, so every keystroke after that was dropped in
|
|
silence. Gboard's suggestion strip is its own composing state, not a
|
|
read of our buffer, which is what made the earlier pass conclude the
|
|
buffer held the text. Fixed by letting parley clamp a tap outside the
|
|
layout to the nearest cursor position (a press that reaches `select`
|
|
has already been hit-tested to the widget, so there is no "outside"),
|
|
plus a `debug_assert!` in `insert_str` so an insert with no caret fails
|
|
at the mistake instead of dropping input -- it immediately caught
|
|
`layout_tests::composing_text_after_a_keyboard_resize_...` typing into
|
|
an unfocused field. Three new tests in `edit.rs`
|
|
(`tapping_an_empty_field_places_a_caret_so_typing_lands`,
|
|
`tapping_past_the_end_of_the_text_clamps_to_the_end`,
|
|
`dragging_without_a_previous_selection_selects_nothing`); the first
|
|
fails on the pre-fix code. Emulator evidence: `adb shell input text`
|
|
after `tap 'Message'` now shows the text in the bar
|
|
(`/tmp/final-typing.png`) and logs `iris text render: chars=5 ...
|
|
glyphs=5`, against `glyphs=0` on every keystroke before.
|
|
|
|
**The old, superseded diagnosis, kept because it was wrong in an
|
|
instructive way:** The composer bar stays empty even once the
|
|
buffer genuinely holds the typed text (confirmed indirectly: Gboard's
|
|
own suggestion strip reacts correctly to each keystroke). A new unit
|
|
test proves the widget tree's own layout math resolves the field's
|
|
region correctly across a keyboard resize, so the bug is downstream of
|
|
that -- most likely `UiRenderState::redraw`'s single-widget redraw path,
|
|
or specific to this emulator's forced `force-gles` backend (untested on
|
|
Vulkan or the real phone). RUST.md's P0 box, item 2, has the full
|
|
writeup, what was ruled out, and where to look next. **Also unverified
|
|
because of this**: item 3's composer rebuild (one `Stack`-based widget,
|
|
a capped/scrollable height, bottom padding tied to the IME/nav-bar
|
|
inset) -- structurally in place and unit-tested, but its own visual
|
|
correctness cannot be screenshotted until text actually renders.
|
|
- [~] **The composer has no touch-drag scroll for overflowing text.**
|
|
**The mechanism is in, the composer is not, 2026-09-06.** `Scroll::drag`
|
|
takes its pan from the same `sense::DragGesture` `List` uses and
|
|
`WidgetLike::scrollable()` registers it beside the wheel handler, so
|
|
every scroll area in the codebase now pans on a finger (no fling -- see
|
|
IRIS.md). A vertical drag inside a focused field no longer extends a
|
|
selection, matching Android's `EditText`. But the composer field is
|
|
**not** wrapped in `.scrollable()` -- the note above was describing
|
|
intent, not the code -- and wrapping it was tried and reverted: `Scroll`
|
|
measures against the window rather than its own offered box, so inside
|
|
the `MaxSize` that caps the field at six lines it pans itself out of the
|
|
bar entirely (emulator, 474 characters, the bar collapsed to its
|
|
padding). docs/RUST.md's plan box has the numbers and the next step.
|
|
|
|
## From the phone, 2026-09-06, 11:39 (build delivered 02:07, commit 543f6d9)
|
|
|
|
Iris's report on the build with the composing-text, tap-vs-swipe and
|
|
atlas-reset fixes, with a screenshot, verbatim. Each is open until an
|
|
agent ticks it here with the evidence.
|
|
|
|
- [x] **"The app definitely does not start with keyboard spacing
|
|
correct. This is how it looks without me doing anything initially."**
|
|
**Not an inset bug at all -- fixed 2026-09-06.** The black third is the
|
|
bench shell's own empty *benchmark report* pane: `bench_client.rs`'s
|
|
root tree gave it `.height(rest(1))` beside `content.height(rest(2))`,
|
|
so an empty `TextEdit` reserved a third of the window at every launch
|
|
and pushed the composer up by exactly that. Measured on this checkout's
|
|
emulator at the phone's own size (1080x2424, density 420, gesture nav),
|
|
which reproduced Iris's screenshot exactly: new `iris insets:` log line
|
|
reported `bottom=63 ime_bottom=0` at launch (a nav bar, no keyboard --
|
|
so the inset the composer was fed was never large), while `ui-trace
|
|
show -m Message --field box` put the field at `31,1488..1048,1540` on a
|
|
2282px-tall surface, 789px clear of the bottom -- that pane's third.
|
|
**Unit mixing checked explicitly and cleared**: `set_bottom_inset` takes
|
|
physical px and stores `Len::abs`, `MainActivity.java`'s `1`/`0`
|
|
`ime_bottom` only ever reaches `insets.bottom.max(ime_bottom)` and
|
|
`> 0.0`, and every `dp` in the composer resolves at layout time. Fix:
|
|
the report pane is sized to its content (`.max_height(dp(260))
|
|
.scrollable()`), and moved above the transcript so it cannot eat the
|
|
composer's nav-bar clearance. After: field box `31,2277..1048,2329`,
|
|
grey bar ending at device y2361 with the 63px nav strip below it
|
|
(`/tmp/fix1.png` this pass).
|
|
The screenshot shows the composer bar (the grey band) sitting about
|
|
two thirds of the way down a 704x1568 screen, with black below it to
|
|
the bottom, and the transcript ending at "Claude / Results" just above
|
|
it -- at launch, no keyboard. So the composer's bottom padding, which
|
|
the 2026-09-06 rebuild tied to the IME/nav-bar inset, is being fed a
|
|
large value at start on the phone. Suspects, in order: the initial
|
|
inset delivery on the phone (GrapheneOS, gesture navigation) versus
|
|
the emulator; `ime_bottom` now carrying a `1`/`0` boolean through a
|
|
field the composer may still read as pixels or dp; a stale value from
|
|
before the first `on_insets_changed`. Reproduce with the phone's
|
|
screen size and density on the emulator before guessing.
|
|
- [~] **"Swiping still gets caught by the grey bar but keeps working
|
|
after I go past it."** Improved 2026-09-06 by the focused-field rule
|
|
below, still needs her phone to close. `attr.rs`'s `on_press` treated an
|
|
already-focused composer as the plain drag-to-select case, so a swipe
|
|
starting inside it dragged a highlight through the typed text for the
|
|
whole gesture; it now abandons that the moment the press passes
|
|
`DRAG_SLOP` vertically (Android `EditText`'s own rule), which removes one
|
|
of the two things that made the bar feel like it caught the swipe. The
|
|
residual `DRAG_SLOP` measured from the boundary crossing, described
|
|
below, is unchanged. Original note follows.
|
|
Not closeable from the emulator, annotated
|
|
2026-09-06 after the `DragGesture` merge. `attr.rs`'s `on_press` never
|
|
calls `capture_pointer` and never consumes a `Pressing` frame past
|
|
`DRAG_SLOP` (it just stops watching), so once the finger's *current*
|
|
position leaves the composer's box and enters the list's, `List`
|
|
starts receiving ordinary hit-tested `Pressing` frames there --
|
|
`DragArbiter::is_idle()`'s 2026-09-05 recovery (a missed `PressStart`)
|
|
picks it up rather than leaving it stuck. What this does **not** do is
|
|
what "wherever it began" implies literally: `DragArbiter::press_start`
|
|
restarts from the *boundary-crossing* position, not from the original
|
|
touch-down inside the composer, so the pan still needs a fresh
|
|
`DRAG_SLOP` of travel measured from the boundary rather than from the
|
|
start of the gesture -- composer and list are adjacent, non-overlapping
|
|
widgets (`lib.rs`'s `(list, composer_bar).span(Dir::DOWN)`), and only
|
|
the composer forwarding its own drag to the list would remove that
|
|
residual slop entirely, which is more than this pass's merge changes.
|
|
RUST.md's merge-pass box has the reasoning in full and an emulator
|
|
swipe confirming the composer's own box never moves/resizes during it;
|
|
whether the residual slop is still perceptible as "caught" needs Iris's
|
|
phone, since the emulator's per-widget boundary is a few dp wide and
|
|
easy to cross without noticing on a real screen too.
|
|
- [ ] **"Flinging still does not work."** No longer expected to reproduce
|
|
after the `DragGesture` merge (`e12c708`, pointer capture +
|
|
`CursorSense::Drop`), 2026-09-06. Emulator evidence (RUST.md's
|
|
merge-pass box, check (b)): a real `ui-trace` finger swipe followed by
|
|
screenshot-hash sampling caught a post-release frame distinct from the
|
|
drag's own last frame in one run, and every run showed 28-32
|
|
`render()` frames per gesture against an idle baseline of 0 and ~8
|
|
expected from the drag alone -- redraw kept being requested well past
|
|
the finger lifting, which only happens while a fling is still
|
|
animating. Left unticked in spirit until Iris's phone confirms it,
|
|
since only she can say whether it *feels* like a fling now; the
|
|
emulator's screenshot timing could not always catch the tail of a
|
|
fast-settling one visually (same caveat noted in RUST.md).
|
|
- [~] **"Text still disappears if I leave and come back to the app."**
|
|
**Instrumented 2026-09-06 so the phone can answer it**, since no
|
|
emulator here has a Vulkan adapter. `iris/src/android/view.rs` now logs
|
|
one `log::info!` line per surface event with the glyph/atlas counts:
|
|
`iris surface: surface_destroyed, tearing the renderer down
|
|
(glyphs_cached=387 atlas_pages=1)`, `iris surface: surface_changed
|
|
1080x2424 already_live=false glyphs_cached=387 atlas_pages=1`, `iris
|
|
surface: new renderer built (Gl), clearing glyph atlas: glyphs=387
|
|
pages=1`, plus `iris insets: ... window=(1080, 2424)` on every insets
|
|
change. That is the emulator's own healthy app-switch cycle, verified
|
|
this pass (home, reopen, screenshot: all text intact,
|
|
`/tmp/appswitch.png`). **The one line to look for on the phone is
|
|
`already_live=`**: `true` on the return from backgrounding would mean
|
|
the surface came back *without* a `surface_destroyed`, so
|
|
`surface_changed` reconfigured a renderer whose Vulkan swapchain and
|
|
atlas textures belong to a window that is gone -- the reuse branch
|
|
never clears the atlas, by design. `false` with no `new renderer built`
|
|
line after it would mean the renderer failed to rebuild. Either answer
|
|
names the fix; guessing between them from here does not.
|
|
The `GlyphAtlas::clear`/`Textures::reset` fix was verified on the
|
|
emulator under `force-gles` only; the phone runs Vulkan. So either the
|
|
reset is not reached on the phone's path (a different surface-
|
|
lifecycle sequence -- `surface_destroyed`/`surface_created` ordering,
|
|
or the renderer not being rebuilt but its textures lost), or the CPU
|
|
glyph cache and the GPU atlas still disagree after it. Needs logging
|
|
of the renderer lifecycle on the phone build, readable from `adb
|
|
logcat` when Iris next runs it, since no emulator here has a Vulkan
|
|
adapter under host GPU.
|
|
|
|
## Build
|
|
|
|
- [x] **Benchmarks**, not unit tests, run on demand (2026-09-05; a
|
|
`benches/` or a script under `iris/`, never in `cargo test`). The
|
|
scenario that matters most is a **message list** — chat apps and this
|
|
app's transcript alike — stressed with many messages and many images.
|
|
One case in particular: **resizing an input box** (typing enough text to
|
|
grow it) that pushes a long list of messages above it must stay very
|
|
fast and recalculate almost nothing — a move of everything above, not a
|
|
re-layout. That is exactly the O(1) move chain in LAYOUT.md; the
|
|
benchmark is what proves it. Done when the numbers are in this file with
|
|
the command, and the input-box case reports draws re-run, not just frame
|
|
time.
|
|
|
|
**Built as two rigs**, chosen per scenario by whether a real `wgpu`
|
|
device is needed (`UiRenderState`/`Widgets` touch no GPU or window, so
|
|
most of this runs as an ordinary binary — the same property
|
|
`layout_tests.rs` relies on):
|
|
|
|
- `iris/benches/message_list.rs` — a plain `Instant`-timed binary
|
|
(`[[bench]] harness = false` in `iris/Cargo.toml`), not criterion: see
|
|
the file's own header for why (short version — every scenario here
|
|
reduces to a *count* `UiRenderState::take_counters` already produces,
|
|
which criterion's statistical machinery adds nothing to and which a
|
|
new dependency is not worth pulling in for). Covers (a) first-frame
|
|
cost of a message list of N wrapped-text rows (one in 20 also carrying
|
|
a small in-memory image) for N = 100/1,000/10,000; (b) per-frame cost
|
|
of scrolling that list, 200 ticks; (c) the input-box case — a
|
|
fixed-height field at the bottom of the screen growing by a line 40
|
|
times, with the message list above it filling the rest of the screen.
|
|
Run: `cd iris && cargo bench --bench message_list` (always release —
|
|
`cargo bench` builds the `bench` profile, which is optimized).
|
|
- `iris/examples/bench_images.rs` — needs a real device, so it runs
|
|
through `iris/run-headless.sh bench_images`, printing
|
|
`UiRenderNode::take_image_bind_group_creates()` (a new counter, added
|
|
in `core/src/render/texture.rs` and `core/src/render/mod.rs`,
|
|
mirroring `UiRenderState::take_counters`) each frame. Covers (d): 1,000
|
|
image rows, checked both cold (does bind-group creation reach zero
|
|
once loaded) and after appending one more image once settled (does
|
|
*that* stay cheap) — the second question is what actually matters for
|
|
a live transcript and is what turned up the two Fix items above.
|
|
- `iris/run-bench.sh [list|images]` runs either or both and is what to
|
|
run before/after touching `Scroll`, `Span`, `Sized`, the move-offset
|
|
chain, or `GpuTextures`.
|
|
|
|
**Numbers (2026-09-05, release, `cargo bench`/`run-headless.sh`, this
|
|
VM: AMD Ryzen 7 3800X, 8 cores, rustc 1.98.0 nightly-2026-09-03):**
|
|
|
|
cd iris && cargo bench --bench message_list
|
|
(a) first frame, N=100: 30.30ms draws=227 rewrites=15 moves=0
|
|
(a) first frame, N=1000: 186.04ms draws=2252 rewrites=150 moves=0
|
|
(a) first frame, N=10000:1770.36ms draws=22502 rewrites=1500 moves=0
|
|
(b) scroll, N=100/1000/10000, 200 ticks each:
|
|
draws=200 rewrites=0 moves=200 (identical at every N)
|
|
per-tick average: 0.0002ms (identical at every N)
|
|
(c) input grows 40 lines, N=100/1000/10000 rows above it:
|
|
draws=320 rewrites=40 moves=160 (identical at every N)
|
|
per-line average: 0.0012-0.0013ms (identical at every N)
|
|
|
|
cd iris && ./run-bench.sh images (2026-09-05, before the fix)
|
|
frame=1 bind_group_creates=1000 (cold load)
|
|
frame=2 bind_group_creates=1000 (see Fix item above)
|
|
frame=3 bind_group_creates=0
|
|
frame=4 bind_group_creates=0
|
|
(append one image here)
|
|
frame=5 bind_group_creates=1001 (see Fix item above)
|
|
frame=6 bind_group_creates=0
|
|
|
|
cd iris && ./run-bench.sh images (2026-09-05, after the fix)
|
|
frame=1 bind_group_creates=1000 (cold load, unchanged -- genuine work)
|
|
frame=2 bind_group_creates=0
|
|
frame=3 bind_group_creates=0
|
|
frame=4 bind_group_creates=0
|
|
(append one image here)
|
|
frame=5 bind_group_creates=1 (one image's own create_image, O(1))
|
|
frame=6 bind_group_creates=0
|
|
|
|
**Reading it**: (a) is real, necessary work — shaping and laying out N
|
|
never-before-seen text rows — and scales with N as it must, ~10x cost
|
|
per 10x N. (b) and (c) are the pass conditions that matter: both are
|
|
**exactly flat across N = 100 to 10,000**, confirming LAYOUT.md's O(1)
|
|
move chain holds for both scrolling and for a growing input box pushing
|
|
the message list — draws/moves per tick or per line do not grow with
|
|
list size, and the per-operation cost (a fraction of a microsecond) is
|
|
nowhere near a frame budget. (d)'s cold-load and steady-state halves
|
|
behave as designed; its *append* half did not, until the fix above moved
|
|
masks/move_offsets out of the per-image bind group — now flat at O(1)
|
|
the same way (b) and (c) are.
|
|
|
|
- **I5's transcript screen (`iris/transcript-ui/`, 2026-09-05) — what it
|
|
left, each recorded at the point in the code it would go rather than
|
|
silently dropped. See RUST.md's I5 box for the full account of what
|
|
*was* built (the screen, `SpanStyle`, cross-row selection, the growing
|
|
composer).**
|
|
- [x] **Android integration for this screen — done, 2026-09-05.**
|
|
`iris-android-app`'s `transcript-screen` Cargo feature
|
|
(`transcript_client.rs`) runs this screen against a real `ai-server`
|
|
through `client-core`, confirmed on-device: real scrolling, real
|
|
touch-drag panning, tap-by-name on the composer. Two real bugs found
|
|
and fixed along the way (a missing `INTERNET` permission; a
|
|
background-thread redraw request that crashed via a `Looper`
|
|
requirement, fixed by routing through `View::post_delayed` — see
|
|
`IRIS.md`'s `Tasks::redraw_handle` entry). See RUST.md's I5 box,
|
|
"The Android integration, done 2026-09-05" for the full account.
|
|
- [x] **A render-time number for iris, comparable to Compose's
|
|
`transcript-bench.sh` report — instrumentation done and a real number
|
|
obtained, 2026-09-05 (later the same day); the clean comparable loop
|
|
is not.** `iris_core::FrameReport` (`iris/core/src/render/
|
|
frame_report.rs`, `IRIS.md`'s new entry) times every frame from
|
|
`render()`'s redraw start to after `queue.submit`+`present()`, exposed
|
|
as two named on-screen controls ("Frame report", "Reset frame
|
|
report"). Driven against a real on-device touch-drag it read
|
|
`frames=34 janky%=61.76 p50=26.5ms p90=48.0ms p99=98.1ms
|
|
worst=98.1ms` — real, not inferred, but accumulated across several
|
|
gestures rather than one clean 24-swipe loop, because of the new
|
|
finding below. See RUST.md's I5 box, "Update, 2026-09-05, later the
|
|
same day" for the full account.
|
|
- [ ] **New, 2026-09-05: intermittent touch delivery to iris's
|
|
`SurfaceView` under this checkout's `EMU_GPU=software` emulator.**
|
|
The same swipe coordinates, confirmed (by scanning a screenshot
|
|
column for the first non-black pixel) to sit over real row text,
|
|
sometimes produced 30+ real frames and a screenshot diff and
|
|
sometimes produced zero of either, across otherwise-identical
|
|
`ui-trace` invocations. Not the already-understood "already at that
|
|
scroll edge" case (reproduced with content confirmed taller than the
|
|
viewport, in both directions). Leading candidate, not yet confirmed:
|
|
this checkout's emulator was independently observed at ~78% of one
|
|
CPU core, continuously, while idle on-screen — SwiftShader's software
|
|
rasterisation is CPU-bound by design, and a synthetic touch competing
|
|
with that load for delivery is plausible but unmeasured *during* a
|
|
failing gesture (the standing rule against diagnosing from
|
|
after-the-fact measurements applies here). Needs a sampler (load,
|
|
`dumpsys input`, a `-i 0` `ui-trace` capture) running while a failing
|
|
gesture is driven, and ideally a comparison under `-gpu host` (real
|
|
Vulkan) to see whether it is specific to software rendering. This is
|
|
what blocks the clean, comparable 24-swipe loop above.
|
|
- [x] **Long-press-then-drag-to-select — confirmed on-device, 2026-09-05
|
|
(later the same day).** `ui-trace` gained a `holddrag X1 Y1 X2 Y2
|
|
HOLD_MS MOVE_MS` action (`emulator-tools`, additive, extends the same
|
|
`MotionEvent`/`injectInputEvent` mechanism `swipe` already used):
|
|
press, hold past `LONG_PRESS`, move, release, as one continuous touch.
|
|
Driven against a real row (`holddrag 300 1850 300 2050 600 300`) it
|
|
produced `iris selection: begin at row ...` then a sequence of
|
|
`iris selection: extend to row ...` log lines
|
|
(`transcript-ui/src/selection.rs`, a new small `log` dependency since
|
|
selection has no accessibility label of its own yet — see the next
|
|
item), and a screenshot taken right after shows the expected
|
|
highlighted selection spanning multiple rows. `DragArbiter`'s own
|
|
unit tests already covered this sequence against a synthetic clock;
|
|
this is the first time it has been driven by a real device touch.
|
|
- [x] **Touch-drag panning over a row's own rendered text — done,
|
|
2026-09-05.** `row.rs` used to register `CursorSense::click_or_drag()`
|
|
on each row's `TextEdit` for cross-row selection; `TextEdit::draw`'s
|
|
`painter.child_layer()` (`iris/src/widget/text/edit.rs:87`) meant that
|
|
registration won `core/src/sense.rs::run_sensors`'s per-layer
|
|
arbitration on every frame it was pressed, not just the frame the
|
|
press started, so a list pan gesture registered on `List` itself never
|
|
got a turn while a row was under the finger. Fixed with
|
|
`iris::sense::DragArbiter` (recorded in `IRIS.md`), one small state
|
|
machine per list deciding pan vs. select the way Android does (a
|
|
vertical drag pans immediately; a stationary press held `LONG_PRESS`
|
|
(500ms) starts a selection which further drag extends; a horizontal
|
|
drag while something is already selected extends immediately) —
|
|
`transcript-ui/src/selection.rs`'s `Selection::drag` is the one place
|
|
every row's drag now routes through. 8 new unit tests
|
|
(`iris/src/sense.rs`'s `drag_arbiter_tests`); `cargo fmt/clippy/test
|
|
--workspace` and `cargo ndk` (both `iris` and `transcript-ui`) all
|
|
clean; `run-headless.sh` screenshot byte-identical to before the
|
|
change (38578 bytes). See RUST.md's I5 box, "Gap closed, 2026-09-05".
|
|
- [x] **Intermittent touch-scroll dropout — root-caused and fixed,
|
|
2026-09-05.** Not the coalesced-`ACTION_MOVE` hypothesis the earlier
|
|
pass suspected (ruled out): a gesture's `ACTION_DOWN` can land on a
|
|
row's own padding/gap or its header, which no `CursorSense` covers,
|
|
so `DragArbiter` never gets `press_start` and sits in `Idle`
|
|
(answers `Undecided` forever) for that whole gesture. Fixed via a new
|
|
`DragArbiter::is_idle()` that `Selection::drag`
|
|
(`transcript-ui/src/selection.rs`) checks to recover a missed press
|
|
on the next `Pressing` frame. Four new unit tests. See RUST.md's I5
|
|
box, "Touch-scroll dropout root-caused, 2026-09-05", for the trace and
|
|
what a peer session sharing this checkout's emulator mid-pass
|
|
prevented from being re-verified end-to-end (the aggregate
|
|
`iris-scroll.sh` three-run confirmation and a re-taken FrameReport
|
|
row) — a future pass should finish that once the emulator is free.
|
|
- [ ] **Row-level accessibility names.** The composer carries
|
|
`.label("Message")`; transcript rows do not carry a `.label()` of
|
|
their own yet, so `Widgets::named()` (I4) does not include them —
|
|
`row.rs`'s `build_text_row` is where one would go, keyed to something
|
|
stable per row (its sender + a short excerpt, matching what a screen
|
|
reader announcing a chat message would say).
|
|
- [ ] **A tappable link and a background chip behind inline code.**
|
|
Both need per-range glyph geometry that `TextEditCtx` does not expose
|
|
outside `iris::widget::text` (`edit.rs`'s `layout()` helper is
|
|
private) — see `markdown.rs`'s module doc for the exact shape the fix
|
|
would take (the same primitive `TextEdit::draw`'s own selection
|
|
highlight already uses internally,
|
|
`iris/src/widget/text/edit.rs:99`).
|
|
- [ ] **`Selection`'s anchor-row shortcut.** The row a drag started in
|
|
is selected in full (`select_all`) the moment the drag leaves it,
|
|
rather than "from the click point to whichever edge points away from
|
|
the drag" — needs the same private `layout()` access as the item
|
|
above. `selection.rs`'s module doc has the exact reasoning.
|
|
- [ ] **No syntax highlighting inside a fenced code block.**
|
|
`client_core::highlight` exists (built for the file explorer) and
|
|
could feed per-token `SpanStyle`s into a code block's span; wiring it
|
|
in was not attempted this pass.
|
|
|
|
- [ ] **Masks defined relative to each other.** Wanted: mask A multiplies
|
|
by something *and also* applies mask B — a mask can reference a parent
|
|
mask, the way the move chain references a parent offset. Today masks
|
|
are independent regions. Design it beside the move chain (same shape:
|
|
a parent index and a bounded walk in the shader); do it when a real
|
|
widget needs it, not before.
|
|
- [ ] **Positions as a single float per scroll.** Iris raised, and half
|
|
rejected, letting a scroll update one float rather than positions:
|
|
input handling cares about most elements in a list, so absolute
|
|
positions must be computed on the CPU anyway. LAYOUT.md's design
|
|
already lands here (GPU walks the chain, CPU resolves on demand for
|
|
hit tests). Keep the CPU resolution lazy and per query; do not
|
|
materialise every row's absolute position per frame.
|
|
- [ ] **Animations, last.** Cosmetic, so after everything above. Must be
|
|
**modular — a piece of the library rather than a core part forced into
|
|
everything, the same way input is**. Whatever the mechanism, a widget
|
|
that does not animate must pay nothing and import nothing for it.
|
|
|
|
## Build (for the port)
|
|
|
|
Widgets `RUST.md`'s "The port, in order (decided 2026-09-05)" needs and
|
|
iris does not have yet, one entry per gap, named against the P-step that
|
|
first needs it. Move an entry up to "Fix" or tick it in place once built;
|
|
do not duplicate it there.
|
|
|
|
- [ ] **A history-paging cushion measured in on-screen viewports, not a
|
|
row count.** (**P1**.) `iris::widget::List` has no equivalent of the
|
|
Compose app's `HISTORY_SCREENS` — AGENTS.md's "Things that have
|
|
bitten" is explicit that a fixed row count under-fills a screen on a
|
|
tool-heavy transcript and over-fills one on a text-heavy one, so
|
|
whatever loads the next page has to ask the list how many viewports
|
|
are actually on screen, not assume a constant.
|
|
- [ ] **A scaled thumbnail/image widget for an in-transcript image.**
|
|
(**P1**.) `SessionImage.kt`'s bitmap decode-and-downscale has no iris
|
|
counterpart; iris's own image widget (used by `bench_images.rs`) draws
|
|
a loaded texture but does nothing about sourcing or scaling one from a
|
|
server-produced attachment.
|
|
- [ ] **A modal/dialog primitive.** (**P1**, reused by **P3** and
|
|
**P5**.) Needed for the session settings dialog, `UsageDialog`'s
|
|
equivalent, and the delete-with-`deleteForeign` confirmation with its
|
|
toggle switch. Build once, wherever it is first needed, rather than
|
|
once per screen that wants one.
|
|
- [ ] **A horizontal gauge/bar widget.** (**P1**.) For
|
|
`SessionUsageBar`'s equivalent — a bounded fill reflecting a fraction,
|
|
nothing fancier.
|
|
- [ ] **A `BusyItem` equivalent: a dimmed row carrying an operation
|
|
label that does not block its list's own scroll/drag.** (**P3**.) The
|
|
Compose version tried an overlay first and it swallowed the drag along
|
|
with the tap (AGENTS.md's "Shared appearance") — worth not repeating
|
|
that attempt in iris before building the row-level version directly.
|
|
- [ ] **A toggle switch.** (**P3**.) For the delete dialog's
|
|
`deleteForeign` control; iris has no switch/checkbox widget yet as far
|
|
as this pass found.
|
|
|
|
## Reconsider
|
|
|
|
- [ ] **`WidgetView`.** Iris is unsure of it: what she wants is an easy way
|
|
to compose a widget from others (a button is the main case). With
|
|
sizing folded into `draw`, composing may be easy enough that `View` is
|
|
redundant. Decide after the layout change lands, by writing a button
|
|
both ways and keeping the one that is shorter to explain; delete the
|
|
other rather than keeping two ways.
|
|
|
|
## Build (asked for by Iris, 2026-09-06): a density-independent length unit
|
|
|
|
- [x] **A third length kind beside relative and pixels, so display scales
|
|
"just work".** Done 2026-09-06 — `Len::dp`/`len_fns::dp`, resolved
|
|
against `UiRenderState`/`Painter::density()` at `apply_rest` time; text
|
|
additionally rasterises at the resolved (physical) size instead of
|
|
scaling a low-resolution bitmap afterward, which was making text blurry.
|
|
`Span::gap`/`Padding` moved from `f32` to `Len` so they take `dp(...)`
|
|
too; transcript-ui's row/composer padding and one example migrated.
|
|
`em` was not added — nothing in this pass needed a text-relative unit,
|
|
and `dp`'s own doc says why it and physical pixels are kept as separate
|
|
fields rather than one the caller pre-multiplies. Not yet verified on
|
|
Iris's own phone at two densities (this pass had no device) — see
|
|
docs/RUST.md's P0 box and docs/IRIS.md's 2026-09-06 entry for what to
|
|
check. Iris's words: "another length type similar to absolute &
|
|
relative, so instead there would be relative, pixels, and another unit
|
|
like em or whatever is standard. That way different display scales
|
|
should just work." Today a length is either a fraction of the parent
|
|
(`rest`/relative) or physical pixels, and the phone drew 16 px text at
|
|
roughly a third of its intended size until the P0 fixes applied the
|
|
display's scale factor globally. That global scale is a stopgap for the
|
|
benchmark; the real shape is a unit resolved against the display's
|
|
density at layout time — Android's `dp` / CSS's reference pixel is the
|
|
standard (1 unit = 1/160 in), with `em` as the text-relative option —
|
|
so a widget author writes `16.dp()` once and never sees the scale.
|
|
Done when: `Length` (or whatever the enum is called) has the third
|
|
variant; every place that resolves a length takes the density; the
|
|
examples and `transcript-ui` use the new unit for text sizes, padding
|
|
and control sizes; the emulator at two densities and the phone draw the
|
|
same layout at the same physical size. After the bench setup is
|
|
finished, before P1 draws any new screen.
|
|
|
|
## From the phone, bench v2 (2026-09-06): streaming re-lays out the whole message
|
|
|
|
- [ ] **Streaming a delta into a long message costs a full text layout of
|
|
that message.** Iris's phone report (`docs/bench/iris-phone-v2-2026-09-06.md`):
|
|
the stream phase is the one place iris is behind Compose (p50 18.2 ms vs
|
|
13.4 ms; p99 level at ~43 ms). `TranscriptScreen::apply` replaces only
|
|
the last row, but that row is the growing message, and replacing it
|
|
re-renders its markdown and re-shapes the entire paragraph run through
|
|
parley on every event. Compose pays a reparse (8.6 ms mean) for the
|
|
same event. What "done" looks like: a streamed delta re-lays out only
|
|
the block it lands in (the last paragraph or code block), with earlier
|
|
blocks' layouts kept -- which needs a row to be a column of per-block
|
|
`Text`s rather than one `TextEdit` for the whole message, or parley's
|
|
layout to be split at block boundaries; measured by the stream phase's
|
|
p50 dropping below Compose's on the phone. Do this after the four bench
|
|
v2 defects (stale primitives, finger fling, decay curve, IME show) are
|
|
closed, since they are what make the run unrepresentative today.
|