TEXTURES.md: how iris should render an unbounded number of images
Written for review before iris's render core changes. Covers the bindless binding-array problem, the gpu-probe measurements (emulator and sourced real-hardware findings), what growth already costs today in the current code, the egui_wgpu/Vello prior art, and the recommendation with its open questions -- not yet implemented. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Sonnet 5 committed
1 parent
79b9cd789a
commit
6e49ce8c92
1 file changed
+215
+215
@@ -0,0 +1,215 @@
|
||||
# How iris should render an unbounded number of images
|
||||
|
||||
Iris (the person) asked whether iris's (the library's) approach to
|
||||
"draw however many images happen to be on screen" — relevant here because a
|
||||
transcript can hold an unbounded number of attached screenshots — actually
|
||||
works on mobile, her recollection being that it does not. Checked rather
|
||||
than assumed, on 2026-09-04, on the `rustify` branch of `ai-app-2`. This
|
||||
file is that investigation and the resulting recommendation, written for a
|
||||
second agent to review before anything in iris's render core changes — no
|
||||
code has been written against this yet.
|
||||
|
||||
## The problem
|
||||
|
||||
Every texture iris ever creates — every `Image` widget
|
||||
(`iris/src/widget/image.rs`) and every glyph atlas page — gets a permanent
|
||||
slot in one array via `Textures::add` (`iris/core/src/primitive/texture.rs:65`).
|
||||
Both of iris's texture-sampling primitives (`TEXTURE` and `GLYPH`) read that
|
||||
array by index: `core/src/render/shader.wgsl:56` declares
|
||||
`var views: binding_array<texture_2d<f32>>`, sized by
|
||||
`UiLimits::default()` (`core/src/render/mod.rs:347`) at **100,000 textures,
|
||||
1,000 samplers**. Getting a device to accept that layout needs three wgpu
|
||||
features — `TEXTURE_BINDING_ARRAY`,
|
||||
`SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING`,
|
||||
`PARTIALLY_BOUND_BINDING_ARRAY` — which correspond to Vulkan's
|
||||
`VK_EXT_descriptor_indexing` ("bindless"), promoted to Vulkan core at 1.2.
|
||||
|
||||
A transcript with an unbounded number of image attachments is exactly the
|
||||
case that grows this array without bound: each attachment becomes its own
|
||||
`Image` widget, which takes its own permanent array slot until dropped.
|
||||
|
||||
## What was measured
|
||||
|
||||
**A new rig, `rigs/gpu-probe`**, asks a device for exactly iris's features
|
||||
and limits with no window and no APK — a plain executable pushed with
|
||||
`adb push` and run from `/data/local/tmp`. It has two parts:
|
||||
`wgpu::Adapter::request_device` with iris's exact `Features`/`Limits`
|
||||
(`src/main.rs`), and a raw Vulkan query bypassing wgpu entirely via `ash`
|
||||
(`src/vk.rs`), to tell "the driver doesn't have it" apart from "wgpu didn't
|
||||
detect it."
|
||||
|
||||
- **On this VM's own GPU** (Vulkan via Venus onto an RX 7900 XT):
|
||||
`IRIS DEVICE: ok`. Not the case that matters — nobody's phone is a
|
||||
discrete desktop GPU — but it is why the design was never checked before
|
||||
now: it always worked in the one place it was tried.
|
||||
- **On the Android emulator's guest Vulkan**, both ICDs it ships
|
||||
(`vk_swiftshader_icd.json` and, cold-booted, `lvp_icd.json`/lavapipe):
|
||||
`request_device` **fails** —
|
||||
`Unsupported features were requested: TEXTURE_BINDING_ARRAY |
|
||||
SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING |
|
||||
PARTIALLY_BOUND_BINDING_ARRAY`. The raw `ash` query on lavapipe shows the
|
||||
driver itself reporting all seven descriptor-indexing sub-features as
|
||||
`true` at device API version 1.3 — so wgpu-hal's own feature detection is
|
||||
being more conservative than the driver here, for a reason not chased
|
||||
further (a likely instance-version negotiation gap, since the extension
|
||||
only promoted to core at 1.2). That part is a wgpu-hal/emulator question,
|
||||
not the finding that matters, and is **not** why this design is rejected.
|
||||
|
||||
**The finding that matters is about real phones, sourced rather than
|
||||
recalled:**
|
||||
|
||||
- The **Android Vulkan Profile 2025** — Google and Khronos's current
|
||||
baseline, covering **80.1% of active Vulkan-capable Android devices** as
|
||||
of October 2025
|
||||
([developer.android.com/ndk/guides/graphics/android-vulkan-profile](https://developer.android.com/ndk/guides/graphics/android-vulkan-profile)) —
|
||||
does **not** require `VK_EXT_descriptor_indexing` or any descriptor-
|
||||
indexing feature. It requires `shaderSampledImageArrayDynamicIndexing`
|
||||
(indexing by a value uniform across the invocation — Vulkan 1.0 baseline,
|
||||
unrelated to bindless) and stops there; true of the 2021 and 2022
|
||||
profiles as well.
|
||||
- Arm's own developer documentation states **"`VK_EXT_descriptor_indexing`
|
||||
is supported on all Valhall and 5th Gen GPUs"**
|
||||
([developer.arm.com/mobile-graphics-and-gaming/vulkan-api-best-practices-on-arm-gpus](https://developer.arm.com/mobile-graphics-and-gaming/vulkan-api-best-practices-on-arm-gpus)) —
|
||||
Mali generations from roughly 2019 (Mali-G77) onward, with no claim made
|
||||
for Bifrost, Midgard or Utgard, which are still common in budget and
|
||||
older Android phones that are still in daily use.
|
||||
- A search engine's summarized claim of "1% support on Android" for this
|
||||
extension was checked against its cited source (an Arm blog post from
|
||||
2021) and **was not actually there** — that number does not appear in
|
||||
any primary source found and should not be repeated. The baseline-
|
||||
profile finding above is the one with an attributable source; use it
|
||||
instead.
|
||||
|
||||
So this is not a software-renderer artifact. A real, currently-shipping
|
||||
share of the Android fleet lacks the feature iris's texture pipeline asks
|
||||
for unconditionally, and neither the emulator's failure nor the current
|
||||
official hardware baseline gives any reason to expect that to change soon.
|
||||
|
||||
## What growth already costs today, before any redesign
|
||||
|
||||
Checked directly in `core/src/render/mod.rs` and `core/src/render/texture.rs`,
|
||||
because "does this redesign make things worse" needs the current baseline
|
||||
first:
|
||||
|
||||
- The `RenderPipeline` (`UiRenderNode::new`) is created **once** and never
|
||||
rebuilt for any reason related to texture count — its bind group
|
||||
*layouts* declare fixed slot counts (`limits.max_textures`,
|
||||
`limits.max_samplers`) up front and that never changes at runtime. Growth
|
||||
was never at risk of recreating the pipeline, in the current design or
|
||||
any redesign discussed below.
|
||||
- What **does** get rebuilt: `UiRenderNode::update` calls
|
||||
`self.textures.update(&mut ui.textures)`, and if that reports any change,
|
||||
rebuilds `self.rsc_group` — one `BindGroup` whose entries are
|
||||
`BindingResource::TextureViewArray(&tex_manager.views())`, collected
|
||||
fresh over **every currently-live texture**, plus the sampler array and
|
||||
the mask buffer. This happens on every texture `Push`, `Set`, or `Free`
|
||||
— an image added anywhere in the whole app rebuilds one shared structure
|
||||
referencing every other image too.
|
||||
- The one path already excluded from this, on purpose, is a `Patch` —
|
||||
writing into an existing texture's pixels without changing which
|
||||
textures exist. The code says why directly
|
||||
(`core/src/render/texture.rs`, in `GpuTextures::update`): *"A patch
|
||||
changes texture contents, not the binding array, so it must not report
|
||||
`changed` — rebuilding the bind group per glyph is the cost this exists
|
||||
to avoid."* This is exactly the mechanism I1 built for the glyph atlas:
|
||||
growing an existing atlas page costs a `write_texture` into a sub-rect,
|
||||
nothing else.
|
||||
|
||||
So today, growth that stays inside an existing texture (glyphs added to an
|
||||
atlas page) is already free. Growth that adds a *new* texture — a new atlas
|
||||
page, or any standalone image — already rebuilds the one shared array
|
||||
regardless of how the array is populated, before any change discussed
|
||||
below. That existing cost is O(live texture count) in CPU work to collect
|
||||
the view list and in however expensive the driver finds a
|
||||
descriptor-set-sized-for-N-descriptors to be.
|
||||
|
||||
## Prior art, checked rather than assumed
|
||||
|
||||
Two independent projects were checked to see whether "atlas for images"
|
||||
is actually how this is normally done, rather than a guess:
|
||||
|
||||
- **egui_wgpu** (`crates/egui-wgpu/src/renderer.rs` in emilk/egui), the
|
||||
closest prior art to iris — an immediate-mode wgpu-backed UI library that
|
||||
ships on Android. It keeps a `HashMap<TextureId, Texture>` and gives
|
||||
**each texture its own ordinary `BindGroup`** — one texture, one sampler,
|
||||
no array, no descriptor indexing of any kind. Draw calls are batched by
|
||||
texture id and the bind group is switched between batches within the
|
||||
render pass.
|
||||
- **Vello** — the renderer Masonry (E1/E2's Linebender stack) draws
|
||||
through — hit the identical problem and wrote down why in their own
|
||||
roadmap document
|
||||
([github.com/linebender/vello/blob/main/doc/roadmap_2023.md](https://github.com/linebender/vello/blob/main/doc/roadmap_2023.md)):
|
||||
*"The number of images that may appear in a scene is not bounded, which
|
||||
is not a good fit for the basic descriptor binding model... Until then,
|
||||
we'll do a workaround of having a single atlas image containing all the
|
||||
images in the scene."* Their reason is broader than Android — WebGPU 1.0
|
||||
has no descriptor indexing at all — but it reaches the same conclusion
|
||||
for the same shape of problem: atlas, not a bigger bindless array.
|
||||
|
||||
**This is also a live hazard, not a solved one.** Vello's own changelog
|
||||
(Sparse Strips v0.2.0) lists a fix titled *"WebGL image-atlas allocation
|
||||
and growth on Mali-G52 GPUs, avoiding application-not-responding errors"*
|
||||
— an actual ANR, from atlas growth, on an actual mid-range Android GPU,
|
||||
in the renderer Masonry is built on. The same release added
|
||||
`AtlasSpaceDiagnostics`/`AtlasLayerDiagnostics` (per-layer free-space,
|
||||
utilization, fragmentation) because growth needed instrumenting in
|
||||
production, not because it turned out to be free.
|
||||
|
||||
## Recommendation (not yet implemented)
|
||||
|
||||
1. **Small, plentiful textures** — glyphs (already done, I1), thumbnails,
|
||||
downscaled attachment previews, icons — go through a shared atlas, the
|
||||
same technique as `core/src/render/atlas.rs` generalized beyond glyphs.
|
||||
Adding one to an existing page is a `Patch`, already free per the
|
||||
section above.
|
||||
2. **Large or one-off images** — a photo attachment opened at full
|
||||
resolution, anything that would fragment a shared page — get their
|
||||
**own ordinary, non-array bind group**, the egui_wgpu way. Creating one
|
||||
is O(1): it references only itself, and does not touch any other
|
||||
texture's binding, unlike today's shared array where every push
|
||||
rebuilds a structure listing everything.
|
||||
3. **Opening a new atlas page** is the one case that still resembles
|
||||
today's rebuild — infrequent (bounded by how many *pages* are needed,
|
||||
not by how many images have ever been attached) but not free, and
|
||||
Vello's Mali-G52 fix says this specifically deserves care: it should
|
||||
never be allowed to block a frame, and it is worth having the
|
||||
equivalent of Vello's atlas diagnostics before trusting it under load.
|
||||
4. **Net effect**: dropping `TEXTURE_BINDING_ARRAY`,
|
||||
`SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING`, and
|
||||
`PARTIALLY_BOUND_BINDING_ARRAY` from iris's device request entirely.
|
||||
Every path above is plain Vulkan 1.0 / GLES-level texture sampling.
|
||||
This is also what fixes the emulator failure measured above, regardless
|
||||
of the unresolved wgpu-hal question: a device that never asks for the
|
||||
feature cannot be refused for lacking it.
|
||||
|
||||
## What this touches, and what is still open
|
||||
|
||||
Implementing this reworks iris's rendering core: the shader's binding
|
||||
group layout (`shader.wgsl`), `Textures` and `GpuTextures`
|
||||
(`core/src/primitive/texture.rs`, `core/src/render/texture.rs`), both
|
||||
texture-sampling primitives, and `core/src/ui/painter.rs`'s draw-call
|
||||
batching (today one draw call can reference any texture by index; the
|
||||
per-texture-bind-group path needs draws grouped by which bind group they
|
||||
use). Nothing has been started.
|
||||
|
||||
Open questions a reviewer should weigh in on:
|
||||
|
||||
- **The size threshold** between "goes in an atlas page" and "gets its own
|
||||
bind group." Too low and ordinary attachment thumbnails end up as
|
||||
one-off bind groups, losing the batching benefit the atlas exists for;
|
||||
too high and a page fragments on a handful of medium images.
|
||||
- **Eviction policy** for atlas pages once the working set does not fit —
|
||||
today's `GlyphAtlas` never evicts, because a font's glyph set is small
|
||||
and bounded; images are not. An LRU at the page level, or at the
|
||||
individual-image level within a page, has not been designed.
|
||||
- **Whether iris should keep any binding array at all**, even a small
|
||||
fixed one (say, capped at a few dozen slots) for atlas pages themselves,
|
||||
or whether every atlas page should also be its own ordinary bind group
|
||||
like standalone images — the array's only remaining justification would
|
||||
be avoiding a bind-group-per-draw-call switch cost that has not been
|
||||
measured on this project's actual target hardware.
|
||||
- **How this interacts with I2/E2's virtualised list** (I3): a
|
||||
bottom-anchored transcript composes only visible rows, so the live
|
||||
texture set should already be bounded by what is on screen rather than
|
||||
by the whole conversation — worth confirming that invariant holds before
|
||||
relying on it to keep atlas/bind-group churn small.
|
||||
Reference in new issue
Block a user