From 6e49ce8c925dfafda3b73e7c28dd719a8c7de021 Mon Sep 17 00:00:00 2001 From: iris <2+iris@noreply.localhost> Date: Fri, 4 Sep 2026 21:36:56 -0400 Subject: [PATCH] TEXTURES.md: how iris should render an unbounded number of images Written for review before iris's render core changes. Covers the bindless binding-array problem, the gpu-probe measurements (emulator and sourced real-hardware findings), what growth already costs today in the current code, the egui_wgpu/Vello prior art, and the recommendation with its open questions -- not yet implemented. Co-Authored-By: Claude Sonnet 5 --- TEXTURES.md | 215 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 215 insertions(+) create mode 100644 TEXTURES.md diff --git a/TEXTURES.md b/TEXTURES.md new file mode 100644 index 0000000..1f9448d --- /dev/null +++ b/TEXTURES.md @@ -0,0 +1,215 @@ +# How iris should render an unbounded number of images + +Iris (the person) asked whether iris's (the library's) approach to +"draw however many images happen to be on screen" — relevant here because a +transcript can hold an unbounded number of attached screenshots — actually +works on mobile, her recollection being that it does not. Checked rather +than assumed, on 2026-09-04, on the `rustify` branch of `ai-app-2`. This +file is that investigation and the resulting recommendation, written for a +second agent to review before anything in iris's render core changes — no +code has been written against this yet. + +## The problem + +Every texture iris ever creates — every `Image` widget +(`iris/src/widget/image.rs`) and every glyph atlas page — gets a permanent +slot in one array via `Textures::add` (`iris/core/src/primitive/texture.rs:65`). +Both of iris's texture-sampling primitives (`TEXTURE` and `GLYPH`) read that +array by index: `core/src/render/shader.wgsl:56` declares +`var views: binding_array>`, sized by +`UiLimits::default()` (`core/src/render/mod.rs:347`) at **100,000 textures, +1,000 samplers**. Getting a device to accept that layout needs three wgpu +features — `TEXTURE_BINDING_ARRAY`, +`SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING`, +`PARTIALLY_BOUND_BINDING_ARRAY` — which correspond to Vulkan's +`VK_EXT_descriptor_indexing` ("bindless"), promoted to Vulkan core at 1.2. + +A transcript with an unbounded number of image attachments is exactly the +case that grows this array without bound: each attachment becomes its own +`Image` widget, which takes its own permanent array slot until dropped. + +## What was measured + +**A new rig, `rigs/gpu-probe`**, asks a device for exactly iris's features +and limits with no window and no APK — a plain executable pushed with +`adb push` and run from `/data/local/tmp`. It has two parts: +`wgpu::Adapter::request_device` with iris's exact `Features`/`Limits` +(`src/main.rs`), and a raw Vulkan query bypassing wgpu entirely via `ash` +(`src/vk.rs`), to tell "the driver doesn't have it" apart from "wgpu didn't +detect it." + +- **On this VM's own GPU** (Vulkan via Venus onto an RX 7900 XT): + `IRIS DEVICE: ok`. Not the case that matters — nobody's phone is a + discrete desktop GPU — but it is why the design was never checked before + now: it always worked in the one place it was tried. +- **On the Android emulator's guest Vulkan**, both ICDs it ships + (`vk_swiftshader_icd.json` and, cold-booted, `lvp_icd.json`/lavapipe): + `request_device` **fails** — + `Unsupported features were requested: TEXTURE_BINDING_ARRAY | + SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING | + PARTIALLY_BOUND_BINDING_ARRAY`. The raw `ash` query on lavapipe shows the + driver itself reporting all seven descriptor-indexing sub-features as + `true` at device API version 1.3 — so wgpu-hal's own feature detection is + being more conservative than the driver here, for a reason not chased + further (a likely instance-version negotiation gap, since the extension + only promoted to core at 1.2). That part is a wgpu-hal/emulator question, + not the finding that matters, and is **not** why this design is rejected. + +**The finding that matters is about real phones, sourced rather than +recalled:** + +- The **Android Vulkan Profile 2025** — Google and Khronos's current + baseline, covering **80.1% of active Vulkan-capable Android devices** as + of October 2025 + ([developer.android.com/ndk/guides/graphics/android-vulkan-profile](https://developer.android.com/ndk/guides/graphics/android-vulkan-profile)) — + does **not** require `VK_EXT_descriptor_indexing` or any descriptor- + indexing feature. It requires `shaderSampledImageArrayDynamicIndexing` + (indexing by a value uniform across the invocation — Vulkan 1.0 baseline, + unrelated to bindless) and stops there; true of the 2021 and 2022 + profiles as well. +- Arm's own developer documentation states **"`VK_EXT_descriptor_indexing` + is supported on all Valhall and 5th Gen GPUs"** + ([developer.arm.com/mobile-graphics-and-gaming/vulkan-api-best-practices-on-arm-gpus](https://developer.arm.com/mobile-graphics-and-gaming/vulkan-api-best-practices-on-arm-gpus)) — + Mali generations from roughly 2019 (Mali-G77) onward, with no claim made + for Bifrost, Midgard or Utgard, which are still common in budget and + older Android phones that are still in daily use. +- A search engine's summarized claim of "1% support on Android" for this + extension was checked against its cited source (an Arm blog post from + 2021) and **was not actually there** — that number does not appear in + any primary source found and should not be repeated. The baseline- + profile finding above is the one with an attributable source; use it + instead. + +So this is not a software-renderer artifact. A real, currently-shipping +share of the Android fleet lacks the feature iris's texture pipeline asks +for unconditionally, and neither the emulator's failure nor the current +official hardware baseline gives any reason to expect that to change soon. + +## What growth already costs today, before any redesign + +Checked directly in `core/src/render/mod.rs` and `core/src/render/texture.rs`, +because "does this redesign make things worse" needs the current baseline +first: + +- The `RenderPipeline` (`UiRenderNode::new`) is created **once** and never + rebuilt for any reason related to texture count — its bind group + *layouts* declare fixed slot counts (`limits.max_textures`, + `limits.max_samplers`) up front and that never changes at runtime. Growth + was never at risk of recreating the pipeline, in the current design or + any redesign discussed below. +- What **does** get rebuilt: `UiRenderNode::update` calls + `self.textures.update(&mut ui.textures)`, and if that reports any change, + rebuilds `self.rsc_group` — one `BindGroup` whose entries are + `BindingResource::TextureViewArray(&tex_manager.views())`, collected + fresh over **every currently-live texture**, plus the sampler array and + the mask buffer. This happens on every texture `Push`, `Set`, or `Free` + — an image added anywhere in the whole app rebuilds one shared structure + referencing every other image too. +- The one path already excluded from this, on purpose, is a `Patch` — + writing into an existing texture's pixels without changing which + textures exist. The code says why directly + (`core/src/render/texture.rs`, in `GpuTextures::update`): *"A patch + changes texture contents, not the binding array, so it must not report + `changed` — rebuilding the bind group per glyph is the cost this exists + to avoid."* This is exactly the mechanism I1 built for the glyph atlas: + growing an existing atlas page costs a `write_texture` into a sub-rect, + nothing else. + +So today, growth that stays inside an existing texture (glyphs added to an +atlas page) is already free. Growth that adds a *new* texture — a new atlas +page, or any standalone image — already rebuilds the one shared array +regardless of how the array is populated, before any change discussed +below. That existing cost is O(live texture count) in CPU work to collect +the view list and in however expensive the driver finds a +descriptor-set-sized-for-N-descriptors to be. + +## Prior art, checked rather than assumed + +Two independent projects were checked to see whether "atlas for images" +is actually how this is normally done, rather than a guess: + +- **egui_wgpu** (`crates/egui-wgpu/src/renderer.rs` in emilk/egui), the + closest prior art to iris — an immediate-mode wgpu-backed UI library that + ships on Android. It keeps a `HashMap` and gives + **each texture its own ordinary `BindGroup`** — one texture, one sampler, + no array, no descriptor indexing of any kind. Draw calls are batched by + texture id and the bind group is switched between batches within the + render pass. +- **Vello** — the renderer Masonry (E1/E2's Linebender stack) draws + through — hit the identical problem and wrote down why in their own + roadmap document + ([github.com/linebender/vello/blob/main/doc/roadmap_2023.md](https://github.com/linebender/vello/blob/main/doc/roadmap_2023.md)): + *"The number of images that may appear in a scene is not bounded, which + is not a good fit for the basic descriptor binding model... Until then, + we'll do a workaround of having a single atlas image containing all the + images in the scene."* Their reason is broader than Android — WebGPU 1.0 + has no descriptor indexing at all — but it reaches the same conclusion + for the same shape of problem: atlas, not a bigger bindless array. + +**This is also a live hazard, not a solved one.** Vello's own changelog +(Sparse Strips v0.2.0) lists a fix titled *"WebGL image-atlas allocation +and growth on Mali-G52 GPUs, avoiding application-not-responding errors"* +— an actual ANR, from atlas growth, on an actual mid-range Android GPU, +in the renderer Masonry is built on. The same release added +`AtlasSpaceDiagnostics`/`AtlasLayerDiagnostics` (per-layer free-space, +utilization, fragmentation) because growth needed instrumenting in +production, not because it turned out to be free. + +## Recommendation (not yet implemented) + +1. **Small, plentiful textures** — glyphs (already done, I1), thumbnails, + downscaled attachment previews, icons — go through a shared atlas, the + same technique as `core/src/render/atlas.rs` generalized beyond glyphs. + Adding one to an existing page is a `Patch`, already free per the + section above. +2. **Large or one-off images** — a photo attachment opened at full + resolution, anything that would fragment a shared page — get their + **own ordinary, non-array bind group**, the egui_wgpu way. Creating one + is O(1): it references only itself, and does not touch any other + texture's binding, unlike today's shared array where every push + rebuilds a structure listing everything. +3. **Opening a new atlas page** is the one case that still resembles + today's rebuild — infrequent (bounded by how many *pages* are needed, + not by how many images have ever been attached) but not free, and + Vello's Mali-G52 fix says this specifically deserves care: it should + never be allowed to block a frame, and it is worth having the + equivalent of Vello's atlas diagnostics before trusting it under load. +4. **Net effect**: dropping `TEXTURE_BINDING_ARRAY`, + `SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING`, and + `PARTIALLY_BOUND_BINDING_ARRAY` from iris's device request entirely. + Every path above is plain Vulkan 1.0 / GLES-level texture sampling. + This is also what fixes the emulator failure measured above, regardless + of the unresolved wgpu-hal question: a device that never asks for the + feature cannot be refused for lacking it. + +## What this touches, and what is still open + +Implementing this reworks iris's rendering core: the shader's binding +group layout (`shader.wgsl`), `Textures` and `GpuTextures` +(`core/src/primitive/texture.rs`, `core/src/render/texture.rs`), both +texture-sampling primitives, and `core/src/ui/painter.rs`'s draw-call +batching (today one draw call can reference any texture by index; the +per-texture-bind-group path needs draws grouped by which bind group they +use). Nothing has been started. + +Open questions a reviewer should weigh in on: + +- **The size threshold** between "goes in an atlas page" and "gets its own + bind group." Too low and ordinary attachment thumbnails end up as + one-off bind groups, losing the batching benefit the atlas exists for; + too high and a page fragments on a handful of medium images. +- **Eviction policy** for atlas pages once the working set does not fit — + today's `GlyphAtlas` never evicts, because a font's glyph set is small + and bounded; images are not. An LRU at the page level, or at the + individual-image level within a page, has not been designed. +- **Whether iris should keep any binding array at all**, even a small + fixed one (say, capped at a few dozen slots) for atlas pages themselves, + or whether every atlas page should also be its own ordinary bind group + like standalone images — the array's only remaining justification would + be avoiding a bind-group-per-draw-call switch cost that has not been + measured on this project's actual target hardware. +- **How this interacts with I2/E2's virtualised list** (I3): a + bottom-anchored transcript composes only visible rows, so the live + texture set should already be bounded by what is on screen rather than + by the whole conversation — worth confirming that invariant holds before + relying on it to keep atlas/bind-group churn small.