iris: the GPU test's crash was the Vulkan loader unloading Mesa, not wgpu

`mask_sdf` SIGSEGVd after printing `test result: ok`, and the workaround
was to hand the device to the process with `mem::forget` on the reading
that "dropping a wgpu device on Venus segfaults". Every part of that
except the symptom was wrong.

`rigs/gpu-probe`'s new `teardown` bin is the experiment, one variable per
mode: the same open-and-close exits 0 on the main thread and SIGSEGVs on
a spawned one; it needs no GPU work and no device, only an instance; raw
`ash` does it with no wgpu involved at all; and keeping the instance
alive fixes it. Destroying the last VkInstance makes the loader dlclose
the ICD, and Mesa's ICD here registers a pthread_key_create destructor
into its own text without `-z nodelete`, so glibc calls it through
unmapped memory when the thread exits. libtest runs every #[test] on a
spawned thread, which is the whole reason this looked like a drop bug.
`VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1` confirms the mechanism.

So the fix is one `wgpu::Instance` for the process -- what wgpu asks for
anyway -- and the device, queue and everything else drop normally again.
The escape and its paragraph of reasons are gone.

Also: the machine-level graphics notes duplicated in docs/RUST.md,
run-headless.sh and two source comments now point at the
`this-machine-graphics` skill, which is the only copy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed 2026-09-08 12:22:47 -04:00
1 parent f014e8d9cf
commit b9924e7617
6 files changed
+229 -160

No files matched your search

+72 -104
View File
@@ -7499,118 +7499,28 @@ a correct `iris` build look like it built the wrong thing. Read the test
*names* in the log to tell two workspaces apart, not a path-relative
size.
### Venus went away for an hour, and nothing said so (2026-09-08)
### The GPU went away, and nothing said so (2026-09-08)
**The machine-wide version of this is the `this-machine-graphics`
skill**, written the same day so an agent in any repo gets it: what this
VM's GPU is, the probe that asks it what it supports, this outage's
signature, the wgpu-on-Venus teardown crash, and the rule about never
`.expect()`ing a `PRIMARY` adapter. What stays here is the part specific
to iris and to this port.
**This machine's graphics live in the `this-machine-graphics` skill** --
what the VM's GPU is, the probe that asks it what it supports, the
transient outage that drops everything to llvmpipe, and the rule about
never `.expect()`ing a `PRIMARY` adapter. It is the only copy; what is
below is only what iris itself changed because of it.
**It came back on its own and nothing in the host config changed.** Iris
asked whether something had, since she still passes Venus as true. What
happened, and what was ruled out, so the next occurrence is not
re-investigated from scratch:
The symptom, around 02:10 on 2026-09-08 with the VM at load 68 and
several agents building: `vulkaninfo` reported `Failed to detect any
valid GPUs in the current config` and `vkEnumeratePhysicalDevices failed
with ERROR_INITIALIZATION_FAILED`; Mesa printed **`No virgl contexts
available on host`**; and `wgpu` reported `NotFound { active_backends:
VULKAN, no_adapter_backends: VULKAN, supported_backends: VULKAN | GL }`.
So **both** paths through the virtio-gpu died at once -- Venus for
Vulkan and virgl for GL -- and GL then fell through to llvmpipe, which
is what actually rendered that hour's layer-2 screenshots.
That string is Mesa's virgl DRM winsys, next to `DRM_IOCTL_VIRTGPU_
CONTEXT_INIT failed with %s` in `libgallium`: the *host* refused a new
context. After a reboot the same host config gives
`Virtio-GPU Venus (AMD Radeon RX 7900 XT (RADV NAVI31))`, Mesa 26.2.2,
driverID `MESA_VENUS`.
What was ruled out, by measurement rather than by reasoning:
- **A guest-side context cap.** 48 concurrent short-lived Vulkan clients
all succeed, and 120 concurrent *long-lived* ones (each holding a
`VkDevice` open at once, a throwaway holder) all succeed. So the
ceiling, if there is one, is not near the handful of GPU-using
processes that were running.
- **A Mesa upgrade.** `mesa 1:26.1.7 -> 1:26.2.2` landed 2026-09-05, two
days before Venus was last seen working here.
- **Anything in iris.** It was `vulkaninfo`'s answer too, from a
process that has never linked against this repo.
**What this host's virtio-gpu actually offers**, asked of the kernel
rather than assumed (`rigs/virtgpu-probe`): bitmask `0x16` --
**VIRGL, VIRGL2 and VENUS**. Capset 6, the **DRM "native context"**, is
not offered. That is the answer to "is there something to do with qemu
instead of Venus", asked by Iris 2026-09-08: native context is the thing
worth wanting -- RADV running *in the guest* against a passed-through DRM
context instead of Venus proxying every Vulkan call, and it is where
Mesa's effort has gone. Whether it would avoid the teardown crash below
is **untested** -- it is a different driver stack, so it is a reasonable
thing to try rather than a known fix -- but it needs the **host** side to
offer it (virglrenderer built with
its amdgpu DRM renderer, and a qemu that exposes `context_types=drm`;
crosvm has it further along). The guest would also need `vulkan-radeon`
installed, which it does not have today -- only `vulkan-virtio`. The
other two options are VFIO passthrough (complete, but the host loses the
GPU) and dropping `venus=true` (leaves virgl/GL only, i.e. no Vulkan at
all, which is the wrong direction since Vulkan is the phone's path).
So it is host-side and transient, and this VM cannot see the host to say
more: no `dmesg` (the guest's kernel buffer is not readable to this
user), and no view of the host's `amdgpu`. **The honest state is "we do
not know which host-side resource ran out"** -- worth capturing the host
side of it if it recurs, since that is the half that would answer it.
**A second, unrelated Venus fault, found the moment it came back
(2026-09-08).** `iris/tests/mask_sdf.rs` had passed the day before and
now `SIGSEGV`d -- and it had passed *because Venus was down*, so it
silently ran on GL. What it actually is, narrowed by measurement:
- **The test's work completes and its answer is right** (worst
CPU/shader disagreement 5.8e-6). The crash is at process teardown,
dropping wgpu's device: a call through an unmapped address on a
wgpu-created thread, per gdb.
- **It is wgpu's teardown, not Venus's device lifecycle.** A plain
Vulkan program creating and destroying five `VkDevice`s and its
instance on the same adapter exits cleanly (`rigs/virtgpu-probe`).
- **It is Venus-specific.** The same test binary, with Vulkan hidden
(`VK_DRIVER_FILES=/nonexistent`) so wgpu falls back to GL, exits
cleanly.
Worked around in the test rather than fixed, at `Gpu::leak` with the
reason written there: one device for the whole test, handed to the
process instead of dropped. **Compute was investigated and is not
involved** -- an early version of that test used a compute pass, which
was wrong for its own reason (the paragraph after this one), but the
render-pass rewrite crashes identically, and Venus here reports full
compute anyway (`maxComputeWorkGroupInvocations` 1024,
`maxComputeSharedMemorySize` 65536, Vulkan 1.4 -- `rigs/virtgpu-probe`
again). The 2026-09-05 "no compute" finding is about the **Android
emulator's SwiftShader GL path** reporting ES 3.0, which is a different
machine; it says nothing about this VM. This was got wrong out loud
first, so it is written down: the compute pass was blamed for the crash
before the rewrite showed the crash was not about compute at all.
The test was rewritten to a render pass regardless, and that part is not
a workaround: it now asks for `iris_core::device_limits()` -- what iris
itself requests -- and calls the function from the fragment stage, which
is where the renderer calls it. A test that needs a capability the thing
under test has never needed is testing the wrong device.
**What was fixed, because the failure was silent.** Two things, both the
rule that a degraded state must be distinguishable from a healthy one:
For about an hour on 2026-09-08 both paths through the virtio-gpu were
refused by the host at once -- Venus for Vulkan and virgl for GL -- and
GL fell through to **llvmpipe**, which is what actually rendered that
hour's layer-2 screenshots. Two things in iris made that silent, and both
are the same rule: a degraded state must be distinguishable from a
healthy one.
1. `default::render::UiRenderer::new` had the defect the Android backend
was fixed for in `85869d0` -- `Backends::PRIMARY` and an `.expect` --
so layer 2 aborted with `Could not get adapter!` instead of falling
back. It now probes and rebuilds on `Backends::GL`, as Android does.
2. **The desktop had no logger at all**, so every `log::` call on that
side -- including that new fallback warning -- went to `log`'s
no-op default. `DefaultApp::run` installs a stderr logger now
side -- including that new fallback warning -- went to `log`'s no-op
default. `DefaultApp::run` installs a stderr logger now
(`src/default/logging.rs`, no new dependency), and the renderer says
which adapter won at `info`:
@@ -7624,6 +7534,64 @@ rule that a degraded state must be distinguishable from a healthy one:
screenshot is being taken to judge. **Check it before trusting a
layer-2 screenshot or any frame number from that window.**
### "wgpu crashes on drop" was the Vulkan loader unloading Mesa (2026-09-08)
`iris/tests/mask_sdf.rs` `SIGSEGV`d **after** printing `test result: ok`,
and the first reading of that -- written down here, then corrected -- was
"dropping a wgpu device on Venus segfaults", worked around by handing the
device to the process with `std::mem::forget`. That was wrong in every
part except the symptom, and `rigs/gpu-probe`'s `teardown` bin is the
experiment that says so. Its four modes, each one variable:
| mode | what it does | result |
|---|---|---|
| `main` | wgpu instance + device on the main thread, dropped | exits 0 |
| `thread` | the same on a spawned thread | **SIGSEGV** |
| `keep` | the same, but the instance is never dropped | exits 0 |
| `raw` | raw Vulkan (`ash`), instance + device, spawned thread | **SIGSEGV** |
So: **no wgpu is required** (raw `ash` does it), **no GPU work is
required** (no pipeline, no submit, no device even -- an instance is
enough), and it is **the thread's exit, not the drop**, that faults.
The mechanism, confirmed by three measurements. Destroying the last
`VkInstance` makes the Vulkan loader `dlclose` the ICD. Mesa's ICD here
(`/usr/lib/libvulkan_virtio.so`) calls `pthread_key_create` with a
destructor pointing into its own text and is **not** linked
`-z nodelete`, so the key outlives the mapping; glibc then calls that
destructor through an unmapped address when the thread exits. The
faulting PC (`0x7ffff77e6880`, gdb) lands inside that library's mapping
as reported before it was unloaded, the caller is glibc's thread-exit
path, and `VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1` makes every
mode exit cleanly. It is a long-standing incompatibility between
`pthread_key_create` and `dlclose` rather than anything about this VM;
Venus only got the blame because libtest runs each `#[test]` on a
spawned thread and this was the only GPU test.
**The fix is one `wgpu::Instance` for the process**, which is what wgpu
asks for anyway -- `mask_sdf.rs`'s `vulkan_instance()`, a `OnceLock`. The
device, queue and everything else drop normally again, and the
`mem::forget` escape and its paragraph of reasons are gone. Anything else
in this workspace that opens a GPU on a spawned thread wants the same
instance rather than its own; a second GPU test creating and dropping its
own instance brings the crash straight back.
Two notes for whoever reads this next. The desktop renderer's GLES
fallback is already safe by construction -- `instance = Instance::new(…)`
builds the replacement before the old one drops, so the ICD's refcount
never reaches zero -- and that is worth not undoing. And
`#![recursion_limit = "256"]` at the top of `mask_sdf.rs` is not related
to any of this: proving `OnceLock<wgpu::Instance>: Sync` overflows
rustc's default trait-recursion budget of 128.
**Compute was never involved**, which is written down because it was said
out loud first: an early version of the test used a compute pass, that
was wrong for its own reason (`device_limits()` deliberately zeroes the
compute limits, since nothing in iris writes a `@compute` stage), and the
render-pass rewrite crashed identically. The 2026-09-05 "no compute"
finding is about the **Android emulator's SwiftShader GL path** reporting
ES 3.0 -- a different machine entirely.
### Vulkan in the emulator (measured 2026-09-04)
**Settled 2026-09-04: the guest gets Vulkan from SwiftShader, and the