iris: the GPU test's crash was the Vulkan loader unloading Mesa, not wgpu
`mask_sdf` SIGSEGVd after printing `test result: ok`, and the workaround was to hand the device to the process with `mem::forget` on the reading that "dropping a wgpu device on Venus segfaults". Every part of that except the symptom was wrong. `rigs/gpu-probe`'s new `teardown` bin is the experiment, one variable per mode: the same open-and-close exits 0 on the main thread and SIGSEGVs on a spawned one; it needs no GPU work and no device, only an instance; raw `ash` does it with no wgpu involved at all; and keeping the instance alive fixes it. Destroying the last VkInstance makes the loader dlclose the ICD, and Mesa's ICD here registers a pthread_key_create destructor into its own text without `-z nodelete`, so glibc calls it through unmapped memory when the thread exits. libtest runs every #[test] on a spawned thread, which is the whole reason this looked like a drop bug. `VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1` confirms the mechanism. So the fix is one `wgpu::Instance` for the process -- what wgpu asks for anyway -- and the device, queue and everything else drop normally again. The escape and its paragraph of reasons are gone. Also: the machine-level graphics notes duplicated in docs/RUST.md, run-headless.sh and two source comments now point at the `this-machine-graphics` skill, which is the only copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f014e8d9cf
commit
b9924e7617
6 files changed
+229
-160
No files matched your search
+72
-104
@@ -7499,118 +7499,28 @@ a correct `iris` build look like it built the wrong thing. Read the test
|
||||
*names* in the log to tell two workspaces apart, not a path-relative
|
||||
size.
|
||||
|
||||
### Venus went away for an hour, and nothing said so (2026-09-08)
|
||||
### The GPU went away, and nothing said so (2026-09-08)
|
||||
|
||||
**The machine-wide version of this is the `this-machine-graphics`
|
||||
skill**, written the same day so an agent in any repo gets it: what this
|
||||
VM's GPU is, the probe that asks it what it supports, this outage's
|
||||
signature, the wgpu-on-Venus teardown crash, and the rule about never
|
||||
`.expect()`ing a `PRIMARY` adapter. What stays here is the part specific
|
||||
to iris and to this port.
|
||||
**This machine's graphics live in the `this-machine-graphics` skill** --
|
||||
what the VM's GPU is, the probe that asks it what it supports, the
|
||||
transient outage that drops everything to llvmpipe, and the rule about
|
||||
never `.expect()`ing a `PRIMARY` adapter. It is the only copy; what is
|
||||
below is only what iris itself changed because of it.
|
||||
|
||||
**It came back on its own and nothing in the host config changed.** Iris
|
||||
asked whether something had, since she still passes Venus as true. What
|
||||
happened, and what was ruled out, so the next occurrence is not
|
||||
re-investigated from scratch:
|
||||
|
||||
The symptom, around 02:10 on 2026-09-08 with the VM at load 68 and
|
||||
several agents building: `vulkaninfo` reported `Failed to detect any
|
||||
valid GPUs in the current config` and `vkEnumeratePhysicalDevices failed
|
||||
with ERROR_INITIALIZATION_FAILED`; Mesa printed **`No virgl contexts
|
||||
available on host`**; and `wgpu` reported `NotFound { active_backends:
|
||||
VULKAN, no_adapter_backends: VULKAN, supported_backends: VULKAN | GL }`.
|
||||
So **both** paths through the virtio-gpu died at once -- Venus for
|
||||
Vulkan and virgl for GL -- and GL then fell through to llvmpipe, which
|
||||
is what actually rendered that hour's layer-2 screenshots.
|
||||
|
||||
That string is Mesa's virgl DRM winsys, next to `DRM_IOCTL_VIRTGPU_
|
||||
CONTEXT_INIT failed with %s` in `libgallium`: the *host* refused a new
|
||||
context. After a reboot the same host config gives
|
||||
`Virtio-GPU Venus (AMD Radeon RX 7900 XT (RADV NAVI31))`, Mesa 26.2.2,
|
||||
driverID `MESA_VENUS`.
|
||||
|
||||
What was ruled out, by measurement rather than by reasoning:
|
||||
|
||||
- **A guest-side context cap.** 48 concurrent short-lived Vulkan clients
|
||||
all succeed, and 120 concurrent *long-lived* ones (each holding a
|
||||
`VkDevice` open at once, a throwaway holder) all succeed. So the
|
||||
ceiling, if there is one, is not near the handful of GPU-using
|
||||
processes that were running.
|
||||
- **A Mesa upgrade.** `mesa 1:26.1.7 -> 1:26.2.2` landed 2026-09-05, two
|
||||
days before Venus was last seen working here.
|
||||
- **Anything in iris.** It was `vulkaninfo`'s answer too, from a
|
||||
process that has never linked against this repo.
|
||||
|
||||
**What this host's virtio-gpu actually offers**, asked of the kernel
|
||||
rather than assumed (`rigs/virtgpu-probe`): bitmask `0x16` --
|
||||
**VIRGL, VIRGL2 and VENUS**. Capset 6, the **DRM "native context"**, is
|
||||
not offered. That is the answer to "is there something to do with qemu
|
||||
instead of Venus", asked by Iris 2026-09-08: native context is the thing
|
||||
worth wanting -- RADV running *in the guest* against a passed-through DRM
|
||||
context instead of Venus proxying every Vulkan call, and it is where
|
||||
Mesa's effort has gone. Whether it would avoid the teardown crash below
|
||||
is **untested** -- it is a different driver stack, so it is a reasonable
|
||||
thing to try rather than a known fix -- but it needs the **host** side to
|
||||
offer it (virglrenderer built with
|
||||
its amdgpu DRM renderer, and a qemu that exposes `context_types=drm`;
|
||||
crosvm has it further along). The guest would also need `vulkan-radeon`
|
||||
installed, which it does not have today -- only `vulkan-virtio`. The
|
||||
other two options are VFIO passthrough (complete, but the host loses the
|
||||
GPU) and dropping `venus=true` (leaves virgl/GL only, i.e. no Vulkan at
|
||||
all, which is the wrong direction since Vulkan is the phone's path).
|
||||
|
||||
So it is host-side and transient, and this VM cannot see the host to say
|
||||
more: no `dmesg` (the guest's kernel buffer is not readable to this
|
||||
user), and no view of the host's `amdgpu`. **The honest state is "we do
|
||||
not know which host-side resource ran out"** -- worth capturing the host
|
||||
side of it if it recurs, since that is the half that would answer it.
|
||||
|
||||
**A second, unrelated Venus fault, found the moment it came back
|
||||
(2026-09-08).** `iris/tests/mask_sdf.rs` had passed the day before and
|
||||
now `SIGSEGV`d -- and it had passed *because Venus was down*, so it
|
||||
silently ran on GL. What it actually is, narrowed by measurement:
|
||||
|
||||
- **The test's work completes and its answer is right** (worst
|
||||
CPU/shader disagreement 5.8e-6). The crash is at process teardown,
|
||||
dropping wgpu's device: a call through an unmapped address on a
|
||||
wgpu-created thread, per gdb.
|
||||
- **It is wgpu's teardown, not Venus's device lifecycle.** A plain
|
||||
Vulkan program creating and destroying five `VkDevice`s and its
|
||||
instance on the same adapter exits cleanly (`rigs/virtgpu-probe`).
|
||||
- **It is Venus-specific.** The same test binary, with Vulkan hidden
|
||||
(`VK_DRIVER_FILES=/nonexistent`) so wgpu falls back to GL, exits
|
||||
cleanly.
|
||||
|
||||
Worked around in the test rather than fixed, at `Gpu::leak` with the
|
||||
reason written there: one device for the whole test, handed to the
|
||||
process instead of dropped. **Compute was investigated and is not
|
||||
involved** -- an early version of that test used a compute pass, which
|
||||
was wrong for its own reason (the paragraph after this one), but the
|
||||
render-pass rewrite crashes identically, and Venus here reports full
|
||||
compute anyway (`maxComputeWorkGroupInvocations` 1024,
|
||||
`maxComputeSharedMemorySize` 65536, Vulkan 1.4 -- `rigs/virtgpu-probe`
|
||||
again). The 2026-09-05 "no compute" finding is about the **Android
|
||||
emulator's SwiftShader GL path** reporting ES 3.0, which is a different
|
||||
machine; it says nothing about this VM. This was got wrong out loud
|
||||
first, so it is written down: the compute pass was blamed for the crash
|
||||
before the rewrite showed the crash was not about compute at all.
|
||||
|
||||
The test was rewritten to a render pass regardless, and that part is not
|
||||
a workaround: it now asks for `iris_core::device_limits()` -- what iris
|
||||
itself requests -- and calls the function from the fragment stage, which
|
||||
is where the renderer calls it. A test that needs a capability the thing
|
||||
under test has never needed is testing the wrong device.
|
||||
|
||||
**What was fixed, because the failure was silent.** Two things, both the
|
||||
rule that a degraded state must be distinguishable from a healthy one:
|
||||
For about an hour on 2026-09-08 both paths through the virtio-gpu were
|
||||
refused by the host at once -- Venus for Vulkan and virgl for GL -- and
|
||||
GL fell through to **llvmpipe**, which is what actually rendered that
|
||||
hour's layer-2 screenshots. Two things in iris made that silent, and both
|
||||
are the same rule: a degraded state must be distinguishable from a
|
||||
healthy one.
|
||||
|
||||
1. `default::render::UiRenderer::new` had the defect the Android backend
|
||||
was fixed for in `85869d0` -- `Backends::PRIMARY` and an `.expect` --
|
||||
so layer 2 aborted with `Could not get adapter!` instead of falling
|
||||
back. It now probes and rebuilds on `Backends::GL`, as Android does.
|
||||
2. **The desktop had no logger at all**, so every `log::` call on that
|
||||
side -- including that new fallback warning -- went to `log`'s
|
||||
no-op default. `DefaultApp::run` installs a stderr logger now
|
||||
side -- including that new fallback warning -- went to `log`'s no-op
|
||||
default. `DefaultApp::run` installs a stderr logger now
|
||||
(`src/default/logging.rs`, no new dependency), and the renderer says
|
||||
which adapter won at `info`:
|
||||
|
||||
@@ -7624,6 +7534,64 @@ rule that a degraded state must be distinguishable from a healthy one:
|
||||
screenshot is being taken to judge. **Check it before trusting a
|
||||
layer-2 screenshot or any frame number from that window.**
|
||||
|
||||
### "wgpu crashes on drop" was the Vulkan loader unloading Mesa (2026-09-08)
|
||||
|
||||
`iris/tests/mask_sdf.rs` `SIGSEGV`d **after** printing `test result: ok`,
|
||||
and the first reading of that -- written down here, then corrected -- was
|
||||
"dropping a wgpu device on Venus segfaults", worked around by handing the
|
||||
device to the process with `std::mem::forget`. That was wrong in every
|
||||
part except the symptom, and `rigs/gpu-probe`'s `teardown` bin is the
|
||||
experiment that says so. Its four modes, each one variable:
|
||||
|
||||
| mode | what it does | result |
|
||||
|---|---|---|
|
||||
| `main` | wgpu instance + device on the main thread, dropped | exits 0 |
|
||||
| `thread` | the same on a spawned thread | **SIGSEGV** |
|
||||
| `keep` | the same, but the instance is never dropped | exits 0 |
|
||||
| `raw` | raw Vulkan (`ash`), instance + device, spawned thread | **SIGSEGV** |
|
||||
|
||||
So: **no wgpu is required** (raw `ash` does it), **no GPU work is
|
||||
required** (no pipeline, no submit, no device even -- an instance is
|
||||
enough), and it is **the thread's exit, not the drop**, that faults.
|
||||
|
||||
The mechanism, confirmed by three measurements. Destroying the last
|
||||
`VkInstance` makes the Vulkan loader `dlclose` the ICD. Mesa's ICD here
|
||||
(`/usr/lib/libvulkan_virtio.so`) calls `pthread_key_create` with a
|
||||
destructor pointing into its own text and is **not** linked
|
||||
`-z nodelete`, so the key outlives the mapping; glibc then calls that
|
||||
destructor through an unmapped address when the thread exits. The
|
||||
faulting PC (`0x7ffff77e6880`, gdb) lands inside that library's mapping
|
||||
as reported before it was unloaded, the caller is glibc's thread-exit
|
||||
path, and `VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1` makes every
|
||||
mode exit cleanly. It is a long-standing incompatibility between
|
||||
`pthread_key_create` and `dlclose` rather than anything about this VM;
|
||||
Venus only got the blame because libtest runs each `#[test]` on a
|
||||
spawned thread and this was the only GPU test.
|
||||
|
||||
**The fix is one `wgpu::Instance` for the process**, which is what wgpu
|
||||
asks for anyway -- `mask_sdf.rs`'s `vulkan_instance()`, a `OnceLock`. The
|
||||
device, queue and everything else drop normally again, and the
|
||||
`mem::forget` escape and its paragraph of reasons are gone. Anything else
|
||||
in this workspace that opens a GPU on a spawned thread wants the same
|
||||
instance rather than its own; a second GPU test creating and dropping its
|
||||
own instance brings the crash straight back.
|
||||
|
||||
Two notes for whoever reads this next. The desktop renderer's GLES
|
||||
fallback is already safe by construction -- `instance = Instance::new(…)`
|
||||
builds the replacement before the old one drops, so the ICD's refcount
|
||||
never reaches zero -- and that is worth not undoing. And
|
||||
`#![recursion_limit = "256"]` at the top of `mask_sdf.rs` is not related
|
||||
to any of this: proving `OnceLock<wgpu::Instance>: Sync` overflows
|
||||
rustc's default trait-recursion budget of 128.
|
||||
|
||||
**Compute was never involved**, which is written down because it was said
|
||||
out loud first: an early version of the test used a compute pass, that
|
||||
was wrong for its own reason (`device_limits()` deliberately zeroes the
|
||||
compute limits, since nothing in iris writes a `@compute` stage), and the
|
||||
render-pass rewrite crashed identically. The 2026-09-05 "no compute"
|
||||
finding is about the **Android emulator's SwiftShader GL path** reporting
|
||||
ES 3.0 -- a different machine entirely.
|
||||
|
||||
### Vulkan in the emulator (measured 2026-09-04)
|
||||
|
||||
**Settled 2026-09-04: the guest gets Vulkan from SwiftShader, and the
|
||||
|
||||
Reference in new issue
Block a user