diff --git a/docs/RUST.md b/docs/RUST.md index e85bf2c..7adb9de 100644 --- a/docs/RUST.md +++ b/docs/RUST.md @@ -7499,118 +7499,28 @@ a correct `iris` build look like it built the wrong thing. Read the test *names* in the log to tell two workspaces apart, not a path-relative size. -### Venus went away for an hour, and nothing said so (2026-09-08) +### The GPU went away, and nothing said so (2026-09-08) -**The machine-wide version of this is the `this-machine-graphics` -skill**, written the same day so an agent in any repo gets it: what this -VM's GPU is, the probe that asks it what it supports, this outage's -signature, the wgpu-on-Venus teardown crash, and the rule about never -`.expect()`ing a `PRIMARY` adapter. What stays here is the part specific -to iris and to this port. +**This machine's graphics live in the `this-machine-graphics` skill** -- +what the VM's GPU is, the probe that asks it what it supports, the +transient outage that drops everything to llvmpipe, and the rule about +never `.expect()`ing a `PRIMARY` adapter. It is the only copy; what is +below is only what iris itself changed because of it. -**It came back on its own and nothing in the host config changed.** Iris -asked whether something had, since she still passes Venus as true. What -happened, and what was ruled out, so the next occurrence is not -re-investigated from scratch: - -The symptom, around 02:10 on 2026-09-08 with the VM at load 68 and -several agents building: `vulkaninfo` reported `Failed to detect any -valid GPUs in the current config` and `vkEnumeratePhysicalDevices failed -with ERROR_INITIALIZATION_FAILED`; Mesa printed **`No virgl contexts -available on host`**; and `wgpu` reported `NotFound { active_backends: -VULKAN, no_adapter_backends: VULKAN, supported_backends: VULKAN | GL }`. -So **both** paths through the virtio-gpu died at once -- Venus for -Vulkan and virgl for GL -- and GL then fell through to llvmpipe, which -is what actually rendered that hour's layer-2 screenshots. - -That string is Mesa's virgl DRM winsys, next to `DRM_IOCTL_VIRTGPU_ -CONTEXT_INIT failed with %s` in `libgallium`: the *host* refused a new -context. After a reboot the same host config gives -`Virtio-GPU Venus (AMD Radeon RX 7900 XT (RADV NAVI31))`, Mesa 26.2.2, -driverID `MESA_VENUS`. - -What was ruled out, by measurement rather than by reasoning: - -- **A guest-side context cap.** 48 concurrent short-lived Vulkan clients - all succeed, and 120 concurrent *long-lived* ones (each holding a - `VkDevice` open at once, a throwaway holder) all succeed. So the - ceiling, if there is one, is not near the handful of GPU-using - processes that were running. -- **A Mesa upgrade.** `mesa 1:26.1.7 -> 1:26.2.2` landed 2026-09-05, two - days before Venus was last seen working here. -- **Anything in iris.** It was `vulkaninfo`'s answer too, from a - process that has never linked against this repo. - -**What this host's virtio-gpu actually offers**, asked of the kernel -rather than assumed (`rigs/virtgpu-probe`): bitmask `0x16` -- -**VIRGL, VIRGL2 and VENUS**. Capset 6, the **DRM "native context"**, is -not offered. That is the answer to "is there something to do with qemu -instead of Venus", asked by Iris 2026-09-08: native context is the thing -worth wanting -- RADV running *in the guest* against a passed-through DRM -context instead of Venus proxying every Vulkan call, and it is where -Mesa's effort has gone. Whether it would avoid the teardown crash below -is **untested** -- it is a different driver stack, so it is a reasonable -thing to try rather than a known fix -- but it needs the **host** side to -offer it (virglrenderer built with -its amdgpu DRM renderer, and a qemu that exposes `context_types=drm`; -crosvm has it further along). The guest would also need `vulkan-radeon` -installed, which it does not have today -- only `vulkan-virtio`. The -other two options are VFIO passthrough (complete, but the host loses the -GPU) and dropping `venus=true` (leaves virgl/GL only, i.e. no Vulkan at -all, which is the wrong direction since Vulkan is the phone's path). - -So it is host-side and transient, and this VM cannot see the host to say -more: no `dmesg` (the guest's kernel buffer is not readable to this -user), and no view of the host's `amdgpu`. **The honest state is "we do -not know which host-side resource ran out"** -- worth capturing the host -side of it if it recurs, since that is the half that would answer it. - -**A second, unrelated Venus fault, found the moment it came back -(2026-09-08).** `iris/tests/mask_sdf.rs` had passed the day before and -now `SIGSEGV`d -- and it had passed *because Venus was down*, so it -silently ran on GL. What it actually is, narrowed by measurement: - -- **The test's work completes and its answer is right** (worst - CPU/shader disagreement 5.8e-6). The crash is at process teardown, - dropping wgpu's device: a call through an unmapped address on a - wgpu-created thread, per gdb. -- **It is wgpu's teardown, not Venus's device lifecycle.** A plain - Vulkan program creating and destroying five `VkDevice`s and its - instance on the same adapter exits cleanly (`rigs/virtgpu-probe`). -- **It is Venus-specific.** The same test binary, with Vulkan hidden - (`VK_DRIVER_FILES=/nonexistent`) so wgpu falls back to GL, exits - cleanly. - -Worked around in the test rather than fixed, at `Gpu::leak` with the -reason written there: one device for the whole test, handed to the -process instead of dropped. **Compute was investigated and is not -involved** -- an early version of that test used a compute pass, which -was wrong for its own reason (the paragraph after this one), but the -render-pass rewrite crashes identically, and Venus here reports full -compute anyway (`maxComputeWorkGroupInvocations` 1024, -`maxComputeSharedMemorySize` 65536, Vulkan 1.4 -- `rigs/virtgpu-probe` -again). The 2026-09-05 "no compute" finding is about the **Android -emulator's SwiftShader GL path** reporting ES 3.0, which is a different -machine; it says nothing about this VM. This was got wrong out loud -first, so it is written down: the compute pass was blamed for the crash -before the rewrite showed the crash was not about compute at all. - -The test was rewritten to a render pass regardless, and that part is not -a workaround: it now asks for `iris_core::device_limits()` -- what iris -itself requests -- and calls the function from the fragment stage, which -is where the renderer calls it. A test that needs a capability the thing -under test has never needed is testing the wrong device. - -**What was fixed, because the failure was silent.** Two things, both the -rule that a degraded state must be distinguishable from a healthy one: +For about an hour on 2026-09-08 both paths through the virtio-gpu were +refused by the host at once -- Venus for Vulkan and virgl for GL -- and +GL fell through to **llvmpipe**, which is what actually rendered that +hour's layer-2 screenshots. Two things in iris made that silent, and both +are the same rule: a degraded state must be distinguishable from a +healthy one. 1. `default::render::UiRenderer::new` had the defect the Android backend was fixed for in `85869d0` -- `Backends::PRIMARY` and an `.expect` -- so layer 2 aborted with `Could not get adapter!` instead of falling back. It now probes and rebuilds on `Backends::GL`, as Android does. 2. **The desktop had no logger at all**, so every `log::` call on that - side -- including that new fallback warning -- went to `log`'s - no-op default. `DefaultApp::run` installs a stderr logger now + side -- including that new fallback warning -- went to `log`'s no-op + default. `DefaultApp::run` installs a stderr logger now (`src/default/logging.rs`, no new dependency), and the renderer says which adapter won at `info`: @@ -7624,6 +7534,64 @@ rule that a degraded state must be distinguishable from a healthy one: screenshot is being taken to judge. **Check it before trusting a layer-2 screenshot or any frame number from that window.** +### "wgpu crashes on drop" was the Vulkan loader unloading Mesa (2026-09-08) + +`iris/tests/mask_sdf.rs` `SIGSEGV`d **after** printing `test result: ok`, +and the first reading of that -- written down here, then corrected -- was +"dropping a wgpu device on Venus segfaults", worked around by handing the +device to the process with `std::mem::forget`. That was wrong in every +part except the symptom, and `rigs/gpu-probe`'s `teardown` bin is the +experiment that says so. Its four modes, each one variable: + +| mode | what it does | result | +|---|---|---| +| `main` | wgpu instance + device on the main thread, dropped | exits 0 | +| `thread` | the same on a spawned thread | **SIGSEGV** | +| `keep` | the same, but the instance is never dropped | exits 0 | +| `raw` | raw Vulkan (`ash`), instance + device, spawned thread | **SIGSEGV** | + +So: **no wgpu is required** (raw `ash` does it), **no GPU work is +required** (no pipeline, no submit, no device even -- an instance is +enough), and it is **the thread's exit, not the drop**, that faults. + +The mechanism, confirmed by three measurements. Destroying the last +`VkInstance` makes the Vulkan loader `dlclose` the ICD. Mesa's ICD here +(`/usr/lib/libvulkan_virtio.so`) calls `pthread_key_create` with a +destructor pointing into its own text and is **not** linked +`-z nodelete`, so the key outlives the mapping; glibc then calls that +destructor through an unmapped address when the thread exits. The +faulting PC (`0x7ffff77e6880`, gdb) lands inside that library's mapping +as reported before it was unloaded, the caller is glibc's thread-exit +path, and `VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1` makes every +mode exit cleanly. It is a long-standing incompatibility between +`pthread_key_create` and `dlclose` rather than anything about this VM; +Venus only got the blame because libtest runs each `#[test]` on a +spawned thread and this was the only GPU test. + +**The fix is one `wgpu::Instance` for the process**, which is what wgpu +asks for anyway -- `mask_sdf.rs`'s `vulkan_instance()`, a `OnceLock`. The +device, queue and everything else drop normally again, and the +`mem::forget` escape and its paragraph of reasons are gone. Anything else +in this workspace that opens a GPU on a spawned thread wants the same +instance rather than its own; a second GPU test creating and dropping its +own instance brings the crash straight back. + +Two notes for whoever reads this next. The desktop renderer's GLES +fallback is already safe by construction -- `instance = Instance::new(…)` +builds the replacement before the old one drops, so the ICD's refcount +never reaches zero -- and that is worth not undoing. And +`#![recursion_limit = "256"]` at the top of `mask_sdf.rs` is not related +to any of this: proving `OnceLock: Sync` overflows +rustc's default trait-recursion budget of 128. + +**Compute was never involved**, which is written down because it was said +out loud first: an early version of the test used a compute pass, that +was wrong for its own reason (`device_limits()` deliberately zeroes the +compute limits, since nothing in iris writes a `@compute` stage), and the +render-pass rewrite crashed identically. The 2026-09-05 "no compute" +finding is about the **Android emulator's SwiftShader GL path** reporting +ES 3.0 -- a different machine entirely. + ### Vulkan in the emulator (measured 2026-09-04) **Settled 2026-09-04: the guest gets Vulkan from SwiftShader, and the diff --git a/iris/run-headless.sh b/iris/run-headless.sh index f86062b..a706a92 100755 --- a/iris/run-headless.sh +++ b/iris/run-headless.sh @@ -33,10 +33,10 @@ # through positionally without disturbing the existing `-- cargo args` # convention above. # -# The VM has a virtio-gpu render node (Vulkan 1.4 through Venus, GL 4.6 -# through virgl), so wgpu runs on the host's real GPU -- what is missing is -# only a compositor to give winit a surface. So: a headless sway, the same -# trick `emu` uses for the Android emulator, and `grim` to see the result. +# The VM has a real GPU and no display (the `this-machine-graphics` skill +# says what it is and how it fails), so what is missing here is only a +# compositor to give winit a surface. So: a headless sway, the same trick +# `emu` uses for the Android emulator, and `grim` to see the result. # # It is deliberately *not* `emu`'s compositor. sway tiles, so adding a window # to the one an emulator is sitting in resizes that emulator's window, and a diff --git a/iris/src/default/logging.rs b/iris/src/default/logging.rs index d9536f7..7776151 100644 --- a/iris/src/default/logging.rs +++ b/iris/src/default/logging.rs @@ -4,7 +4,7 @@ //! a no-op logger, and nothing in `desktop-app` or the examples ever //! installed a real one. That is how iris came to have a renderer that //! silently fell back to GLES (and, on this VM, on to llvmpipe when the -//! host lost its virtio-gpu contexts) with **no record anywhere of what +//! host took its GPU away) with **no record anywhere of what //! drew the frame** -- a layer-2 screenshot off llvmpipe and one off the //! host GPU are the same PNG, and the difference is exactly what a //! screenshot is being taken to judge. diff --git a/iris/src/default/render.rs b/iris/src/default/render.rs index 895c066..1163027 100644 --- a/iris/src/default/render.rs +++ b/iris/src/default/render.rs @@ -100,9 +100,9 @@ impl UiRenderer { // the same reason: a machine can advertise a Vulkan ICD with no // device behind it, and refusing to draw at all because the only // usable adapter is a GLES one is iris's bug rather than the - // machine's. On this VM the virtio-gpu Venus device disappears - // whenever the host runs out of virgl contexts, so `run-headless. - // sh` -- layer 2 of the test rig -- aborted with `Could not get + // machine's. On this VM the Vulkan device disappears whenever + // the host refuses a virtio-gpu context, so `run-headless.sh` -- + // layer 2 of the test rig -- aborted with `Could not get // adapter!` while GL was sitting there working. Probed before the // surface exists, matching Android, where an instance carrying // both backends fails worse than one carrying the wrong one. @@ -135,8 +135,8 @@ impl UiRenderer { // Say which adapter won, the way the Android backend's own report // does. Without it a layer-2 screenshot or frame time from this // window carries no record of what drew it, and the two cases that - // matter look identical in the PNG: the host GPU through Venus, - // and llvmpipe after this VM lost its virtio-gpu contexts. That + // matter look identical in the PNG: the host's real GPU, and + // llvmpipe after this VM lost its virtio-gpu contexts. That // happened on 2026-09-08, and the only reason anyone noticed is // that the fallback above did not exist yet and the app aborted // instead. A silent fallback needs this line to stay honest. diff --git a/iris/tests/mask_sdf.rs b/iris/tests/mask_sdf.rs index e5d95ff..7a4ebd6 100644 --- a/iris/tests/mask_sdf.rs +++ b/iris/tests/mask_sdf.rs @@ -28,10 +28,17 @@ //! a capability the thing under test has never needed is testing the //! wrong device, which is reason enough. //! -//! It is **not** why that version crashed, and the record should not say -//! it was: the render-pass rewrite crashes in exactly the same place, and -//! this VM's Venus reports full compute anyway. See `Gpu::leak` for what -//! the crash actually is. +//! It is **not** why that version crashed; see [`vulkan_instance`] for +//! what that crash actually was and why nothing here has to work around +//! it any more. + +// `OnceLock` needs `Instance: Sync`, and wgpu's type +// graph is deep enough that proving it overflows rustc's default trait +// recursion limit of 128. Nothing here is recursive; the limit is a +// compile-time budget, and this is the documented way to raise it. +#![recursion_limit = "256"] + +use std::sync::OnceLock; use iris_core::{SHAPE_SHADER, rounded_rect_coverage, util::Vec2}; use pollster::FutureExt; @@ -98,8 +105,6 @@ fn mask_sdf_matches_the_shader() { } } - gpu.leak(); - let (pos, radius, want, got) = worst_at; assert!( worst <= TOLERANCE, @@ -263,27 +268,52 @@ fn run_shader(gpu: &Gpu, radius: f32) -> Vec { coverage } -/// One device for the whole test, **never dropped** -- see [`Gpu::open`]. +/// One `wgpu::Instance` for the process, created on first use and never +/// destroyed. +/// +/// **Why it is a static rather than a value the test owns.** Destroying +/// the last `VkInstance` makes the Vulkan loader `dlclose` the ICD, and +/// Mesa's ICD here registers a `pthread_key_create` destructor pointing +/// into its own text without being linked `-z nodelete`. glibc then calls +/// that destructor when the thread exits -- through an address that is no +/// longer mapped. libtest runs every `#[test]` on a spawned thread, so a +/// test that opens and closes an instance segfaults *after* printing its +/// result, which reads exactly like the test failing. Measured +/// 2026-09-08 with `rigs/gpu-probe`'s `teardown` bin: it +/// needs no wgpu (raw `ash` does it too), no GPU work, and no device -- +/// an instance created and destroyed on a spawned thread is enough, and +/// keeping any one instance alive is enough to prevent it. +/// +/// Devices, queues and everything else drop normally; only the instance +/// is held, which is what wgpu asks for anyway (one instance per +/// process). So this costs one instance for the length of a test binary +/// and buys ordinary drops everywhere else. +fn vulkan_instance() -> &'static wgpu::Instance { + static INSTANCE: OnceLock = OnceLock::new(); + INSTANCE.get_or_init(wgpu::Instance::default) +} + +/// The device this test draws with. struct Gpu { - instance: wgpu::Instance, device: wgpu::Device, queue: wgpu::Queue, } impl Gpu { /// Opens the device this test draws with, and reports which adapter - /// answered, because that is not a detail here: this same test passes - /// on GL and **`SIGSEGV`s on Venus**, so a run that does not say which - /// one it got cannot be read. + /// answered, because that is not a detail here: a run on llvmpipe and + /// a run on the host's GPU are otherwise indistinguishable in the + /// log, and only one of them is a check of what the phone will do. fn open() -> Self { - let instance = wgpu::Instance::default(); + let instance = vulkan_instance(); let adapter = instance .request_adapter(&wgpu::RequestAdapterOptions::default()) .block_on() .expect( "no wgpu adapter on this machine, so the CPU/shader SDF agreement went \ - unchecked. This VM has a virtio-gpu render node (see iris/run-headless.sh); \ - if that is gone, fix it rather than deleting this test.", + unchecked. This VM has a virtio-gpu render node (the `this-machine-graphics` \ + skill says what it is and how it fails); if that is gone, fix it rather \ + than deleting this test.", ); let info = adapter.get_info(); eprintln!( @@ -298,38 +328,7 @@ impl Gpu { }) .block_on() .expect("could not get a device from the adapter"); - Self { - instance, - device, - queue, - } - } - - /// Hands the device to the process rather than dropping it. - /// - /// **The escape, and why there is no alternative here.** Dropping a - /// `wgpu` device against this VM's Venus adapter segfaults: a call - /// through an unmapped address on a wgpu-created thread, measured - /// 2026-09-08 with gdb. It is `wgpu`'s teardown and not the driver's - /// device lifecycle -- a plain Vulkan program creating and destroying - /// five `VkDevice`s and its instance on the same adapter exits - /// cleanly, and this same test binary exits cleanly when Vulkan is - /// hidden and it falls back to GL. Nothing this test can do about - /// wgpu's drop order makes that call valid, and the alternative is a - /// test that reports a crash after it has already produced its - /// answer, which is indistinguishable from the test failing. - /// - /// Safe because the process is about to exit: the leak is one device - /// and one instance, for the microseconds between here and `main` - /// returning. **Delete this the moment the teardown crash is fixed** - /// -- `cargo test -p iris --test mask_sdf` failing with `SIGSEGV` - /// after printing `test result: ok` is what it looks like when it is - /// still needed, and an ordinary pass is what it looks like when it - /// is not. - fn leak(self) { - std::mem::forget(self.queue); - std::mem::forget(self.device); - std::mem::forget(self.instance); + Self { device, queue } } } diff --git a/rigs/gpu-probe/src/bin/teardown.rs b/rigs/gpu-probe/src/bin/teardown.rs new file mode 100644 index 0000000..fb6a30b --- /dev/null +++ b/rigs/gpu-probe/src/bin/teardown.rs @@ -0,0 +1,102 @@ +//! Why a GPU test segfaults *after* it has passed, and what stops it. +//! +//! Measured here 2026-09-08, on this VM's Venus adapter. Destroying the +//! last `VkInstance` makes the Vulkan loader `dlclose` the ICD; Mesa's +//! ICD (`/usr/lib/libvulkan_virtio.so`) registers a `pthread_key_create` +//! destructor pointing into its own text and is not linked `-z nodelete`, +//! so glibc calls that destructor through unmapped memory when the thread +//! that used Vulkan exits. libtest runs every `#[test]` on a spawned +//! thread, which is why it looked like "wgpu crashes on drop": the drop +//! itself completes, and the crash lands as the thread unwinds. +//! +//! The four modes are the experiment, and each is one variable: +//! +//! | mode | what it does | 2026-09-08 | +//! |---|---|---| +//! | `main` | wgpu instance + device on the main thread, dropped | exits 0 | +//! | `thread` | the same on a spawned thread | **SIGSEGV** | +//! | `keep` | the same, but the instance is never dropped | exits 0 | +//! | `raw` | raw Vulkan (`ash`), instance + device, spawned thread | **SIGSEGV** | +//! +//! `raw` is the one that says whose bug it is: no wgpu is involved, so +//! there is nothing for wgpu or a caller to fix in its drop order. `keep` +//! is the fix -- hold one `wgpu::Instance` for the process, which is what +//! wgpu asks for anyway. `iris/tests/mask_sdf.rs` does exactly that. +//! +//! `VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1` also makes every mode +//! exit cleanly, which is the confirmation that the unload is the +//! mechanism -- but it is an environment variable every caller would have +//! to remember, so it belongs in this comment rather than in a script. + +use ash::vk; +use pollster::block_on; +use wgpu::*; + +fn main() { + let mode = std::env::args().nth(1).unwrap_or_else(|| "thread".into()); + let body = match mode.as_str() { + "main" => return wgpu_open_and_close(false), + "thread" => || wgpu_open_and_close(false), + "keep" => || wgpu_open_and_close(true), + "raw" => raw_vulkan_open_and_close, + other => panic!("unknown mode {other:?}: main | thread | keep | raw"), + }; + std::thread::spawn(body).join().expect("the spawned thread"); + // Not reached when the thread's exit takes the process with it. + eprintln!("thread joined"); +} + +/// A wgpu instance and device, opened and closed. `keep_instance` is the +/// fix under test: everything else still drops normally. +fn wgpu_open_and_close(keep_instance: bool) { + let instance = Instance::default(); + let adapter = + block_on(instance.request_adapter(&RequestAdapterOptions::default())).expect("no adapter"); + let info = adapter.get_info(); + eprintln!( + "adapter: {} ({:?}, {})", + info.name, info.backend, info.driver + ); + let (device, queue) = + block_on(adapter.request_device(&DeviceDescriptor::default())).expect("no device"); + + drop(queue); + drop(device); + drop(adapter); + if keep_instance { + std::mem::forget(instance); + } else { + drop(instance); + } + eprintln!("wgpu closed"); +} + +/// The same shape with no wgpu in it at all, which is what makes this a +/// loader/driver bug rather than a wgpu one. +fn raw_vulkan_open_and_close() { + unsafe { + let entry = ash::Entry::load().expect("vulkan loader"); + let app = vk::ApplicationInfo::default().api_version(vk::make_api_version(0, 1, 1, 0)); + let instance = entry + .create_instance( + &vk::InstanceCreateInfo::default().application_info(&app), + None, + ) + .expect("instance"); + let phys = instance.enumerate_physical_devices().expect("devices")[0]; + let priorities = [1.0f32]; + let queues = [vk::DeviceQueueCreateInfo::default() + .queue_family_index(0) + .queue_priorities(&priorities)]; + let device = instance + .create_device( + phys, + &vk::DeviceCreateInfo::default().queue_create_infos(&queues), + None, + ) + .expect("device"); + device.destroy_device(None); + instance.destroy_instance(None); + } + eprintln!("raw vulkan closed"); +}