iris: cut test debug info, say which adapter drew, and log on the desktop
Three findings from one morning, all of them things that were invisible rather than wrong. docs/RUST.md's two new sections have the full account. **`cargo test --workspace` was taking half an hour, and it was debug info.** rustc's default `debug = true`, times eight test binaries each statically linking the whole wgpu + naga + winit + parley graph, means every one of them gets a private copy of that graph's DWARF written into it: the linkers for one run had written ~54 GB between them and were still going at thirty minutes -- the worst single one 16.9 GB for one test binary -- leaving an 88 GB target/. It was not CPU: the machine was 87% idle, and rust-lld's threads were in D state in btrfs `handle_reserve_ticket`, blocked on space reservation at 83% full. So `debug = "line-tables-only"` on both `profile.dev` and `profile.test` -- both, because `cargo test` builds dependencies under one and the test targets under the other. Cold, with all 19 suites run: 69 s and a 3.7 GB target. Backtraces keep file and line; `RUSTFLAGS="-C debuginfo=2"` per run buys back variable inspection when a debugger actually needs it. **The desktop had no logger at all**, so every `log::` call on that side went to `log`'s no-op default -- including the GLES fallback warning added hours earlier. `DefaultApp::run` installs a stderr logger (`src/default/logging.rs`, no new dependency: a level and a line is a page of code against env_logger plus its filter dialect), and the renderer now says which adapter won at `info`. That line is the point: with a silent fallback, a layer-2 screenshot rendered by llvmpipe and one rendered by the host's GPU are the same PNG, and which one it was is exactly what the screenshot is being taken to judge. **`tests/mask_sdf.rs` is a render pass now, not a compute pass.** It asked for `adapter.limits()` because `iris_core::device_limits()` deliberately zeroes the six `max_compute_*` fields -- a decision on record since 2026-09-05, which this quietly worked around instead of following. It now asks for what iris asks for and calls the function from the fragment stage, where the renderer calls it. The compute pass was *not* why it crashed, and the record should not say it was: the rewrite crashes identically. What the crash is: dropping a wgpu device on this VM's Venus adapter segfaults, after the test has produced its answer (worst CPU/shader disagreement 5.8e-6). Narrowed -- plain Vulkan creating and destroying five VkDevices on the same adapter is clean, and the same binary with Vulkan hidden falls back to GL and exits clean. Worked around at `Gpu::leak`, with the reason and the delete-me condition written there. `rigs/virtgpu-probe` is the new rig behind the Venus half: which capsets the host offers (0x16 -- VIRGL, VIRGL2, VENUS; no capset 6, so no DRM native context without host-side work), whether the device has compute (it does: 1024 invocations/workgroup -- the "no compute" finding on record is about the Android emulator's SwiftShader, a different machine), and whether plain Vulkan teardown is clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c6da735134
commit
0ccc444246
8 files changed
+722
-112
No files matched your search
+177
@@ -7440,6 +7440,183 @@ re-derived:
|
||||
7. Decisions belong here with a date and what was rejected, the way
|
||||
`PLAN.md` does it. Do not put design into commit messages alone.
|
||||
|
||||
### `cargo test --workspace` was taking half an hour, and it was debug info (2026-09-08)
|
||||
|
||||
Iris asked why. It is worth reading before concluding that this machine
|
||||
needs more disk, because that was the first guess here too and it is the
|
||||
smaller half.
|
||||
|
||||
**What it was.** rustc's default for `dev` is `debug = true`, and this
|
||||
workspace has **eight test binaries**, each statically linking the whole
|
||||
wgpu + naga + winit + parley graph. Every one of them therefore gets its
|
||||
own full copy of that graph's DWARF written into it at link time.
|
||||
Measured: the linkers for one `cargo test --workspace` had written
|
||||
**~54 GB between them** and were still going at 30 minutes, the worst
|
||||
single one **16.9 GB for one test binary**, leaving an **88 GB**
|
||||
`target/`.
|
||||
|
||||
**What it was not: CPU.** The machine was **87% idle** with 8-12%
|
||||
iowait throughout, and every `rustc` was in `__futex_wait` with no busy
|
||||
thread. An earlier guess in this same session -- that concurrent agents'
|
||||
builds were loading the machine -- was wrong, and checking `ps` for a
|
||||
busy process would have killed it in a minute. `rust-lld`'s worker
|
||||
threads were in **`D` state in btrfs `handle_reserve_ticket`**: blocked
|
||||
on space reservation, because the filesystem was at 83% full. So the
|
||||
write volume is the disease and the full disk is what turned slow into
|
||||
stalled -- they move about 20 MB/s between them in that state.
|
||||
|
||||
**How to tell it apart from a busy build**, since the two look identical
|
||||
from the outside (`cargo` printing `Compiling ...` and not finishing):
|
||||
|
||||
ps -eo pid,stat,etime,pcpu,comm | grep -E 'rustc|rust-lld'
|
||||
cat /proc/<lld-pid>/wchan # handle_reserve_ticket = btrfs, not you
|
||||
for l in $(pgrep -x rust-lld); do awk '/^write_bytes/{print $2/1048576}' /proc/$l/io; done
|
||||
|
||||
A linker at 0.5% CPU having written gigabytes is not compiling.
|
||||
|
||||
**The fix, in `iris/Cargo.toml`.** `debug = "line-tables-only"` on both
|
||||
`profile.dev` and `profile.test` -- both, because `cargo test` builds
|
||||
dependencies under `dev` and the test targets themselves under `test`, so
|
||||
setting only `dev` leaves the eight big binaries at the default. It keeps
|
||||
the file and line of every frame, which is what a panicking test prints
|
||||
and what gdb needs to name the frames of a segfault (both were used
|
||||
today); it gives up inspecting variables in a debugger, which is worth
|
||||
asking for per-run when it is wanted:
|
||||
|
||||
RUSTFLAGS="-C debuginfo=2" cargo test -p iris --test mask_sdf
|
||||
|
||||
**Measured after**, cold (`cargo clean` first), in `iris/`: the whole
|
||||
workspace compiles *and runs all 19 test suites* in **69 s**, with a
|
||||
**3.7 GB** `target/`. Before, an *incremental* run — everything already
|
||||
compiled, only the eight binaries left to link — had not finished after
|
||||
thirty minutes. `cargo clean` alongside the profile change freed
|
||||
**104 GB** (83% -> 67% full).
|
||||
|
||||
**One trap while measuring this, worth not repeating**: the session's
|
||||
working directory is the repo root, not `iris/`, so a `du -sh target`
|
||||
appended to a build command reports the root workspace's 29 MB and makes
|
||||
a correct `iris` build look like it built the wrong thing. Read the test
|
||||
*names* in the log to tell two workspaces apart, not a path-relative
|
||||
size.
|
||||
|
||||
### Venus went away for an hour, and nothing said so (2026-09-08)
|
||||
|
||||
**It came back on its own and nothing in the host config changed.** Iris
|
||||
asked whether something had, since she still passes Venus as true. What
|
||||
happened, and what was ruled out, so the next occurrence is not
|
||||
re-investigated from scratch:
|
||||
|
||||
The symptom, around 02:10 on 2026-09-08 with the VM at load 68 and
|
||||
several agents building: `vulkaninfo` reported `Failed to detect any
|
||||
valid GPUs in the current config` and `vkEnumeratePhysicalDevices failed
|
||||
with ERROR_INITIALIZATION_FAILED`; Mesa printed **`No virgl contexts
|
||||
available on host`**; and `wgpu` reported `NotFound { active_backends:
|
||||
VULKAN, no_adapter_backends: VULKAN, supported_backends: VULKAN | GL }`.
|
||||
So **both** paths through the virtio-gpu died at once -- Venus for
|
||||
Vulkan and virgl for GL -- and GL then fell through to llvmpipe, which
|
||||
is what actually rendered that hour's layer-2 screenshots.
|
||||
|
||||
That string is Mesa's virgl DRM winsys, next to `DRM_IOCTL_VIRTGPU_
|
||||
CONTEXT_INIT failed with %s` in `libgallium`: the *host* refused a new
|
||||
context. After a reboot the same host config gives
|
||||
`Virtio-GPU Venus (AMD Radeon RX 7900 XT (RADV NAVI31))`, Mesa 26.2.2,
|
||||
driverID `MESA_VENUS`.
|
||||
|
||||
What was ruled out, by measurement rather than by reasoning:
|
||||
|
||||
- **A guest-side context cap.** 48 concurrent short-lived Vulkan clients
|
||||
all succeed, and 120 concurrent *long-lived* ones (each holding a
|
||||
`VkDevice` open at once, a throwaway holder) all succeed. So the
|
||||
ceiling, if there is one, is not near the handful of GPU-using
|
||||
processes that were running.
|
||||
- **A Mesa upgrade.** `mesa 1:26.1.7 -> 1:26.2.2` landed 2026-09-05, two
|
||||
days before Venus was last seen working here.
|
||||
- **Anything in iris.** It was `vulkaninfo`'s answer too, from a
|
||||
process that has never linked against this repo.
|
||||
|
||||
**What this host's virtio-gpu actually offers**, asked of the kernel
|
||||
rather than assumed (`rigs/virtgpu-probe`): bitmask `0x16` --
|
||||
**VIRGL, VIRGL2 and VENUS**. Capset 6, the **DRM "native context"**, is
|
||||
not offered. That is the answer to "is there something to do with qemu
|
||||
instead of Venus", asked by Iris 2026-09-08: native context is the thing
|
||||
worth wanting -- RADV running *in the guest* against a passed-through DRM
|
||||
context instead of Venus proxying every Vulkan call, and it is where
|
||||
Mesa's effort has gone. Whether it would avoid the teardown crash below
|
||||
is **untested** -- it is a different driver stack, so it is a reasonable
|
||||
thing to try rather than a known fix -- but it needs the **host** side to
|
||||
offer it (virglrenderer built with
|
||||
its amdgpu DRM renderer, and a qemu that exposes `context_types=drm`;
|
||||
crosvm has it further along). The guest would also need `vulkan-radeon`
|
||||
installed, which it does not have today -- only `vulkan-virtio`. The
|
||||
other two options are VFIO passthrough (complete, but the host loses the
|
||||
GPU) and dropping `venus=true` (leaves virgl/GL only, i.e. no Vulkan at
|
||||
all, which is the wrong direction since Vulkan is the phone's path).
|
||||
|
||||
So it is host-side and transient, and this VM cannot see the host to say
|
||||
more: no `dmesg` (the guest's kernel buffer is not readable to this
|
||||
user), and no view of the host's `amdgpu`. **The honest state is "we do
|
||||
not know which host-side resource ran out"** -- worth capturing the host
|
||||
side of it if it recurs, since that is the half that would answer it.
|
||||
|
||||
**A second, unrelated Venus fault, found the moment it came back
|
||||
(2026-09-08).** `iris/tests/mask_sdf.rs` had passed the day before and
|
||||
now `SIGSEGV`d -- and it had passed *because Venus was down*, so it
|
||||
silently ran on GL. What it actually is, narrowed by measurement:
|
||||
|
||||
- **The test's work completes and its answer is right** (worst
|
||||
CPU/shader disagreement 5.8e-6). The crash is at process teardown,
|
||||
dropping wgpu's device: a call through an unmapped address on a
|
||||
wgpu-created thread, per gdb.
|
||||
- **It is wgpu's teardown, not Venus's device lifecycle.** A plain
|
||||
Vulkan program creating and destroying five `VkDevice`s and its
|
||||
instance on the same adapter exits cleanly (`rigs/virtgpu-probe`).
|
||||
- **It is Venus-specific.** The same test binary, with Vulkan hidden
|
||||
(`VK_DRIVER_FILES=/nonexistent`) so wgpu falls back to GL, exits
|
||||
cleanly.
|
||||
|
||||
Worked around in the test rather than fixed, at `Gpu::leak` with the
|
||||
reason written there: one device for the whole test, handed to the
|
||||
process instead of dropped. **Compute was investigated and is not
|
||||
involved** -- an early version of that test used a compute pass, which
|
||||
was wrong for its own reason (the paragraph after this one), but the
|
||||
render-pass rewrite crashes identically, and Venus here reports full
|
||||
compute anyway (`maxComputeWorkGroupInvocations` 1024,
|
||||
`maxComputeSharedMemorySize` 65536, Vulkan 1.4 -- `rigs/virtgpu-probe`
|
||||
again). The 2026-09-05 "no compute" finding is about the **Android
|
||||
emulator's SwiftShader GL path** reporting ES 3.0, which is a different
|
||||
machine; it says nothing about this VM. This was got wrong out loud
|
||||
first, so it is written down: the compute pass was blamed for the crash
|
||||
before the rewrite showed the crash was not about compute at all.
|
||||
|
||||
The test was rewritten to a render pass regardless, and that part is not
|
||||
a workaround: it now asks for `iris_core::device_limits()` -- what iris
|
||||
itself requests -- and calls the function from the fragment stage, which
|
||||
is where the renderer calls it. A test that needs a capability the thing
|
||||
under test has never needed is testing the wrong device.
|
||||
|
||||
**What was fixed, because the failure was silent.** Two things, both the
|
||||
rule that a degraded state must be distinguishable from a healthy one:
|
||||
|
||||
1. `default::render::UiRenderer::new` had the defect the Android backend
|
||||
was fixed for in `85869d0` -- `Backends::PRIMARY` and an `.expect` --
|
||||
so layer 2 aborted with `Could not get adapter!` instead of falling
|
||||
back. It now probes and rebuilds on `Backends::GL`, as Android does.
|
||||
2. **The desktop had no logger at all**, so every `log::` call on that
|
||||
side -- including that new fallback warning -- went to `log`'s
|
||||
no-op default. `DefaultApp::run` installs a stderr logger now
|
||||
(`src/default/logging.rs`, no new dependency), and the renderer says
|
||||
which adapter won at `info`:
|
||||
|
||||
INFO iris::default::render: iris renderer: Virtio-GPU Venus
|
||||
(AMD Radeon RX 7900 XT (RADV NAVI31)) (Vulkan, venus Mesa
|
||||
26.2.2-arch1.1) on Backends(VULKAN | METAL | DX12 | BROWSER_WEBGPU)
|
||||
|
||||
That line is the point: with the fallback in place and no logger, a
|
||||
`run-headless.sh` screenshot rendered by llvmpipe and one rendered by
|
||||
the host's GPU are the same PNG, and the difference is exactly what a
|
||||
screenshot is being taken to judge. **Check it before trusting a
|
||||
layer-2 screenshot or any frame number from that window.**
|
||||
|
||||
### Vulkan in the emulator (measured 2026-09-04)
|
||||
|
||||
**Settled 2026-09-04: the guest gets Vulkan from SwiftShader, and the
|
||||
|
||||
Reference in new issue
Block a user