irisandClaude Opus 5
b678bf3ccd
`--do "tap 'Save'"` finds whatever currently carries that label -- the text, or the padded control around it, preferring the clickable one -- resolves its box from the accessibility tree at the moment of the gesture, and presses its centre. It is the name the control already has for assistive technology, so there is nothing extra to keep in step with it. `tap X Y` still works and is now the exception. A coordinate is a position measured once by hand: anything that moves the control makes the tap land on whatever now sits there, and the script then reports a number that was never measured, which reads exactly like a result rather than like a failure. ai-app's two benchmark scripts pressed a header button at `tap 723 205` and that button has now moved; Iris asked on 2026-09-03 that the fix be in the tool rather than a habit each script remembers. Two things make the failure loud, which is the whole point. A label that is not on screen ends the recording with `# error` and a non-zero exit, and `record` now prints the error lines rather than the head of a trace that can be thousands of frames long. And the sampling loop waits for the action thread before exiting -- a script whose last step outlasted the recording used to have its outcome discarded, including that error. The label is looked for over three seconds rather than once: the accessibility connection has no window at all for the first frames after it is made, and a control revealed by the previous step arrives a frame or two later. The wait is bounded and the failure is still loud. Exercised against a running emulator: a tap by label, a tap that finds nothing, and a failure that lands after the recording's own duration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
199 lines
9.5 KiB
Markdown
199 lines
9.5 KiB
Markdown
# emulator-tools
|
|
|
|
Everything this machine uses to drive an Android emulator, in one place:
|
|
which emulator a command means, whether there is room to start it, reading a
|
|
screen as text, and taking a screenshot that does not cost a thousand tokens.
|
|
|
|
./install.sh
|
|
|
|
symlinks `bin/*` into `~/.local/bin` and `share/ui-trace` into
|
|
`~/.local/share`, so editing this repo is editing what runs. It keeps
|
|
anything it replaces as `<name>.bak`.
|
|
|
|
## What is here
|
|
|
|
| | |
|
|
|---|---|
|
|
| `emu` | this checkout's emulator: `name`, `serial`, `list`, `up`, `down`, `check` |
|
|
| `adb` | adb, aimed at this checkout's emulator, with screenshots scaled |
|
|
| `ui-trace` | records a screen as text at 60Hz, and drives it |
|
|
| `lib/project-avd.sh` | the rules all three share |
|
|
| `share/ui-trace/` | the device recorder (`UiTrace.java`) and its build |
|
|
| `share/gradle-init/` | the init script that stops Gradle installing on every device |
|
|
|
|
## One emulator per checkout, named after it
|
|
|
|
`emu` and `adb` both work out which AVD you mean from the enclosing git
|
|
checkout: in `~/repos/ai-app-2` that is `ai-app-2`. Nothing has to be
|
|
configured, and `AVD_NAME` overrides it where the guess is wrong.
|
|
|
|
This is the rule that lets several agent sessions work here at once. The
|
|
emulator used to be the one resource that could not be parallelised: taking
|
|
it meant messaging the other sessions, waiting, and handing it back — and
|
|
installing an app onto somebody else's running emulator steals the foreground
|
|
from whatever they were looking at.
|
|
|
|
It is also why `adb` fills in `-s`. With two emulators attached, a bare
|
|
`adb shell pm list packages` comes back **empty** rather than failing, which
|
|
reads as the app having been uninstalled rather than as the question being
|
|
ambiguous. An explicit `-s`, `ANDROID_SERIAL`, or a subcommand that is not
|
|
about one device (`adb devices`) all turn the defaulting off, and nothing is
|
|
guessed when this checkout's emulator is not running: adb's own error is
|
|
better than a wrapper picking a stranger's device.
|
|
|
|
## Gradle is the hole the wrapper cannot cover
|
|
|
|
`adb` can only aim a call that goes through it, and Gradle's Android tasks
|
|
do not: `installDebug`, `uninstallDebug` and `connectedAndroidTest` ask the
|
|
adb server for every attached device and act on all of them. On 2026-08-31
|
|
one session ran `./gradlew :androidApp:installDebug` with two emulators up
|
|
and replaced the app on both. Gradle reported success; from the other
|
|
session it looked like its own build had never landed.
|
|
|
|
So `install.sh` also links `share/gradle-init/emulator-tools.gradle` into
|
|
`~/.gradle/init.d`, where Gradle applies it to **every** build on this
|
|
machine — including checkouts nobody has adopted it in, which are exactly
|
|
the ones this is protecting. Before any of those tasks runs it calls
|
|
|
|
emu check
|
|
|
|
in the project's own directory. That is the same rule the `adb` wrapper
|
|
follows, in one place: what must not be touched is *another checkout's
|
|
emulator*. A physical phone is nobody's checkout, one emulator on its own is
|
|
nobody else's business, and a caller who named a device has already narrowed
|
|
the fan-out to it — so `check` passes all of those, and refuses only when a
|
|
stranger's emulator is in range. `ANDROID_SERIAL` pointing at one is refused
|
|
too, because that is what a serial exported into a long-lived shell decays
|
|
into once an emulator restarts and another checkout's takes the port.
|
|
|
|
Say which device you mean at the moment you use it rather than exporting it:
|
|
|
|
ANDROID_SERIAL=$(emu serial) ./gradlew :androidApp:installDebug
|
|
|
|
`EMU_ANY_DEVICE=1` means every attached device anyway, for the case that is
|
|
genuinely what you want.
|
|
|
|
## Rendering on the GPU with no display
|
|
|
|
This machine has no screen, and for a long time that meant the emulator
|
|
rasterised in software. It does not have to: the VM sees the host's GPU
|
|
through virtio-gpu, and `/dev/dri/renderD128` is readable without being in
|
|
the `video` group.
|
|
|
|
What was in the way is that the emulator's `-gpu host` renderer speaks GLX,
|
|
so with no `DISPLAY` it cannot open a context at all -- it says
|
|
`GlxEnginegetDefaultDisplay: Failed to open display 0` and falls back. So
|
|
`emu up` starts a **headless sway** whose only job is to provide an
|
|
Xwayland, and Xwayland's GLX runs on the render node. The guest's GLES then
|
|
reports `virgl (AMD Radeon RX 7900 XT)` instead of a software rasteriser.
|
|
One compositor is started for the machine and reused, on a named socket
|
|
under `XDG_RUNTIME_DIR`; it needs no seat, no DRM master and no card node.
|
|
|
|
Measured 2026-08-31, scrolling the same list in the stock **Settings** app
|
|
on the same AVD with the same gesture -- so this is the platform's own cost,
|
|
with no app of ours in it:
|
|
|
|
| | janky frames | legacy janky | 90th percentile | slow UI-thread frames |
|
|
|---|---|---|---|---|
|
|
| software | 57.4% | 100% | 73ms | 18 |
|
|
| on the GPU | 2.3% | 5.7% | 26ms | 0 |
|
|
|
|
Read **both** jank columns. The legacy figure counts a frame against the
|
|
16.7ms deadline and the modern one against what the compositor actually
|
|
needed, so on a fast enough path the modern number can barely move while the
|
|
legacy one halves. When comparing two builds of an app in here, the legacy
|
|
column is usually the one carrying the signal.
|
|
|
|
An emulator also costs about **620 MB less** resident this way: measured
|
|
twenty seconds after boot on one AVD, 3537 MB on software against 2918 MB on
|
|
the GPU. (An earlier draft of this file said 1.1 GB. That number came from
|
|
`emu list` before the prefix bug in it was fixed, and was one emulator's size
|
|
reported against another's.)
|
|
|
|
This matters for more than speed. Before it, frame timings measured in here
|
|
said nothing at all about an app: every app was bounded by the rasteriser
|
|
rather than by anything it was doing, and the stock Settings app missed
|
|
*every single frame's* deadline while scrolling its own list. Now the
|
|
platform floor is low enough that an app's own jank is visible above it.
|
|
|
|
`GPU_HOST_FEATURES` carries the one workaround: host Vulkan is switched off,
|
|
because gfxstream asks Venus for a memory type for its ColorBuffers that
|
|
Venus does not offer, and the emulator dies before adb sees it. GLES is
|
|
unaffected. It is worth retrying whenever mesa moves -- the failure is loud
|
|
and costs one boot to find.
|
|
|
|
If sway is missing, or the compositor will not start, `emu up` says so and
|
|
starts on software rendering as it always did. None of this is essential.
|
|
|
|
## Refusing to start one
|
|
|
|
`emu up` checks `MemAvailable` first and refuses if starting an emulator
|
|
would leave the machine short, printing what is attached, what is large, and
|
|
what usually frees enough. `EMU_FORCE=1` overrides it.
|
|
|
|
The reason is a real incident rather than tidiness. On 2026-08-30 an emulator
|
|
was started on this box with 2.8 GB available; the OOM killer ran, and what
|
|
it took was not the emulator that had just started — it walked the user slice
|
|
and killed pipewire, dbus-broker and *another session's* emulator first. The
|
|
cost of one too many lands on somebody else's work, minutes later, looking
|
|
like an unrelated crash.
|
|
|
|
Measured numbers behind the defaults: a headless x86_64 AVD is about 3.8 GB
|
|
resident, this VM has 23 GB with a swap that is routinely full, two emulators
|
|
are comfortable, and three were not.
|
|
|
|
## Reading a screen without screenshots
|
|
|
|
`ui-trace` drives a small resident recorder on the device and samples the
|
|
accessibility tree at 60Hz, so an animation or a settling layout is visible
|
|
rather than happening between two samples.
|
|
|
|
ui-trace record -d 3000 --do "tap 'Session settings'" -o /tmp/t.txt
|
|
ui-trace elements /tmp/t.txt # what is on screen, to pick from
|
|
ui-trace show /tmp/t.txt # what moved
|
|
ui-trace show /tmp/t.txt -m 'Send|Stop' --field box
|
|
|
|
Bounds come out in **device** pixels, so they feed straight back into a
|
|
`--do tap`. Prefer this to a screenshot for anything positional: two buttons
|
|
that measured identically at 171x105 looked different in a screenshot,
|
|
because the filled one read as larger than the disabled grey one. Screenshots
|
|
are for appearance — colour, weight, whether it looks right.
|
|
|
|
### Press things by name, never by coordinate
|
|
|
|
`tap 'Save'` finds whatever currently carries that label — the text, or the
|
|
padded control around it, preferring the one that is clickable — resolves its
|
|
box from the tree **at the moment of the gesture**, and presses its centre. It
|
|
is the name the control already has for assistive technology, so nothing has
|
|
to be kept in step with it.
|
|
|
|
`tap X Y` still exists and is the exception. A coordinate is a position
|
|
measured once by hand, and anything that moves the control — a button added to
|
|
the row, a font size, a density, another device — makes the tap land on
|
|
whatever now sits there. The script then reports a number that was never
|
|
measured, which reads exactly like a result rather than like a failure. Asked
|
|
for by Iris on 2026-09-03, after ai-app's two benchmark scripts pressed a
|
|
header button at `tap 723 205` and that button moved.
|
|
|
|
A label that is not on screen ends the recording with `# error action failed`
|
|
and a non-zero exit, so a run that could not press what it meant to press
|
|
produces no numbers at all.
|
|
|
|
The check that none is left in a project's scripts is one grep:
|
|
|
|
grep -n "tap [0-9]" path/to/*.sh
|
|
|
|
`adb exec-out screencap -p` still works and comes back scaled to 800px on its
|
|
long edge, which is ~380 tokens to read instead of ~1460 and still legible
|
|
for layout, contrast and small labels. `ADB_SCREENCAP_MAX_EDGE` raises it for
|
|
one call.
|
|
|
|
## Using it from a project
|
|
|
|
A project's own `run-android.sh` should build and install; the emulator half
|
|
belongs here:
|
|
|
|
SERIAL=$(emu up)
|
|
./gradlew :androidApp:assembleDebug
|
|
adb install -r <apk> # already aimed at this checkout's emulator
|