commit 5d77599f3fb66d55609ce184f57166bf73b7b3ee Author: iris <2+iris@noreply.localhost> Date: Sun Aug 30 13:06:17 2026 -0400 Collect the emulator tooling into one repo ui-trace, the adb wrapper and its device recorder were loose files under ~/.local, and each Android checkout carried its own copy of the same emulator boot sequence. This puts them together, with an install script that symlinks them back so editing the repo is editing what runs. Two things are new rather than moved. `emu` is the emulator lifecycle -- name, serial, list, up, down -- keyed on the AVD named after the enclosing checkout, which is the rule that lets several sessions work here at once; and `adb` now fills in `-s` from that same rule, because with two emulators attached a bare `adb shell pm list packages` comes back empty rather than failing, which reads as the app being uninstalled rather than the question being ambiguous. `emu up` refuses when the machine has no room. On 2026-08-30 an emulator started with 2.8 GB available invoked the OOM killer, and what it took was not the emulator that had just started: it walked the user slice and killed pipewire, dbus-broker and another session's emulator first. Co-Authored-By: Claude Opus 5 diff --git a/README.md b/README.md new file mode 100644 index 0000000..71b1eac --- /dev/null +++ b/README.md @@ -0,0 +1,89 @@ +# emulator-tools + +Everything this machine uses to drive an Android emulator, in one place: +which emulator a command means, whether there is room to start it, reading a +screen as text, and taking a screenshot that does not cost a thousand tokens. + + ./install.sh + +symlinks `bin/*` into `~/.local/bin` and `share/ui-trace` into +`~/.local/share`, so editing this repo is editing what runs. It keeps +anything it replaces as `.bak`. + +## What is here + +| | | +|---|---| +| `emu` | this checkout's emulator: `name`, `serial`, `list`, `up`, `down` | +| `adb` | adb, aimed at this checkout's emulator, with screenshots scaled | +| `ui-trace` | records a screen as text at 60Hz, and drives it | +| `lib/project-avd.sh` | the rules all three share | +| `share/ui-trace/` | the device recorder (`UiTrace.java`) and its build | + +## One emulator per checkout, named after it + +`emu` and `adb` both work out which AVD you mean from the enclosing git +checkout: in `~/repos/ai-app-2` that is `ai-app-2`. Nothing has to be +configured, and `AVD_NAME` overrides it where the guess is wrong. + +This is the rule that lets several agent sessions work here at once. The +emulator used to be the one resource that could not be parallelised: taking +it meant messaging the other sessions, waiting, and handing it back — and +installing an app onto somebody else's running emulator steals the foreground +from whatever they were looking at. + +It is also why `adb` fills in `-s`. With two emulators attached, a bare +`adb shell pm list packages` comes back **empty** rather than failing, which +reads as the app having been uninstalled rather than as the question being +ambiguous. An explicit `-s`, `ANDROID_SERIAL`, or a subcommand that is not +about one device (`adb devices`) all turn the defaulting off, and nothing is +guessed when this checkout's emulator is not running: adb's own error is +better than a wrapper picking a stranger's device. + +## Refusing to start one + +`emu up` checks `MemAvailable` first and refuses if starting an emulator +would leave the machine short, printing what is attached, what is large, and +what usually frees enough. `EMU_FORCE=1` overrides it. + +The reason is a real incident rather than tidiness. On 2026-08-30 an emulator +was started on this box with 2.8 GB available; the OOM killer ran, and what +it took was not the emulator that had just started — it walked the user slice +and killed pipewire, dbus-broker and *another session's* emulator first. The +cost of one too many lands on somebody else's work, minutes later, looking +like an unrelated crash. + +Measured numbers behind the defaults: a headless x86_64 AVD is about 3.8 GB +resident, this VM has 23 GB with a swap that is routinely full, two emulators +are comfortable, and three were not. + +## Reading a screen without screenshots + +`ui-trace` drives a small resident recorder on the device and samples the +accessibility tree at 60Hz, so an animation or a settling layout is visible +rather than happening between two samples. + + ui-trace record -d 3000 --do 'tap 540 800' -o /tmp/t.txt + ui-trace elements /tmp/t.txt # what is on screen, to pick from + ui-trace show /tmp/t.txt # what moved + ui-trace show /tmp/t.txt -m 'Send|Stop' --field box + +Bounds come out in **device** pixels, so they feed straight back into a +`--do tap`. Prefer this to a screenshot for anything positional: two buttons +that measured identically at 171x105 looked different in a screenshot, +because the filled one read as larger than the disabled grey one. Screenshots +are for appearance — colour, weight, whether it looks right. + +`adb exec-out screencap -p` still works and comes back scaled to 800px on its +long edge, which is ~380 tokens to read instead of ~1460 and still legible +for layout, contrast and small labels. `ADB_SCREENCAP_MAX_EDGE` raises it for +one call. + +## Using it from a project + +A project's own `run-android.sh` should build and install; the emulator half +belongs here: + + SERIAL=$(emu up) + ./gradlew :androidApp:assembleDebug + adb install -r # already aimed at this checkout's emulator diff --git a/bin/adb b/bin/adb new file mode 100755 index 0000000..87ea217 --- /dev/null +++ b/bin/adb @@ -0,0 +1,119 @@ +#!/bin/bash +# adb, aimed at this checkout's own emulator, with screenshots scaled down +# before anything reads them. +# +# This is a wrapper, not a separate screenshot command, because a separate +# command is a rule to remember and this has to hold for every agent on +# this machine without anybody opting in. `adb exec-out screencap -p` is +# the normal way to take a screenshot here, so that is the thing that has +# to come back small. +# +# What it costs to read one is set by pixels, not bytes: an image is +# charged at about (width x height) / 750 tokens, after being scaled to fit +# 1568px on its long edge. So a 1080x2424 phone screenshot is ~1460 tokens +# however well the PNG is compressed, and the only lever that moves is the +# size. At 800px on the long edge it is ~380 -- and a 356x800 frame is +# still legible enough to check layout, contrast, alignment and small +# labels, which is what these are looked at for. Raise it for one call with +# ADB_SCREENCAP_MAX_EDGE if something genuinely needs the detail. +# +# The other half is which device a call means. Several agent sessions share +# this machine, each with an emulator named after its checkout, and a bare +# `adb shell` with two of them attached does not fail cleanly: `adb shell pm +# list packages` comes back *empty*, which reads as the app having been +# uninstalled rather than as the question being ambiguous. So when a call +# does not name a device and this checkout's own emulator is running, this +# aims at it. Nothing is guessed when it is not running: adb's own error is +# better than a wrapper picking somebody else's phone. +# +# Everything that is not a screencap is handed straight through, unchanged, +# including its exit status -- a wrapper that breaks `adb install` to save +# tokens on screenshots would not survive the week. +set -euo pipefail + +. "$(dirname "$(readlink -f "$0")")/../lib/project-avd.sh" + +# See `android_adb`: found without consulting PATH, which is how this script +# was called and would be how it called itself for ever. +real=$(android_adb) || { + echo "adb: no platform-tools under any known SDK; set ANDROID_HOME" >&2 + exit 127 +} + +# Aim at this checkout's emulator, unless the caller has already said which +# device they mean or is asking something that is not about one. +# +# `-s` and friends are only valid before the subcommand, so the scan stops at +# the first word that is not one of adb's own options -- otherwise +# `adb shell echo -s` would look like a device selection. +target=() +if [ -z "${ANDROID_SERIAL:-}" ]; then + subcommand="" + named=false + for arg in "$@"; do + case "$arg" in + -s | -d | -e | -t | -H | -P | -L) + named=true + break + ;; + -*) ;; + *) + subcommand="$arg" + break + ;; + esac + done + case "$subcommand" in + # Not about one device: `devices` lists them all, and the rest talk to + # the adb server itself. Passing -s to these is at best ignored. + devices | start-server | kill-server | version | help | connect | disconnect) ;; + *) + if [ "$named" = false ]; then + serial=$(avd_serial "$real" "$(project_avd)" || true) + [ -n "$serial" ] && target=(-s "$serial") + fi + ;; + esac +fi + +screencap=false +for arg in "$@"; do + if [ "$arg" = screencap ]; then + screencap=true + break + fi +done + +# ffmpeg missing is a reason to hand over the full-size image, not to fail: +# the caller asked for a screenshot and is entitled to one. +if [ "$screencap" != true ] || ! command -v ffmpeg >/dev/null 2>&1; then + exec "$real" "${target[@]}" "$@" +fi + +max=${ADB_SCREENCAP_MAX_EDGE:-800} +tmp=$(mktemp -t adb-screencap.XXXXXX.png) +trap 'rm -f "$tmp"' EXIT +status=0 +"$real" "${target[@]}" "$@" >"$tmp" || status=$? + +# Only a PNG that actually arrived on stdout is ours to shrink. `adb shell +# screencap -p /sdcard/x.png` writes to the device and produces nothing +# here, and feeding that to ffmpeg would turn a working command into a +# failing one; so anything that is not a PNG goes back out byte for byte, +# with the real adb's exit status. +if [ "$status" -ne 0 ] || [ ! -s "$tmp" ] || + [ "$(head -c 8 "$tmp" | od -An -tx1 | tr -d ' \n')" != "89504e470d0a1a0a" ]; then + cat "$tmp" + exit "$status" +fi + +# Never enlarge (min(1,...)), keep the aspect ratio, and keep both edges +# even, which some encoders insist on. If ffmpeg cannot make sense of it, +# the original still goes out: a smaller screenshot is worth having, a +# missing one is not. +scale="min(1,$max/max(iw,ih))" +if ! ffmpeg -y -loglevel error -i "$tmp" \ + -vf "scale='trunc(iw*$scale/2)*2':'trunc(ih*$scale/2)*2':flags=area" \ + -f image2pipe -c:v png -; then + cat "$tmp" +fi diff --git a/bin/emu b/bin/emu new file mode 100755 index 0000000..3449aa9 --- /dev/null +++ b/bin/emu @@ -0,0 +1,187 @@ +#!/bin/bash +# This checkout's emulator: which one it is, whether there is room for it, +# and getting it up and down. +# +# The AVD is named after the enclosing git checkout and nobody has to say so +# (see `project_avd`). That is the whole point: several agent sessions work on +# this machine at once, and an emulator named after the checkout is one that +# cannot be somebody else's to install onto or take the foreground from. +# +# `up` refuses rather than tries when memory is short, which is not caution +# for its own sake. On 2026-08-30 an emulator started with 2.8 GB available +# invoked the OOM killer, and what it took was not the emulator that had just +# started: it walked the user slice and killed pipewire, dbus-broker and +# another session's running emulator first. The cost of one too many lands on +# somebody else's work, minutes later, looking like an unrelated crash. +set -euo pipefail + +. "$(dirname "$(readlink -f "$0")")/../lib/project-avd.sh" + +# What one costs, measured: 3.8 GB resident for a headless x86_64 AVD, plus +# what it grows into while an app builds and installs. +EMU_EXPECTED_MB=${EMU_EXPECTED_MB:-4200} +# What must be left afterwards. A machine at exactly zero swaps itself to a +# halt rather than failing cleanly. +EMU_HEADROOM_MB=${EMU_HEADROOM_MB:-1200} +# Two are comfortable on 23 GB; three is what invoked the OOM killer. +EMU_MAX_RUNNING=${EMU_MAX_RUNNING:-2} + +DEVICE_PROFILE=${DEVICE_PROFILE:-pixel_10} +SYSTEM_IMAGE=${SYSTEM_IMAGE:-system-images;android-36;google_apis;x86_64} + +sdk=$(android_sdk) || { + echo "emu: no Android SDK found; set ANDROID_HOME" >&2 + exit 127 +} +adb=$(android_adb) +avd=$(project_avd) +export ANDROID_AVD_HOME="${ANDROID_AVD_HOME:-$HOME/.android/avd}" + +usage() { + cat >&2 <<'USAGE' +usage: emu + + name the AVD this directory means + serial its adb serial, if it is running (exit 1 if not) + list every emulator attached, with its AVD and what it is costing + up start it, refusing if the machine has no room + down stop it + +Environment: AVD_NAME overrides the name, EMU_FORCE=1 overrides the memory +refusal, DEVICE_PROFILE and SYSTEM_IMAGE decide what `up` creates. +USAGE + exit 2 +} + +# Every emulator's serial, AVD and resident size -- the last being the number +# the refusal below is about, so it is worth seeing before asking for another. +cmd_list() { + found=false + while IFS=$'\t' read -r serial name; do + found=true + port=${serial#emulator-} + rss=$(ps -eo rss,args | awk -v p="-port $port" '$0 ~ p && $0 ~ /qemu-system/ {print int($1/1024); exit}') + printf ' %-16s %-24s %s\n' "$serial" "$name" "${rss:+$rss MB}" + done < <(running_avds "$adb") + [ "$found" = true ] || echo " (none attached)" +} + +# Whether starting one now is likely to cost somebody else their work. +# +# Reported in full rather than as a bare refusal: the caller cannot see this +# machine, and "not enough memory" without the numbers or a way forward is a +# message that gets overridden blind. +check_room() { + available=$(mem_available_mb) + needed=$((EMU_EXPECTED_MB + EMU_HEADROOM_MB)) + running=$(running_avds "$adb" | wc -l) + problem="" + if [ "$running" -ge "$EMU_MAX_RUNNING" ]; then + problem="$running emulator(s) are already running, and $EMU_MAX_RUNNING is as many as this machine takes" + elif [ "$available" -lt "$needed" ]; then + problem="$available MB available, and one of these needs about $EMU_EXPECTED_MB MB plus $EMU_HEADROOM_MB MB of headroom" + fi + [ -z "$problem" ] && return 0 + + if [ "${EMU_FORCE:-}" = 1 ]; then + echo "emu: $problem -- starting anyway because EMU_FORCE=1" >&2 + return 0 + fi + { + echo "emu: not starting '$avd' -- $problem." + echo + echo " Emulators attached:" + cmd_list + echo + echo " Biggest processes:" + ps -eo rss,comm --sort=-rss | awk 'NR>1 && NR<=6 {printf " %-24s %6.0f MB\n", $2, $1/1024}' + echo + echo " What usually frees enough:" + echo " gradle --stop (an idle Gradle daemon holds 1-2 GB between builds)" + echo " emu down (in the checkout that owns an emulator you are done with)" + echo " ask the session that owns one -- they are named after their checkout" + echo " EMU_FORCE=1 emu up (if you have decided the numbers above are wrong)" + } >&2 + exit 1 +} + +cmd_up() { + if serial=$(avd_serial "$adb" "$avd"); then + echo "emu: '$avd' is already running ($serial)" >&2 + echo "$serial" + return 0 + fi + check_room + + if [ ! -f "$ANDROID_AVD_HOME/$avd.ini" ]; then + echo "emu: creating AVD '$avd' ($DEVICE_PROFILE, $SYSTEM_IMAGE)" >&2 + "$sdk/cmdline-tools/latest/bin/sdkmanager" "emulator" "$SYSTEM_IMAGE" >/dev/null || + echo "emu: sdkmanager could not confirm the system image; trying anyway" >&2 + echo no | "$sdk/cmdline-tools/latest/bin/avdmanager" create avd \ + -n "$avd" -k "$SYSTEM_IMAGE" --device "$DEVICE_PROFILE" --sdcard 512M >&2 + fi + + # The host keyboard, so anything with a text field can be typed into + # rather than tapped out on the on-screen one. + config="$ANDROID_AVD_HOME/$avd.avd/config.ini" + if [ -f "$config" ] && ! grep -qx 'hw.keyboard=yes' "$config"; then + grep -v '^hw\.keyboard=' "$config" >"$config.tmp" + echo "hw.keyboard=yes" >>"$config.tmp" + mv "$config.tmp" "$config" + fi + + # A stray process for this AVD that never registered with adb. The + # bracketed first character keeps the pattern from matching the shell + # running this script, which would kill it mid-flight. + pkill -f "[e]mulator.*-avd $avd" >/dev/null 2>&1 || true + + log="/tmp/$avd-emulator.log" + : >"$log" + if [ -n "${DISPLAY:-}" ] || [ -n "${WAYLAND_DISPLAY:-}" ]; then + echo "emu: starting '$avd' with GPU acceleration" >&2 + "$sdk/emulator/emulator" -avd "$avd" -gpu host -no-audio >"$log" 2>&1 & + else + echo "emu: no display -- starting '$avd' headless" >&2 + "$sdk/emulator/emulator" -avd "$avd" -gpu swiftshader_indirect -no-audio -no-window \ + >"$log" 2>&1 & + fi + pid=$! + + for _ in $(seq 150); do + if ! kill -0 "$pid" 2>/dev/null; then + echo "emu: the emulator exited before it finished booting:" >&2 + tail -20 "$log" >&2 + exit 1 + fi + if serial=$(avd_serial "$adb" "$avd"); then + booted=$("$adb" -s "$serial" shell getprop sys.boot_completed 2>/dev/null | tr -d '\r') + if [ "$booted" = 1 ]; then + echo "emu: '$avd' booted ($serial)" >&2 + echo "$serial" + return 0 + fi + fi + sleep 2 + done + echo "emu: '$avd' did not finish booting in five minutes; log at $log" >&2 + exit 1 +} + +cmd_down() { + if ! serial=$(avd_serial "$adb" "$avd"); then + echo "emu: '$avd' is not running" >&2 + return 0 + fi + "$adb" -s "$serial" emu kill >/dev/null + rm -f "${XDG_RUNTIME_DIR:-/tmp}/emulator-tools/$avd.serial" + echo "emu: stopped '$avd' ($serial)" >&2 +} + +case "${1:-}" in + name) echo "$avd" ;; + serial) avd_serial "$adb" "$avd" || { echo "emu: '$avd' is not running" >&2; exit 1; } ;; + list) cmd_list ;; + up) cmd_up ;; + down) cmd_down ;; + *) usage ;; +esac diff --git a/bin/ui-trace b/bin/ui-trace new file mode 100755 index 0000000..3a59415 --- /dev/null +++ b/bin/ui-trace @@ -0,0 +1,383 @@ +#!/usr/bin/env python3 +"""Record what an Android screen does, as text, and ask questions of it later. + +Why this exists. `adb shell uiautomator dump` is the usual way to read a +screen's layout, and it costs about two seconds a call -- so an animation, a +scroll or a settling layout happens entirely in the gap between two samples, +and the only thing it can answer is what the screen looked like once it had +stopped. This drives a small resident recorder on the device instead (see +UiTrace.java beside this script), which connects to the accessibility service +once and then samples at whatever rate is asked for; 60Hz is comfortable. + +The other half is that a screen has a few hundred nodes and a question is +almost always about two or three of them. So recording and reading are separate +steps: `record` saves everything, and `show` selects. A trace can be +re-interrogated with a different selector without touching the device again, +which matters because the interesting question is usually the one you think of +after seeing the first answer. + + ui-trace record -d 3000 --do 'tap 540 800' -o /tmp/t.txt + ui-trace elements /tmp/t.txt # what is on screen, to pick from + ui-trace show /tmp/t.txt # what moved (the default question) + ui-trace show /tmp/t.txt -m 'Called|ask' # a timeline for those + +Nothing here is specific to any app. +""" + +import argparse +import os +import re +import subprocess +import sys +import tempfile +from pathlib import Path + +SHARE = Path.home() / ".local/share/ui-trace" +DEVICE_JAR = "/data/local/tmp/uitrace.jar" + + +def find_adb(): + """The real adb, without assuming PATH has been set up for Android.""" + for candidate in [ + os.environ.get("ADB"), + "adb", + os.environ.get("ANDROID_HOME", "") + "/platform-tools/adb", + os.environ.get("ANDROID_SDK_ROOT", "") + "/platform-tools/adb", + str(Path.home() / "Android/Sdk/platform-tools/adb"), + ]: + if not candidate or candidate.startswith("/platform-tools"): + continue + which = subprocess.run(["sh", "-c", f"command -v {candidate!r}"], + capture_output=True, text=True) + if which.returncode == 0: + return which.stdout.strip() + sys.exit("ui-trace: no adb found; set ANDROID_HOME or ADB") + + +def ensure_jar(): + """Build the device recorder if it is missing or older than its source.""" + jar, source = SHARE / "uitrace.jar", SHARE / "UiTrace.java" + if not source.exists(): + sys.exit(f"ui-trace: recorder source missing at {source}") + if not jar.exists() or jar.stat().st_mtime < source.stat().st_mtime: + print("ui-trace: building the device recorder...", file=sys.stderr) + subprocess.run([str(SHARE / "build.sh")], check=True, + stdout=subprocess.DEVNULL) + return jar + + +# --- recording ------------------------------------------------------------- + +def record(args): + jar = ensure_jar() + adb = find_adb() + target = ["-s", args.serial] if args.serial else [] + pushed = subprocess.run([adb, *target, "push", str(jar), DEVICE_JAR], + capture_output=True, text=True) + if pushed.returncode != 0: + # Two emulators running is the usual cause, and adb says so plainly; there is + # nothing to add to its own message except which flag fixes it. + sys.exit(f"ui-trace: {pushed.stderr.strip() or 'adb push failed'}" + f"{'' if target else ' (use -s SERIAL to pick a device)'}") + remote = f"CLASSPATH={DEVICE_JAR} app_process /system/bin UiTrace" + remote += f" --duration {args.duration} --interval {args.interval}" + for step in args.do: + remote += f" --do {step!r}" + out = Path(args.out) if args.out else Path( + tempfile.mkstemp(prefix="ui-trace-", suffix=".txt")[1]) + with out.open("w") as handle: + result = subprocess.run([adb, *target, "shell", remote], stdout=handle, + stderr=subprocess.STDOUT) + text = out.read_text() + if result.returncode != 0 or "# error" in text: + print(text[:2000], file=sys.stderr) + sys.exit("ui-trace: the device recorder failed") + frames, _ = parse(out) + print(f"ui-trace: {len(frames)} frames over {frames[-1][0] if frames else 0}ms " + f"-> {out}", file=sys.stderr) + return out + + +# --- reading --------------------------------------------------------------- + +class Node: + __slots__ = ("depth", "box", "flags", "cls", "rid", "text", "desc", "label") + + def __init__(self, parts): + self.depth = int(parts[1]) + self.box = tuple(int(v) for v in parts[2:6]) + self.flags, self.cls, self.rid, self.text, self.desc = parts[6:11] + self.label = (self.text or self.desc or self.rid.split("/")[-1] + or self.cls or "?") + + def haystack(self): + return "\t".join((self.label, self.rid, self.cls, self.text, self.desc)) + + +def parse(path): + """A trace file into (frames, actions). A frame is (t_ms, [Node]). + + A missing or unreadable trace exits with one line rather than a stack trace: the + caller is usually reading this in a terminal or a context window, and a traceback + says nothing a sentence cannot. + """ + frames, actions, current = [], [], None + try: + text = Path(path).read_text() + except OSError as problem: + sys.exit(f"ui-trace: cannot read {path}: {problem.strerror}") + for line in text.splitlines(): + if line.startswith("F "): + _, _, at = line.split(" ", 2) + current = (int(at), []) + frames.append(current) + elif line.startswith("N\t"): + parts = line.split("\t") + # A label may itself be empty, so pad rather than trusting the count. + parts += [""] * (11 - len(parts)) + if current: + current[1].append(Node(parts)) + elif line.startswith("A "): + at, what = line[2:].split("\t", 1) + actions.append((int(at), what)) + return frames, actions + + +def keyed(nodes, pattern): + """Selected nodes by a stable key: the label, plus an index when repeated.""" + seen, out = {}, {} + for node in nodes: + if pattern and not pattern.search(node.haystack()): + continue + count = seen.get(node.label, 0) + seen[node.label] = count + 1 + out[node.label if count == 0 else f"{node.label} #{count + 1}"] = node + return out + + +def sizes(frames, pattern): + """{key: set of (width, height)} -- see [summarise] for why this is tracked.""" + seen = {} + for _, nodes in frames: + for key, node in keyed(nodes, pattern).items(): + box = node.box + seen.setdefault(key, set()).add((box[2] - box[0], box[3] - box[1])) + return seen + + +def series(frames, pattern, field): + """{key: [(t, value or None)]} over every frame, for the chosen field.""" + pick = {"top": lambda b: b[1], "bottom": lambda b: b[3], + "left": lambda b: b[0], "right": lambda b: b[2], + "box": lambda b: b}[field] + tracks, order = {}, [] + for index, (at, nodes) in enumerate(frames): + present = keyed(nodes, pattern) + for key, node in present.items(): + if key not in tracks: + # Backfilled as absent, so every track is one value per frame and + # a row can be read across by index. An element that appears + # part-way through is the common case, not the exotic one. + tracks[key] = [(frames[i][0], None) for i in range(index)] + order.append(key) + tracks[key].append((at, pick(node.box))) + for key in tracks: + if key not in present: + tracks[key].append((at, None)) + return order, tracks + + +def render(value): + if value is None: + return "-" + if isinstance(value, tuple): + return "{},{}..{},{}".format(*value) + return str(value) + + +def cap(rows, limit): + """The first [limit] rows, saying how many were left out. + + Every listing here shortens the same way, because the caller is usually an + agent with a context window and a screen has hundreds of nodes. + """ + if limit <= 0 or len(rows) <= limit: + return rows + print(f" (showing {limit} of {len(rows)}; --limit 0 for all)") + return rows[:limit] + + +def cmd_elements(args): + frames, _ = parse(args.trace) + counts = {} + for _, nodes in frames: + for node in nodes: + # A node whose only name is its class name names nothing -- a screen has + # hundreds of those and listing them is the noise this tool exists to cut. + if not args.all and not (node.text or node.desc or node.rid): + continue + entry = counts.setdefault(node.label, [0, node.cls, node.rid, node.flags]) + entry[0] += 1 + ranked = sorted(counts.items(), key=lambda kv: -kv[1][0]) + print(f"{len(counts)} named labels over {len(frames)} frames" + + ("" if args.all else " (--all also lists unnamed structure)")) + for label, (count, cls, rid, flags) in cap(ranked, args.limit): + mark = f" [{flags}]" if flags != "-" else "" + print(f" {count:5d}x {label[:44]:<44} {cls}{mark}" + + (f" {rid.split('/')[-1]}" if rid else "")) + + +def cmd_show(args): + frames, actions = parse(args.trace) + if not frames: + sys.exit("ui-trace: no frames in that trace") + pattern = re.compile(args.match) if args.match else None + order, tracks = series(frames, pattern, args.field) + if not pattern: + summarise(order, tracks, sizes(frames, pattern), actions, args) + return + timeline(frames, order, tracks, actions, args) + + +def summarise(order, tracks, shapes, actions, args): + """What moved -- the question a trace is usually recorded to answer. + + Everything that held still is dropped, because a screen is mostly things + that held still and reading them is the cost this tool exists to avoid. + """ + for at, what in actions: + print(f" action t={at}ms {what}") + moved = [] + for key in order: + points = [(t, v) for t, v in tracks[key] if v is not None] + if not points: + continue + values = [v for _, v in points] + if len(set(values)) > 1: + moved.append((key, points)) + total = len(order) + print(f"{len(moved)} of {total} elements changed {args.field} during the trace") + for key, points in cap(moved, args.limit): + values = [v for _, v in points] + changes = [(t, v) for i, (t, v) in enumerate(points) + if i == 0 or v != points[i - 1][1]] + span = f"t={changes[1][0]}..{changes[-1][0]}ms" if len(changes) > 1 else "" + net = values[-1] - values[0] if not isinstance(values[0], tuple) else None + arrow = f"{render(values[0])} -> {render(values[-1])}" + delta = f" ({net:+d})" if net is not None else "" + # A label is only a name, and two different nodes can share one -- a text + # and the padded control wrapping it both read as their text. When the + # size changes too, the two readings are probably not the same node, and + # the move is partly the difference between them rather than travel. + shape = shapes.get(key, set()) + note = f" [size changed {len(shape)}x -- may be different nodes]" if len(shape) > 1 else "" + print(f" {key[:40]:<40} {arrow}{delta} {len(changes) - 1} steps {span}{note}") + if not moved: + print(" (nothing moved)") + + +def timeline(frames, order, tracks, actions, args): + """A row per change, not a row per frame. + + Consecutive frames that say the same thing are one row with a time range: + at 60Hz a settled second is sixty identical rows, and the reader wants the + moment it stopped being settled. + """ + # "Nothing matched" and "matched but never moved" are different answers, and + # an empty table says neither of them. + if not order: + print(f" nothing matched /{args.match}/ in any of {len(frames)} frames." + f" Try `ui-trace elements` to see what is there.") + return + if len(order) > args.max_columns: + print(f" {len(order)} elements match, which is too many to read across." + f" Narrowing to the first {args.max_columns}; use a tighter -m," + f" or --max-columns to widen.") + order = order[:args.max_columns] + width = max(12, max((len(k) for k in order), default=12)) + width = min(width, 24) + columns = [k[:width] for k in order] + print(" " + "t (ms)".rjust(13) + " | " + " | ".join(c.ljust(width) for c in columns)) + print(" " + "-" * 13 + "-+-" + "-+-".join("-" * width for _ in columns)) + pending = list(actions) + previous, start, last = None, None, None + rows = [[t for t, _ in tracks[order[0]]]] if order else [[]] + + def flush(): + if previous is None: + return + label = f"{start}" if start == last else f"{start}-{last}" + print(" " + label.rjust(13) + " | " + + " | ".join(render(v).ljust(width) for v in previous)) + + rows_out = 0 + for index, (at, _) in enumerate(frames): + values = tuple(tracks[k][index][1] for k in order) + while pending and pending[0][0] <= at: + flush() + previous = None + print(f" >>> {pending[0][1]} @ t={pending[0][0]}ms") + pending.pop(0) + if values != previous: + flush() + rows_out += 1 + if args.limit > 0 and rows_out > args.limit: + print(f" ... more changes after t={at}ms (--limit 0 for all)") + return + previous, start = values, at + last = at + flush() + for at, what in pending: + print(f" >>> {what} @ t={at}ms") + + +def main(): + parser = argparse.ArgumentParser( + prog="ui-trace", description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + subs = parser.add_subparsers(dest="command", required=True) + + rec = subs.add_parser("record", help="record a trace from a device") + rec.add_argument("-s", "--serial", help="adb device serial") + rec.add_argument("-d", "--duration", type=int, default=3000, help="ms to record") + rec.add_argument("-i", "--interval", type=int, default=16, + help="ms between samples (0 = as fast as possible)") + rec.add_argument("--do", action="append", default=[], metavar="ACTION", + help="'wait MS' | 'tap X Y' | 'swipe X1 Y1 X2 Y2 [MS]', repeatable") + rec.add_argument("-o", "--out", help="where to save the trace") + rec.add_argument("-m", "--match", help="show a timeline for these afterwards") + rec.add_argument("--field", default="top", + choices=["top", "bottom", "left", "right", "box"]) + rec.add_argument("--max-columns", type=int, default=6) + rec.add_argument("--limit", type=int, default=10, help="max rows printed; 0 for all") + + for name, help_text in [("show", "what moved, or a timeline with -m"), + ("elements", "every label in a trace, to pick from")]: + sub = subs.add_parser(name, help=help_text) + sub.add_argument("trace") + if name == "show": + sub.add_argument("-m", "--match", help="regex over label, id, class, text") + sub.add_argument("--field", default="top", + choices=["top", "bottom", "left", "right", "box"]) + sub.add_argument("--max-columns", type=int, default=6) + sub.add_argument("--limit", type=int, default=10, + help="max rows printed; 0 for all") + else: + sub.add_argument("--limit", type=int, default=20, + help="max rows printed; 0 for all") + sub.add_argument("--all", action="store_true", + help="include nodes with no name of their own") + + args = parser.parse_args() + if args.command == "record": + out = record(args) + args.trace = out + cmd_show(args) + elif args.command == "show": + cmd_show(args) + else: + cmd_elements(args) + + +if __name__ == "__main__": + main() diff --git a/install.sh b/install.sh new file mode 100755 index 0000000..7ebec9c --- /dev/null +++ b/install.sh @@ -0,0 +1,55 @@ +#!/bin/sh +# Puts these tools where a shell and an agent will find them, by symlink. +# +# Symlinks rather than copies so that editing the repo is editing what runs -- +# the alternative is a second copy that drifts, which is exactly what this +# repo was made to end (ui-trace, the adb wrapper and three near-identical +# emulator boot sequences all lived in different places). +# +# Idempotent: run it again after a pull, or after adding a command. +set -eu + +here=$(cd "$(dirname "$0")" && pwd) +bindir=${BINDIR:-$HOME/.local/bin} +sharedir=${SHAREDIR:-$HOME/.local/share} + +mkdir -p "$bindir" "$sharedir" + +for target in "$here"/bin/*; do + name=$(basename "$target") + link="$bindir/$name" + # A real file here is the pre-repo copy of one of these tools. Kept, once, + # under .bak: it is somebody's working state until they have looked at it, + # and a tool that silently deletes the thing it replaces is one nobody can + # undo. + if [ -e "$link" ] && [ ! -L "$link" ]; then + echo "install: keeping the existing $name as $name.bak" + mv "$link" "$link.bak" + fi + ln -sfn "$target" "$link" + echo "install: $link -> $target" +done + +# The device recorder's source and build script, which ui-trace reads from a +# fixed path and rebuilds when the source is newer than the jar. Linked as a +# directory so a rebuild lands in the repo rather than beside it. +if [ -e "$sharedir/ui-trace" ] && [ ! -L "$sharedir/ui-trace" ]; then + echo "install: keeping the existing share/ui-trace as ui-trace.bak" + mv "$sharedir/ui-trace" "$sharedir/ui-trace.bak" +fi +ln -sfn "$here/share/ui-trace" "$sharedir/ui-trace" +echo "install: $sharedir/ui-trace -> $here/share/ui-trace" + +case ":$PATH:" in + *":$bindir:"*) ;; + *) echo "install: note -- $bindir is not on PATH" ;; +esac + +# The adb wrapper only wins if it comes before the SDK's own on PATH. Said +# rather than fixed: which shell profile to edit is not this script's to +# decide, and getting it wrong is a broken login shell. +real=$(command -v adb 2>/dev/null || true) +if [ "$real" != "$bindir/adb" ]; then + echo "install: note -- 'adb' resolves to $real, not $bindir/adb;" + echo " put $bindir before the SDK's platform-tools on PATH." +fi diff --git a/lib/project-avd.sh b/lib/project-avd.sh new file mode 100644 index 0000000..ed53c4a --- /dev/null +++ b/lib/project-avd.sh @@ -0,0 +1,96 @@ +# Which emulator belongs to the directory you are standing in, and whether +# there is room to start it. Sourced by every command in this repo. +# +# One file rather than the same three functions copied into each script: the +# rule about *which* emulator a call means has to be the same for `adb`, +# `ui-trace` and `emu`, or the tool that reads the screen and the tool that +# taps it can end up on different devices. + +# The AVD for a directory: the basename of the enclosing git checkout. +# +# Derived rather than configured, because the point is that nobody has to say +# it. Several agent sessions work here at once out of ~/repos/ai-app, +# ~/repos/ai-app-2 and ~/repos/dev-updater, and an emulator named after the +# checkout is one that cannot be somebody else's. Override with AVD_NAME for +# the case this cannot guess -- a worktree meant to share its parent's. +project_avd() { + if [ -n "${AVD_NAME:-}" ]; then + echo "$AVD_NAME" + return + fi + root=$(git rev-parse --show-toplevel 2>/dev/null || true) + basename "${root:-$PWD}" +} + +# Every emulator attached right now, as "serialavd" lines. +# +# Takes the real adb rather than finding one, because its caller is usually +# the adb wrapper and a wrapper that calls itself never returns. +running_avds() { + _adb=$1 + "$_adb" devices 2>/dev/null | awk '$2 == "device" {print $1}' | while read -r serial; do + name=$("$_adb" -s "$serial" emu avd name 2>/dev/null | head -n1 | tr -d '\r') + [ -n "$name" ] && printf '%s\t%s\n' "$serial" "$name" + done +} + +# The serial AVD "$2" is on, or nothing. +# +# Cached, because asking costs one console round trip per attached device and +# this runs on every single adb call -- which includes a `tap` inside a loop. +# The cache is checked against `adb devices` first, so an emulator that has +# gone away cannot leave a stale serial behind; that check is one round trip +# whatever the answer. +avd_serial() { + _adb=$1 + _avd=$2 + _dir="${XDG_RUNTIME_DIR:-/tmp}/emulator-tools" + _cache="$_dir/$_avd.serial" + if [ -r "$_cache" ]; then + _cached=$(cat "$_cache") + if "$_adb" devices 2>/dev/null | awk '$2 == "device" {print $1}' | + grep -qx "$_cached"; then + echo "$_cached" + return 0 + fi + fi + _found=$(running_avds "$_adb" | awk -F'\t' -v want="$_avd" '$2 == want {print $1; exit}') + [ -z "$_found" ] && return 1 + mkdir -p "$_dir" && printf '%s' "$_found" >"$_cache" 2>/dev/null || true + echo "$_found" +} + +# What the kernel thinks is available, in MiB. MemAvailable rather than +# MemFree: free memory on this box is mostly page cache, and refusing to start +# an emulator because the cache is warm would refuse always. +mem_available_mb() { + awk '/^MemAvailable:/ {print int($2 / 1024)}' /proc/meminfo +} + +# The Android SDK, without consulting PATH. +# +# PATH is how a wrapper here got called, so searching it again is how a +# wrapper calls itself for ever. The ambient $ANDROID_HOME on this machine +# still points at a root-owned /opt/android-sdk with no platform-tools, +# cmdline-tools or emulator under it, which is why the user-owned SDK is +# checked before it rather than after. +android_sdk() { + for dir in \ + "$HOME/Android/Sdk" \ + "${ANDROID_HOME:-}" \ + "${ANDROID_SDK_ROOT:-}" \ + /opt/android-sdk; do + if [ -n "$dir" ] && [ -d "$dir/platform-tools" ]; then + echo "$dir" + return 0 + fi + done + return 1 +} + +# The real adb, as opposed to whichever wrapper is asking. +android_adb() { + sdk=$(android_sdk) || return 1 + [ -x "$sdk/platform-tools/adb" ] || return 1 + echo "$sdk/platform-tools/adb" +} diff --git a/share/ui-trace/UiTrace.java b/share/ui-trace/UiTrace.java new file mode 100644 index 0000000..9e9bc39 --- /dev/null +++ b/share/ui-trace/UiTrace.java @@ -0,0 +1,230 @@ +// The on-device half of ui-trace: samples the accessibility tree in a loop and +// prints one text frame per sample. +// +// This exists because `uiautomator dump` costs about two seconds per call -- +// almost all of it starting a JVM and connecting to the accessibility service -- +// which is half a hertz, so an animation happens entirely between two samples. +// Connecting once and then sampling in a loop is the whole trick; everything +// else here is bookkeeping around it. +// +// Runs under app_process as the shell user, the same way /system/bin/uiautomator +// does. Three pieces of the connection are hidden API and are reached by +// reflection: android.app.UiAutomationConnection, the (Looper, connection) +// constructor of UiAutomation, and its connect() method. There is no public way +// for a shell process to obtain a UiAutomation -- the public entry points all +// require an Instrumentation and therefore an app -- and this is the same path +// the platform's own uiautomator command takes. + +import android.accessibilityservice.AccessibilityServiceInfo; +import android.app.UiAutomation; +import android.graphics.Rect; +import android.os.HandlerThread; +import android.os.Looper; +import android.os.SystemClock; +import android.view.InputDevice; +import android.view.MotionEvent; +import android.view.accessibility.AccessibilityNodeInfo; +import java.io.PrintWriter; +import java.lang.reflect.Constructor; +import java.util.ArrayList; +import java.util.List; + +public final class UiTrace { + private static final int MAX_DEPTH = 60; + + private final UiAutomation automation; + private final PrintWriter out; + private final long origin = SystemClock.uptimeMillis(); + + private UiTrace(UiAutomation automation, PrintWriter out) { + this.automation = automation; + this.out = out; + } + + // prepareMainLooper() is deprecated because an app never needs it -- the + // Android environment makes the main looper. This is not an app: app_process + // starts a bare VM, so there is no main looper and no supported way to ask + // for one. The framework needs it anyway (see below), so the deprecated call + // is the only route, and it is scoped to this method alone. + @SuppressWarnings("deprecation") + public static void main(String[] args) { + // The accessibility client builds a Handler on the main looper the first + // time a callback arrives, and a bare app_process has no main looper, so + // connecting from here died with an NPE inside the framework and the + // shell reported only "Killed". So the main thread prepares the looper + // and then does nothing but run it; the trace itself is a worker. + Looper.prepareMainLooper(); + Thread worker = new Thread(() -> record(args), "ui-trace"); + worker.start(); + Looper.loop(); + } + + private static void record(String[] args) { + PrintWriter out = new PrintWriter(new java.io.BufferedWriter(new java.io.OutputStreamWriter(System.out), 1 << 16)); + try { + long duration = 3000; + long interval = 0; + List script = new ArrayList<>(); + for (int i = 0; i < args.length; i++) { + switch (args[i]) { + case "--duration" -> duration = Long.parseLong(args[++i]); + case "--interval" -> interval = Long.parseLong(args[++i]); + case "--do" -> script.add(args[++i].trim().split("\\s+")); + default -> throw new IllegalArgumentException("unknown option: " + args[i]); + } + } + UiTrace trace = new UiTrace(connect(), out); + trace.run(duration, interval, script); + out.println("# done"); + } catch (Throwable problem) { + out.println("# error " + problem); + for (StackTraceElement frame : problem.getStackTrace()) out.println("# at " + frame); + out.flush(); + System.exit(1); + } + out.flush(); + // The accessibility connection keeps non-daemon threads alive, so a + // returning main() would hang here rather than ending the trace. + System.exit(0); + } + + private static UiAutomation connect() throws Exception { + HandlerThread thread = new HandlerThread("ui-trace"); + thread.start(); + Object connection = Class.forName("android.app.UiAutomationConnection") + .getDeclaredConstructor().newInstance(); + Constructor ctor = UiAutomation.class.getDeclaredConstructor( + Looper.class, Class.forName("android.app.IUiAutomationConnection")); + ctor.setAccessible(true); + UiAutomation automation = (UiAutomation) ctor.newInstance(thread.getLooper(), connection); + UiAutomation.class.getDeclaredMethod("connect").invoke(automation); + // Without these two the tree is the one a screen reader would use: + // decorative and off-stage nodes are dropped and view ids are withheld, + // and both are things a layout question is usually about. + AccessibilityServiceInfo info = automation.getServiceInfo(); + info.flags |= AccessibilityServiceInfo.FLAG_INCLUDE_NOT_IMPORTANT_VIEWS + | AccessibilityServiceInfo.FLAG_REPORT_VIEW_IDS; + automation.setServiceInfo(info); + return automation; + } + + private void run(long duration, long interval, List script) { + // The actions run beside the sampling rather than between samples: an + // action that blocked the loop would leave a hole in the recording at + // exactly the moment being recorded. + Thread actor = new Thread(() -> { + for (String[] step : script) { + try { + perform(step); + } catch (Exception problem) { + synchronized (out) { out.println("# action failed: " + problem); } + } + } + }, "ui-trace-actions"); + actor.setDaemon(true); + actor.start(); + + long end = origin + duration; + int frame = 0; + while (SystemClock.uptimeMillis() < end) { + long at = SystemClock.uptimeMillis() - origin; + AccessibilityNodeInfo root = automation.getRootInActiveWindow(); + synchronized (out) { + out.println("F " + frame + " " + at); + if (root == null) out.println("# no active window"); + else emit(root, 0); + out.flush(); + } + frame++; + if (interval > 0) { + long slack = interval - (SystemClock.uptimeMillis() - origin - at); + if (slack > 0) SystemClock.sleep(slack); + } + } + } + + private void emit(AccessibilityNodeInfo node, int depth) { + if (node == null || depth > MAX_DEPTH) return; + Rect box = new Rect(); + node.getBoundsInScreen(box); + // Tab-separated with the free text last, because a label may contain + // anything at all including the separator of a friendlier format. + StringBuilder line = new StringBuilder("N\t"); + line.append(depth).append('\t') + .append(box.left).append('\t').append(box.top).append('\t') + .append(box.right).append('\t').append(box.bottom).append('\t') + .append(flags(node)).append('\t') + .append(shortName(node.getClassName())).append('\t') + .append(clean(node.getViewIdResourceName())).append('\t') + .append(clean(node.getText())).append('\t') + .append(clean(node.getContentDescription())); + out.println(line); + for (int i = 0; i < node.getChildCount(); i++) emit(node.getChild(i), depth + 1); + } + + private static String flags(AccessibilityNodeInfo node) { + StringBuilder marks = new StringBuilder(); + if (node.isClickable()) marks.append('c'); + if (node.isScrollable()) marks.append('s'); + if (node.isFocused()) marks.append('f'); + if (!node.isEnabled()) marks.append('d'); + if (!node.isVisibleToUser()) marks.append('h'); + return marks.length() == 0 ? "-" : marks.toString(); + } + + private static String shortName(CharSequence name) { + String text = clean(name); + int dot = text.lastIndexOf('.'); + return dot < 0 ? text : text.substring(dot + 1); + } + + private static String clean(CharSequence value) { + if (value == null) return ""; + return value.toString().replace('\t', ' ').replace('\n', '⏎').replace('\r', ' '); + } + + private void perform(String[] step) throws Exception { + long at = SystemClock.uptimeMillis() - origin; + synchronized (out) { out.println("A " + at + "\t" + String.join(" ", step)); out.flush(); } + switch (step[0]) { + case "wait" -> SystemClock.sleep(Long.parseLong(step[1])); + case "tap" -> tap(Integer.parseInt(step[1]), Integer.parseInt(step[2])); + case "swipe" -> swipe(Integer.parseInt(step[1]), Integer.parseInt(step[2]), + Integer.parseInt(step[3]), Integer.parseInt(step[4]), + step.length > 5 ? Long.parseLong(step[5]) : 300); + default -> throw new IllegalArgumentException("unknown action: " + step[0]); + } + } + + private void send(int action, long down, int x, int y) { + MotionEvent event = MotionEvent.obtain(down, SystemClock.uptimeMillis(), action, x, y, 0); + event.setSource(InputDevice.SOURCE_TOUCHSCREEN); + automation.injectInputEvent(event, true); + event.recycle(); + } + + private void tap(int x, int y) { + long down = SystemClock.uptimeMillis(); + send(MotionEvent.ACTION_DOWN, down, x, y); + SystemClock.sleep(60); + send(MotionEvent.ACTION_UP, down, x, y); + } + + // Moves are emitted on a fixed 10ms cadence so the velocity tracker behind a + // fling sees a real gesture. `input swipe` interpolates over far fewer + // points, which is why it so often scrolls without ever flinging. + private void swipe(int x1, int y1, int x2, int y2, long millis) { + long down = SystemClock.uptimeMillis(); + send(MotionEvent.ACTION_DOWN, down, x1, y1); + int steps = (int) Math.max(2, millis / 10); + for (int i = 1; i <= steps; i++) { + float part = (float) i / steps; + long due = down + (long) (millis * part); + long slack = due - SystemClock.uptimeMillis(); + if (slack > 0) SystemClock.sleep(slack); + send(MotionEvent.ACTION_MOVE, down, Math.round(x1 + (x2 - x1) * part), + Math.round(y1 + (y2 - y1) * part)); + } + send(MotionEvent.ACTION_UP, down, x2, y2); + } +} diff --git a/share/ui-trace/build.sh b/share/ui-trace/build.sh new file mode 100755 index 0000000..351cfdb --- /dev/null +++ b/share/ui-trace/build.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# Builds uitrace.jar from UiTrace.java. Run after editing the recorder; ui-trace +# also runs this by itself when the jar is missing or older than the source. +set -euo pipefail +cd "$(dirname "$0")" +# The ambient ANDROID_HOME on this machine points at a root-owned SDK that has +# no build-tools under it, so an SDK is only accepted once d8 is confirmed in it. +d8="" +for root in "$HOME/Android/Sdk" "${ANDROID_SDK_ROOT:-}" "${ANDROID_HOME:-}"; do + [ -n "$root" ] || continue + found=$(ls -d "$root"/build-tools/*/d8 2>/dev/null | sort -V | tail -1) + if [ -n "$found" ]; then + d8=$found + platform=$(ls -d "$root"/platforms/android-* 2>/dev/null | sort -V | tail -1) + break + fi +done +if [ -z "$d8" ] || [ -z "${platform:-}" ]; then + echo "ui-trace: no Android SDK with both build-tools and a platform" >&2 + exit 1 +fi + +rm -rf classes && mkdir -p classes +javac --release 17 -Xlint:all -Werror -cp "$platform/android.jar" -d classes UiTrace.java +"$d8" --min-api 30 --output . classes/UiTrace.class +python3 -c 'import zipfile,sys; z=zipfile.ZipFile("uitrace.jar","w",zipfile.ZIP_DEFLATED); z.write("classes.dex"); z.close()' +rm -rf classes classes.dex +echo "built $(pwd)/uitrace.jar"