Press things by name in ui-trace, not by coordinate

`--do "tap 'Save'"` finds whatever currently carries that label -- the text,
or the padded control around it, preferring the clickable one -- resolves its
box from the accessibility tree at the moment of the gesture, and presses its
centre. It is the name the control already has for assistive technology, so
there is nothing extra to keep in step with it.

`tap X Y` still works and is now the exception. A coordinate is a position
measured once by hand: anything that moves the control makes the tap land on
whatever now sits there, and the script then reports a number that was never
measured, which reads exactly like a result rather than like a failure.
ai-app's two benchmark scripts pressed a header button at `tap 723 205` and
that button has now moved; Iris asked on 2026-09-03 that the fix be in the
tool rather than a habit each script remembers.

Two things make the failure loud, which is the whole point. A label that is
not on screen ends the recording with `# error` and a non-zero exit, and
`record` now prints the error lines rather than the head of a trace that can
be thousands of frames long. And the sampling loop waits for the action
thread before exiting -- a script whose last step outlasted the recording
used to have its outcome discarded, including that error.

The label is looked for over three seconds rather than once: the
accessibility connection has no window at all for the first frames after it
is made, and a control revealed by the previous step arrives a frame or two
later. The wait is bounded and the failure is still loud.

Exercised against a running emulator: a tap by label, a tap that finds
nothing, and a failure that lands after the recording's own duration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
irisandClaude Opus 5 committed 2026-09-03 23:52:43 -04:00
1 parent 69a22c54e7
commit b678bf3ccd
3 files changed
+159 -5

No files matched your search

+14 -2
View File
@@ -16,11 +16,18 @@ re-interrogated with a different selector without touching the device again,
which matters because the interesting question is usually the one you think of
after seeing the first answer.
ui-trace record -d 3000 --do 'tap 540 800' -o /tmp/t.txt
ui-trace record -d 3000 --do "tap 'Session settings'" -o /tmp/t.txt
ui-trace elements /tmp/t.txt # what is on screen, to pick from
ui-trace show /tmp/t.txt # what moved (the default question)
ui-trace show /tmp/t.txt -m 'Called|ask' # a timeline for those
Press things by **name**, not by coordinate. `tap 'Save'` resolves the label
against the tree at the moment of the gesture, so it survives anything that
moves the control and fails loudly when the control is genuinely not there.
`tap X Y` still works and is the exception: a coordinate is a position measured
once by hand, and the first thing that moves the control makes the tap land on
whatever now sits there -- which reads exactly like a result.
Nothing here is specific to any app.
"""
@@ -102,7 +109,12 @@ def record(args):
stderr=subprocess.STDOUT)
text = out.read_text()
if result.returncode != 0 or "# error" in text:
print(text[:2000], file=sys.stderr)
# The error lines rather than the head of the file. An action that failed -- a
# `tap` whose label is not on screen -- reports itself part way through a trace
# that may be thousands of frames long, so printing the beginning showed a
# perfectly ordinary first frame and said nothing about what went wrong.
problems = [line for line in text.splitlines() if line.startswith(("# error", "# at"))]
print("\n".join(problems[:40]) if problems else text[:2000], file=sys.stderr)
sys.exit("ui-trace: the device recorder failed")
frames, _ = parse(out)
print(f"ui-trace: {len(frames)} frames over {frames[-1][0] if frames else 0}ms "