Press things by name in ui-trace, not by coordinate
`--do "tap 'Save'"` finds whatever currently carries that label -- the text, or the padded control around it, preferring the clickable one -- resolves its box from the accessibility tree at the moment of the gesture, and presses its centre. It is the name the control already has for assistive technology, so there is nothing extra to keep in step with it. `tap X Y` still works and is now the exception. A coordinate is a position measured once by hand: anything that moves the control makes the tap land on whatever now sits there, and the script then reports a number that was never measured, which reads exactly like a result rather than like a failure. ai-app's two benchmark scripts pressed a header button at `tap 723 205` and that button has now moved; Iris asked on 2026-09-03 that the fix be in the tool rather than a habit each script remembers. Two things make the failure loud, which is the whole point. A label that is not on screen ends the recording with `# error` and a non-zero exit, and `record` now prints the error lines rather than the head of a trace that can be thousands of frames long. And the sampling loop waits for the action thread before exiting -- a script whose last step outlasted the recording used to have its outcome discarded, including that error. The label is looked for over three seconds rather than once: the accessibility connection has no window at all for the first frames after it is made, and a control revealed by the previous step arrives a frame or two later. The wait is bounded and the failure is still loud. Exercised against a running emulator: a tap by label, a tap that finds nothing, and a failure that lands after the recording's own duration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
69a22c54e7
commit
b678bf3ccd
3 files changed
+159
-5
No files matched your search
@@ -148,7 +148,7 @@ are comfortable, and three were not.
|
||||
accessibility tree at 60Hz, so an animation or a settling layout is visible
|
||||
rather than happening between two samples.
|
||||
|
||||
ui-trace record -d 3000 --do 'tap 540 800' -o /tmp/t.txt
|
||||
ui-trace record -d 3000 --do "tap 'Session settings'" -o /tmp/t.txt
|
||||
ui-trace elements /tmp/t.txt # what is on screen, to pick from
|
||||
ui-trace show /tmp/t.txt # what moved
|
||||
ui-trace show /tmp/t.txt -m 'Send|Stop' --field box
|
||||
@@ -159,6 +159,30 @@ that measured identically at 171x105 looked different in a screenshot,
|
||||
because the filled one read as larger than the disabled grey one. Screenshots
|
||||
are for appearance — colour, weight, whether it looks right.
|
||||
|
||||
### Press things by name, never by coordinate
|
||||
|
||||
`tap 'Save'` finds whatever currently carries that label — the text, or the
|
||||
padded control around it, preferring the one that is clickable — resolves its
|
||||
box from the tree **at the moment of the gesture**, and presses its centre. It
|
||||
is the name the control already has for assistive technology, so nothing has
|
||||
to be kept in step with it.
|
||||
|
||||
`tap X Y` still exists and is the exception. A coordinate is a position
|
||||
measured once by hand, and anything that moves the control — a button added to
|
||||
the row, a font size, a density, another device — makes the tap land on
|
||||
whatever now sits there. The script then reports a number that was never
|
||||
measured, which reads exactly like a result rather than like a failure. Asked
|
||||
for by Iris on 2026-09-03, after ai-app's two benchmark scripts pressed a
|
||||
header button at `tap 723 205` and that button moved.
|
||||
|
||||
A label that is not on screen ends the recording with `# error action failed`
|
||||
and a non-zero exit, so a run that could not press what it meant to press
|
||||
produces no numbers at all.
|
||||
|
||||
The check that none is left in a project's scripts is one grep:
|
||||
|
||||
grep -n "tap [0-9]" path/to/*.sh
|
||||
|
||||
`adb exec-out screencap -p` still works and comes back scaled to 800px on its
|
||||
long edge, which is ~380 tokens to read instead of ~1460 and still legible
|
||||
for layout, contrast and small labels. `ADB_SCREENCAP_MAX_EDGE` raises it for
|
||||
|
||||
Reference in new issue
Block a user