Sam Rolfe
be22eba386
The Maps omnibox hint rotates: it read "Search here" in the Phase 5 run and "Try petrol stations, ATMs" in the AUD-128 re-verification, so the recipe's text match on it failed. The resource id does not rotate. - internal/ui: add Find/FindID and the "id:" selector; Find dispatches an id: selector to the resource id and everything else to text as before. - internal/recipe: wait/tap/wait_gone/tap_if and the tap helper use Find, so a recipe may name a resource id. Guards stay text-only. - recipes/maps_navigate.recipe: wait/tap id:search_omnibox_text_box. - tests for the id selector; docs and README updated.
321 lines
14 KiB
Markdown
321 lines
14 KiB
Markdown
# Phase 5 — the Maps recipe, and the wedged-app check
|
|
|
|
Phase 5 of `AGENTS.md` section 11: the `maps_navigate` recipe, plus the wedged-app
|
|
detection the task asked for.
|
|
|
|
Date: 2026-10-16. Phone: `Pixel_8a (akita)` over Tailscale at
|
|
`100.101.49.17:46689` (the port moves on every reboot; nothing hardcodes it).
|
|
|
|
## What was built
|
|
|
|
| File | What it holds |
|
|
|---|---|
|
|
| `recipes/maps_navigate.recipe` | The recipe: open the search box, type the place, confirm the result list names it. |
|
|
| `internal/ui/dump.go` | `Find` / `FindID`: the `id:` selector, for a label that rotates. |
|
|
| `internal/ui/change.go` | `ScreenSignature` and `WaitForChange`: the screen fingerprint and the bounded change check. |
|
|
| `internal/recipe/recipe.go` | The new `expect_change <duration>` verb. |
|
|
| `internal/recipe/engine.go` | Runs the check, and reads the screen **before** the step above it. |
|
|
| `internal/ui/change_test.go`, `internal/recipe/recipe_test.go` | Unit tests for the fingerprint and the verb. |
|
|
|
|
## The recipe
|
|
|
|
One file, plain text, no coordinates anywhere (FR-4):
|
|
|
|
```
|
|
task: maps_navigate
|
|
app: com.google.android.apps.maps
|
|
param: query
|
|
|
|
guard "Manage your Google Account" => Maps is showing its account sheet; press back on the phone and retry
|
|
guard "Turn on Incognito mode" => Maps is showing its account sheet; press back on the phone and retry
|
|
|
|
steps:
|
|
- launch
|
|
- foreground 30s
|
|
- wait id:search_omnibox_text_box 30s retry 2
|
|
- tap id:search_omnibox_text_box
|
|
- expect_change 15s retry 2
|
|
- type "{query}"
|
|
- expect_change 15s retry 2
|
|
- wait "{query}" 30s else "the Maps result list did not name the place"
|
|
```
|
|
|
|
Run it with:
|
|
|
|
```sh
|
|
./bin/artemis run maps_navigate query="Melbourne Museum"
|
|
```
|
|
|
|
## The search box is addressed by id, not by its label
|
|
|
|
The first version of the recipe waited for the text **`Search here`**. On
|
|
2026-10-17 a re-verification run failed at that step: the box was on screen,
|
|
but its label had rotated to **`Try petrol stations, ATMs`**:
|
|
|
|
```
|
|
step 3 wait [Search here]: element matching "Search here" did not appear within 30s;
|
|
foreground=com.google.android.apps.maps;
|
|
screen shows: "Layers", "Enter compass mode", "Directions", "Ivanhoe", ...
|
|
```
|
|
|
|
The Maps omnibox carries its hint as a **rotating content description**, so a
|
|
text match on it is not stable. A fresh dump of the live node shows the stable
|
|
half and the rotating half side by side:
|
|
|
|
```
|
|
<node resource-id="com.google.android.apps.maps:id/search_omnibox_text_box"
|
|
class="android.widget.EditText" content-desc="Try petrol stations, ATMs"
|
|
clickable="true" enabled="true" bounds="[31,142][663,268]"/>
|
|
```
|
|
|
|
The recipe now names the resource id with the **`id:` selector**:
|
|
`wait id:search_omnibox_text_box` and `tap id:search_omnibox_text_box`. This is
|
|
still a named match with the tap coordinate derived from the matched element's
|
|
bounds — not a fixed coordinate, so FR-4 holds. `internal/ui/dump.go` dispatches
|
|
`Find`: a selector that starts with `id:` is matched against the resource id
|
|
(full or short name), and every other selector against text and content
|
|
description as before. `guards` remain text-only, because a guard names visible
|
|
screen text.
|
|
|
|
The rule to keep: **use a text selector where a stable label exists, and an
|
|
`id:` selector where the label rotates.** The final `wait "{query}"` stays a
|
|
text match on purpose — the result row must name the place the caller asked for.
|
|
|
|
## The labels came from the phone, not from a guess
|
|
|
|
A fresh `uiautomator dump` of the Maps home and the search screen on the real
|
|
phone:
|
|
|
|
```
|
|
# Map home
|
|
[EditText] Search here id=search_omnibox_text_box bounds=[31,142][663,268]
|
|
[View] Voice search bounds=[663,142][789,268]
|
|
[View] Lens in Maps bounds=[789,142][915,268]
|
|
[FrameLayout] Account and settings. bounds=[915,142][1049,268]
|
|
[FrameLayout] Explore / You / Contribute / Business bounds=[0,2168][1080,2337]
|
|
|
|
# Search screen, after tapping "Search here"
|
|
[Button] Navigate up bounds=[31,142][157,268]
|
|
[EditText] Search here id=search_omnibox_text_box bounds=[157,142][921,268]
|
|
[EditText] <query> id=search_omnibox_edit_text bounds=[157,142][921,268]
|
|
[Button] Voice search bounds=[923,142][1049,268]
|
|
[ImageView] Clear bounds=[954,173][1017,236]
|
|
[RecyclerView] <results> id=typed_suggest_container bounds=[0,284][1080,1577]
|
|
|
|
# One suggestion row in that list
|
|
[TextView] 7.6 km bounds=[0,378][210,431]
|
|
[TextView] Melbourne Museum bounds=[210,296][668,366]
|
|
[TextView] Nicholson Street, Carlton VIC bounds=[210,366][823,429]
|
|
[FrameLayout] Activate to enter suggestion Melbourne Museum into search bar
|
|
bounds=[933,300][1059,426]
|
|
```
|
|
|
|
Three things this dump settled:
|
|
|
|
* The search box is the `EditText` labelled **`Search here`** (resource id
|
|
`search_omnibox_text_box`). Tapping it opens the search screen with the box
|
|
focused.
|
|
* Typing shows the results list immediately; there is no separate "press enter"
|
|
step. The list is the `RecyclerView` with id `typed_suggest_container`.
|
|
* Each suggestion row carries the place as its title **and** as the label of its
|
|
own row button: `Activate to enter suggestion <place> into search bar`. So the
|
|
final `wait "{query}"` matches a real row, not the empty box. Because that row
|
|
button is clickable and small, it is the match the run log reports.
|
|
|
|
The account sheet was opened on purpose to take its labels for the guards
|
|
(`Manage your Google Account`, `Turn on Incognito mode`). Maps on this phone is
|
|
signed in as `samuelrolfe@gmail.com`, so there is no sign-in screen in the
|
|
search path and no guard was invented for one.
|
|
|
|
## The wedged-app check (`expect_change`)
|
|
|
|
An app can stop consuming input while it keeps drawing: Android delivers the
|
|
tap, the app never acts on it (`docs/phase-3-spotify-recipe.md`, attempts 9, 10
|
|
and 13). Before Phase 5 the recipe reported that as the label it was waiting
|
|
for, which names the wrong problem.
|
|
|
|
`expect_change <duration>` reads the screen, waits a bounded time for it to
|
|
change, and fails with the distinct reason:
|
|
|
|
```
|
|
the app stopped consuming input: the screen did not change within 6s;
|
|
foreground=com.google.android.apps.maps; screen shows: "Layers", ...
|
|
```
|
|
|
|
The screen fingerprint is the foreground package plus every visible label and
|
|
its bounds, sorted so that the order uiautomator happens to report never looks
|
|
like a change (`internal/ui/change.go`).
|
|
|
|
### The baseline must be read *before* the input
|
|
|
|
The first live run of the recipe failed the check against a perfectly healthy
|
|
Maps:
|
|
|
|
```
|
|
step 4 tap [Search here]: ok (5.463s)
|
|
step 5 expect_change [15s]: the app stopped consuming input: the screen did not change within 15s
|
|
```
|
|
|
|
The tap had worked and the search screen had opened. The bug was ordering: a
|
|
`uiautomator dump` is not instant, so a baseline taken *after* the tap already
|
|
showed the app's response, and the "after" screen could never differ.
|
|
|
|
The engine now reads the screen immediately **before** the step above runs, and
|
|
hands that signature to the `expect_change` step that follows it
|
|
(`internal/recipe/engine.go`). A dump runs a JVM on the phone, so this read
|
|
happens only when a recipe asks for the check. Each `retry` re-reads the screen
|
|
before it re-presses the step above.
|
|
|
|
## The live run
|
|
|
|
```sh
|
|
$ ./bin/artemis run maps_navigate query="Melbourne Museum"
|
|
task maps_navigate (Search Google Maps for a place and show the matching result list.)
|
|
step 1 launch []: ok (616ms)
|
|
step 2 foreground []: ok (157ms)
|
|
step 3 wait [Search here]: ok (2.672s)
|
|
step 4 tap [Search here]: ok (5.647s)
|
|
step 5 expect_change [15s]: ok (3.156s)
|
|
step 6 type [Melbourne Museum]: ok (447ms)
|
|
step 7 expect_change [15s]: ok (3.077s)
|
|
step 8 wait [Melbourne Museum]: ok (2.692s)
|
|
OK maps_navigate in 27.606s
|
|
```
|
|
|
|
The run log (`logs/runs.jsonl`) records `"ok":true`, and the step that proves the
|
|
DoD reports:
|
|
|
|
```json
|
|
{"step":8,"verb":"wait","args":["{query}"],
|
|
"detail":"found android.widget.FrameLayout[-] text=\"Activate to enter suggestion Melbourne Museum into search bar\" bounds=[933,300][1059,426]",
|
|
"ok":true,"duration_ms":2692}
|
|
```
|
|
|
|
And the screen after the run, read back with a fresh dump — the result list
|
|
names the searched place:
|
|
|
|
```
|
|
$ ./bin/artemis dump
|
|
38 text nodes
|
|
[TextView] 7.6 km bounds=[0,378][210,431]
|
|
[TextView] Melbourne Museum bounds=[210,296][668,366] <- the place, first row
|
|
[TextView] Nicholson Street, Carlton VIC bounds=[210,366][823,429]
|
|
[FrameLayout] Activate to enter suggestion Melbourne Museum into search bar
|
|
[TextView] 7.6 km bounds=[0,539][210,592]
|
|
[TextView] Melbourne Museum - Museum Market Cafe bounds=[210,457][923,527]
|
|
[TextView] Nicholson Street, Carlton VIC bounds=[210,527][823,590]
|
|
[TextView] 7.5 km bounds=[0,700][210,753]
|
|
[TextView] Melbourne Museum - Museum Shop bounds=[210,618][923,688]
|
|
[TextView] Nicholson Street, Carlton VIC bounds=[210,688][823,751]
|
|
[TextView] 8.2 km bounds=[0,861][210,914]
|
|
[TextView] Museum of Chinese Australian History bounds=[210,779][923,849]
|
|
[TextView] Cohen Place, Melbourne VIC bounds=[210,849][803,912]
|
|
[TextView] 7.6 km bounds=[0,1022][210,1075]
|
|
[TextView] Children's Gallery (Melbourne Museum) bounds=[210,940][923,1010]
|
|
[TextView] Carlton VIC bounds=[210,1010][451,1073]
|
|
```
|
|
|
|
## The wedge reason, demonstrated live
|
|
|
|
A real wedged app could **not** be reproduced in Phase 5: the Phase 3 wedge came
|
|
from a memory-starved phone under load, and it cannot be forced on demand. What
|
|
was verified instead is the detection path itself, with a probe recipe (not
|
|
shipped) whose step above is a real input that Maps answers with no screen
|
|
change — an ENTER key event on the map home, where nothing has focus:
|
|
|
|
```sh
|
|
$ ARTEMIS_RECIPES=<probe dir> ./bin/artemis run maps_wedge_probe
|
|
step 4 key [enter]: ok (150ms)
|
|
step 5 expect_change [6s]: the app stopped consuming input: the screen did not change within 6s; foreground=com.google.android.apps.maps; screen shows: "Layers", "Re-centre map to your location", "Directions", "Carlton", "Mostly cloudy, 23°, Good, AQC", "23°"
|
|
FAILED maps_wedge_probe after 17.857s: ...
|
|
```
|
|
|
|
The bounded retry was verified in the same way (`expect_change 6s retry 2`):
|
|
the engine re-pressed the step above twice more and then stopped, three attempts
|
|
in total, so a wedged app ends the run instead of looping (FR-11):
|
|
|
|
```sh
|
|
step 5 expect_change [6s]: the app stopped consuming input: ...
|
|
step 5 expect_change: retrying, pressing key again
|
|
step 4 key [enter]: ok (223ms)
|
|
step 5 expect_change [6s]: the app stopped consuming input: ...
|
|
step 5 expect_change: retrying, pressing key again
|
|
step 4 key [enter]: ok (150ms)
|
|
step 5 expect_change [6s]: the app stopped consuming input: ...
|
|
FAILED maps_wedge_probe after 46.099s: ...
|
|
```
|
|
|
|
**Limit, stated plainly.** The check cannot tell a wedged app from an input that
|
|
legitimately changes nothing: it keys on "the screen did not change". A recipe
|
|
must place `expect_change` only after an input that must change the screen. The
|
|
`maps_navigate` recipe does that twice — the tap that opens the search screen,
|
|
and the typing that draws the result list.
|
|
|
|
## Verification status
|
|
|
|
| Item | State |
|
|
|---|---|
|
|
| Screen fingerprint, stability and sensitivity | unit tested (`internal/ui`) |
|
|
| `expect_change` parsing, `retry`, `else` | unit tested (`internal/recipe`) |
|
|
| All shipped recipes parse | unit tested (`internal/recipe`) |
|
|
| `maps_navigate` opens a real search and names the place | **live, verified** — 8 steps, 27.6s, result list above |
|
|
| `maps_navigate` re-verified after the label rotated | **live, verified** — 8 steps, 27.2s, id selector; see “Re-verification 2026-10-17” below |
|
|
| The wedge reason fires on an input that changes nothing | **live, verified** — probe above |
|
|
| A wedged app was reproduced | **no** — could not be forced; see the limit above |
|
|
| The wedge check is bounded by `retry` | **live, verified** — 3 attempts, then stop |
|
|
|
|
## Re-verification 2026-10-17
|
|
|
|
AUD-128 re-ran the recipe on the current tree and the current phone session.
|
|
The text selector failed (the label had rotated, above), so the recipe moved to
|
|
the `id:` selector and the run went green:
|
|
|
|
```sh
|
|
$ ./bin/artemis run maps_navigate query="Melbourne Museum"
|
|
task maps_navigate (Search Google Maps for a place and show the matching result list.)
|
|
step 1 launch []: ok (676ms)
|
|
step 2 foreground []: ok (166ms)
|
|
step 3 wait [id:search_omnibox_text_box]: ok (2.795s)
|
|
step 4 tap [id:search_omnibox_text_box]: ok (5.768s)
|
|
step 5 expect_change [15s]: ok (3.21s)
|
|
step 6 type [Melbourne Museum]: ok (420ms)
|
|
step 7 expect_change [15s]: ok (3.479s)
|
|
step 8 wait [Melbourne Museum]: ok (2.992s)
|
|
OK maps_navigate in 27.177s
|
|
```
|
|
|
|
The run log records `"ok":true`, and the wedge probe fired on a live no-op input
|
|
(`key escape` on the map home), three attempts in total, then stopped:
|
|
|
|
```
|
|
step 3 key [escape]: ok (151ms)
|
|
step 4 expect_change [6s]: the app stopped consuming input: the screen did not change within 6s;
|
|
foreground=com.google.android.apps.maps; screen shows: "Layers", "Enter compass mode", ...
|
|
step 4 expect_change: retrying, pressing key again
|
|
step 3 key [escape]: ok (207ms)
|
|
step 4 expect_change [6s]: the app stopped consuming input: ...
|
|
step 4 expect_change: retrying, pressing key again
|
|
step 3 key [escape]: ok (180ms)
|
|
step 4 expect_change [6s]: the app stopped consuming input: ...
|
|
FAILED maps_wedge_probe after 44.346s: ...
|
|
```
|
|
|
|
## Reproducing it
|
|
|
|
```sh
|
|
cd ~/paseo/projects/artemis_control
|
|
./bin/artemis run maps_navigate query="Melbourne Museum"
|
|
tail -1 logs/runs.jsonl
|
|
./bin/artemis dump
|
|
```
|
|
|
|
The phone must be awake and unlocked (a secure keyguard is a loud refusal,
|
|
`AGENTS.md` section 13 Q2). From the map home this takes about 28 seconds.
|
|
|
|
## Not done
|
|
|
|
* The recipe stops at the result list, which is what the Phase 5 DoD asks for.
|
|
It does not tap a result to open the place, so it does not start navigation.
|
|
* Phase 6 (the action API) and Phase 7 (resilience) have not been started, on
|
|
instruction.
|