artemis_control/docs/phase-5-maps-recipe.md

Phase 5 — the Maps recipe, and the wedged-app check

Phase 5 of AGENTS.md section 11: the maps_navigate recipe, plus the wedged-app detection the task asked for.

Date: 2026-10-16. Phone: Pixel_8a (akita) over Tailscale at 100.101.49.17:46689 (the port moves on every reboot; nothing hardcodes it).

What was built

File	What it holds
recipes/maps_navigate.recipe	The recipe: open the search box, type the place, confirm the result list names it.
internal/ui/dump.go	Find / FindID: the id: selector, for a label that rotates.
internal/ui/change.go	ScreenSignature and WaitForChange: the screen fingerprint and the bounded change check.
internal/recipe/recipe.go	The new expect_change <duration> verb.
internal/recipe/engine.go	Runs the check, and reads the screen before the step above it.
internal/ui/change_test.go, internal/recipe/recipe_test.go	Unit tests for the fingerprint and the verb.

The recipe

One file, plain text, no coordinates anywhere (FR-4):

task: maps_navigate
app: com.google.android.apps.maps
param: query

guard "Manage your Google Account" => Maps is showing its account sheet; press back on the phone and retry
guard "Turn on Incognito mode"     => Maps is showing its account sheet; press back on the phone and retry

steps:
  - launch
  - foreground 30s
  - wait id:search_omnibox_text_box 30s retry 2
  - tap id:search_omnibox_text_box
  - expect_change 15s retry 2
  - type "{query}"
  - expect_change 15s retry 2
  - wait "{query}" 30s else "the Maps result list did not name the place"

Run it with:

./bin/artemis run maps_navigate query="Melbourne Museum"

The search box is addressed by id, not by its label

The first version of the recipe waited for the text Search here. On 2026-10-17 a re-verification run failed at that step: the box was on screen, but its label had rotated to Try petrol stations, ATMs:

step 3 wait [Search here]: element matching "Search here" did not appear within 30s;
  foreground=com.google.android.apps.maps;
  screen shows: "Layers", "Enter compass mode", "Directions", "Ivanhoe", ...

The Maps omnibox carries its hint as a rotating content description, so a text match on it is not stable. A fresh dump of the live node shows the stable half and the rotating half side by side:

<node resource-id="com.google.android.apps.maps:id/search_omnibox_text_box"
      class="android.widget.EditText" content-desc="Try petrol stations, ATMs"
      clickable="true" enabled="true" bounds="[31,142][663,268]"/>

The recipe now names the resource id with the id: selector: wait id:search_omnibox_text_box and tap id:search_omnibox_text_box. This is still a named match with the tap coordinate derived from the matched element's bounds — not a fixed coordinate, so FR-4 holds. internal/ui/dump.go dispatches Find: a selector that starts with id: is matched against the resource id (full or short name), and every other selector against text and content description as before. guards remain text-only, because a guard names visible screen text.

The rule to keep: use a text selector where a stable label exists, and an id: selector where the label rotates. The final wait "{query}" stays a text match on purpose — the result row must name the place the caller asked for.

The labels came from the phone, not from a guess

A fresh uiautomator dump of the Maps home and the search screen on the real phone:

# Map home
[EditText]  Search here          id=search_omnibox_text_box   bounds=[31,142][663,268]
[View]      Voice search                                       bounds=[663,142][789,268]
[View]      Lens in Maps                                       bounds=[789,142][915,268]
[FrameLayout] Account and settings.                            bounds=[915,142][1049,268]
[FrameLayout] Explore / You / Contribute / Business            bounds=[0,2168][1080,2337]

# Search screen, after tapping "Search here"
[Button]      Navigate up                                      bounds=[31,142][157,268]
[EditText]    Search here        id=search_omnibox_text_box   bounds=[157,142][921,268]
[EditText]    <query>            id=search_omnibox_edit_text  bounds=[157,142][921,268]
[Button]      Voice search                                     bounds=[923,142][1049,268]
[ImageView]   Clear                                            bounds=[954,173][1017,236]
[RecyclerView] <results>         id=typed_suggest_container    bounds=[0,284][1080,1577]

# One suggestion row in that list
[TextView]    7.6 km                                           bounds=[0,378][210,431]
[TextView]    Melbourne Museum                                 bounds=[210,296][668,366]
[TextView]    Nicholson Street, Carlton VIC                    bounds=[210,366][823,429]
[FrameLayout] Activate to enter suggestion Melbourne Museum into search bar
                                                               bounds=[933,300][1059,426]

Three things this dump settled:

• The search box is the EditText labelled Search here (resource id search_omnibox_text_box). Tapping it opens the search screen with the box focused.
• Typing shows the results list immediately; there is no separate "press enter" step. The list is the RecyclerView with id typed_suggest_container.
• Each suggestion row carries the place as its title and as the label of its own row button: Activate to enter suggestion <place> into search bar. So the final wait "{query}" matches a real row, not the empty box. Because that row button is clickable and small, it is the match the run log reports.

The account sheet was opened on purpose to take its labels for the guards (Manage your Google Account, Turn on Incognito mode). Maps on this phone is signed in as samuelrolfe@gmail.com, so there is no sign-in screen in the search path and no guard was invented for one.

The wedged-app check (expect_change)

An app can stop consuming input while it keeps drawing: Android delivers the tap, the app never acts on it (docs/phase-3-spotify-recipe.md, attempts 9, 10 and 13). Before Phase 5 the recipe reported that as the label it was waiting for, which names the wrong problem.

expect_change <duration> reads the screen, waits a bounded time for it to change, and fails with the distinct reason:

the app stopped consuming input: the screen did not change within 6s;
foreground=com.google.android.apps.maps; screen shows: "Layers", ...

The screen fingerprint is the foreground package plus every visible label and its bounds, sorted so that the order uiautomator happens to report never looks like a change (internal/ui/change.go).

The baseline must be read before the input

The first live run of the recipe failed the check against a perfectly healthy Maps:

step 4 tap [Search here]: ok (5.463s)
step 5 expect_change [15s]: the app stopped consuming input: the screen did not change within 15s

The tap had worked and the search screen had opened. The bug was ordering: a uiautomator dump is not instant, so a baseline taken after the tap already showed the app's response, and the "after" screen could never differ.

The engine now reads the screen immediately before the step above runs, and hands that signature to the expect_change step that follows it (internal/recipe/engine.go). A dump runs a JVM on the phone, so this read happens only when a recipe asks for the check. Each retry re-reads the screen before it re-presses the step above.

The live run

$ ./bin/artemis run maps_navigate query="Melbourne Museum"
task maps_navigate (Search Google Maps for a place and show the matching result list.)
  step 1 launch []: ok (616ms)
  step 2 foreground []: ok (157ms)
  step 3 wait [Search here]: ok (2.672s)
  step 4 tap [Search here]: ok (5.647s)
  step 5 expect_change [15s]: ok (3.156s)
  step 6 type [Melbourne Museum]: ok (447ms)
  step 7 expect_change [15s]: ok (3.077s)
  step 8 wait [Melbourne Museum]: ok (2.692s)
OK maps_navigate in 27.606s

The run log (logs/runs.jsonl) records "ok":true, and the step that proves the DoD reports:

{"step":8,"verb":"wait","args":["{query}"],
 "detail":"found android.widget.FrameLayout[-] text=\"Activate to enter suggestion Melbourne Museum into search bar\" bounds=[933,300][1059,426]",
 "ok":true,"duration_ms":2692}

And the screen after the run, read back with a fresh dump — the result list names the searched place:

$ ./bin/artemis dump
38 text nodes
[TextView]     7.6 km                                          bounds=[0,378][210,431]
[TextView]     Melbourne Museum                                bounds=[210,296][668,366]   <- the place, first row
[TextView]     Nicholson Street, Carlton VIC                    bounds=[210,366][823,429]
[FrameLayout]  Activate to enter suggestion Melbourne Museum into search bar
[TextView]     7.6 km                                          bounds=[0,539][210,592]
[TextView]     Melbourne Museum - Museum Market Cafe           bounds=[210,457][923,527]
[TextView]     Nicholson Street, Carlton VIC                    bounds=[210,527][823,590]
[TextView]     7.5 km                                          bounds=[0,700][210,753]
[TextView]     Melbourne Museum - Museum Shop                  bounds=[210,618][923,688]
[TextView]     Nicholson Street, Carlton VIC                    bounds=[210,688][823,751]
[TextView]     8.2 km                                          bounds=[0,861][210,914]
[TextView]     Museum of Chinese Australian History            bounds=[210,779][923,849]
[TextView]     Cohen Place, Melbourne VIC                       bounds=[210,849][803,912]
[TextView]     7.6 km                                          bounds=[0,1022][210,1075]
[TextView]     Children's Gallery (Melbourne Museum)           bounds=[210,940][923,1010]
[TextView]     Carlton VIC                                      bounds=[210,1010][451,1073]

The wedge reason, demonstrated live

A real wedged app could not be reproduced in Phase 5: the Phase 3 wedge came from a memory-starved phone under load, and it cannot be forced on demand. What was verified instead is the detection path itself, with a probe recipe (not shipped) whose step above is a real input that Maps answers with no screen change — an ENTER key event on the map home, where nothing has focus:

$ ARTEMIS_RECIPES=<probe dir> ./bin/artemis run maps_wedge_probe
  step 4 key [enter]: ok (150ms)
  step 5 expect_change [6s]: the app stopped consuming input: the screen did not change within 6s; foreground=com.google.android.apps.maps; screen shows: "Layers", "Re-centre map to your location", "Directions", "Carlton", "Mostly cloudy, 23°, Good, AQC", "23°"
FAILED maps_wedge_probe after 17.857s: ...

The bounded retry was verified in the same way (expect_change 6s retry 2): the engine re-pressed the step above twice more and then stopped, three attempts in total, so a wedged app ends the run instead of looping (FR-11):

  step 5 expect_change [6s]: the app stopped consuming input: ...
  step 5 expect_change: retrying, pressing key again
  step 4 key [enter]: ok (223ms)
  step 5 expect_change [6s]: the app stopped consuming input: ...
  step 5 expect_change: retrying, pressing key again
  step 4 key [enter]: ok (150ms)
  step 5 expect_change [6s]: the app stopped consuming input: ...
FAILED maps_wedge_probe after 46.099s: ...

Limit, stated plainly. The check cannot tell a wedged app from an input that legitimately changes nothing: it keys on "the screen did not change". A recipe must place expect_change only after an input that must change the screen. The maps_navigate recipe does that twice — the tap that opens the search screen, and the typing that draws the result list.

Verification status

Item	State
Screen fingerprint, stability and sensitivity	unit tested (internal/ui)
expect_change parsing, retry, else	unit tested (internal/recipe)
All shipped recipes parse	unit tested (internal/recipe)
maps_navigate opens a real search and names the place	live, verified — 8 steps, 27.6s, result list above
maps_navigate re-verified after the label rotated	live, verified — 8 steps, 27.2s, id selector; see “Re-verification 2026-10-17” below
The wedge reason fires on an input that changes nothing	live, verified — probe above
A wedged app was reproduced	no — could not be forced; see the limit above
The wedge check is bounded by retry	live, verified — 3 attempts, then stop

Re-verification 2026-10-17

AUD-128 re-ran the recipe on the current tree and the current phone session. The text selector failed (the label had rotated, above), so the recipe moved to the id: selector and the run went green:

$ ./bin/artemis run maps_navigate query="Melbourne Museum"
task maps_navigate (Search Google Maps for a place and show the matching result list.)
  step 1 launch []: ok (676ms)
  step 2 foreground []: ok (166ms)
  step 3 wait [id:search_omnibox_text_box]: ok (2.795s)
  step 4 tap [id:search_omnibox_text_box]: ok (5.768s)
  step 5 expect_change [15s]: ok (3.21s)
  step 6 type [Melbourne Museum]: ok (420ms)
  step 7 expect_change [15s]: ok (3.479s)
  step 8 wait [Melbourne Museum]: ok (2.992s)
OK maps_navigate in 27.177s

The run log records "ok":true, and the wedge probe fired on a live no-op input (key escape on the map home), three attempts in total, then stopped:

step 3 key [escape]: ok (151ms)
step 4 expect_change [6s]: the app stopped consuming input: the screen did not change within 6s;
  foreground=com.google.android.apps.maps; screen shows: "Layers", "Enter compass mode", ...
step 4 expect_change: retrying, pressing key again
step 3 key [escape]: ok (207ms)
step 4 expect_change [6s]: the app stopped consuming input: ...
step 4 expect_change: retrying, pressing key again
step 3 key [escape]: ok (180ms)
step 4 expect_change [6s]: the app stopped consuming input: ...
FAILED maps_wedge_probe after 44.346s: ...

Reproducing it

cd ~/paseo/projects/artemis_control
./bin/artemis run maps_navigate query="Melbourne Museum"
tail -1 logs/runs.jsonl
./bin/artemis dump

The phone must be awake and unlocked (a secure keyguard is a loud refusal, AGENTS.md section 13 Q2). From the map home this takes about 28 seconds.

Not done

• The recipe stops at the result list, which is what the Phase 5 DoD asks for. It does not tap a result to open the place, so it does not start navigation.
• Phase 6 (the action API) and Phase 7 (resilience) have not been started, on instruction.