sam/artemis_control
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Phase 5 — maps_navigate, and read the screen before the input

Browse Source 
The Maps recipe: open the search box, type the place, and confirm the
result list names it. Every label comes from a live uiautomator dump
(search_omnibox_text_box, typed_suggest_container, the account sheet),
never from a guess. Text only, one step_timeout/total_timeout, guards for
the account sheet.

The first live run exposed an ordering bug in expect_change. A dump is not
instant, so a baseline taken after the input already showed the app's
response and the check could never see a wedge; it failed against a
healthy Maps. The engine now reads the screen before the step above runs
and hands that signature to the check. Each retry re-reads it.

Live: maps_navigate ok in 27.6s, result list names "Melbourne Museum".
The wedge reason and its bounded retry are demonstrated with a probe
recipe whose input changes nothing; a real wedge could not be forced, and
docs/phase-5-maps-recipe.md says so plainly.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Sam Rolfe
2026-10-16 16:18:06 +11:00
parent 43cd4282d0
commit a7c07c645e
7 changed files with 420 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files

247
docs/phase-5-maps-recipe.md Normal file
View File

@@ -0,0 +1,247 @@
# Phase 5 — the Maps recipe, and the wedged-app check
Phase 5 of `AGENTS.md` section 11: the `maps_navigate` recipe, plus the wedged-app
detection the task asked for.
Date: 2026-10-16. Phone: `Pixel_8a (akita)` over Tailscale at
`100.101.49.17:46689` (the port moves on every reboot; nothing hardcodes it).
## What was built
| File | What it holds |
|---|---|
| `recipes/maps_navigate.recipe` | The recipe: open the search box, type the place, confirm the result list names it. |
| `internal/ui/change.go` | `ScreenSignature` and `WaitForChange`: the screen fingerprint and the bounded change check. |
| `internal/recipe/recipe.go` | The new `expect_change <duration>` verb. |
| `internal/recipe/engine.go` | Runs the check, and reads the screen **before** the step above it. |
| `internal/ui/change_test.go`, `internal/recipe/recipe_test.go` | Unit tests for the fingerprint and the verb. |
## The recipe
One file, plain text, no coordinates anywhere (FR-4):
```
task: maps_navigate
app: com.google.android.apps.maps
param: query
guard "Manage your Google Account" => Maps is showing its account sheet; press back on the phone and retry
guard "Turn on Incognito mode" => Maps is showing its account sheet; press back on the phone and retry
steps:
- launch
- foreground 30s
- wait "Search here" 30s retry 2
- tap "Search here"
- expect_change 15s retry 2
- type "{query}"
- expect_change 15s retry 2
- wait "{query}" 30s else "the Maps result list did not name the place"
```
Run it with:
```sh
./bin/artemis run maps_navigate query="Melbourne Museum"
```
## The labels came from the phone, not from a guess
A fresh `uiautomator dump` of the Maps home and the search screen on the real
phone:
```
# Map home
[EditText] Search here id=search_omnibox_text_box bounds=[31,142][663,268]
[View] Voice search bounds=[663,142][789,268]
[View] Lens in Maps bounds=[789,142][915,268]
[FrameLayout] Account and settings. bounds=[915,142][1049,268]
[FrameLayout] Explore / You / Contribute / Business bounds=[0,2168][1080,2337]
# Search screen, after tapping "Search here"
[Button] Navigate up bounds=[31,142][157,268]
[EditText] Search here id=search_omnibox_text_box bounds=[157,142][921,268]
[EditText] <query> id=search_omnibox_edit_text bounds=[157,142][921,268]
[Button] Voice search bounds=[923,142][1049,268]
[ImageView] Clear bounds=[954,173][1017,236]
[RecyclerView] <results> id=typed_suggest_container bounds=[0,284][1080,1577]
# One suggestion row in that list
[TextView] 7.6 km bounds=[0,378][210,431]
[TextView] Melbourne Museum bounds=[210,296][668,366]
[TextView] Nicholson Street, Carlton VIC bounds=[210,366][823,429]
[FrameLayout] Activate to enter suggestion Melbourne Museum into search bar
bounds=[933,300][1059,426]
```
Three things this dump settled:
* The search box is the `EditText` labelled **`Search here`** (resource id
`search_omnibox_text_box`). Tapping it opens the search screen with the box
focused.
* Typing shows the results list immediately; there is no separate "press enter"
step. The list is the `RecyclerView` with id `typed_suggest_container`.
* Each suggestion row carries the place as its title **and** as the label of its
own row button: `Activate to enter suggestion <place> into search bar`. So the
final `wait "{query}"` matches a real row, not the empty box. Because that row
button is clickable and small, it is the match the run log reports.
The account sheet was opened on purpose to take its labels for the guards
(`Manage your Google Account`, `Turn on Incognito mode`). Maps on this phone is
signed in as `samuelrolfe@gmail.com`, so there is no sign-in screen in the
search path and no guard was invented for one.
## The wedged-app check (`expect_change`)
An app can stop consuming input while it keeps drawing: Android delivers the
tap, the app never acts on it (`docs/phase-3-spotify-recipe.md`, attempts 9, 10
and 13). Before Phase 5 the recipe reported that as the label it was waiting
for, which names the wrong problem.
`expect_change <duration>` reads the screen, waits a bounded time for it to
change, and fails with the distinct reason:
```
the app stopped consuming input: the screen did not change within 6s;
foreground=com.google.android.apps.maps; screen shows: "Layers", ...
```
The screen fingerprint is the foreground package plus every visible label and
its bounds, sorted so that the order uiautomator happens to report never looks
like a change (`internal/ui/change.go`).
### The baseline must be read *before* the input
The first live run of the recipe failed the check against a perfectly healthy
Maps:
```
step 4 tap [Search here]: ok (5.463s)
step 5 expect_change [15s]: the app stopped consuming input: the screen did not change within 15s
```
The tap had worked and the search screen had opened. The bug was ordering: a
`uiautomator dump` is not instant, so a baseline taken *after* the tap already
showed the app's response, and the "after" screen could never differ.
The engine now reads the screen immediately **before** the step above runs, and
hands that signature to the `expect_change` step that follows it
(`internal/recipe/engine.go`). A dump runs a JVM on the phone, so this read
happens only when a recipe asks for the check. Each `retry` re-reads the screen
before it re-presses the step above.
## The live run
```sh
$ ./bin/artemis run maps_navigate query="Melbourne Museum"
task maps_navigate (Search Google Maps for a place and show the matching result list.)
step 1 launch []: ok (616ms)
step 2 foreground []: ok (157ms)
step 3 wait [Search here]: ok (2.672s)
step 4 tap [Search here]: ok (5.647s)
step 5 expect_change [15s]: ok (3.156s)
step 6 type [Melbourne Museum]: ok (447ms)
step 7 expect_change [15s]: ok (3.077s)
step 8 wait [Melbourne Museum]: ok (2.692s)
OK maps_navigate in 27.606s
```
The run log (`logs/runs.jsonl`) records `"ok":true`, and the step that proves the
DoD reports:
```json
{"step":8,"verb":"wait","args":["{query}"],
"detail":"found android.widget.FrameLayout[-] text=\"Activate to enter suggestion Melbourne Museum into search bar\" bounds=[933,300][1059,426]",
"ok":true,"duration_ms":2692}
```
And the screen after the run, read back with a fresh dump — the result list
names the searched place:
```
$ ./bin/artemis dump
38 text nodes
[TextView] 7.6 km bounds=[0,378][210,431]
[TextView] Melbourne Museum bounds=[210,296][668,366] <- the place, first row
[TextView] Nicholson Street, Carlton VIC bounds=[210,366][823,429]
[FrameLayout] Activate to enter suggestion Melbourne Museum into search bar
[TextView] 7.6 km bounds=[0,539][210,592]
[TextView] Melbourne Museum - Museum Market Cafe bounds=[210,457][923,527]
[TextView] Nicholson Street, Carlton VIC bounds=[210,527][823,590]
[TextView] 7.5 km bounds=[0,700][210,753]
[TextView] Melbourne Museum - Museum Shop bounds=[210,618][923,688]
[TextView] Nicholson Street, Carlton VIC bounds=[210,688][823,751]
[TextView] 8.2 km bounds=[0,861][210,914]
[TextView] Museum of Chinese Australian History bounds=[210,779][923,849]
[TextView] Cohen Place, Melbourne VIC bounds=[210,849][803,912]
[TextView] 7.6 km bounds=[0,1022][210,1075]
[TextView] Children's Gallery (Melbourne Museum) bounds=[210,940][923,1010]
[TextView] Carlton VIC bounds=[210,1010][451,1073]
```
## The wedge reason, demonstrated live
A real wedged app could **not** be reproduced in Phase 5: the Phase 3 wedge came
from a memory-starved phone under load, and it cannot be forced on demand. What
was verified instead is the detection path itself, with a probe recipe (not
shipped) whose step above is a real input that Maps answers with no screen
change — an ENTER key event on the map home, where nothing has focus:
```sh
$ ARTEMIS_RECIPES=<probe dir> ./bin/artemis run maps_wedge_probe
step 4 key [enter]: ok (150ms)
step 5 expect_change [6s]: the app stopped consuming input: the screen did not change within 6s; foreground=com.google.android.apps.maps; screen shows: "Layers", "Re-centre map to your location", "Directions", "Carlton", "Mostly cloudy, 23°, Good, AQC", "23°"
FAILED maps_wedge_probe after 17.857s: ...
```
The bounded retry was verified in the same way (`expect_change 6s retry 2`):
the engine re-pressed the step above twice more and then stopped, three attempts
in total, so a wedged app ends the run instead of looping (FR-11):
```sh
step 5 expect_change [6s]: the app stopped consuming input: ...
step 5 expect_change: retrying, pressing key again
step 4 key [enter]: ok (223ms)
step 5 expect_change [6s]: the app stopped consuming input: ...
step 5 expect_change: retrying, pressing key again
step 4 key [enter]: ok (150ms)
step 5 expect_change [6s]: the app stopped consuming input: ...
FAILED maps_wedge_probe after 46.099s: ...
```
**Limit, stated plainly.** The check cannot tell a wedged app from an input that
legitimately changes nothing: it keys on "the screen did not change". A recipe
must place `expect_change` only after an input that must change the screen. The
`maps_navigate` recipe does that twice — the tap that opens the search screen,
and the typing that draws the result list.
## Verification status
| Item | State |
|---|---|
| Screen fingerprint, stability and sensitivity | unit tested (`internal/ui`) |
| `expect_change` parsing, `retry`, `else` | unit tested (`internal/recipe`) |
| All shipped recipes parse | unit tested (`internal/recipe`) |
| `maps_navigate` opens a real search and names the place | **live, verified** — 8 steps, 27.6s, result list above |
| The wedge reason fires on an input that changes nothing | **live, verified** — probe above |
| A wedged app was reproduced | **no** — could not be forced; see the limit above |
| The wedge check is bounded by `retry` | **live, verified** — 3 attempts, then stop |
## Reproducing it
```sh
cd ~/paseo/projects/artemis_control
./bin/artemis run maps_navigate query="Melbourne Museum"
tail -1 logs/runs.jsonl
./bin/artemis dump
```
The phone must be awake and unlocked (a secure keyguard is a loud refusal,
`AGENTS.md` section 13 Q2). From the map home this takes about 28 seconds.
## Not done
* The recipe stops at the result list, which is what the Phase 5 DoD asks for.
It does not tap a result to open the place, so it does not start navigation.
* Phase 6 (the action API) and Phase 7 (resilience) have not been started, on
instruction.