# Phase 3 — the recipes file and `spotify_play` Phase 3 of `AGENTS.md` section 11: the first named recipe, plus the resolved answer to section 13 Q2 (the locked screen). Date: 2026-10-16. Phone: `Pixel_8a (akita)` over Tailscale at `100.101.49.17:46689` (the port moves on every reboot; nothing hardcodes it). ## What was built | File | What it holds | |---|---| | `recipes/spotify_play.recipe` | The recipe, written to be edited by hand. | | `internal/recipe/recipe.go` | The file format: directives, guards, steps, parameters. | | `internal/recipe/engine.go` | Runs a recipe step by step with timeouts and a run log. | | `internal/ui/rows.go` | Picks a list row by text and taps its centre. | | `internal/ui/keyguard.go` | Reads the lock state and refuses to act when locked. | | `cmd/artemis/run.go` | `artemis recipes`, `artemis run k=v`, JSON run log. | ## The recipes file One file per task in `recipes/`, plain text, no build step. It names the app, the parameters the caller must supply, the screens Artemis must never tap on, and the ordered steps. ``` task: spotify_play app: com.spotify.music param: query row_id: com.spotify.music:id/row_root guard "Log in" => Spotify is signed out; sign in on the phone and retry steps: - launch - wait "Search" 30s - tap "Search" - tap_if "Clear search query" - wait "What do you want to listen to?" 30s - tap "What do you want to listen to?" - type "{query}" - wait "Clear search query" 30s - wait_gone "Recents" 30s - tap_row "{query}" 40s - wait exact "Pause" 40s ``` Steps express intent. There is no coordinate anywhere in the file (FR-4). Run it with: ```sh ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk" ``` The verbs are `launch`, `foreground`, `wait`, `wait_gone`, `tap`, `tap_if`, `tap_row`, `type`, `key`, `back`, `home` and `sleep`. A `wait` step may add `retry N` (see below). `tap exact "Pause"` matches the whole text, not a substring. `tap_if` taps only when the text is already on screen, which lets one recipe cope with an app that remembers its last screen. `foreground` waits until the recipe's app is really in front, so a word like "Search" cannot be matched from a different app that happens to be open — every recipe should use it straight after `launch`. ## Why `tap_row` exists A plain text tap picked the wrong element on the real Spotify screen. On the results list, `artemis tap "Honky Tonk Women"` tapped ``` ImageButton[context_menu_button] text="More options for song Honky Tonk Women - Live" bounds=[828,2388][954,2400] ``` That is the smallest clickable match, and it sits at the bottom edge of the screen. It was the wrong target: a "More options" button, not the song. `tap_row` scores every result row against the query and taps the row itself: * A candidate row is a clickable container that carries a title **and** a subtitle. A one-line search suggestion (Spotify's "rolling stones honky tonk women single") is therefore not a result row. * Text inside a clickable child is skipped, because that is an action button ("More options", "Add to library", "Add suggestion"), not the row's own text. * A row clipped by the bottom of the list is dropped. It is much shorter than a row that is fully drawn, and its centre sits under the navigation area. * Rows are ranked by F1 score over the query words, so an exact title beats a noisy deluxe-edition title. Equal scores go to the highest row, because search results are ranked by relevance. On the real screen for `Rolling Stones Honky Tonk`, the top hit wins: ``` Honky Tonk Women / Song • The Rolling Stones bounds=[0,794][1080,962] F1=0.727 ``` and the tap lands at the row's centre, `(540,878)` — on the title, clear of the action buttons. `internal/ui/rows_test.go` runs this against a trimmed copy of the real dump. ## Failing cleanly (FR-8) Every failure names its cause, and they are distinguishable: | Situation | Message | |---|---| | App missing | `app is not installed: com.spotify.music` | | Signed out | `Spotify is signed out; sign in on the phone and retry (phone is showing "Log in")` | | Update nag | `Spotify wants an update; update it on the phone and retry` | | Element missing | `text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Log in", "Sign up free"` | | Phone locked | `phone is locked; unlock it and retry` | A guard is checked when a step cannot find its target, so a bad screen produces its own reason instead of a generic "element not found". ## Timeouts (FR-11) `step_timeout: 20s` is the default per step, `total_timeout: 420s` bounds the whole run. A step may name its own timeout (`wait "Search" 30s`). The engine checks the remaining budget before each step and stops when it runs out. Every `adb` call is already bounded by `adb.DefaultTimeout`. ## Logging (FR-10) Every run appends one JSON line to `logs/runs.jsonl` (override with `ARTEMIS_LOG`): task, params, each step with its duration, the result and the total duration. Two real entries: ```json {"time":"2026-10-16T13:23:31+11:00","task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"},"ok":false,"error":"phone is locked; unlock it and retry","steps":null,"duration_ms":832} ``` The same steps print to the terminal as they run. ## The locked screen (section 13 Q2) **Decision: Artemis fails loudly. No PIN is stored and none is ever typed.** `ui.KeyguardState` reads two things: whether the lock screen is showing (`dumpsys window`, the `KeyguardServiceDelegate` block) and whether the device has a secure lock at all (`cmd lock_settings is-user-secure`). Acting commands wake the screen first and then refuse if the keyguard is showing and secure. A non-secure keyguard is still dismissed, as before. An unreadable lock state is a failure, not a guess. ### Evidence The prescribed test (put the phone to sleep, wake it, Artemis must refuse): ```sh adb -s 100.101.49.17:46689 shell input keyevent 26 # POWER = sleep adb -s 100.101.49.17:46689 shell input keyevent 224 # WAKEUP ``` ``` $ adb shell dumpsys window policy | grep -E 'showing=|screenState' showing=true screenState=SCREEN_STATE_ON $ adb shell cmd lock_settings is-user-secure true $ ./bin/artemis keyguard keyguard showing=true secure=true artemis: phone is locked; unlock it and retry exit=1 $ ./bin/artemis tap "Search" artemis: phone is locked; unlock it and retry exit=1 $ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk" task spotify_play (Search Spotify for a song and play it.) FAILED spotify_play after 833ms: phone is locked; unlock it and retry exit=1 ``` No PIN was entered, nothing was read from the phone, and the lock screen was still on screen afterwards — no blind tap happened. The screen dump after the refusals still showed the lock screen clock and date. A dozing phone reports `showing=false`, because the lock screen is not drawn yet. That is why every acting command wakes the screen **before** it checks the keyguard; reading the state first would say "unlocked" about a locked phone. ## Verification status | Item | State | |---|---| | Recipes file format, guards, parameters, timeouts | unit tested (`internal/recipe`) | | Row selection against a real Spotify dump | unit tested (`internal/ui`) | | Keyguard parsing and secure-lock detection | unit tested (`internal/ui`) | | Step durations reach the run log | unit tested (`internal/recipe`) | | All acting commands refuse on a locked phone | **live, verified** (above) | | `artemis run spotify_play` refuses on a locked phone | **live, verified** (above) | | `spotify_play` plays the right song | **live, verified end to end** — 13 steps, 35.3s | | `spotify_play` picks the right row on the real results list | **live, verified** — `tapped row "Honky Tonk Women" at (540,878)` | | Spotify search reaches the result list | live (Home → Search → type → results) | ## Confirming playback The screen is not a trustworthy place to read playback state. Spotify's transport button keeps the accessibility label **"Play"** while music is actually playing, so the first version of this recipe asserted on a label that never appears and failed on a phone that was playing the track perfectly. The last step asks Android instead: ``` - playing "{query}" 30s retry 3 ``` `playing` waits until the app's media session reports `PLAYING` **and** its metadata contains the query's words. It fails with the state and the track it found, for example: ``` com.spotify.music is not playing "Rolling Stones Honky Tonk" within 30s (state=PAUSED, playing "Honky Tonk Women, The Rolling Stones, ...") ``` ## Why the top row wins Spotify ranks results by relevance, so the highest acceptable row is the one a human would press. Two rules keep it that way: - A row's own text is its title and subtitle. Text inside a clickable child is an action button, and **content descriptions are state**, not text: the row of the track that is already loaded carries `content-desc="Paused"`. Counting that made the top hit score *worse* than an identical duplicate further down, and `tap_row` pressed the duplicate at `y=1443` while the right row sat at `y=878`. `internal/ui/rows_test.go` reproduces that screen exactly. - Scores within `rowScoreTolerance` (0.15) are treated as equal, and then the higher row wins. A clearly better text match still wins outright. ## Retrying a dropped tap A step can ask for the step above it to be pressed again: ``` - tap_row "{query}" 40s - wait exact "Pause" 45s retry 3 ``` `retry N` is only allowed on a `wait` step, needs a step above it to repeat, and never taps anything that was not matched by text. It exists because a busy phone can deliver a tap and then not act on it. The retry only re-runs the step above, so nothing else can be pressed by accident. ## Live run log Six attempts, in order. Each failure was clean, named its screen and tapped nothing random — which is the point of the phase. 1. **Refused, locked phone.** `phone is locked; unlock it and retry`. The phone had locked itself on its 5-minute screen timeout while the code was being written. 2. **Refused after the prescribed sleep/wake test.** `keyguard showing=true secure=true`, exit 1 from `keyguard`, `tap` and `run`. The lock screen was still on screen afterwards. 3. **Refused, dozing phone that only *looked* unlocked.** A dozing phone reports `showing=false`. Pressing WAKEUP then revealed the secure keyguard, and the run refused. This is why every acting command wakes first. 4. **Wrong app matched "Search".** The phone was sitting on the Android Settings app. `wait "Search"` was satisfied by Settings' own "Search Settings" list entry, so the recipe tapped the Settings search bar. Fixed by adding the `foreground` step, which waits until the recipe's app really is in front before any text is trusted. 5. **Blocked by a biometric prompt.** Settings was left showing a `Verify that it's you` dialog (`Touch the fingerprint sensor` / `Use PIN`), which is drawn over Spotify. The run failed with: `text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Cancel", "App logo", "Settings", "Verify that it's you" ...`. Artemis did not touch the prompt, and it must not: it needs Sam's fingerprint or PIN, exactly like the lock screen. 6. **Same prompt again**, until Sam cleared it. 7. **Through to the row, but no playback.** After the prompt was cleared the recipe reached step 11 (`tap_row`) with every earlier step ok, and the run log shows it chose exactly the right element: `tapped row "Honky Tonk Women" at (540,878) from Honky Tonk Women / Song • The Rolling Stones`. Only the last step failed: `text "Pause" did not appear within 40s`, with the search results still on screen. The tap was delivered (the dispatcher lists the Spotify window as the touch target) and Spotify did nothing with it. 8. **Duration logging fixed, retry added.** `wait ... retry N` now re-runs the step above and waits again. Steps 9–12 made four attempts each. 9. **A dropped tap is not the whole story.** With retries, `tap_if "Clear search query"` reported ok four times and never cleared the box. A probe showed typing no longer reached the app either (`input text "zz"` left the field unchanged) and neither did taps on the tab bar, while `keyevent 3` (HOME) still worked. The app was deaf to input while still drawing. 10. **It recovers on its own, then wedges again.** After a 75-second rest with no dumps and no input, the screen had moved to Home — queued input had been processed — so a fresh run was started immediately. It failed at `tap "Search"`: the tap reported ok, the screen never left Home. 11. **Wrong row chosen, quietly.** After Sam restarted Spotify the whole recipe ran fast and ended at `playing` with `state=PAUSED`. The run log showed why: `tapped row "Honky Tonk Women" at (540,1443)` — the fifth result, not the top hit at `(540,878)`. Once the track is loaded its top row carries a `Paused` state indicator, and counting that description as row text dropped the top hit's F1 below an identical duplicate further down. Fixed, and the top row now loses no points for carrying state. 12. **Success.** With the ranking fixed, and with playback confirmed through the media session rather than the on-screen label (which stays `Play` even while music plays): ```sh $ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk" step 12 tap_row [Rolling Stones Honky Tonk]: ok (2.922s) step 13 playing [Rolling Stones Honky Tonk]: ok (1.714s) OK spotify_play in 35.311s ``` ``` tapped row "Honky Tonk Women" at (540,878) from Honky Tonk Women / Song • The Rolling Stones com.spotify.music is playing "Honky Tonk Women, The Rolling Stones, Honky Tonk Women / You Can't Always Get What You Want" ``` Independent reads, not Artemis's own claim: ``` $ adb shell dumpsys media_session | grep -A9 'package=com.spotify.music' state=PlaybackState {state=PLAYING(3), position=546, ...} $ adb shell dumpsys audio | grep -i 'state:started' AudioPlaybackConfiguration piid:22575 ... type:AAudio u/pid:10340/11987 state:started ... sampleRate=44100 ``` uid 10340 is Spotify, so audio really was being produced. A run was started with the track deliberately **paused** first, so the pass proves the row tap started playback rather than finding it already playing. ### Diagnosis This is the phone, not the recipe. Spotify's UI thread is starved: the phone is at `MemFree ~300MB` with 3.2GB of swap in use, load average ~4, and Spotify sustains ~70% CPU. When it is in that state it accepts no input at all — not injected taps, not injected keys — while `keyevent 3` (HOME, handled by the system) still works, and it recovers only after minutes of quiet. A force-stop would clear it, and that is a restart on Sam's phone, so it is not something Artemis does on its own. Artemis made this worse than it needed to be: `uiautomator dump` starts a JVM on the phone, and the wait helpers were polling every 400ms. `ui.PollInterval` is now 1500ms, which cuts that load by roughly two thirds. 13. **A second query met the wedged app again.** Straight after the successful run, `spotify_play query="Beatles Hey Jude"` failed at the clear-the-search-box step: the app never acted on the tap, and a 90-second rest did not clear it. Not a recipe fault — the same wedge as attempts 9 and 10, and this time the evidence is complete: ``` $ adb shell input motionevent DOWN 993 186 $ adb shell dumpsys input | sed -n '/Input Dispatcher State/,/Input Reader State/p' TouchStatesByDisplay: LogicalDisplayId 0: TouchedWindows: 0: name='44f7f5b com.spotify.music/com.spotify.music.MainActivity', targetFlags=FOREGROUND | SPLIT ... FINGER $ adb shell dumpsys activity lastanr | tail -2 $ adb shell ps -A -o PID,NAME,%CPU | grep spotify 11987 com.spotify.music 41.5 $ adb shell grep -E 'MemFree|SwapFree' /proc/meminfo MemFree: 253508 kB SwapFree: 338264 kB ``` The touch reaches Spotify's window; the app does not consume it; Android has not declared an ANR; the phone has 250MB free and has used 3.5GB of its 3.9GB swap. The first run of the heartbeat, from a quieter phone, completed in 35 seconds. This is the phone's resource state, not the recipe, and detecting "the app has stopped taking input" belongs to Phase 7 (resilience). ### Reproducing it ```sh cd ~/paseo/projects/artemis_control ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk" tail -1 logs/runs.jsonl adb shell dumpsys media_session | grep -A9 'package=com.spotify.music' ``` The phone must be awake and unlocked. On a quiet phone this takes about 35 seconds; on the loaded phone above it ranged from 90 seconds to over four minutes, which is what `total_timeout: 420s` is there to bound. Phase 4 (song radio) and Phase 5 (maps) have not been started, on instruction.