artemis_control/docs/phase-3-spotify-recipe.md

Phase 3 — the recipes file and spotify_play

Phase 3 of AGENTS.md section 11: the first named recipe, plus the resolved answer to section 13 Q2 (the locked screen).

Date: 2026-10-16. Phone: Pixel_8a (akita) over Tailscale at 100.101.49.17:46689 (the port moves on every reboot; nothing hardcodes it).

What was built

File	What it holds
recipes/spotify_play.recipe	The recipe, written to be edited by hand.
internal/recipe/recipe.go	The file format: directives, guards, steps, parameters.
internal/recipe/engine.go	Runs a recipe step by step with timeouts and a run log.
internal/ui/rows.go	Picks a list row by text and taps its centre.
internal/ui/keyguard.go	Reads the lock state and refuses to act when locked.
cmd/artemis/run.go	artemis recipes, artemis run <task> k=v, JSON run log.

The recipes file

One file per task in recipes/, plain text, no build step. It names the app, the parameters the caller must supply, the screens Artemis must never tap on, and the ordered steps.

task: spotify_play
app: com.spotify.music
param: query
row_id: com.spotify.music:id/row_root

guard "Log in" => Spotify is signed out; sign in on the phone and retry

steps:
  - launch
  - wait "Search" 30s
  - tap "Search"
  - tap_if "Clear search query"
  - wait "What do you want to listen to?" 30s
  - tap "What do you want to listen to?"
  - type "{query}"
  - wait "Clear search query" 30s
  - wait_gone "Recents" 30s
  - tap_row "{query}" 40s
  - wait exact "Pause" 40s

Steps express intent. There is no coordinate anywhere in the file (FR-4).

Run it with:

./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"

The verbs are launch, foreground, wait, wait_gone, tap, tap_if, tap_row, type, key, back, home and sleep. tap exact "Pause" matches the whole text, not a substring. tap_if taps only when the text is already on screen, which lets one recipe cope with an app that remembers its last screen. foreground waits until the recipe's app is really in front, so a word like "Search" cannot be matched from a different app that happens to be open — every recipe should use it straight after launch.

Why tap_row exists

A plain text tap picked the wrong element on the real Spotify screen. On the results list, artemis tap "Honky Tonk Women" tapped

ImageButton[context_menu_button] text="More options for song Honky Tonk Women - Live"
  bounds=[828,2388][954,2400]

That is the smallest clickable match, and it sits at the bottom edge of the screen. It was the wrong target: a "More options" button, not the song.

tap_row scores every result row against the query and taps the row itself:

• A candidate row is a clickable container that carries a title and a subtitle. A one-line search suggestion (Spotify's "rolling stones honky tonk women single") is therefore not a result row.
• Text inside a clickable child is skipped, because that is an action button ("More options", "Add to library", "Add suggestion"), not the row's own text.
• A row clipped by the bottom of the list is dropped. It is much shorter than a row that is fully drawn, and its centre sits under the navigation area.
• Rows are ranked by F1 score over the query words, so an exact title beats a noisy deluxe-edition title. Equal scores go to the highest row, because search results are ranked by relevance.

On the real screen for Rolling Stones Honky Tonk, the top hit wins:

Honky Tonk Women / Song • The Rolling Stones   bounds=[0,794][1080,962]  F1=0.727

and the tap lands at the row's centre, (540,878) — on the title, clear of the action buttons. internal/ui/rows_test.go runs this against a trimmed copy of the real dump.

Failing cleanly (FR-8)

Every failure names its cause, and they are distinguishable:

Situation	Message
App missing	app is not installed: com.spotify.music
Signed out	Spotify is signed out; sign in on the phone and retry (phone is showing "Log in")
Update nag	Spotify wants an update; update it on the phone and retry
Element missing	text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Log in", "Sign up free"
Phone locked	phone is locked; unlock it and retry

A guard is checked when a step cannot find its target, so a bad screen produces its own reason instead of a generic "element not found".

Timeouts (FR-11)

step_timeout: 20s is the default per step, total_timeout: 150s bounds the whole run. A step may name its own timeout (wait "Search" 30s). The engine checks the remaining budget before each step and stops when it runs out. Every adb call is already bounded by adb.DefaultTimeout.

Logging (FR-10)

Every run appends one JSON line to logs/runs.jsonl (override with ARTEMIS_LOG): task, params, each step with its duration, the result and the total duration. Two real entries:

{"time":"2026-10-16T13:23:31+11:00","task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"},"ok":false,"error":"phone is locked; unlock it and retry","steps":null,"duration_ms":832}

The same steps print to the terminal as they run.

The locked screen (section 13 Q2)

Decision: Artemis fails loudly. No PIN is stored and none is ever typed.

ui.KeyguardState reads two things: whether the lock screen is showing (dumpsys window, the KeyguardServiceDelegate block) and whether the device has a secure lock at all (cmd lock_settings is-user-secure). Acting commands wake the screen first and then refuse if the keyguard is showing and secure. A non-secure keyguard is still dismissed, as before. An unreadable lock state is a failure, not a guess.

Evidence

The prescribed test (put the phone to sleep, wake it, Artemis must refuse):

adb -s 100.101.49.17:46689 shell input keyevent 26    # POWER = sleep
adb -s 100.101.49.17:46689 shell input keyevent 224   # WAKEUP

$ adb shell dumpsys window policy | grep -E 'showing=|screenState'
      showing=true
      screenState=SCREEN_STATE_ON
$ adb shell cmd lock_settings is-user-secure
true

$ ./bin/artemis keyguard
keyguard showing=true secure=true
artemis: phone is locked; unlock it and retry
exit=1

$ ./bin/artemis tap "Search"
artemis: phone is locked; unlock it and retry
exit=1

$ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
task spotify_play (Search Spotify for a song and play it.)
FAILED spotify_play after 833ms: phone is locked; unlock it and retry
exit=1

No PIN was entered, nothing was read from the phone, and the lock screen was still on screen afterwards — no blind tap happened. The screen dump after the refusals still showed the lock screen clock and date.

A dozing phone reports showing=false, because the lock screen is not drawn yet. That is why every acting command wakes the screen before it checks the keyguard; reading the state first would say "unlocked" about a locked phone.

Verification status

Item	State
Recipes file format, guards, parameters, timeouts	unit tested (internal/recipe)
Row selection against a real Spotify dump	unit tested (internal/ui)
Keyguard parsing and secure-lock detection	unit tested (internal/ui)
Step durations reach the run log	unit tested (internal/recipe)
All acting commands refuse on a locked phone	live, verified (above)
artemis run spotify_play refuses on a locked phone	live, verified (above)
spotify_play plays the right song	not yet run end to end — see below
Spotify search reaches the result list	live (Home → Search → type → results)

Live run log

Six attempts, in order. Each failure was clean, named its screen and tapped nothing random — which is the point of the phase.

1. Refused, locked phone. phone is locked; unlock it and retry. The phone had locked itself on its 5-minute screen timeout while the code was being written.
2. Refused after the prescribed sleep/wake test. keyguard showing=true secure=true, exit 1 from keyguard, tap and run. The lock screen was still on screen afterwards.
3. Refused, dozing phone that only looked unlocked. A dozing phone reports showing=false. Pressing WAKEUP then revealed the secure keyguard, and the run refused. This is why every acting command wakes first.
4. Wrong app matched "Search". The phone was sitting on the Android Settings app. wait "Search" was satisfied by Settings' own "Search Settings" list entry, so the recipe tapped the Settings search bar. Fixed by adding the foreground step, which waits until the recipe's app really is in front before any text is trusted.
5. Blocked by a biometric prompt. Settings was left showing a Verify that it's you dialog (Touch the fingerprint sensor / Use PIN), which is drawn over Spotify. The run failed with: text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Cancel", "App logo", "Settings", "Verify that it's you" .... Artemis did not touch the prompt, and it must not: it needs Sam's fingerprint or PIN, exactly like the lock screen.
6. Still blocked by the same prompt on the next attempt.

What is left

The live end-to-end run of spotify_play (Phase 3 DoD) has not completed. It needs the phone unlocked and free of the biometric prompt, and then:

./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
./bin/artemis fg
./bin/artemis dump | grep -i pause
tail -1 logs/runs.jsonl

Phase 4 (song radio) and Phase 5 (maps) have not been started, on instruction.