artemis_control/docs/phase-3-spotify-recipe.md

Phase 3 — the recipes file and spotify_play

Phase 3 of AGENTS.md section 11: the first named recipe, plus the resolved answer to section 13 Q2 (the locked screen).

Date: 2026-10-16. Phone: Pixel_8a (akita) over Tailscale at 100.101.49.17:46689 (the port moves on every reboot; nothing hardcodes it).

What was built

File	What it holds
recipes/spotify_play.recipe	The recipe, written to be edited by hand.
internal/recipe/recipe.go	The file format: directives, guards, steps, parameters.
internal/recipe/engine.go	Runs a recipe step by step with timeouts and a run log.
internal/ui/rows.go	Picks a list row by text and taps its centre.
internal/ui/keyguard.go	Reads the lock state and refuses to act when locked.
cmd/artemis/run.go	artemis recipes, artemis run <task> k=v, JSON run log.

The recipes file

One file per task in recipes/, plain text, no build step. It names the app, the parameters the caller must supply, the screens Artemis must never tap on, and the ordered steps.

task: spotify_play
app: com.spotify.music
param: query
row_id: com.spotify.music:id/row_root

guard "Log in" => Spotify is signed out; sign in on the phone and retry

steps:
  - launch
  - wait "Search" 30s
  - tap "Search"
  - tap_if "Clear search query"
  - wait "What do you want to listen to?" 30s
  - tap "What do you want to listen to?"
  - type "{query}"
  - wait "Clear search query" 30s
  - wait_gone "Recents" 30s
  - tap_row "{query}" 40s
  - wait exact "Pause" 40s

Steps express intent. There is no coordinate anywhere in the file (FR-4).

Run it with:

./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"

The verbs are launch, foreground, wait, wait_gone, tap, tap_if, tap_row, type, key, back, home and sleep. A wait step may add retry N (see below). tap exact "Pause" matches the whole text, not a substring. tap_if taps only when the text is already on screen, which lets one recipe cope with an app that remembers its last screen. foreground waits until the recipe's app is really in front, so a word like "Search" cannot be matched from a different app that happens to be open — every recipe should use it straight after launch.

Why tap_row exists

A plain text tap picked the wrong element on the real Spotify screen. On the results list, artemis tap "Honky Tonk Women" tapped

ImageButton[context_menu_button] text="More options for song Honky Tonk Women - Live"
  bounds=[828,2388][954,2400]

That is the smallest clickable match, and it sits at the bottom edge of the screen. It was the wrong target: a "More options" button, not the song.

tap_row scores every result row against the query and taps the row itself:

• A candidate row is a clickable container that carries a title and a subtitle. A one-line search suggestion (Spotify's "rolling stones honky tonk women single") is therefore not a result row.
• Text inside a clickable child is skipped, because that is an action button ("More options", "Add to library", "Add suggestion"), not the row's own text.
• A row clipped by the bottom of the list is dropped. It is much shorter than a row that is fully drawn, and its centre sits under the navigation area.
• Rows are ranked by F1 score over the query words, so an exact title beats a noisy deluxe-edition title. Equal scores go to the highest row, because search results are ranked by relevance.

On the real screen for Rolling Stones Honky Tonk, the top hit wins:

Honky Tonk Women / Song • The Rolling Stones   bounds=[0,794][1080,962]  F1=0.727

and the tap lands at the row's centre, (540,878) — on the title, clear of the action buttons. internal/ui/rows_test.go runs this against a trimmed copy of the real dump.

Failing cleanly (FR-8)

Every failure names its cause, and they are distinguishable:

Situation	Message
App missing	app is not installed: com.spotify.music
Signed out	Spotify is signed out; sign in on the phone and retry (phone is showing "Log in")
Update nag	Spotify wants an update; update it on the phone and retry
Element missing	text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Log in", "Sign up free"
Phone locked	phone is locked; unlock it and retry

A guard is checked when a step cannot find its target, so a bad screen produces its own reason instead of a generic "element not found".

Timeouts (FR-11)

step_timeout: 20s is the default per step, total_timeout: 420s bounds the whole run. A step may name its own timeout (wait "Search" 30s). The engine checks the remaining budget before each step and stops when it runs out. Every adb call is already bounded by adb.DefaultTimeout.

Logging (FR-10)

Every run appends one JSON line to logs/runs.jsonl (override with ARTEMIS_LOG): task, params, each step with its duration, the result and the total duration. Two real entries:

{"time":"2026-10-16T13:23:31+11:00","task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"},"ok":false,"error":"phone is locked; unlock it and retry","steps":null,"duration_ms":832}

The same steps print to the terminal as they run.

The locked screen (section 13 Q2)

Decision: Artemis fails loudly. No PIN is stored and none is ever typed.

ui.KeyguardState reads two things: whether the lock screen is showing (dumpsys window, the KeyguardServiceDelegate block) and whether the device has a secure lock at all (cmd lock_settings is-user-secure). Acting commands wake the screen first and then refuse if the keyguard is showing and secure. A non-secure keyguard is still dismissed, as before. An unreadable lock state is a failure, not a guess.

Evidence

The prescribed test (put the phone to sleep, wake it, Artemis must refuse):

adb -s 100.101.49.17:46689 shell input keyevent 26    # POWER = sleep
adb -s 100.101.49.17:46689 shell input keyevent 224   # WAKEUP

$ adb shell dumpsys window policy | grep -E 'showing=|screenState'
      showing=true
      screenState=SCREEN_STATE_ON
$ adb shell cmd lock_settings is-user-secure
true

$ ./bin/artemis keyguard
keyguard showing=true secure=true
artemis: phone is locked; unlock it and retry
exit=1

$ ./bin/artemis tap "Search"
artemis: phone is locked; unlock it and retry
exit=1

$ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
task spotify_play (Search Spotify for a song and play it.)
FAILED spotify_play after 833ms: phone is locked; unlock it and retry
exit=1

No PIN was entered, nothing was read from the phone, and the lock screen was still on screen afterwards — no blind tap happened. The screen dump after the refusals still showed the lock screen clock and date.

A dozing phone reports showing=false, because the lock screen is not drawn yet. That is why every acting command wakes the screen before it checks the keyguard; reading the state first would say "unlocked" about a locked phone.

Verification status

Item	State
Recipes file format, guards, parameters, timeouts	unit tested (internal/recipe)
Row selection against a real Spotify dump	unit tested (internal/ui)
Keyguard parsing and secure-lock detection	unit tested (internal/ui)
Step durations reach the run log	unit tested (internal/recipe)
All acting commands refuse on a locked phone	live, verified (above)
artemis run spotify_play refuses on a locked phone	live, verified (above)
spotify_play plays the right song	not yet run end to end — see below
spotify_play picks the right row on the real results list	live, verified — tapped row "Honky Tonk Women" at (540,878)
Spotify search reaches the result list	live (Home → Search → type → results)

Retrying a dropped tap

A step can ask for the step above it to be pressed again:

- tap_row "{query}" 40s
- wait exact "Pause" 45s retry 3

retry N is only allowed on a wait step, needs a step above it to repeat, and never taps anything that was not matched by text. It exists because a busy phone can deliver a tap and then not act on it. The retry only re-runs the step above, so nothing else can be pressed by accident.

Live run log

Six attempts, in order. Each failure was clean, named its screen and tapped nothing random — which is the point of the phase.

1. Refused, locked phone. phone is locked; unlock it and retry. The phone had locked itself on its 5-minute screen timeout while the code was being written.
2. Refused after the prescribed sleep/wake test. keyguard showing=true secure=true, exit 1 from keyguard, tap and run. The lock screen was still on screen afterwards.
3. Refused, dozing phone that only looked unlocked. A dozing phone reports showing=false. Pressing WAKEUP then revealed the secure keyguard, and the run refused. This is why every acting command wakes first.
4. Wrong app matched "Search". The phone was sitting on the Android Settings app. wait "Search" was satisfied by Settings' own "Search Settings" list entry, so the recipe tapped the Settings search bar. Fixed by adding the foreground step, which waits until the recipe's app really is in front before any text is trusted.
5. Blocked by a biometric prompt. Settings was left showing a Verify that it's you dialog (Touch the fingerprint sensor / Use PIN), which is drawn over Spotify. The run failed with: text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Cancel", "App logo", "Settings", "Verify that it's you" .... Artemis did not touch the prompt, and it must not: it needs Sam's fingerprint or PIN, exactly like the lock screen.
6. Same prompt again, until Sam cleared it.
7. Through to the row, but no playback. After the prompt was cleared the recipe reached step 11 (tap_row) with every earlier step ok, and the run log shows it chose exactly the right element: tapped row "Honky Tonk Women" at (540,878) from Honky Tonk Women / Song • The Rolling Stones. Only the last step failed: text "Pause" did not appear within 40s, with the search results still on screen. The tap was delivered (the dispatcher lists the Spotify window as the touch target) and Spotify did nothing with it.
8. Duration logging fixed, retry added. wait ... retry N now re-runs the step above and waits again. Steps 9–12 made four attempts each.
9. A dropped tap is not the whole story. With retries, tap_if "Clear search query" reported ok four times and never cleared the box. A probe showed typing no longer reached the app either (input text "zz" left the field unchanged) and neither did taps on the tab bar, while keyevent 3 (HOME) still worked. The app was deaf to input while still drawing.
10. It recovers on its own, then wedges again. After a 75-second rest with no dumps and no input, the screen had moved to Home — queued input had been processed — so a fresh run was started immediately. It failed at tap "Search": the tap reported ok, the screen never left Home.

Diagnosis

This is the phone, not the recipe. Spotify's UI thread is starved: the phone is at MemFree ~300MB with 3.2GB of swap in use, load average ~4, and Spotify sustains ~70% CPU. When it is in that state it accepts no input at all — not injected taps, not injected keys — while keyevent 3 (HOME, handled by the system) still works, and it recovers only after minutes of quiet. A force-stop would clear it, and that is a restart on Sam's phone, so it is not something Artemis does on its own.

Artemis made this worse than it needed to be: uiautomator dump starts a JVM on the phone, and the wait helpers were polling every 400ms. ui.PollInterval is now 1500ms, which cuts that load by roughly two thirds.

What is left

The live end-to-end run of spotify_play (Phase 3 DoD) has not completed. It needs the phone's Spotify restarted — swipe it away from Recents, or force-stop it — and the phone left awake and unlocked. Then:

./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
./bin/artemis fg
./bin/artemis dump | grep -i pause
tail -1 logs/runs.jsonl

Phase 4 (song radio) and Phase 5 (maps) have not been started, on instruction.