artemis_control/docs/phase-3-spotify-recipe.md

# Phase 3 — the recipes file and `spotify_play`

Phase 3 of `AGENTS.md` section 11: the first named recipe, plus the resolved
answer to section 13 Q2 (the locked screen).

Date: 2026-10-16. Phone: `Pixel_8a (akita)` over Tailscale at
`100.101.49.17:46689` (the port moves on every reboot; nothing hardcodes it).

## What was built

| File | What it holds |
|---|---|
| `recipes/spotify_play.recipe` | The recipe, written to be edited by hand. |
| `internal/recipe/recipe.go` | The file format: directives, guards, steps, parameters. |
| `internal/recipe/engine.go` | Runs a recipe step by step with timeouts and a run log. |
| `internal/ui/rows.go` | Picks a list row by text and taps its centre. |
| `internal/ui/keyguard.go` | Reads the lock state and refuses to act when locked. |
| `cmd/artemis/run.go` | `artemis recipes`, `artemis run <task> k=v`, JSON run log. |

## The recipes file

One file per task in `recipes/`, plain text, no build step. It names the app,
the parameters the caller must supply, the screens Artemis must never tap on,
and the ordered steps.

```
task: spotify_play
app: com.spotify.music
param: query
row_id: com.spotify.music:id/row_root

guard "Log in" => Spotify is signed out; sign in on the phone and retry

steps:
  - launch
  - wait "Search" 30s
  - tap "Search"
  - tap_if "Clear search query"
  - wait "What do you want to listen to?" 30s
  - tap "What do you want to listen to?"
  - type "{query}"
  - wait "Clear search query" 30s
  - wait_gone "Recents" 30s
  - tap_row "{query}" 40s
  - wait exact "Pause" 40s
```

Steps express intent. There is no coordinate anywhere in the file (FR-4).

Run it with:

```sh
./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
```

The verbs are `launch`, `foreground`, `wait`, `wait_gone`, `tap`, `tap_if`,
`tap_row`, `type`, `key`, `back`, `home` and `sleep`. A `wait` step may add
`retry N` (see below). `tap exact "Pause"`
matches the whole text, not a substring. `tap_if` taps only when the text is
already on screen, which lets one recipe cope with an app that remembers its
last screen. `foreground` waits until the recipe's app is really in front, so a
word like "Search" cannot be matched from a different app that happens to be
open — every recipe should use it straight after `launch`.

## Why `tap_row` exists

A plain text tap picked the wrong element on the real Spotify screen. On the
results list, `artemis tap "Honky Tonk Women"` tapped

```
ImageButton[context_menu_button] text="More options for song Honky Tonk Women - Live"
  bounds=[828,2388][954,2400]
```

That is the smallest clickable match, and it sits at the bottom edge of the
screen. It was the wrong target: a "More options" button, not the song.

`tap_row` scores every result row against the query and taps the row itself:

* A candidate row is a clickable container that carries a title **and** a
  subtitle. A one-line search suggestion (Spotify's "rolling stones honky tonk
  women single") is therefore not a result row.
* Text inside a clickable child is skipped, because that is an action button
  ("More options", "Add to library", "Add suggestion"), not the row's own text.
* A row clipped by the bottom of the list is dropped. It is much shorter than a
  row that is fully drawn, and its centre sits under the navigation area.
* Rows are ranked by F1 score over the query words, so an exact title beats a
  noisy deluxe-edition title. Equal scores go to the highest row, because search
  results are ranked by relevance.

On the real screen for `Rolling Stones Honky Tonk`, the top hit wins:

```
Honky Tonk Women / Song • The Rolling Stones   bounds=[0,794][1080,962]  F1=0.727
```

and the tap lands at the row's centre, `(540,878)` — on the title, clear of the
action buttons. `internal/ui/rows_test.go` runs this against a trimmed copy of
the real dump.

## Failing cleanly (FR-8)

Every failure names its cause, and they are distinguishable:

| Situation | Message |
|---|---|
| App missing | `app is not installed: com.spotify.music` |
| Signed out | `Spotify is signed out; sign in on the phone and retry (phone is showing "Log in")` |
| Update nag | `Spotify wants an update; update it on the phone and retry` |
| Element missing | `text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Log in", "Sign up free"` |
| Phone locked | `phone is locked; unlock it and retry` |

A guard is checked when a step cannot find its target, so a bad screen produces
its own reason instead of a generic "element not found".

## Timeouts (FR-11)

`step_timeout: 20s` is the default per step, `total_timeout: 420s` bounds the
whole run. A step may name its own timeout (`wait "Search" 30s`). The engine
checks the remaining budget before each step and stops when it runs out. Every
`adb` call is already bounded by `adb.DefaultTimeout`.

## Logging (FR-10)

Every run appends one JSON line to `logs/runs.jsonl` (override with
`ARTEMIS_LOG`): task, params, each step with its duration, the result and the
total duration. Two real entries:

```json
{"time":"2026-10-16T13:23:31+11:00","task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"},"ok":false,"error":"phone is locked; unlock it and retry","steps":null,"duration_ms":832}
```

The same steps print to the terminal as they run.

## The locked screen (section 13 Q2)

**Decision: Artemis fails loudly. No PIN is stored and none is ever typed.**

`ui.KeyguardState` reads two things: whether the lock screen is showing
(`dumpsys window`, the `KeyguardServiceDelegate` block) and whether the device
has a secure lock at all (`cmd lock_settings is-user-secure`). Acting commands
wake the screen first and then refuse if the keyguard is showing and secure. A
non-secure keyguard is still dismissed, as before. An unreadable lock state is a
failure, not a guess.

### Evidence

The prescribed test (put the phone to sleep, wake it, Artemis must refuse):

```sh
adb -s 100.101.49.17:46689 shell input keyevent 26    # POWER = sleep
adb -s 100.101.49.17:46689 shell input keyevent 224   # WAKEUP
```

```
$ adb shell dumpsys window policy | grep -E 'showing=|screenState'
      showing=true
      screenState=SCREEN_STATE_ON
$ adb shell cmd lock_settings is-user-secure
true

$ ./bin/artemis keyguard
keyguard showing=true secure=true
artemis: phone is locked; unlock it and retry
exit=1

$ ./bin/artemis tap "Search"
artemis: phone is locked; unlock it and retry
exit=1

$ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
task spotify_play (Search Spotify for a song and play it.)
FAILED spotify_play after 833ms: phone is locked; unlock it and retry
exit=1
```

No PIN was entered, nothing was read from the phone, and the lock screen was
still on screen afterwards — no blind tap happened. The screen dump after the
refusals still showed the lock screen clock and date.

A dozing phone reports `showing=false`, because the lock screen is not drawn
yet. That is why every acting command wakes the screen **before** it checks the
keyguard; reading the state first would say "unlocked" about a locked phone.

## Verification status

| Item | State |
|---|---|
| Recipes file format, guards, parameters, timeouts | unit tested (`internal/recipe`) |
| Row selection against a real Spotify dump | unit tested (`internal/ui`) |
| Keyguard parsing and secure-lock detection | unit tested (`internal/ui`) |
| Step durations reach the run log | unit tested (`internal/recipe`) |
| All acting commands refuse on a locked phone | **live, verified** (above) |
| `artemis run spotify_play` refuses on a locked phone | **live, verified** (above) |
| `spotify_play` plays the right song | **not yet run end to end** — see below |
| `spotify_play` picks the right row on the real results list | **live, verified** — `tapped row "Honky Tonk Women" at (540,878)` |
| Spotify search reaches the result list | live (Home → Search → type → results) |

## Retrying a dropped tap

A step can ask for the step above it to be pressed again:

```
- tap_row "{query}" 40s
- wait exact "Pause" 45s retry 3
```

`retry N` is only allowed on a `wait` step, needs a step above it to repeat, and
never taps anything that was not matched by text. It exists because a busy phone
can deliver a tap and then not act on it. The retry only re-runs the step above,
so nothing else can be pressed by accident.

## Live run log

Six attempts, in order. Each failure was clean, named its screen and tapped
nothing random — which is the point of the phase.

1. **Refused, locked phone.** `phone is locked; unlock it and retry`. The phone
   had locked itself on its 5-minute screen timeout while the code was being
   written.
2. **Refused after the prescribed sleep/wake test.** `keyguard showing=true
   secure=true`, exit 1 from `keyguard`, `tap` and `run`. The lock screen was
   still on screen afterwards.
3. **Refused, dozing phone that only *looked* unlocked.** A dozing phone reports
   `showing=false`. Pressing WAKEUP then revealed the secure keyguard, and the
   run refused. This is why every acting command wakes first.
4. **Wrong app matched "Search".** The phone was sitting on the Android
   Settings app. `wait "Search"` was satisfied by Settings' own "Search
   Settings" list entry, so the recipe tapped the Settings search bar. Fixed by
   adding the `foreground` step, which waits until the recipe's app really is in
   front before any text is trusted.
5. **Blocked by a biometric prompt.** Settings was left showing a
   `Verify that it's you` dialog (`Touch the fingerprint sensor` / `Use PIN`),
   which is drawn over Spotify. The run failed with:
   `text "Search" did not appear within 30s; foreground=com.spotify.music;
   screen shows: "Cancel", "App logo", "Settings", "Verify that it's you" ...`.
   Artemis did not touch the prompt, and it must not: it needs Sam's fingerprint
   or PIN, exactly like the lock screen.
6. **Same prompt again**, until Sam cleared it.
7. **Through to the row, but no playback.** After the prompt was cleared the
   recipe reached step 11 (`tap_row`) with every earlier step ok, and the run
   log shows it chose exactly the right element:
   `tapped row "Honky Tonk Women" at (540,878) from Honky Tonk Women /
   Song • The Rolling Stones`. Only the last step failed:
   `text "Pause" did not appear within 40s`, with the search results still on
   screen. The tap was delivered (the dispatcher lists the Spotify window as the
   touch target) and Spotify did nothing with it.
8. **Duration logging fixed, retry added.** `wait ... retry N` now re-runs the
   step above and waits again. Steps 9–12 made four attempts each.
9. **A dropped tap is not the whole story.** With retries, `tap_if "Clear
   search query"` reported ok four times and never cleared the box. A probe
   showed typing no longer reached the app either (`input text "zz"` left the
   field unchanged) and neither did taps on the tab bar, while `keyevent 3`
   (HOME) still worked. The app was deaf to input while still drawing.
10. **It recovers on its own, then wedges again.** After a 75-second rest with
    no dumps and no input, the screen had moved to Home — queued input had been
    processed — so a fresh run was started immediately. It failed at
    `tap "Search"`: the tap reported ok, the screen never left Home.

### Diagnosis

This is the phone, not the recipe. Spotify's UI thread is starved: the phone is
at `MemFree ~300MB` with 3.2GB of swap in use, load average ~4, and Spotify
sustains ~70% CPU. When it is in that state it accepts no input at all — not
injected taps, not injected keys — while `keyevent 3` (HOME, handled by the
system) still works, and it recovers only after minutes of quiet. A force-stop
would clear it, and that is a restart on Sam's phone, so it is not something
Artemis does on its own.

Artemis made this worse than it needed to be: `uiautomator dump` starts a JVM on
the phone, and the wait helpers were polling every 400ms. `ui.PollInterval` is
now 1500ms, which cuts that load by roughly two thirds.

### What is left

The live end-to-end run of `spotify_play` (Phase 3 DoD) has not completed. It
needs the phone's Spotify restarted — swipe it away from Recents, or force-stop
it — and the phone left awake and unlocked. Then:

```sh
./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
./bin/artemis fg
./bin/artemis dump | grep -i pause
tail -1 logs/runs.jsonl
```

Phase 4 (song radio) and Phase 5 (maps) have not been started, on instruction.