artemis_control/docs/phase-3-spotify-recipe.md

# Phase 3 — the recipes file and `spotify_play`

Phase 3 of `AGENTS.md` section 11: the first named recipe, plus the resolved
answer to section 13 Q2 (the locked screen).

Date: 2026-10-16. Phone: `Pixel_8a (akita)` over Tailscale at
`100.101.49.17:46689` (the port moves on every reboot; nothing hardcodes it).

## What was built

| File | What it holds |
|---|---|
| `recipes/spotify_play.recipe` | The recipe, written to be edited by hand. |
| `internal/recipe/recipe.go` | The file format: directives, guards, steps, parameters. |
| `internal/recipe/engine.go` | Runs a recipe step by step with timeouts and a run log. |
| `internal/ui/rows.go` | Picks a list row by text and taps its centre. |
| `internal/ui/keyguard.go` | Reads the lock state and refuses to act when locked. |
| `cmd/artemis/run.go` | `artemis recipes`, `artemis run <task> k=v`, JSON run log. |

## The recipes file

One file per task in `recipes/`, plain text, no build step. It names the app,
the parameters the caller must supply, the screens Artemis must never tap on,
and the ordered steps.

```
task: spotify_play
app: com.spotify.music
param: query
row_id: com.spotify.music:id/row_root

guard "Log in" => Spotify is signed out; sign in on the phone and retry

steps:
  - launch
  - wait "Search" 30s
  - tap "Search"
  - tap_if "Clear search query"
  - wait "What do you want to listen to?" 30s
  - tap "What do you want to listen to?"
  - type "{query}"
  - wait "Clear search query" 30s
  - wait_gone "Recents" 30s
  - tap_row "{query}" 40s
  - wait exact "Pause" 40s
```

Steps express intent. There is no coordinate anywhere in the file (FR-4).

Run it with:

```sh
./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
```

The verbs are `launch`, `foreground`, `wait`, `wait_gone`, `tap`, `tap_if`,
`tap_row`, `type`, `key`, `back`, `home` and `sleep`. A `wait` step may add
`retry N` (see below). `tap exact "Pause"`
matches the whole text, not a substring. `tap_if` taps only when the text is
already on screen, which lets one recipe cope with an app that remembers its
last screen. `foreground` waits until the recipe's app is really in front, so a
word like "Search" cannot be matched from a different app that happens to be
open — every recipe should use it straight after `launch`.

## Why `tap_row` exists

A plain text tap picked the wrong element on the real Spotify screen. On the
results list, `artemis tap "Honky Tonk Women"` tapped

```
ImageButton[context_menu_button] text="More options for song Honky Tonk Women - Live"
  bounds=[828,2388][954,2400]
```

That is the smallest clickable match, and it sits at the bottom edge of the
screen. It was the wrong target: a "More options" button, not the song.

`tap_row` scores every result row against the query and taps the row itself:

* A candidate row is a clickable container that carries a title **and** a
  subtitle. A one-line search suggestion (Spotify's "rolling stones honky tonk
  women single") is therefore not a result row.
* Text inside a clickable child is skipped, because that is an action button
  ("More options", "Add to library", "Add suggestion"), not the row's own text.
* A row clipped by the bottom of the list is dropped. It is much shorter than a
  row that is fully drawn, and its centre sits under the navigation area.
* Rows are ranked by F1 score over the query words, so an exact title beats a
  noisy deluxe-edition title. Equal scores go to the highest row, because search
  results are ranked by relevance.

On the real screen for `Rolling Stones Honky Tonk`, the top hit wins:

```
Honky Tonk Women / Song • The Rolling Stones   bounds=[0,794][1080,962]  F1=0.727
```

and the tap lands at the row's centre, `(540,878)` — on the title, clear of the
action buttons. `internal/ui/rows_test.go` runs this against a trimmed copy of
the real dump.

## Failing cleanly (FR-8)

Every failure names its cause, and they are distinguishable:

| Situation | Message |
|---|---|
| App missing | `app is not installed: com.spotify.music` |
| Signed out | `Spotify is signed out; sign in on the phone and retry (phone is showing "Log in")` |
| Update nag | `Spotify wants an update; update it on the phone and retry` |
| Element missing | `text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Log in", "Sign up free"` |
| Phone locked | `phone is locked; unlock it and retry` |

A guard is checked when a step cannot find its target, so a bad screen produces
its own reason instead of a generic "element not found".

## Timeouts (FR-11)

`step_timeout: 20s` is the default per step, `total_timeout: 420s` bounds the
whole run. A step may name its own timeout (`wait "Search" 30s`). The engine
checks the remaining budget before each step and stops when it runs out. Every
`adb` call is already bounded by `adb.DefaultTimeout`.

## Logging (FR-10)

Every run appends one JSON line to `logs/runs.jsonl` (override with
`ARTEMIS_LOG`): task, params, each step with its duration, the result and the
total duration. Two real entries:

```json
{"time":"2026-10-16T13:23:31+11:00","task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"},"ok":false,"error":"phone is locked; unlock it and retry","steps":null,"duration_ms":832}
```

The same steps print to the terminal as they run.

## The locked screen (section 13 Q2)

**Decision: Artemis fails loudly. No PIN is stored and none is ever typed.**

`ui.KeyguardState` reads two things: whether the lock screen is showing
(`dumpsys window`, the `KeyguardServiceDelegate` block) and whether the device
has a secure lock at all (`cmd lock_settings is-user-secure`). Acting commands
wake the screen first and then refuse if the keyguard is showing and secure. A
non-secure keyguard is still dismissed, as before. An unreadable lock state is a
failure, not a guess.

### Evidence

The prescribed test (put the phone to sleep, wake it, Artemis must refuse):

```sh
adb -s 100.101.49.17:46689 shell input keyevent 26    # POWER = sleep
adb -s 100.101.49.17:46689 shell input keyevent 224   # WAKEUP
```

```
$ adb shell dumpsys window policy | grep -E 'showing=|screenState'
      showing=true
      screenState=SCREEN_STATE_ON
$ adb shell cmd lock_settings is-user-secure
true

$ ./bin/artemis keyguard
keyguard showing=true secure=true
artemis: phone is locked; unlock it and retry
exit=1

$ ./bin/artemis tap "Search"
artemis: phone is locked; unlock it and retry
exit=1

$ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
task spotify_play (Search Spotify for a song and play it.)
FAILED spotify_play after 833ms: phone is locked; unlock it and retry
exit=1
```

No PIN was entered, nothing was read from the phone, and the lock screen was
still on screen afterwards — no blind tap happened. The screen dump after the
refusals still showed the lock screen clock and date.

A dozing phone reports `showing=false`, because the lock screen is not drawn
yet. That is why every acting command wakes the screen **before** it checks the
keyguard; reading the state first would say "unlocked" about a locked phone.

## Verification status

| Item | State |
|---|---|
| Recipes file format, guards, parameters, timeouts | unit tested (`internal/recipe`) |
| Row selection against a real Spotify dump | unit tested (`internal/ui`) |
| Keyguard parsing and secure-lock detection | unit tested (`internal/ui`) |
| Step durations reach the run log | unit tested (`internal/recipe`) |
| All acting commands refuse on a locked phone | **live, verified** (above) |
| `artemis run spotify_play` refuses on a locked phone | **live, verified** (above) |
| `spotify_play` plays the right song | **live, verified end to end** — 13 steps, 35.3s |
| `spotify_play` picks the right row on the real results list | **live, verified** — `tapped row "Honky Tonk Women" at (540,878)` |
| Spotify search reaches the result list | live (Home → Search → type → results) |

## Confirming playback

The screen is not a trustworthy place to read playback state. Spotify's
transport button keeps the accessibility label **"Play"** while music is
actually playing, so the first version of this recipe asserted on a label that
never appears and failed on a phone that was playing the track perfectly.

The last step asks Android instead:

```
- playing "{query}" 30s retry 3
```

`playing` waits until the app's media session reports `PLAYING` **and** its
metadata contains the query's words. It fails with the state and the track it
found, for example:

```
com.spotify.music is not playing "Rolling Stones Honky Tonk" within 30s
  (state=PAUSED, playing "Honky Tonk Women, The Rolling Stones, ...")
```

## Why the top row wins

Spotify ranks results by relevance, so the highest acceptable row is the one a
human would press. Two rules keep it that way:

- A row's own text is its title and subtitle. Text inside a clickable child is
  an action button, and **content descriptions are state**, not text: the row of
  the track that is already loaded carries `content-desc="Paused"`. Counting
  that made the top hit score *worse* than an identical duplicate further down,
  and `tap_row` pressed the duplicate at `y=1443` while the right row sat at
  `y=878`. `internal/ui/rows_test.go` reproduces that screen exactly.
- Scores within `rowScoreTolerance` (0.15) are treated as equal, and then the
  higher row wins. A clearly better text match still wins outright.

## Retrying a dropped tap

A step can ask for the step above it to be pressed again:

```
- tap_row "{query}" 40s
- wait exact "Pause" 45s retry 3
```

`retry N` is only allowed on a `wait` step, needs a step above it to repeat, and
never taps anything that was not matched by text. It exists because a busy phone
can deliver a tap and then not act on it. The retry only re-runs the step above,
so nothing else can be pressed by accident.

## Live run log

Six attempts, in order. Each failure was clean, named its screen and tapped
nothing random — which is the point of the phase.

1. **Refused, locked phone.** `phone is locked; unlock it and retry`. The phone
   had locked itself on its 5-minute screen timeout while the code was being
   written.
2. **Refused after the prescribed sleep/wake test.** `keyguard showing=true
   secure=true`, exit 1 from `keyguard`, `tap` and `run`. The lock screen was
   still on screen afterwards.
3. **Refused, dozing phone that only *looked* unlocked.** A dozing phone reports
   `showing=false`. Pressing WAKEUP then revealed the secure keyguard, and the
   run refused. This is why every acting command wakes first.
4. **Wrong app matched "Search".** The phone was sitting on the Android
   Settings app. `wait "Search"` was satisfied by Settings' own "Search
   Settings" list entry, so the recipe tapped the Settings search bar. Fixed by
   adding the `foreground` step, which waits until the recipe's app really is in
   front before any text is trusted.
5. **Blocked by a biometric prompt.** Settings was left showing a
   `Verify that it's you` dialog (`Touch the fingerprint sensor` / `Use PIN`),
   which is drawn over Spotify. The run failed with:
   `text "Search" did not appear within 30s; foreground=com.spotify.music;
   screen shows: "Cancel", "App logo", "Settings", "Verify that it's you" ...`.
   Artemis did not touch the prompt, and it must not: it needs Sam's fingerprint
   or PIN, exactly like the lock screen.
6. **Same prompt again**, until Sam cleared it.
7. **Through to the row, but no playback.** After the prompt was cleared the
   recipe reached step 11 (`tap_row`) with every earlier step ok, and the run
   log shows it chose exactly the right element:
   `tapped row "Honky Tonk Women" at (540,878) from Honky Tonk Women /
   Song • The Rolling Stones`. Only the last step failed:
   `text "Pause" did not appear within 40s`, with the search results still on
   screen. The tap was delivered (the dispatcher lists the Spotify window as the
   touch target) and Spotify did nothing with it.
8. **Duration logging fixed, retry added.** `wait ... retry N` now re-runs the
   step above and waits again. Steps 9–12 made four attempts each.
9. **A dropped tap is not the whole story.** With retries, `tap_if "Clear
   search query"` reported ok four times and never cleared the box. A probe
   showed typing no longer reached the app either (`input text "zz"` left the
   field unchanged) and neither did taps on the tab bar, while `keyevent 3`
   (HOME) still worked. The app was deaf to input while still drawing.
10. **It recovers on its own, then wedges again.** After a 75-second rest with
    no dumps and no input, the screen had moved to Home — queued input had been
    processed — so a fresh run was started immediately. It failed at
    `tap "Search"`: the tap reported ok, the screen never left Home.
11. **Wrong row chosen, quietly.** After Sam restarted Spotify the whole recipe
    ran fast and ended at `playing` with `state=PAUSED`. The run log showed why:
    `tapped row "Honky Tonk Women" at (540,1443)` — the fifth result, not the top
    hit at `(540,878)`. Once the track is loaded its top row carries a `Paused`
    state indicator, and counting that description as row text dropped the top
    hit's F1 below an identical duplicate further down. Fixed, and the top row
    now loses no points for carrying state.
12. **Success.** With the ranking fixed, and with playback confirmed through the
    media session rather than the on-screen label (which stays `Play` even while
    music plays):

    ```sh
    $ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
      step 12 tap_row [Rolling Stones Honky Tonk]: ok (2.922s)
      step 13 playing [Rolling Stones Honky Tonk]: ok (1.714s)
    OK spotify_play in 35.311s
    ```

    ```
    tapped row "Honky Tonk Women" at (540,878)
      from Honky Tonk Women / Song • The Rolling Stones
    com.spotify.music is playing
      "Honky Tonk Women, The Rolling Stones, Honky Tonk Women / You Can't Always Get What You Want"
    ```

    Independent reads, not Artemis's own claim:

    ```
    $ adb shell dumpsys media_session | grep -A9 'package=com.spotify.music'
          state=PlaybackState {state=PLAYING(3), position=546, ...}
    $ adb shell dumpsys audio | grep -i 'state:started'
      AudioPlaybackConfiguration piid:22575 ... type:AAudio u/pid:10340/11987
        state:started ... sampleRate=44100
    ```

    uid 10340 is Spotify, so audio really was being produced. A run was started
    with the track deliberately **paused** first, so the pass proves the row tap
    started playback rather than finding it already playing.

### Diagnosis

This is the phone, not the recipe. Spotify's UI thread is starved: the phone is
at `MemFree ~300MB` with 3.2GB of swap in use, load average ~4, and Spotify
sustains ~70% CPU. When it is in that state it accepts no input at all — not
injected taps, not injected keys — while `keyevent 3` (HOME, handled by the
system) still works, and it recovers only after minutes of quiet. A force-stop
would clear it, and that is a restart on Sam's phone, so it is not something
Artemis does on its own.

Artemis made this worse than it needed to be: `uiautomator dump` starts a JVM on
the phone, and the wait helpers were polling every 400ms. `ui.PollInterval` is
now 1500ms, which cuts that load by roughly two thirds.

13. **A second query met the wedged app again.** Straight after the successful
    run, `spotify_play query="Beatles Hey Jude"` failed at the clear-the-search-box
    step: the app never acted on the tap, and a 90-second rest did not clear it.
    Not a recipe fault — the same wedge as attempts 9 and 10, and this time the
    evidence is complete:

    ```
    $ adb shell input motionevent DOWN 993 186
    $ adb shell dumpsys input | sed -n '/Input Dispatcher State/,/Input Reader State/p'
      TouchStatesByDisplay:
        LogicalDisplayId 0:
          TouchedWindows:
            0: name='44f7f5b com.spotify.music/com.spotify.music.MainActivity',
               targetFlags=FOREGROUND | SPLIT   ... FINGER

    $ adb shell dumpsys activity lastanr | tail -2
      <no ANR has occurred since boot>
    $ adb shell ps -A -o PID,NAME,%CPU | grep spotify
      11987 com.spotify.music  41.5
    $ adb shell grep -E 'MemFree|SwapFree' /proc/meminfo
      MemFree:  253508 kB
      SwapFree: 338264 kB
    ```

    The touch reaches Spotify's window; the app does not consume it; Android has
    not declared an ANR; the phone has 250MB free and has used 3.5GB of its 3.9GB
    swap. The first run of the heartbeat, from a quieter phone, completed in 35
    seconds. This is the phone's resource state, not the recipe, and detecting
    "the app has stopped taking input" belongs to Phase 7 (resilience).

### Reproducing it

```sh
cd ~/paseo/projects/artemis_control
./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
tail -1 logs/runs.jsonl
adb shell dumpsys media_session | grep -A9 'package=com.spotify.music'
```

The phone must be awake and unlocked. On a quiet phone this takes about 35
seconds; on the loaded phone above it ranged from 90 seconds to over four
minutes, which is what `total_timeout: 420s` is there to bound.

Phase 4 (song radio) and Phase 5 (maps) have not been started, on instruction.