sam/artemis_control
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Phase 3 recipes file, spotify_play, and locked-screen refusal

Browse Source 
Phase 3 of AGENTS.md section 11.

Recipes file (FR-5)
- recipes/ holds one hand-editable file per task; recipes/spotify_play.recipe
  is the first one.
- internal/recipe parses directives, guards, parameters and steps, and runs
  them with a per-step and a whole-run timeout (FR-11).
- Steps name text ("tap the element whose text is Search"), never coordinates.
- guards name screens Artemis must never tap on, so a signed-out or
  first-run Spotify fails with its own reason (FR-8).
- Every run appends task, params, steps, result and duration to
  logs/runs.jsonl (FR-10).

spotify_play
- launch, search, play, then wait for the transport button to say "Pause".
- internal/ui/rows.go adds tap_row: a result row is a clickable container with
  a title and a subtitle, so a one-line search suggestion is not a result, and
  the tap lands on the row's centre rather than on its "More options" button.
  This fixes a real wrong tap: a plain text tap hit a 126x12 pixel
  "More options for song Honky Tonk Women - Live" button.

Locked screen (AGENTS.md section 13 Q2, decided)
- Artemis fails loudly on a secure keyguard and never types a PIN. No secret is
  stored. A non-secure keyguard is still dismissed.
- internal/ui/keyguard.go reads the KeyguardServiceDelegate block plus
  cmd lock_settings is-user-secure.
- Every acting command wakes the screen before it checks the keyguard: a dozing
  phone reports showing=false, because the lock screen is not drawn yet.
- AGENTS.md records the decision and that storing a PIN for auto-unlock is
  rejected for now.

Verified live: the phone locked itself mid-session, and keyguard, tap and
run spotify_play all refused with "phone is locked; unlock it and retry".
The live end-to-end playback run still needs an unlocked phone.
This commit is contained in:
Sam Rolfe
2026-10-16 13:24:22 +11:00
parent 7767651c11
commit 2604fea483
15 changed files with 2192 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

205
docs/phase-3-spotify-recipe.md Normal file
View File

@@ -0,0 +1,205 @@
# Phase 3 — the recipes file and `spotify_play`
Phase 3 of `AGENTS.md` section 11: the first named recipe, plus the resolved
answer to section 13 Q2 (the locked screen).
Date: 2026-10-16. Phone: `Pixel_8a (akita)` over Tailscale at
`100.101.49.17:46689` (the port moves on every reboot; nothing hardcodes it).
## What was built
| File | What it holds |
|---|---|
| `recipes/spotify_play.recipe` | The recipe, written to be edited by hand. |
| `internal/recipe/recipe.go` | The file format: directives, guards, steps, parameters. |
| `internal/recipe/engine.go` | Runs a recipe step by step with timeouts and a run log. |
| `internal/ui/rows.go` | Picks a list row by text and taps its centre. |
| `internal/ui/keyguard.go` | Reads the lock state and refuses to act when locked. |
| `cmd/artemis/run.go` | `artemis recipes`, `artemis run <task> k=v`, JSON run log. |
## The recipes file
One file per task in `recipes/`, plain text, no build step. It names the app,
the parameters the caller must supply, the screens Artemis must never tap on,
and the ordered steps.
```
task: spotify_play
app: com.spotify.music
param: query
row_id: com.spotify.music:id/row_root
guard "Log in" => Spotify is signed out; sign in on the phone and retry
steps:
- launch
- wait "Search" 30s
- tap "Search"
- tap_if "Clear search query"
- wait "What do you want to listen to?" 30s
- tap "What do you want to listen to?"
- type "{query}"
- wait "Clear search query" 30s
- wait_gone "Recents" 30s
- tap_row "{query}" 40s
- wait exact "Pause" 40s
```
Steps express intent. There is no coordinate anywhere in the file (FR-4).
Run it with:
```sh
./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
```
The verbs are `launch`, `wait`, `wait_gone`, `tap`, `tap_if`, `tap_row`, `type`,
`key`, `back`, `home` and `sleep`. `tap exact "Pause"` matches the whole text,
not a substring. `tap_if` taps only when the text is already on screen, which
lets one recipe cope with an app that remembers its last screen.
## Why `tap_row` exists
A plain text tap picked the wrong element on the real Spotify screen. On the
results list, `artemis tap "Honky Tonk Women"` tapped
```
ImageButton[context_menu_button] text="More options for song Honky Tonk Women - Live"
bounds=[828,2388][954,2400]
```
That is the smallest clickable match, and it sits at the bottom edge of the
screen. It was the wrong target: a "More options" button, not the song.
`tap_row` scores every result row against the query and taps the row itself:
* A candidate row is a clickable container that carries a title **and** a
subtitle. A one-line search suggestion (Spotify's "rolling stones honky tonk
women single") is therefore not a result row.
* Text inside a clickable child is skipped, because that is an action button
("More options", "Add to library", "Add suggestion"), not the row's own text.
* A row clipped by the bottom of the list is dropped. It is much shorter than a
row that is fully drawn, and its centre sits under the navigation area.
* Rows are ranked by F1 score over the query words, so an exact title beats a
noisy deluxe-edition title. Equal scores go to the highest row, because search
results are ranked by relevance.
On the real screen for `Rolling Stones Honky Tonk`, the top hit wins:
```
Honky Tonk Women / Song • The Rolling Stones bounds=[0,794][1080,962] F1=0.727
```
and the tap lands at the row's centre, `(540,878)` — on the title, clear of the
action buttons. `internal/ui/rows_test.go` runs this against a trimmed copy of
the real dump.
## Failing cleanly (FR-8)
Every failure names its cause, and they are distinguishable:
| Situation | Message |
|---|---|
| App missing | `app is not installed: com.spotify.music` |
| Signed out | `Spotify is signed out; sign in on the phone and retry (phone is showing "Log in")` |
| Update nag | `Spotify wants an update; update it on the phone and retry` |
| Element missing | `text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Log in", "Sign up free"` |
| Phone locked | `phone is locked; unlock it and retry` |
A guard is checked when a step cannot find its target, so a bad screen produces
its own reason instead of a generic "element not found".
## Timeouts (FR-11)
`step_timeout: 20s` is the default per step, `total_timeout: 150s` bounds the
whole run. A step may name its own timeout (`wait "Search" 30s`). The engine
checks the remaining budget before each step and stops when it runs out. Every
`adb` call is already bounded by `adb.DefaultTimeout`.
## Logging (FR-10)
Every run appends one JSON line to `logs/runs.jsonl` (override with
`ARTEMIS_LOG`): task, params, each step with its duration, the result and the
total duration. Two real entries:
```json
{"time":"2026-10-16T13:23:31+11:00","task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"},"ok":false,"error":"phone is locked; unlock it and retry","steps":null,"duration_ms":832}
```
The same steps print to the terminal as they run.
## The locked screen (section 13 Q2)
**Decision: Artemis fails loudly. No PIN is stored and none is ever typed.**
`ui.KeyguardState` reads two things: whether the lock screen is showing
(`dumpsys window`, the `KeyguardServiceDelegate` block) and whether the device
has a secure lock at all (`cmd lock_settings is-user-secure`). Acting commands
wake the screen first and then refuse if the keyguard is showing and secure. A
non-secure keyguard is still dismissed, as before. An unreadable lock state is a
failure, not a guess.
### Evidence
The prescribed test (put the phone to sleep, wake it, Artemis must refuse):
```sh
adb -s 100.101.49.17:46689 shell input keyevent 26 # POWER = sleep
adb -s 100.101.49.17:46689 shell input keyevent 224 # WAKEUP
```
```
$ adb shell dumpsys window policy | grep -E 'showing=|screenState'
showing=true
screenState=SCREEN_STATE_ON
$ adb shell cmd lock_settings is-user-secure
true
$ ./bin/artemis keyguard
keyguard showing=true secure=true
artemis: phone is locked; unlock it and retry
exit=1
$ ./bin/artemis tap "Search"
artemis: phone is locked; unlock it and retry
exit=1
$ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
task spotify_play (Search Spotify for a song and play it.)
FAILED spotify_play after 833ms: phone is locked; unlock it and retry
exit=1
```
No PIN was entered, nothing was read from the phone, and the lock screen was
still on screen afterwards — no blind tap happened. The screen dump after the
refusals still showed the lock screen clock and date.
A dozing phone reports `showing=false`, because the lock screen is not drawn
yet. That is why every acting command wakes the screen **before** it checks the
keyguard; reading the state first would say "unlocked" about a locked phone.
## Verification status
| Item | State |
|---|---|
| Recipes file format, guards, parameters, timeouts | unit tested (`internal/recipe`) |
| Row selection against a real Spotify dump | unit tested (`internal/ui`) |
| Keyguard parsing and secure-lock detection | unit tested (`internal/ui`) |
| All acting commands refuse on a locked phone | **live, verified** (above) |
| `artemis run spotify_play` refuses on a locked phone | **live, verified** (above) |
| `spotify_play` plays the right song | **not yet run end to end** — see below |
| Spotify search reaches the result list | live (Home → Search → type → results) |
The live end-to-end run of `spotify_play` (Phase 3 DoD) has not been completed.
The phone locked itself part-way through the session (5-minute screen timeout,
secure keyguard) and Artemis will not unlock it by design. It needs a human to
unlock the phone, and then:
```sh
./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
./bin/artemis fg
./bin/artemis dump | grep -i pause
tail -1 logs/runs.jsonl
```
Phase 4 (song radio) and Phase 5 (maps) have not been started, on instruction.