From 2604fea483a8bc5a21c7cbb4e4a7603a29e34c7d Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Fri, 16 Oct 2026 13:24:22 +1100 Subject: [PATCH] feat: Phase 3 recipes file, spotify_play, and locked-screen refusal Phase 3 of AGENTS.md section 11. Recipes file (FR-5) - recipes/ holds one hand-editable file per task; recipes/spotify_play.recipe is the first one. - internal/recipe parses directives, guards, parameters and steps, and runs them with a per-step and a whole-run timeout (FR-11). - Steps name text ("tap the element whose text is Search"), never coordinates. - guards name screens Artemis must never tap on, so a signed-out or first-run Spotify fails with its own reason (FR-8). - Every run appends task, params, steps, result and duration to logs/runs.jsonl (FR-10). spotify_play - launch, search, play, then wait for the transport button to say "Pause". - internal/ui/rows.go adds tap_row: a result row is a clickable container with a title and a subtitle, so a one-line search suggestion is not a result, and the tap lands on the row's centre rather than on its "More options" button. This fixes a real wrong tap: a plain text tap hit a 126x12 pixel "More options for song Honky Tonk Women - Live" button. Locked screen (AGENTS.md section 13 Q2, decided) - Artemis fails loudly on a secure keyguard and never types a PIN. No secret is stored. A non-secure keyguard is still dismissed. - internal/ui/keyguard.go reads the KeyguardServiceDelegate block plus cmd lock_settings is-user-secure. - Every acting command wakes the screen before it checks the keyguard: a dozing phone reports showing=false, because the lock screen is not drawn yet. - AGENTS.md records the decision and that storing a PIN for auto-unlock is rejected for now. Verified live: the phone locked itself mid-session, and keyguard, tap and run spotify_play all refused with "phone is locked; unlock it and retry". The live end-to-end playback run still needs an unlocked phone. --- .gitignore | 3 + AGENTS.md | 19 +- README.md | 39 ++- cmd/artemis/main.go | 96 +++++++- cmd/artemis/run.go | 122 +++++++++ docs/phase-3-spotify-recipe.md | 205 ++++++++++++++++ internal/recipe/engine.go | 376 ++++++++++++++++++++++++++++ internal/recipe/recipe.go | 437 +++++++++++++++++++++++++++++++++ internal/recipe/recipe_test.go | 194 +++++++++++++++ internal/ui/actions.go | 40 ++- internal/ui/keyguard.go | 114 +++++++++ internal/ui/keyguard_test.go | 81 ++++++ internal/ui/rows.go | 252 +++++++++++++++++++ internal/ui/rows_test.go | 174 +++++++++++++ recipes/spotify_play.recipe | 63 +++++ 15 files changed, 2192 insertions(+), 23 deletions(-) create mode 100644 cmd/artemis/run.go create mode 100644 docs/phase-3-spotify-recipe.md create mode 100644 internal/recipe/engine.go create mode 100644 internal/recipe/recipe.go create mode 100644 internal/recipe/recipe_test.go create mode 100644 internal/ui/keyguard.go create mode 100644 internal/ui/keyguard_test.go create mode 100644 internal/ui/rows.go create mode 100644 internal/ui/rows_test.go create mode 100644 recipes/spotify_play.recipe diff --git a/.gitignore b/.gitignore index 23c0a75..7dada89 100644 --- a/.gitignore +++ b/.gitignore @@ -34,3 +34,6 @@ local.properties # Pi agent generated memory (local state, not source) .pi/memory-vault/ + +# Artemis run log (local state, not source) +logs/ diff --git a/AGENTS.md b/AGENTS.md index 92d73ff..86ab744 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -176,12 +176,23 @@ Resolved: was USB-only and is removed in this NixOS release anyway. Go is likewise run from a one-off `nix shell nixpkgs#go`, not installed system-wide. Never run `nixos-rebuild` on `.13`: it runs the Hub, MQTT and every service. +6. **Locked screen — Artemis fails loudly. It never unlocks the phone.** It + detects a secure keyguard (PIN, pattern or password) before acting and + refuses with `phone is locked; unlock it and retry`. It does not dismiss a + secure keyguard and it does not type anything into one. A *non-secure* + keyguard (the phone has no PIN at all) is still dismissed, because that + screen guards nothing. **No PIN is stored, read or transmitted**, and there + is no secret for one. **Storing a PIN for auto-unlock is rejected for now** — + the phone is a pocket device, and Artemis is a convenience, not a reason to + keep Sam's unlock credential in a service on `.13`. This is revisit-able if + the daily friction turns out to be worse than the risk. Remaining: 1. Which tasks after Spotify and maps? (WhatsApp message? Other music apps?) -2. Locked screen: must Artemis unlock it, or does Sam leave it awake? -3. Action API shape: one generic `/act`, or one endpoint per task? -4. **Service packaging** for Artemis on `.13` — container or systemd — is still +2. Action API shape: one generic `/act`, or one endpoint per task? +3. **Service packaging** for Artemis on `.13` — container or systemd — is still open. ADB can be awkward in a container, so this is decided in Phase 6. The tooling-placement half of the old Q4 is resolved above. -5. Speed: what is an acceptable time for a 6-step recipe? +4. Speed: what is an acceptable time for a 6-step recipe? +5. The recipes file has grown `guard`, `tap_if` and `wait_gone` verbs. If it + keeps growing, decide whether the format stays bespoke or moves to YAML. diff --git a/README.md b/README.md index 6046e1c..38176d3 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ order. |---|---| | 1 — Prove the ADB link | ✅ proven — see [`docs/phase-1-adb-link.md`](docs/phase-1-adb-link.md) | | 2 — Act primitives | ✅ done — tap by text, type, scroll, back, home, launch | -| 3 — Spotify recipe | not started | +| 3 — Spotify recipe | 🟡 built — recipes file, guards, `spotify_play`; the live end-to-end run is still to do, see [`docs/phase-3-spotify-recipe.md`](docs/phase-3-spotify-recipe.md) | | 4 — Song radio | not started | | 5 — Maps recipe | not started | | 6 — Action API | not started | @@ -25,6 +25,9 @@ order. cmd/artemis/ command line entry point internal/adb/ timeout-bounded wrapper around the adb binary internal/ui/ read the screen (uiautomator dump) and act on it (input) +internal/recipe/ the recipes file format and its runner +recipes/ one hand-editable file per task +logs/ run log (JSON lines, not committed) docs/ phase evidence and notes ``` @@ -42,13 +45,15 @@ nix shell nixpkgs#go -c go test ./... ## Use The phone address defaults to `100.101.49.17`. Override with `ARTEMIS_PHONE`, -pin a serial with `ARTEMIS_ADB_SERIAL`, or point at a different adb with -`ARTEMIS_ADB`. +pin a serial with `ARTEMIS_ADB_SERIAL`, point at a different adb with +`ARTEMIS_ADB`, point at another recipes directory with `ARTEMIS_RECIPES`, and +move the run log with `ARTEMIS_LOG`. ```sh ./bin/artemis connect # scan for the wireless-debugging port, then connect ./bin/artemis devices ./bin/artemis dump # text nodes and bounds +./bin/artemis keyguard # lock-screen state; exits 1 when locked ./bin/artemis launch com.spotify.music ./bin/artemis tap "Search" # find by text, tap its centre (FR-4) ./bin/artemis type "Rolling Stones Honky Tonk" @@ -58,11 +63,31 @@ pin a serial with `ARTEMIS_ADB_SERIAL`, or point at a different adb with ./bin/artemis demo # Phase 2 DoD: launch an app, tap a named element ``` +## Recipes + +A recipe is a plain-text file in `recipes/` that Sam edits by hand. Steps name +what to find on screen; there are no coordinates anywhere. + +```sh +./bin/artemis recipes # list them +./bin/artemis run spotify_play query="Rolling Stones Honky Tonk" +``` + +Read [`docs/phase-3-spotify-recipe.md`](docs/phase-3-spotify-recipe.md) for the +file format, the verbs, and why list rows need their own step. + ## Rules that the code enforces - **Find by text, never fixed coordinates.** `tap` resolves a text match to - bounds and taps the centre. + bounds and taps the centre; `tap_row` scores result rows against the query + and taps the winning row's centre. - **Never tap blindly.** A missing or disabled element is a loud failure, not a - guessed coordinate. -- **Wake first.** Every acting command wakes the screen (FR-7). -- **Bound every command.** No adb call runs without a timeout (FR-11). + guessed coordinate. A named bad screen (`guard`) fails with its own reason. +- **Never touch a locked phone.** A secure keyguard stops Artemis with + `phone is locked; unlock it and retry`. No PIN is stored, read or typed + (AGENTS.md section 13 Q2). +- **Wake first.** Every acting command wakes the screen before it acts (FR-7), + and it checks the keyguard only after the screen is on. +- **Bound every command.** No adb call runs without a timeout, and a recipe has + a per-step and a whole-run timeout (FR-11). +- **Log every run.** Each run appends a JSON line to `logs/runs.jsonl` (FR-10). diff --git a/cmd/artemis/main.go b/cmd/artemis/main.go index 39b6cef..894e97a 100644 --- a/cmd/artemis/main.go +++ b/cmd/artemis/main.go @@ -1,7 +1,8 @@ // Command artemis drives an Android phone over ADB wireless. // -// This is the Phase 2 surface: act primitives on the command line. The Hub's -// HTTP action API arrives in Phase 6 and reuses these packages. +// This is the command line surface: connection, act primitives, and the named +// recipes in recipes/. The Hub's HTTP action API arrives in Phase 6 and reuses +// these packages. package main import ( @@ -22,6 +23,7 @@ connection connect [host] connect to the phone (scans ports if host given) devices list adb devices wake wake the screen + keyguard report the lock-screen state reading dump print visible text nodes with bounds @@ -37,6 +39,10 @@ acting back press back home press home +recipes + recipes list the recipes in the recipes directory + run [k=v ...] run a named recipe, e.g. run spotify_play query="..." + proof demo Phase 2 DoD: launch an app, tap a named element @@ -44,6 +50,8 @@ environment ARTEMIS_PHONE phone tailscale IP (default 100.101.49.17) ARTEMIS_ADB_SERIAL pin a specific adb serial ARTEMIS_ADB path to the adb binary (default: PATH) + ARTEMIS_RECIPES recipes directory (default: recipes) + ARTEMIS_LOG run log file (default: logs/runs.jsonl) ` func main() { @@ -64,6 +72,8 @@ func main() { err = cmdDevices(r) case "wake": err = cmdWake(r) + case "keyguard": + err = cmdKeyguard(r) case "dump": err = cmdDump(r) case "find": @@ -81,9 +91,23 @@ func main() { case "scroll": err = cmdScroll(r, args) case "back": - err = ensurePhone(r, func() error { return ui.Back(r) }) + err = ensurePhone(r, func() error { + if err := wakeAndGuard(r); err != nil { + return err + } + return ui.Back(r) + }) case "home": - err = ensurePhone(r, func() error { return ui.Home(r) }) + err = ensurePhone(r, func() error { + if err := wakeAndGuard(r); err != nil { + return err + } + return ui.Home(r) + }) + case "recipes": + err = cmdRecipes(args) + case "run": + err = cmdRun(r, args) case "demo": err = cmdDemo(r) case "-h", "--help", "help": @@ -119,6 +143,19 @@ func ensurePhone(r *adb.Runner, fn func() error) error { return fn() } +// wakeAndGuard wakes the screen and refuses to act while a secure keyguard is +// showing. Artemis has no PIN and never types one (AGENTS.md section 13 Q2). +// +// Waking first matters. A dozing phone reports `showing=false` because the lock +// screen is not drawn yet, so the keyguard state is only trustworthy once the +// screen is on. +func wakeAndGuard(r *adb.Runner) error { + if err := ui.Wake(r); err != nil { + return err + } + return ui.RequireUnlocked(r) +} + func cmdConnect(r *adb.Runner, args []string) error { if len(args) > 0 { msg, err := r.Connect(args[0]) @@ -156,6 +193,30 @@ func cmdWake(r *adb.Runner) error { return err } fmt.Println("screen awake") + showing, secure, err := ui.KeyguardState(r) + if err != nil { + return err + } + if showing && secure { + fmt.Println("phone is locked (secure keyguard); Artemis will not act until it is unlocked") + return nil + } + fmt.Println("phone is unlocked") + return nil + }) +} + +// cmdKeyguard reports the lock-screen state without touching it. +func cmdKeyguard(r *adb.Runner) error { + return ensurePhone(r, func() error { + showing, secure, err := ui.KeyguardState(r) + if err != nil { + return err + } + fmt.Printf("keyguard showing=%t secure=%t\n", showing, secure) + if showing && secure { + return ui.ErrLocked + } return nil }) } @@ -219,7 +280,7 @@ func cmdLaunch(r *adb.Runner, args []string) error { return fmt.Errorf("usage: artemis launch ") } return ensurePhone(r, func() error { - if err := ui.Wake(r); err != nil { + if err := wakeAndGuard(r); err != nil { return err } start := time.Now() @@ -236,7 +297,7 @@ func cmdTap(r *adb.Runner, args []string) error { return fmt.Errorf("usage: artemis tap ") } return ensurePhone(r, func() error { - if err := ui.Wake(r); err != nil { + if err := wakeAndGuard(r); err != nil { return err } node, err := ui.TapText(r, args[0], false) @@ -253,7 +314,12 @@ func cmdType(r *adb.Runner, args []string) error { if len(args) != 1 { return fmt.Errorf("usage: artemis type ") } - return ensurePhone(r, func() error { return ui.Type(r, args[0]) }) + return ensurePhone(r, func() error { + if err := wakeAndGuard(r); err != nil { + return err + } + return ui.Type(r, args[0]) + }) } func cmdSwipe(r *adb.Runner, args []string) error { @@ -276,21 +342,31 @@ func cmdSwipe(r *adb.Runner, args []string) error { } ms = v } - return ensurePhone(r, func() error { return ui.Swipe(r, nums[0], nums[1], nums[2], nums[3], ms) }) + return ensurePhone(r, func() error { + if err := wakeAndGuard(r); err != nil { + return err + } + return ui.Swipe(r, nums[0], nums[1], nums[2], nums[3], ms) + }) } func cmdScroll(r *adb.Runner, args []string) error { if len(args) != 1 { return fmt.Errorf("usage: artemis scroll ") } - return ensurePhone(r, func() error { return ui.Scroll(r, args[0]) }) + return ensurePhone(r, func() error { + if err := wakeAndGuard(r); err != nil { + return err + } + return ui.Scroll(r, args[0]) + }) } // cmdDemo is the Phase 2 Definition of Done, runnable on demand: // Artemis opens a named app and taps a named element on it. func cmdDemo(r *adb.Runner) error { return ensurePhone(r, func() error { - if err := ui.Wake(r); err != nil { + if err := wakeAndGuard(r); err != nil { return err } diff --git a/cmd/artemis/run.go b/cmd/artemis/run.go new file mode 100644 index 0000000..6c569a2 --- /dev/null +++ b/cmd/artemis/run.go @@ -0,0 +1,122 @@ +package main + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "time" + + "gitea.lab.audasmedia.com.au/sam/artemis_control/internal/adb" + "gitea.lab.audasmedia.com.au/sam/artemis_control/internal/recipe" +) + +// recipesDir is where hand-editable recipes live. +func recipesDir() string { + if d := os.Getenv("ARTEMIS_RECIPES"); d != "" { + return d + } + return "recipes" +} + +// runLogPath is the JSON-lines action log (FR-10). +func runLogPath() string { + if p := os.Getenv("ARTEMIS_LOG"); p != "" { + return p + } + return filepath.Join("logs", "runs.jsonl") +} + +func cmdRecipes(args []string) error { + recipes, err := recipe.List(recipesDir()) + if err != nil { + return err + } + for _, r := range recipes { + fmt.Printf("%-16s %s\n", r.Task, r.Description) + } + return nil +} + +// cmdRun runs a named recipe: artemis run spotify_play query="Rolling Stones". +func cmdRun(r *adb.Runner, args []string) error { + if len(args) < 1 { + return fmt.Errorf("usage: artemis run [name=value ...]") + } + task := args[0] + + params := map[string]string{} + for _, kv := range args[1:] { + name, value, ok := strings.Cut(kv, "=") + if !ok { + return fmt.Errorf("parameter %q must be name=value", kv) + } + params[strings.TrimSpace(name)] = value + } + + rec, err := recipe.Find(recipesDir(), task) + if err != nil { + return err + } + if missing := rec.MissingParams(params); len(missing) > 0 { + return fmt.Errorf("recipe %s needs %s", task, strings.Join(missing, ", ")) + } + + var res *recipe.RunResult + err = ensurePhone(r, func() error { + res = recipe.Run(r, rec, recipe.Options{Params: params, Log: os.Stdout}) + if logErr := appendRunLog(runLogPath(), res); logErr != nil { + fmt.Fprintf(os.Stderr, "artemis: could not write the run log: %v\n", logErr) + } + if !res.OK { + return fmt.Errorf("%s", res.Error) + } + return nil + }) + + fmt.Printf("\n%s\n", summaryLine(task, res)) + return err +} + +func summaryLine(task string, res *recipe.RunResult) string { + if res == nil { + return "task " + task + " did not run" + } + steps := len(res.Steps) + if res.OK { + return fmt.Sprintf("ok: %s ran %d steps in %s", task, steps, res.Duration.Round(time.Millisecond)) + } + return fmt.Sprintf("failed: %s stopped at step %d in %s: %s", + task, steps, res.Duration.Round(time.Millisecond), res.Error) +} + +// appendRunLog adds one JSON line per run: task, params, steps, result and +// duration (FR-10). +func appendRunLog(path string, res *recipe.RunResult) error { + if res == nil { + return nil + } + if dir := filepath.Dir(path); dir != "" && dir != "." { + if err := os.MkdirAll(dir, 0o755); err != nil { + return err + } + } + f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + return err + } + defer f.Close() + + entry := struct { + Time string `json:"time"` + *recipe.RunResult + }{Time: time.Now().Format(time.RFC3339), RunResult: res} + + raw, err := json.Marshal(entry) + if err != nil { + return err + } + _, err = f.Write(append(raw, '\n')) + return err +} diff --git a/docs/phase-3-spotify-recipe.md b/docs/phase-3-spotify-recipe.md new file mode 100644 index 0000000..68fc332 --- /dev/null +++ b/docs/phase-3-spotify-recipe.md @@ -0,0 +1,205 @@ +# Phase 3 — the recipes file and `spotify_play` + +Phase 3 of `AGENTS.md` section 11: the first named recipe, plus the resolved +answer to section 13 Q2 (the locked screen). + +Date: 2026-10-16. Phone: `Pixel_8a (akita)` over Tailscale at +`100.101.49.17:46689` (the port moves on every reboot; nothing hardcodes it). + +## What was built + +| File | What it holds | +|---|---| +| `recipes/spotify_play.recipe` | The recipe, written to be edited by hand. | +| `internal/recipe/recipe.go` | The file format: directives, guards, steps, parameters. | +| `internal/recipe/engine.go` | Runs a recipe step by step with timeouts and a run log. | +| `internal/ui/rows.go` | Picks a list row by text and taps its centre. | +| `internal/ui/keyguard.go` | Reads the lock state and refuses to act when locked. | +| `cmd/artemis/run.go` | `artemis recipes`, `artemis run k=v`, JSON run log. | + +## The recipes file + +One file per task in `recipes/`, plain text, no build step. It names the app, +the parameters the caller must supply, the screens Artemis must never tap on, +and the ordered steps. + +``` +task: spotify_play +app: com.spotify.music +param: query +row_id: com.spotify.music:id/row_root + +guard "Log in" => Spotify is signed out; sign in on the phone and retry + +steps: + - launch + - wait "Search" 30s + - tap "Search" + - tap_if "Clear search query" + - wait "What do you want to listen to?" 30s + - tap "What do you want to listen to?" + - type "{query}" + - wait "Clear search query" 30s + - wait_gone "Recents" 30s + - tap_row "{query}" 40s + - wait exact "Pause" 40s +``` + +Steps express intent. There is no coordinate anywhere in the file (FR-4). + +Run it with: + +```sh +./bin/artemis run spotify_play query="Rolling Stones Honky Tonk" +``` + +The verbs are `launch`, `wait`, `wait_gone`, `tap`, `tap_if`, `tap_row`, `type`, +`key`, `back`, `home` and `sleep`. `tap exact "Pause"` matches the whole text, +not a substring. `tap_if` taps only when the text is already on screen, which +lets one recipe cope with an app that remembers its last screen. + +## Why `tap_row` exists + +A plain text tap picked the wrong element on the real Spotify screen. On the +results list, `artemis tap "Honky Tonk Women"` tapped + +``` +ImageButton[context_menu_button] text="More options for song Honky Tonk Women - Live" + bounds=[828,2388][954,2400] +``` + +That is the smallest clickable match, and it sits at the bottom edge of the +screen. It was the wrong target: a "More options" button, not the song. + +`tap_row` scores every result row against the query and taps the row itself: + +* A candidate row is a clickable container that carries a title **and** a + subtitle. A one-line search suggestion (Spotify's "rolling stones honky tonk + women single") is therefore not a result row. +* Text inside a clickable child is skipped, because that is an action button + ("More options", "Add to library", "Add suggestion"), not the row's own text. +* A row clipped by the bottom of the list is dropped. It is much shorter than a + row that is fully drawn, and its centre sits under the navigation area. +* Rows are ranked by F1 score over the query words, so an exact title beats a + noisy deluxe-edition title. Equal scores go to the highest row, because search + results are ranked by relevance. + +On the real screen for `Rolling Stones Honky Tonk`, the top hit wins: + +``` +Honky Tonk Women / Song • The Rolling Stones bounds=[0,794][1080,962] F1=0.727 +``` + +and the tap lands at the row's centre, `(540,878)` — on the title, clear of the +action buttons. `internal/ui/rows_test.go` runs this against a trimmed copy of +the real dump. + +## Failing cleanly (FR-8) + +Every failure names its cause, and they are distinguishable: + +| Situation | Message | +|---|---| +| App missing | `app is not installed: com.spotify.music` | +| Signed out | `Spotify is signed out; sign in on the phone and retry (phone is showing "Log in")` | +| Update nag | `Spotify wants an update; update it on the phone and retry` | +| Element missing | `text "Search" did not appear within 30s; foreground=com.spotify.music; screen shows: "Log in", "Sign up free"` | +| Phone locked | `phone is locked; unlock it and retry` | + +A guard is checked when a step cannot find its target, so a bad screen produces +its own reason instead of a generic "element not found". + +## Timeouts (FR-11) + +`step_timeout: 20s` is the default per step, `total_timeout: 150s` bounds the +whole run. A step may name its own timeout (`wait "Search" 30s`). The engine +checks the remaining budget before each step and stops when it runs out. Every +`adb` call is already bounded by `adb.DefaultTimeout`. + +## Logging (FR-10) + +Every run appends one JSON line to `logs/runs.jsonl` (override with +`ARTEMIS_LOG`): task, params, each step with its duration, the result and the +total duration. Two real entries: + +```json +{"time":"2026-10-16T13:23:31+11:00","task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"},"ok":false,"error":"phone is locked; unlock it and retry","steps":null,"duration_ms":832} +``` + +The same steps print to the terminal as they run. + +## The locked screen (section 13 Q2) + +**Decision: Artemis fails loudly. No PIN is stored and none is ever typed.** + +`ui.KeyguardState` reads two things: whether the lock screen is showing +(`dumpsys window`, the `KeyguardServiceDelegate` block) and whether the device +has a secure lock at all (`cmd lock_settings is-user-secure`). Acting commands +wake the screen first and then refuse if the keyguard is showing and secure. A +non-secure keyguard is still dismissed, as before. An unreadable lock state is a +failure, not a guess. + +### Evidence + +The prescribed test (put the phone to sleep, wake it, Artemis must refuse): + +```sh +adb -s 100.101.49.17:46689 shell input keyevent 26 # POWER = sleep +adb -s 100.101.49.17:46689 shell input keyevent 224 # WAKEUP +``` + +``` +$ adb shell dumpsys window policy | grep -E 'showing=|screenState' + showing=true + screenState=SCREEN_STATE_ON +$ adb shell cmd lock_settings is-user-secure +true + +$ ./bin/artemis keyguard +keyguard showing=true secure=true +artemis: phone is locked; unlock it and retry +exit=1 + +$ ./bin/artemis tap "Search" +artemis: phone is locked; unlock it and retry +exit=1 + +$ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk" +task spotify_play (Search Spotify for a song and play it.) +FAILED spotify_play after 833ms: phone is locked; unlock it and retry +exit=1 +``` + +No PIN was entered, nothing was read from the phone, and the lock screen was +still on screen afterwards — no blind tap happened. The screen dump after the +refusals still showed the lock screen clock and date. + +A dozing phone reports `showing=false`, because the lock screen is not drawn +yet. That is why every acting command wakes the screen **before** it checks the +keyguard; reading the state first would say "unlocked" about a locked phone. + +## Verification status + +| Item | State | +|---|---| +| Recipes file format, guards, parameters, timeouts | unit tested (`internal/recipe`) | +| Row selection against a real Spotify dump | unit tested (`internal/ui`) | +| Keyguard parsing and secure-lock detection | unit tested (`internal/ui`) | +| All acting commands refuse on a locked phone | **live, verified** (above) | +| `artemis run spotify_play` refuses on a locked phone | **live, verified** (above) | +| `spotify_play` plays the right song | **not yet run end to end** — see below | +| Spotify search reaches the result list | live (Home → Search → type → results) | + +The live end-to-end run of `spotify_play` (Phase 3 DoD) has not been completed. +The phone locked itself part-way through the session (5-minute screen timeout, +secure keyguard) and Artemis will not unlock it by design. It needs a human to +unlock the phone, and then: + +```sh +./bin/artemis run spotify_play query="Rolling Stones Honky Tonk" +./bin/artemis fg +./bin/artemis dump | grep -i pause +tail -1 logs/runs.jsonl +``` + +Phase 4 (song radio) and Phase 5 (maps) have not been started, on instruction. diff --git a/internal/recipe/engine.go b/internal/recipe/engine.go new file mode 100644 index 0000000..0f75004 --- /dev/null +++ b/internal/recipe/engine.go @@ -0,0 +1,376 @@ +package recipe + +import ( + "fmt" + "io" + "strings" + "time" + + "gitea.lab.audasmedia.com.au/sam/artemis_control/internal/adb" + "gitea.lab.audasmedia.com.au/sam/artemis_control/internal/ui" +) + +// StepResult is the record of one executed step (FR-10). +type StepResult struct { + Index int `json:"step"` + Verb string `json:"verb"` + Args []string `json:"args,omitempty"` + Detail string `json:"detail,omitempty"` + OK bool `json:"ok"` + Error string `json:"error,omitempty"` + DurationMS int64 `json:"duration_ms"` + Duration time.Duration `json:"-"` +} + +// RunResult is the record of one whole recipe run (FR-10). +type RunResult struct { + Task string `json:"task"` + Params map[string]string `json:"params,omitempty"` + OK bool `json:"ok"` + Error string `json:"error,omitempty"` + Steps []StepResult `json:"steps"` + DurationMS int64 `json:"duration_ms"` + Duration time.Duration `json:"-"` +} + +// Options carry the caller's parameters and where to write the step log. +type Options struct { + Params map[string]string + Log io.Writer +} + +// Run executes a recipe against the phone. +// +// It refuses to start on a locked phone or a missing app, bounds every step +// (FR-11), and reports the reason for the first failing step. A run never +// returns an error silently: the returned RunResult always says what happened. +func Run(r *adb.Runner, rec *Recipe, opts Options) *RunResult { + started := time.Now() + params := opts.Params + if params == nil { + params = map[string]string{} + } + res := &RunResult{Task: rec.Task, Params: params} + defer func() { + res.Duration = time.Since(started) + res.DurationMS = res.Duration.Milliseconds() + }() + + logf := func(format string, args ...any) { + if opts.Log != nil { + fmt.Fprintf(opts.Log, format+"\n", args...) + } + } + + fail := func(err error) *RunResult { + res.OK = false + res.Error = err.Error() + logf("FAILED %s after %s: %s", rec.Task, time.Since(started).Round(time.Millisecond), res.Error) + return res + } + + logf("task %s (%s)", rec.Task, rec.Description) + + if missing := rec.MissingParams(params); len(missing) > 0 { + return fail(fmt.Errorf("missing required parameter(s): %s", strings.Join(missing, ", "))) + } + if rec.App != "" { + installed, err := ui.AppInstalled(r, rec.App) + if err != nil { + return fail(fmt.Errorf("could not check whether %s is installed: %w", rec.App, err)) + } + if !installed { + return fail(fmt.Errorf("app is not installed: %s", rec.App)) + } + } + + if err := ui.Wake(r); err != nil { + return fail(err) + } + // A secure keyguard is a hard stop: Artemis has no credential and must + // never type one (AGENTS.md section 13 Q2). + if err := ui.RequireUnlocked(r); err != nil { + return fail(err) + } + + w, h, _ := ui.ScreenSize(r) // best effort: bounds checking degrades gracefully + + deadline := started.Add(rec.TotalTimeout) + for i, step := range rec.Steps { + if remaining := time.Until(deadline); remaining <= 0 { + return fail(fmt.Errorf("recipe timed out after %s at step %d (%s)", rec.TotalTimeout, i+1, step.Verb)) + } + sr := executeStep(r, rec, step, i, params, w, h, deadline, logf) + res.Steps = append(res.Steps, sr) + if !sr.OK { + return fail(fmt.Errorf("step %d (%s %s) failed: %s", i+1, step.Verb, strings.Join(step.Args, " "), sr.Error)) + } + } + + res.OK = true + logf("OK %s in %s", rec.Task, time.Since(started).Round(time.Millisecond)) + return res +} + +func executeStep(r *adb.Runner, rec *Recipe, step Step, index int, params map[string]string, + w, h int, deadline time.Time, logf func(string, ...any)) StepResult { + + started := time.Now() + sr := StepResult{Index: index + 1, Verb: step.Verb, Args: step.Args} + defer func() { + sr.Duration = time.Since(started) + sr.DurationMS = sr.Duration.Milliseconds() + }() + + timeout := step.Timeout + if timeout <= 0 { + timeout = rec.StepTimeout + } + if remaining := time.Until(deadline); timeout > remaining { + timeout = remaining + } + if timeout <= 0 { + sr.Error = "no time left in the recipe budget" + return sr + } + + args := make([]string, len(step.Args)) + for i, a := range step.Args { + v, err := Substitute(a, params) + if err != nil { + sr.Error = err.Error() + return sr + } + args[i] = v + } + + stepErr := func() error { + switch step.Verb { + case "launch": + pkg := rec.App + if len(args) == 1 { + pkg = args[0] + } + if pkg == "" { + return fmt.Errorf("no app named: add an \"app:\" setting or pass a package") + } + if err := ui.Launch(r, pkg); err != nil { + return err + } + sr.Detail = "launched " + pkg + return nil + + case "wait": + node, err := ui.WaitForText(r, args[0], step.Exact, timeout) + if err != nil { + return describeFailure(r, rec, err) + } + sr.Detail = "found " + node.Label() + return nil + + case "wait_gone": + if err := ui.WaitForNoText(r, args[0], step.Exact, timeout); err != nil { + return describeFailure(r, rec, err) + } + sr.Detail = fmt.Sprintf("%q is gone", args[0]) + return nil + + case "tap": + return tapByText(r, rec, args[0], step.Exact, timeout, &sr) + + case "tap_if": + // Tap only if the text is already on screen. This makes a recipe + // tolerant of a screen that remembers state, without ever tapping + // something that was not matched by text. + h, err := ui.Dump(r) + if err != nil { + return err + } + matches := h.FindText(args[0], step.Exact) + if len(matches) == 0 { + sr.Detail = fmt.Sprintf("%q not on screen, skipped", args[0]) + return nil + } + node, err := ui.TapNode(r, matches) + if err != nil { + return err + } + cx, cy, _ := node.Center() + sr.Detail = fmt.Sprintf("tapped %q at (%d,%d)", args[0], cx, cy) + return nil + + case "tap_row": + return tapRow(r, rec, args[0], timeout, w, h, &sr) + + case "type": + if err := ui.Type(r, args[0]); err != nil { + return err + } + sr.Detail = fmt.Sprintf("typed %d characters", len(args[0])) + return nil + + case "key": + return sendKey(r, args[0], &sr) + + case "back": + if err := ui.Back(r); err != nil { + return err + } + sr.Detail = "pressed back" + return nil + + case "home": + if err := ui.Home(r); err != nil { + return err + } + sr.Detail = "pressed home" + return nil + + case "sleep": + time.Sleep(timeout) + sr.Detail = "slept " + timeout.String() + return nil + } + return fmt.Errorf("unknown step verb %q", step.Verb) + }() + + if stepErr != nil { + sr.OK = false + sr.Error = stepErr.Error() + logf(" step %d %s %v: %s", sr.Index, step.Verb, args, sr.Error) + return sr + } + + sr.OK = true + logf(" step %d %s %v: ok (%s)", sr.Index, step.Verb, args, sr.Duration.Round(time.Millisecond)) + return sr +} + +// tapByText finds an element by text and taps its centre (FR-4). +// +// The element is looked up twice: once to wait for it, and again immediately +// before the tap, so the coordinates come from the newest dump. On a busy +// phone the first dump can be seconds old by the time the tap is sent. +func tapByText(r *adb.Runner, rec *Recipe, query string, exact bool, timeout time.Duration, sr *StepResult) error { + if _, err := ui.WaitForText(r, query, exact, timeout); err != nil { + return describeFailure(r, rec, err) + } + h, err := ui.Dump(r) + if err != nil { + return err + } + matches := h.FindText(query, exact) + if len(matches) == 0 { + return describeFailure(r, rec, fmt.Errorf("element with text %q disappeared before the tap", query)) + } + node, err := ui.TapNode(r, matches) + if err != nil { + return err + } + cx, cy, _ := node.Center() + sr.Detail = fmt.Sprintf("tapped %q at (%d,%d) %s", query, cx, cy, node.Label()) + return nil +} + +// tapRow taps the list row that best matches the query. It retries the dump +// until a matching row is on screen, then taps a point derived from that row's +// own bounds. +func tapRow(r *adb.Runner, rec *Recipe, query string, timeout time.Duration, w, h int, sr *StepResult) error { + deadline := time.Now().Add(timeout) + var lastErr error + for time.Now().Before(deadline) { + hier, err := ui.Dump(r) + if err == nil { + row, rerr := hier.BestRow(query, rec.RowID, w, h) + if rerr == nil { + if err := ui.Tap(r, row.Centre.X, row.Centre.Y); err != nil { + return err + } + sr.Detail = fmt.Sprintf("tapped row %q at (%d,%d) from %s", row.Title, row.Centre.X, row.Centre.Y, + strings.Join(row.Texts, " / ")) + return nil + } + lastErr = rerr + } else { + lastErr = err + } + time.Sleep(500 * time.Millisecond) + } + if lastErr == nil { + lastErr = fmt.Errorf("no result row matches %q", query) + } + return describeFailure(r, rec, lastErr) +} + +func sendKey(r *adb.Runner, name string, sr *StepResult) error { + codes := map[string]int{ + "enter": ui.KeyEnter, + "back": ui.KeyBack, + "home": ui.KeyHome, + "power": 26, + "wakeup": ui.KeyWakeup, + "escape": 111, + } + code, ok := codes[strings.ToLower(name)] + if !ok { + return fmt.Errorf("unknown key %q", name) + } + if err := ui.KeyEvent(r, code); err != nil { + return err + } + sr.Detail = "pressed " + strings.ToLower(name) + return nil +} + +// describeFailure turns a step failure into a clean, named failure (FR-8). +// +// A guard match wins: "Spotify is signed out" is far more useful than "text +// Search did not appear". Otherwise the message names the screen that was on +// the phone when the step gave up. +func describeFailure(r *adb.Runner, rec *Recipe, err error) error { + if err == nil { + return nil + } + h, dumpErr := ui.Dump(r) + if dumpErr == nil { + for _, g := range rec.Guards { + if len(h.FindText(g.Text, false)) > 0 { + return fmt.Errorf("%s (phone is showing %q): %w", g.Reason, g.Text, err) + } + } + } + screen := describeScreen(r, h, dumpErr) + if screen == "" { + return err + } + return fmt.Errorf("%w; %s", err, screen) +} + +// describeScreen names the screen Artemis gave up on: the foreground package +// plus the first visible lines of text. +func describeScreen(r *adb.Runner, h *ui.Hierarchy, dumpErr error) string { + var parts []string + if fg, err := ui.ForegroundPackage(r); err == nil { + parts = append(parts, "foreground="+fg) + } + if dumpErr == nil && h != nil { + var labels []string + for _, n := range h.TextNodes() { + label := strings.TrimSpace(n.Text) + if label == "" { + label = strings.TrimSpace(n.ContentDesc) + } + if label == "" || len(label) > 60 { + continue + } + labels = append(labels, fmt.Sprintf("%q", label)) + if len(labels) == 6 { + break + } + } + if len(labels) > 0 { + parts = append(parts, "screen shows: "+strings.Join(labels, ", ")) + } + } + return strings.Join(parts, "; ") +} diff --git a/internal/recipe/recipe.go b/internal/recipe/recipe.go new file mode 100644 index 0000000..b7d0255 --- /dev/null +++ b/internal/recipe/recipe.go @@ -0,0 +1,437 @@ +// Package recipe loads and runs named, deterministic, hand-editable tasks. +// +// A recipe is one plain-text file in recipes/, written so that Sam can change +// it by hand without a build step. It names the app, the parameters it needs, +// the screens it must never tap on, and the ordered steps that do the work. +// Steps talk about text ("tap the element whose text is Search"), never about +// coordinates, so a recipe never taps blindly (FR-4, FR-5, FR-8). +// +// # File format +// +// A recipe file is a list of lines. Blank lines and lines starting with "#" +// are ignored. +// +// Directives are "key: value" lines placed before the steps: +// +// task: spotify_play unique name, must match the file name +// description: ... one line, for humans +// app: com.spotify.music the package the recipe drives +// param: query a parameter the caller must supply (repeatable) +// row_id: com.spotify:id/row_root container id for list rows (optional) +// step_timeout: 20s default per-step timeout (FR-11) +// total_timeout: 120s whole-recipe timeout (FR-11) +// +// A guard names a screen Artemis must never act on. Guards are checked when a +// step cannot find its target, so the failure names the real problem: +// +// guard "Log in" => Spotify is signed out; sign in on the phone and retry +// +// Steps are "- " lines, run in order: +// +// - launch start the recipe's app +// - wait "Search" 30s wait for an element whose text matches +// - wait_gone "Recents" 30s wait until no element matches the text +// - tap "Search" tap the element whose text matches (FR-4) +// - tap exact "Pause" tap an element whose text matches exactly +// - tap_if "Clear search query" tap it when it is on screen, else skip +// - tap_row "{query}" 30s tap the list row that best matches the text +// - type "{query}" type into the focused field +// - key enter send a named key event +// - back / home press a system key +// - sleep 1s wait a fixed time +// +// {name} placeholders are replaced with the value of parameter "name". The +// trailing "30s" on wait/tap/tap_row steps is that step's timeout. +package recipe + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "time" +) + +// Recipe is one parsed task. +type Recipe struct { + Task string + Description string + App string + Params []string + RowID string + StepTimeout time.Duration + TotalTimeout time.Duration + Guards []Guard + Steps []Step + Source string +} + +// Guard is a screen Artemis must never act on, with the reason to report. +type Guard struct { + Text string + Reason string +} + +// Step is one ordered action. +type Step struct { + Verb string + Args []string + Exact bool + Timeout time.Duration + Line int +} + +// Defaults used when a recipe does not set its own timeouts. +const ( + DefaultStepTimeout = 20 * time.Second + DefaultTotalTimeout = 120 * time.Second +) + +// validKeys are the directive names a recipe file may use. +var validKeys = map[string]bool{ + "task": true, + "description": true, + "app": true, + "param": true, + "row_id": true, + "step_timeout": true, + "total_timeout": true, +} + +// verbs lists the step verbs and how many arguments each takes. +var verbs = map[string]struct { + min, max int + durArg bool // the single argument is a duration + allowExact bool +}{ + "launch": {0, 1, false, false}, + "wait": {1, 2, false, true}, + "wait_gone": {1, 2, false, true}, + "tap": {1, 2, false, true}, + "tap_if": {1, 1, false, true}, + "tap_row": {1, 2, false, true}, + "type": {1, 1, false, false}, + "key": {1, 1, false, false}, + "back": {0, 0, false, false}, + "home": {0, 0, false, false}, + "sleep": {1, 1, true, false}, +} + +// keyNames are the key event names a recipe may use. +var keyNames = map[string]bool{ + "enter": true, + "back": true, + "home": true, + "power": true, + "wakeup": true, + "escape": true, +} + +// Load reads and parses one recipe file. +func Load(path string) (*Recipe, error) { + raw, err := os.ReadFile(path) + if err != nil { + return nil, err + } + rec, err := Parse(string(raw)) + if err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + rec.Source = path + if rec.Task == "" { + rec.Task = strings.TrimSuffix(filepath.Base(path), filepath.Ext(path)) + } + return rec, nil +} + +// List loads every recipe in dir, sorted by task name. +func List(dir string) ([]*Recipe, error) { + entries, err := os.ReadDir(dir) + if err != nil { + return nil, fmt.Errorf("read recipes directory %s: %w", dir, err) + } + var out []*Recipe + for _, e := range entries { + if e.IsDir() || strings.HasPrefix(e.Name(), ".") { + continue + } + rec, err := Load(filepath.Join(dir, e.Name())) + if err != nil { + return nil, err + } + out = append(out, rec) + } + sort.Slice(out, func(i, j int) bool { return out[i].Task < out[j].Task }) + return out, nil +} + +// Find loads the recipe for one task name from dir. +func Find(dir, task string) (*Recipe, error) { + recipes, err := List(dir) + if err != nil { + return nil, err + } + for _, r := range recipes { + if r.Task == task { + return r, nil + } + } + names := make([]string, 0, len(recipes)) + for _, r := range recipes { + names = append(names, r.Task) + } + if len(names) == 0 { + return nil, fmt.Errorf("no recipes in %s", dir) + } + return nil, fmt.Errorf("no recipe named %q in %s (have: %s)", task, dir, strings.Join(names, ", ")) +} + +// Parse turns the body of a recipe file into a Recipe. +func Parse(src string) (*Recipe, error) { + rec := &Recipe{ + StepTimeout: DefaultStepTimeout, + TotalTimeout: DefaultTotalTimeout, + } + + for i, raw := range strings.Split(src, "\n") { + lineNo := i + 1 + line := strings.TrimSpace(raw) + switch { + case line == "", strings.HasPrefix(line, "#"): + continue + case line == "steps:": + continue + case strings.HasPrefix(line, "-"): + step, err := parseStep(strings.TrimSpace(strings.TrimPrefix(line, "-")), lineNo) + if err != nil { + return nil, err + } + rec.Steps = append(rec.Steps, *step) + case strings.HasPrefix(line, "guard "): + guard, err := parseGuard(line, lineNo) + if err != nil { + return nil, err + } + rec.Guards = append(rec.Guards, *guard) + default: + if err := rec.parseDirective(line, lineNo); err != nil { + return nil, err + } + } + } + + if len(rec.Steps) == 0 { + return nil, fmt.Errorf("recipe has no steps") + } + return rec, nil +} + +func (rec *Recipe) parseDirective(line string, lineNo int) error { + key, value, ok := strings.Cut(line, ":") + if !ok { + return fmt.Errorf("line %d: expected \"key: value\", a \"guard ...\" line or a \"- step\" line, got %q", lineNo, line) + } + key = strings.ToLower(strings.TrimSpace(key)) + value = strings.TrimSpace(value) + if !validKeys[key] { + return fmt.Errorf("line %d: unknown setting %q", lineNo, key) + } + + switch key { + case "task": + rec.Task = value + case "description": + rec.Description = value + case "app": + rec.App = value + case "param": + if value == "" { + return fmt.Errorf("line %d: param needs a name", lineNo) + } + rec.Params = append(rec.Params, value) + case "row_id": + rec.RowID = value + case "step_timeout": + d, err := parseDuration(value) + if err != nil { + return fmt.Errorf("line %d: step_timeout: %w", lineNo, err) + } + rec.StepTimeout = d + case "total_timeout": + d, err := parseDuration(value) + if err != nil { + return fmt.Errorf("line %d: total_timeout: %w", lineNo, err) + } + rec.TotalTimeout = d + } + return nil +} + +func parseGuard(line string, lineNo int) (*Guard, error) { + body := strings.TrimSpace(strings.TrimPrefix(line, "guard")) + text, reason, ok := strings.Cut(body, "=>") + if !ok { + return nil, fmt.Errorf("line %d: a guard line needs \"guard \\\"text\\\" => reason\"", lineNo) + } + tokens, err := tokenize(text, lineNo) + if err != nil { + return nil, err + } + if len(tokens) != 1 { + return nil, fmt.Errorf("line %d: a guard needs exactly one quoted text, got %d", lineNo, len(tokens)) + } + reason = strings.TrimSpace(reason) + if reason == "" { + return nil, fmt.Errorf("line %d: a guard needs a reason after =>", lineNo) + } + return &Guard{Text: tokens[0], Reason: reason}, nil +} + +func parseStep(body string, lineNo int) (*Step, error) { + tokens, err := tokenize(body, lineNo) + if err != nil { + return nil, err + } + if len(tokens) == 0 { + return nil, fmt.Errorf("line %d: empty step", lineNo) + } + + verb := strings.ToLower(tokens[0]) + spec, ok := verbs[verb] + if !ok { + return nil, fmt.Errorf("line %d: unknown step verb %q", lineNo, tokens[0]) + } + step := &Step{Verb: verb, Line: lineNo} + args := tokens[1:] + + if spec.allowExact && len(args) > 0 && strings.EqualFold(args[0], "exact") { + step.Exact = true + args = args[1:] + } + + // A trailing duration is the step timeout, except for sleep where the + // duration is the step itself. + if !spec.durArg && len(args) > spec.min { + if d, derr := parseDuration(args[len(args)-1]); derr == nil { + step.Timeout = d + args = args[:len(args)-1] + } + } + + if len(args) < spec.min || len(args) > spec.max { + return nil, fmt.Errorf("line %d: %s takes %s, got %d", lineNo, verb, argRange(spec.min, spec.max), len(args)) + } + step.Args = args + + if spec.durArg { + d, derr := parseDuration(args[0]) + if derr != nil { + return nil, fmt.Errorf("line %d: sleep: %w", lineNo, derr) + } + step.Timeout = d + } + if verb == "key" && !keyNames[strings.ToLower(args[0])] { + return nil, fmt.Errorf("line %d: unknown key %q", lineNo, args[0]) + } + return step, nil +} + +func argRange(min, max int) string { + switch { + case min == max: + return fmt.Sprintf("exactly %d argument(s)", min) + case min == 0: + return fmt.Sprintf("up to %d argument(s)", max) + default: + return fmt.Sprintf("%d or %d arguments", min, max) + } +} + +func parseDuration(s string) (time.Duration, error) { + d, err := time.ParseDuration(strings.TrimSpace(s)) + if err != nil { + return 0, fmt.Errorf("%q is not a duration (use 20s, 1500ms, 2m)", s) + } + if d <= 0 { + return 0, fmt.Errorf("duration %q must be positive", s) + } + return d, nil +} + +// tokenize splits a recipe line into words, treating a double-quoted run as one +// word. Backslash escapes the next character inside quotes. +func tokenize(s string, lineNo int) ([]string, error) { + var out []string + for i := 0; i < len(s); { + switch { + case s[i] == ' ' || s[i] == '\t': + i++ + case s[i] == '"': + var b strings.Builder + j := i + 1 + for j < len(s) && s[j] != '"' { + if s[j] == '\\' && j+1 < len(s) { + j++ + } + b.WriteByte(s[j]) + j++ + } + if j >= len(s) { + return nil, fmt.Errorf("line %d: unterminated quoted string", lineNo) + } + out = append(out, b.String()) + i = j + 1 + default: + j := i + for j < len(s) && s[j] != ' ' && s[j] != '\t' { + j++ + } + out = append(out, s[i:j]) + i = j + } + } + return out, nil +} + +// Substitute replaces {name} placeholders with parameter values. +// +// Every declared parameter must be present, and every placeholder must be +// declared: an unset value would otherwise be typed into the phone as an empty +// string. +func Substitute(text string, params map[string]string) (string, error) { + var out strings.Builder + for i := 0; i < len(text); { + if text[i] != '{' { + out.WriteByte(text[i]) + i++ + continue + } + end := strings.IndexByte(text[i:], '}') + if end < 0 { + return "", fmt.Errorf("unclosed { in %q", text) + } + name := text[i+1 : i+end] + value, ok := params[name] + if !ok { + return "", fmt.Errorf("no value for parameter %q in %q", name, text) + } + if strings.TrimSpace(value) == "" { + return "", fmt.Errorf("parameter %q is empty", name) + } + out.WriteString(value) + i += end + 1 + } + return out.String(), nil +} + +// MissingParams returns the declared parameters that the caller did not supply. +func (rec *Recipe) MissingParams(params map[string]string) []string { + var missing []string + for _, name := range rec.Params { + if strings.TrimSpace(params[name]) == "" { + missing = append(missing, name) + } + } + return missing +} diff --git a/internal/recipe/recipe_test.go b/internal/recipe/recipe_test.go new file mode 100644 index 0000000..41e4b1d --- /dev/null +++ b/internal/recipe/recipe_test.go @@ -0,0 +1,194 @@ +package recipe + +import ( + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +const sample = `# a comment +task: spotify_play +description: Search Spotify for a song and play it. +app: com.spotify.music +param: query +row_id: com.spotify.music:id/row_root +step_timeout: 20s +total_timeout: 150s + +guard "Log in" => Spotify is signed out +guard "Update Spotify" => Spotify wants an update + +steps: + - launch + - wait "Search" 30s + - tap "Search" + - tap_if "Clear search query" + - wait_gone "Recents" 30s + - type "{query}" + - tap_row "{query}" 40s + - wait exact "Pause" 40s + - key enter + - sleep 1s +` + +func TestParseSampleRecipe(t *testing.T) { + rec, err := Parse(sample) + if err != nil { + t.Fatalf("Parse: %v", err) + } + if rec.Task != "spotify_play" || rec.App != "com.spotify.music" { + t.Fatalf("unexpected header: %+v", rec) + } + if len(rec.Params) != 1 || rec.Params[0] != "query" { + t.Fatalf("params = %v", rec.Params) + } + if rec.StepTimeout != 20*time.Second || rec.TotalTimeout != 150*time.Second { + t.Fatalf("timeouts = %s / %s", rec.StepTimeout, rec.TotalTimeout) + } + if len(rec.Guards) != 2 || rec.Guards[0].Text != "Log in" || rec.Guards[0].Reason != "Spotify is signed out" { + t.Fatalf("guards = %+v", rec.Guards) + } + if len(rec.Steps) != 10 { + t.Fatalf("steps = %d, want 10", len(rec.Steps)) + } + + wait := rec.Steps[1] + if wait.Verb != "wait" || wait.Args[0] != "Search" || wait.Timeout != 30*time.Second { + t.Fatalf("wait step = %+v", wait) + } + exact := rec.Steps[7] + if exact.Verb != "wait" || !exact.Exact || exact.Args[0] != "Pause" || exact.Timeout != 40*time.Second { + t.Fatalf("exact wait step = %+v", exact) + } + sleep := rec.Steps[9] + if sleep.Verb != "sleep" || sleep.Timeout != time.Second { + t.Fatalf("sleep step = %+v", sleep) + } +} + +func TestParseRejectsBadInput(t *testing.T) { + cases := map[string]string{ + "unknown verb": "steps:\n - teleport \"Home\"\n", + "missing settings": "steps:\n - wait \"x\"\n - frobnicate: 3\n", + "bad duration": "step_timeout: soon\nsteps:\n - wait \"x\"\n", + "no steps": "task: nothing\n", + "guard needs text": "guard only a reason => nope\nsteps:\n - home\n", + "guard needs why": "guard \"Log in\" =>\nsteps:\n - home\n", + "unknown key name": "steps:\n - key volup\n", + "unclosed quote": "steps:\n - wait \"Search\n", + "too many args": "steps:\n - type \"a\" \"b\"\n", + "unknown setting": "colour: blue\nsteps:\n - home\n", + } + for name, src := range cases { + if _, err := Parse(src); err == nil { + t.Errorf("%s: expected a parse error", name) + } + } +} + +func TestSubstitute(t *testing.T) { + got, err := Substitute("play {query} now", map[string]string{"query": "Honky Tonk Women"}) + if err != nil { + t.Fatalf("Substitute: %v", err) + } + if got != "play Honky Tonk Women now" { + t.Fatalf("Substitute = %q", got) + } + for _, bad := range []map[string]string{{}, {"query": " "}} { + if _, err := Substitute("{query}", bad); err == nil { + t.Fatalf("expected an error for params %v", bad) + } + } + if _, err := Substitute("{other}", map[string]string{"query": "x"}); err == nil { + t.Fatal("expected an error for an undeclared placeholder") + } +} + +func TestMissingParams(t *testing.T) { + rec, err := Parse(sample) + if err != nil { + t.Fatalf("Parse: %v", err) + } + if got := rec.MissingParams(map[string]string{"query": "x"}); len(got) != 0 { + t.Fatalf("MissingParams = %v", got) + } + if got := rec.MissingParams(nil); len(got) != 1 || got[0] != "query" { + t.Fatalf("MissingParams = %v", got) + } +} + +func TestParseRecipeWithoutTimeoutsUsesDefaults(t *testing.T) { + rec, err := Parse("steps:\n - home\n") + if err != nil { + t.Fatalf("Parse: %v", err) + } + if rec.StepTimeout != DefaultStepTimeout || rec.TotalTimeout != DefaultTotalTimeout { + t.Fatalf("defaults = %s / %s", rec.StepTimeout, rec.TotalTimeout) + } +} + +// TestSpotifyRecipeIsValid keeps the shipped recipe honest: a typo in +// recipes/spotify_play.recipe fails the test suite, not the phone. +func TestSpotifyRecipeIsValid(t *testing.T) { + dir := filepath.Join("..", "..", "recipes") + rec, err := Load(filepath.Join(dir, "spotify_play.recipe")) + if err != nil { + t.Fatalf("Load: %v", err) + } + if rec.Task != "spotify_play" { + t.Fatalf("task = %q", rec.Task) + } + if rec.App != "com.spotify.music" { + t.Fatalf("app = %q", rec.App) + } + if len(rec.MissingParams(map[string]string{"query": "x"})) != 0 { + t.Fatal("spotify_play must accept a query parameter") + } + if len(rec.Guards) < 3 { + t.Fatalf("spotify_play needs guards for its bad screens, got %d", len(rec.Guards)) + } + // Every step must be runnable: the parsed steps are what the engine gets. + for i, s := range rec.Steps { + if _, ok := verbs[s.Verb]; !ok { + t.Fatalf("step %d uses unknown verb %q", i+1, s.Verb) + } + } + // The recipe must wait for playback, not assume it. + last := rec.Steps[len(rec.Steps)-1] + if last.Verb != "wait" || last.Args[0] != "Pause" { + t.Fatalf("the last step must confirm playback, got %+v", last) + } +} + +func TestLoadAndList(t *testing.T) { + dir := t.TempDir() + write := func(name, body string) { + t.Helper() + if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + write("b.recipe", "task: beta\nsteps:\n - home\n") + write("a.recipe", "task: alpha\nsteps:\n - home\n") + + list, err := List(dir) + if err != nil { + t.Fatalf("List: %v", err) + } + if len(list) != 2 || list[0].Task != "alpha" || list[1].Task != "beta" { + t.Fatalf("List = %+v", list) + } + + rec, err := Find(dir, "beta") + if err != nil { + t.Fatalf("Find: %v", err) + } + if rec.Task != "beta" { + t.Fatalf("Find = %+v", rec) + } + if _, err := Find(dir, "gamma"); err == nil || !strings.Contains(err.Error(), "alpha") { + t.Fatalf("Find unknown task: %v", err) + } +} diff --git a/internal/ui/actions.go b/internal/ui/actions.go index 64ec237..4d68d0e 100644 --- a/internal/ui/actions.go +++ b/internal/ui/actions.go @@ -128,6 +128,17 @@ func TapText(r *adb.Runner, query string, exact bool) (*Node, error) { if len(matches) == 0 { return nil, fmt.Errorf("element with text %q not found on screen; refusing to tap blindly", query) } + return TapNode(r, matches) +} + +// TapNode taps the centre of the most tappable node among matches. +// +// Callers that already hold a hierarchy use this instead of TapText, so the +// coordinates come from the dump they just read rather than an older one. +func TapNode(r *adb.Runner, matches []*Node) (*Node, error) { + if len(matches) == 0 { + return nil, fmt.Errorf("no element was matched, refusing to tap blindly") + } node, err := pickBestMatch(matches) if err != nil { return nil, err @@ -170,6 +181,24 @@ func WaitForText(r *adb.Runner, query string, exact bool, timeout time.Duration) return nil, fmt.Errorf("text %q did not appear within %s", query, timeout) } +// WaitForNoText polls the screen until no element matches query. Recipes use it +// to wait for one list to be replaced by another, for example a search result +// list replacing the "Recents" list. +func WaitForNoText(r *adb.Runner, query string, exact bool, timeout time.Duration) error { + if timeout <= 0 { + timeout = 10 * time.Second + } + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + h, err := Dump(r) + if err == nil && len(h.FindText(query, exact)) == 0 { + return nil + } + time.Sleep(400 * time.Millisecond) + } + return fmt.Errorf("text %q was still on screen after %s", query, timeout) +} + // ScreenSize reads the physical screen size. func ScreenSize(r *adb.Runner) (width, height int, err error) { out, err := r.Shell("wm size") @@ -247,6 +276,10 @@ func Scroll(r *adb.Runner, direction string) error { } // Wake makes sure the screen is on before any action (FR-7). +// +// It dismisses the keyguard only when that keyguard is not secure. A secure +// keyguard (PIN, pattern or password) is left exactly as it is: Artemis never +// tries to get past it (AGENTS.md section 13 Q2). func Wake(r *adb.Runner) error { if _, err := r.Shell("input keyevent " + strconv.Itoa(KeyWakeup)); err != nil { return fmt.Errorf("wake keyevent: %w", err) @@ -255,8 +288,11 @@ func Wake(r *adb.Runner) error { for time.Now().Before(deadline) { out, err := r.Shell("dumpsys power | grep mWakefulness=") if err == nil && strings.Contains(out, "mWakefulness=Awake") { - // Dismiss a non-secure keyguard; harmless when the phone is unlocked. - _, _ = r.Shell("wm dismiss-keyguard") + // Only a non-secure keyguard may be dismissed. If the state cannot + // be read we leave the keyguard alone rather than guess. + if showing, secure, kerr := KeyguardState(r); kerr == nil && (!showing || !secure) { + _, _ = r.Shell("wm dismiss-keyguard") + } return nil } time.Sleep(300 * time.Millisecond) diff --git a/internal/ui/keyguard.go b/internal/ui/keyguard.go new file mode 100644 index 0000000..447c93a --- /dev/null +++ b/internal/ui/keyguard.go @@ -0,0 +1,114 @@ +package ui + +import ( + "errors" + "fmt" + "strings" + + "gitea.lab.audasmedia.com.au/sam/artemis_control/internal/adb" +) + +// ErrLocked is returned when a secure keyguard is showing. Artemis never tries +// to get past it: no PIN, pattern or password is stored, read or typed +// (AGENTS.md section 13 Q2). +var ErrLocked = errors.New("phone is locked; unlock it and retry") + +// KeyguardState reports whether the lock screen is showing and, separately, +// whether the device has a secure lock (PIN, pattern or password) at all. +// +// The two are independent: a phone with no PIN still shows a keyguard when it +// wakes, but that one may be dismissed. A secure one may not. +func KeyguardState(r *adb.Runner) (showing bool, secure bool, err error) { + dump, err := r.Shell("dumpsys window") + if err != nil { + return false, false, fmt.Errorf("read window state: %w", err) + } + showing, err = keyguardShowing(dump) + if err != nil { + return false, false, err + } + secure, err = userHasSecureLock(r) + if err != nil { + return false, false, err + } + return showing, secure, nil +} + +// keyguardShowing parses `dumpsys window` for the KeyguardServiceDelegate +// block. An unparsable dump is an error: guessing here would mean acting on a +// phone whose lock state is unknown. +func keyguardShowing(dump string) (bool, error) { + showing := false + sawDelegate := false + sawShowing := false + inDelegate := false + + for _, raw := range strings.Split(dump, "\n") { + line := strings.TrimSpace(raw) + switch { + case strings.HasPrefix(line, "KeyguardServiceDelegate"): + inDelegate, sawDelegate = true, true + continue + case strings.HasPrefix(line, "isKeyguardShowing="): + if strings.TrimPrefix(line, "isKeyguardShowing=") == "true" { + showing = true + } + continue + } + if !inDelegate { + continue + } + // The delegate block is a flat list of key=value lines. + if !strings.Contains(line, "=") { + inDelegate = false + continue + } + if strings.HasPrefix(line, "showing=") { + showing = strings.TrimPrefix(line, "showing=") == "true" + sawShowing = true + } + } + + if !sawDelegate || !sawShowing { + return false, fmt.Errorf("could not read the keyguard state from dumpsys window") + } + return showing, nil +} + +// userHasSecureLock asks the lock settings service whether the current user has +// a secure lock screen. It never asks for or handles the credential itself. +func userHasSecureLock(r *adb.Runner) (bool, error) { + out, err := r.Shell("cmd lock_settings is-user-secure") + if err != nil { + return false, fmt.Errorf("read lock state: %w", err) + } + switch strings.TrimSpace(out) { + case "true": + return true, nil + case "false": + return false, nil + } + return false, fmt.Errorf("unexpected lock state %q from cmd lock_settings is-user-secure", strings.TrimSpace(out)) +} + +// RequireUnlocked refuses to act while a secure keyguard is showing. +// +// A non-secure keyguard (the phone has no PIN at all) is dismissed, because +// that screen guards nothing. A secure one is left alone: Artemis stops and +// says so. +func RequireUnlocked(r *adb.Runner) error { + showing, secure, err := KeyguardState(r) + if err != nil { + return fmt.Errorf("cannot confirm the phone is unlocked: %w", err) + } + if !showing { + return nil + } + if secure { + return ErrLocked + } + if _, err := r.Shell("wm dismiss-keyguard"); err != nil { + return fmt.Errorf("dismiss the non-secure keyguard: %w", err) + } + return nil +} diff --git a/internal/ui/keyguard_test.go b/internal/ui/keyguard_test.go new file mode 100644 index 0000000..98a9450 --- /dev/null +++ b/internal/ui/keyguard_test.go @@ -0,0 +1,81 @@ +package ui + +import "testing" + +// A real (trimmed) KeyguardServiceDelegate block from `dumpsys window`. +const keyguardLockedDump = `WINDOW MANAGER POLICY STATE (dumpsys window policy) + mSafeMode=false mSystemReady=true + mKeyguardOccluded=false mPendingKeyguardOccluded=false + KeyguardServiceDelegate + showing=true + inputRestricted=true + occluded=false + trusted=false + simSecure=false + dreaming=false + systemReady=true + deviceHasKeyguard=true + enabled=true + userId=0 + bootCompleted=true + screenState=SCREEN_STATE_ON + interactiveState=INTERACTIVE_STATE_AWAKE + Looper state: + Looper (android.ui, tid 22) {4c5c23e} +` + +const keyguardUnlockedDump = `WINDOW MANAGER POLICY STATE (dumpsys window policy) + mKeyguardOccluded=false mPendingKeyguardOccluded=false + KeyguardServiceDelegate + showing=false + inputRestricted=false + occluded=false + trusted=false + simSecure=false + dreaming=false + systemReady=true + deviceHasKeyguard=true + enabled=true + userId=0 + bootCompleted=true + screenState=SCREEN_STATE_ON + interactiveState=INTERACTIVE_STATE_AWAKE + Looper state: + Looper (android.ui, tid 22) {4c5c23e} +` + +func TestKeyguardShowingParsesTheDelegateBlock(t *testing.T) { + showing, err := keyguardShowing(keyguardLockedDump) + if err != nil { + t.Fatalf("keyguardShowing: %v", err) + } + if !showing { + t.Fatal("want showing=true") + } + + showing, err = keyguardShowing(keyguardUnlockedDump) + if err != nil { + t.Fatalf("keyguardShowing: %v", err) + } + if showing { + t.Fatal("want showing=false") + } +} + +func TestKeyguardShowingRejectsAnUnparsableDump(t *testing.T) { + // Acting on a phone whose lock state is unknown is not allowed. + if _, err := keyguardShowing("dumpsys window: service not found\n"); err == nil { + t.Fatal("expected an error when the delegate block is missing") + } +} + +func TestKeyguardShowingReadsIsKeyguardShowing(t *testing.T) { + dump := " KeyguardServiceDelegate\n showing=false\n mDisableSecureWindows=false\n isKeyguardShowing=true\n" + showing, err := keyguardShowing(dump) + if err != nil { + t.Fatalf("keyguardShowing: %v", err) + } + if !showing { + t.Fatal("want showing=true from isKeyguardShowing") + } +} diff --git a/internal/ui/rows.go b/internal/ui/rows.go new file mode 100644 index 0000000..915e7a3 --- /dev/null +++ b/internal/ui/rows.go @@ -0,0 +1,252 @@ +package ui + +import ( + "fmt" + "math" + "strings" + "unicode" +) + +// Rows are the only safe way to press a list result. +// +// A search result row is a clickable container that holds a title and a +// subtitle ("Honky Tonk Women" / "Song • The Rolling Stones"). The container is +// the text-matched element, and its centre is the tap point: that lands on the +// title, never on the row's "More options" or "Add to library" buttons, which +// sit in a small action bar at the row's right edge. +// +// Selecting the row instead of the smallest clickable child matters. On the +// real Spotify screen a plain text match resolved to a 126x12 pixel "More +// options for song ..." button, because that was the smallest clickable node. + +// Row is one pressable list row. +type Row struct { + Node *Node + Texts []string + Score float64 + Title string + Top int + Centre Point +} + +// Point is a screen coordinate. +type Point struct { + X, Y int +} + +// BestRow chooses the row that best matches query. +// +// rowID, when set, restricts candidates to elements with that resource id +// (either the full "package:id/name" form or a suffix of it). +func (h *Hierarchy) BestRow(query, rowID string, screenW, screenH int) (*Row, error) { + query = strings.TrimSpace(query) + if query == "" { + return nil, fmt.Errorf("row match needs a non-empty query") + } + qTokens := tokenize(query) + if len(qTokens) == 0 { + return nil, fmt.Errorf("query %q has no matchable words", query) + } + + candidates := h.rowCandidates(rowID, screenW, screenH) + if len(candidates) == 0 { + return nil, fmt.Errorf("no result row is fully drawn on screen; refusing to tap blindly") + } + + var best *Row + for _, node := range candidates { + rect, err := node.BoundsRect() + if err != nil { + continue + } + texts := rowTexts(node) + score, matched := rowScore(qTokens, texts) + // More than half of the query words must be on the row, and the row + // must not be buried under unrelated words. + if matched*2 < len(qTokens) || score < minRowScore { + continue + } + cx, cy, err := node.Center() + if err != nil { + continue + } + candidate := &Row{ + Node: node, + Texts: texts, + Score: score, + Title: texts[0], + Top: rect.Y1, + Centre: Point{X: cx, Y: cy}, + } + if best == nil || betterRow(candidate, best) { + best = candidate + } + } + + if best == nil { + return nil, fmt.Errorf("no result row matches %q; refusing to tap blindly", query) + } + return best, nil +} + +// minRowScore is the F1 floor for an acceptable row match. +const minRowScore = 0.5 + +// rowCandidates returns the fully drawn rows on screen: clickable containers +// that carry at least a title and a subtitle. +// +// Two shapes are rejected on purpose. A one-line search *suggestion* is not a +// result. And a row clipped by the bottom of the list is much shorter than a +// row that is fully scrolled into view, so its centre is not a point a human +// could press; rows shorter than half the tallest candidate are dropped. +func (h *Hierarchy) rowCandidates(rowID string, screenW, screenH int) []*Node { + var found []*Node + tallest := 0 + h.Walk(func(n *Node) { + if !n.Clickable || !n.Enabled || !rowMatchesID(n, rowID) { + return + } + rect, err := n.BoundsRect() + if err != nil || !onScreen(rect, screenW, screenH) { + return + } + if len(rowTexts(n)) < 2 { + return + } + if h := rect.Y2 - rect.Y1; h > tallest { + tallest = h + } + found = append(found, n) + }) + + var out []*Node + for _, n := range found { + rect, err := n.BoundsRect() + if err != nil { + continue + } + if (rect.Y2-rect.Y1)*2 < tallest { + continue + } + out = append(out, n) + } + return out +} + +// betterRow ranks rows: the best text match wins, and equal matches go to the +// row nearest the top, because search results are ranked by relevance. +func betterRow(a, b *Row) bool { + if math.Abs(a.Score-b.Score) > 1e-9 { + return a.Score > b.Score + } + return a.Top < b.Top +} + +func rowMatchesID(n *Node, rowID string) bool { + if rowID == "" { + return true + } + return n.ResourceID == rowID || strings.HasSuffix(n.ResourceID, rowID) +} + +// textDescendants returns the visible text lines inside a node, in reading +// order, without duplicates. It is used for dumps and diagnostics. +func textDescendants(n *Node) []string { + var out []string + seen := map[string]bool{} + n.Walk(func(c *Node) { + for _, t := range []string{c.Text, c.ContentDesc} { + t = strings.TrimSpace(t) + if t == "" || seen[t] { + continue + } + seen[t] = true + out = append(out, t) + } + }) + return out +} + +// rowTexts returns the text that belongs to the row itself: its title and +// subtitle. +// +// Anything inside a clickable child is an action button ("More options", "Add +// to library", "Add suggestion"), not the row's own text, so that subtree is +// skipped. This is what keeps a one-line search suggestion from looking like a +// two-line result. +func rowTexts(n *Node) []string { + var out []string + seen := map[string]bool{} + add := func(s string) { + s = strings.TrimSpace(s) + if s == "" || seen[s] { + return + } + seen[s] = true + out = append(out, s) + } + + var visit func(c *Node, root bool) + visit = func(c *Node, root bool) { + if !root && c.Clickable { + return + } + add(c.Text) + if !c.Clickable { + add(c.ContentDesc) + } + for i := range c.Children { + visit(&c.Children[i], false) + } + } + visit(n, true) + return out +} + +// onScreen reports whether the whole rectangle is inside the display. A row +// clipped by the screen edge would be tapped at a point the user cannot see, +// so it is not a candidate. +func onScreen(r Rect, w, h int) bool { + if w <= 0 || h <= 0 { + return true + } + return r.X1 >= 0 && r.Y1 >= 0 && r.X2 <= w && r.Y2 <= h +} + +// rowScore returns the F1 score of the query words against the row's words, +// plus how many distinct query words were found. +func rowScore(query []string, texts []string) (score float64, matched int) { + row := map[string]bool{} + for _, t := range texts { + for _, w := range tokenize(t) { + row[w] = true + } + } + hit := map[string]bool{} + for _, w := range query { + if row[w] { + hit[w] = true + } + } + if len(hit) == 0 || len(row) == 0 { + return 0, 0 + } + precision := float64(len(hit)) / float64(len(row)) + recall := float64(len(hit)) / float64(len(query)) + if precision+recall == 0 { + return 0, len(hit) + } + return 2 * precision * recall / (precision + recall), len(hit) +} + +// tokenize splits text into lowercase words. +func tokenize(s string) []string { + fields := strings.FieldsFunc(s, func(r rune) bool { + return !unicode.IsLetter(r) && !unicode.IsDigit(r) + }) + out := make([]string, 0, len(fields)) + for _, f := range fields { + out = append(out, strings.ToLower(f)) + } + return out +} diff --git a/internal/ui/rows_test.go b/internal/ui/rows_test.go new file mode 100644 index 0000000..745b9f5 --- /dev/null +++ b/internal/ui/rows_test.go @@ -0,0 +1,174 @@ +package ui + +import ( + "strings" + "testing" +) + +// spotifyRowDump is the shape of a real Spotify search-results screen, trimmed +// to the rows that matter. The first row is a search *suggestion* (one line of +// text); the rest are result rows (title + subtitle). +const spotifyRowDump = ` + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +` + +const spotifyRowID = "com.spotify.music:id/row_root" + +func spotifyRows(t *testing.T) *Hierarchy { + t.Helper() + h, err := parseHierarchy([]byte(spotifyRowDump)) + if err != nil { + t.Fatalf("parseHierarchy: %v", err) + } + return h +} + +func TestBestRowPicksTheTopResultOfARealScreen(t *testing.T) { + h := spotifyRows(t) + row, err := h.BestRow("Rolling Stones Honky Tonk", spotifyRowID, 1080, 2400) + if err != nil { + t.Fatalf("BestRow: %v", err) + } + if row.Title != "Honky Tonk Women" { + t.Fatalf("chose row %q, want the canonical top hit; rows: %v", row.Title, row.Texts) + } + // The tap must land on the row, on the title, not on a row button. + if row.Centre.X < 200 || row.Centre.X > 828 { + t.Fatalf("tap x=%d is outside the title", row.Centre.X) + } + if row.Centre.Y < 794 || row.Centre.Y > 962 { + t.Fatalf("tap y=%d is outside the row", row.Centre.Y) + } +} + +func TestBestRowIgnoresSearchSuggestions(t *testing.T) { + h := spotifyRows(t) + // The suggestion row carries every query word, but it is a single line of + // text, so it is not a result row. + query := "rolling stones honky tonk women single" + row, err := h.BestRow(query, spotifyRowID, 1080, 2400) + if err != nil { + t.Fatalf("BestRow: %v", err) + } + if row.Title == "rolling stones honky tonk women single" { + t.Fatal("chose the search suggestion; a suggestion has no subtitle and is not a result") + } +} + +func TestBestRowIgnoresRowsClippedByTheScreenEdge(t *testing.T) { + // The last row is cut off at the bottom of the list: only 33 pixels tall, + // with its centre underneath the navigation area. It matches the query + // better than any other row, and it must still lose, because tapping it + // would press something the user cannot see. + h := spotifyRows(t) + row, err := h.BestRow("Honky Tonk Women Live", spotifyRowID, 1080, 2400) + if err != nil { + t.Fatalf("BestRow: %v", err) + } + if row.Title == "Honky Tonk Women - Live" { + t.Fatal("chose a row clipped by the bottom of the list") + } +} + +func TestBestRowRefusesAnUnknownQuery(t *testing.T) { + h := spotifyRows(t) + if _, err := h.BestRow("Chopin Nocturne", spotifyRowID, 1080, 2400); err == nil { + t.Fatal("expected an error when no row matches the query") + } +} + +func TestBestRowWorksWithoutARowID(t *testing.T) { + h := spotifyRows(t) + row, err := h.BestRow("The Rolling Stones Honky Tonk Women", "", 1080, 2400) + if err != nil { + t.Fatalf("BestRow without a row id: %v", err) + } + if row.Title != "Honky Tonk Women" { + t.Fatalf("chose %q", row.Title) + } +} + +func TestRowScorePrefersPrecision(t *testing.T) { + exact, _ := rowScore(tokenize("honky tonk women"), []string{"Honky Tonk Women", "Song • The Rolling Stones"}) + noisy, _ := rowScore(tokenize("honky tonk women"), []string{"Honky Tonk Women Live Licks Tour 2009 Remastered Deluxe Edition", "Song • The Rolling Stones, Sheryl Crow"}) + if exact <= noisy { + t.Fatalf("an exact title (%f) must beat a noisy one (%f)", exact, noisy) + } +} + +func TestRowTextsSkipActionButtons(t *testing.T) { + h := spotifyRows(t) + + // The suggestion row: one line of text plus an "Add suggestion" button. + var suggestion *Node + h.Walk(func(n *Node) { + if n.ResourceID == spotifyRowID && n.Bounds == "[0,251][1080,425]" { + suggestion = n + } + }) + if suggestion == nil { + t.Fatal("suggestion row not found in the sample dump") + } + texts := rowTexts(suggestion) + if len(texts) != 1 { + t.Fatalf("a suggestion row's own text = %v, want 1 line", texts) + } + + // A result row: title + subtitle, without the row's action buttons. + var result *Node + h.Walk(func(n *Node) { + if n.ResourceID == spotifyRowID && n.Bounds == "[0,794][1080,962]" { + result = n + } + }) + texts = rowTexts(result) + if len(texts) != 2 || texts[0] != "Honky Tonk Women" || texts[1] != "Song • The Rolling Stones" { + t.Fatalf("a result row's own text = %v", texts) + } + for _, bad := range texts { + if strings.Contains(bad, "More options") || strings.Contains(bad, "Add ") { + t.Fatalf("row text %q came from an action button", bad) + } + } +} + +func TestTokenize(t *testing.T) { + got := tokenize("Honky Tonk Women - Mono Version (Remastered)") + want := []string{"honky", "tonk", "women", "mono", "version", "remastered"} + if len(got) != len(want) { + t.Fatalf("tokenize = %v, want %v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("tokenize = %v, want %v", got, want) + } + } +} diff --git a/recipes/spotify_play.recipe b/recipes/spotify_play.recipe new file mode 100644 index 0000000..53eedf1 --- /dev/null +++ b/recipes/spotify_play.recipe @@ -0,0 +1,63 @@ +# spotify_play — search Spotify for a song and play it. +# +# Edit this file by hand. Steps name the text Artemis must find on screen; it +# taps the centre of the element it matched. There are no coordinates here on +# purpose, so this recipe survives a Spotify redesign (FR-4). +# +# Run it with: +# artemis run spotify_play query="Rolling Stones Honky Tonk" + +task: spotify_play +description: Search Spotify for a song and play it. +app: com.spotify.music + +# The caller must supply these. +param: query + +# Spotify's list rows: title + subtitle inside one clickable container. Naming +# the container keeps the tap on the row itself and off the row's buttons. +row_id: com.spotify.music:id/row_root + +step_timeout: 20s +total_timeout: 150s + +# Screens Artemis must never tap on. Each one fails the run with the reason, so +# a wrong tap is impossible (FR-8). +guard "Log in" => Spotify is signed out; sign in on the phone and retry +guard "Sign up free" => Spotify is signed out; sign in on the phone and retry +guard "Log in to Spotify" => Spotify is signed out; sign in on the phone and retry +guard "Continue with Facebook" => Spotify is showing its welcome screen; sign in on the phone and retry +guard "Continue with Google" => Spotify is showing its welcome screen; sign in on the phone and retry +guard "Update Spotify" => Spotify wants an update; update it on the phone and retry +guard "What's new" => Spotify is showing an announcement screen; dismiss it on the phone and retry + +steps: + # 1. Make sure Spotify is up. Launching an app that is already open is a + # no-op, so this is safe to repeat. + - launch + - wait "Search" 30s + + # 2. Open the search tab. Spotify remembers the last tab, so a search screen + # with the previous query can come back: clear that box only when the + # clear button is actually on screen. + - tap "Search" + - tap_if "Clear search query" + - wait "What do you want to listen to?" 30s + - tap "What do you want to listen to?" + + # 3. Type the query. The clear button proves the text landed. + - type "{query}" + - wait "Clear search query" 30s + + # 4. Wait for the results to replace the "Recents" list, so the next step can + # never match something from the history instead of a search result. + - wait_gone "Recents" 30s + + # 5. Play the result that matches the query. tap_row scores every visible + # result row against the query words and taps the best one; equal matches + # go to the highest row, which is how Spotify ranks them. + - tap_row "{query}" 40s + + # 6. Playback started when the transport button says "Pause". Exact match, so + # Spotify's "Paused" labels do not count. + - wait exact "Pause" 40s