feat: confirm playback from the media session, and stop demoting the top row
Phase 3 DoD reached end to end: `artemis run spotify_play query="Rolling Stones Honky Tonk"` played the right song in 13 steps / 35.3s, with playback paused first so the pass proves the row tap started it. Two things stopped it before that, both found on the real phone: - The playback check asserted on an on-screen "Pause" label that Spotify never shows: its transport button keeps the label "Play" while music is playing. The recipe now ends with a `playing` step that reads the app's media session (state PLAYING) and confirms the track title contains the query words. A failure reports the state and the track it found. - tap_row pressed the wrong row. Once a track is loaded its row carries a `Paused` state indicator, and counting that content description as row text dropped the top hit below an identical duplicate further down, so the tap went to y=1443 instead of y=878. Rows are now scored on their title and subtitle only (state lives in content descriptions), and scores within rowScoreTolerance are treated as equal so the higher row wins, because Spotify already ranks by relevance. internal/ui/rows_test.go reproduces that results screen, including the state indicator and the duplicate row. docs/phase-3-spotify-recipe.md records the successful run, the independent media-session and audio-output reads, and the failure that preceded it.
This commit is contained in:
@@ -192,10 +192,46 @@ keyguard; reading the state first would say "unlocked" about a locked phone.
|
||||
| Step durations reach the run log | unit tested (`internal/recipe`) |
|
||||
| All acting commands refuse on a locked phone | **live, verified** (above) |
|
||||
| `artemis run spotify_play` refuses on a locked phone | **live, verified** (above) |
|
||||
| `spotify_play` plays the right song | **not yet run end to end** — see below |
|
||||
| `spotify_play` plays the right song | **live, verified end to end** — 13 steps, 35.3s |
|
||||
| `spotify_play` picks the right row on the real results list | **live, verified** — `tapped row "Honky Tonk Women" at (540,878)` |
|
||||
| Spotify search reaches the result list | live (Home → Search → type → results) |
|
||||
|
||||
## Confirming playback
|
||||
|
||||
The screen is not a trustworthy place to read playback state. Spotify's
|
||||
transport button keeps the accessibility label **"Play"** while music is
|
||||
actually playing, so the first version of this recipe asserted on a label that
|
||||
never appears and failed on a phone that was playing the track perfectly.
|
||||
|
||||
The last step asks Android instead:
|
||||
|
||||
```
|
||||
- playing "{query}" 30s retry 3
|
||||
```
|
||||
|
||||
`playing` waits until the app's media session reports `PLAYING` **and** its
|
||||
metadata contains the query's words. It fails with the state and the track it
|
||||
found, for example:
|
||||
|
||||
```
|
||||
com.spotify.music is not playing "Rolling Stones Honky Tonk" within 30s
|
||||
(state=PAUSED, playing "Honky Tonk Women, The Rolling Stones, ...")
|
||||
```
|
||||
|
||||
## Why the top row wins
|
||||
|
||||
Spotify ranks results by relevance, so the highest acceptable row is the one a
|
||||
human would press. Two rules keep it that way:
|
||||
|
||||
- A row's own text is its title and subtitle. Text inside a clickable child is
|
||||
an action button, and **content descriptions are state**, not text: the row of
|
||||
the track that is already loaded carries `content-desc="Paused"`. Counting
|
||||
that made the top hit score *worse* than an identical duplicate further down,
|
||||
and `tap_row` pressed the duplicate at `y=1443` while the right row sat at
|
||||
`y=878`. `internal/ui/rows_test.go` reproduces that screen exactly.
|
||||
- Scores within `rowScoreTolerance` (0.15) are treated as equal, and then the
|
||||
higher row wins. A clearly better text match still wins outright.
|
||||
|
||||
## Retrying a dropped tap
|
||||
|
||||
A step can ask for the step above it to be pressed again:
|
||||
@@ -256,6 +292,44 @@ nothing random — which is the point of the phase.
|
||||
no dumps and no input, the screen had moved to Home — queued input had been
|
||||
processed — so a fresh run was started immediately. It failed at
|
||||
`tap "Search"`: the tap reported ok, the screen never left Home.
|
||||
11. **Wrong row chosen, quietly.** After Sam restarted Spotify the whole recipe
|
||||
ran fast and ended at `playing` with `state=PAUSED`. The run log showed why:
|
||||
`tapped row "Honky Tonk Women" at (540,1443)` — the fifth result, not the top
|
||||
hit at `(540,878)`. Once the track is loaded its top row carries a `Paused`
|
||||
state indicator, and counting that description as row text dropped the top
|
||||
hit's F1 below an identical duplicate further down. Fixed, and the top row
|
||||
now loses no points for carrying state.
|
||||
12. **Success.** With the ranking fixed, and with playback confirmed through the
|
||||
media session rather than the on-screen label (which stays `Play` even while
|
||||
music plays):
|
||||
|
||||
```sh
|
||||
$ ./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
|
||||
step 12 tap_row [Rolling Stones Honky Tonk]: ok (2.922s)
|
||||
step 13 playing [Rolling Stones Honky Tonk]: ok (1.714s)
|
||||
OK spotify_play in 35.311s
|
||||
```
|
||||
|
||||
```
|
||||
tapped row "Honky Tonk Women" at (540,878)
|
||||
from Honky Tonk Women / Song • The Rolling Stones
|
||||
com.spotify.music is playing
|
||||
"Honky Tonk Women, The Rolling Stones, Honky Tonk Women / You Can't Always Get What You Want"
|
||||
```
|
||||
|
||||
Independent reads, not Artemis's own claim:
|
||||
|
||||
```
|
||||
$ adb shell dumpsys media_session | grep -A9 'package=com.spotify.music'
|
||||
state=PlaybackState {state=PLAYING(3), position=546, ...}
|
||||
$ adb shell dumpsys audio | grep -i 'state:started'
|
||||
AudioPlaybackConfiguration piid:22575 ... type:AAudio u/pid:10340/11987
|
||||
state:started ... sampleRate=44100
|
||||
```
|
||||
|
||||
uid 10340 is Spotify, so audio really was being produced. A run was started
|
||||
with the track deliberately **paused** first, so the pass proves the row tap
|
||||
started playback rather than finding it already playing.
|
||||
|
||||
### Diagnosis
|
||||
|
||||
@@ -271,17 +345,17 @@ Artemis made this worse than it needed to be: `uiautomator dump` starts a JVM on
|
||||
the phone, and the wait helpers were polling every 400ms. `ui.PollInterval` is
|
||||
now 1500ms, which cuts that load by roughly two thirds.
|
||||
|
||||
### What is left
|
||||
|
||||
The live end-to-end run of `spotify_play` (Phase 3 DoD) has not completed. It
|
||||
needs the phone's Spotify restarted — swipe it away from Recents, or force-stop
|
||||
it — and the phone left awake and unlocked. Then:
|
||||
### Reproducing it
|
||||
|
||||
```sh
|
||||
cd ~/paseo/projects/artemis_control
|
||||
./bin/artemis run spotify_play query="Rolling Stones Honky Tonk"
|
||||
./bin/artemis fg
|
||||
./bin/artemis dump | grep -i pause
|
||||
tail -1 logs/runs.jsonl
|
||||
adb shell dumpsys media_session | grep -A9 'package=com.spotify.music'
|
||||
```
|
||||
|
||||
The phone must be awake and unlocked. On a quiet phone this takes about 35
|
||||
seconds; on the loaded phone above it ranged from 90 seconds to over four
|
||||
minutes, which is what `total_timeout: 420s` is there to bound.
|
||||
|
||||
Phase 4 (song radio) and Phase 5 (maps) have not been started, on instruction.
|
||||
|
||||
Reference in New Issue
Block a user