118 lines
4.6 KiB
Markdown
118 lines
4.6 KiB
Markdown
# Phase 2 — Act primitives
|
|
|
|
Status: **done**. Verified live against the Pixel 8a on 2026-10-16.
|
|
|
|
Definition of Done: *Artemis opens a named app and taps a named element on it.*
|
|
|
|
---
|
|
|
|
## 1. Primitives
|
|
|
|
All primitives live in `internal/ui`. Every acting command wakes the screen
|
|
first (FR-7) and every adb call is bounded by a timeout (FR-11).
|
|
|
|
| Primitive | Command | How it works |
|
|
|---|---|---|
|
|
| Tap by text | `artemis tap <text>` | dump → match text/content-desc → tap element centre (FR-4) |
|
|
| Type | `artemis type <text>` | `input text`, with `%s` space encoding and shell-safe quoting |
|
|
| Swipe | `artemis swipe x1 y1 x2 y2 [ms]` | `input swipe` |
|
|
| Scroll | `artemis scroll up\|down` | picks the tallest vertical scrollable node, swipes inside it |
|
|
| Back | `artemis back` | key event 4 |
|
|
| Home | `artemis home` | key event 3 |
|
|
| Launch app | `artemis launch <package>` | `monkey -p <pkg> ... 1`, then waits for foreground |
|
|
| Wake | `artemis wake` | key event 224, polls `dumpsys power` until `Awake` |
|
|
| Read screen | `artemis dump` | `uiautomator dump`, parsed as XML, text nodes with bounds |
|
|
| Find (no tap) | `artemis find <text>` | dump → matching elements |
|
|
| Foreground app | `artemis fg` | `dumpsys window` → `mCurrentFocus` |
|
|
|
|
## 2. Definition of Done run
|
|
|
|
`artemis demo` launches Settings, backs out to the Settings home, taps
|
|
**Network and internet**, then proves the tap navigated by waiting for
|
|
**Aeroplane mode**, which only exists on the target page.
|
|
|
|
```
|
|
$ ./bin/artemis demo
|
|
connected to 100.101.49.17:46689
|
|
step 1: launch com.android.settings
|
|
step 2: tap "Network and internet"
|
|
tapped android.widget.TextView[title] text="Network and internet" bounds=[221,663][748,727] at (484,695)
|
|
step 3: wait for "Aeroplane mode"
|
|
step 4: foreground package is com.android.settings
|
|
demo OK
|
|
```
|
|
|
|
## 3. Other live checks
|
|
|
|
```
|
|
$ ./bin/artemis home ; ./bin/artemis fg
|
|
com.google.android.apps.nexuslauncher
|
|
|
|
$ ./bin/artemis launch com.spotify.music
|
|
launched com.spotify.music in 698ms
|
|
|
|
$ ./bin/artemis scroll down # Spotify home: tabs stay, list content changes
|
|
before: [TextView] Julius Caesar Part 1: The Roman Dictator
|
|
after: [TextView] It's New Music Friday!
|
|
|
|
$ ./bin/artemis tap "Search Settings" ; ./bin/artemis type "wifi"
|
|
tapped "Search Settings" at (412,236) — android.widget.TextView[search_bar_title] ...
|
|
|
|
$ ./bin/artemis dump | grep -i wifi
|
|
[EditText] wifi bounds=[126,121][933,310]
|
|
|
|
$ ./bin/artemis find "wi"
|
|
android.widget.EditText[open_search_view_edit_text] text="wifi" bounds=[126,121][933,310] centre=(529,215)
|
|
android.widget.TextView[title] text="Use Wi-Fi" bounds=[231,450][453,520] centre=(342,485)
|
|
```
|
|
|
|
Every primitive in the table has now been exercised against the phone: `wake`,
|
|
`dump`, `find`, `fg`, `launch`, `tap`, `type`, `scroll` (which uses `swipe`),
|
|
`back`, `home`, and `demo`.
|
|
|
|
## 4. Failure is loud (FR-8)
|
|
|
|
The first demo run used the wrong string, `"Network & internet"`, while the
|
|
Pixel spells it `"Network and internet"`. Artemis refused to act:
|
|
|
|
```
|
|
artemis: element with text "Network & internet" not found on screen; refusing to tap blindly
|
|
```
|
|
|
|
No coordinate was guessed. A test in `internal/ui/dump_test.go` pins the same
|
|
behaviour: a set of matches whose only member is disabled yields an error, not a
|
|
tap.
|
|
|
|
## 5. Design notes
|
|
|
|
- **Text first, coordinates derived.** `TapText` finds matches, ranks them
|
|
(clickable first, then enabled, then smallest area), reads the winner's
|
|
bounds, and taps the centre. Coordinates are never written down by hand.
|
|
- **Scroll targets vertical lists only.** An earlier version picked the
|
|
smallest scrollable node, which on Spotify was a short horizontal carousel; a
|
|
vertical swipe there became a mis-gesture. The current version picks the
|
|
tallest scrollable node that is taller than it is wide and covers at least a
|
|
third of the screen, else it falls back to the screen rectangle.
|
|
- **Launch waits for a result.** `Launch` does not return success until the
|
|
package is actually the foreground app, or it fails after 10s.
|
|
- **Timeout everywhere.** `adb.Runner` runs each command through
|
|
`context.WithTimeout`; a recipe can never hang.
|
|
|
|
## 6. Tests
|
|
|
|
```sh
|
|
nix shell nixpkgs#go -c go vet ./...
|
|
nix shell nixpkgs#go -c go test ./...
|
|
```
|
|
|
|
`internal/ui` unit tests cover bounds parsing, centre maths, exact vs substring
|
|
text matching, content descriptions, disabled matches, and shell-safe input
|
|
escaping. The live checks above cover the phone.
|
|
|
|
## 7. Not done / next
|
|
|
|
- No recipes file yet; that is Phase 3.
|
|
- No HTTP action API yet; that is Phase 6.
|
|
- Locked-screen handling is unsettled (§13 Q2): `wake` dismisses a non-secure
|
|
keyguard but cannot unlock a PIN-locked phone.
|