feat: Phase 2 act primitives
Add the Artemis Go CLI and its packages: - internal/adb: timeout-bounded adb wrapper, device list, wireless-debugging port scan and auto-connect (never hardcodes the moving port). - internal/ui: uiautomator dump parsing, find-by-text, tap the centre of a text match (FR-4), type, swipe, scroll, back, home, launch, wake. - cmd/artemis: CLI for the primitives plus 'demo', the Phase 2 DoD run. Failures are loud: a missing or disabled element errors instead of tapping a guessed coordinate. The screen is woken before acting, and every adb call has a deadline. Also record the section 13 Q4 decision (adb lives in the user profile on .13, wireless ADB needs no udev rules; service packaging still open until Phase 6), add unit tests, and document the phase evidence.
This commit is contained in:
103
docs/phase-2-act-primitives.md
Normal file
103
docs/phase-2-act-primitives.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# Phase 2 — Act primitives
|
||||
|
||||
Status: **done**. Verified live against the Pixel 8a on 2026-10-16.
|
||||
|
||||
Definition of Done: *Artemis opens a named app and taps a named element on it.*
|
||||
|
||||
---
|
||||
|
||||
## 1. Primitives
|
||||
|
||||
All primitives live in `internal/ui`. Every acting command wakes the screen
|
||||
first (FR-7) and every adb call is bounded by a timeout (FR-11).
|
||||
|
||||
| Primitive | Command | How it works |
|
||||
|---|---|---|
|
||||
| Tap by text | `artemis tap <text>` | dump → match text/content-desc → tap element centre (FR-4) |
|
||||
| Type | `artemis type <text>` | `input text`, with `%s` space encoding and shell-safe quoting |
|
||||
| Swipe | `artemis swipe x1 y1 x2 y2 [ms]` | `input swipe` |
|
||||
| Scroll | `artemis scroll up\|down` | picks the tallest vertical scrollable node, swipes inside it |
|
||||
| Back | `artemis back` | key event 4 |
|
||||
| Home | `artemis home` | key event 3 |
|
||||
| Launch app | `artemis launch <package>` | `monkey -p <pkg> ... 1`, then waits for foreground |
|
||||
| Wake | `artemis wake` | key event 224, polls `dumpsys power` until `Awake` |
|
||||
| Read screen | `artemis dump` | `uiautomator dump`, parsed as XML, text nodes with bounds |
|
||||
| Find (no tap) | `artemis find <text>` | dump → matching elements |
|
||||
| Foreground app | `artemis fg` | `dumpsys window` → `mCurrentFocus` |
|
||||
|
||||
## 2. Definition of Done run
|
||||
|
||||
`artemis demo` launches Settings, backs out to the Settings home, taps
|
||||
**Network and internet**, then proves the tap navigated by waiting for
|
||||
**Aeroplane mode**, which only exists on the target page.
|
||||
|
||||
```
|
||||
$ ./bin/artemis demo
|
||||
connected to 100.101.49.17:46689
|
||||
step 1: launch com.android.settings
|
||||
step 2: tap "Network and internet"
|
||||
tapped android.widget.TextView[title] text="Network and internet" bounds=[221,663][748,727] at (484,695)
|
||||
step 3: wait for "Aeroplane mode"
|
||||
step 4: foreground package is com.android.settings
|
||||
demo OK
|
||||
```
|
||||
|
||||
## 3. Other live checks
|
||||
|
||||
```
|
||||
$ ./bin/artemis home ; ./bin/artemis fg
|
||||
com.google.android.apps.nexuslauncher
|
||||
|
||||
$ ./bin/artemis launch com.spotify.music
|
||||
launched com.spotify.music in 698ms
|
||||
|
||||
$ ./bin/artemis scroll down # Spotify home: tabs stay, list content changes
|
||||
before: [TextView] Julius Caesar Part 1: The Roman Dictator
|
||||
after: [TextView] It's New Music Friday!
|
||||
```
|
||||
|
||||
## 4. Failure is loud (FR-8)
|
||||
|
||||
The first demo run used the wrong string, `"Network & internet"`, while the
|
||||
Pixel spells it `"Network and internet"`. Artemis refused to act:
|
||||
|
||||
```
|
||||
artemis: element with text "Network & internet" not found on screen; refusing to tap blindly
|
||||
```
|
||||
|
||||
No coordinate was guessed. A test in `internal/ui/dump_test.go` pins the same
|
||||
behaviour: a set of matches whose only member is disabled yields an error, not a
|
||||
tap.
|
||||
|
||||
## 5. Design notes
|
||||
|
||||
- **Text first, coordinates derived.** `TapText` finds matches, ranks them
|
||||
(clickable first, then enabled, then smallest area), reads the winner's
|
||||
bounds, and taps the centre. Coordinates are never written down by hand.
|
||||
- **Scroll targets vertical lists only.** An earlier version picked the
|
||||
smallest scrollable node, which on Spotify was a short horizontal carousel; a
|
||||
vertical swipe there became a mis-gesture. The current version picks the
|
||||
tallest scrollable node that is taller than it is wide and covers at least a
|
||||
third of the screen, else it falls back to the screen rectangle.
|
||||
- **Launch waits for a result.** `Launch` does not return success until the
|
||||
package is actually the foreground app, or it fails after 10s.
|
||||
- **Timeout everywhere.** `adb.Runner` runs each command through
|
||||
`context.WithTimeout`; a recipe can never hang.
|
||||
|
||||
## 6. Tests
|
||||
|
||||
```sh
|
||||
nix shell nixpkgs#go -c go vet ./...
|
||||
nix shell nixpkgs#go -c go test ./...
|
||||
```
|
||||
|
||||
`internal/ui` unit tests cover bounds parsing, centre maths, exact vs substring
|
||||
text matching, content descriptions, disabled matches, and shell-safe input
|
||||
escaping. The live checks above cover the phone.
|
||||
|
||||
## 7. Not done / next
|
||||
|
||||
- No recipes file yet; that is Phase 3.
|
||||
- No HTTP action API yet; that is Phase 6.
|
||||
- Locked-screen handling is unsettled (§13 Q2): `wake` dismisses a non-secure
|
||||
keyguard but cannot unlock a PIN-locked phone.
|
||||
Reference in New Issue
Block a user