artemis_control/docs/phase-2-act-primitives.md

Phase 2 — Act primitives

Status: done. Verified live against the Pixel 8a on 2026-10-16.

Definition of Done: Artemis opens a named app and taps a named element on it.

---

1. Primitives

All primitives live in internal/ui. Every acting command wakes the screen first (FR-7) and every adb call is bounded by a timeout (FR-11).

Primitive	Command	How it works
Tap by text	artemis tap <text>	dump → match text/content-desc → tap element centre (FR-4)
Type	artemis type <text>	input text, with %s space encoding and shell-safe quoting
Swipe	artemis swipe x1 y1 x2 y2 [ms]	input swipe
Scroll	artemis scroll up|down	picks the tallest vertical scrollable node, swipes inside it
Back	artemis back	key event 4
Home	artemis home	key event 3
Launch app	artemis launch <package>	monkey -p <pkg> ... 1, then waits for foreground
Wake	artemis wake	key event 224, polls dumpsys power until Awake
Read screen	artemis dump	uiautomator dump, parsed as XML, text nodes with bounds
Find (no tap)	artemis find <text>	dump → matching elements
Foreground app	artemis fg	dumpsys window → mCurrentFocus

2. Definition of Done run

artemis demo launches Settings, backs out to the Settings home, taps Network and internet, then proves the tap navigated by waiting for Aeroplane mode, which only exists on the target page.

$ ./bin/artemis demo
connected to 100.101.49.17:46689
step 1: launch com.android.settings
step 2: tap "Network and internet"
        tapped android.widget.TextView[title] text="Network and internet" bounds=[221,663][748,727] at (484,695)
step 3: wait for "Aeroplane mode"
step 4: foreground package is com.android.settings
demo OK

3. Other live checks

$ ./bin/artemis home ; ./bin/artemis fg
com.google.android.apps.nexuslauncher

$ ./bin/artemis launch com.spotify.music
launched com.spotify.music in 698ms

$ ./bin/artemis scroll down      # Spotify home: tabs stay, list content changes
before: [TextView] Julius Caesar Part 1: The Roman Dictator
after:  [TextView] It's New Music Friday!

$ ./bin/artemis tap "Search Settings" ; ./bin/artemis type "wifi"
tapped "Search Settings" at (412,236) — android.widget.TextView[search_bar_title] ...

$ ./bin/artemis dump | grep -i wifi
[EditText] wifi   bounds=[126,121][933,310]

$ ./bin/artemis find "wi"
android.widget.EditText[open_search_view_edit_text] text="wifi" bounds=[126,121][933,310]  centre=(529,215)
android.widget.TextView[title] text="Use Wi-Fi" bounds=[231,450][453,520]  centre=(342,485)

Every primitive in the table has now been exercised against the phone: wake, dump, find, fg, launch, tap, type, scroll (which uses swipe), back, home, and demo.

4. Failure is loud (FR-8)

The first demo run used the wrong string, "Network & internet", while the Pixel spells it "Network and internet". Artemis refused to act:

artemis: element with text "Network & internet" not found on screen; refusing to tap blindly

No coordinate was guessed. A test in internal/ui/dump_test.go pins the same behaviour: a set of matches whose only member is disabled yields an error, not a tap.

5. Design notes

• Text first, coordinates derived. TapText finds matches, ranks them (clickable first, then enabled, then smallest area), reads the winner's bounds, and taps the centre. Coordinates are never written down by hand.
• Scroll targets vertical lists only. An earlier version picked the smallest scrollable node, which on Spotify was a short horizontal carousel; a vertical swipe there became a mis-gesture. The current version picks the tallest scrollable node that is taller than it is wide and covers at least a third of the screen, else it falls back to the screen rectangle.
• Launch waits for a result. Launch does not return success until the package is actually the foreground app, or it fails after 10s.
• Timeout everywhere. adb.Runner runs each command through context.WithTimeout; a recipe can never hang.

6. Tests

nix shell nixpkgs#go -c go vet ./...
nix shell nixpkgs#go -c go test ./...

internal/ui unit tests cover bounds parsing, centre maths, exact vs substring text matching, content descriptions, disabled matches, and shell-safe input escaping. The live checks above cover the phone.

7. Not done / next

• No recipes file yet; that is Phase 3.
• No HTTP action API yet; that is Phase 6.
• Locked-screen handling is unsettled (§13 Q2): wake dismisses a non-secure keyguard but cannot unlock a PIN-locked phone.