artemis_control/AGENTS.md

Artemis Control

Build spec for Paperclip. Follow the phases in section 11 in order. Do not skip a phase's Definition of Done. When something is unclear, ask Sam — do not guess.

---

1. Purpose and outcome

Artemis is the phone's hands. It drives the phone screen from .13 over Tailscale using ADB wireless, so the Hub can do things no deep link allows.

Outcome (the single success statement):

Sam says "Open Spotify, search Rolling Stones Honky Tonk, play it, make a song radio" and it happens on the phone, reliably.

Google Assistant cannot do this, because it cannot search inside an app. Artemis can, because it uses the app's own UI.

2. Scope

In scope

• Hold an ADB session to the phone over Tailscale.
• Read the screen (element text and positions).
• Act: launch, tap, type, swipe, scroll, back, home.
• Named recipes for common tasks (for example, play a song in Spotify).
• A small action API for the Hub.

Out of scope

• Understanding speech. The Hub does that.
• Native phone actions (calls, contacts, SMS). Those belong to the Kotlin app.
• The phone app itself. That is kotlin_phone_app_admin.
• Voice and outbound conversations. That is voice_outbound.

3. Users and roles

• No direct user. Artemis is driven by the Hub.
• Sam may edit the recipes file by hand.

4. Functional requirements

ID	Requirement
FR-1	Hold an ADB session to the phone over Tailscale. Reconnect after a drop.
FR-2	Read the screen with adb shell uiautomator dump — element text and bounds.
FR-3	Act with adb shell input: tap, type, swipe, scroll, back, home, key events.
FR-4	Find an element by text and tap its centre. Do not use fixed coordinates where text exists.
FR-5	Named recipes for known tasks (spotify_play, spotify_radio, maps_navigate).
FR-6	Action API: POST /act {"task": "...", "params": {...}} returns ok or failed with a reason.
FR-7	Wake the screen before acting.
FR-8	Fail cleanly: element not found, app not installed, app logged out, timeout. Never tap blindly.
FR-9	Health: report ADB connected and phone reachable.
FR-10	Log every action: task, steps, result, duration.
FR-11	Per-step timeout. A recipe never runs forever.

5. Screens and UX flow

No UI in v1. Artemis is headless. It exposes an API and reads a recipes file.

• Recipes file — a plain file (YAML or JSON) that Sam can edit to add or change a task. This replaces a UI for now.
• Optional later — a small desktop page to view recipes and recent runs. Build it only if the recipes file becomes painful.

Example flow

Hub: POST /act {"task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"}}
Artemis: wake screen → launch Spotify → tap "Search" → type query
         → wait → tap first result → tap Play
Artemis: {"status":"ok","duration_ms": 4200}

6. Look and feel

Not applicable in v1 (no UI). If a desktop page is added later, reuse the Family Home Lab design system: /home/sam/Docker/Containers/family-home-lab/portal/static/tokens.css.

7. Data model

Files and SQLite inside the service. Suggested:

Store	Holds
recipes/	One file per task. Steps and selectors. Editable.
action_log (SQLite)	task, params, steps, result, duration, created-at
session	ADB connection state

8. Integrations

System	Use
ADB	The control channel. Wireless, over Tailscale.
Tailscale	Reaches the phone from .13 when away from home.
Hub	Calls /act. Nothing comes back to the Hub.
Kotlin app	The partition: native actions go there, in-app navigation comes here.
OmniRoute	Only if a future step needs a vision fallback. Not in v1.

9. Technology (locked)

Item	Value
Language	Go
Transport	ADB wireless over Tailscale
Host	.13, Docker (or a systemd service — decide in Phase 1)
Container	This project's own.
Port	Allocate one (suggest 8793) for the action API.
URL	none (no UI). A status page is optional later.
Auth	A bearer token for the Hub, from environment.d.

10. Rules and constraints

1. No LLM in the action path. Recipes are deterministic.
2. Find by text, not fixed coordinates, where text exists.
3. Never tap blindly. If a step cannot find its target, fail.
4. Screen must be awake. Wake it first.
5. Secrets come from environment.d.
6. Log every action.
7. Own folder, own container, own port.
8. Do not put the Hub in the loop for the phone call itself. Calls are the Kotlin app.

11. Build order

Phase 1 — Prove the ADB link Install ADB on .13. Pair the phone. Connect over Tailscale. Screenshot and dump the UI tree. DoD: the view tree of a real phone screen is fetched. This is the risk phase — prove it first.

Phase 2 — Act primitives Tap by text, type, scroll, back, home, launch an app. DoD: Artemis opens a named app and taps a named element on it.

Phase 3 — Spotify recipe Launch, search, play. DoD: spotify_play with a real query plays the right song.

Phase 4 — Song radio Open the menu and choose "song radio". DoD: spotify_radio starts a real radio.

Phase 5 — Maps recipe Open maps and search a place. DoD: maps_navigate opens a real search.

Phase 6 — Action API POST /act, bearer token, health, timeouts. DoD: the Hub calls /act and the phone responds.

Phase 7 — Resilience Wake the screen, reconnect after a drop, fail cleanly on a missing element. DoD: a dropped ADB session recovers without a restart.

Phase 8 — Documentation DoD: project-docs run; docs updated.

12. Acceptance criteria

• spotify_play plays the correct song from a spoken query, end to end.
• spotify_radio starts a radio.
• A missing element produces a clean failure, not a wrong tap.
• A dropped ADB session recovers automatically.
• The Hub can call /act and get a result.
• Every action is logged.

13. Decisions and remaining questions

Resolved:

1. ADB over Tailscale is the mechanism.
2. No UI in v1. A recipes file replaces a UI.
3. Find by text from the UI dump, not screenshots or vision.
4. Partition: native actions → Kotlin app; in-app navigation → Artemis.
5. ADB tooling lives in the user profile on .13, installed with nix profile install nixpkgs#android-tools — not a nixos-rebuild system module. Wireless ADB over TCP needs no udev rules; programs.adb was USB-only and is removed in this NixOS release anyway. Go is likewise run from a one-off nix shell nixpkgs#go, not installed system-wide. Never run nixos-rebuild on .13: it runs the Hub, MQTT and every service.
6. Locked screen — Artemis fails loudly. It never unlocks the phone. It detects a secure keyguard (PIN, pattern or password) before acting and refuses with phone is locked; unlock it and retry. It does not dismiss a secure keyguard and it does not type anything into one. A non-secure keyguard (the phone has no PIN at all) is still dismissed, because that screen guards nothing. No PIN is stored, read or transmitted, and there is no secret for one. Storing a PIN for auto-unlock is rejected for now — the phone is a pocket device, and Artemis is a convenience, not a reason to keep Sam's unlock credential in a service on .13. This is revisit-able if the daily friction turns out to be worse than the risk.

Remaining:

1. Which tasks after Spotify and maps? (WhatsApp message? Other music apps?)
2. Action API shape: one generic /act, or one endpoint per task?
3. Service packaging for Artemis on .13 — container or systemd — is still open. ADB can be awkward in a container, so this is decided in Phase 6. The tooling-placement half of the old Q4 is resolved above.
4. Speed: what is an acceptable time for a 6-step recipe?
5. The recipes file has grown guard, tap_if and wait_gone verbs. If it keeps growing, decide whether the format stays bespoke or moves to YAML.