Initial spec: AGENTS.md build spec + archived original notes
This commit is contained in:
179
AGENTS.md
Normal file
179
AGENTS.md
Normal file
@@ -0,0 +1,179 @@
|
||||
# Artemis Control
|
||||
|
||||
> Build spec for Paperclip. Follow the phases in section 11 in order. Do not skip a
|
||||
> phase's Definition of Done. When something is unclear, ask Sam — do not guess.
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose and outcome
|
||||
|
||||
Artemis is the phone's **hands**. It drives the phone screen from `.13` over
|
||||
Tailscale using **ADB wireless**, so the Hub can do things no deep link allows.
|
||||
|
||||
**Outcome (the single success statement):**
|
||||
> Sam says *"Open Spotify, search Rolling Stones Honky Tonk, play it, make a song
|
||||
> radio"* and it happens on the phone, reliably.
|
||||
|
||||
Google Assistant cannot do this, because it cannot search inside an app. Artemis
|
||||
can, because it uses the app's own UI.
|
||||
|
||||
## 2. Scope
|
||||
|
||||
**In scope**
|
||||
- Hold an ADB session to the phone over Tailscale.
|
||||
- Read the screen (element text and positions).
|
||||
- Act: launch, tap, type, swipe, scroll, back, home.
|
||||
- Named **recipes** for common tasks (for example, play a song in Spotify).
|
||||
- A small action API for the Hub.
|
||||
|
||||
**Out of scope**
|
||||
- Understanding speech. The Hub does that.
|
||||
- Native phone actions (calls, contacts, SMS). Those belong to the Kotlin app.
|
||||
- The phone app itself. That is `kotlin_phone_app_admin`.
|
||||
- Voice and outbound conversations. That is `voice_outbound`.
|
||||
|
||||
## 3. Users and roles
|
||||
|
||||
- **No direct user.** Artemis is driven by the **Hub**.
|
||||
- Sam may edit the **recipes file** by hand.
|
||||
|
||||
## 4. Functional requirements
|
||||
|
||||
| ID | Requirement |
|
||||
|---|---|
|
||||
| FR-1 | Hold an ADB session to the phone over Tailscale. Reconnect after a drop. |
|
||||
| FR-2 | Read the screen with `adb shell uiautomator dump` — element text and bounds. |
|
||||
| FR-3 | Act with `adb shell input`: tap, type, swipe, scroll, back, home, key events. |
|
||||
| FR-4 | **Find an element by text and tap its centre.** Do not use fixed coordinates where text exists. |
|
||||
| FR-5 | Named recipes for known tasks (`spotify_play`, `spotify_radio`, `maps_navigate`). |
|
||||
| FR-6 | Action API: `POST /act {"task": "...", "params": {...}}` returns `ok` or `failed` with a reason. |
|
||||
| FR-7 | Wake the screen before acting. |
|
||||
| FR-8 | Fail cleanly: element not found, app not installed, app logged out, timeout. Never tap blindly. |
|
||||
| FR-9 | Health: report ADB connected and phone reachable. |
|
||||
| FR-10 | Log every action: task, steps, result, duration. |
|
||||
| FR-11 | Per-step timeout. A recipe never runs forever. |
|
||||
|
||||
## 5. Screens and UX flow
|
||||
|
||||
**No UI in v1.** Artemis is headless. It exposes an API and reads a recipes file.
|
||||
|
||||
- **Recipes file** — a plain file (YAML or JSON) that Sam can edit to add or change
|
||||
a task. This replaces a UI for now.
|
||||
- **Optional later** — a small desktop page to view recipes and recent runs. Build
|
||||
it only if the recipes file becomes painful.
|
||||
|
||||
**Example flow**
|
||||
```
|
||||
Hub: POST /act {"task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"}}
|
||||
Artemis: wake screen → launch Spotify → tap "Search" → type query
|
||||
→ wait → tap first result → tap Play
|
||||
Artemis: {"status":"ok","duration_ms": 4200}
|
||||
```
|
||||
|
||||
## 6. Look and feel
|
||||
|
||||
Not applicable in v1 (no UI). If a desktop page is added later, reuse the Family
|
||||
Home Lab design system:
|
||||
`/home/sam/Docker/Containers/family-home-lab/portal/static/tokens.css`.
|
||||
|
||||
## 7. Data model
|
||||
|
||||
Files and SQLite inside the service. Suggested:
|
||||
|
||||
| Store | Holds |
|
||||
|---|---|
|
||||
| `recipes/` | One file per task. Steps and selectors. Editable. |
|
||||
| `action_log` (SQLite) | task, params, steps, result, duration, created-at |
|
||||
| `session` | ADB connection state |
|
||||
|
||||
## 8. Integrations
|
||||
|
||||
| System | Use |
|
||||
|---|---|
|
||||
| **ADB** | The control channel. Wireless, over Tailscale. |
|
||||
| **Tailscale** | Reaches the phone from `.13` when away from home. |
|
||||
| **Hub** | Calls `/act`. Nothing comes back to the Hub. |
|
||||
| **Kotlin app** | The partition: native actions go there, in-app navigation comes here. |
|
||||
| **OmniRoute** | Only if a future step needs a vision fallback. Not in v1. |
|
||||
|
||||
## 9. Technology (locked)
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Language | **Go** |
|
||||
| Transport | **ADB wireless over Tailscale** |
|
||||
| Host | `.13`, Docker (or a systemd service — decide in Phase 1) |
|
||||
| Container | **This project's own.** |
|
||||
| Port | Allocate one (suggest **8793**) for the action API. |
|
||||
| URL | none (no UI). A status page is optional later. |
|
||||
| Auth | A bearer token for the Hub, from `environment.d`. |
|
||||
|
||||
## 10. Rules and constraints
|
||||
|
||||
1. **No LLM in the action path.** Recipes are deterministic.
|
||||
2. **Find by text**, not fixed coordinates, where text exists.
|
||||
3. **Never tap blindly.** If a step cannot find its target, fail.
|
||||
4. Screen must be awake. Wake it first.
|
||||
5. Secrets come from `environment.d`.
|
||||
6. Log every action.
|
||||
7. Own folder, own container, own port.
|
||||
8. Do not put the Hub in the loop for the phone call itself. Calls are the Kotlin app.
|
||||
|
||||
## 11. Build order
|
||||
|
||||
**Phase 1 — Prove the ADB link**
|
||||
Install ADB on `.13`. Pair the phone. Connect over Tailscale. Screenshot and dump the
|
||||
UI tree.
|
||||
*DoD:* the view tree of a real phone screen is fetched. **This is the risk phase —
|
||||
prove it first.**
|
||||
|
||||
**Phase 2 — Act primitives**
|
||||
Tap by text, type, scroll, back, home, launch an app.
|
||||
*DoD:* Artemis opens a named app and taps a named element on it.
|
||||
|
||||
**Phase 3 — Spotify recipe**
|
||||
Launch, search, play.
|
||||
*DoD:* `spotify_play` with a real query plays the right song.
|
||||
|
||||
**Phase 4 — Song radio**
|
||||
Open the menu and choose "song radio".
|
||||
*DoD:* `spotify_radio` starts a real radio.
|
||||
|
||||
**Phase 5 — Maps recipe**
|
||||
Open maps and search a place.
|
||||
*DoD:* `maps_navigate` opens a real search.
|
||||
|
||||
**Phase 6 — Action API**
|
||||
`POST /act`, bearer token, health, timeouts.
|
||||
*DoD:* the Hub calls `/act` and the phone responds.
|
||||
|
||||
**Phase 7 — Resilience**
|
||||
Wake the screen, reconnect after a drop, fail cleanly on a missing element.
|
||||
*DoD:* a dropped ADB session recovers without a restart.
|
||||
|
||||
**Phase 8 — Documentation**
|
||||
*DoD:* project-docs run; docs updated.
|
||||
|
||||
## 12. Acceptance criteria
|
||||
|
||||
- `spotify_play` plays the correct song from a spoken query, end to end.
|
||||
- `spotify_radio` starts a radio.
|
||||
- A missing element produces a clean failure, not a wrong tap.
|
||||
- A dropped ADB session recovers automatically.
|
||||
- The Hub can call `/act` and get a result.
|
||||
- Every action is logged.
|
||||
|
||||
## 13. Decisions and remaining questions
|
||||
|
||||
Resolved:
|
||||
1. **ADB over Tailscale** is the mechanism.
|
||||
2. **No UI in v1.** A recipes file replaces a UI.
|
||||
3. **Find by text** from the UI dump, not screenshots or vision.
|
||||
4. **Partition:** native actions → Kotlin app; in-app navigation → Artemis.
|
||||
|
||||
Remaining:
|
||||
1. Which tasks after Spotify and maps? (WhatsApp message? Other music apps?)
|
||||
2. Locked screen: must Artemis unlock it, or does Sam leave it awake?
|
||||
3. Action API shape: one generic `/act`, or one endpoint per task?
|
||||
4. Docker or a systemd service on `.13`? ADB can be awkward in a container.
|
||||
5. Speed: what is an acceptable time for a 6-step recipe?
|
||||
Reference in New Issue
Block a user