From c0d3e635322572384797cb7e59c3c225ccde7d2b Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Sun, 11 Oct 2026 17:24:04 +1100 Subject: [PATCH] Initial spec: AGENTS.md build spec + archived original notes --- .gitignore | 36 ++++++++++ AGENTS.md | 179 ++++++++++++++++++++++++++++++++++++++++++++++ ORIGINAL_NOTES.md | 23 ++++++ 3 files changed, 238 insertions(+) create mode 100644 .gitignore create mode 100644 AGENTS.md create mode 100644 ORIGINAL_NOTES.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..23c0a75 --- /dev/null +++ b/.gitignore @@ -0,0 +1,36 @@ +# Secrets +.env +.env.* +*.local +*.keystore + +# Go +bin/ +*.exe +*.test +*.out + +# Python +__pycache__/ +*.pyc +.venv/ +venv/ + +# Node / web +node_modules/ +dist/ +*.visual-check.* + +# Android / Kotlin +.gradle/ +build/ +local.properties +*.apk + +# OS / editors +.DS_Store +*.swp + + +# Pi agent generated memory (local state, not source) +.pi/memory-vault/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..a32abdd --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,179 @@ +# Artemis Control + +> Build spec for Paperclip. Follow the phases in section 11 in order. Do not skip a +> phase's Definition of Done. When something is unclear, ask Sam — do not guess. + +--- + +## 1. Purpose and outcome + +Artemis is the phone's **hands**. It drives the phone screen from `.13` over +Tailscale using **ADB wireless**, so the Hub can do things no deep link allows. + +**Outcome (the single success statement):** +> Sam says *"Open Spotify, search Rolling Stones Honky Tonk, play it, make a song +> radio"* and it happens on the phone, reliably. + +Google Assistant cannot do this, because it cannot search inside an app. Artemis +can, because it uses the app's own UI. + +## 2. Scope + +**In scope** +- Hold an ADB session to the phone over Tailscale. +- Read the screen (element text and positions). +- Act: launch, tap, type, swipe, scroll, back, home. +- Named **recipes** for common tasks (for example, play a song in Spotify). +- A small action API for the Hub. + +**Out of scope** +- Understanding speech. The Hub does that. +- Native phone actions (calls, contacts, SMS). Those belong to the Kotlin app. +- The phone app itself. That is `kotlin_phone_app_admin`. +- Voice and outbound conversations. That is `voice_outbound`. + +## 3. Users and roles + +- **No direct user.** Artemis is driven by the **Hub**. +- Sam may edit the **recipes file** by hand. + +## 4. Functional requirements + +| ID | Requirement | +|---|---| +| FR-1 | Hold an ADB session to the phone over Tailscale. Reconnect after a drop. | +| FR-2 | Read the screen with `adb shell uiautomator dump` — element text and bounds. | +| FR-3 | Act with `adb shell input`: tap, type, swipe, scroll, back, home, key events. | +| FR-4 | **Find an element by text and tap its centre.** Do not use fixed coordinates where text exists. | +| FR-5 | Named recipes for known tasks (`spotify_play`, `spotify_radio`, `maps_navigate`). | +| FR-6 | Action API: `POST /act {"task": "...", "params": {...}}` returns `ok` or `failed` with a reason. | +| FR-7 | Wake the screen before acting. | +| FR-8 | Fail cleanly: element not found, app not installed, app logged out, timeout. Never tap blindly. | +| FR-9 | Health: report ADB connected and phone reachable. | +| FR-10 | Log every action: task, steps, result, duration. | +| FR-11 | Per-step timeout. A recipe never runs forever. | + +## 5. Screens and UX flow + +**No UI in v1.** Artemis is headless. It exposes an API and reads a recipes file. + +- **Recipes file** — a plain file (YAML or JSON) that Sam can edit to add or change + a task. This replaces a UI for now. +- **Optional later** — a small desktop page to view recipes and recent runs. Build + it only if the recipes file becomes painful. + +**Example flow** +``` +Hub: POST /act {"task":"spotify_play","params":{"query":"Rolling Stones Honky Tonk"}} +Artemis: wake screen → launch Spotify → tap "Search" → type query + → wait → tap first result → tap Play +Artemis: {"status":"ok","duration_ms": 4200} +``` + +## 6. Look and feel + +Not applicable in v1 (no UI). If a desktop page is added later, reuse the Family +Home Lab design system: +`/home/sam/Docker/Containers/family-home-lab/portal/static/tokens.css`. + +## 7. Data model + +Files and SQLite inside the service. Suggested: + +| Store | Holds | +|---|---| +| `recipes/` | One file per task. Steps and selectors. Editable. | +| `action_log` (SQLite) | task, params, steps, result, duration, created-at | +| `session` | ADB connection state | + +## 8. Integrations + +| System | Use | +|---|---| +| **ADB** | The control channel. Wireless, over Tailscale. | +| **Tailscale** | Reaches the phone from `.13` when away from home. | +| **Hub** | Calls `/act`. Nothing comes back to the Hub. | +| **Kotlin app** | The partition: native actions go there, in-app navigation comes here. | +| **OmniRoute** | Only if a future step needs a vision fallback. Not in v1. | + +## 9. Technology (locked) + +| Item | Value | +|---|---| +| Language | **Go** | +| Transport | **ADB wireless over Tailscale** | +| Host | `.13`, Docker (or a systemd service — decide in Phase 1) | +| Container | **This project's own.** | +| Port | Allocate one (suggest **8793**) for the action API. | +| URL | none (no UI). A status page is optional later. | +| Auth | A bearer token for the Hub, from `environment.d`. | + +## 10. Rules and constraints + +1. **No LLM in the action path.** Recipes are deterministic. +2. **Find by text**, not fixed coordinates, where text exists. +3. **Never tap blindly.** If a step cannot find its target, fail. +4. Screen must be awake. Wake it first. +5. Secrets come from `environment.d`. +6. Log every action. +7. Own folder, own container, own port. +8. Do not put the Hub in the loop for the phone call itself. Calls are the Kotlin app. + +## 11. Build order + +**Phase 1 — Prove the ADB link** +Install ADB on `.13`. Pair the phone. Connect over Tailscale. Screenshot and dump the +UI tree. +*DoD:* the view tree of a real phone screen is fetched. **This is the risk phase — +prove it first.** + +**Phase 2 — Act primitives** +Tap by text, type, scroll, back, home, launch an app. +*DoD:* Artemis opens a named app and taps a named element on it. + +**Phase 3 — Spotify recipe** +Launch, search, play. +*DoD:* `spotify_play` with a real query plays the right song. + +**Phase 4 — Song radio** +Open the menu and choose "song radio". +*DoD:* `spotify_radio` starts a real radio. + +**Phase 5 — Maps recipe** +Open maps and search a place. +*DoD:* `maps_navigate` opens a real search. + +**Phase 6 — Action API** +`POST /act`, bearer token, health, timeouts. +*DoD:* the Hub calls `/act` and the phone responds. + +**Phase 7 — Resilience** +Wake the screen, reconnect after a drop, fail cleanly on a missing element. +*DoD:* a dropped ADB session recovers without a restart. + +**Phase 8 — Documentation** +*DoD:* project-docs run; docs updated. + +## 12. Acceptance criteria + +- `spotify_play` plays the correct song from a spoken query, end to end. +- `spotify_radio` starts a radio. +- A missing element produces a clean failure, not a wrong tap. +- A dropped ADB session recovers automatically. +- The Hub can call `/act` and get a result. +- Every action is logged. + +## 13. Decisions and remaining questions + +Resolved: +1. **ADB over Tailscale** is the mechanism. +2. **No UI in v1.** A recipes file replaces a UI. +3. **Find by text** from the UI dump, not screenshots or vision. +4. **Partition:** native actions → Kotlin app; in-app navigation → Artemis. + +Remaining: +1. Which tasks after Spotify and maps? (WhatsApp message? Other music apps?) +2. Locked screen: must Artemis unlock it, or does Sam leave it awake? +3. Action API shape: one generic `/act`, or one endpoint per task? +4. Docker or a systemd service on `.13`? ADB can be awkward in a container. +5. Speed: what is an acceptable time for a 6-step recipe? diff --git a/ORIGINAL_NOTES.md b/ORIGINAL_NOTES.md new file mode 100644 index 0000000..137fd74 --- /dev/null +++ b/ORIGINAL_NOTES.md @@ -0,0 +1,23 @@ +# Artemis Control — Original Notes (archived) + +> Historical pre-planning notes. **Superseded by `AGENTS.md`.** Do not build from this. + + +> Kept for provenance. This folder was once `voice_artemis_minimax` and mixed three +> things. It was split. Do not build from these notes. + +``` +Original (thin) AGENTS.md +- Artemis with MiniMax and LLM to create phone conversations. +- Might be good as general media. +- Integrate phone and WhatsApp. +- Interface: target person, phone, WhatsApp; tone and objective of the conversation. + +The split (2026-10-10) +- Artemis screen control → THIS project (artemis_control) +- Phone app + admin (Kotlin) → kotlin_phone_app_admin +- Outbound voice / conversations → voice_outbound + +GOOGLE_NOTES.md (Jeeves phone app) was moved to kotlin_phone_app_admin, because it +describes the phone app, not the phone control layer. +```