commit a3b93f6cf35c518979effbd1b2b425fc9290406d Author: Sam Rolfe Date: Sun Oct 11 17:24:04 2026 +1100 Initial spec: AGENTS.md build spec + archived original notes diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..23c0a75 --- /dev/null +++ b/.gitignore @@ -0,0 +1,36 @@ +# Secrets +.env +.env.* +*.local +*.keystore + +# Go +bin/ +*.exe +*.test +*.out + +# Python +__pycache__/ +*.pyc +.venv/ +venv/ + +# Node / web +node_modules/ +dist/ +*.visual-check.* + +# Android / Kotlin +.gradle/ +build/ +local.properties +*.apk + +# OS / editors +.DS_Store +*.swp + + +# Pi agent generated memory (local state, not source) +.pi/memory-vault/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..7ffb140 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,180 @@ +# Kotlin Phone App + Admin + +> Build spec for Paperclip. Follow the phases in section 11 in order. Do not skip a +> phase's Definition of Done. When something is unclear, ask Sam — do not guess. + +--- + +## 1. Purpose and outcome + +Sam's Android app. It is the **Hub's way in from the phone**, and the **trusted +admin surface** for approvals and parental controls. + +**Outcome (the single success statement):** +> Sam talks to the Hub from his phone, hears the answer, and can approve or change +> anything from the same app. + +## 2. Scope + +**In scope** +- Capture voice and send it to the Hub. +- Show the Hub's reply: text, spoken audio, and notifications. +- Show approval prompts ("Did you ask for this?") and answer them. +- Admin controls: parental controls, by calling the internet control service. +- Native phone actions the Hub requests (call, open a link, open an app by deep link). + +**Out of scope** +- The **kids' phone client.** That stays under `internet_use_control`. +- **Artemis** (screen control). Separate project. +- The **Hub** service itself. +- Understanding speech. The Hub does that. + +**Decision to confirm:** this is **one app with two areas** — Hub (voice) and Admin. +Not two apps. + +## 3. Users and roles + +- **Sam only.** This app is not for Jo, Harry, or Finn. +- The kids' client is a different app, built under `internet_use_control`. + +## 4. Functional requirements + +| ID | Requirement | +|---|---| +| FR-1 | Capture voice. Start with **hold-to-talk**; add a wake word later. | +| FR-2 | Send the transcript to the Hub (`POST /ask`) and show the reply as text. | +| FR-3 | Play the Hub's reply as **TTS on the phone**, when the reply asks for it. | +| FR-4 | Show **approval prompts** and return approve or deny to the Hub. | +| FR-5 | Admin: list the kids' devices and their state, and lock, unlock, or grant time. | +| FR-6 | Admin calls the **internet use control** API directly. The app holds no control logic. | +| FR-7 | Gate admin actions behind a **keycode or biometric**. | +| FR-8 | Perform native actions the Hub requests: place a call, open a deep link. | +| FR-9 | Show Hub `/notify` messages as Android notifications. | +| FR-10 | Settings: Hub URL and token. Test the connection. | +| FR-11 | Offline: fail cleanly, and never claim a command succeeded. | + +## 5. Screens and UX flow + +**Screens** +1. **Home (Hub)** — a large talk button, the last reply, and recent history. +2. **Approvals** — pending "Did you ask for this?" items with Approve / Deny. +3. **Admin** — the kids' devices, each with its state and quick actions. +4. **Settings** — Hub URL, token, keycode, test connection. + +**Main flow** +``` +Sam holds the button and speaks → the Hub replies → the phone speaks it +If the Hub needs approval → a card appears → Sam taps Approve +If Sam opens Admin → he locks a phone → the internet control service applies it +``` + +## 6. Look and feel + +Reuse the Family Home Lab design system, adapted to Android. + +- Source: `/home/sam/Docker/Containers/family-home-lab/DESIGN.md` +- Tokens: `/home/sam/Docker/Containers/family-home-lab/portal/static/tokens.css` + +Map the tokens to the app theme: canvas `#f6f5f4`, one action blue `#0075de`, Inter, +8px spacing, 4px inputs, pill buttons, hairline dividers, dark mode. The login/consent +moment is the one dark indigo band. + +## 7. Data model + +On the device only. No server database. + +| Store | Holds | +|---|---| +| Settings | Hub URL, Hub token (encrypted), keycode hash | +| Cache | recent replies, pending approvals | +| Log | last actions, for debugging | + +## 8. Integrations + +| System | Use | +|---|---| +| **Hub** | `POST /ask`, approval callbacks, `/notify`. Frozen in `central_ai_llm_master/docs/hub-api.md`. | +| **Internet use control** | Admin lock, unlock, grant. Direct API call. See `internet_use_control/SYSTEM_MAP.md`. | +| **Android telephony** | Native calls and contacts. | +| **Deep links** | Opening a specific song or place. | +| **Artemis** | Not called from here. The Hub calls it. Native actions stay in this app. | + +## 9. Technology (locked) + +| Item | Value | +|---|---| +| Language | **Kotlin** (Android) | +| Voice | `SpeechRecognizer` first; a wake-word engine later | +| Transport | HTTP to the Hub. **LAN, or Tailscale when away** — decide in Phase 1. | +| Push | FCM, or polling for approvals — decide in Phase 4 | +| Min SDK | Decide in Phase 1. Suggest API 29 (Android 10). | +| Repo | `sam/kotlin_phone_app_admin` | + +## 10. Rules and constraints + +1. **Admin actions need the keycode or biometric.** Never a spoken keycode alone. +2. The app **never** holds control logic. It calls the service. +3. The Hub token lives in encrypted storage. Never in logs. +4. The app never claims success on a failed call. +5. Models go through **OmniRoute** — but this app talks to the Hub, not to models. +6. Own folder, own repo. + +## 11. Build order + +**Phase 1 — Skeleton and settings** +Empty app, settings screen, Hub URL + token, test connection. +*DoD:* the app reaches the Hub and shows its health. + +**Phase 2 — Voice in** +Hold-to-talk, send to the Hub, show the text reply. +*DoD:* a spoken question returns a Hub answer on screen. + +**Phase 3 — Voice out** +Play the reply as TTS on the phone. +*DoD:* the phone speaks the Hub's reply. + +**Phase 4 — Approvals** +Fetch pending approvals, show the card, approve or deny. +*DoD:* a real "did you ask for this?" from the Hub is answered from the phone. + +**Phase 5 — Admin** +List kids' devices via the internet control API. Lock, unlock, grant. +*DoD:* a real device locks in under one second from the app. + +**Phase 6 — Security gate** +Keycode or biometric before admin actions. +*DoD:* admin actions are blocked without the gate. + +**Phase 7 — Notifications and deep links** +Hub `/notify` as notifications. Native actions and deep links. +*DoD:* a Hub notification arrives, and a deep link opens the right song. + +**Phase 8 — Wake word (optional)** +Replace hold-to-talk with a wake word. +*DoD:* the wake word starts listening without touching the phone. + +**Phase 9 — Documentation** +*DoD:* project-docs run; docs updated. + +## 12. Acceptance criteria + +- Sam asks a question by voice and hears the answer on the phone. +- An approval prompt can be answered from the phone. +- A kids' device can be locked from the app in under one second. +- Admin actions require the keycode or biometric. +- A Hub notification arrives on the phone. +- The app never reports success after a failure. + +## 13. Decisions and remaining questions + +Resolved: +1. **One app for Sam** (Hub + Admin). The kids' client is under `internet_use_control`. +2. **Kotlin**, Android. +3. Admin calls the internet control API directly. No Hub in that path. + +Remaining: +1. **Hold-to-talk first, wake word later** — agree? +2. How does the app reach the Hub: LAN only, or Tailscale when away? +3. Approvals by FCM push, or polling? +4. Locked-screen behaviour: can approvals be answered from the lock screen? +5. Min SDK version. diff --git a/GOOGLE_NOTES.md b/GOOGLE_NOTES.md new file mode 100644 index 0000000..7cddb6b --- /dev/null +++ b/GOOGLE_NOTES.md @@ -0,0 +1,61 @@ + +> **Editor's note (2026-10-10).** This note describes the Kotlin "Jeeves" phone +> app. It moved here from `artemis_control`, because it belongs to the phone app, +> not to the screen-control layer. +> +> **What still holds:** +> - The phone app records your voice and sends it to the home server. +> - The server decides what to do, and sends instructions back. +> - Native phone actions (contacts, dialing, audio) run in the app. +> +> **What changed:** +> - The "Home AI Server" is now **the Hub** (`central_ai_llm_master`). +> - **Artemis is separate.** It controls the screen over **ADB + Tailscale** from +> `.13`. It does **not** go through this app. See `../artemis_control/`. +> - The Hub calls Artemis **directly**. The phone app is only a way **in**. + +🌐 High-Level System Architecture Flow + +[🗣️ Your Voice] ──> (📱 Jeeves App) ──> [🏠 Home AI Server (LLM)] + │ + ┌───────────────────────────────┴───────────────────────────────┐ + ▼ [If Native Phone Feature] ▼ [If Screen Control] + (📱 Jeeves App) (🤖 ARTEMIS CONTROL) + │ │ + [⚡ Native Kotlin Execution] [🖱️ ADB input over Tailscale] + (Contacts, Phone Calls, Mic Audio) (Spotify, Maps, other apps) + + NOTE: The two right-hand branches are now SEPARATE projects. + Jeeves = kotlin_phone_app_admin. Artemis = artemis_control. + +🔄 Step-by-Step Execution Sequence + + +1. Input & Intent Catching + +• Action: You speak into your phone: "Find Sarah and call her." +• Process: Your custom Jeeves Kotlin App converts the speech to text and instantly forwards it to your Home AI Server over your local network or Tailscale tunnel. + +2. The LLM Brain Check + +• Process: Your Home LLM interprets the request. It identifies the intent and figures out the best way to execute it: + • Option A (Native System Tasks): If it requires basic phone features (Contacts, Dialing), it builds a Code Function Hook to send back to your app. + • Option B (Third-Party Tasks): If it requires a closed app interface (Telegram, Spotify), it sends instructions to Google ARTEMIS. + +🔀 The Split Execution Paths + + +Path A: Handling Android Native Tasks (Contacts & Calling) + +1. The Hook-Back: The Home LLM Server sends a quick JSON payload back to your running phone app:json +{ "action": "QUERY_CONTACT", "target": "Sarah" } +Use code with caution. +2. Local Lookup: The Jeeves App intercepts this and uses Android's native database hooks to find Sarah's number (+1-555-0199) in milliseconds. +3. The Call Setup: The app automatically triggers a phone call using native Android system commands without touching the screen. +4. The Audio Loop: Your app records your microphone audio, passes it to your server for real-time processing, receives the generated MiniMax TTS audio file back, and plays it instantly over your phone's speakerphone. + +Path B: Handling Third-Party Apps (Telegram, Spotify, etc.) + +1. The Hand-Off: Your Home LLM Server realizes it cannot natively inject data into a closed application. It kicks the instruction over to Google ARTEMIS running on your Linux host. +2. The Remote Bridge: ARTEMIS connects to your phone over the air via the encrypted Tailscale wire. +3. Visual Interaction: ARTEMIS physically wakes up your phone screen, launches Telegram or Spotify, types out text strings, and clicks the layout buttons live in front of your eyes like a human operator. diff --git a/ORIGINAL_NOTES.md b/ORIGINAL_NOTES.md new file mode 100644 index 0000000..bbd46a0 --- /dev/null +++ b/ORIGINAL_NOTES.md @@ -0,0 +1,14 @@ +# Kotlin Phone App Admin — Original Notes (archived) + +> Historical pre-planning notes. **Superseded by `AGENTS.md`.** Do not build from this. + + +> Kept for provenance. Do not build from this. + +The phone-app architecture is described in **`GOOGLE_NOTES.md`** in this folder, +with an editor's note marking what still holds. In short: the phone records voice +and sends it to the home server (now the Hub); the server decides; native phone +actions run in the app; and **Artemis is separate** (ADB over Tailscale), not part +of this app. + +This folder was created by splitting `voice_artemis_minimax` on 2026-10-10.