Initial spec: AGENTS.md build spec + archived original notes
This commit is contained in:
36
.gitignore
vendored
Normal file
36
.gitignore
vendored
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
# Secrets
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
*.local
|
||||||
|
*.keystore
|
||||||
|
|
||||||
|
# Go
|
||||||
|
bin/
|
||||||
|
*.exe
|
||||||
|
*.test
|
||||||
|
*.out
|
||||||
|
|
||||||
|
# Python
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
.venv/
|
||||||
|
venv/
|
||||||
|
|
||||||
|
# Node / web
|
||||||
|
node_modules/
|
||||||
|
dist/
|
||||||
|
*.visual-check.*
|
||||||
|
|
||||||
|
# Android / Kotlin
|
||||||
|
.gradle/
|
||||||
|
build/
|
||||||
|
local.properties
|
||||||
|
*.apk
|
||||||
|
|
||||||
|
# OS / editors
|
||||||
|
.DS_Store
|
||||||
|
*.swp
|
||||||
|
|
||||||
|
|
||||||
|
# Pi agent generated memory (local state, not source)
|
||||||
|
.pi/memory-vault/
|
||||||
180
AGENTS.md
Normal file
180
AGENTS.md
Normal file
@@ -0,0 +1,180 @@
|
|||||||
|
# Kotlin Phone App + Admin
|
||||||
|
|
||||||
|
> Build spec for Paperclip. Follow the phases in section 11 in order. Do not skip a
|
||||||
|
> phase's Definition of Done. When something is unclear, ask Sam — do not guess.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Purpose and outcome
|
||||||
|
|
||||||
|
Sam's Android app. It is the **Hub's way in from the phone**, and the **trusted
|
||||||
|
admin surface** for approvals and parental controls.
|
||||||
|
|
||||||
|
**Outcome (the single success statement):**
|
||||||
|
> Sam talks to the Hub from his phone, hears the answer, and can approve or change
|
||||||
|
> anything from the same app.
|
||||||
|
|
||||||
|
## 2. Scope
|
||||||
|
|
||||||
|
**In scope**
|
||||||
|
- Capture voice and send it to the Hub.
|
||||||
|
- Show the Hub's reply: text, spoken audio, and notifications.
|
||||||
|
- Show approval prompts ("Did you ask for this?") and answer them.
|
||||||
|
- Admin controls: parental controls, by calling the internet control service.
|
||||||
|
- Native phone actions the Hub requests (call, open a link, open an app by deep link).
|
||||||
|
|
||||||
|
**Out of scope**
|
||||||
|
- The **kids' phone client.** That stays under `internet_use_control`.
|
||||||
|
- **Artemis** (screen control). Separate project.
|
||||||
|
- The **Hub** service itself.
|
||||||
|
- Understanding speech. The Hub does that.
|
||||||
|
|
||||||
|
**Decision to confirm:** this is **one app with two areas** — Hub (voice) and Admin.
|
||||||
|
Not two apps.
|
||||||
|
|
||||||
|
## 3. Users and roles
|
||||||
|
|
||||||
|
- **Sam only.** This app is not for Jo, Harry, or Finn.
|
||||||
|
- The kids' client is a different app, built under `internet_use_control`.
|
||||||
|
|
||||||
|
## 4. Functional requirements
|
||||||
|
|
||||||
|
| ID | Requirement |
|
||||||
|
|---|---|
|
||||||
|
| FR-1 | Capture voice. Start with **hold-to-talk**; add a wake word later. |
|
||||||
|
| FR-2 | Send the transcript to the Hub (`POST /ask`) and show the reply as text. |
|
||||||
|
| FR-3 | Play the Hub's reply as **TTS on the phone**, when the reply asks for it. |
|
||||||
|
| FR-4 | Show **approval prompts** and return approve or deny to the Hub. |
|
||||||
|
| FR-5 | Admin: list the kids' devices and their state, and lock, unlock, or grant time. |
|
||||||
|
| FR-6 | Admin calls the **internet use control** API directly. The app holds no control logic. |
|
||||||
|
| FR-7 | Gate admin actions behind a **keycode or biometric**. |
|
||||||
|
| FR-8 | Perform native actions the Hub requests: place a call, open a deep link. |
|
||||||
|
| FR-9 | Show Hub `/notify` messages as Android notifications. |
|
||||||
|
| FR-10 | Settings: Hub URL and token. Test the connection. |
|
||||||
|
| FR-11 | Offline: fail cleanly, and never claim a command succeeded. |
|
||||||
|
|
||||||
|
## 5. Screens and UX flow
|
||||||
|
|
||||||
|
**Screens**
|
||||||
|
1. **Home (Hub)** — a large talk button, the last reply, and recent history.
|
||||||
|
2. **Approvals** — pending "Did you ask for this?" items with Approve / Deny.
|
||||||
|
3. **Admin** — the kids' devices, each with its state and quick actions.
|
||||||
|
4. **Settings** — Hub URL, token, keycode, test connection.
|
||||||
|
|
||||||
|
**Main flow**
|
||||||
|
```
|
||||||
|
Sam holds the button and speaks → the Hub replies → the phone speaks it
|
||||||
|
If the Hub needs approval → a card appears → Sam taps Approve
|
||||||
|
If Sam opens Admin → he locks a phone → the internet control service applies it
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. Look and feel
|
||||||
|
|
||||||
|
Reuse the Family Home Lab design system, adapted to Android.
|
||||||
|
|
||||||
|
- Source: `/home/sam/Docker/Containers/family-home-lab/DESIGN.md`
|
||||||
|
- Tokens: `/home/sam/Docker/Containers/family-home-lab/portal/static/tokens.css`
|
||||||
|
|
||||||
|
Map the tokens to the app theme: canvas `#f6f5f4`, one action blue `#0075de`, Inter,
|
||||||
|
8px spacing, 4px inputs, pill buttons, hairline dividers, dark mode. The login/consent
|
||||||
|
moment is the one dark indigo band.
|
||||||
|
|
||||||
|
## 7. Data model
|
||||||
|
|
||||||
|
On the device only. No server database.
|
||||||
|
|
||||||
|
| Store | Holds |
|
||||||
|
|---|---|
|
||||||
|
| Settings | Hub URL, Hub token (encrypted), keycode hash |
|
||||||
|
| Cache | recent replies, pending approvals |
|
||||||
|
| Log | last actions, for debugging |
|
||||||
|
|
||||||
|
## 8. Integrations
|
||||||
|
|
||||||
|
| System | Use |
|
||||||
|
|---|---|
|
||||||
|
| **Hub** | `POST /ask`, approval callbacks, `/notify`. Frozen in `central_ai_llm_master/docs/hub-api.md`. |
|
||||||
|
| **Internet use control** | Admin lock, unlock, grant. Direct API call. See `internet_use_control/SYSTEM_MAP.md`. |
|
||||||
|
| **Android telephony** | Native calls and contacts. |
|
||||||
|
| **Deep links** | Opening a specific song or place. |
|
||||||
|
| **Artemis** | Not called from here. The Hub calls it. Native actions stay in this app. |
|
||||||
|
|
||||||
|
## 9. Technology (locked)
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|---|---|
|
||||||
|
| Language | **Kotlin** (Android) |
|
||||||
|
| Voice | `SpeechRecognizer` first; a wake-word engine later |
|
||||||
|
| Transport | HTTP to the Hub. **LAN, or Tailscale when away** — decide in Phase 1. |
|
||||||
|
| Push | FCM, or polling for approvals — decide in Phase 4 |
|
||||||
|
| Min SDK | Decide in Phase 1. Suggest API 29 (Android 10). |
|
||||||
|
| Repo | `sam/kotlin_phone_app_admin` |
|
||||||
|
|
||||||
|
## 10. Rules and constraints
|
||||||
|
|
||||||
|
1. **Admin actions need the keycode or biometric.** Never a spoken keycode alone.
|
||||||
|
2. The app **never** holds control logic. It calls the service.
|
||||||
|
3. The Hub token lives in encrypted storage. Never in logs.
|
||||||
|
4. The app never claims success on a failed call.
|
||||||
|
5. Models go through **OmniRoute** — but this app talks to the Hub, not to models.
|
||||||
|
6. Own folder, own repo.
|
||||||
|
|
||||||
|
## 11. Build order
|
||||||
|
|
||||||
|
**Phase 1 — Skeleton and settings**
|
||||||
|
Empty app, settings screen, Hub URL + token, test connection.
|
||||||
|
*DoD:* the app reaches the Hub and shows its health.
|
||||||
|
|
||||||
|
**Phase 2 — Voice in**
|
||||||
|
Hold-to-talk, send to the Hub, show the text reply.
|
||||||
|
*DoD:* a spoken question returns a Hub answer on screen.
|
||||||
|
|
||||||
|
**Phase 3 — Voice out**
|
||||||
|
Play the reply as TTS on the phone.
|
||||||
|
*DoD:* the phone speaks the Hub's reply.
|
||||||
|
|
||||||
|
**Phase 4 — Approvals**
|
||||||
|
Fetch pending approvals, show the card, approve or deny.
|
||||||
|
*DoD:* a real "did you ask for this?" from the Hub is answered from the phone.
|
||||||
|
|
||||||
|
**Phase 5 — Admin**
|
||||||
|
List kids' devices via the internet control API. Lock, unlock, grant.
|
||||||
|
*DoD:* a real device locks in under one second from the app.
|
||||||
|
|
||||||
|
**Phase 6 — Security gate**
|
||||||
|
Keycode or biometric before admin actions.
|
||||||
|
*DoD:* admin actions are blocked without the gate.
|
||||||
|
|
||||||
|
**Phase 7 — Notifications and deep links**
|
||||||
|
Hub `/notify` as notifications. Native actions and deep links.
|
||||||
|
*DoD:* a Hub notification arrives, and a deep link opens the right song.
|
||||||
|
|
||||||
|
**Phase 8 — Wake word (optional)**
|
||||||
|
Replace hold-to-talk with a wake word.
|
||||||
|
*DoD:* the wake word starts listening without touching the phone.
|
||||||
|
|
||||||
|
**Phase 9 — Documentation**
|
||||||
|
*DoD:* project-docs run; docs updated.
|
||||||
|
|
||||||
|
## 12. Acceptance criteria
|
||||||
|
|
||||||
|
- Sam asks a question by voice and hears the answer on the phone.
|
||||||
|
- An approval prompt can be answered from the phone.
|
||||||
|
- A kids' device can be locked from the app in under one second.
|
||||||
|
- Admin actions require the keycode or biometric.
|
||||||
|
- A Hub notification arrives on the phone.
|
||||||
|
- The app never reports success after a failure.
|
||||||
|
|
||||||
|
## 13. Decisions and remaining questions
|
||||||
|
|
||||||
|
Resolved:
|
||||||
|
1. **One app for Sam** (Hub + Admin). The kids' client is under `internet_use_control`.
|
||||||
|
2. **Kotlin**, Android.
|
||||||
|
3. Admin calls the internet control API directly. No Hub in that path.
|
||||||
|
|
||||||
|
Remaining:
|
||||||
|
1. **Hold-to-talk first, wake word later** — agree?
|
||||||
|
2. How does the app reach the Hub: LAN only, or Tailscale when away?
|
||||||
|
3. Approvals by FCM push, or polling?
|
||||||
|
4. Locked-screen behaviour: can approvals be answered from the lock screen?
|
||||||
|
5. Min SDK version.
|
||||||
61
GOOGLE_NOTES.md
Normal file
61
GOOGLE_NOTES.md
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
|
||||||
|
> **Editor's note (2026-10-10).** This note describes the Kotlin "Jeeves" phone
|
||||||
|
> app. It moved here from `artemis_control`, because it belongs to the phone app,
|
||||||
|
> not to the screen-control layer.
|
||||||
|
>
|
||||||
|
> **What still holds:**
|
||||||
|
> - The phone app records your voice and sends it to the home server.
|
||||||
|
> - The server decides what to do, and sends instructions back.
|
||||||
|
> - Native phone actions (contacts, dialing, audio) run in the app.
|
||||||
|
>
|
||||||
|
> **What changed:**
|
||||||
|
> - The "Home AI Server" is now **the Hub** (`central_ai_llm_master`).
|
||||||
|
> - **Artemis is separate.** It controls the screen over **ADB + Tailscale** from
|
||||||
|
> `.13`. It does **not** go through this app. See `../artemis_control/`.
|
||||||
|
> - The Hub calls Artemis **directly**. The phone app is only a way **in**.
|
||||||
|
|
||||||
|
🌐 High-Level System Architecture Flow
|
||||||
|
|
||||||
|
[🗣️ Your Voice] ──> (📱 Jeeves App) ──> [🏠 Home AI Server (LLM)]
|
||||||
|
│
|
||||||
|
┌───────────────────────────────┴───────────────────────────────┐
|
||||||
|
▼ [If Native Phone Feature] ▼ [If Screen Control]
|
||||||
|
(📱 Jeeves App) (🤖 ARTEMIS CONTROL)
|
||||||
|
│ │
|
||||||
|
[⚡ Native Kotlin Execution] [🖱️ ADB input over Tailscale]
|
||||||
|
(Contacts, Phone Calls, Mic Audio) (Spotify, Maps, other apps)
|
||||||
|
|
||||||
|
NOTE: The two right-hand branches are now SEPARATE projects.
|
||||||
|
Jeeves = kotlin_phone_app_admin. Artemis = artemis_control.
|
||||||
|
|
||||||
|
🔄 Step-by-Step Execution Sequence
|
||||||
|
|
||||||
|
|
||||||
|
1. Input & Intent Catching
|
||||||
|
|
||||||
|
• Action: You speak into your phone: "Find Sarah and call her."
|
||||||
|
• Process: Your custom Jeeves Kotlin App converts the speech to text and instantly forwards it to your Home AI Server over your local network or Tailscale tunnel.
|
||||||
|
|
||||||
|
2. The LLM Brain Check
|
||||||
|
|
||||||
|
• Process: Your Home LLM interprets the request. It identifies the intent and figures out the best way to execute it:
|
||||||
|
• Option A (Native System Tasks): If it requires basic phone features (Contacts, Dialing), it builds a Code Function Hook to send back to your app.
|
||||||
|
• Option B (Third-Party Tasks): If it requires a closed app interface (Telegram, Spotify), it sends instructions to Google ARTEMIS.
|
||||||
|
|
||||||
|
🔀 The Split Execution Paths
|
||||||
|
|
||||||
|
|
||||||
|
Path A: Handling Android Native Tasks (Contacts & Calling)
|
||||||
|
|
||||||
|
1. The Hook-Back: The Home LLM Server sends a quick JSON payload back to your running phone app:json
|
||||||
|
{ "action": "QUERY_CONTACT", "target": "Sarah" }
|
||||||
|
Use code with caution.
|
||||||
|
2. Local Lookup: The Jeeves App intercepts this and uses Android's native database hooks to find Sarah's number (+1-555-0199) in milliseconds.
|
||||||
|
3. The Call Setup: The app automatically triggers a phone call using native Android system commands without touching the screen.
|
||||||
|
4. The Audio Loop: Your app records your microphone audio, passes it to your server for real-time processing, receives the generated MiniMax TTS audio file back, and plays it instantly over your phone's speakerphone.
|
||||||
|
|
||||||
|
Path B: Handling Third-Party Apps (Telegram, Spotify, etc.)
|
||||||
|
|
||||||
|
1. The Hand-Off: Your Home LLM Server realizes it cannot natively inject data into a closed application. It kicks the instruction over to Google ARTEMIS running on your Linux host.
|
||||||
|
2. The Remote Bridge: ARTEMIS connects to your phone over the air via the encrypted Tailscale wire.
|
||||||
|
3. Visual Interaction: ARTEMIS physically wakes up your phone screen, launches Telegram or Spotify, types out text strings, and clicks the layout buttons live in front of your eyes like a human operator.
|
||||||
14
ORIGINAL_NOTES.md
Normal file
14
ORIGINAL_NOTES.md
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# Kotlin Phone App Admin — Original Notes (archived)
|
||||||
|
|
||||||
|
> Historical pre-planning notes. **Superseded by `AGENTS.md`.** Do not build from this.
|
||||||
|
|
||||||
|
|
||||||
|
> Kept for provenance. Do not build from this.
|
||||||
|
|
||||||
|
The phone-app architecture is described in **`GOOGLE_NOTES.md`** in this folder,
|
||||||
|
with an editor's note marking what still holds. In short: the phone records voice
|
||||||
|
and sends it to the home server (now the Hub); the server decides; native phone
|
||||||
|
actions run in the app; and **Artemis is separate** (ADB over Tailscale), not part
|
||||||
|
of this app.
|
||||||
|
|
||||||
|
This folder was created by splitting `voice_artemis_minimax` on 2026-10-10.
|
||||||
Reference in New Issue
Block a user