Add overview, agent-api, clients docs for Jervis voice agent

2026-08-30 14:42:11 +10:00
commit 4b86c18622
5 changed files with 283 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -0,0 +1,3 @@
+# secrets
+.env
+*.env
--- a/README.md
+++ b/README.md
@@ -0,0 +1,29 @@
+# voice-assistant — Jervis Voice Assistant
+
+A fully-local, private voice assistant. Say **"Hey Jervis"** and make a request; it answers,
+controls your house, manages your shopping list, and speaks back through your speakers —
+driven by a single LLM "brain" that calls tools.
+
+## Docs
+
+- **[`docs/overview.md`](docs/overview.md)** — the whole architecture in one page (diagram, hosts/IPs/ports, component map).
+- **[`docs/agent-api.md`](docs/agent-api.md)** — the command contract: send text, get a spoken reply. Use from any AI model, script, N8N, ESP32, or web app.
+- **[`docs/clients.md`](docs/clients.md)** — type-it, web page, and Android gateways.
+
+## Quick use
+
+```bash
+# speak a command (reply is spoken through speakers too)
+curl -X POST http://192.168.20.13:8501/voice \
+  -H "Content-Type: application/json" \
+  -d '{"text":"add chicken to the shopping list"}'
+```
+
+## Related repos
+
+- `sam/voice_assistant` — ESP32 firmware (wake word, mic)
+- `sam/voice_bridge` — STT bridge + whisper
+- `sam/speech_piper` — TTS → Snapcast
+- `sam/voice-assistant` — **this** overview + API + clients
+
+> Secrets live in `.env` (gitignored) — never commit them.
--- a/docs/agent-api.md
+++ b/docs/agent-api.md
@@ -0,0 +1,85 @@
+# Voice-Agent Command API
+
+> The stable contract for sending a command to the Jervis LLM service and getting a spoken reply.
+> Works for **any** AI model, script, N8N workflow, ESP32, web page, or Android app — not just Home Assistant.
+
+---
+
+## Quick start
+
+**Send a command (HTTP):**
+```bash
+curl -X POST http://192.168.20.13:8501/voice \
+  -H "Content-Type: application/json" \
+  -d '{"text":"add chicken to the shopping list"}'
+# → {"text":"add chicken to the shopping list","reply":"Done, I've added chicken to your shopping list."}
+```
+
+The reply is **also spoken** through your speakers via Snapcast.
+
+**Send a command (MQTT):**
+```bash
+mosquitto_pub -h 192.168.20.30 -u mqtt-user -P <pass> -t homeassistant/voice/text -m "what time is it"
+```
+Same effect — the agent picks up the topic and speaks the reply. (Use whichever transport is easier for the caller.)
+
+---
+
+## Endpoint
+
+| Method | Path | Body | Returns |
+|---|---|---|---|
+| POST | `/voice` | `{"text": "<command>"}` | `{"text": ..., "reply": ...}` |
+| GET  | `/health` | — | `{"status":"ok","model":"..."}` |
+
+- **Base URL:** `http://192.168.20.13:8501`
+- **Optional:** `{"message": "..."}` can be used instead of `{"text": "..."}`.
+
+---
+
+## What the LLM can do (tools it will call)
+
+| Example command | Tool | Backend |
+|---|---|---|
+| "what time is it" | `get_time` | built-in |
+| "what's the weather" | `get_weather` | Open-Meteo |
+| "add milk to the shopping list" | `add_shopping_item` | KitchenOwl |
+| "turn on the lounge lights" | `control_ha` | Home Assistant REST |
+| "remind me to water the plants" | `set_reminder` | HA notification |
+| "feed the fish" | `feed_fish` | HA `input_button.fish_feeder` |
+| "put on your Donald Trump voice and tell me a joke" | `speak_as` | piper (per-character) |
+| "send {topic}/{payload}" (advanced) | `ha_service_cmd` | HA any service |
+| any open question | `answer` | OmniRoute LLM |
+| "play {artist/album}" | `play_music` | Mopidy *(planned)* |
+
+The LLM picks the tool automatically from natural language — you don't pick it.
+
+---
+
+## Voices (for `speak_as`)
+
+`trump`, `donald`, `hal`, `hal 9000`, `picard`, `captain`, `eminem`, `glados`, `ryan`, `alan`, `bt`/`bt7274`, `mech`.
+
+---
+
+## Notes for other AI agents
+
+- The same text can be sent by any model/caller — the agent does the reasoning + tool dispatch + speaking.
+- For non-voice callers (N8N, scripts) that just want the **reply text without speaking**, the agent currently speaks by design (voice assistant). A `?speak=0` / `{"speak": false}` option can be added if needed.
+- Timeouts: LLM tool loops are bounded (max 5). Expect a few seconds for tool calls.
+
+---
+
+## Config / env (`/home/sam/voice-agent/.env`)
+
+| Var | Meaning |
+|---|---|
+| `LLM_BASE`, `LLM_MODEL`, `LLM_KEY` | OmniRoute endpoint + model |
+| `HA_URL`, `HA_TOKEN` | Home Assistant REST + long-lived token |
+| `KITCHENOWL_URL/_HOUSEHOLD/_LIST/_USER/_PASS` | KitchenOwl shopping write |
+| `MQTT_BROKER/_TOPIC/_USER/_PASS` | text intake + output |
+| `SPEAK_CMD`, `DEFAULT_VOICE` | piper TTS command/voice |
+| `LAT`, `LON` | weather location |
+| `AGENT_PORT` | HTTP port (`8501`) |
+
+Secrets live in `.env` (gitignored) — never commit them.
--- a/docs/clients.md
+++ b/docs/clients.md
@@ -0,0 +1,73 @@
+# Clients & Gateways
+
+> Ways to talk to Jervis — **type it**, a **web page**, or your **Android** — all sending text to the same agent (`POST :8501/voice` or MQTT). You do not need the ESP32 to use the brain; any of these work.
+
+---
+
+## 1. Type it (terminal)
+
+```bash
+curl -X POST http://192.168.20.13:8501/voice \
+  -H "Content-Type: application/json" \
+  -d '{"text":"add eggs to the shopping list"}'
+```
+Alias for convenience (add to `~/.bashrc` / `~/.zshrc`):
+```bash
+jervis() { curl -s -m60 -X POST http://192.168.20.13:8501/voice -H "Content-Type: application/json" -d "{\"text\":\"$*\"}"; echo; }
+# usage: jervis add milk to the shopping list
+```
+
+---
+
+## 2. Web page (type + voice input)
+
+A single-file HTML page that:
+- lets you type a command (POSTs to the agent), and
+- uses your browser's **Web Speech API** so you can *speak* into it (works in Chrome/Edge, incl. desktop).
+
+Save as `voice.html` and open it, or serve it. Example (`speech-to-text` → POST):
+
+```html
+<!doctype html><html><head><meta charset="utf-8"><title>Jervis</title></head>
+<body style="font-family:sans-serif;max-width:640px;margin:2em auto">
+<h1>Jervis</h1>
+<input id="t" style="width:80%;padding:.5em" placeholder="Type a command…">
+<button onclick="send()">Send</button>
+<button onclick="mic()">🎤 Speak</button>
+<div id="r" style="margin-top:1em;white-space:pre-wrap"></div>
+<script>
+const API="http://192.168.20.13:8501/voice";
+async function send(tx){const r=await fetch(API,{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({text:tx})});const j=await r.json();document.getElementById("r").textContent=(j.reply||j.error||"");}
+document.addEventListener("keydown",e=>{if(e.key==="Enter")send(document.getElementById("t").value)});
+function mic(){const S=window.SpeechRecognition||window.webkitSpeechRecognition;if(!S){alert("Speech recognition unsupported in this browser");return;}const sr=new S();sr.lang="en-AU";sr.interimResults=false;sr.onresult=e=>{const t=e.results[0][0].transcript;document.getElementById("t").value=t;send(t)};sr.start();}
+</script></body></html>
+```
+> ⚠️ The page calls the LAN IP directly — fine on your home network. If served off the LAN, add CORS to the agent and use a LAN-only endpoint or Tunnel/WireGuard.
+
+---
+
+## 3. Android
+
+Best options, in order of simplicity:
+
+- **Browser speech** — open the `voice.html` above on your phone and tap 🎤 (works in Chrome on Android; uses your phone's speech recognition). **Zero install.**
+- **Home Assistant Companion (mobile app)** — you already have HA. Configure an `assist`/text pipeline that forwards to the agent's reply (HA already consumes the same MQTT topic the agent uses). This ties into the app you already run.
+- **KWGT / Tasker / MacroDroid** — a quick action that captures voice → POSTs to `:8501/voice` (Tasker "HTTP Post" with your chosen TTS engine).
+
+All three just send text to the same endpoint — they don't care about the ESP32.
+
+---
+
+## Other callers
+
+- **N8N / Prefect** — HTTP node → `POST :8501/voice` (or MQTT publish) to trigger + hear.
+- **MQTT** — publish to `homeassistant/voice/text` on broker `.30`; the agent speaks the reply.
+- **Another AI model / assistant** — call `POST :8501/voice` like any function (see `agent-api.md`).
+
+---
+
+## Planning notes (not yet implemented)
+
+- Per-person Apprise notifications (sam/jo/harry/finn) — `send_message` tool (planned).
+- Calendar read via N8N webhook returning the day's events — `calendar` tool (planned).
+- Native Android app / always-listening mic — future.
--- a/docs/overview.md
+++ b/docs/overview.md
@@ -0,0 +1,93 @@
+# Jervis Voice Assistant — System Overview
+
+> Single source of truth for the local voice-assistant system on Sam's home lab.
+> Companion docs: **`agent-api.md`** (how to send a command / get a reply) and **`clients.md`** (type-it, web, Android gateways).
+> Component repos: `voice_assistant` (ESP32 firmware), `voice_bridge` (STT bridge), `voice_agent` (this brain), `speech_piper` (TTS).
+
+---
+
+## What it does
+
+A fully-local, private voice assistant. You say **"Hey Jervis"**, then a natural-language request; it answers, controls the house, manages your shopping list, and speaks back through your speakers. The **design principle: one LLM "brain"** decides what to do, and calls tools. Home Assistant is *one of those tools* — not the decision maker.
+
+---
+
+## Architecture
+
+```
+┌─ Input ───────────────────────────────────────────────────────────────┐
+│  ESP32-S3  ("Hey Jervis" wake word, INMP441 mic, WS2812 LED ring)     │
+│    └→ MQTT voice/audio_stream  (broker .13:1883)                      │
+│  voice_bridge (docker .13) — wake word, buffer audio                  │
+│    └→ HTTP :5000                                                      │
+│  voice_whisper (docker .13, faster-whisper) — STT audio→text          │
+│    └→ publish text → MQTT homeassistant/voice/text  (broker .30)      │
+│  ALSO: any client → POST http://.13:8501/voice {text}  (see clients.md)│
+└───────────────────────────────────────────────────────────────────────┘
+                          │  text
+                          ▼
+┌─ Brain ── voice-agent (systemd service on .13:8501) ─────────────────┐
+│  LLM = OmniRoute (OpenAI-compatible, tool-calling)                   │
+│  Model: auto/best-fast                                               │
+│  LLM decides intent → calls a TOOL:                                  │
+│    • get_time / get_weather / answer          (Open-Meteo, LLM)      │
+│    • add_shopping_item    → KitchenOwl API   (shopping list)         │
+│    • control_ha / ha_service_cmd  → Home Assistant REST API          │
+│    • feed_fish            → HA input_button.fish_feeder              │
+│    • set_reminder         → HA persistent_notification               │
+│    • speak_as(voice,text) → piper per-character voice                │
+│    • play_music           → Mopidy (planned)                         │
+└───────────────────────────────────────────────────────────────────────┘
+                          │  reply text
+                          ▼
+┌─ Output ─────────────────────────────────────────────────────────────┐
+│  piper_tts (docker .13) → speak_direct.sh → Snapcast → all speakers  │
+└───────────────────────────────────────────────────────────────────────┘
+```
+
+**Key design decision:** the agent is the single brain; **Home Assistant is a passive device the agent commands via its REST API** — no per-action HA automation/subscription is needed for the agent's commands (one long-lived HA token enables all HA control).
+
+---
+
+## Components (`host/IP/port/repo`)
+
+| Component | Host/IP | Port | Repo / path | Role |
+|---|---|---|---|---|
+| ESP32-S3 | — | — | `sam/voice_assistant` | Wake word, mic, LED |
+| voice_bridge | `.13` | docker | `sam/voice_bridge` | Wake word + audio buffer |
+| voice_whisper | `.13` | `:5000` | `sam/voice_bridge` | faster-whisper STT |
+| **voice-agent** | `.13` | **`:8501`** | `sam/voice-assistant` | LLM brain + tools |
+| OmniRoute | `.13` | `:20129` (API) | — | LLM gateway (OpenAI-compatible) |
+| piper_tts | `.13` | `:10200` | `sam/speech_piper` | TTS (Wyoming) |
+| Snapcast | `.13` | `:1780` | — | Multi-room audio |
+| Mopidy / librespot | `.13` | `:6600`/`:6680` | — | Music sources |
+| Home Assistant | `.30` | `:8123` (REST) | — | Home control |
+| KitchenOwl | `.35` | `owl.lab.audasmedia.com.au` | — | Shopping list |
+| mosquitto | `.13` + `.30` | `:1883` | — | MQTT brokers |
+
+---
+
+## Data flow summary
+
+1. ESP32 streams audio to **voice_bridge** (MQTT on `.13`).
+2. **voice_bridge** detects the wake word, buffers, sends to **voice_whisper**.
+3. **voice_whisper** (faster-whisper) returns text.
+4. Text is published to MQTT `homeassistant/voice/text` (broker `.30`) **and/or** POSTed to the agent.
+5. **voice-agent** (LLM) decides intent and calls a tool.
+6. Reply text → **piper** → **Snapcast** → speakers.
+
+---
+
+## Run / manage
+
+- Agent service (persistent, auto-starts): `systemctl status voice-agent` (NixOS systemd unit declared in `/etc/nixos/configuration.nix`). Restart: `sudo systemctl restart voice-agent`.
+- Logs: `journalctl -u voice-agent` / `journalctl -u snapserver`.
+- Firewall: port `8501` is in `allowedTCPPorts` in the NixOS config.
+
+## Repository map (other related projects)
+
+- `sam/voice_assistant` — ESP32 firmware
+- `sam/voice_bridge` — STT bridge + whisper
+- `sam/speech_piper` — TTS scripts/models → Snapcast
+- `sam/voice-assistant` — **this** overview + API contract + clients
+- `sam/family_home_lab` — console portal / dsh