From 4b86c18622426319d646f9b5609fa073f5174ef3 Mon Sep 17 00:00:00 2001 From: sam Date: Sun, 30 Aug 2026 14:42:11 +1000 Subject: [PATCH] Add overview, agent-api, clients docs for Jervis voice agent --- .gitignore | 3 ++ README.md | 29 +++++++++++++++ docs/agent-api.md | 85 +++++++++++++++++++++++++++++++++++++++++++ docs/clients.md | 73 +++++++++++++++++++++++++++++++++++++ docs/overview.md | 93 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 283 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 docs/agent-api.md create mode 100644 docs/clients.md create mode 100644 docs/overview.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..1f1dfe6 --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +# secrets +.env +*.env \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..489cb89 --- /dev/null +++ b/README.md @@ -0,0 +1,29 @@ +# voice-assistant — Jervis Voice Assistant + +A fully-local, private voice assistant. Say **"Hey Jervis"** and make a request; it answers, +controls your house, manages your shopping list, and speaks back through your speakers — +driven by a single LLM "brain" that calls tools. + +## Docs + +- **[`docs/overview.md`](docs/overview.md)** — the whole architecture in one page (diagram, hosts/IPs/ports, component map). +- **[`docs/agent-api.md`](docs/agent-api.md)** — the command contract: send text, get a spoken reply. Use from any AI model, script, N8N, ESP32, or web app. +- **[`docs/clients.md`](docs/clients.md)** — type-it, web page, and Android gateways. + +## Quick use + +```bash +# speak a command (reply is spoken through speakers too) +curl -X POST http://192.168.20.13:8501/voice \ + -H "Content-Type: application/json" \ + -d '{"text":"add chicken to the shopping list"}' +``` + +## Related repos + +- `sam/voice_assistant` — ESP32 firmware (wake word, mic) +- `sam/voice_bridge` — STT bridge + whisper +- `sam/speech_piper` — TTS → Snapcast +- `sam/voice-assistant` — **this** overview + API + clients + +> Secrets live in `.env` (gitignored) — never commit them. \ No newline at end of file diff --git a/docs/agent-api.md b/docs/agent-api.md new file mode 100644 index 0000000..7dc9103 --- /dev/null +++ b/docs/agent-api.md @@ -0,0 +1,85 @@ +# Voice-Agent Command API + +> The stable contract for sending a command to the Jervis LLM service and getting a spoken reply. +> Works for **any** AI model, script, N8N workflow, ESP32, web page, or Android app — not just Home Assistant. + +--- + +## Quick start + +**Send a command (HTTP):** +```bash +curl -X POST http://192.168.20.13:8501/voice \ + -H "Content-Type: application/json" \ + -d '{"text":"add chicken to the shopping list"}' +# → {"text":"add chicken to the shopping list","reply":"Done, I've added chicken to your shopping list."} +``` + +The reply is **also spoken** through your speakers via Snapcast. + +**Send a command (MQTT):** +```bash +mosquitto_pub -h 192.168.20.30 -u mqtt-user -P -t homeassistant/voice/text -m "what time is it" +``` +Same effect — the agent picks up the topic and speaks the reply. (Use whichever transport is easier for the caller.) + +--- + +## Endpoint + +| Method | Path | Body | Returns | +|---|---|---|---| +| POST | `/voice` | `{"text": ""}` | `{"text": ..., "reply": ...}` | +| GET | `/health` | — | `{"status":"ok","model":"..."}` | + +- **Base URL:** `http://192.168.20.13:8501` +- **Optional:** `{"message": "..."}` can be used instead of `{"text": "..."}`. + +--- + +## What the LLM can do (tools it will call) + +| Example command | Tool | Backend | +|---|---|---| +| "what time is it" | `get_time` | built-in | +| "what's the weather" | `get_weather` | Open-Meteo | +| "add milk to the shopping list" | `add_shopping_item` | KitchenOwl | +| "turn on the lounge lights" | `control_ha` | Home Assistant REST | +| "remind me to water the plants" | `set_reminder` | HA notification | +| "feed the fish" | `feed_fish` | HA `input_button.fish_feeder` | +| "put on your Donald Trump voice and tell me a joke" | `speak_as` | piper (per-character) | +| "send {topic}/{payload}" (advanced) | `ha_service_cmd` | HA any service | +| any open question | `answer` | OmniRoute LLM | +| "play {artist/album}" | `play_music` | Mopidy *(planned)* | + +The LLM picks the tool automatically from natural language — you don't pick it. + +--- + +## Voices (for `speak_as`) + +`trump`, `donald`, `hal`, `hal 9000`, `picard`, `captain`, `eminem`, `glados`, `ryan`, `alan`, `bt`/`bt7274`, `mech`. + +--- + +## Notes for other AI agents + +- The same text can be sent by any model/caller — the agent does the reasoning + tool dispatch + speaking. +- For non-voice callers (N8N, scripts) that just want the **reply text without speaking**, the agent currently speaks by design (voice assistant). A `?speak=0` / `{"speak": false}` option can be added if needed. +- Timeouts: LLM tool loops are bounded (max 5). Expect a few seconds for tool calls. + +--- + +## Config / env (`/home/sam/voice-agent/.env`) + +| Var | Meaning | +|---|---| +| `LLM_BASE`, `LLM_MODEL`, `LLM_KEY` | OmniRoute endpoint + model | +| `HA_URL`, `HA_TOKEN` | Home Assistant REST + long-lived token | +| `KITCHENOWL_URL/_HOUSEHOLD/_LIST/_USER/_PASS` | KitchenOwl shopping write | +| `MQTT_BROKER/_TOPIC/_USER/_PASS` | text intake + output | +| `SPEAK_CMD`, `DEFAULT_VOICE` | piper TTS command/voice | +| `LAT`, `LON` | weather location | +| `AGENT_PORT` | HTTP port (`8501`) | + +Secrets live in `.env` (gitignored) — never commit them. \ No newline at end of file diff --git a/docs/clients.md b/docs/clients.md new file mode 100644 index 0000000..257a9d5 --- /dev/null +++ b/docs/clients.md @@ -0,0 +1,73 @@ +# Clients & Gateways + +> Ways to talk to Jervis — **type it**, a **web page**, or your **Android** — all sending text to the same agent (`POST :8501/voice` or MQTT). You do not need the ESP32 to use the brain; any of these work. + +--- + +## 1. Type it (terminal) + +```bash +curl -X POST http://192.168.20.13:8501/voice \ + -H "Content-Type: application/json" \ + -d '{"text":"add eggs to the shopping list"}' +``` +Alias for convenience (add to `~/.bashrc` / `~/.zshrc`): +```bash +jervis() { curl -s -m60 -X POST http://192.168.20.13:8501/voice -H "Content-Type: application/json" -d "{\"text\":\"$*\"}"; echo; } +# usage: jervis add milk to the shopping list +``` + +--- + +## 2. Web page (type + voice input) + +A single-file HTML page that: +- lets you type a command (POSTs to the agent), and +- uses your browser's **Web Speech API** so you can *speak* into it (works in Chrome/Edge, incl. desktop). + +Save as `voice.html` and open it, or serve it. Example (`speech-to-text` → POST): + +```html +Jervis + +

Jervis

+ + + +
+ +``` +> ⚠️ The page calls the LAN IP directly — fine on your home network. If served off the LAN, add CORS to the agent and use a LAN-only endpoint or Tunnel/WireGuard. + +--- + +## 3. Android + +Best options, in order of simplicity: + +- **Browser speech** — open the `voice.html` above on your phone and tap 🎤 (works in Chrome on Android; uses your phone's speech recognition). **Zero install.** +- **Home Assistant Companion (mobile app)** — you already have HA. Configure an `assist`/text pipeline that forwards to the agent's reply (HA already consumes the same MQTT topic the agent uses). This ties into the app you already run. +- **KWGT / Tasker / MacroDroid** — a quick action that captures voice → POSTs to `:8501/voice` (Tasker "HTTP Post" with your chosen TTS engine). + +All three just send text to the same endpoint — they don't care about the ESP32. + +--- + +## Other callers + +- **N8N / Prefect** — HTTP node → `POST :8501/voice` (or MQTT publish) to trigger + hear. +- **MQTT** — publish to `homeassistant/voice/text` on broker `.30`; the agent speaks the reply. +- **Another AI model / assistant** — call `POST :8501/voice` like any function (see `agent-api.md`). + +--- + +## Planning notes (not yet implemented) + +- Per-person Apprise notifications (sam/jo/harry/finn) — `send_message` tool (planned). +- Calendar read via N8N webhook returning the day's events — `calendar` tool (planned). +- Native Android app / always-listening mic — future. \ No newline at end of file diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..432df24 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,93 @@ +# Jervis Voice Assistant — System Overview + +> Single source of truth for the local voice-assistant system on Sam's home lab. +> Companion docs: **`agent-api.md`** (how to send a command / get a reply) and **`clients.md`** (type-it, web, Android gateways). +> Component repos: `voice_assistant` (ESP32 firmware), `voice_bridge` (STT bridge), `voice_agent` (this brain), `speech_piper` (TTS). + +--- + +## What it does + +A fully-local, private voice assistant. You say **"Hey Jervis"**, then a natural-language request; it answers, controls the house, manages your shopping list, and speaks back through your speakers. The **design principle: one LLM "brain"** decides what to do, and calls tools. Home Assistant is *one of those tools* — not the decision maker. + +--- + +## Architecture + +``` +┌─ Input ───────────────────────────────────────────────────────────────┐ +│ ESP32-S3 ("Hey Jervis" wake word, INMP441 mic, WS2812 LED ring) │ +│ └→ MQTT voice/audio_stream (broker .13:1883) │ +│ voice_bridge (docker .13) — wake word, buffer audio │ +│ └→ HTTP :5000 │ +│ voice_whisper (docker .13, faster-whisper) — STT audio→text │ +│ └→ publish text → MQTT homeassistant/voice/text (broker .30) │ +│ ALSO: any client → POST http://.13:8501/voice {text} (see clients.md)│ +└───────────────────────────────────────────────────────────────────────┘ + │ text + ▼ +┌─ Brain ── voice-agent (systemd service on .13:8501) ─────────────────┐ +│ LLM = OmniRoute (OpenAI-compatible, tool-calling) │ +│ Model: auto/best-fast │ +│ LLM decides intent → calls a TOOL: │ +│ • get_time / get_weather / answer (Open-Meteo, LLM) │ +│ • add_shopping_item → KitchenOwl API (shopping list) │ +│ • control_ha / ha_service_cmd → Home Assistant REST API │ +│ • feed_fish → HA input_button.fish_feeder │ +│ • set_reminder → HA persistent_notification │ +│ • speak_as(voice,text) → piper per-character voice │ +│ • play_music → Mopidy (planned) │ +└───────────────────────────────────────────────────────────────────────┘ + │ reply text + ▼ +┌─ Output ─────────────────────────────────────────────────────────────┐ +│ piper_tts (docker .13) → speak_direct.sh → Snapcast → all speakers │ +└───────────────────────────────────────────────────────────────────────┘ +``` + +**Key design decision:** the agent is the single brain; **Home Assistant is a passive device the agent commands via its REST API** — no per-action HA automation/subscription is needed for the agent's commands (one long-lived HA token enables all HA control). + +--- + +## Components (`host/IP/port/repo`) + +| Component | Host/IP | Port | Repo / path | Role | +|---|---|---|---|---| +| ESP32-S3 | — | — | `sam/voice_assistant` | Wake word, mic, LED | +| voice_bridge | `.13` | docker | `sam/voice_bridge` | Wake word + audio buffer | +| voice_whisper | `.13` | `:5000` | `sam/voice_bridge` | faster-whisper STT | +| **voice-agent** | `.13` | **`:8501`** | `sam/voice-assistant` | LLM brain + tools | +| OmniRoute | `.13` | `:20129` (API) | — | LLM gateway (OpenAI-compatible) | +| piper_tts | `.13` | `:10200` | `sam/speech_piper` | TTS (Wyoming) | +| Snapcast | `.13` | `:1780` | — | Multi-room audio | +| Mopidy / librespot | `.13` | `:6600`/`:6680` | — | Music sources | +| Home Assistant | `.30` | `:8123` (REST) | — | Home control | +| KitchenOwl | `.35` | `owl.lab.audasmedia.com.au` | — | Shopping list | +| mosquitto | `.13` + `.30` | `:1883` | — | MQTT brokers | + +--- + +## Data flow summary + +1. ESP32 streams audio to **voice_bridge** (MQTT on `.13`). +2. **voice_bridge** detects the wake word, buffers, sends to **voice_whisper**. +3. **voice_whisper** (faster-whisper) returns text. +4. Text is published to MQTT `homeassistant/voice/text` (broker `.30`) **and/or** POSTed to the agent. +5. **voice-agent** (LLM) decides intent and calls a tool. +6. Reply text → **piper** → **Snapcast** → speakers. + +--- + +## Run / manage + +- Agent service (persistent, auto-starts): `systemctl status voice-agent` (NixOS systemd unit declared in `/etc/nixos/configuration.nix`). Restart: `sudo systemctl restart voice-agent`. +- Logs: `journalctl -u voice-agent` / `journalctl -u snapserver`. +- Firewall: port `8501` is in `allowedTCPPorts` in the NixOS config. + +## Repository map (other related projects) + +- `sam/voice_assistant` — ESP32 firmware +- `sam/voice_bridge` — STT bridge + whisper +- `sam/speech_piper` — TTS scripts/models → Snapcast +- `sam/voice-assistant` — **this** overview + API contract + clients +- `sam/family_home_lab` — console portal / dsh \ No newline at end of file