118 lines
7.9 KiB
Markdown
118 lines
7.9 KiB
Markdown
# Jervis Voice Assistant — System Overview
|
|
|
|
> Single source of truth for the local voice-assistant system on Sam's home lab.
|
|
> Companion docs: **`agent-api.md`** (how to send a command / get a reply) and **`clients.md`** (type-it, web, Android gateways).
|
|
> Component repos: `voice_assistant` (ESP32 firmware), `voice_bridge` (STT bridge), `voice_agent` (this brain), `speech_piper` (TTS).
|
|
|
|
---
|
|
|
|
## What it does
|
|
|
|
A fully-local, private voice assistant. You say **"Hey Jervis"**, then a natural-language request; it answers, controls the house, manages your shopping list, and speaks back through your speakers. The **design principle: one LLM "brain"** decides what to do, and calls tools. Home Assistant is *one of those tools* — not the decision maker.
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─ Input ───────────────────────────────────────────────────────────────┐
|
|
│ ESP32-S3 ("Hey Jervis" wake word, INMP441 mic, WS2812 LED ring) │
|
|
│ └→ MQTT voice/audio_stream (broker .13:1883) │
|
|
│ voice_bridge (docker .13) — wake word, buffer audio │
|
|
│ └→ HTTP :5000 │
|
|
│ voice_whisper (docker .13, faster-whisper) — STT audio→text │
|
|
│ └→ publish text → MQTT homeassistant/voice/text (broker .30) │
|
|
│ ALSO: any client → POST http://.13:8501/voice {text} (see clients.md)│
|
|
└───────────────────────────────────────────────────────────────────────┘
|
|
│ text
|
|
▼
|
|
┌─ Brain ── voice-agent (systemd service on .13:8501) ─────────────────┐
|
|
│ LLM = OmniRoute (OpenAI-compatible, tool-calling) │
|
|
│ Model: auto/best-fast │
|
|
│ LLM decides intent → calls a TOOL: │
|
|
│ • get_time / get_weather / answer (Open-Meteo, LLM) │
|
|
│ • add_shopping_item → KitchenOwl API (shopping list) │
|
|
│ • control_ha / ha_service_cmd → Home Assistant REST API │
|
|
│ • feed_fish → HA input_button.fish_feeder │
|
|
│ • set_reminder → HA persistent_notification │
|
|
│ • speak_as(voice,text) → piper per-character voice │
|
|
│ • play_music → Mopidy (Spotify, WORKING) │
|
|
│ • pause/stop/next → Mopidy MPD (planned) │
|
|
└───────────────────────────────────────────────────────────────────────┘
|
|
│ reply text
|
|
▼
|
|
┌─ Output ─────────────────────────────────────────────────────────────┐
|
|
│ piper_tts (docker .13) → speak_direct.sh (pure-python resampler snapcast_send.py) → Snapcast → speakers │
|
|
└───────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Key design decision:** the agent is the single brain; **Home Assistant is a passive device the agent commands via its REST API** — no per-action HA automation/subscription is needed for the agent's commands (one long-lived HA token enables all HA control).
|
|
|
|
---
|
|
|
|
## Components (`host/IP/port/repo`)
|
|
|
|
| Component | Host/IP | Port | Repo / path | Role |
|
|
|---|---|---|---|---|
|
|
| ESP32-S3 | — | — | `sam/voice_assistant` | Wake word, mic, LED |
|
|
| voice_bridge | `.13` | docker | `sam/voice_bridge` | Wake word + audio buffer |
|
|
| voice_whisper | `.13` | `:5000` | `sam/voice_bridge` | faster-whisper STT |
|
|
| **voice-agent** | `.13` | **`:8501`** | `sam/voice-assistant` | LLM brain + tools |
|
|
| OmniRoute | `.13` | `:20129` (API) | — | LLM gateway (OpenAI-compatible) |
|
|
| piper_tts | `.13` | `:10200` | `sam/speech_piper` | TTS (Wyoming) |
|
|
| Snapcast | `.13` | `:1780` | — | Multi-room audio |
|
|
| Mopidy / librespot | `.13` | `:6600`/`:6680` | — | Music: Spotify via gst-plugin-spotify + device-auth credentials |
|
|
| Home Assistant | `.30` | `:8123` (REST) | — | Home control |
|
|
| KitchenOwl | `.35` | `owl.lab.audasmedia.com.au` | — | Shopping list |
|
|
| mosquitto | `.13` + `.30` | `:1883` | — | MQTT brokers |
|
|
|
|
---
|
|
|
|
## Data flow summary
|
|
|
|
1. ESP32 streams audio to **voice_bridge** (MQTT on `.13`).
|
|
2. **voice_bridge** detects the wake word, buffers, sends to **voice_whisper**.
|
|
3. **voice_whisper** (faster-whisper) returns text.
|
|
4. Text is published to MQTT `homeassistant/voice/text` (broker `.30`) **and/or** POSTed to the agent.
|
|
5. **voice-agent** (LLM) decides intent and calls a tool.
|
|
6. Reply text → **piper** → **Snapcast** → speakers.
|
|
|
|
---
|
|
|
|
## Music playback (Mopidy + Spotify) — working 2026-08-30
|
|
|
|
`play_music` → Mopidy (MPD :6600) → Spotify → Snapcast → speakers.
|
|
|
|
**Two pieces were required (both were the blocker):**
|
|
1. **`gst-plugin-spotify`** — the `spotifyaudiosrc` GStreamer element that decodes `spotify:track` URIs. Added `pkgs.gst_all_1.gst-plugins-rs` to `systemd.services.mopidy.environment.GST_PLUGIN_PATH` in `.13` `configuration.nix` (also isolated Mopidy's GST path to its own 1.28.4 plugins + a void `GST_PLUGIN_SYSTEM_PATH` to avoid the system pipewire plugin poisoning the registry).
|
|
2. **`credentials.json`** — generated via librespot **device-authorization flow** (`get_creds_device`, built from librespot dev branch — has `DeviceAuthClient`). The old PKCE redirect flow (`get_creds`) is **broken** in librespot (deterministic `code_verifier was incorrect`). Device flow prints `spotify.com/pair` + code; user approves on any device. File at `/var/lib/mopidy/spotify/credentials-cache/credentials.json` (mopidy-owned).
|
|
|
|
Also fixed along the way: mopidy-local disabled (empty `library.db` crashed search: `no such table: tracks`); Mopidy-Spotify creds re-authed via mopidy.com (old grant revoked).
|
|
|
|
**librespot (NixOS speaker's Spotify Connect output) is separate and untouched.**
|
|
|
|
## Run / manage
|
|
|
|
- Agent service (persistent, auto-starts): `systemctl status voice-agent` (NixOS systemd unit declared in `/etc/nixos/configuration.nix`). Restart: `sudo systemctl restart voice-agent`.
|
|
- Logs: `journalctl -u voice-agent` / `journalctl -u snapserver`.
|
|
- Firewall: port `8501` is in `allowedTCPPorts` in the NixOS config.
|
|
|
|
## Repository map (other related projects)
|
|
|
|
- `sam/voice_assistant` — ESP32 firmware
|
|
- `sam/voice_bridge` — STT bridge + whisper
|
|
- `sam/speech_piper` — TTS scripts/models → Snapcast
|
|
- `sam/voice-assistant` — **this** overview + API contract + clients
|
|
- `sam/family_home_lab` — console portal / dsh
|
|
---
|
|
|
|
## Diagrams
|
|
|
|
- **Interactive map** (Archify HTML, pan/zoom/search/dark-light): [`voice-assistant-map.html`](./voice-assistant-map.html)
|
|
- Published live: <https://maps.lab.audasmedia.com.au/homeassistant/docs/voice-assistant-map.html>
|
|
- **Mermaid source**: [`voice-assistant.mmd`](./voice-assistant.mmd) · rendered **SVG**: [`diagrams/voice-assistant.svg`](./diagrams/voice-assistant.svg)
|
|
- **Archify JSON IR** (single source of truth for the interactive map): [`archify/voice-assistant.architecture.json`](./archify/voice-assistant.architecture.json)
|
|
|
|
> Regenerate the interactive map from the JSON:
|
|
> `node ~/.agents/skills/archify/bin/archify.mjs deliver architecture docs/archify/voice-assistant.architecture.json docs/voice-assistant-map.html --quality showcase --json`
|