Jervis Voice Assistant — System Overview

Single source of truth for the local voice-assistant system on Sam's home lab. Companion docs: agent-api.md (how to send a command / get a reply) and clients.md (type-it, web, Android gateways). Component repos: voice_assistant (ESP32 firmware), voice_bridge (STT bridge), voice_agent (this brain), speech_piper (TTS).

What it does

A fully-local, private voice assistant. You say "Hey Jervis", then a natural-language request; it answers, controls the house, manages your shopping list, and speaks back through your speakers. The design principle: one LLM "brain" decides what to do, and calls tools. Home Assistant is one of those tools — not the decision maker.

Architecture

┌─ Input ───────────────────────────────────────────────────────────────┐
│  ESP32-S3  ("Hey Jervis" wake word, INMP441 mic, WS2812 LED ring)     │
│    └→ MQTT voice/audio_stream  (broker .13:1883)                      │
│  voice_bridge (docker .13) — wake word, buffer audio                  │
│    └→ HTTP :5000                                                      │
│  voice_whisper (docker .13, faster-whisper) — STT audio→text          │
│    └→ publish text → MQTT homeassistant/voice/text  (broker .30)      │
│  ALSO: any client → POST http://.13:8501/voice {text}  (see clients.md)│
└───────────────────────────────────────────────────────────────────────┘
                          │  text
                          ▼
┌─ Brain ── voice-agent (systemd service on .13:8501) ─────────────────┐
│  LLM = OmniRoute (OpenAI-compatible, tool-calling)                   │
│  Model: auto/best-fast                                               │
│  LLM decides intent → calls a TOOL:                                  │
│    • get_time / get_weather / answer          (Open-Meteo, LLM)      │
│    • add_shopping_item    → KitchenOwl API   (shopping list)         │
│    • control_ha / ha_service_cmd  → Home Assistant REST API          │
│    • feed_fish            → HA input_button.fish_feeder              │
│    • set_reminder         → HA persistent_notification               │
│    • speak_as(voice,text) → piper per-character voice                │
│    • play_music           → Mopidy (Spotify, WORKING)               │
│    • pause/stop/next      → Mopidy MPD (planned)                    │
└───────────────────────────────────────────────────────────────────────┘
                          │  reply text
                          ▼
┌─ Output ─────────────────────────────────────────────────────────────┐
│  piper_tts (docker .13) → speak_direct.sh (pure-python resampler snapcast_send.py) → Snapcast → speakers  │
└───────────────────────────────────────────────────────────────────────┘

Key design decision: the agent is the single brain; Home Assistant is a passive device the agent commands via its REST API — no per-action HA automation/subscription is needed for the agent's commands (one long-lived HA token enables all HA control).

Components (`host/IP/port/repo`)

Component	Host/IP	Port	Repo / path	Role
ESP32-S3	—	—	`sam/voice_assistant`	Wake word, mic, LED
voice_bridge	`.13`	docker	`sam/voice_bridge`	Wake word + audio buffer
voice_whisper	`.13`	`:5000`	`sam/voice_bridge`	faster-whisper STT
voice-agent	`.13`	`:8501`	`sam/voice-assistant`	LLM brain + tools
OmniRoute	`.13`	`:20129` (API)	—	LLM gateway (OpenAI-compatible)
piper_tts	`.13`	`:10200`	`sam/speech_piper`	TTS (Wyoming)
Snapcast	`.13`	`:1780`	—	Multi-room audio
Mopidy / librespot	`.13`	`:6600`/`:6680`	—	Music: Spotify via gst-plugin-spotify + device-auth credentials
Home Assistant	`.30`	`:8123` (REST)	—	Home control
KitchenOwl	`.35`	`owl.lab.audasmedia.com.au`	—	Shopping list
mosquitto	`.13` + `.30`	`:1883`	—	MQTT brokers

Data flow summary

ESP32 streams audio to voice_bridge (MQTT on .13).
voice_bridge detects the wake word, buffers, sends to voice_whisper.
voice_whisper (faster-whisper) returns text.
Text is published to MQTT homeassistant/voice/text (broker .30) and/or POSTed to the agent.
voice-agent (LLM) decides intent and calls a tool.
Reply text → piper → Snapcast → speakers.

Music playback (Mopidy + Spotify) — working 2026-08-30

play_music → Mopidy (MPD :6600) → Spotify → Snapcast → speakers.

Two pieces were required (both were the blocker):

gst-plugin-spotify — the spotifyaudiosrc GStreamer element that decodes spotify:track URIs. Added pkgs.gst_all_1.gst-plugins-rs to systemd.services.mopidy.environment.GST_PLUGIN_PATH in .13 configuration.nix (also isolated Mopidy's GST path to its own 1.28.4 plugins + a void GST_PLUGIN_SYSTEM_PATH to avoid the system pipewire plugin poisoning the registry).
credentials.json — generated via librespot device-authorization flow (get_creds_device, built from librespot dev branch — has DeviceAuthClient). The old PKCE redirect flow (get_creds) is broken in librespot (deterministic code_verifier was incorrect). Device flow prints spotify.com/pair + code; user approves on any device. File at /var/lib/mopidy/spotify/credentials-cache/credentials.json (mopidy-owned).

Also fixed along the way: mopidy-local disabled (empty library.db crashed search: no such table: tracks); Mopidy-Spotify creds re-authed via mopidy.com (old grant revoked).

librespot (NixOS speaker's Spotify Connect output) is separate and untouched.

Run / manage

Agent service (persistent, auto-starts): systemctl status voice-agent (NixOS systemd unit declared in /etc/nixos/configuration.nix). Restart: sudo systemctl restart voice-agent.
Logs: journalctl -u voice-agent / journalctl -u snapserver.
Firewall: port 8501 is in allowedTCPPorts in the NixOS config.

sam/voice_assistant — ESP32 firmware
sam/voice_bridge — STT bridge + whisper
sam/speech_piper — TTS scripts/models → Snapcast
sam/voice-assistant — this overview + API contract + clients
sam/family_home_lab — console portal / dsh

Diagrams

Interactive map (Archify HTML, pan/zoom/search/dark-light): voice-assistant-map.html
- Published live: https://maps.lab.audasmedia.com.au/homeassistant/docs/voice-assistant-map.html
Mermaid source: voice-assistant.mmd · rendered SVG: diagrams/voice-assistant.svg
Archify JSON IR (single source of truth for the interactive map): archify/voice-assistant.architecture.json

Regenerate the interactive map from the JSON: node ~/.agents/skills/archify/bin/archify.mjs deliver architecture docs/archify/voice-assistant.architecture.json docs/voice-assistant-map.html --quality showcase --json

7.9 KiB Raw Permalink Blame History