voice-assistant/docs/overview.md

Jervis Voice Assistant — System Overview

Single source of truth for the local voice-assistant system on Sam's home lab. Companion docs: agent-api.md (how to send a command / get a reply) and clients.md (type-it, web, Android gateways). Component repos: voice_assistant (ESP32 firmware), voice_bridge (STT bridge), voice_agent (this brain), speech_piper (TTS).

---

What it does

A fully-local, private voice assistant. You say "Hey Jervis", then a natural-language request; it answers, controls the house, manages your shopping list, and speaks back through your speakers. The design principle: one LLM "brain" decides what to do, and calls tools. Home Assistant is one of those tools — not the decision maker.

---

Architecture

┌─ Input ───────────────────────────────────────────────────────────────┐
│  ESP32-S3  ("Hey Jervis" wake word, INMP441 mic, WS2812 LED ring)     │
│    └→ MQTT voice/audio_stream  (broker .13:1883)                      │
│  voice_bridge (docker .13) — wake word, buffer audio                  │
│    └→ HTTP :5000                                                      │
│  voice_whisper (docker .13, faster-whisper) — STT audio→text          │
│    └→ publish text → MQTT homeassistant/voice/text  (broker .30)      │
│  ALSO: any client → POST http://.13:8501/voice {text}  (see clients.md)│
└───────────────────────────────────────────────────────────────────────┘
                          │  text
                          ▼
┌─ Brain ── voice-agent (systemd service on .13:8501) ─────────────────┐
│  LLM = OmniRoute (OpenAI-compatible, tool-calling)                   │
│  Model: auto/best-fast                                               │
│  LLM decides intent → calls a TOOL:                                  │
│    • get_time / get_weather / answer          (Open-Meteo, LLM)      │
│    • add_shopping_item    → KitchenOwl API   (shopping list)         │
│    • control_ha / ha_service_cmd  → Home Assistant REST API          │
│    • feed_fish            → HA input_button.fish_feeder              │
│    • set_reminder         → HA persistent_notification               │
│    • speak_as(voice,text) → piper per-character voice                │
│    • play_music           → Mopidy (planned)                         │
└───────────────────────────────────────────────────────────────────────┘
                          │  reply text
                          ▼
┌─ Output ─────────────────────────────────────────────────────────────┐
│  piper_tts (docker .13) → speak_direct.sh (pure-python resampler snapcast_send.py) → Snapcast → speakers  │
└───────────────────────────────────────────────────────────────────────┘

Key design decision: the agent is the single brain; Home Assistant is a passive device the agent commands via its REST API — no per-action HA automation/subscription is needed for the agent's commands (one long-lived HA token enables all HA control).

---

Components (host/IP/port/repo)

Component	Host/IP	Port	Repo / path	Role
ESP32-S3	—	—	sam/voice_assistant	Wake word, mic, LED
voice_bridge	.13	docker	sam/voice_bridge	Wake word + audio buffer
voice_whisper	.13	:5000	sam/voice_bridge	faster-whisper STT
voice-agent	.13	:8501	sam/voice-assistant	LLM brain + tools
OmniRoute	.13	:20129 (API)	—	LLM gateway (OpenAI-compatible)
piper_tts	.13	:10200	sam/speech_piper	TTS (Wyoming)
Snapcast	.13	:1780	—	Multi-room audio
Mopidy / librespot	.13	:6600/:6680	—	Music sources
Home Assistant	.30	:8123 (REST)	—	Home control
KitchenOwl	.35	owl.lab.audasmedia.com.au	—	Shopping list
mosquitto	.13 + .30	:1883	—	MQTT brokers

---

Data flow summary

1. ESP32 streams audio to voice_bridge (MQTT on .13).
2. voice_bridge detects the wake word, buffers, sends to voice_whisper.
3. voice_whisper (faster-whisper) returns text.
4. Text is published to MQTT homeassistant/voice/text (broker .30) and/or POSTed to the agent.
5. voice-agent (LLM) decides intent and calls a tool.
6. Reply text → piper → Snapcast → speakers.

---

Run / manage

• Agent service (persistent, auto-starts): systemctl status voice-agent (NixOS systemd unit declared in /etc/nixos/configuration.nix). Restart: sudo systemctl restart voice-agent.
• Logs: journalctl -u voice-agent / journalctl -u snapserver.
• Firewall: port 8501 is in allowedTCPPorts in the NixOS config.

Repository map (other related projects)

• sam/voice_assistant — ESP32 firmware
• sam/voice_bridge — STT bridge + whisper
• sam/speech_piper — TTS scripts/models → Snapcast
• sam/voice-assistant — this overview + API contract + clients
• sam/family_home_lab — console portal / dsh