Home Voice Assistant System (Jervis)

Human-readable overview of the local voice-assistant project.

AI/project tracking: Vikunja → Jervis Voice Assistant (Whisper ESP HA Flow) (project id 5)

AI documentation (for bots): Outline → Voice Assistant (Whisper ESP HA) collection → Project Overview

API/docs (Gitea): sam/voice-assistant → overview / agent-api / clients

Tasks/progress: see the Vikunja project; docs live in Outline; this note is the human map.

What it is (TL;DR)

A private, local voice assistant. Say "Hey Jervis" (ESP32 wake word) then a natural-language request — a local LLM agent decides intent and calls tools: it controls Home Assistant, manages shopping/tasks/calendar, plays music, answers questions, and speaks back through the house speakers (piper → Snapcast).

Command it from anywhere: POST http://192.168.20.13:8501/voice {"text": "add milk to the shopping list"} — the reply is spoken.

Architecture

ESP32 (wake word, mic) → MQTT voice/audio_stream
  → voice_bridge + whisper (STT → text)
  → Jervis agent (FastAPI :8501, LLM tool-calling)
       ├─ Home Assistant  (lights, timers, calendar, HA todo lists)
       ├─ OmniRoute LLM   (voice-fast: OpenRouter → opencode-go → deepseek flash)
       ├─ Mopidy → Spotify → Snapcast  (music)
       ├─ Apprise         (family WhatsApp/Telegram/ntfy/email)
       ├─ Vikunja         (project tasks)
       └─ reply → piper → Snapcast speakers

Components (host : port)

Component	Host	Port	Role
ESP32-S3 (voice_assistant.ino)	devices	—	wake word "Hey Jervis", mic, LED
voice_bridge / voice_whisper	.13	docker / :5000	speech → text (faster-whisper)
Jervis agent	.13	:8501	LLM tool-calling brain (systemd)
OmniRoute	.13	:20129	LLM gateway (voice-fast route)
Home Assistant	.30	:8123	control, timers, calendar, todo lists
Mopidy + Spotify	.13	:6600 / librespot	music
piper_tts	.13	:10200	TTS → Snapcast
Snapcast	.13	:1780	multi-room audio
Apprise	.35	:8210	notifications
Vikunja	.35	vikunja.lab	project tasks/stages (tracker)
Outline	.13	:3000	project documentation (AI)

Status (2026-09-05)

Voice working end-to-end: ESP32 wake word → whisper → agent → piper → Snapcast (fixed docker piper, HA→.13 SSH auth, resampler, MQTT client-id).
HA integration: lights/scenes via Assist; shopping lists via HA todo (add_shopping_item); timers + spoken reminders (HA timer → automation → piper); calendar alerts spoken 30 min before (HA automation); add-calendar tool.
Music: play/pause/skip/volume via Mopidy → Spotify → Snapcast (credentials.json device-auth).
Q&A + web search: Tavily-backed web_search; fast LLM route voice-fast (~2–3s).
Family messaging: Apprise (sam/finn/harry/joanna; WhatsApp/email/Telegram/ntfy).
AI tooling: project-ops skill gives every pi agent Vikunja + Outline access (keys in 10-secrets.conf).

Tools & systems (employment/skills)

FastAPI · Python · Docker · MQTT · Home Assistant (REST/Assist/automations) · Whisper · piper TTS · Snapcast · Mopidy / librespot / Spotify · OmniRoute LLM routing · Tavily web search · Apprise · Vikunja & Outline APIs · NixOS (systemd services, flakes) · git-filter-repo (secret hygiene) · ESP32 (Arduino/ESP-IDF).

Key configs & links

Vikunja project: https://vikunja.lab.audasmedia.com.au → Jervis Voice Assistant (Whisper ESP HA Flow)
Outline collection: https://outline.lab.audasmedia.com.au → Voice Assistant (Whisper ESP HA)
Gitea: sam/voice-assistant (overview / agent-api / clients docs)
Repo layout / plan: ~/chats/homeassistant (plan.md, AGENT.md); agent at /home/sam/voice-agent/
Secrets: ~/.config/environment.d/10-secrets.conf on .27/.13/.51 (VIKUNJA_TOKEN, OL_API_KEY, etc.) — see also the secrets_fixes note.

4.2 KiB Raw Blame History Unescape Escape