---
created: 2026-08-30
modified: 2026-10-08
type: area
status: active
tags:
  - voice
  - system
  - ai
  - project
aliases:
  - Jervis
  - Voice Assistant
---

# Home Voice Assistant System (Jervis)

> **Human-readable overview of the local voice-assistant project.**
> - **AI/project tracking:** Vikunja → *Jervis Voice Assistant (Whisper ESP HA Flow)* (project id 5)
> - **AI documentation (for bots):** Outline → *Voice Assistant (Whisper ESP HA)* collection → [Project Overview](/doc/project-overview-genev9IHFT)
> - **API/docs (Gitea):** `sam/voice-assistant` → overview / agent-api / clients
> - **Tasks/progress:** see the Vikunja project; docs live in Outline; this note is the human map.

## What it is (TL;DR)
A private, local voice assistant. Say **"Hey Jervis"** (ESP32 wake word) then a natural-language request — a local **LLM agent** decides intent and calls tools: it controls Home Assistant, manages shopping/tasks/calendar, plays music, answers questions, and speaks back through the house speakers (piper → Snapcast).

**Command it from anywhere:** `POST http://192.168.20.13:8501/voice {"text": "add milk to the shopping list"}` — the reply is spoken.

## MiniMax voice migration (2026-10-08)
Speech-to-text and text-to-speech now run on **MiniMax cloud APIs**, with a
**local fallback** so the house still works offline. Wake-word detection stays
local; only the spoken command is sent to the cloud.

- **STT**: `voice_stt_shim` (`:5001`) — cloud primary (OpenRouter `whisper-1`;
  MiniMax `asr-1.0` ready), local `base.en` fallback (`:5002`).
- **TTS**: `tts_router` behind the unchanged `speak_direct.sh` — MiniMax T2A v2
  with piper fallback.
- **Personality**: the agent emits a speech plan (inline sound tags + one
  emotion/delivery config). Kill switch: `VOICE_PERSONALITY=0`.
- **Result**: about 239 MiB of resident RAM freed, better voices, and voice
  input survives an internet outage.
- Full project note: [[MiniMax Voice — Cloud STT and TTS with Local Fallback]].
- Interactive map: [Archify Map](https://maps.lab.audasmedia.com.au/minimax_voice_tts_stt/docs/minimax-voice-map.html).

## Architecture
```
ESP32 (wake word, mic) → MQTT voice/audio_stream
  → voice_bridge (OpenWakeWord, local)
  → voice_stt_shim (:5001) — cloud STT + local base.en fallback (:5002)
  → Jervis agent (FastAPI :8501, LLM tool-calling)
       ├─ Home Assistant  (lights, timers, calendar, HA todo lists)
       ├─ OmniRoute LLM   (voice-fast: OpenRouter → opencode-go → deepseek flash)
       ├─ Mopidy → Spotify → Snapcast  (music)
       ├─ Apprise         (family WhatsApp/Telegram/ntfy/email)
       ├─ Vikunja         (project tasks)
       └─ reply → speak_direct.sh → tts_router (MiniMax T2A / piper) → Snapcast
```

## Components (host : port)
| Component                      | Host    | Port              | Role                                  |
| ------------------------------ | ------- | ----------------- | ------------------------------------- |
| ESP32-S3 (voice_assistant.ino) | devices | —                 | wake word "Hey Jervis", mic, LED      |
| voice_bridge                   | .13     | docker            | wake word + audio buffer              |
| voice_stt_shim                 | .13     | :5001             | cloud STT + local fallback chain      |
| voice_whisper_fallback         | .13     | :5002             | local base.en STT fallback            |
| tts_router                     | .13     | —                 | MiniMax T2A / piper routing           |
| **Jervis agent**               | .13     | **:8501**         | LLM tool-calling brain (systemd)      |
| OmniRoute                      | .13     | :20129            | LLM gateway (voice-fast route)        |
| Home Assistant                 | .30     | :8123             | control, timers, calendar, todo lists |
| Mopidy + Spotify               | .13     | :6600 / librespot | music                                 |
| piper_tts                      | .13     | :10200            | TTS → Snapcast                        |
| Snapcast                       | .13     | :1780             | multi-room audio                      |
| Apprise                        | .35     | :8210             | notifications                         |
| Vikunja                        | .35     | vikunja.lab       | project tasks/stages (tracker)        |
| Outline                        | .13     | :3000             | project documentation (AI)            |

## Status (2026-10-08)
- **Voice working end-to-end**: ESP32 wake word → whisper → agent → piper → Snapcast (fixed docker piper, HA→.13 SSH auth, resampler, MQTT client-id).
- **HA integration**: lights/scenes via Assist; shopping lists via HA todo (`add_shopping_item`); timers + spoken reminders (HA `timer` → automation → piper); calendar alerts spoken 30 min before (HA automation); add-calendar tool.
- **Music**: play/pause/skip/volume via Mopidy → Spotify → Snapcast (credentials.json device-auth).
- **Q&A + web search**: Tavily-backed `web_search`; fast LLM route `voice-fast` (~2–3s).
- **Family messaging**: Apprise (sam/finn/harry/joanna; WhatsApp/email/Telegram/ntfy).
- **AI tooling**: `project-ops` skill gives every pi agent Vikunja + Outline access (keys in `10-secrets.conf`).

## Tools & systems (employment/skills)
FastAPI · Python · Docker · MQTT · Home Assistant (REST/Assist/automations) · MiniMax T2A & Speech-to-Text · OpenRouter Whisper · faster-whisper · piper TTS · Snapcast · Mopidy / librespot / Spotify · OmniRoute LLM routing · Tavily web search · Apprise · Vikunja & Outline APIs · NixOS (systemd services, flakes) · git-filter-repo (secret hygiene) · ESP32 (Arduino/ESP-IDF) · Archify.

## Key configs & links
- **Vikunja project**: `https://vikunja.lab.audasmedia.com.au` → *Jervis Voice Assistant (Whisper ESP HA Flow)*
- **Outline collection**: `https://outline.lab.audasmedia.com.au` → *Voice Assistant (Whisper ESP HA)*
- **MiniMax Voice project**: `/home/sam/paseo/projects/minimax_voice_tts_stt/` on `.13`; map at `https://maps.lab.audasmedia.com.au/minimax_voice_tts_stt/docs/`
- **Gitea**: `sam/voice-assistant` (overview / agent-api / clients docs)
- **Repo layout / plan**: `~/chats/homeassistant` (plan.md, AGENT.md); agent at `/home/sam/voice-agent/`
- **Secrets**: `~/.config/environment.d/10-secrets.conf` on .27/.13/.51 (VIKUNJA_TOKEN, OL_API_KEY, etc.) — see also the `secrets_fixes` note.

See also: [[Docker Containers]], [[Home Network Map Overview]], [[MiniMax Voice — Cloud STT and TTS with Local Fallback]].