From 057487f876f6a108e8733b54e74e4ab6d0a8a854 Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Thu, 8 Oct 2026 16:21:28 +1100 Subject: [PATCH] sam-4screen-desktop 2026-10-8:16:21:28 --- .obsidian/workspace.json | 4 +- ...— Cloud STT and TTS with Local Fallback.md | 101 ++++++++++++++++++ 010 inbox/_sort log.md | 3 + .../Home Voice Assistant System.md | 35 ++++-- 4 files changed, 134 insertions(+), 9 deletions(-) create mode 100644 010 inbox/MiniMax Voice — Cloud STT and TTS with Local Fallback.md diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index bd36b82..bbadd2d 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -210,8 +210,9 @@ "templater-obsidian:Templater": false } }, - "active": "26813e16e829b798", + "active": "8018aa52f0591df9", "lastOpenFiles": [ + "010 inbox/MiniMax Voice — Cloud STT and TTS with Local Fallback.md", "300 areas/355 AI Research and Techniques/Ai memory management.md", "200 projects/Tools Software WebUI/Family Console.md", "200 projects/Tools Software WebUI/Family Home Lab.md", @@ -239,7 +240,6 @@ "400 Personal Family/410 Business Ideas/Business Ideas Display Screens.md", "010 inbox/Business Idea - Franchises display screens.md", "400 Personal Family/410 Business Ideas/Business Idea - Franchises display screens.md", - "010 inbox/Business Ideas Display Screens.md", "010 inbox/_WATCHER", "300 areas/355 AI Research and Techniques", "300 areas/355 Ai Research and Techniques", diff --git a/010 inbox/MiniMax Voice — Cloud STT and TTS with Local Fallback.md b/010 inbox/MiniMax Voice — Cloud STT and TTS with Local Fallback.md new file mode 100644 index 0000000..90ba3de --- /dev/null +++ b/010 inbox/MiniMax Voice — Cloud STT and TTS with Local Fallback.md @@ -0,0 +1,101 @@ +--- +title: MiniMax Voice +type: note +status: active +proposed_folder: 200 projects/Tools Software WebUI +proposed_toc: "100 Table of Contents/Projects.md → Tools & Software" +proposed_tags: + - migration + - home-assistant + - project + - esp32 + - architecture + - minimax + - voice + - ai + - showcase +confidence: 0.66 +approved: false +proposed_for: 324fa7828911 +created: "2026-10-08" +tags: + - project + - voice + - ai + - showcase +aliases: + - MiniMax Voice +skills: + - API integration (MiniMax T2A, Speech-to-Text) + - Python standard-library services + - Docker and Docker Compose + - Audio DSP (PCM decode, linear resampling) + - MQTT and Snapcast + - Benchmarking and A/B measurement +tools: + - MiniMax cloud APIs + - OpenRouter Whisper + - faster-whisper + - piper TTS + - Snapcast + - Home Assistant + - Archify + - NixOS +--- + +# MiniMax Voice — Overview + +## 💡 What is this project? + +A migration of the **Jervis** home voice assistant to **MiniMax cloud** speech +services, while keeping a working **local fallback** so the house keeps working +without internet. + +Wake-word detection stays on the local machine. Only the spoken command is sent +to the cloud. This keeps cost low and keeps the continuous microphone stream +private. + +The assistant hears the wake word, transcribes the spoken command, decides an +intent with an LLM, calls a tool, and speaks the reply through the house +speakers. + +## 🌟 Key Highlights (Portfolio & Employment) + +- **Concept**: Replace a heavy local speech model with a cloud API, without ever + losing voice input when the internet is down. +- **Tools & Skills Showcase**: MiniMax and OpenRouter APIs, Python standard + library, Docker, MQTT, Snapcast, audio resampling, drop-in service + compatibility, graceful fallback design, reproducible benchmarks. +- **Practical Value**: Frees a few hundred MB of RAM on the home server, improves + voice quality, and keeps the house working during outages. + +## 📐 Architecture & Flow + +```mermaid +flowchart LR + ESP[ESP32-S3 wake word + mic] --> BRIDGE[voice_bridge] + BRIDGE --> SHIM[voice_stt_shim :5001] + SHIM -->|primary| CLOUD[Cloud STT] + SHIM -->|fallback| LOCAL[voice_whisper_fallback :5002] + SHIM --> AGENT[voice-agent Jervis :8501] + AGENT --> HA[Home Assistant] + AGENT --> TTS[tts_router] + TTS -->|fallback| PIPER[piper_tts] + TTS --> SNAP[Snapcast :4953] + PIPER --> SNAP + SNAP --> SPEAKERS[House speakers] +``` + +## 🔗 Related Notes & Links + +- Interactive Map: [Archify Map](https://maps.lab.audasmedia.com.au/minimax_voice_tts_stt/docs/minimax-voice-map.html) +- Project workspace: `/home/sam/paseo/projects/minimax_voice_tts_stt/` on `.13` +- Vikunja: *Jervis Voice Assistant (Whisper ESP HA Flow)* project +- Outline: *Voice Assistant (Whisper ESP HA)* collection +- Related: [[Home Voice Assistant System]] + + diff --git a/010 inbox/_sort log.md b/010 inbox/_sort log.md index e6d34f9..b8dbdd1 100644 --- a/010 inbox/_sort log.md +++ b/010 inbox/_sort log.md @@ -20023,3 +20023,6 @@ What the watcher did, newest last. ### 2026-10-03 09:34:31 - **re-proposing** `Family Holiday Camping Wilsons Prom.md` — you edited it - **proposed** `Family Holiday Camping Wilsons Prom.md` → `400 Personal Family/470 Holidays Travel` (folder already set — kept) (confidence 0.70) + +### 2026-10-08 16:16:18 +- **proposed** `MiniMax Voice — Cloud STT and TTS with Local Fallback.md` → `200 projects/Tools Software WebUI` (confidence 0.66) diff --git a/300 areas/360 Dev-Ops Network Computers/Home Voice Assistant System.md b/300 areas/360 Dev-Ops Network Computers/Home Voice Assistant System.md index 251d30c..4083918 100644 --- a/300 areas/360 Dev-Ops Network Computers/Home Voice Assistant System.md +++ b/300 areas/360 Dev-Ops Network Computers/Home Voice Assistant System.md @@ -1,6 +1,6 @@ --- created: 2026-08-30 -modified: 2026-09-05 +modified: 2026-10-08 type: area status: active tags: @@ -26,24 +26,44 @@ A private, local voice assistant. Say **"Hey Jervis"** (ESP32 wake word) then a **Command it from anywhere:** `POST http://192.168.20.13:8501/voice {"text": "add milk to the shopping list"}` — the reply is spoken. +## MiniMax voice migration (2026-10-08) +Speech-to-text and text-to-speech now run on **MiniMax cloud APIs**, with a +**local fallback** so the house still works offline. Wake-word detection stays +local; only the spoken command is sent to the cloud. + +- **STT**: `voice_stt_shim` (`:5001`) — cloud primary (OpenRouter `whisper-1`; + MiniMax `asr-1.0` ready), local `base.en` fallback (`:5002`). +- **TTS**: `tts_router` behind the unchanged `speak_direct.sh` — MiniMax T2A v2 + with piper fallback. +- **Personality**: the agent emits a speech plan (inline sound tags + one + emotion/delivery config). Kill switch: `VOICE_PERSONALITY=0`. +- **Result**: about 239 MiB of resident RAM freed, better voices, and voice + input survives an internet outage. +- Full project note: [[MiniMax Voice — Cloud STT and TTS with Local Fallback]]. +- Interactive map: [Archify Map](https://maps.lab.audasmedia.com.au/minimax_voice_tts_stt/docs/minimax-voice-map.html). + ## Architecture ``` ESP32 (wake word, mic) → MQTT voice/audio_stream - → voice_bridge + whisper (STT → text) + → voice_bridge (OpenWakeWord, local) + → voice_stt_shim (:5001) — cloud STT + local base.en fallback (:5002) → Jervis agent (FastAPI :8501, LLM tool-calling) ├─ Home Assistant (lights, timers, calendar, HA todo lists) ├─ OmniRoute LLM (voice-fast: OpenRouter → opencode-go → deepseek flash) ├─ Mopidy → Spotify → Snapcast (music) ├─ Apprise (family WhatsApp/Telegram/ntfy/email) ├─ Vikunja (project tasks) - └─ reply → piper → Snapcast speakers + └─ reply → speak_direct.sh → tts_router (MiniMax T2A / piper) → Snapcast ``` ## Components (host : port) | Component | Host | Port | Role | | ------------------------------ | ------- | ----------------- | ------------------------------------- | | ESP32-S3 (voice_assistant.ino) | devices | — | wake word "Hey Jervis", mic, LED | -| voice_bridge / voice_whisper | .13 | docker / :5000 | speech → text (faster-whisper) | +| voice_bridge | .13 | docker | wake word + audio buffer | +| voice_stt_shim | .13 | :5001 | cloud STT + local fallback chain | +| voice_whisper_fallback | .13 | :5002 | local base.en STT fallback | +| tts_router | .13 | — | MiniMax T2A / piper routing | | **Jervis agent** | .13 | **:8501** | LLM tool-calling brain (systemd) | | OmniRoute | .13 | :20129 | LLM gateway (voice-fast route) | | Home Assistant | .30 | :8123 | control, timers, calendar, todo lists | @@ -54,7 +74,7 @@ ESP32 (wake word, mic) → MQTT voice/audio_stream | Vikunja | .35 | vikunja.lab | project tasks/stages (tracker) | | Outline | .13 | :3000 | project documentation (AI) | -## Status (2026-09-05) +## Status (2026-10-08) - **Voice working end-to-end**: ESP32 wake word → whisper → agent → piper → Snapcast (fixed docker piper, HA→.13 SSH auth, resampler, MQTT client-id). - **HA integration**: lights/scenes via Assist; shopping lists via HA todo (`add_shopping_item`); timers + spoken reminders (HA `timer` → automation → piper); calendar alerts spoken 30 min before (HA automation); add-calendar tool. - **Music**: play/pause/skip/volume via Mopidy → Spotify → Snapcast (credentials.json device-auth). @@ -63,13 +83,14 @@ ESP32 (wake word, mic) → MQTT voice/audio_stream - **AI tooling**: `project-ops` skill gives every pi agent Vikunja + Outline access (keys in `10-secrets.conf`). ## Tools & systems (employment/skills) -FastAPI · Python · Docker · MQTT · Home Assistant (REST/Assist/automations) · Whisper · piper TTS · Snapcast · Mopidy / librespot / Spotify · OmniRoute LLM routing · Tavily web search · Apprise · Vikunja & Outline APIs · NixOS (systemd services, flakes) · git-filter-repo (secret hygiene) · ESP32 (Arduino/ESP-IDF). +FastAPI · Python · Docker · MQTT · Home Assistant (REST/Assist/automations) · MiniMax T2A & Speech-to-Text · OpenRouter Whisper · faster-whisper · piper TTS · Snapcast · Mopidy / librespot / Spotify · OmniRoute LLM routing · Tavily web search · Apprise · Vikunja & Outline APIs · NixOS (systemd services, flakes) · git-filter-repo (secret hygiene) · ESP32 (Arduino/ESP-IDF) · Archify. ## Key configs & links - **Vikunja project**: `https://vikunja.lab.audasmedia.com.au` → *Jervis Voice Assistant (Whisper ESP HA Flow)* - **Outline collection**: `https://outline.lab.audasmedia.com.au` → *Voice Assistant (Whisper ESP HA)* +- **MiniMax Voice project**: `/home/sam/paseo/projects/minimax_voice_tts_stt/` on `.13`; map at `https://maps.lab.audasmedia.com.au/minimax_voice_tts_stt/docs/` - **Gitea**: `sam/voice-assistant` (overview / agent-api / clients docs) - **Repo layout / plan**: `~/chats/homeassistant` (plan.md, AGENT.md); agent at `/home/sam/voice-agent/` - **Secrets**: `~/.config/environment.d/10-secrets.conf` on .27/.13/.51 (VIKUNJA_TOKEN, OL_API_KEY, etc.) — see also the `secrets_fixes` note. -See also: [[Docker Containers]], [[Home Network Map Overview]]. \ No newline at end of file +See also: [[Docker Containers]], [[Home Network Map Overview]], [[MiniMax Voice — Cloud STT and TTS with Local Fallback]]. \ No newline at end of file