voice_assistant/README.md

AI Voice Assistant Gateway (ESP32-S3 + Docker)

🎯 Project Objective

A private, local voice assistant system that separates high-bandwidth audio streaming from smart home command logic.

• Frontend: ESP32-S3 acts as a dumb "microphone-to-MQTT" bridge.
• Backend: Docker container performs Wake Word detection (OpenWakeWord) and Speech-to-Text (Faster-Whisper).
• Controller: Home Assistant receives clean text commands for execution.

🏗️ Architecture

The system uses a Dual-Broker strategy to prevent flooding Home Assistant with raw audio data.

graph LR
    Mic[INMP441] -->|I2S| ESP32
    ESP32 -->|Raw Audio (Voice Broker)| MQTT_Voice[Mosquitto .13]
    
    subgraph Docker Server [.13]
        MQTT_Voice -->|Sub Audio| Bridge[Python Bridge]
        Bridge -->|HTTP| Whisper[Whisper API]
        Whisper -->|Text| Bridge
    end
    
    Bridge -->|Text Command (HA Broker)| MQTT_HA[Mosquitto .30]
    MQTT_HA -->|Trigger| HA[Home Assistant]
    HA -->|Action| Lights/Scripts

📂 File Structure

1. Firmware (Client)

Location: ~/Documents/Arduino/voice_assistant/

• voice_assistant.ino: Main ESP32 firmware. Handles I2S reading, VAD (Voice Activity Detection), and MQTT streaming.
• Dependencies: PubSubClient, Freenove_WS2812_Lib_for_ESP32.

2. Backend (Server)

Location: ~/voice_bridge/ (On Server .13)

• docker-compose.yml: Orchestrates the Whisper and Bridge containers.
• mqtt_audio_bridge.py: The "Brain".
  • Listens to Audio on .13.
  • Buffers audio and checks for Wake Word ("Hey Jarvis").
  • Sends valid audio to Whisper.
  • Publishes resulting text to Home Assistant Broker .30.
• app.py: A lightweight Flask wrapper for faster-whisper (running in separate container).

🚦 LED Status Indicators (ESP32)

The LED Ring provides real-time feedback on the device state.

Green now means "MQTT connected and publishing", not merely "Wi-Fi joined". This is the Directive 3 fix: the old ring could sit on green (streaming) while the socket was dead and nothing reached the broker.

Color	State	Trigger
Red blink	Wi-Fi down	Not joined to the access point. Never green.
Amber blink	MQTT down	Wi-Fi up but the broker link is down. Never green.
Green (steady)	Idle + healthy	MQTT connected, subscribed, heartbeats publishing.
Green (bright)	Recording	Sound detected above threshold, streaming audio.
Blue	Processing	Silence detected, end of stream sent to server, waiting for reply.
Yellow	Acknowledged	Server replied OK (command executed).

🔌 MQTT reliability (Hub Fixes — Directive 3)

• Auto-reconnect + re-subscribe on every (re)connect. PubSubClient reopens the socket but does not restore subscriptions. connectMqtt() re-subscribes to voice/status on every connect and logs the result. Without this the client can hold a live socket with no subscription and go silently deaf.
• Heartbeat every 5 minutes on voice/audio_stream. The watchdog (Directive 1) watches that topic, so topic silence means "the publisher is dead", not "nobody spoke". The heartbeat is a valid silent PCM frame (256 bytes), so voice_bridge treats it as silence and ignores it.
• Presence. The device sets a retained Last Will offline on voice/status and publishes retained online on connect. This gives the watchdog a way to attribute a failure to the ESP32 layer.
• Drop logging. On every connected→disconnected transition the serial console prints the PubSubClient state code (for example CONNECTION_LOST) or the Wi-Fi disconnect reason code.

⚙️ Configuration Details

Hardware Config (ESP32-S3)

• Mic SCK: GPIO 4
• Mic WS: GPIO 5
• Mic SD: GPIO 6
• LED Pin: GPIO 1
• VAD Threshold: 1000 (Adjust based on noise floor)
• Soft Gain: x6 (Multiplies raw input to match Whisper requirements)

config.h and led_control.h are stale early drafts and are not included by the sketch. voice_assistant.ino is the single source of truth. Do not add a second copy of the broker/topic/pin config.

Docker Environment

• Network: Internal bridge network between Bridge and Whisper.
• Models:
  • Wake Word: hey_jarvis (OpenWakeWord)
  • STT: small.en (Faster-Whisper, int8 quantization)

🚀 Setup & Commands

1. Flash ESP32

Credentials in voice_assistant.ino are placeholders in this repo. Set the real values locally before flashing, and never commit them. Flashing needs the ESP32 physically connected over USB to a machine with arduino-cli (this is the step that cannot be done from the server):

arduino-cli upload -v -p /dev/ttyACM0 --fqbn esp32:esp32:esp32s3:CDCOnBoot=cdc,USBMode=hwcdc,PSRAM=disabled,FlashMode=dio,DebugLevel=info --input-dir ./build

Watch the serial console (arduino-cli monitor -p /dev/ttyACM0 -c baudrate=115200) to confirm the [mqtt] re-subscribe and [heartbeat] lines.

2. Start Backend

cd ~/voice_bridge
docker compose up -d

3. Monitor Logs

docker compose logs -f voice_bridge

4. Monitor Audio Stream (Debug)

mosquitto_sub -h 192.168.20.13 -u "MQTT_USER_REDACTED" -P "pass" -t "voice/audio_stream" | awk '{printf "."}'