voice_assistant/README.md

# AI Voice Assistant Gateway (ESP32-S3 + Docker)

## 🎯 Project Objective
A private, local voice assistant system that separates high-bandwidth audio streaming from smart home command logic. 
- **Frontend:** ESP32-S3 acts as a dumb "microphone-to-MQTT" bridge.
- **Backend:** Docker container performs Wake Word detection (OpenWakeWord) and Speech-to-Text (Faster-Whisper).
- **Controller:** Home Assistant receives clean text commands for execution.

## 🏗️ Architecture
The system uses a **Dual-Broker** strategy to prevent flooding Home Assistant with raw audio data.

```mermaid
graph LR
    Mic[INMP441] -->|I2S| ESP32
    ESP32 -->|Raw Audio (Voice Broker)| MQTT_Voice[Mosquitto .13]
    
    subgraph Docker Server [.13]
        MQTT_Voice -->|Sub Audio| Bridge[Python Bridge]
        Bridge -->|HTTP| Whisper[Whisper API]
        Whisper -->|Text| Bridge
    end
    
    Bridge -->|Text Command (HA Broker)| MQTT_HA[Mosquitto .30]
    MQTT_HA -->|Trigger| HA[Home Assistant]
    HA -->|Action| Lights/Scripts
```

## 📂 File Structure

### 1. Firmware (Client)
Location: `~/Documents/Arduino/voice_assistant/`
*   **`voice_assistant.ino`**: Main ESP32 firmware. Handles I2S reading, VAD (Voice Activity Detection), and MQTT streaming.
*   **Dependencies**: `PubSubClient`, `Freenove_WS2812_Lib_for_ESP32`.

### 2. Backend (Server)
Location: `~/voice_bridge/` (On Server .13)
*   **`docker-compose.yml`**: Orchestrates the Whisper and Bridge containers.
*   **`mqtt_audio_bridge.py`**: The "Brain".
    *   Listens to Audio on `.13`.
    *   Buffers audio and checks for Wake Word ("Hey Jarvis").
    *   Sends valid audio to Whisper.
    *   Publishes resulting text to Home Assistant Broker `.30`.
*   **`app.py`**: A lightweight Flask wrapper for `faster-whisper` (running in separate container).

## 🚦 LED Status Indicators (ESP32)
The LED Ring provides real-time feedback on the device state.

**Green now means "MQTT connected and publishing", not merely "Wi-Fi joined".**
This is the Directive 3 fix: the old ring could sit on green (streaming) while the
socket was dead and nothing reached the broker.

| Color | State | Trigger |
| :--- | :--- | :--- |
| **Red blink** | Wi-Fi down | Not joined to the access point. Never green. |
| **Amber blink** | MQTT down | Wi-Fi up but the broker link is down. Never green. |
| **Green (steady)** | Idle + healthy | MQTT connected, subscribed, heartbeats publishing. |
| **Green (bright)** | Recording | Sound detected above threshold, streaming audio. |
| **Blue** | Processing | Silence detected, end of stream sent to server, waiting for reply. |
| **Yellow** | Acknowledged | Server replied `OK` (command executed). |

## 🔌 MQTT reliability (Hub Fixes — Directive 3)

- **Auto-reconnect + re-subscribe on every (re)connect.** PubSubClient reopens
the socket but does not restore subscriptions. `connectMqtt()` re-subscribes to
`voice/status` on every connect and logs the result. Without this the client can
hold a live socket with no subscription and go silently deaf.
- **Heartbeat every 5 minutes on `voice/audio_stream`.** The watchdog (Directive 1)
watches that topic, so topic silence means "the publisher is dead", not "nobody
spoke". The heartbeat is a valid silent PCM frame (256 bytes), so `voice_bridge`
treats it as silence and ignores it.
- **Presence.** The device sets a retained Last Will `offline` on `voice/status`
and publishes retained `online` on connect. This gives the watchdog a way to
attribute a failure to the ESP32 layer.
- **Drop logging.** On every connected→disconnected transition the serial console
prints the PubSubClient state code (for example `CONNECTION_LOST`) or the Wi-Fi
disconnect reason code.

## ⚙️ Configuration Details

### Hardware Config (ESP32-S3)
*   **Mic SCK:** GPIO 4
*   **Mic WS:** GPIO 5
*   **Mic SD:** GPIO 6
*   **LED Pin:** GPIO 1
*   **VAD Threshold:** `1000` (Adjust based on noise floor)
*   **Soft Gain:** `x6` (Multiplies raw input to match Whisper requirements)

> `config.h` and `led_control.h` are stale early drafts and are **not** included
> by the sketch. `voice_assistant.ino` is the single source of truth. Do not add
> a second copy of the broker/topic/pin config.

### Docker Environment
*   **Network:** Internal bridge network between Bridge and Whisper.
*   **Models:** 
    *   Wake Word: `hey_jarvis` (OpenWakeWord)
    *   STT: `small.en` (Faster-Whisper, int8 quantization)

## 🚀 Setup & Commands

**1. Flash ESP32**

Credentials in `voice_assistant.ino` are placeholders in this repo. Set the real
values locally before flashing, and never commit them. Flashing needs the ESP32
physically connected over USB to a machine with `arduino-cli` (this is the step
that cannot be done from the server):

```bash
arduino-cli upload -v -p /dev/ttyACM0 --fqbn esp32:esp32:esp32s3:CDCOnBoot=cdc,USBMode=hwcdc,PSRAM=disabled,FlashMode=dio,DebugLevel=info --input-dir ./build
```

Watch the serial console (`arduino-cli monitor -p /dev/ttyACM0 -c baudrate=115200`)
to confirm the `[mqtt] re-subscribe` and `[heartbeat]` lines.

**2. Start Backend**
```bash
cd ~/voice_bridge
docker compose up -d
```

**3. Monitor Logs**
```bash
docker compose logs -f voice_bridge
```

**4. Monitor Audio Stream (Debug)**
```bash
mosquitto_sub -h 192.168.20.13 -u "MQTT_USER_REDACTED" -P "pass" -t "voice/audio_stream" | awk '{printf "."}'
```