hub_fixes/watchdog/README.md

# Voice intake watchdog (Hub Fixes — Directive 1)

Watches the house voice path and alerts Sam through Apprise when intake dies.
It **never heals anything** — no smart plug, no power-cycle, no restart.
Notification only. Sam fixes the failed layer by hand.

This is the fix for the 2026-10-13 outage: the `.13` broker restarted,
`voice_bridge` went deaf and the ESP32 stopped publishing, and nobody noticed
for two days.

## What it watches

"Voice intake alive" means all three layers are healthy, checked in this order:

| Layer | Signal | Failure message says |
|---|---|---|
| **broker** | `.13:1883` reachable (live MQTT session or TCP probe) | check the Mosquitto container; **do not** power-cycle the ESP32 |
| **bridge** | `voice_bridge` container running **and** subscribed to `voice/audio_stream` (Docker socket) | start the bridge; **do not** power-cycle the ESP32 |
| **publisher** | a message on `voice/audio_stream` within `SILENCE_TIMEOUT_SECONDS`, or retained `voice/status` presence | "Likely the ESP32 — try a power-cycle." |

Layer attribution is the point: if something else broke, the ESP32 is not blamed.

### Why 15 minutes is meaningful

The Directive 3 firmware publishes a **heartbeat on `voice/audio_stream` every
5 minutes**, so 15 minutes of silence is three missed beats — not "nobody
spoke". The firmware also sets a retained `voice/status` presence (`online` on
connect, `offline` as a Last Will) for immediate attribution.

Until a heartbeat or presence has been seen, the watchdog does **not** treat
silence as a publisher failure (that state means the heartbeat firmware is not
flashed yet, and silence is normal). This is controlled by
`REQUIRE_PUBLISHER_EVIDENCE` (default `1`).

## Repeat policy

- First failure → alert immediately.
- Same layer still down → repeat once per `REPEAT_INTERVAL_SECONDS` (default one
  day), for as long as it stays broken.
- A different layer fails → that is a new alert (immediate).
- Recovery → one recovery notice, then the state is cleared and alerts stop.
- State is persisted in a Docker volume, so a watchdog restart neither re-spams
  nor forgets an ongoing outage.

First alerts use tag `APPRISE_TAG` (`sam_ntfy`); daily repeats escalate to
`APPRISE_TAG_REPEAT` (`sam`). Both live in the Apprise config on `.35`.

## Files

```
watchdog/
  watchdog.py            the service (paho + stdlib only)
  Dockerfile             python:3.12-slim + paho-mqtt==2.1.0
  docker-compose.yml     host network, read-only Docker socket, state volume
  start_watchdog.sh      sources ~/.config/environment.d/10-secrets.conf, then compose up
  tests/test_watchdog.py unit tests for classification, repeats, messages
```

## Deploy

```bash
cd /home/sam/paseo/projects/hub_fixes/watchdog
./start_watchdog.sh                      # normal operation
DRY_RUN=1 ./start_watchdog.sh            # log alerts, send nothing
SILENCE_TIMEOUT_SECONDS=60 CHECK_INTERVAL_SECONDS=5 ./start_watchdog.sh   # test mode
```

It runs as container `voice_intake_watchdog`, `network_mode: host`, no listening
port, `restart: unless-stopped`. The Docker socket is mounted **read-only** and
only ever read (`/containers/voice_bridge/json` and its logs).

## Test

```bash
python3 -m unittest discover -s watchdog/tests -v
```

## Verify (evidence)

All commands below were run on `.13` on 2026-10-14.

**1. Stop `voice_bridge` → alert arrives within 15 minutes**

```
$ docker stop voice_bridge
$ # watchdog, real Apprise, 5 s checks
WARNING ALERT kind=alert layer=bridge -> tag=sam_ntfy
INFO apprise notify OK HTTP 200 tag=sam_ntfy: Voice intake DOWN — voice_bridge is not running
```

The bridge layer is checked directly, so the alert fires on the next check
(seconds), well inside the 15-minute budget.

**2. Start it again → alerts stop**

```
$ docker start voice_bridge
WARNING ALERT kind=recovery layer=bridge -> tag=sam_ntfy
INFO apprise notify OK HTTP 200 tag=sam_ntfy: Voice intake recovered
$ # no further alerts once healthy
```

**3. Broker down → names the broker, not the ESP32**

```
$ # watchdog pointed at an unreachable broker address
WARNING ALERT kind=alert layer=broker -> tag=sam_ntfy
TITLE: Voice intake DOWN — .13 MQTT broker unreachable
... Do NOT power-cycle the ESP32 — the broker layer failed, not the device.
```

**4. Daily repeat (shortened interval)**

```
$ # SILENCE_TIMEOUT=1 REPEAT_INTERVAL=6 CHECK_INTERVAL=2, dry-run
ALERT kind=alert  layer=publisher -> tag=sam_ntfy   (first)
ALERT kind=repeat layer=publisher -> tag=sam       (after the interval)
```

The unit tests also drive three consecutive daily repeats
(`test_daily_repeats_keep_firing`).

## Not in scope / boundaries

- No auto-heal of any kind.
- The `.13:1883` plane is untouched (plain, anonymous). The watchdog connects as
  a normal subscriber and publishes nothing.
- The broker was **not** stopped to prove layer attribution; an unreachable
  broker address exercises the same code path without disturbing n8n or the
  kids' 8883 devices.