Directive 1: voice intake watchdog (layer attribution, daily repeat, Apprise)

Build the Hub Fixes Directive 1 watchdog as a small standalone service on .13.

- Watch voice/audio_stream (paho); 15 min silence = 3 missed heartbeats.
- Attribute failures in order: broker -> voice_bridge -> ESP32/no-publisher.
  The ESP32 is never blamed when the broker or the bridge broke.
- Alert via Apprise on .35:8210 (tag sam_ntfy; daily repeat escalates to sam).
- Persist state; repeat once a day while broken; recovery notice; alerts stop.
- No auto-heal of any kind. Notification only.
- Require publisher evidence (heartbeat or retained presence) before treating
  silence as a failure, so the unflashed Directive 3 firmware cannot cause the
  cry-wolf false alarms the spec warns about.
- Docker: host network, read-only Docker socket, state volume, unless-stopped.
- 23 unit tests; live evidence captured in watchdog/README.md.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

watchdog/README.md

@@ -0,0 +1,130 @@
+# Voice intake watchdog (Hub Fixes — Directive 1)
+
+Watches the house voice path and alerts Sam through Apprise when intake dies.
+It **never heals anything** — no smart plug, no power-cycle, no restart.
+Notification only. Sam fixes the failed layer by hand.
+
+This is the fix for the 2026-10-13 outage: the `.13` broker restarted,
+`voice_bridge` went deaf and the ESP32 stopped publishing, and nobody noticed
+for two days.
+
+## What it watches
+
+"Voice intake alive" means all three layers are healthy, checked in this order:
+
+| Layer | Signal | Failure message says |
+|---|---|---|
+| **broker** | `.13:1883` reachable (live MQTT session or TCP probe) | check the Mosquitto container; **do not** power-cycle the ESP32 |
+| **bridge** | `voice_bridge` container running **and** subscribed to `voice/audio_stream` (Docker socket) | start the bridge; **do not** power-cycle the ESP32 |
+| **publisher** | a message on `voice/audio_stream` within `SILENCE_TIMEOUT_SECONDS`, or retained `voice/status` presence | "Likely the ESP32 — try a power-cycle." |
+
+Layer attribution is the point: if something else broke, the ESP32 is not blamed.
+
+### Why 15 minutes is meaningful
+
+The Directive 3 firmware publishes a **heartbeat on `voice/audio_stream` every
+5 minutes**, so 15 minutes of silence is three missed beats — not "nobody
+spoke". The firmware also sets a retained `voice/status` presence (`online` on
+connect, `offline` as a Last Will) for immediate attribution.
+
+Until a heartbeat or presence has been seen, the watchdog does **not** treat
+silence as a publisher failure (that state means the heartbeat firmware is not
+flashed yet, and silence is normal). This is controlled by
+`REQUIRE_PUBLISHER_EVIDENCE` (default `1`).
+
+## Repeat policy
+
+- First failure → alert immediately.
+- Same layer still down → repeat once per `REPEAT_INTERVAL_SECONDS` (default one
+  day), for as long as it stays broken.
+- A different layer fails → that is a new alert (immediate).
+- Recovery → one recovery notice, then the state is cleared and alerts stop.
+- State is persisted in a Docker volume, so a watchdog restart neither re-spams
+  nor forgets an ongoing outage.
+
+First alerts use tag `APPRISE_TAG` (`sam_ntfy`); daily repeats escalate to
+`APPRISE_TAG_REPEAT` (`sam`). Both live in the Apprise config on `.35`.
+
+## Files
+
+```
+watchdog/
+  watchdog.py            the service (paho + stdlib only)
+  Dockerfile             python:3.12-slim + paho-mqtt==2.1.0
+  docker-compose.yml     host network, read-only Docker socket, state volume
+  start_watchdog.sh      sources ~/.config/environment.d/10-secrets.conf, then compose up
+  tests/test_watchdog.py unit tests for classification, repeats, messages
+```
+
+## Deploy
+
+```bash
+cd /home/sam/paseo/projects/hub_fixes/watchdog
+./start_watchdog.sh                      # normal operation
+DRY_RUN=1 ./start_watchdog.sh            # log alerts, send nothing
+SILENCE_TIMEOUT_SECONDS=60 CHECK_INTERVAL_SECONDS=5 ./start_watchdog.sh   # test mode
+```
+
+It runs as container `voice_intake_watchdog`, `network_mode: host`, no listening
+port, `restart: unless-stopped`. The Docker socket is mounted **read-only** and
+only ever read (`/containers/voice_bridge/json` and its logs).
+
+## Test
+
+```bash
+python3 -m unittest discover -s watchdog/tests -v
+```
+
+## Verify (evidence)
+
+All commands below were run on `.13` on 2026-10-14.
+
+**1. Stop `voice_bridge` → alert arrives within 15 minutes**
+
+```
+$ docker stop voice_bridge
+$ # watchdog, real Apprise, 5 s checks
+WARNING ALERT kind=alert layer=bridge -> tag=sam_ntfy
+INFO apprise notify OK HTTP 200 tag=sam_ntfy: Voice intake DOWN — voice_bridge is not running
+```
+
+The bridge layer is checked directly, so the alert fires on the next check
+(seconds), well inside the 15-minute budget.
+
+**2. Start it again → alerts stop**
+
+```
+$ docker start voice_bridge
+WARNING ALERT kind=recovery layer=bridge -> tag=sam_ntfy
+INFO apprise notify OK HTTP 200 tag=sam_ntfy: Voice intake recovered
+$ # no further alerts once healthy
+```
+
+**3. Broker down → names the broker, not the ESP32**
+
+```
+$ # watchdog pointed at an unreachable broker address
+WARNING ALERT kind=alert layer=broker -> tag=sam_ntfy
+TITLE: Voice intake DOWN — .13 MQTT broker unreachable
+... Do NOT power-cycle the ESP32 — the broker layer failed, not the device.
+```
+
+**4. Daily repeat (shortened interval)**
+
+```
+$ # SILENCE_TIMEOUT=1 REPEAT_INTERVAL=6 CHECK_INTERVAL=2, dry-run
+ALERT kind=alert  layer=publisher -> tag=sam_ntfy   (first)
+ALERT kind=repeat layer=publisher -> tag=sam       (after the interval)
+```
+
+The unit tests also drive three consecutive daily repeats
+(`test_daily_repeats_keep_firing`).
+
+## Not in scope / boundaries
+
+- No auto-heal of any kind.
+- The `.13:1883` plane is untouched (plain, anonymous). The watchdog connects as
+  a normal subscriber and publishes nothing.
+- The broker was **not** stopped to prove layer attribution; an unreachable
+  broker address exercises the same code path without disturbing n8n or the
+  kids' 8883 devices.