hub_fixes/watchdog

Voice intake watchdog (Hub Fixes — Directive 1)

Watches the house voice path and alerts Sam through Apprise when intake dies. It never heals anything — no smart plug, no power-cycle, no restart. Notification only. Sam fixes the failed layer by hand.

This is the fix for the 2026-10-13 outage: the .13 broker restarted, voice_bridge went deaf and the ESP32 stopped publishing, and nobody noticed for two days.

What it watches

"Voice intake alive" means all three layers are healthy, checked in this order:

Layer	Signal	Failure message says
broker	.13:1883 reachable (live MQTT session or TCP probe)	check the Mosquitto container; do not power-cycle the ESP32
bridge	voice_bridge container running and subscribed to voice/audio_stream (Docker socket)	start the bridge; do not power-cycle the ESP32
publisher	a message on voice/audio_stream within SILENCE_TIMEOUT_SECONDS, or retained voice/status presence	"Likely the ESP32 — try a power-cycle."

Layer attribution is the point: if something else broke, the ESP32 is not blamed.

Why 15 minutes is meaningful

The Directive 3 firmware publishes a heartbeat on voice/audio_stream every 5 minutes, so 15 minutes of silence is three missed beats — not "nobody spoke". The firmware also sets a retained voice/status presence (online on connect, offline as a Last Will) for immediate attribution.

Until a heartbeat or presence has been seen, the watchdog does not treat silence as a publisher failure (that state means the heartbeat firmware is not flashed yet, and silence is normal). This is controlled by REQUIRE_PUBLISHER_EVIDENCE (default 1).

Repeat policy

• First failure → alert immediately.
• Same layer still down → repeat once per REPEAT_INTERVAL_SECONDS (default one day), for as long as it stays broken.
• A different layer fails → that is a new alert (immediate).
• Recovery → one recovery notice, then the state is cleared and alerts stop.
• State is persisted in a Docker volume, so a watchdog restart neither re-spams nor forgets an ongoing outage.

First alerts use tag APPRISE_TAG (sam_ntfy); daily repeats escalate to APPRISE_TAG_REPEAT (sam). Both live in the Apprise config on .35.

Files

watchdog/
  watchdog.py            the service (paho + stdlib only)
  Dockerfile             python:3.12-slim + paho-mqtt==2.1.0
  docker-compose.yml     host network, read-only Docker socket, state volume
  start_watchdog.sh      sources ~/.config/environment.d/10-secrets.conf, then compose up
  tests/test_watchdog.py unit tests for classification, repeats, messages

Deploy

cd /home/sam/paseo/projects/hub_fixes/watchdog
./start_watchdog.sh                      # normal operation
DRY_RUN=1 ./start_watchdog.sh            # log alerts, send nothing
SILENCE_TIMEOUT_SECONDS=60 CHECK_INTERVAL_SECONDS=5 ./start_watchdog.sh   # test mode

It runs as container voice_intake_watchdog, network_mode: host, no listening port, restart: unless-stopped. The Docker socket is mounted read-only and only ever read (/containers/voice_bridge/json and its logs).

Test

python3 -m unittest discover -s watchdog/tests -v

Verify (evidence)

All commands below were run on .13 on 2026-10-14.

1. Stop voice_bridge → alert arrives within 15 minutes

$ docker stop voice_bridge
$ # watchdog, real Apprise, 5 s checks
WARNING ALERT kind=alert layer=bridge -> tag=sam_ntfy
INFO apprise notify OK HTTP 200 tag=sam_ntfy: Voice intake DOWN — voice_bridge is not running

The bridge layer is checked directly, so the alert fires on the next check (seconds), well inside the 15-minute budget.

2. Start it again → alerts stop

$ docker start voice_bridge
WARNING ALERT kind=recovery layer=bridge -> tag=sam_ntfy
INFO apprise notify OK HTTP 200 tag=sam_ntfy: Voice intake recovered
$ # no further alerts once healthy

3. Broker down → names the broker, not the ESP32

$ # watchdog pointed at an unreachable broker address
WARNING ALERT kind=alert layer=broker -> tag=sam_ntfy
TITLE: Voice intake DOWN — .13 MQTT broker unreachable
... Do NOT power-cycle the ESP32 — the broker layer failed, not the device.

4. Daily repeat (shortened interval)

$ # SILENCE_TIMEOUT=1 REPEAT_INTERVAL=6 CHECK_INTERVAL=2, dry-run
ALERT kind=alert  layer=publisher -> tag=sam_ntfy   (first)
ALERT kind=repeat layer=publisher -> tag=sam       (after the interval)

The unit tests also drive three consecutive daily repeats (test_daily_repeats_keep_firing).

Not in scope / boundaries

• No auto-heal of any kind.
• The .13:1883 plane is untouched (plain, anonymous). The watchdog connects as a normal subscriber and publishes nothing.
• The broker was not stopped to prove layer attribution; an unreachable broker address exercises the same code path without disturbing n8n or the kids' 8883 devices.