commit d1503eccbb9d0374db99d486ef197ee76712ac2d Author: Sam Rolfe Date: Wed Oct 14 10:40:16 2026 +1100 Initial: hub fixes spec (voice watchdog, MQTT reconnect rule, ESP32 firmware) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..c5bdf13 --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +.env +*.local +__pycache__/ +node_modules/ +.pi/memory-vault/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..af6a65b --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,129 @@ +# Hub Fixes + +> Catch-all for fixes and hardening to the Hub and the services that talk to it. +> Each directive below is a self-contained task. Run them one at a time. +> +> Build spec for Paperclip. Follow each directive's DoD. Stop and ask if something +> is unclear. + +--- + +## 1. Purpose and outcome + +The Hub is live and the projects around it are running. This project fixes the +things that *break* in that system — the failures that go unnoticed for days. + +**Outcome:** when something in the voice/automation path dies, Sam finds out the +same day — and the same class of bug cannot recur silently. + +## 2. Scope + +**In scope** +- Watchdogs and alerting for the Hub and the voice path. +- MQTT reliability across every client. +- Firmware fixes for the ESP32 (in `sam/voice_assistant`). + +**Out of scope** +- New Hub features — those belong in `central_ai_llm_master`. +- The workstream projects (email, school, internet control, Artemis). + +## 3. Context — the outage this exists for + +On **2026-10-13** the shared `.13` Mosquitto restarted (internet-control added the +8883 TLS listener). Two things then failed **silently**: + +1. **`voice_bridge` went deaf.** It subscribed to MQTT once at startup with no + `on_connect` handler. paho reconnected but never restored the subscription, so + the bridge held a live socket with no subscription — no error, no log. + *Fixed 2026-10-14:* added an `on_connect` re-subscribe. +2. **The ESP32 stopped publishing.** Green LED (Wi-Fi joined) but nothing on + `voice/audio_stream`. A power-cycle fixed it. Same bug class, still to do. + +**Neither was noticed for two days.** Sam discovered it by talking to the house. +That is the failure this project fixes. + +--- + +## Directive 1 — Voice intake watchdog (REQUIRED) + +**Requirement** + +- Detect when voice intake is dead: **no message on `voice/audio_stream`** (`.13` + broker) for **15 minutes**. +- Send an alert **via Apprise to Sam**. +- **Repeat once a day while it stays broken.** Not once — daily, until fixed. +- The alert must say *what failed* and *what to do*: + > "Voice intake silent since HH:MM. Likely the ESP32 — try a power-cycle." +- **Do NOT auto-heal.** No smart plug, no automatic power-cycling. Sam does that + by hand. (This was considered and rejected — keep it a notification.) +- **Do not blame the ESP32 if something else broke.** Check the broker and the + bridge first, and say *which* layer failed. + +**DoD** + +- Stop `voice_bridge` → an Apprise alert arrives within 15 minutes. +- Leave it stopped → the alert repeats the next day, and the day after. +- Start it again → the alerts stop. +- Stop the broker instead → the alert names the **broker**, not the ESP32. + +--- + +## Directive 2 — MQTT reconnect rule (REQUIRED) + +Every MQTT client in the lab must **reconnect AND re-subscribe**. paho reconnects +on its own but does **not** restore subscriptions — the client then holds a live +socket with no subscription and goes silently deaf. + +Confirmed instances so far: + +| Client | Status | +|---|---| +| `voice_bridge` | ✅ fixed 2026-10-14 (`on_connect` re-subscribe) | +| ESP32 firmware | ❌ to do — see Directive 3 | +| Hub | check | +| `internet_use_control` service | check | + +**DoD** + +- Restart the `.13` broker. Within 60 seconds, **every** client is receiving + messages again — verified, not assumed. +- Every client has an `on_connect` handler that re-subscribes, and logs when it does. + +--- + +## Directive 3 — ESP32 firmware (repo: `sam/voice_assistant`) + +The unit shows **green** (Wi-Fi joined) while publishing nothing. Green is +misleading. + +- [ ] **Auto-reconnect MQTT and re-subscribe on every (re)connect.** +- [ ] **Publish a heartbeat** (e.g. every 5 min) so silence is a reliable signal — + the watchdog in Directive 1 depends on this. +- [ ] **Show MQTT state on the LED.** Green must mean *connected and publishing*, + not merely *on Wi-Fi*. +- [ ] Log the reason on the serial console when the broker connection drops. + +**DoD** + +- Restart the broker. The ESP32 reconnects and publishes again within 60 seconds + **without a power-cycle**. +- Pull the Wi-Fi access point. The LED stops showing green. + +--- + +## 4. Rules + +- Models go through **OmniRoute**. Secrets from `environment.d`. +- Alerts go through **Apprise** (`.35:8210`). +- Never change the `.13` broker's **1883** plane — it is plain and anonymous, and + n8n and the voice path depend on it. Kids' devices use **8883** only. +- One commit and push per directive. +- Bound every docker command (`timeout`, `--entrypoint sh` when running a shell + inside an image). + +## 5. Open questions + +1. Where does the watchdog live — a small service, or an n8n flow, or part of the Hub? +2. Alert target: which Apprise channel (email / ntfy / WhatsApp)? +3. How long should silence be tolerated before alerting? (Suggest 15 min.) +4. Is the ESP32 on a smart plug anyway? (Still no auto-heal — but useful to know.)