hub_fixes/AGENTS.md

Hub Fixes

Catch-all for fixes and hardening to the Hub and the services that talk to it. Each directive below is a self-contained task. Run them one at a time.

Build spec for Paperclip. Follow each directive's DoD. Stop and ask if something is unclear.

---

1. Purpose and outcome

The Hub is live and the projects around it are running. This project fixes the things that break in that system — the failures that go unnoticed for days.

Outcome: when something in the voice/automation path dies, Sam finds out the same day — and the same class of bug cannot recur silently.

2. Scope

In scope

• Watchdogs and alerting for the Hub and the voice path.
• MQTT reliability across every client.
• Firmware fixes for the ESP32 (in sam/voice_assistant).

Out of scope

• New Hub features — those belong in central_ai_llm_master.
• The workstream projects (email, school, internet control, Artemis).

3. Context — the outage this exists for

On 2026-10-13 the shared .13 Mosquitto restarted (internet-control added the 8883 TLS listener). Two things then failed silently:

1. voice_bridge went deaf. It subscribed to MQTT once at startup with no on_connect handler. paho reconnected but never restored the subscription, so the bridge held a live socket with no subscription — no error, no log. Fixed 2026-10-14: added an on_connect re-subscribe.
2. The ESP32 stopped publishing. Green LED (Wi-Fi joined) but nothing on voice/audio_stream. A power-cycle fixed it. Same bug class, still to do.

Neither was noticed for two days. Sam discovered it by talking to the house. That is the failure this project fixes.

---

Directive 1 — Voice intake watchdog (REQUIRED)

Requirement

• Detect when voice intake is dead: no message on voice/audio_stream (.13 broker) for 15 minutes.
• Send an alert via Apprise to Sam.
• Repeat once a day while it stays broken. Not once — daily, until fixed.
• The alert must say what failed and what to do:

  "Voice intake silent since HH:MM. Likely the ESP32 — try a power-cycle."

• Do NOT auto-heal. No smart plug, no automatic power-cycling. Sam does that by hand. (This was considered and rejected — keep it a notification.)
• Do not blame the ESP32 if something else broke. Check the broker and the bridge first, and say which layer failed.

DoD

• Stop voice_bridge → an Apprise alert arrives within 15 minutes.
• Leave it stopped → the alert repeats the next day, and the day after.
• Start it again → the alerts stop.
• Stop the broker instead → the alert names the broker, not the ESP32.

---

Directive 2 — MQTT reconnect rule (REQUIRED)

Every MQTT client in the lab must reconnect AND re-subscribe. paho reconnects on its own but does not restore subscriptions — the client then holds a live socket with no subscription and goes silently deaf.

Confirmed instances so far:

Client	Status
voice_bridge	✅ fixed 2026-10-14 (on_connect re-subscribe)
ESP32 firmware	❌ to do — see Directive 3
Hub	check
internet_use_control service	check

DoD

• Restart the .13 broker. Within 60 seconds, every client is receiving messages again — verified, not assumed.
• Every client has an on_connect handler that re-subscribes, and logs when it does.

---

Directive 3 — ESP32 firmware (repo: sam/voice_assistant)

The unit shows green (Wi-Fi joined) while publishing nothing. Green is misleading.

• Auto-reconnect MQTT and re-subscribe on every (re)connect.
• Publish a heartbeat (e.g. every 5 min) so silence is a reliable signal — the watchdog in Directive 1 depends on this.
• Show MQTT state on the LED. Green must mean connected and publishing, not merely on Wi-Fi.
• Log the reason on the serial console when the broker connection drops.

DoD

• Restart the broker. The ESP32 reconnects and publishes again within 60 seconds without a power-cycle.
• Pull the Wi-Fi access point. The LED stops showing green.

---

4. Rules

• Models go through OmniRoute. Secrets from environment.d.
• Alerts go through Apprise (.35:8210).
• Never change the .13 broker's 1883 plane — it is plain and anonymous, and n8n and the voice path depend on it. Kids' devices use 8883 only.
• One commit and push per directive.
• Bound every docker command (timeout, --entrypoint sh when running a shell inside an image).

5. Open questions

1. Where does the watchdog live — a small service, or an n8n flow, or part of the Hub?
2. Alert target: which Apprise channel (email / ntfy / WhatsApp)?
3. How long should silence be tolerated before alerting? (Suggest 15 min.)
4. Is the ESP32 on a smart plug anyway? (Still no auto-heal — but useful to know.)