Initial: hub fixes spec (voice watchdog, MQTT reconnect rule, ESP32 firmware)
This commit is contained in:
129
AGENTS.md
Normal file
129
AGENTS.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Hub Fixes
|
||||
|
||||
> Catch-all for fixes and hardening to the Hub and the services that talk to it.
|
||||
> Each directive below is a self-contained task. Run them one at a time.
|
||||
>
|
||||
> Build spec for Paperclip. Follow each directive's DoD. Stop and ask if something
|
||||
> is unclear.
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose and outcome
|
||||
|
||||
The Hub is live and the projects around it are running. This project fixes the
|
||||
things that *break* in that system — the failures that go unnoticed for days.
|
||||
|
||||
**Outcome:** when something in the voice/automation path dies, Sam finds out the
|
||||
same day — and the same class of bug cannot recur silently.
|
||||
|
||||
## 2. Scope
|
||||
|
||||
**In scope**
|
||||
- Watchdogs and alerting for the Hub and the voice path.
|
||||
- MQTT reliability across every client.
|
||||
- Firmware fixes for the ESP32 (in `sam/voice_assistant`).
|
||||
|
||||
**Out of scope**
|
||||
- New Hub features — those belong in `central_ai_llm_master`.
|
||||
- The workstream projects (email, school, internet control, Artemis).
|
||||
|
||||
## 3. Context — the outage this exists for
|
||||
|
||||
On **2026-10-13** the shared `.13` Mosquitto restarted (internet-control added the
|
||||
8883 TLS listener). Two things then failed **silently**:
|
||||
|
||||
1. **`voice_bridge` went deaf.** It subscribed to MQTT once at startup with no
|
||||
`on_connect` handler. paho reconnected but never restored the subscription, so
|
||||
the bridge held a live socket with no subscription — no error, no log.
|
||||
*Fixed 2026-10-14:* added an `on_connect` re-subscribe.
|
||||
2. **The ESP32 stopped publishing.** Green LED (Wi-Fi joined) but nothing on
|
||||
`voice/audio_stream`. A power-cycle fixed it. Same bug class, still to do.
|
||||
|
||||
**Neither was noticed for two days.** Sam discovered it by talking to the house.
|
||||
That is the failure this project fixes.
|
||||
|
||||
---
|
||||
|
||||
## Directive 1 — Voice intake watchdog (REQUIRED)
|
||||
|
||||
**Requirement**
|
||||
|
||||
- Detect when voice intake is dead: **no message on `voice/audio_stream`** (`.13`
|
||||
broker) for **15 minutes**.
|
||||
- Send an alert **via Apprise to Sam**.
|
||||
- **Repeat once a day while it stays broken.** Not once — daily, until fixed.
|
||||
- The alert must say *what failed* and *what to do*:
|
||||
> "Voice intake silent since HH:MM. Likely the ESP32 — try a power-cycle."
|
||||
- **Do NOT auto-heal.** No smart plug, no automatic power-cycling. Sam does that
|
||||
by hand. (This was considered and rejected — keep it a notification.)
|
||||
- **Do not blame the ESP32 if something else broke.** Check the broker and the
|
||||
bridge first, and say *which* layer failed.
|
||||
|
||||
**DoD**
|
||||
|
||||
- Stop `voice_bridge` → an Apprise alert arrives within 15 minutes.
|
||||
- Leave it stopped → the alert repeats the next day, and the day after.
|
||||
- Start it again → the alerts stop.
|
||||
- Stop the broker instead → the alert names the **broker**, not the ESP32.
|
||||
|
||||
---
|
||||
|
||||
## Directive 2 — MQTT reconnect rule (REQUIRED)
|
||||
|
||||
Every MQTT client in the lab must **reconnect AND re-subscribe**. paho reconnects
|
||||
on its own but does **not** restore subscriptions — the client then holds a live
|
||||
socket with no subscription and goes silently deaf.
|
||||
|
||||
Confirmed instances so far:
|
||||
|
||||
| Client | Status |
|
||||
|---|---|
|
||||
| `voice_bridge` | ✅ fixed 2026-10-14 (`on_connect` re-subscribe) |
|
||||
| ESP32 firmware | ❌ to do — see Directive 3 |
|
||||
| Hub | check |
|
||||
| `internet_use_control` service | check |
|
||||
|
||||
**DoD**
|
||||
|
||||
- Restart the `.13` broker. Within 60 seconds, **every** client is receiving
|
||||
messages again — verified, not assumed.
|
||||
- Every client has an `on_connect` handler that re-subscribes, and logs when it does.
|
||||
|
||||
---
|
||||
|
||||
## Directive 3 — ESP32 firmware (repo: `sam/voice_assistant`)
|
||||
|
||||
The unit shows **green** (Wi-Fi joined) while publishing nothing. Green is
|
||||
misleading.
|
||||
|
||||
- [ ] **Auto-reconnect MQTT and re-subscribe on every (re)connect.**
|
||||
- [ ] **Publish a heartbeat** (e.g. every 5 min) so silence is a reliable signal —
|
||||
the watchdog in Directive 1 depends on this.
|
||||
- [ ] **Show MQTT state on the LED.** Green must mean *connected and publishing*,
|
||||
not merely *on Wi-Fi*.
|
||||
- [ ] Log the reason on the serial console when the broker connection drops.
|
||||
|
||||
**DoD**
|
||||
|
||||
- Restart the broker. The ESP32 reconnects and publishes again within 60 seconds
|
||||
**without a power-cycle**.
|
||||
- Pull the Wi-Fi access point. The LED stops showing green.
|
||||
|
||||
---
|
||||
|
||||
## 4. Rules
|
||||
|
||||
- Models go through **OmniRoute**. Secrets from `environment.d`.
|
||||
- Alerts go through **Apprise** (`.35:8210`).
|
||||
- Never change the `.13` broker's **1883** plane — it is plain and anonymous, and
|
||||
n8n and the voice path depend on it. Kids' devices use **8883** only.
|
||||
- One commit and push per directive.
|
||||
- Bound every docker command (`timeout`, `--entrypoint sh` when running a shell
|
||||
inside an image).
|
||||
|
||||
## 5. Open questions
|
||||
|
||||
1. Where does the watchdog live — a small service, or an n8n flow, or part of the Hub?
|
||||
2. Alert target: which Apprise channel (email / ntfy / WhatsApp)?
|
||||
3. How long should silence be tolerated before alerting? (Suggest 15 min.)
|
||||
4. Is the ESP32 on a smart plug anyway? (Still no auto-heal — but useful to know.)
|
||||
Reference in New Issue
Block a user