130 lines
4.8 KiB
Markdown
130 lines
4.8 KiB
Markdown
# Hub Fixes
|
|
|
|
> Catch-all for fixes and hardening to the Hub and the services that talk to it.
|
|
> Each directive below is a self-contained task. Run them one at a time.
|
|
>
|
|
> Build spec for Paperclip. Follow each directive's DoD. Stop and ask if something
|
|
> is unclear.
|
|
|
|
---
|
|
|
|
## 1. Purpose and outcome
|
|
|
|
The Hub is live and the projects around it are running. This project fixes the
|
|
things that *break* in that system — the failures that go unnoticed for days.
|
|
|
|
**Outcome:** when something in the voice/automation path dies, Sam finds out the
|
|
same day — and the same class of bug cannot recur silently.
|
|
|
|
## 2. Scope
|
|
|
|
**In scope**
|
|
- Watchdogs and alerting for the Hub and the voice path.
|
|
- MQTT reliability across every client.
|
|
- Firmware fixes for the ESP32 (in `sam/voice_assistant`).
|
|
|
|
**Out of scope**
|
|
- New Hub features — those belong in `central_ai_llm_master`.
|
|
- The workstream projects (email, school, internet control, Artemis).
|
|
|
|
## 3. Context — the outage this exists for
|
|
|
|
On **2026-10-13** the shared `.13` Mosquitto restarted (internet-control added the
|
|
8883 TLS listener). Two things then failed **silently**:
|
|
|
|
1. **`voice_bridge` went deaf.** It subscribed to MQTT once at startup with no
|
|
`on_connect` handler. paho reconnected but never restored the subscription, so
|
|
the bridge held a live socket with no subscription — no error, no log.
|
|
*Fixed 2026-10-14:* added an `on_connect` re-subscribe.
|
|
2. **The ESP32 stopped publishing.** Green LED (Wi-Fi joined) but nothing on
|
|
`voice/audio_stream`. A power-cycle fixed it. Same bug class, still to do.
|
|
|
|
**Neither was noticed for two days.** Sam discovered it by talking to the house.
|
|
That is the failure this project fixes.
|
|
|
|
---
|
|
|
|
## Directive 1 — Voice intake watchdog (REQUIRED)
|
|
|
|
**Requirement**
|
|
|
|
- Detect when voice intake is dead: **no message on `voice/audio_stream`** (`.13`
|
|
broker) for **15 minutes**.
|
|
- Send an alert **via Apprise to Sam**.
|
|
- **Repeat once a day while it stays broken.** Not once — daily, until fixed.
|
|
- The alert must say *what failed* and *what to do*:
|
|
> "Voice intake silent since HH:MM. Likely the ESP32 — try a power-cycle."
|
|
- **Do NOT auto-heal.** No smart plug, no automatic power-cycling. Sam does that
|
|
by hand. (This was considered and rejected — keep it a notification.)
|
|
- **Do not blame the ESP32 if something else broke.** Check the broker and the
|
|
bridge first, and say *which* layer failed.
|
|
|
|
**DoD**
|
|
|
|
- Stop `voice_bridge` → an Apprise alert arrives within 15 minutes.
|
|
- Leave it stopped → the alert repeats the next day, and the day after.
|
|
- Start it again → the alerts stop.
|
|
- Stop the broker instead → the alert names the **broker**, not the ESP32.
|
|
|
|
---
|
|
|
|
## Directive 2 — MQTT reconnect rule (REQUIRED)
|
|
|
|
Every MQTT client in the lab must **reconnect AND re-subscribe**. paho reconnects
|
|
on its own but does **not** restore subscriptions — the client then holds a live
|
|
socket with no subscription and goes silently deaf.
|
|
|
|
Confirmed instances so far:
|
|
|
|
| Client | Status |
|
|
|---|---|
|
|
| `voice_bridge` | ✅ fixed 2026-10-14 (`on_connect` re-subscribe) |
|
|
| ESP32 firmware | ❌ to do — see Directive 3 |
|
|
| Hub | check |
|
|
| `internet_use_control` service | check |
|
|
|
|
**DoD**
|
|
|
|
- Restart the `.13` broker. Within 60 seconds, **every** client is receiving
|
|
messages again — verified, not assumed.
|
|
- Every client has an `on_connect` handler that re-subscribes, and logs when it does.
|
|
|
|
---
|
|
|
|
## Directive 3 — ESP32 firmware (repo: `sam/voice_assistant`)
|
|
|
|
The unit shows **green** (Wi-Fi joined) while publishing nothing. Green is
|
|
misleading.
|
|
|
|
- [ ] **Auto-reconnect MQTT and re-subscribe on every (re)connect.**
|
|
- [ ] **Publish a heartbeat** (e.g. every 5 min) so silence is a reliable signal —
|
|
the watchdog in Directive 1 depends on this.
|
|
- [ ] **Show MQTT state on the LED.** Green must mean *connected and publishing*,
|
|
not merely *on Wi-Fi*.
|
|
- [ ] Log the reason on the serial console when the broker connection drops.
|
|
|
|
**DoD**
|
|
|
|
- Restart the broker. The ESP32 reconnects and publishes again within 60 seconds
|
|
**without a power-cycle**.
|
|
- Pull the Wi-Fi access point. The LED stops showing green.
|
|
|
|
---
|
|
|
|
## 4. Rules
|
|
|
|
- Models go through **OmniRoute**. Secrets from `environment.d`.
|
|
- Alerts go through **Apprise** (`.35:8210`).
|
|
- Never change the `.13` broker's **1883** plane — it is plain and anonymous, and
|
|
n8n and the voice path depend on it. Kids' devices use **8883** only.
|
|
- One commit and push per directive.
|
|
- Bound every docker command (`timeout`, `--entrypoint sh` when running a shell
|
|
inside an image).
|
|
|
|
## 5. Open questions
|
|
|
|
1. Where does the watchdog live — a small service, or an n8n flow, or part of the Hub?
|
|
2. Alert target: which Apprise channel (email / ntfy / WhatsApp)?
|
|
3. How long should silence be tolerated before alerting? (Suggest 15 min.)
|
|
4. Is the ESP32 on a smart plug anyway? (Still no auto-heal — but useful to know.)
|