# Hub Fixes

> Catch-all for fixes and hardening to the Hub and the services that talk to it.
> Each directive below is a self-contained task. Run them one at a time.
>
> Build spec for Paperclip. Follow each directive's DoD. Stop and ask if something
> is unclear.

---

## 1. Purpose and outcome

The Hub is live and the projects around it are running. This project fixes the
things that *break* in that system — the failures that go unnoticed for days.

**Outcome:** when something in the voice/automation path dies, Sam finds out the
same day — and the same class of bug cannot recur silently.

## 2. Scope

**In scope**
- Watchdogs and alerting for the Hub and the voice path.
- MQTT reliability across every client.
- Firmware fixes for the ESP32 (in `sam/voice_assistant`).

**Out of scope**
- New Hub features — those belong in `central_ai_llm_master`.
- The workstream projects (email, school, internet control, Artemis).

## 3. Context — the outage this exists for

On **2026-10-13** the shared `.13` Mosquitto restarted (internet-control added the
8883 TLS listener). Two things then failed **silently**:

1. **`voice_bridge` went deaf.** It subscribed to MQTT once at startup with no
   `on_connect` handler. paho reconnected but never restored the subscription, so
   the bridge held a live socket with no subscription — no error, no log.
   *Fixed 2026-10-14:* added an `on_connect` re-subscribe.
2. **The ESP32 stopped publishing.** Green LED (Wi-Fi joined) but nothing on
   `voice/audio_stream`. A power-cycle fixed it. Same bug class, still to do.

**Neither was noticed for two days.** Sam discovered it by talking to the house.
That is the failure this project fixes.

---

## Directive 1 — Voice intake watchdog (REQUIRED)

**Requirement**

- Detect when voice intake is dead: **no message on `voice/audio_stream`** (`.13`
  broker) for **15 minutes**.
- Send an alert **via Apprise to Sam**.
- **Repeat once a day while it stays broken.** Not once — daily, until fixed.
- The alert must say *what failed* and *what to do*:
  > "Voice intake silent since HH:MM. Likely the ESP32 — try a power-cycle."
- **Do NOT auto-heal.** No smart plug, no automatic power-cycling. Sam does that
  by hand. (This was considered and rejected — keep it a notification.)
- **Do not blame the ESP32 if something else broke.** Check the broker and the
  bridge first, and say *which* layer failed.

**DoD**

- Stop `voice_bridge` → an Apprise alert arrives within 15 minutes.
- Leave it stopped → the alert repeats the next day, and the day after.
- Start it again → the alerts stop.
- Stop the broker instead → the alert names the **broker**, not the ESP32.

---

## Directive 2 — MQTT reconnect rule (REQUIRED)

Every MQTT client in the lab must **reconnect AND re-subscribe**. paho reconnects
on its own but does **not** restore subscriptions — the client then holds a live
socket with no subscription and goes silently deaf.

Confirmed instances so far:

| Client | Status |
|---|---|
| `voice_bridge` | ✅ fixed 2026-10-14 (`on_connect` re-subscribe) |
| ESP32 firmware | ❌ to do — see Directive 3 |
| Hub | check |
| `internet_use_control` service | check |

**DoD**

- Restart the `.13` broker. Within 60 seconds, **every** client is receiving
  messages again — verified, not assumed.
- Every client has an `on_connect` handler that re-subscribes, and logs when it does.

---

## Directive 3 — ESP32 firmware (repo: `sam/voice_assistant`)

The unit shows **green** (Wi-Fi joined) while publishing nothing. Green is
misleading.

- [ ] **Auto-reconnect MQTT and re-subscribe on every (re)connect.**
- [ ] **Publish a heartbeat** (e.g. every 5 min) so silence is a reliable signal —
      the watchdog in Directive 1 depends on this.
- [ ] **Show MQTT state on the LED.** Green must mean *connected and publishing*,
      not merely *on Wi-Fi*.
- [ ] Log the reason on the serial console when the broker connection drops.

**DoD**

- Restart the broker. The ESP32 reconnects and publishes again within 60 seconds
  **without a power-cycle**.
- Pull the Wi-Fi access point. The LED stops showing green.

---

## 4. Rules

- Models go through **OmniRoute**. Secrets from `environment.d`.
- Alerts go through **Apprise** (`.35:8210`).
- Never change the `.13` broker's **1883** plane — it is plain and anonymous, and
  n8n and the voice path depend on it. Kids' devices use **8883** only.
- One commit and push per directive.
- Bound every docker command (`timeout`, `--entrypoint sh` when running a shell
  inside an image).

## 5. Open questions

1. Where does the watchdog live — a small service, or an n8n flow, or part of the Hub?
2. Alert target: which Apprise channel (email / ntfy / WhatsApp)?
3. How long should silence be tolerated before alerting? (Suggest 15 min.)
4. Is the ESP32 on a smart plug anyway? (Still no auto-heal — but useful to know.)