---
created: 2026-09-06
modified: 2026-10-06
title: Error Tracking
summary: Log of network/system errors encountered and their fixes
type: area
client: sam
project: devops
status: active
priority: 5
---

# Error Tracking

A running log of errors encountered across the network, with root causes and fixes. Append new entries as they occur.

---

# 2026-09-06 — Router: "Another management entity is configuring the Broadband Router"

## Symptom
While in the router admin (192.168.20.1), saw:

> Broadband Router Error — Another management entity is currently configuring the Broadband Router. Please close the Broadband Router Configuration window, and try again in a few minutes.

Error appeared repeatedly; also coincided with `.13` losing all network connectivity (wired + Wi-Fi + Tailscale unreachable from `.27`).

## Root cause
Not a second admin session and not a rogue device — **`.13`'s own flapping Wi-Fi adapter (`wlp4s0`) was seen by the router as a "management entity"**:

- `.13` has **two MACs / two IPs on the router table** (root of the confusion):
  - **Wired** `enp42s0` = MAC **`34:5a:60:a4:cc:8c`** → `192.168.20.13` (LAN)
  - **Wi-Fi** `wlp4s0` = MAC **`a0:f3:c1:d0:e3:2b`** → `192.168.20.46` (2.4G)
- Wi-Fi kept going DORMANT / re-registering and the default route flipped onto it → router interpreted the churn as a second configuring device.
- `sudo reboot` on `.13` cleared it. Error has **not returned** since.

## Verification / non-causes
- Router login log showed **no login except `.27`** → no foreign admin session.
- Client table showed only known devices (ESP32s, Pis, `.27`/`.35`/`.51`/`.13`, etc.).
- Not caused by Langfuse/Clickhouse removal (unrelated; those never touch networking).

## Potential fix — pin the Wi-Fi MAC address
If the error (or Wi-Fi flakiness on `.13`) recurs, the likely fix is to stop the Wi-Fi adapter's MAC churn:

- On `.13`, disable **MAC address randomization** in NetworkManager for `wlp4s0`, forcing a stable MAC so the router sees one consistent identity per interface:
  ```
  # /etc/NetworkManager/conf.d/ (or via home-manager)
  [connection]
  wifi.cloned-mac-address=permanent   # or "stable"
  # per-interface: wifi.mac-randomization-disabled
  ```
- On NixOS this is done in `hardware-configuration.nix` / `networking` options (e.g. `networking.wireless` or NetworkManager `connection` settings in home-manager), or simply disable the Wi-Fi interface entirely if wired is preferred: `sudo nmcli radio wifi off`.

## Related notes
- `.13` network layout: `enp42s0` = wired `.13`, `wlp4s0` = Wi-Fi `.46` (2.4G). Default route prefers Wi-Fi (lower metric) — worth reviewing if flakiness returns.
- `192.168.80.0/20` bridge on `.13` is normal Docker (`wherewoof-db_default` subnet) — not an issue.
## 2026-09-06 — Voice agent: "feed the fish" did nothing, LED red

### Symptom
The voice command "feed the fish" returned no action. The ESP32 LED went red.

### Root cause
Two failures blocked the request path.

- OmniRoute combo `voice-fast` used old model IDs. The new IDs are `openai/*`.
  The combo returned 503 "all accounts inactive".
- The agent MQTT intake connected once. A network flap broke that connection.
  The agent did not retry, so it lost the ESP32 text feed.

### Fix
- Recreated the `voice-fast` combo with three healthy OpenRouter models.
- Changed `app.py` MQTT intake to retry and reconnect. Backup: `app.py.bak-mqtt-20260906`.
- Run `sudo systemctl restart voice-agent` to load the change.
- DeepSeek key is valid but the account balance is zero. Add credit at platform.deepseek.com.

---

# 2026-09-09 — Garage S3 web returns 404/403 for site images (anonymous hosting)

## Symptom
A Go + htmx site (being built for `kontra.day`) tried to serve uploaded images through **Garage S3 web** (`:3902`) and got `404` on the bucket root and `403 Forbidden: Garage does not support anonymous access yet` in `docker logs family-home-lab-garage-1`.

## Root cause
Two things misled the diagnosis:

1. **Wrong claim (from an earlier AI):** it said `[s3_web]` was absent and the web server wasn't running. **False** — `[s3_web]` IS in `garage.toml` (`bind_addr = "[::]:3902"`, `root_domain = ".web.lab.audasmedia.com.au"`, `index = "index.html"`) and the log shows `Web server listening on http://[::]:3902`. Verified: `ss -tln` shows 3902 bound (Garage web server), 3903 admin, 3909 garage-webui.
2. **Real issue:** Garage S3 Web is **not anonymous static hosting**. It requires **per-bucket web access + a read-scoped access key**. Without it, every request gets `403 Forbidden: Garage does not support anonymous access yet`, and the bucket root 404s with no index/perm.

## Fix / correct pattern
- **Do NOT** point site/app code at `:3902` for anonymous browsing.
- **Use the S3 API** (`:3900`, region `garage`) with a **read-scoped key** for uploads/reads.
- **Serve images** through the app or **Caddy** (reverse-proxy to Garage with the read key, or a Caddy `file_server` static dir) — not via `:3902` directly.
- If web-serving a bucket is truly needed, enable **per-bucket web access + a guest/read key** in Garage (not a global switch).

## Verification / non-causes
- `[s3_web]` present; web server confirmed running (log line + port 3902 bound) → the earlier "no s3_web / not running" diagnosis was wrong.
- Port map: 3900 S3 API · 3902 S3 web · 3903 admin API · 3909 garage-webui. No port conflict.
- Documented in the Garage section of [[Backup System — Borg, Kopia & Restic]] and the kontra.day site handoff.

## 2026-09-14 — Voice: ESP32 green, never blue; then no response

### Symptom
The ESP32 LED turned green but never blue. The bridge, whisper, and broker were up.

### Root cause
Three failures in one chain.

- The voice_bridge OpenWakeWord process was wedged. It used 0.01% CPU and ignored audio.
- The ESP32 MQTT client was in a stale state. The broker dropped it for keepalive timeouts. It published no audio.
- The voice-agent process ran for 7 days without the MQTT reconnect patch. Journald restarted and lost the agent stdout socket, so the agent logs vanished. The agent never re-subscribed to the .30 topic.

### Fix
- Restart the bridge container: `docker restart voice_bridge`
- Power-cycle the ESP32
- Restart the agent service: `sudo systemctl restart voice-agent`
- Verify: journal shows `[mqtt] listening on 192.168.20.30:1883 homeassistant/voice/text`

### Note
The agent MQTT intake now retries and reconnects. A future broker flap heals itself. Report the LED state to exercise the chain end to end. Keepalive timeouts on ESP32 suggest a weak WiFi link; consider a stable keepalive or fixed install if it recurs.

---

# 2026-10-05 — Music stack silent: Mopidy has no Spotify, "NixOS-Speaker" missing from Spotify

## Symptom
- No audio from any Snapcast speaker.
- The Spotify client did not list the `NixOS-Speaker` Connect device.
- The Mopidy / Iris Spotify library was empty.

## Root cause
One network fault caused two failures.

- The `.13` reboot at 20:16 broke DNS. NetworkManager assumed `enp42s0` and wrote a profile without the gateway. The IPv4 default route dropped.
- `librespot` could not reach `clienttoken.spotify.com`. systemd restarted it **6,361 times** between 20:16 and 05:32.
- `mopidy` refreshes its Spotify OAuth token **once**, at backend start. The refresh failed at 20:16:14 and never retried. `mopidy.service` stayed `active` with a dead Spotify backend for 10.5 hours.

`idle` on all Snapcast streams is normal when nothing plays. It is not a fault indicator.

## Fix
Restart the FIFO writers first, then Snapserver.

```bash
# Mopidy Spotify. This alone restored the outage.
sudo systemctl restart mopidy

# Use this instead if Snapcast is also silent.
sudo systemctl restart librespot mopidy && sleep 3 && sudo systemctl restart snapserver
```

If Spotify stays empty after the restart, the stored refresh token was revoked. Re-run the mopidy-spotify OAuth flow (`mopidy spotify`).

## Verify
```bash
systemctl is-active snapserver mopidy librespot
journalctl -u mopidy -b | grep -iE 'Logged into Spotify|OAuth'
journalctl -u librespot -b | grep -c 'Main process exited'   # expect 0
```

Journal signatures on `.13`:

| Log line | Meaning |
|---|---|
| `OAuth token refresh failed: Unknown error.` | Spotify backend dead. Restart Mopidy. |
| `Failed to load Spotify user profile` | Same fault, next line. |
| `Logged into Spotify Web API as guzzbinkles` | Healthy. |
| `Refreshed 152 Spotify playlists` | Healthy, library loaded. |

## Notes
- Root cause fixed in commit `47f956e`: `networkmanager.unmanaged = [ "enp42s0" ]` plus `name_servers` in `resolvconf.extraConfig`.
- Ports on `.13`: 1704 Snapcast, 1705 control, 1780 web/JSON-RPC, 4953 announcements, 57621 librespot Connect handshake.
- **Open, low impact:** `.13` publishes no `_spotify-connect._tcp` mDNS record. librespot's `libmdns` and `avahi-daemon` both bind UDP 5353. Spotify Connect still worked through Spotify's cloud, so no change was made.
- **Open, cosmetic:** boot ordering still starts `snapserver` before its FIFO writers. It logs `End of file, length: 0` at boot and self-heals.

## Related notes
- [[Home Network Map Overview]]