sam-4screen-desktop 2026-10-6:11:21:28

300 areas/360 Dev-Ops Network Computers/Error Tracking.md

@@ -1,6 +1,6 @@
 ---
 created: 2026-09-06
-modified: 2026-09-06
+modified: 2026-10-06
 title: Error Tracking
 summary: Log of network/system errors encountered and their fixes
 type: area
@@ -117,3 +117,59 @@ Three failures in one chain.
 
 ### Note
 The agent MQTT intake now retries and reconnects. A future broker flap heals itself. Report the LED state to exercise the chain end to end. Keepalive timeouts on ESP32 suggest a weak WiFi link; consider a stable keepalive or fixed install if it recurs.
+
+---
+
+# 2026-10-05 — Music stack silent: Mopidy has no Spotify, "NixOS-Speaker" missing from Spotify
+
+## Symptom
+- No audio from any Snapcast speaker.
+- The Spotify client did not list the `NixOS-Speaker` Connect device.
+- The Mopidy / Iris Spotify library was empty.
+
+## Root cause
+One network fault caused two failures.
+
+- The `.13` reboot at 20:16 broke DNS. NetworkManager assumed `enp42s0` and wrote a profile without the gateway. The IPv4 default route dropped.
+- `librespot` could not reach `clienttoken.spotify.com`. systemd restarted it **6,361 times** between 20:16 and 05:32.
+- `mopidy` refreshes its Spotify OAuth token **once**, at backend start. The refresh failed at 20:16:14 and never retried. `mopidy.service` stayed `active` with a dead Spotify backend for 10.5 hours.
+
+`idle` on all Snapcast streams is normal when nothing plays. It is not a fault indicator.
+
+## Fix
+Restart the FIFO writers first, then Snapserver.
+
+```bash
+# Mopidy Spotify. This alone restored the outage.
+sudo systemctl restart mopidy
+
+# Use this instead if Snapcast is also silent.
+sudo systemctl restart librespot mopidy && sleep 3 && sudo systemctl restart snapserver
+```
+
+If Spotify stays empty after the restart, the stored refresh token was revoked. Re-run the mopidy-spotify OAuth flow (`mopidy spotify`).
+
+## Verify
+```bash
+systemctl is-active snapserver mopidy librespot
+journalctl -u mopidy -b | grep -iE 'Logged into Spotify|OAuth'
+journalctl -u librespot -b | grep -c 'Main process exited'   # expect 0
+```
+
+Journal signatures on `.13`:
+
+| Log line | Meaning |
+|---|---|
+| `OAuth token refresh failed: Unknown error.` | Spotify backend dead. Restart Mopidy. |
+| `Failed to load Spotify user profile` | Same fault, next line. |
+| `Logged into Spotify Web API as guzzbinkles` | Healthy. |
+| `Refreshed 152 Spotify playlists` | Healthy, library loaded. |
+
+## Notes
+- Root cause fixed in commit `47f956e`: `networkmanager.unmanaged = [ "enp42s0" ]` plus `name_servers` in `resolvconf.extraConfig`.
+- Ports on `.13`: 1704 Snapcast, 1705 control, 1780 web/JSON-RPC, 4953 announcements, 57621 librespot Connect handshake.
+- **Open, low impact:** `.13` publishes no `_spotify-connect._tcp` mDNS record. librespot's `libmdns` and `avahi-daemon` both bind UDP 5353. Spotify Connect still worked through Spotify's cloud, so no change was made.
+- **Open, cosmetic:** boot ordering still starts `snapserver` before its FIFO writers. It logs `End of file, length: 0` at boot and self-heals.
+
+## Related notes
+- [[Home Network Map Overview]]