sam/paperclip-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

environment facts: pi alignment done by a pinned package input on .27 and .51

Browse Source
This commit is contained in:
Sam
2026-11-12 14:28:15 +11:00
parent 5f8069d857
commit b81cb91831
11 changed files with 1001 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
.pi/memory-vault/packs/paperclip-harness/80-sessions/nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static.md Normal file
View File

@@ -0,0 +1,52 @@
---
type: session
id: session.paperclip-harness.nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static
title: NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race
status: active
source_of_truth: false
freshness: current
last_reviewed: 2026-11-12
tags:
- pack/paperclip-harness
- agent-memory/session
- nixos
- networking
- networkmanager
- enp42s0
- nixos-rebuild
- postmortem
- 192.168.20.13
---
# NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race
Host `nixos-desktop` at 192.168.20.13 (NixOS 26.11, system `/etc/nixos`) went offline after an in-place `nixos-rebuild switch` following a nixpkgs bump.
Facts discovered:
- `nixos-rebuild switch` restarts every changed unit. Because of the nixpkgs bump, this included `NetworkManager.service` and `wpa_supplicant.service`.
- The machine uses legacy static `networking.interfaces` + `defaultGateway` for `enp42s0` (192.168.20.13/24, gateway 192.168.20.1) simultaneously with `networkmanager.unmanaged`; the config at `configuration.nix` line ~334 already comments that this combination is unsupported and works only by boot-order luck.
- The restart triggered `network-addresses-enp42s0.service` pre-stop (deletes address/route), start (adds address and `ip -4 route replace default proto static via 192.168.20.1`), then the NetworkManager restart flushed the route again, leaving no default route.
- Exit code 4 from `nixos-rebuild` did NOT indicate a build failure; the generation switched, but units failed to start because the route was gone.
- Repaired externally (by another AI) with a runtime `ip route add` plus restart of librespot. Post-repair state: pi 1.5.0, current-system `...nixos-system-nixos-desktop-26.11.20261107.f4a03c0`, default via 192.168.20.1, no failed units, librespot/snapserver/NetworkManager/docker/postgresql/tailscaled active, Paperclip `/api/health` 200, tuios sessions restored.
Known-good recommendation (agreed, not yet executed): let NetworkManager own `enp42s0` with a manual static profile (.13/24, gateway .1, DNS .35/.13/1.1.1.1) and remove `networkmanager.unmanaged` + legacy `interfaces`/`defaultGateway` lines. Must be validated at runtime with `nmcli` before writing to config.
Other operator notes: use `nixos-rebuild boot` for large changes on live servers; `.27` and `.51` get default routes via DHCP so the .13 failure mode cannot repeat; `.27`'s flake.lock is currently bumped to a non-full-system-evaluable state and should be reverted to `flake.lock.bak-pi-align` before anyone runs a rebuild.
## Evidence
- Confidence: 90%
- Reason: Rich discovery snapshot: concrete failure sequence, journal evidence, recommended fix, and machine state that future agents on this repo should know.
## Related
- [[packs/paperclip-harness/00-system/pi-agent/memory-manifest|Memory Manifest]]
## Related learned memories <!-- same-turn-2026-11-12 03:14:45 -->
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|Pi coding agent 1.5.0 builds standalone from cache; .27 and .51 don't need a full system bump]]
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Apply large NixOS changes on the live Paperclip/Docker server (.13)]]
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|use `nixos-rebuild boot` for wide changes on .13; hand enp42s0 to NetworkManager static profile]]
- session: [[packs/paperclip-harness/80-sessions/nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static|NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race]]
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-35-2026-11-12-03-14-45|Rich discovery: .35 2026-11-12 03:14:45]]
- context: [[packs/paperclip-harness/20-context/nix|.35]]

131
.pi/memory-vault/packs/paperclip-harness/80-sessions/rich-discovery-35-2026-11-12-03-14-45.md Normal file
View File

@@ -0,0 +1,131 @@
---
type: session
id: session.paperclip-harness.rich-discovery-35-2026-11-12-03-14-45
title: Rich discovery: .35 2026-11-12 03:14:45
status: active
source_of_truth: false
freshness: current
last_reviewed: 2026-11-12
tags:
- pack/paperclip-harness
- agent-memory/session
- autolearn
- rich-discovery
- repo/35
---
# Rich discovery: .35 2026-11-12 03:14:45
Sanitized rich discovery snapshot captured automatically from the completed turn.
## Source paths observed
- `etc/nixos/hosts/sam-4screen-desktop/configuration.nix`
- `./home/sam/home.nix`
- `Chrome/Spotify/VSCode`
- `./hosts/sam-thinkpad/configuration.nix`
- `nix/store/02h6ym8j8qvxpjps56n9gx4gxqpg26a0-source`
- `etc/nixos/flake.lock.bak-pi-align`
- `etc/nixos/home.nix.bak-pi-align`
- `etc/nixos/home/sam/home.nix.bak-pi-align`
- `nix/store/3kfmdqn711fqcc2y1qnjssancqf7b36l-nodejs-24.22.2`
- `nix/store/4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9-pi-coding-agent-1.5.0`
- `etc/systemd/system/network-addresses-enp42s0.service`
- `nix/store/qrz69l7h7d78vf8qgjbflq263ddymfss-unit-network-addresses-enp42s0.service/network-addresses-enp42s0.service`
- `nix/store/rbvrn9kch96rr5nm7jdp3913d51nm09g-glibc-locales-2.44-57/lib/locale/locale-archive`
- `nix/store/wc6ky2sqq8shrmaf14x7spbdpqi6ls9w-iproute2-7.2.0/bin`
- `nix/store/a50ml7za6h61m4mv4cdbvkl33c6sx5yn-coreutils-9.12/bin`
- `nix/store/5zsc280i2i7bqgjn7kln87w1ss6li22f-findutils-4.11.0/bin`
- `nix/store/5sgjjiqn90zmvlvlgadw7ss00nn3hpjb-gnugrep-3.12/bin`
- `nix/store/x275j18p7dx6kv7vfj45saapbhbljia4-gnused-4.10/bin`
- `nix/store/kh0kk4mi6hr0cdvsc8rzd0jvr2c2vhwk-systemd-262/bin`
- `nix/store/wc6ky2sqq8shrmaf14x7spbdpqi6ls9w-iproute2-7.2.0/sbin`
- `nix/store/a50ml7za6h61m4mv4cdbvkl33c6sx5yn-coreutils-9.12/sbin`
- `nix/store/5zsc280i2i7bqgjn7kln87w1ss6li22f-findutils-4.11.0/sbin`
- `nix/store/5sgjjiqn90zmvlvlgadw7ss00nn3hpjb-gnugrep-3.12/sbin`
- `nix/store/x275j18p7dx6kv7vfj45saapbhbljia4-gnused-4.10/sbin`
- `nix/store/kh0kk4mi6hr0cdvsc8rzd0jvr2c2vhwk-systemd-262/sbin`
- `nix/store/s92hlmg73b3zlw38dfgm13v9cah2bm80-tzdata-2026f/share/zoneinfo`
- `nix/store/4p9n1mpnjahlm9j51kal5a7nmj7b8v88-unit-script-network-addresses-enp42s0-start/bin/network-addresses-enp42s0-start`
- `nix/store/3przkjya9ba0x1y40slkr2g8wkrva8ym-unit-script-network-addresses-enp42s0-pre-stop/bin/network-addresses-enp42s0-pre-stop`
- `nix/store/hasnvc0j36gg6qrdsfcx7yivx3c2bpqr-bash-5.3p20/bin/bash`
- `run/nixos/network/addresses/enp42s0`
- `run/nixos/network/routes/enp42s0`
- `.35/.13/1.1.1.1`
## Final answer snapshot
I broke it, and here is what actually happened. No metaphors, just the sequence.
## What I did wrong
`nixos-rebuild switch` on a machine that hosts Paperclip, Docker and every TUIOS session **restarts every changed unit in place**. A nixpkgs bump changes a great many units. I checked that the config *built* and never asked what it would *restart*. It restarted the network, the network did not come back, and everything on `.13` that needs a route went down with it. The correct verb for a change that large on a live server is `nixos-rebuild boot`: it builds, sets the next generation, then stops. You reboot when you choose. I should have used that, and I will from now on.
The exit code 4 was **not** a build failure. The generation did switch. Exit 4 means units failed to start, and the units failed because the route was gone.
## `.13` right now, measured
```
pi --version 1.5.0
/run/current-system x0yghxby0zwpyi8l15gw8gr8jwm5n4ir-nixos-system-nixos-desktop-26.11.20261107.f4a03c0
ip route show default default via 192.168.20.1 dev enp42s0 metric 100
systemctl --failed (nothing)
librespot / snapserver / NetworkManager / docker / postgresql / tailscaled active
internet, .27 reachable
Paperclip /api/health 200
tuios list-sessions sessions restored
```
So the system is healthy and on the same Pi as planned. The repair was the other AI's, not mine: a runtime `ip route add` plus a restart of librespot. I am telling you that plainly because I did not fix it.
## The root cause, from the journal
```
12:51:46 Stopping Address configuration of enp42s0... ← pre-stop deletes address and route
12:51:47 adding address 192.168.20.13/24... done
12:51:47 setting enp42s0 as default IPv4 gateway... done ← route WAS set
(nothing after this for that unit)
```
The route was set at 12:51:47 and disappeared afterwards. What ran between then and the failure was the restart of `NetworkManager.service` and `wpa_supplicant.service`. Your own comment in `configuration.nix` at line 334 already names this: static `interfaces` + `defaultGateway` with `networkmanager.unmanaged` is a combination NixOS warns is unsupported, and it worked "only by boot-order luck". The reboot-strength change destroyed the luck.
Two managers on one interface is the bug. Not the nixpkgs bump.
## What I recommend for `.13`
Let NetworkManager own `enp42s0` with a static profile (manual, `.13/24`, gateway `.1`, DNS `.35/.13/1.1.1.1`), and delete `networkmanager.unmanaged` plus the legacy `interfaces`/`defaultGateway` lines. One manager, no flush race, no boot-order luck. This is also what the other AI suggested and I agree with it.
The risk to be honest about: if the NetworkManager profile is wrong, the machine loses its address on the next boot. So it must be tested **at runtime with `nmcli` first** — bring the profile up by hand, prove the route and DNS work, then write it into the config. Not the other way round.
## `.27` and `.51`: the Pi part needs no system bump
This is the useful discovery. On `.27`, with the bumped pin already there, the Pi package alone builds in **6 seconds** from `cache.nixos.org`:
```
/nix/store/4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9-pi-coding-agent-1.5.0
```
That is the exact store path `.13` runs. So `.27` and `.51` can have the *same bytes* of Pi without a whole-system bump, and without touching the home-manager font problem that blocks `.27`'s full rebuild. `.13`'s failure mode also cannot repeat on them: both take their default route from DHCP through NetworkManager (`proto dhcp`), not from a legacy static address.
Also: `.27`'s `flake.lock` is currently bumped into a state that will not evaluate to a full system. I should revert it to `flake.lock.bak-pi-align` so the machine is not sitting on a landmine for whoever runs a rebuild next.
Say which you want and I will do only that.
## Evidence
- Confidence: 86%
- Reason: Preserve full sanitized discovery detail from a rich investigation turn
## Related
- [[packs/paperclip-harness/00-system/pi-agent/memory-manifest|Memory Manifest]]
## Related learned memories <!-- same-turn-2026-11-12 03:14:45 -->
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|Pi coding agent 1.5.0 builds standalone from cache; .27 and .51 don't need a full system bump]]
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Apply large NixOS changes on the live Paperclip/Docker server (.13)]]
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|use `nixos-rebuild boot` for wide changes on .13; hand enp42s0 to NetworkManager static profile]]
- session: [[packs/paperclip-harness/80-sessions/nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static|NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race]]
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-35-2026-11-12-03-14-45|Rich discovery: .35 2026-11-12 03:14:45]]
- context: [[packs/paperclip-harness/20-context/nix|.35]]

104
.pi/memory-vault/packs/paperclip-harness/80-sessions/rich-discovery-nix-2026-11-12-01-50-43.md Normal file
View File

@@ -0,0 +1,104 @@
---
type: session
id: session.paperclip-harness.rich-discovery-nix-2026-11-12-01-50-43
title: Rich discovery: nix 2026-11-12 01:50:43
status: active
source_of_truth: false
freshness: current
last_reviewed: 2026-11-12
tags:
- pack/paperclip-harness
- agent-memory/session
- autolearn
- rich-discovery
- repo/nix
---
# Rich discovery: nix 2026-11-12 01:50:43
Sanitized rich discovery snapshot captured automatically from the completed turn.
## Source paths observed
- `nix/store/91ghzifbssyi53z9w3c2v2nliami4ry1-lan-mouse-0.11.0.drv`
- `nix/store/9vrmdr2wnp6y8529alkyb55cimhrlcd9-lan-mouse-0.11.0.drv`
- `nix/store/y0h5slyxhmkikga74pnaz0ybadwaji7b-zzz-0.11.0.drv`
- `avahi/services/snapcast.service`
- `nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0`
- `gitea.lab.audasmedia.com.au/sam/paperclip-harness.git`
## Final answer snapshot
The rebuild failed because my first fix patched the wrong lan-mouse. It is fixed properly now, and the derivation that caused the failure is no longer in the system.
## What was wrong
Your `configuration.nix` used **`lan-mouse` from nixpkgs**, while I had overridden the **flake input's** copy. So the build kept using the unpatched package — which is why the failing derivation was byte-for-byte the same one, `9vrmdr…`, before and after my change.
I then tried an overlay on `pkgs.lan-mouse`, and that did not work either: the panic returned with the same message, because nixpkgs sets `LIBGIT2_NO_VENDOR` in a way the overlay could not remove.
## What works, proven
The flake's own lan-mouse package, built with the variable turned off:
```nix
lm = lan-mouse.packages.${system}.default.overrideAttrs (o: { LIBGIT2_NO_VENDOR = "0"; });
```
Built it before wiring it in:
```
/nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0
```
Then changed two lines: `configuration.nix` now takes `lm` as an argument and uses it in `systemPackages` where `lan-mouse` was.
Result on `.13`:
```
these 12 derivations will be built:
pi: 1.5.0
```
Twelve derivations, not 465 — most of the rest had already been substituted during the earlier attempts. Your rebuild should now go through:
```sh
sudo nixos-rebuild switch --flake /etc/nixos#nixos-desktop
```
If it fails on something else, send the first error block; the method is now established: build the offending package alone with an override before switching.
## About the font
Thank you for the pointer — it may well be the cause on `.27`. Its error names `home-manager.users.sam.fonts.fontconfig.configFile.fonts.settings`, and `.27` declares fonts in two places (system `fonts.packages` and again inside `systemPackages`). With home-manager 2026-11-10 that duplication is what it cannot merge. That is a small fix, but I want `.13` proven first so we are changing one thing at a time.
## State
| | config | evaluates | running | next |
|---|---|---|---|---|
| `.13` | lock, `initContent`, lan-mouse via `lm` | pi **1.5.0**, 12 to build | 0.82.1 | **rebuild now** |
| `.27` | lock, `initContent` | fails in home-manager fontconfig | 0.82.1 | fix the font duplication, or bump nixpkgs only |
| `.51` | unchanged | pi 0.87.1 | 0.87.1 | the same three edits, then rebuild |
Written into `docs/ENVIRONMENT-FACTS.md`, committed as `5f8069d`, pushed.
Tell me what `pi --version` says after the rebuild and I will bring `.27` and `.51` to the same version.
## Evidence
- Confidence: 86%
- Reason: Preserve full sanitized discovery detail from a rich investigation turn
## Related
- [[packs/paperclip-harness/00-system/pi-agent/memory-manifest|Memory Manifest]]
## Related learned memories <!-- same-turn-2026-11-12 01:50:43 -->
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|lan-mouse LIBGIT2_NO_VENDOR build failure: nixpkgs vs flake package mismatch]]
- decision: [[packs/paperclip-harness/50-decisions/wire-lan-mouse-into-configuration-nix-as-flake-lm-override-n|Wire lan-mouse into configuration.nix as flake `lm` override, not nixpkgs package or overlay]]
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|NixOS fleet upgrade runbook: rebuild `.13` first, fix font duplication on `.27`, then `.51`]]
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|Fixed .13 lan-mouse build, updated docs/ENVIRONMENT-FACTS.md, committed 5f8069d and pushed]]
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-nix-2026-11-12-01-50-43|Rich discovery: nix 2026-11-12 01:50:43]]
- context: [[packs/paperclip-harness/20-context/nix|nix]]