environment facts: pi alignment done by a pinned package input on .27 and .51
This commit is contained in:
Sam
213
.pi/memory-vault/packs/paperclip-harness/20-context/nix.md
Normal file
213
.pi/memory-vault/packs/paperclip-harness/20-context/nix.md
Normal file
@@ -0,0 +1,213 @@
|
|||||||
|
---
|
||||||
|
type: context
|
||||||
|
id: context.paperclip-harness.nix
|
||||||
|
title: nix
|
||||||
|
status: active
|
||||||
|
source_of_truth: false
|
||||||
|
freshness: current
|
||||||
|
last_reviewed: 2026-11-12
|
||||||
|
tags:
|
||||||
|
- pack/paperclip-harness
|
||||||
|
- agent-memory/context
|
||||||
|
- autolearn
|
||||||
|
- rich-discovery
|
||||||
|
- repo/nix
|
||||||
|
---
|
||||||
|
|
||||||
|
# nix
|
||||||
|
|
||||||
|
## Rich discovery update (2026-11-12 01:50:43)
|
||||||
|
|
||||||
|
The rebuild failed because my first fix patched the wrong lan-mouse. It is fixed properly now, and the derivation that caused the failure is no longer in the system.
|
||||||
|
|
||||||
|
## What was wrong
|
||||||
|
|
||||||
|
Your `configuration.nix` used **`lan-mouse` from nixpkgs**, while I had overridden the **flake input's** copy. So the build kept using the unpatched package — which is why the failing derivation was byte-for-byte the same one, `9vrmdr…`, before and after my change.
|
||||||
|
|
||||||
|
I then tried an overlay on `pkgs.lan-mouse`, and that did not work either: the panic returned with the same message, because nixpkgs sets `LIBGIT2_NO_VENDOR` in a way the overlay could not remove.
|
||||||
|
|
||||||
|
## What works, proven
|
||||||
|
|
||||||
|
The flake's own lan-mouse package, built with the variable turned off:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
lm = lan-mouse.packages.${system}.default.overrideAttrs (o: { LIBGIT2_NO_VENDOR = "0"; });
|
||||||
|
```
|
||||||
|
|
||||||
|
Built it before wiring it in:
|
||||||
|
|
||||||
|
```
|
||||||
|
/nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Then changed two lines: `configuration.nix` now takes `lm` as an argument and uses it in `systemPackages` where `lan-mouse` was.
|
||||||
|
|
||||||
|
Result on `.13`:
|
||||||
|
|
||||||
|
```
|
||||||
|
these 12 derivations will be built:
|
||||||
|
pi: 1.5.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Twelve derivations, not 465 — most of the rest had already been substituted during the earlier attempts. Your rebuild should now go through:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo nixos-rebuild switch --flake /etc/nixos#nixos-desktop
|
||||||
|
```
|
||||||
|
|
||||||
|
If it fails on something else, send the first error block; the method is now established: build the offending package alone with an override before switching.
|
||||||
|
|
||||||
|
## About the font
|
||||||
|
|
||||||
|
Thank you for the pointer — it may well be the cause on `.27`. Its error names `home-manager.users.sam.fonts.fontconfig.configFile.fonts.settings`, and `.27` declares fonts in two places (system `fonts.packages` and again inside `systemPackages`). With home-manager 2026-11-10 that duplication is what it cannot merge. That is a small fix, but I want `.13` proven first so we are changing one thing at a time.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
| | config | evaluates | running | next |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `.13` | lock, `initContent`, lan-mouse via `lm` | pi **1.5.0**, 12 to build | 0.82.1 | **rebuild now** |
|
||||||
|
| `.27` | lock, `initContent` | fails in home-manager fontconfig | 0.82.1 | fix the font duplication, or bump nixpkgs only |
|
||||||
|
| `.51` | unchanged | pi 0.87.1 | 0.87.1 | the same three edits, then rebuild |
|
||||||
|
|
||||||
|
Written into `docs/ENVIRONMENT-FACTS.md`, committed as `5f8069d`, pushed.
|
||||||
|
|
||||||
|
Tell me what `pi --version` says after the rebuild and I will bring `.27` and `.51` to the same version.
|
||||||
|
|
||||||
|
## Source paths observed
|
||||||
|
|
||||||
|
- `nix/store/91ghzifbssyi53z9w3c2v2nliami4ry1-lan-mouse-0.11.0.drv`
|
||||||
|
- `nix/store/9vrmdr2wnp6y8529alkyb55cimhrlcd9-lan-mouse-0.11.0.drv`
|
||||||
|
- `nix/store/y0h5slyxhmkikga74pnaz0ybadwaji7b-zzz-0.11.0.drv`
|
||||||
|
- `avahi/services/snapcast.service`
|
||||||
|
- `nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0`
|
||||||
|
- `gitea.lab.audasmedia.com.au/sam/paperclip-harness.git`
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 84%
|
||||||
|
- Reason: Update component/repository context with detailed discovered structure and configuration
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[packs/paperclip-harness/00-system/pi-agent/memory-manifest|Memory Manifest]]
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 01:50:43 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|lan-mouse LIBGIT2_NO_VENDOR build failure: nixpkgs vs flake package mismatch]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/wire-lan-mouse-into-configuration-nix-as-flake-lm-override-n|Wire lan-mouse into configuration.nix as flake `lm` override, not nixpkgs package or overlay]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|NixOS fleet upgrade runbook: rebuild `.13` first, fix font duplication on `.27`, then `.51`]]
|
||||||
|
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|Fixed .13 lan-mouse build, updated docs/ENVIRONMENT-FACTS.md, committed 5f8069d and pushed]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-nix-2026-11-12-01-50-43|Rich discovery: nix 2026-11-12 01:50:43]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|nix]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 03:14:45)
|
||||||
|
|
||||||
|
## Rich discovery update (2026-11-12 03:14:45)
|
||||||
|
|
||||||
|
I broke it, and here is what actually happened. No metaphors, just the sequence.
|
||||||
|
|
||||||
|
## What I did wrong
|
||||||
|
|
||||||
|
`nixos-rebuild switch` on a machine that hosts Paperclip, Docker and every TUIOS session **restarts every changed unit in place**. A nixpkgs bump changes a great many units. I checked that the config *built* and never asked what it would *restart*. It restarted the network, the network did not come back, and everything on `.13` that needs a route went down with it. The correct verb for a change that large on a live server is `nixos-rebuild boot`: it builds, sets the next generation, then stops. You reboot when you choose. I should have used that, and I will from now on.
|
||||||
|
|
||||||
|
The exit code 4 was **not** a build failure. The generation did switch. Exit 4 means units failed to start, and the units failed because the route was gone.
|
||||||
|
|
||||||
|
## `.13` right now, measured
|
||||||
|
|
||||||
|
```
|
||||||
|
pi --version 1.5.0
|
||||||
|
/run/current-system x0yghxby0zwpyi8l15gw8gr8jwm5n4ir-nixos-system-nixos-desktop-26.11.20261107.f4a03c0
|
||||||
|
ip route show default default via 192.168.20.1 dev enp42s0 metric 100
|
||||||
|
systemctl --failed (nothing)
|
||||||
|
librespot / snapserver / NetworkManager / docker / postgresql / tailscaled active
|
||||||
|
internet, .27 reachable
|
||||||
|
Paperclip /api/health 200
|
||||||
|
tuios list-sessions sessions restored
|
||||||
|
```
|
||||||
|
|
||||||
|
So the system is healthy and on the same Pi as planned. The repair was the other AI's, not mine: a runtime `ip route add` plus a restart of librespot. I am telling you that plainly because I did not fix it.
|
||||||
|
|
||||||
|
## The root cause, from the journal
|
||||||
|
|
||||||
|
```
|
||||||
|
12:51:46 Stopping Address configuration of enp42s0... ← pre-stop deletes address and route
|
||||||
|
12:51:47 adding address 192.168.20.13/24... done
|
||||||
|
12:51:47 setting enp42s0 as default IPv4 gateway... done ← route WAS set
|
||||||
|
(nothing after this for that unit)
|
||||||
|
```
|
||||||
|
|
||||||
|
The route was set at 12:51:47 and disappeared afterwards. What ran between then and the failure was the restart of `NetworkManager.service` and `wpa_supplicant.service`. Your own comment in `configuration.nix` at line 334 already names this: static `interfaces` + `defaultGateway` with `networkmanager.unmanaged` is a combination NixOS warns is unsupported, and it worked "only by boot-order luck". The reboot-strength change destroyed the luck.
|
||||||
|
|
||||||
|
Two managers on one interface is the bug. Not the nixpkgs bump.
|
||||||
|
|
||||||
|
## What I recommend for `.13`
|
||||||
|
|
||||||
|
Let NetworkManager own `enp42s0` with a static profile (manual, `.13/24`, gateway `.1`, DNS `.35/.13/1.1.1.1`), and delete `networkmanager.unmanaged` plus the legacy `interfaces`/`defaultGateway` lines. One manager, no flush race, no boot-order luck. This is also what the other AI suggested and I agree with it.
|
||||||
|
|
||||||
|
The risk to be honest about: if the NetworkManager profile is wrong, the machine loses its address on the next boot. So it must be tested **at runtime with `nmcli` first** — bring the profile up by hand, prove the route and DNS work, then write it into the config. Not the other way round.
|
||||||
|
|
||||||
|
## `.27` and `.51`: the Pi part needs no system bump
|
||||||
|
|
||||||
|
This is the useful discovery. On `.27`, with the bumped pin already there, the Pi package alone builds in **6 seconds** from `cache.nixos.org`:
|
||||||
|
|
||||||
|
```
|
||||||
|
/nix/store/4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9-pi-coding-agent-1.5.0
|
||||||
|
```
|
||||||
|
|
||||||
|
That is the exact store path `.13` runs. So `.27` and `.51` can have the *same bytes* of Pi without a whole-system bump, and without touching the home-manager font problem that blocks `.27`'s full rebuild. `.13`'s failure mode also cannot repeat on them: both take their default route from DHCP through NetworkManager (`proto dhcp`), not from a legacy static address.
|
||||||
|
|
||||||
|
Also: `.27`'s `flake.lock` is currently bumped into a state that will not evaluate to a full system. I should revert it to `flake.lock.bak-pi-align` so the machine is not sitting on a landmine for whoever runs a rebuild next.
|
||||||
|
|
||||||
|
Say which you want and I will do only that.
|
||||||
|
|
||||||
|
## Source paths observed
|
||||||
|
|
||||||
|
- `etc/nixos/hosts/sam-4screen-desktop/configuration.nix`
|
||||||
|
- `./home/sam/home.nix`
|
||||||
|
- `Chrome/Spotify/VSCode`
|
||||||
|
- `./hosts/sam-thinkpad/configuration.nix`
|
||||||
|
- `nix/store/02h6ym8j8qvxpjps56n9gx4gxqpg26a0-source`
|
||||||
|
- `etc/nixos/flake.lock.bak-pi-align`
|
||||||
|
- `etc/nixos/home.nix.bak-pi-align`
|
||||||
|
- `etc/nixos/home/sam/home.nix.bak-pi-align`
|
||||||
|
- `nix/store/3kfmdqn711fqcc2y1qnjssancqf7b36l-nodejs-24.22.2`
|
||||||
|
- `nix/store/4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9-pi-coding-agent-1.5.0`
|
||||||
|
- `etc/systemd/system/network-addresses-enp42s0.service`
|
||||||
|
- `nix/store/qrz69l7h7d78vf8qgjbflq263ddymfss-unit-network-addresses-enp42s0.service/network-addresses-enp42s0.service`
|
||||||
|
- `nix/store/rbvrn9kch96rr5nm7jdp3913d51nm09g-glibc-locales-2.44-57/lib/locale/locale-archive`
|
||||||
|
- `nix/store/wc6ky2sqq8shrmaf14x7spbdpqi6ls9w-iproute2-7.2.0/bin`
|
||||||
|
- `nix/store/a50ml7za6h61m4mv4cdbvkl33c6sx5yn-coreutils-9.12/bin`
|
||||||
|
- `nix/store/5zsc280i2i7bqgjn7kln87w1ss6li22f-findutils-4.11.0/bin`
|
||||||
|
- `nix/store/5sgjjiqn90zmvlvlgadw7ss00nn3hpjb-gnugrep-3.12/bin`
|
||||||
|
- `nix/store/x275j18p7dx6kv7vfj45saapbhbljia4-gnused-4.10/bin`
|
||||||
|
- `nix/store/kh0kk4mi6hr0cdvsc8rzd0jvr2c2vhwk-systemd-262/bin`
|
||||||
|
- `nix/store/wc6ky2sqq8shrmaf14x7spbdpqi6ls9w-iproute2-7.2.0/sbin`
|
||||||
|
- `nix/store/a50ml7za6h61m4mv4cdbvkl33c6sx5yn-coreutils-9.12/sbin`
|
||||||
|
- `nix/store/5zsc280i2i7bqgjn7kln87w1ss6li22f-findutils-4.11.0/sbin`
|
||||||
|
- `nix/store/5sgjjiqn90zmvlvlgadw7ss00nn3hpjb-gnugrep-3.12/sbin`
|
||||||
|
- `nix/store/x275j18p7dx6kv7vfj45saapbhbljia4-gnused-4.10/sbin`
|
||||||
|
- `nix/store/kh0kk4mi6hr0cdvsc8rzd0jvr2c2vhwk-systemd-262/sbin`
|
||||||
|
- `nix/store/s92hlmg73b3zlw38dfgm13v9cah2bm80-tzdata-2026f/share/zoneinfo`
|
||||||
|
- `nix/store/4p9n1mpnjahlm9j51kal5a7nmj7b8v88-unit-script-network-addresses-enp42s0-start/bin/network-addresses-enp42s0-start`
|
||||||
|
- `nix/store/3przkjya9ba0x1y40slkr2g8wkrva8ym-unit-script-network-addresses-enp42s0-pre-stop/bin/network-addresses-enp42s0-pre-stop`
|
||||||
|
- `nix/store/hasnvc0j36gg6qrdsfcx7yivx3c2bpqr-bash-5.3p20/bin/bash`
|
||||||
|
- `run/nixos/network/addresses/enp42s0`
|
||||||
|
- `run/nixos/network/routes/enp42s0`
|
||||||
|
- `.35/.13/1.1.1.1`
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 84%
|
||||||
|
- Reason: Update component/repository context with detailed discovered structure and configuration
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 03:14:45 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|Pi coding agent 1.5.0 builds standalone from cache; .27 and .51 don't need a full system bump]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Apply large NixOS changes on the live Paperclip/Docker server (.13)]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|use `nixos-rebuild boot` for wide changes on .13; hand enp42s0 to NetworkManager static profile]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static|NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-35-2026-11-12-03-14-45|Rich discovery: .35 2026-11-12 03:14:45]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|.35]]
|
||||||
@@ -1127,3 +1127,32 @@ Follow-up: after .13 reboots into the new generation, report `pi --version`; the
|
|||||||
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Fix a single package build before a NixOS rebuild, then switch]]
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Fix a single package build before a NixOS rebuild, then switch]]
|
||||||
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|office CLI wrappers installed; office bugs fixed; .13 rebuild failure fixed and pushed]]
|
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|office CLI wrappers installed; office bugs fixed; .13 rebuild failure fixed and pushed]]
|
||||||
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-local-2026-11-12-01-11-14|Rich discovery: .local 2026-11-12 01:11:14]]
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-local-2026-11-12-01-11-14|Rich discovery: .local 2026-11-12 01:11:14]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 01:50:43)
|
||||||
|
|
||||||
|
Deliverable summary: resolved the `.13` rebuild blocker by defining `lm` in `flake.nix` as the flake's `lan-mouse` package with `LIBGIT2_NO_VENDOR = "0"` via `overrideAttrs`, and switching `configuration.nix`'s `systemPackages` entry from `lan-mouse` (nixpkgs) to `lm`, removing the non-working overlay.
|
||||||
|
|
||||||
|
Key artifacts: `flake.nix` (`lm` definition), `configuration.nix` (argument + `systemPackages` line ~571, overlay removed; existing comment at line 182 notes lan-mouse on `.13` needs a compositor), `docs/ENVIRONMENT-FACTS.md` (documented state of `.13`/`.27`/`.51`).
|
||||||
|
|
||||||
|
Validation/result: standalone build produced `/nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0`; full system dry-run on `.13` showed 12 derivations to build (vs ~465 earlier), with pi 1.5.0. Not yet switched — user to run `sudo nixos-rebuild switch --flake /etc/nixos#nixos-desktop`.
|
||||||
|
|
||||||
|
Repo: `https://gitea.lab.audasmedia.com.au/sam/paperclip-harness.git`, commit `5f8069d` (`a129c4e..5f8069d main -> main`), pushed.
|
||||||
|
|
||||||
|
Follow-up: `.27` font-duplication fix; `.51` three edits + rebuild.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 90%
|
||||||
|
- Reason: Completed, committed, pushed change with artifacts and explicit pending validation.
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 01:50:43 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|lan-mouse LIBGIT2_NO_VENDOR build failure: nixpkgs vs flake package mismatch]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/wire-lan-mouse-into-configuration-nix-as-flake-lm-override-n|Wire lan-mouse into configuration.nix as flake `lm` override, not nixpkgs package or overlay]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|NixOS fleet upgrade runbook: rebuild `.13` first, fix font duplication on `.27`, then `.51`]]
|
||||||
|
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|Fixed .13 lan-mouse build, updated docs/ENVIRONMENT-FACTS.md, committed 5f8069d and pushed]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-nix-2026-11-12-01-50-43|Rich discovery: nix 2026-11-12 01:50:43]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|nix]]
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
---
|
||||||
|
type: action
|
||||||
|
id: action.paperclip-harness.session-handoff-01a1c288
|
||||||
|
title: Session Handoff 01a1c288
|
||||||
|
status: completed
|
||||||
|
source_of_truth: false
|
||||||
|
freshness: current
|
||||||
|
last_reviewed: 2026-11-12
|
||||||
|
tags:
|
||||||
|
- pack/paperclip-harness
|
||||||
|
- agent-memory/action
|
||||||
|
- session/handoff
|
||||||
|
---
|
||||||
|
|
||||||
|
# Session Handoff 01a1c288
|
||||||
|
|
||||||
|
**Session:** 01a1c288-74f9-75ae-ba9a-a501d66bc3b5
|
||||||
|
**Date:** 2026-11-12
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Assistant: The rebuild failed because my first fix patched the wrong lan-mouse. It is fixed properly now, and the derivation that caused the failure is no longer in the system.
|
||||||
|
|
||||||
|
## What was wrong
|
||||||
|
|
||||||
|
Your `configur
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[packs/paperclip-harness/00-system/pi-agent/memory-manifest|Memory Manifest]]
|
||||||
@@ -463,3 +463,32 @@ Caveat: when done from .27 against an .13 folder, use `--host nixos-desktop --se
|
|||||||
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|Convention: folder and session must live on the same machine; --host only when they differ]]
|
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|Convention: folder and session must live on the same machine; --host only when they differ]]
|
||||||
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-home-2026-11-12-00-35-25|Rich discovery: home 2026-11-12 00:35:25]]
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-home-2026-11-12-00-35-25|Rich discovery: home 2026-11-12 00:35:25]]
|
||||||
- context: [[packs/paperclip-harness/20-context/home|home]]
|
- context: [[packs/paperclip-harness/20-context/home|home]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 03:14:45)
|
||||||
|
|
||||||
|
Decision: on nixos-desktop (.13), stop using `nixos-rebuild switch` for nixpkgs bumps or similarly broad changes; use `nixos-rebuild boot` and reboot manually. Additionally, retire the legacy static `networking.interfaces`/`defaultGateway` config for `enp42s0` and give NetworkManager a manual connection profile (192.168.20.13/24, gateway 192.168.20.1, DNS .35/.13/1.1.1.1).
|
||||||
|
|
||||||
|
Context: after a nixpkgs bump, `switch` restarted NetworkManager/wpa_supplicant, the two network managers raced, and the default route vanished — taking Paperclip, Docker and tuios down until an external agent re-added the route. `configuration.nix` line ~334 already documents the unsupported combination.
|
||||||
|
|
||||||
|
Rationale: `boot` makes restarts an explicit operator action; one network manager eliminates the flush/ordering race instead of relying on boot order.
|
||||||
|
|
||||||
|
Alternatives considered: keep as-is and accept boot-order luck (rejected — failed once); dropping to `ip`-only statics without NetworkManager (riskier for WiFi on the ThinkPad host).
|
||||||
|
|
||||||
|
Consequences/caveats: the NM profile must be verified at runtime with `nmcli` before the config is changed, because a wrong profile means losing the address on reboot. On `.27` and `.51` no system bump is needed for Pi — build the package alone and use it, leaving their lock state untouched.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 85%
|
||||||
|
- Reason: Architectural/operational convention with rationale, alternatives and caveats.
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 03:14:45 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|Pi coding agent 1.5.0 builds standalone from cache; .27 and .51 don't need a full system bump]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Apply large NixOS changes on the live Paperclip/Docker server (.13)]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|use `nixos-rebuild boot` for wide changes on .13; hand enp42s0 to NetworkManager static profile]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static|NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-35-2026-11-12-03-14-45|Rich discovery: .35 2026-11-12 03:14:45]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|.35]]
|
||||||
|
|||||||
@@ -0,0 +1,52 @@
|
|||||||
|
---
|
||||||
|
type: decision
|
||||||
|
id: decision.paperclip-harness.wire-lan-mouse-into-configuration-nix-as-flake-lm-override-n
|
||||||
|
title: Wire lan-mouse into configuration.nix as flake `lm` override, not nixpkgs package or overlay
|
||||||
|
status: active
|
||||||
|
source_of_truth: false
|
||||||
|
freshness: current
|
||||||
|
last_reviewed: 2026-11-12
|
||||||
|
tags:
|
||||||
|
- pack/paperclip-harness
|
||||||
|
- agent-memory/decision
|
||||||
|
- nixos
|
||||||
|
- lan-mouse
|
||||||
|
- flake
|
||||||
|
- systemPackages
|
||||||
|
- overrideAttrs
|
||||||
|
---
|
||||||
|
|
||||||
|
# Wire lan-mouse into configuration.nix as flake `lm` override, not nixpkgs package or overlay
|
||||||
|
|
||||||
|
Decision: in `configuration.nix`, take a flake argument `lm` defined in `flake.nix` as
|
||||||
|
|
||||||
|
```nix
|
||||||
|
lm = lan-mouse.packages.${system}.default.overrideAttrs (o: { LIBGIT2_NO_VENDOR = "0"; });
|
||||||
|
```
|
||||||
|
|
||||||
|
and use `lm` in `systemPackages` where `lan-mouse` used to appear. The stale `lan-mouse` entry / overlay path was removed from `configuration.nix` (now ~line 571:
|
||||||
|
`lm # the flake input, built with LIBGIT2_NO_VENDOR=0 (see flake.nix)`).
|
||||||
|
|
||||||
|
Context: nixpkgs `lan-mouse` panics at build time when `LIBGIT2_NO_VENDOR` is set; overlaying `pkgs.lan-mouse` could not clear the variable set by nixpkgs, so the only working lever was the flake input's own package derivation.
|
||||||
|
|
||||||
|
Alternatives considered and rejected: (a) overriding the flake input while `configuration.nix` still used nixpkgs `lan-mouse` (no effect); (b) overlay on `pkgs.lan-mouse` (same panic).
|
||||||
|
|
||||||
|
Consequences / caveats: confirmed good by building `/nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0` standalone; the system dry-run then showed 12 derivations to build instead of ~465. Note lan-mouse needs a compositor (lost on `.13` per configuration.nix comment).
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 90%
|
||||||
|
- Reason: Architectural/config choice with rationale and rejected alternatives, explicitly encoded in the repo.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[packs/paperclip-harness/00-system/pi-agent/memory-manifest|Memory Manifest]]
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 01:50:43 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|lan-mouse LIBGIT2_NO_VENDOR build failure: nixpkgs vs flake package mismatch]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/wire-lan-mouse-into-configuration-nix-as-flake-lm-override-n|Wire lan-mouse into configuration.nix as flake `lm` override, not nixpkgs package or overlay]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|NixOS fleet upgrade runbook: rebuild `.13` first, fix font duplication on `.27`, then `.51`]]
|
||||||
|
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|Fixed .13 lan-mouse build, updated docs/ENVIRONMENT-FACTS.md, committed 5f8069d and pushed]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-nix-2026-11-12-01-50-43|Rich discovery: nix 2026-11-12 01:50:43]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|nix]]
|
||||||
@@ -4602,3 +4602,250 @@ Committed as `a129c4e` and pushed. After `.13` reboots into the new generation,
|
|||||||
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Fix a single package build before a NixOS rebuild, then switch]]
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Fix a single package build before a NixOS rebuild, then switch]]
|
||||||
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|office CLI wrappers installed; office bugs fixed; .13 rebuild failure fixed and pushed]]
|
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|office CLI wrappers installed; office bugs fixed; .13 rebuild failure fixed and pushed]]
|
||||||
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-local-2026-11-12-01-11-14|Rich discovery: .local 2026-11-12 01:11:14]]
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-local-2026-11-12-01-11-14|Rich discovery: .local 2026-11-12 01:11:14]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 01:50:43)
|
||||||
|
|
||||||
|
On host `.13` (nixos-desktop), `nixos-rebuild switch` failed building `lan-mouse-0.11.0` (derivation `9vrmdr2wnp6y8529alkyb55cimhrlcd9`). Root cause chain:
|
||||||
|
|
||||||
|
1. `configuration.nix` referenced `lan-mouse` from nixpkgs, but the attempted override was applied to the **flake input's** copy (`lan-mouse.packages.${system}.default`). The build therefore kept using the unpatched package — evidence: the failing derivation hash `9vrmdr…` was byte-for-byte identical before and after the change.
|
||||||
|
2. An overlay on `pkgs.lan-mouse` also failed — the panic returned with the same message because nixpkgs sets `LIBGIT2_NO_VENDOR` in a way an overlay cannot remove.
|
||||||
|
3. The fix that works: use the flake's package with `LIBGIT2_NO_VENDOR = "0"` via `overrideAttrs`, and wire that derivation into `configuration.nix`.
|
||||||
|
|
||||||
|
Diagnostic probes used: evaluating the derivation env/`buildInputs` through Nix showed `LIBGIT2_NO_VENDOR = None`, `PKG_CONFIG_PATH` absent, `buildInputs count: 0` on the un-overridden drv. A python snippet parsing the derivation JSON panicked (`AttributeError: 'int' object has no attribute 'get'` on `v.get('env',{})`) — a throwaway probe, not a repo artifact.
|
||||||
|
|
||||||
|
Established method: build the offending package alone with an override and confirm the new store path before switching the whole system.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 90%
|
||||||
|
- Reason: Concrete, evidence-backed diagnosis of a build failure with derivation hashes and the winning fix — reusable by future agents touching this repo/host.
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 01:50:43)
|
||||||
|
|
||||||
|
## Structural/configuration observations (2026-11-12 01:50:43)
|
||||||
|
|
||||||
|
The rebuild failed because my first fix patched the wrong lan-mouse. It is fixed properly now, and the derivation that caused the failure is no longer in the system.
|
||||||
|
|
||||||
|
## What was wrong
|
||||||
|
|
||||||
|
Your `configuration.nix` used **`lan-mouse` from nixpkgs**, while I had overridden the **flake input's** copy. So the build kept using the unpatched package — which is why the failing derivation was byte-for-byte the same one, `9vrmdr…`, before and after my change.
|
||||||
|
|
||||||
|
I then tried an overlay on `pkgs.lan-mouse`, and that did not work either: the panic returned with the same message, because nixpkgs sets `LIBGIT2_NO_VENDOR` in a way the overlay could not remove.
|
||||||
|
|
||||||
|
## What works, proven
|
||||||
|
|
||||||
|
The flake's own lan-mouse package, built with the variable turned off:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
lm = lan-mouse.packages.${system}.default.overrideAttrs (o: { LIBGIT2_NO_VENDOR = "0"; });
|
||||||
|
```
|
||||||
|
|
||||||
|
Built it before wiring it in:
|
||||||
|
|
||||||
|
```
|
||||||
|
/nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Then changed two lines: `configuration.nix` now takes `lm` as an argument and uses it in `systemPackages` where `lan-mouse` was.
|
||||||
|
|
||||||
|
Result on `.13`:
|
||||||
|
|
||||||
|
```
|
||||||
|
these 12 derivations will be built:
|
||||||
|
pi: 1.5.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Twelve derivations, not 465 — most of the rest had already been substituted during the earlier attempts. Your rebuild should now go through:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo nixos-rebuild switch --flake /etc/nixos#nixos-desktop
|
||||||
|
```
|
||||||
|
|
||||||
|
If it fails on something else, send the first error block; the method is now established: build the offending package alone with an override before switching.
|
||||||
|
|
||||||
|
## About the font
|
||||||
|
|
||||||
|
Thank you for the pointer — it may well be the cause on `.27`. Its error names `home-manager.users.sam.fonts.fontconfig.configFile.fonts.settings`, and `.27` declares fonts in two places (system `fonts.packages` and again inside `systemPackages`). With home-manager 2026-11-10 that duplication is what it cannot merge. That is a small fix, but I want `.13` proven first so we are changing one thing at a time.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
| | config | evaluates | running | next |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `.13` | lock, `initContent`, lan-mouse via `lm` | pi **1.5.0**, 12 to build | 0.82.1 | **rebuild now** |
|
||||||
|
| `.27` | lock, `initContent` | fails in home-manager fontconfig | 0.82.1 | fix the font duplication, or bump nixpkgs only |
|
||||||
|
| `.51` | unchanged | pi 0.87.1 | 0.87.1 | the same three edits, then rebuild |
|
||||||
|
|
||||||
|
Written into `docs/ENVIRONMENT-FACTS.md`, committed as `5f8069d`, pushed.
|
||||||
|
|
||||||
|
Tell me what `pi --version` says after the rebuild and I will bring `.27` and `.51` to the same version.
|
||||||
|
|
||||||
|
## Source paths observed
|
||||||
|
|
||||||
|
- `nix/store/91ghzifbssyi53z9w3c2v2nliami4ry1-lan-mouse-0.11.0.drv`
|
||||||
|
- `nix/store/9vrmdr2wnp6y8529alkyb55cimhrlcd9-lan-mouse-0.11.0.drv`
|
||||||
|
- `nix/store/y0h5slyxhmkikga74pnaz0ybadwaji7b-zzz-0.11.0.drv`
|
||||||
|
- `avahi/services/snapcast.service`
|
||||||
|
- `nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0`
|
||||||
|
- `gitea.lab.audasmedia.com.au/sam/paperclip-harness.git`
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 82%
|
||||||
|
- Reason: Captured factual repo structure/configuration patterns from rich discovery
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 01:50:43 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|lan-mouse LIBGIT2_NO_VENDOR build failure: nixpkgs vs flake package mismatch]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/wire-lan-mouse-into-configuration-nix-as-flake-lm-override-n|Wire lan-mouse into configuration.nix as flake `lm` override, not nixpkgs package or overlay]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|NixOS fleet upgrade runbook: rebuild `.13` first, fix font duplication on `.27`, then `.51`]]
|
||||||
|
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|Fixed .13 lan-mouse build, updated docs/ENVIRONMENT-FACTS.md, committed 5f8069d and pushed]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-nix-2026-11-12-01-50-43|Rich discovery: nix 2026-11-12 01:50:43]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|nix]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 03:14:45)
|
||||||
|
|
||||||
|
With the bumped nixpkgs pin present on `.27`, building only the Pi package (`nix build --no-link --print-out-paths` of the pi coding agent) completes in ~6 seconds entirely from `cache.nixos.org`, producing `/nix/store/4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9-pi-coding-agent-1.5.0`. That is the exact store path running on `.13`, so `.27` and `.51` can run the same Pi bytes without a full `nixos-rebuild`, avoiding `.27`'s blocking home-manager font problem and the `.13` network-restart failure mode (`.27`/`.51` use DHCP-provided default routes). A previously built flake output path observed: `/nix/store/02h6ym8j8qvxpjps56n9gx4gxqpg26a0-source`.
|
||||||
|
|
||||||
|
Caveat: `.27`'s `flake.lock` is currently bumped into a state that will not evaluate a full system; backups `flake.lock.bak-pi-align` and `home/sam/home.nix.bak-pi-align` exist on `.27`, and `flake.lock.bak-pi-align` + `home.nix.bak-pi-align` exist on `.13` (under `/etc/nixos`). Revert `.27`'s lock before any full rebuild.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 85%
|
||||||
|
- Reason: Evidence-backed operational discovery with exact store path, timings, and a landmine warning for .27.
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 03:14:45)
|
||||||
|
|
||||||
|
`/etc/nixos` is a multi-host flake: `nixosConfigurations.sam-thinkpad` (ThinkPad L14 Gen 2 AMD, Niri) uses `./hosts/sam-thinkpad/configuration.nix`; sam-desktop (.13) includes the Pi package via home-manager (`hunk.packages.${pkgs.stdenv.hostPlatform.system}.hunk`) in `nixos-rebuild switch` style flake module wiring — `home-manager.useGlobalPkgs = true`, `home-manager.useUserPackages = true`, `home-manager.users.sam` importing `./home/sam/home.nix` (`.27` uses `./home/sam/home.nix` under `/etc/nixos/home/sam/home.nix`). `nixpkgs.config.allowUnfree = true` is set for the host.
|
||||||
|
|
||||||
|
Runtime state on .13: ~190 old generations retained (from a prior observation "old gens kept 190"); 5+ tuios sessions (drone deeca ai, email_agent, factory 8 windows, harness-repo 9 windows, nixos desktop 13 devops).
|
||||||
|
|
||||||
|
This context matters because the home-manager import path differs between hosts, and edits to the Pi pin must touch both.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 80%
|
||||||
|
- Reason: Repository/component overview of the NixOS flake necessary for future edits across hosts.
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 03:14:45)
|
||||||
|
|
||||||
|
## Structural/configuration observations (2026-11-12 03:14:45)
|
||||||
|
|
||||||
|
I broke it, and here is what actually happened. No metaphors, just the sequence.
|
||||||
|
|
||||||
|
## What I did wrong
|
||||||
|
|
||||||
|
`nixos-rebuild switch` on a machine that hosts Paperclip, Docker and every TUIOS session **restarts every changed unit in place**. A nixpkgs bump changes a great many units. I checked that the config *built* and never asked what it would *restart*. It restarted the network, the network did not come back, and everything on `.13` that needs a route went down with it. The correct verb for a change that large on a live server is `nixos-rebuild boot`: it builds, sets the next generation, then stops. You reboot when you choose. I should have used that, and I will from now on.
|
||||||
|
|
||||||
|
The exit code 4 was **not** a build failure. The generation did switch. Exit 4 means units failed to start, and the units failed because the route was gone.
|
||||||
|
|
||||||
|
## `.13` right now, measured
|
||||||
|
|
||||||
|
```
|
||||||
|
pi --version 1.5.0
|
||||||
|
/run/current-system x0yghxby0zwpyi8l15gw8gr8jwm5n4ir-nixos-system-nixos-desktop-26.11.20261107.f4a03c0
|
||||||
|
ip route show default default via 192.168.20.1 dev enp42s0 metric 100
|
||||||
|
systemctl --failed (nothing)
|
||||||
|
librespot / snapserver / NetworkManager / docker / postgresql / tailscaled active
|
||||||
|
internet, .27 reachable
|
||||||
|
Paperclip /api/health 200
|
||||||
|
tuios list-sessions sessions restored
|
||||||
|
```
|
||||||
|
|
||||||
|
So the system is healthy and on the same Pi as planned. The repair was the other AI's, not mine: a runtime `ip route add` plus a restart of librespot. I am telling you that plainly because I did not fix it.
|
||||||
|
|
||||||
|
## The root cause, from the journal
|
||||||
|
|
||||||
|
```
|
||||||
|
12:51:46 Stopping Address configuration of enp42s0... ← pre-stop deletes address and route
|
||||||
|
12:51:47 adding address 192.168.20.13/24... done
|
||||||
|
12:51:47 setting enp42s0 as default IPv4 gateway... done ← route WAS set
|
||||||
|
(nothing after this for that unit)
|
||||||
|
```
|
||||||
|
|
||||||
|
The route was set at 12:51:47 and disappeared afterwards. What ran between then and the failure was the restart of `NetworkManager.service` and `wpa_supplicant.service`. Your own comment in `configuration.nix` at line 334 already names this: static `interfaces` + `defaultGateway` with `networkmanager.unmanaged` is a combination NixOS warns is unsupported, and it worked "only by boot-order luck". The reboot-strength change destroyed the luck.
|
||||||
|
|
||||||
|
Two managers on one interface is the bug. Not the nixpkgs bump.
|
||||||
|
|
||||||
|
## What I recommend for `.13`
|
||||||
|
|
||||||
|
Let NetworkManager own `enp42s0` with a static profile (manual, `.13/24`, gateway `.1`, DNS `.35/.13/1.1.1.1`), and delete `networkmanager.unmanaged` plus the legacy `interfaces`/`defaultGateway` lines. One manager, no flush race, no boot-order luck. This is also what the other AI suggested and I agree with it.
|
||||||
|
|
||||||
|
The risk to be honest about: if the NetworkManager profile is wrong, the machine loses its address on the next boot. So it must be tested **at runtime with `nmcli` first** — bring the profile up by hand, prove the route and DNS work, then write it into the config. Not the other way round.
|
||||||
|
|
||||||
|
## `.27` and `.51`: the Pi part needs no system bump
|
||||||
|
|
||||||
|
This is the useful discovery. On `.27`, with the bumped pin already there, the Pi package alone builds in **6 seconds** from `cache.nixos.org`:
|
||||||
|
|
||||||
|
```
|
||||||
|
/nix/store/4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9-pi-coding-agent-1.5.0
|
||||||
|
```
|
||||||
|
|
||||||
|
That is the exact store path `.13` runs. So `.27` and `.51` can have the *same bytes* of Pi without a whole-system bump, and without touching the home-manager font problem that blocks `.27`'s full rebuild. `.13`'s failure mode also cannot repeat on them: both take their default route from DHCP through NetworkManager (`proto dhcp`), not from a legacy static address.
|
||||||
|
|
||||||
|
Also: `.27`'s `flake.lock` is currently bumped into a state that will not evaluate to a full system. I should revert it to `flake.lock.bak-pi-align` so the machine is not sitting on a landmine for whoever runs a rebuild next.
|
||||||
|
|
||||||
|
Say which you want and I will do only that.
|
||||||
|
|
||||||
|
## Source paths observed
|
||||||
|
|
||||||
|
- `etc/nixos/hosts/sam-4screen-desktop/configuration.nix`
|
||||||
|
- `./home/sam/home.nix`
|
||||||
|
- `Chrome/Spotify/VSCode`
|
||||||
|
- `./hosts/sam-thinkpad/configuration.nix`
|
||||||
|
- `nix/store/02h6ym8j8qvxpjps56n9gx4gxqpg26a0-source`
|
||||||
|
- `etc/nixos/flake.lock.bak-pi-align`
|
||||||
|
- `etc/nixos/home.nix.bak-pi-align`
|
||||||
|
- `etc/nixos/home/sam/home.nix.bak-pi-align`
|
||||||
|
- `nix/store/3kfmdqn711fqcc2y1qnjssancqf7b36l-nodejs-24.22.2`
|
||||||
|
- `nix/store/4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9-pi-coding-agent-1.5.0`
|
||||||
|
- `etc/systemd/system/network-addresses-enp42s0.service`
|
||||||
|
- `nix/store/qrz69l7h7d78vf8qgjbflq263ddymfss-unit-network-addresses-enp42s0.service/network-addresses-enp42s0.service`
|
||||||
|
- `nix/store/rbvrn9kch96rr5nm7jdp3913d51nm09g-glibc-locales-2.44-57/lib/locale/locale-archive`
|
||||||
|
- `nix/store/wc6ky2sqq8shrmaf14x7spbdpqi6ls9w-iproute2-7.2.0/bin`
|
||||||
|
- `nix/store/a50ml7za6h61m4mv4cdbvkl33c6sx5yn-coreutils-9.12/bin`
|
||||||
|
- `nix/store/5zsc280i2i7bqgjn7kln87w1ss6li22f-findutils-4.11.0/bin`
|
||||||
|
- `nix/store/5sgjjiqn90zmvlvlgadw7ss00nn3hpjb-gnugrep-3.12/bin`
|
||||||
|
- `nix/store/x275j18p7dx6kv7vfj45saapbhbljia4-gnused-4.10/bin`
|
||||||
|
- `nix/store/kh0kk4mi6hr0cdvsc8rzd0jvr2c2vhwk-systemd-262/bin`
|
||||||
|
- `nix/store/wc6ky2sqq8shrmaf14x7spbdpqi6ls9w-iproute2-7.2.0/sbin`
|
||||||
|
- `nix/store/a50ml7za6h61m4mv4cdbvkl33c6sx5yn-coreutils-9.12/sbin`
|
||||||
|
- `nix/store/5zsc280i2i7bqgjn7kln87w1ss6li22f-findutils-4.11.0/sbin`
|
||||||
|
- `nix/store/5sgjjiqn90zmvlvlgadw7ss00nn3hpjb-gnugrep-3.12/sbin`
|
||||||
|
- `nix/store/x275j18p7dx6kv7vfj45saapbhbljia4-gnused-4.10/sbin`
|
||||||
|
- `nix/store/kh0kk4mi6hr0cdvsc8rzd0jvr2c2vhwk-systemd-262/sbin`
|
||||||
|
- `nix/store/s92hlmg73b3zlw38dfgm13v9cah2bm80-tzdata-2026f/share/zoneinfo`
|
||||||
|
- `nix/store/4p9n1mpnjahlm9j51kal5a7nmj7b8v88-unit-script-network-addresses-enp42s0-start/bin/network-addresses-enp42s0-start`
|
||||||
|
- `nix/store/3przkjya9ba0x1y40slkr2g8wkrva8ym-unit-script-network-addresses-enp42s0-pre-stop/bin/network-addresses-enp42s0-pre-stop`
|
||||||
|
- `nix/store/hasnvc0j36gg6qrdsfcx7yivx3c2bpqr-bash-5.3p20/bin/bash`
|
||||||
|
- `run/nixos/network/addresses/enp42s0`
|
||||||
|
- `run/nixos/network/routes/enp42s0`
|
||||||
|
- `.35/.13/1.1.1.1`
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 82%
|
||||||
|
- Reason: Captured factual repo structure/configuration patterns from rich discovery
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 03:14:45 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|Pi coding agent 1.5.0 builds standalone from cache; .27 and .51 don't need a full system bump]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Apply large NixOS changes on the live Paperclip/Docker server (.13)]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|use `nixos-rebuild boot` for wide changes on .13; hand enp42s0 to NetworkManager static profile]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static|NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-35-2026-11-12-03-14-45|Rich discovery: .35 2026-11-12 03:14:45]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|.35]]
|
||||||
|
|||||||
@@ -1530,3 +1530,72 @@ Committed as `a129c4e` and pushed. After `.13` reboots into the new generation,
|
|||||||
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Fix a single package build before a NixOS rebuild, then switch]]
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Fix a single package build before a NixOS rebuild, then switch]]
|
||||||
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|office CLI wrappers installed; office bugs fixed; .13 rebuild failure fixed and pushed]]
|
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|office CLI wrappers installed; office bugs fixed; .13 rebuild failure fixed and pushed]]
|
||||||
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-local-2026-11-12-01-11-14|Rich discovery: .local 2026-11-12 01:11:14]]
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-local-2026-11-12-01-11-14|Rich discovery: .local 2026-11-12 01:11:14]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 01:50:43)
|
||||||
|
|
||||||
|
Trigger: bringing the NixOS fleet hosts to the same pi (`pi` package) version / fixing failed `nixos-rebuild switch` runs.
|
||||||
|
|
||||||
|
Prerequisites: flake at `/etc/nixos` with output `#nixos-desktop`; this repo committed & pushed for `docs/ENVIRONMENT-FACTS.md` updates.
|
||||||
|
|
||||||
|
Procedure & current host state:
|
||||||
|
1. **`.13` (nixos-desktop)** — config has: lock, `initContent`, and lan-mouse via `lm` (flake override with `LIBGIT2_NO_VENDOR="0"`). Evaluates to pi **1.5.0**; running 0.82.1. Dry-run shows 12 derivations to build. Next action: rebuild now:
|
||||||
|
`sudo nixos-rebuild switch --flake /etc/nixos#nixos-desktop`
|
||||||
|
Then check `pi --version`.
|
||||||
|
2. **`.27`** — config has lock + `initContent`; evaluation **fails in home-manager fontconfig**: error names `home-manager.users.sam.fonts.fontconfig.configFile.fonts.settings`. Cause hypothesis: `.27` declares fonts in two places — system `fonts.packages` and again inside `systemPackages` — which home-manager 2026-11-10 cannot merge. Fix the duplication (or only bump nixpkgs), then rebuild.
|
||||||
|
3. **`.51`** — unchanged: evaluates pi **0.87.1**, running 0.87.1. Apply "the same three edits" (lock, `initContent`, and the lan-mouse/`lm` wiring) then rebuild.
|
||||||
|
|
||||||
|
Safe/dangerous: `nixos-rebuild switch` mutates live systems — do them one host at a time, `.13` proven first. Change one thing at a time.
|
||||||
|
|
||||||
|
Troubleshooting: if a package build fails, build the offending package alone with an override first (see the lan-mouse `LIBGIT2_NO_VENDOR` case) and confirm the resulting store path, then wire it in. If it fails on something else, capture the first error block before changing anything else.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 85%
|
||||||
|
- Reason: Repeatable multi-host procedure with per-host state, exact commands, and validation steps.
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 01:50:43 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|lan-mouse LIBGIT2_NO_VENDOR build failure: nixpkgs vs flake package mismatch]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/wire-lan-mouse-into-configuration-nix-as-flake-lm-override-n|Wire lan-mouse into configuration.nix as flake `lm` override, not nixpkgs package or overlay]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|NixOS fleet upgrade runbook: rebuild `.13` first, fix font duplication on `.27`, then `.51`]]
|
||||||
|
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|Fixed .13 lan-mouse build, updated docs/ENVIRONMENT-FACTS.md, committed 5f8069d and pushed]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-nix-2026-11-12-01-50-43|Rich discovery: nix 2026-11-12 01:50:43]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|nix]]
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update (2026-11-12 03:14:45)
|
||||||
|
|
||||||
|
Trigger: any nixpkgs bump or otherwise wide-reaching change on nixos-desktop (.13), which hosts Paperclip, Docker, postgres, tailscale, librespot and tuios sessions.
|
||||||
|
|
||||||
|
Prerequisites: ssh access to .13; confirm backups exist (`/etc/nixos/flake.lock.bak-pi-align`, `/etc/nixos/home.nix.bak-pi-align`).
|
||||||
|
|
||||||
|
Procedure:
|
||||||
|
1. Build only: `nixos-rebuild build` or `nix build` the host config to prove evaluation.
|
||||||
|
2. Inspect what will restart: `systemctl list-units` diff / `nixos-rebuild dry-activate` and note units touching network.
|
||||||
|
3. Deploy with `nixos-rebuild boot` (never `switch`) for wide changes; reboot at a chosen maintenance window.
|
||||||
|
4. On reboot, verify: `ip route show default` shows `default via 192.168.20.1 dev enp42s0`; `systemctl --failed` empty; `curl -fsS localhost:<paperclip-port>/api/health` returns 200 (`/api/health` expected 200); `pi --version` is 1.5.0; `tuios list-sessions` shows restored sessions.
|
||||||
|
|
||||||
|
Dangerous: `nixos-rebuild switch` on .13 while `networkmanager.unmanaged` coexists with legacy `networking.interfaces` static config — restarting NetworkManager wipes the default route that `network-addresses-enp42s0` just set. If the route disappears mid-run, exit code 4 reflects failed units, not a build failure.
|
||||||
|
|
||||||
|
Rollback: `nixos-rebuild switch --rollback` (but this too restarts units); last resort reboot an older generation from the bootloader / `nixos-rebuild boot --rollback` then reboot.
|
||||||
|
|
||||||
|
Troubleshooting: if address present but no route, `ip route add default via 192.168.20.1 dev enp42s0` and restart dependent services (e.g. librespot). Long-term: migrate enp42s0 to a NetworkManager manual profile first, tested with `nmcli device modify`/`nmcli connection up` before removing the legacy config.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 85%
|
||||||
|
- Reason: Repeatable procedure derived from the outage with exact verification commands and rollback.
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 03:14:45 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|Pi coding agent 1.5.0 builds standalone from cache; .27 and .51 don't need a full system bump]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Apply large NixOS changes on the live Paperclip/Docker server (.13)]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|use `nixos-rebuild boot` for wide changes on .13; hand enp42s0 to NetworkManager static profile]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static|NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-35-2026-11-12-03-14-45|Rich discovery: .35 2026-11-12 03:14:45]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|.35]]
|
||||||
|
|||||||
@@ -0,0 +1,52 @@
|
|||||||
|
---
|
||||||
|
type: session
|
||||||
|
id: session.paperclip-harness.nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static
|
||||||
|
title: NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race
|
||||||
|
status: active
|
||||||
|
source_of_truth: false
|
||||||
|
freshness: current
|
||||||
|
last_reviewed: 2026-11-12
|
||||||
|
tags:
|
||||||
|
- pack/paperclip-harness
|
||||||
|
- agent-memory/session
|
||||||
|
- nixos
|
||||||
|
- networking
|
||||||
|
- networkmanager
|
||||||
|
- enp42s0
|
||||||
|
- nixos-rebuild
|
||||||
|
- postmortem
|
||||||
|
- 192.168.20.13
|
||||||
|
---
|
||||||
|
|
||||||
|
# NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race
|
||||||
|
|
||||||
|
Host `nixos-desktop` at 192.168.20.13 (NixOS 26.11, system `/etc/nixos`) went offline after an in-place `nixos-rebuild switch` following a nixpkgs bump.
|
||||||
|
|
||||||
|
Facts discovered:
|
||||||
|
- `nixos-rebuild switch` restarts every changed unit. Because of the nixpkgs bump, this included `NetworkManager.service` and `wpa_supplicant.service`.
|
||||||
|
- The machine uses legacy static `networking.interfaces` + `defaultGateway` for `enp42s0` (192.168.20.13/24, gateway 192.168.20.1) simultaneously with `networkmanager.unmanaged`; the config at `configuration.nix` line ~334 already comments that this combination is unsupported and works only by boot-order luck.
|
||||||
|
- The restart triggered `network-addresses-enp42s0.service` pre-stop (deletes address/route), start (adds address and `ip -4 route replace default proto static via 192.168.20.1`), then the NetworkManager restart flushed the route again, leaving no default route.
|
||||||
|
- Exit code 4 from `nixos-rebuild` did NOT indicate a build failure; the generation switched, but units failed to start because the route was gone.
|
||||||
|
- Repaired externally (by another AI) with a runtime `ip route add` plus restart of librespot. Post-repair state: pi 1.5.0, current-system `...nixos-system-nixos-desktop-26.11.20261107.f4a03c0`, default via 192.168.20.1, no failed units, librespot/snapserver/NetworkManager/docker/postgresql/tailscaled active, Paperclip `/api/health` 200, tuios sessions restored.
|
||||||
|
|
||||||
|
Known-good recommendation (agreed, not yet executed): let NetworkManager own `enp42s0` with a manual static profile (.13/24, gateway .1, DNS .35/.13/1.1.1.1) and remove `networkmanager.unmanaged` + legacy `interfaces`/`defaultGateway` lines. Must be validated at runtime with `nmcli` before writing to config.
|
||||||
|
|
||||||
|
Other operator notes: use `nixos-rebuild boot` for large changes on live servers; `.27` and `.51` get default routes via DHCP so the .13 failure mode cannot repeat; `.27`'s flake.lock is currently bumped to a non-full-system-evaluable state and should be reverted to `flake.lock.bak-pi-align` before anyone runs a rebuild.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 90%
|
||||||
|
- Reason: Rich discovery snapshot: concrete failure sequence, journal evidence, recommended fix, and machine state that future agents on this repo should know.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[packs/paperclip-harness/00-system/pi-agent/memory-manifest|Memory Manifest]]
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 03:14:45 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|Pi coding agent 1.5.0 builds standalone from cache; .27 and .51 don't need a full system bump]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Apply large NixOS changes on the live Paperclip/Docker server (.13)]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|use `nixos-rebuild boot` for wide changes on .13; hand enp42s0 to NetworkManager static profile]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static|NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-35-2026-11-12-03-14-45|Rich discovery: .35 2026-11-12 03:14:45]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|.35]]
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
---
|
||||||
|
type: session
|
||||||
|
id: session.paperclip-harness.rich-discovery-35-2026-11-12-03-14-45
|
||||||
|
title: Rich discovery: .35 2026-11-12 03:14:45
|
||||||
|
status: active
|
||||||
|
source_of_truth: false
|
||||||
|
freshness: current
|
||||||
|
last_reviewed: 2026-11-12
|
||||||
|
tags:
|
||||||
|
- pack/paperclip-harness
|
||||||
|
- agent-memory/session
|
||||||
|
- autolearn
|
||||||
|
- rich-discovery
|
||||||
|
- repo/35
|
||||||
|
---
|
||||||
|
|
||||||
|
# Rich discovery: .35 2026-11-12 03:14:45
|
||||||
|
|
||||||
|
Sanitized rich discovery snapshot captured automatically from the completed turn.
|
||||||
|
|
||||||
|
|
||||||
|
## Source paths observed
|
||||||
|
|
||||||
|
- `etc/nixos/hosts/sam-4screen-desktop/configuration.nix`
|
||||||
|
- `./home/sam/home.nix`
|
||||||
|
- `Chrome/Spotify/VSCode`
|
||||||
|
- `./hosts/sam-thinkpad/configuration.nix`
|
||||||
|
- `nix/store/02h6ym8j8qvxpjps56n9gx4gxqpg26a0-source`
|
||||||
|
- `etc/nixos/flake.lock.bak-pi-align`
|
||||||
|
- `etc/nixos/home.nix.bak-pi-align`
|
||||||
|
- `etc/nixos/home/sam/home.nix.bak-pi-align`
|
||||||
|
- `nix/store/3kfmdqn711fqcc2y1qnjssancqf7b36l-nodejs-24.22.2`
|
||||||
|
- `nix/store/4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9-pi-coding-agent-1.5.0`
|
||||||
|
- `etc/systemd/system/network-addresses-enp42s0.service`
|
||||||
|
- `nix/store/qrz69l7h7d78vf8qgjbflq263ddymfss-unit-network-addresses-enp42s0.service/network-addresses-enp42s0.service`
|
||||||
|
- `nix/store/rbvrn9kch96rr5nm7jdp3913d51nm09g-glibc-locales-2.44-57/lib/locale/locale-archive`
|
||||||
|
- `nix/store/wc6ky2sqq8shrmaf14x7spbdpqi6ls9w-iproute2-7.2.0/bin`
|
||||||
|
- `nix/store/a50ml7za6h61m4mv4cdbvkl33c6sx5yn-coreutils-9.12/bin`
|
||||||
|
- `nix/store/5zsc280i2i7bqgjn7kln87w1ss6li22f-findutils-4.11.0/bin`
|
||||||
|
- `nix/store/5sgjjiqn90zmvlvlgadw7ss00nn3hpjb-gnugrep-3.12/bin`
|
||||||
|
- `nix/store/x275j18p7dx6kv7vfj45saapbhbljia4-gnused-4.10/bin`
|
||||||
|
- `nix/store/kh0kk4mi6hr0cdvsc8rzd0jvr2c2vhwk-systemd-262/bin`
|
||||||
|
- `nix/store/wc6ky2sqq8shrmaf14x7spbdpqi6ls9w-iproute2-7.2.0/sbin`
|
||||||
|
- `nix/store/a50ml7za6h61m4mv4cdbvkl33c6sx5yn-coreutils-9.12/sbin`
|
||||||
|
- `nix/store/5zsc280i2i7bqgjn7kln87w1ss6li22f-findutils-4.11.0/sbin`
|
||||||
|
- `nix/store/5sgjjiqn90zmvlvlgadw7ss00nn3hpjb-gnugrep-3.12/sbin`
|
||||||
|
- `nix/store/x275j18p7dx6kv7vfj45saapbhbljia4-gnused-4.10/sbin`
|
||||||
|
- `nix/store/kh0kk4mi6hr0cdvsc8rzd0jvr2c2vhwk-systemd-262/sbin`
|
||||||
|
- `nix/store/s92hlmg73b3zlw38dfgm13v9cah2bm80-tzdata-2026f/share/zoneinfo`
|
||||||
|
- `nix/store/4p9n1mpnjahlm9j51kal5a7nmj7b8v88-unit-script-network-addresses-enp42s0-start/bin/network-addresses-enp42s0-start`
|
||||||
|
- `nix/store/3przkjya9ba0x1y40slkr2g8wkrva8ym-unit-script-network-addresses-enp42s0-pre-stop/bin/network-addresses-enp42s0-pre-stop`
|
||||||
|
- `nix/store/hasnvc0j36gg6qrdsfcx7yivx3c2bpqr-bash-5.3p20/bin/bash`
|
||||||
|
- `run/nixos/network/addresses/enp42s0`
|
||||||
|
- `run/nixos/network/routes/enp42s0`
|
||||||
|
- `.35/.13/1.1.1.1`
|
||||||
|
|
||||||
|
## Final answer snapshot
|
||||||
|
|
||||||
|
I broke it, and here is what actually happened. No metaphors, just the sequence.
|
||||||
|
|
||||||
|
## What I did wrong
|
||||||
|
|
||||||
|
`nixos-rebuild switch` on a machine that hosts Paperclip, Docker and every TUIOS session **restarts every changed unit in place**. A nixpkgs bump changes a great many units. I checked that the config *built* and never asked what it would *restart*. It restarted the network, the network did not come back, and everything on `.13` that needs a route went down with it. The correct verb for a change that large on a live server is `nixos-rebuild boot`: it builds, sets the next generation, then stops. You reboot when you choose. I should have used that, and I will from now on.
|
||||||
|
|
||||||
|
The exit code 4 was **not** a build failure. The generation did switch. Exit 4 means units failed to start, and the units failed because the route was gone.
|
||||||
|
|
||||||
|
## `.13` right now, measured
|
||||||
|
|
||||||
|
```
|
||||||
|
pi --version 1.5.0
|
||||||
|
/run/current-system x0yghxby0zwpyi8l15gw8gr8jwm5n4ir-nixos-system-nixos-desktop-26.11.20261107.f4a03c0
|
||||||
|
ip route show default default via 192.168.20.1 dev enp42s0 metric 100
|
||||||
|
systemctl --failed (nothing)
|
||||||
|
librespot / snapserver / NetworkManager / docker / postgresql / tailscaled active
|
||||||
|
internet, .27 reachable
|
||||||
|
Paperclip /api/health 200
|
||||||
|
tuios list-sessions sessions restored
|
||||||
|
```
|
||||||
|
|
||||||
|
So the system is healthy and on the same Pi as planned. The repair was the other AI's, not mine: a runtime `ip route add` plus a restart of librespot. I am telling you that plainly because I did not fix it.
|
||||||
|
|
||||||
|
## The root cause, from the journal
|
||||||
|
|
||||||
|
```
|
||||||
|
12:51:46 Stopping Address configuration of enp42s0... ← pre-stop deletes address and route
|
||||||
|
12:51:47 adding address 192.168.20.13/24... done
|
||||||
|
12:51:47 setting enp42s0 as default IPv4 gateway... done ← route WAS set
|
||||||
|
(nothing after this for that unit)
|
||||||
|
```
|
||||||
|
|
||||||
|
The route was set at 12:51:47 and disappeared afterwards. What ran between then and the failure was the restart of `NetworkManager.service` and `wpa_supplicant.service`. Your own comment in `configuration.nix` at line 334 already names this: static `interfaces` + `defaultGateway` with `networkmanager.unmanaged` is a combination NixOS warns is unsupported, and it worked "only by boot-order luck". The reboot-strength change destroyed the luck.
|
||||||
|
|
||||||
|
Two managers on one interface is the bug. Not the nixpkgs bump.
|
||||||
|
|
||||||
|
## What I recommend for `.13`
|
||||||
|
|
||||||
|
Let NetworkManager own `enp42s0` with a static profile (manual, `.13/24`, gateway `.1`, DNS `.35/.13/1.1.1.1`), and delete `networkmanager.unmanaged` plus the legacy `interfaces`/`defaultGateway` lines. One manager, no flush race, no boot-order luck. This is also what the other AI suggested and I agree with it.
|
||||||
|
|
||||||
|
The risk to be honest about: if the NetworkManager profile is wrong, the machine loses its address on the next boot. So it must be tested **at runtime with `nmcli` first** — bring the profile up by hand, prove the route and DNS work, then write it into the config. Not the other way round.
|
||||||
|
|
||||||
|
## `.27` and `.51`: the Pi part needs no system bump
|
||||||
|
|
||||||
|
This is the useful discovery. On `.27`, with the bumped pin already there, the Pi package alone builds in **6 seconds** from `cache.nixos.org`:
|
||||||
|
|
||||||
|
```
|
||||||
|
/nix/store/4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9-pi-coding-agent-1.5.0
|
||||||
|
```
|
||||||
|
|
||||||
|
That is the exact store path `.13` runs. So `.27` and `.51` can have the *same bytes* of Pi without a whole-system bump, and without touching the home-manager font problem that blocks `.27`'s full rebuild. `.13`'s failure mode also cannot repeat on them: both take their default route from DHCP through NetworkManager (`proto dhcp`), not from a legacy static address.
|
||||||
|
|
||||||
|
Also: `.27`'s `flake.lock` is currently bumped into a state that will not evaluate to a full system. I should revert it to `flake.lock.bak-pi-align` so the machine is not sitting on a landmine for whoever runs a rebuild next.
|
||||||
|
|
||||||
|
Say which you want and I will do only that.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 86%
|
||||||
|
- Reason: Preserve full sanitized discovery detail from a rich investigation turn
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[packs/paperclip-harness/00-system/pi-agent/memory-manifest|Memory Manifest]]
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 03:14:45 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|Pi coding agent 1.5.0 builds standalone from cache; .27 and .51 don't need a full system bump]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|Apply large NixOS changes on the live Paperclip/Docker server (.13)]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/paperclip-harness-ui-six-workspaces-spec-first-four-panel-li|use `nixos-rebuild boot` for wide changes on .13; hand enp42s0 to NetworkManager static profile]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/nixos-13-outage-postmortem-nixos-rebuild-switch-broke-static|NixOS .13 outage postmortem: nixos-rebuild switch broke static network via NetworkManager race]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-35-2026-11-12-03-14-45|Rich discovery: .35 2026-11-12 03:14:45]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|.35]]
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
---
|
||||||
|
type: session
|
||||||
|
id: session.paperclip-harness.rich-discovery-nix-2026-11-12-01-50-43
|
||||||
|
title: Rich discovery: nix 2026-11-12 01:50:43
|
||||||
|
status: active
|
||||||
|
source_of_truth: false
|
||||||
|
freshness: current
|
||||||
|
last_reviewed: 2026-11-12
|
||||||
|
tags:
|
||||||
|
- pack/paperclip-harness
|
||||||
|
- agent-memory/session
|
||||||
|
- autolearn
|
||||||
|
- rich-discovery
|
||||||
|
- repo/nix
|
||||||
|
---
|
||||||
|
|
||||||
|
# Rich discovery: nix 2026-11-12 01:50:43
|
||||||
|
|
||||||
|
Sanitized rich discovery snapshot captured automatically from the completed turn.
|
||||||
|
|
||||||
|
|
||||||
|
## Source paths observed
|
||||||
|
|
||||||
|
- `nix/store/91ghzifbssyi53z9w3c2v2nliami4ry1-lan-mouse-0.11.0.drv`
|
||||||
|
- `nix/store/9vrmdr2wnp6y8529alkyb55cimhrlcd9-lan-mouse-0.11.0.drv`
|
||||||
|
- `nix/store/y0h5slyxhmkikga74pnaz0ybadwaji7b-zzz-0.11.0.drv`
|
||||||
|
- `avahi/services/snapcast.service`
|
||||||
|
- `nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0`
|
||||||
|
- `gitea.lab.audasmedia.com.au/sam/paperclip-harness.git`
|
||||||
|
|
||||||
|
## Final answer snapshot
|
||||||
|
|
||||||
|
The rebuild failed because my first fix patched the wrong lan-mouse. It is fixed properly now, and the derivation that caused the failure is no longer in the system.
|
||||||
|
|
||||||
|
## What was wrong
|
||||||
|
|
||||||
|
Your `configuration.nix` used **`lan-mouse` from nixpkgs**, while I had overridden the **flake input's** copy. So the build kept using the unpatched package — which is why the failing derivation was byte-for-byte the same one, `9vrmdr…`, before and after my change.
|
||||||
|
|
||||||
|
I then tried an overlay on `pkgs.lan-mouse`, and that did not work either: the panic returned with the same message, because nixpkgs sets `LIBGIT2_NO_VENDOR` in a way the overlay could not remove.
|
||||||
|
|
||||||
|
## What works, proven
|
||||||
|
|
||||||
|
The flake's own lan-mouse package, built with the variable turned off:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
lm = lan-mouse.packages.${system}.default.overrideAttrs (o: { LIBGIT2_NO_VENDOR = "0"; });
|
||||||
|
```
|
||||||
|
|
||||||
|
Built it before wiring it in:
|
||||||
|
|
||||||
|
```
|
||||||
|
/nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Then changed two lines: `configuration.nix` now takes `lm` as an argument and uses it in `systemPackages` where `lan-mouse` was.
|
||||||
|
|
||||||
|
Result on `.13`:
|
||||||
|
|
||||||
|
```
|
||||||
|
these 12 derivations will be built:
|
||||||
|
pi: 1.5.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Twelve derivations, not 465 — most of the rest had already been substituted during the earlier attempts. Your rebuild should now go through:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo nixos-rebuild switch --flake /etc/nixos#nixos-desktop
|
||||||
|
```
|
||||||
|
|
||||||
|
If it fails on something else, send the first error block; the method is now established: build the offending package alone with an override before switching.
|
||||||
|
|
||||||
|
## About the font
|
||||||
|
|
||||||
|
Thank you for the pointer — it may well be the cause on `.27`. Its error names `home-manager.users.sam.fonts.fontconfig.configFile.fonts.settings`, and `.27` declares fonts in two places (system `fonts.packages` and again inside `systemPackages`). With home-manager 2026-11-10 that duplication is what it cannot merge. That is a small fix, but I want `.13` proven first so we are changing one thing at a time.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
| | config | evaluates | running | next |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `.13` | lock, `initContent`, lan-mouse via `lm` | pi **1.5.0**, 12 to build | 0.82.1 | **rebuild now** |
|
||||||
|
| `.27` | lock, `initContent` | fails in home-manager fontconfig | 0.82.1 | fix the font duplication, or bump nixpkgs only |
|
||||||
|
| `.51` | unchanged | pi 0.87.1 | 0.87.1 | the same three edits, then rebuild |
|
||||||
|
|
||||||
|
Written into `docs/ENVIRONMENT-FACTS.md`, committed as `5f8069d`, pushed.
|
||||||
|
|
||||||
|
Tell me what `pi --version` says after the rebuild and I will bring `.27` and `.51` to the same version.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Confidence: 86%
|
||||||
|
- Reason: Preserve full sanitized discovery detail from a rich investigation turn
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[packs/paperclip-harness/00-system/pi-agent/memory-manifest|Memory Manifest]]
|
||||||
|
|
||||||
|
## Related learned memories <!-- same-turn-2026-11-12 01:50:43 -->
|
||||||
|
|
||||||
|
- observation: [[packs/paperclip-harness/60-observations/tmp-repository-structure-and-configuration-patterns|lan-mouse LIBGIT2_NO_VENDOR build failure: nixpkgs vs flake package mismatch]]
|
||||||
|
- decision: [[packs/paperclip-harness/50-decisions/wire-lan-mouse-into-configuration-nix-as-flake-lm-override-n|Wire lan-mouse into configuration.nix as flake `lm` override, not nixpkgs package or overlay]]
|
||||||
|
- runbook: [[packs/paperclip-harness/70-runbooks/tmp-deployment-operations|NixOS fleet upgrade runbook: rebuild `.13` first, fix font duplication on `.27`, then `.51`]]
|
||||||
|
- action: [[packs/paperclip-harness/40-actions/2026-11-10-wrote-docs-ui-layout-md-and-three-live-layout-mockups|Fixed .13 lan-mouse build, updated docs/ENVIRONMENT-FACTS.md, committed 5f8069d and pushed]]
|
||||||
|
- session: [[packs/paperclip-harness/80-sessions/rich-discovery-nix-2026-11-12-01-50-43|Rich discovery: nix 2026-11-12 01:50:43]]
|
||||||
|
- context: [[packs/paperclip-harness/20-context/nix|nix]]
|
||||||
@@ -136,34 +136,57 @@ What was changed:
|
|||||||
- Local git repo only, one commit, no remote. It is not on Gitea. Push it somewhere before
|
- Local git repo only, one commit, no remote. It is not on Gitea. Push it somewhere before
|
||||||
relying on it, or accept that only this machine has it.
|
relying on it, or accept that only this machine has it.
|
||||||
|
|
||||||
## NixOS alignment for Pi, 2026-11-11
|
## NixOS alignment for Pi, measured 2026-11-12
|
||||||
|
|
||||||
Goal: all three machines on the same Pi. The `pi-coding-agent` version comes from the nixpkgs pin,
|
Goal: the same Pi on all three machines. `pi-coding-agent` comes from nixpkgs, so aligning used to
|
||||||
so aligning means bumping `nixpkgs` and `home-manager` in each machine's flake.
|
mean bumping the whole system on each machine. That is what broke `.13`: `nixos-rebuild switch`
|
||||||
|
restarts every changed unit in place, the network units restarted, the default route did not come
|
||||||
|
back, and every service that needs the network died with it. Exit code 4 meant "units failed to
|
||||||
|
start", not "the build failed" - the generation had already switched.
|
||||||
|
|
||||||
Measured after `nix flake update nixpkgs home-manager`:
|
**The rule that follows: on a machine someone is using, a nixpkgs-sized change goes in with
|
||||||
|
`nixos-rebuild boot` and a reboot, never `switch`.**
|
||||||
|
|
||||||
| machine | nixpkgs pin | pi after update | state |
|
### What was done instead, and why it is safe
|
||||||
|
|
||||||
|
Pi is one package. Instead of moving a whole system to a new nixpkgs, a second flake input pinned
|
||||||
|
to one revision provides only that package:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
nixpkgs-pi.url = "github:NixOS/nixpkgs/f4a03c008967841f2a0b3cb7c436fc6fc31671f6";
|
||||||
|
# in the let block
|
||||||
|
piPkgs = import nixpkgs-pi { inherit system; config.allowUnfree = true; };
|
||||||
|
# in home-manager users.${user}.home.packages
|
||||||
|
piPkgs.pi-coding-agent
|
||||||
|
```
|
||||||
|
|
||||||
|
and `pi-coding-agent` was removed from `home/sam/home.nix` so exactly one `pi` is on `PATH`. The
|
||||||
|
system pin on those machines did not move, so nothing else in the system changes and no network or
|
||||||
|
service unit is restarted. Remove the input once the system pins are aligned for real.
|
||||||
|
|
||||||
|
### State, verified by building, not switching
|
||||||
|
|
||||||
|
| machine | method | pi in the built system | store path |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `.13` | 2026-07-29 → 2026-11-07 | **1.5.0** | config ready, 12 derivations to build |
|
| `.13` nixos-desktop | full nixpkgs bump (2026-11-07), **running** | 1.5.0 | `4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9` |
|
||||||
| `.27` | same | 1.5.0 | evaluation fails in home-manager fontconfig |
|
| `.27` sam-4screen-desktop | `nixpkgs-pi` input, built, **not yet switched** | 1.5.0 | `4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9` |
|
||||||
| `.51` | 2026-09-26 | 0.87.1 (today) | not started |
|
| `.51` sam-thinkpad | `nixpkgs-pi` input, built, **not yet switched** | 1.5.0 | `4yxlfj0sw5hlk7x1hiv0igm0dckg8rv9` |
|
||||||
|
|
||||||
Two changes are needed with the bump, both measured:
|
One store path on all three: the same bytes of Pi, not merely the same version number.
|
||||||
|
|
||||||
1. `programs.zsh.initExtra` was renamed: `initContent`.
|
### Two other things measured on the way
|
||||||
2. **lan-mouse 0.11.0 panics** when `LIBGIT2_NO_VENDOR` is set and there is no system libgit2
|
|
||||||
(nixpkgs 2026-11-07 sets it). An overlay on `pkgs.lan-mouse` did **not** fix it — the
|
|
||||||
derivation hash barely moved and the same panic came back. What does work is building the
|
|
||||||
flake's own lan-mouse package with `LIBGIT2_NO_VENDOR = "0"`:
|
|
||||||
|
|
||||||
```nix
|
- `programs.zsh.initExtra` is `initContent` in home-manager 2026-11. Both names evaluate on the
|
||||||
lm = lan-mouse.packages.${system}.default.overrideAttrs (o: { LIBGIT2_NO_VENDOR = "0"; });
|
older home-manager, so the rename is safe in either direction.
|
||||||
```
|
- **lan-mouse 0.11.0 panics** when `LIBGIT2_NO_VENDOR` is set and no system libgit2 exists; nixpkgs
|
||||||
|
2026-11-07 sets it. An overlay on `pkgs.lan-mouse` does **not** help. Building the flake's own
|
||||||
|
package with the variable off does:
|
||||||
|
|
||||||
and using `lm` in `systemPackages` instead of the plain package. Built and verified:
|
```nix
|
||||||
`/nix/store/f6w228jqp5wq6xznf6arq69kck6axyyf-lan-mouse-0.11.0`.
|
lm = lan-mouse.packages.${system}.default.overrideAttrs (o: { LIBGIT2_NO_VENDOR = "0"; });
|
||||||
|
```
|
||||||
|
|
||||||
`.27`'s failure names `home-manager.users.sam.fonts.fontconfig.configFile.fonts.settings`. It is a
|
`.13` now lists `lm` in `systemPackages` instead of `lan-mouse`.
|
||||||
home-manager option merge, not the Pi change. The smaller next attempt there is to bump `nixpkgs`
|
- `.27`'s full-system bump failed in `home-manager.users.sam.fonts.fontconfig.configFile.fonts.settings`
|
||||||
only and leave `home-manager` where it is.
|
(`recursiveUpdateUntil` / `head values`). The `nixpkgs-pi` route avoids it entirely; it is still
|
||||||
|
an open item for the day the system pins are aligned.
|
||||||
|
|||||||
Reference in New Issue
Block a user