docs: secret rotation plan after 2026-11-11 compromise (3 SSH keys, 47 secrets + 20 .env files exposed)

docs/secret-rotation-2026-11-11.md

@@ -0,0 +1,88 @@
+# Secret rotation plan — 2026-11-11
+
+**Why:** the compromised GIMP container had `/home/sam` mounted **read-only for 5 days**
+(2026-11-06 → 2026-11-11) and ran as uid 1000 (`sam`). Everything below was **readable**,
+and LMMS additionally had a reverse shell giving arbitrary command execution as root inside
+a container with a write mount into `/home/sam/Docker/Containers/music`.
+
+Treat **every credential listed here as compromised**. See `incident-2026-11-11.md`.
+
+## Exposure inventory (what was readable)
+
+| Store | Contents | Volume |
+|---|---|---|
+| `~/.ssh/id_ed25519`, `id_ed25519_rsync`, `id_ed25519_school_scan` | **3 private keys** | full host/remote access |
+| `~/.config/environment.d/10-secrets.conf` | **47 keys** (API keys, tokens, passwords) | see table below |
+| …`10-secrets.conf.bak*` | **12 historical copies** — old AND current secrets | doubles the exposure |
+| `~/.aws-credentials.local` | AWS access keys | cloud billing |
+| `~/.secrets` (1.5 KB) | misc credentials | varies |
+| `~/.gemini/mcp-oauth-tokens-v2.json` | OAuth tokens | Google account scope |
+| `~/.npmrc` | npm registry token | supply-chain publish risk |
+| **20 `.env` files** (~200 keys total) | per-project service secrets | see counts below |
+
+Largest `.env` files: `sams-home-network` (43 keys), `wherewoof-admin` (28), `voice-agent` (21),
+`family-home-lab` (19), `langfuse` (14), `outline` (12).
+
+## Priority 1 — rotate TODAY (direct access or money)
+
+| Credential | Why urgent | Rotate where | Then |
+|---|---|---|---|
+| `id_ed25519`, `id_ed25519_rsync`, `id_ed25519_school_scan` | Attacker can SSH as you | `ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_new`; remove old pubkeys from `authorized_keys` on every target | Update `.ssh/config`, rsync targets, school-scan host |
+| `GITHUB_TOKEN` | Code push/steal, supply chain | github.com → Settings → Developer settings → Tokens | `gh auth login` to refresh |
+| `GITEA_TOKEN` | Same, self-hosted repos | Gitea → Settings → Applications → regenerate | update `10-secrets.conf` |
+| `HUB_TOKEN_ADMIN`, `HUB_TOKEN_DSH`, `HUB_TOKEN_CONSOLE`, `HUB_TOKEN_PHONE`, `HUB_TOKEN_ESP32`, `HUB_PROJECT_*_TOKEN` | Full control of your hub/API layer | hub admin UI → revoke + reissue per client | update ESP32 firmware + phone config |
+| `SC_TOKEN` | Service-controller = start/stop anything | regenerate in `10-secrets.conf` | also `family-home-lab/.env`, restart controller |
+| `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `DEEPSEEK_API_KEY`, `GEMINI_API_KEY`, `MOONSHOT_API_KEY`, `MINIMAX_API_KEY` (+`_GROUP_ID`,`_SUBSCRIPTION_KEY`), `OPENCODE_API_KEY`, `TOKENROUTER_API_KEY`, `OMNIROUTE_API_KEY`, `STITCH_API_KEY`, `JEV_API_KEY`, `MOUSE_API_KEY`, `OL_API_KEY`, `TYPESAFE_API_KEY`, `TAVILY_API_KEY` | **Billing** — check usage dashboards for abuse | each provider console | update `10-secrets.conf` + dsh configs |
+| `GMAIL_APP_PASSWORD`, `EMAIL_AGENT_PASSWORD`, `EMAIL_AGENT_SESSION_SECRET` | Mailbox takeover, spam/phishing from your address | Google account → App passwords → revoke all | update voice-agent + email_agent |
+| `COMPASS_PASSWORD`, `PASEO_PASSWORD` | Account takeover | respective apps | update `.env`/conf |
+| `MQTT_USER`/`MQTT_PASS` | Control smart-home devices | Mosquitto `passwd` file | update all ESP/HA clients |
+| `AWS` keys in `~/.aws-credentials.local` | Cloud spend, data | IAM → deactivate access key → create new | update profile |
+| `~/.npmrc` token | Publish malicious packages under your name | npmjs.com → Access Tokens → revoke | `npm login` |
+| `~/.gemini/mcp-oauth-tokens-v2.json` | Google scope | revoke at myaccount.google.com → Security → Third-party access | re-auth |
+| `VIKUNJA_TOKEN`, `N8N_API_KEY`, `N8N_MCP_TOKEN`, `PAPERCLIP_API_KEY`, `PAPERCLIP_BOARD_KEY`, `APPRISE_URLS` | Task/automation control; apprise URLs embed tokens | each app's settings | update consumers |
+
+## Priority 2 — rotate this week (internal blast radius)
+
+- `POSTGRES_PASSWORD` / `POSTGRES_USER` / `POSTGRES_DB` (family-home-lab)
+- `SESSION_SECRET`, `ADMIN_PASSWORD` (portal — invalidate all sessions, log everyone out)
+- `GARAGE_RPC_SECRET`, `GARAGE_ADMIN_TOKEN`, `GARAGE_METRICS_TOKEN`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`
+- `RABBITMQ` / `REDIS` passwords if set in `.env`
+- `langfuse`, `outline`, `worldmonitor`, `omniroute`, `wherewoof-*` DB/API keys
+- `WEBTOP_PASSWORD` (new, created after the incident — **not** exposed, no action)
+
+## Priority 3 — no action (not secrets)
+
+`MOZ_ENABLE_WAYLAND`, `GITEA_URL`, `GMAIL_USER`, `COMPASS_USERNAME`, `HUB_PROJECTS`,
+`PASEO_HOST`, `PI_NTFY_TOPIC` (topic name only), ports/paths.
+
+## Hygiene fixes (do alongside rotation)
+
+1. **Delete the 12 `10-secrets.conf.bak*` files** — they hold superseded secrets and multiply
+   exposure. Keep **one** encrypted backup instead.
+2. `chmod 600` every file under `~/.config/environment.d` and confirm `~/.ssh/id_*` are `600`
+   (currently correct; `.pub` files at 644 are fine).
+3. **Never mount `/home/sam` into a container again** — removed from GIMP already. Mount only
+   the specific directory needed.
+4. Consider `sops-nix` / `agenix` so secrets are encrypted at rest, and so a read-only home
+   mount can never expose them in plaintext.
+5. Audit `.env` files for values that don't need to be secrets at all.
+
+## Rotation order that avoids downtime
+
+1. Revoke first (kills attacker access), even if it breaks things for a few minutes.
+2. Regenerate + write into `10-secrets.conf`.
+3. `systemctl --user daemon-reload` / re-login for `environment.d` to apply.
+4. Restart consumers (portal, controller, dsh, voice-agent, n8n, worldmonitor…).
+5. Verify each service health endpoint, then update the `.env` copies.
+6. Only then delete the old `.bak` files.
+
+## Verification checklist
+
+- [ ] `ssh -T git@github.com` works with a new key; old key rejected on target hosts
+- [ ] Gitea push works with new token
+- [ ] Portal login works; old sessions invalidated (forced logout)
+- [ ] Service-controller `/health` OK with new `SC_TOKEN`
+- [ ] dsh + models respond (new AI provider keys)
+- [ ] MQTT devices reconnect (new broker creds)
+- [ ] Email agent sends a test message
+- [ ] No unexpected spend on provider billing dashboards for 2026-11-06 → 11-11