From 30d87e74a262513d75d4a52df451f9ca008ab2d3 Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Tue, 1 Sep 2026 13:39:08 +1000 Subject: [PATCH] sam-4screen-desktop 2026-9-1:13:39:8 --- .obsidian/workspace.json | 2 +- ...Backup Architecture — Offsite to AWS S3.md | 3 +- .../Migration Work Log.md | 11 ++ .../220 Web Host Migration/Restore Runbook.md | 105 ++++++++++++++++++ 4 files changed, 119 insertions(+), 2 deletions(-) create mode 100644 200 projects/220 Web Host Migration/Restore Runbook.md diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index c329e77..bd80b65 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -212,6 +212,7 @@ }, "active": "8018aa52f0591df9", "lastOpenFiles": [ + "200 projects/220 Web Host Migration/Restore Runbook.md", "200 projects/240 Web Sites/Where Woof.md", "300 areas/380 Social Politics Economics Issues/Chinese Non Silicon Micro Chip.md", "200 projects/220 Web Host Migration/Backup Action Plan — Close the Gaps + Restore.md", @@ -237,7 +238,6 @@ "200 projects/220 Web Host Migration/Phase 2 — Email.md", "200 projects/220 Web Host Migration/Phase 1 — AWS Offsite Backup.md", "200 projects/220 Web Host Migration/Phase 0 — Pre-flight.md", - "200 projects/220 Web Host Migration/Migration Plan — InMotion to Local + AWS Offsite.md", "000 daily/ThinkPad Recommendations", "500 archive/510 Daily", "300 areas/305 Ideas Businesses", diff --git a/200 projects/220 Web Host Migration/Backup Architecture — Offsite to AWS S3.md b/200 projects/220 Web Host Migration/Backup Architecture — Offsite to AWS S3.md index 72358fa..d1add08 100644 --- a/200 projects/220 Web Host Migration/Backup Architecture — Offsite to AWS S3.md +++ b/200 projects/220 Web Host Migration/Backup Architecture — Offsite to AWS S3.md @@ -93,4 +93,5 @@ If photos + archive are genuinely cold (only needed in disaster), upload them wi - [ ] Security: rotate access key after seeding (it was shared in chat); root MFA now enabled --- -*See [[Migration Work Log]] for live track record; [[Migration Plan — InMotion to Local + AWS Offsite]] §9 for the master cost model.* \ No newline at end of file +*See [[Migration Work Log]] for live track record; [[Migration Plan — InMotion to Local + AWS Offsite]] §9 for the master cost model. +*➡ **Restore procedures: [[Restore Runbook]]** (how to bring back each machine, local or S3).* \ No newline at end of file diff --git a/200 projects/220 Web Host Migration/Migration Work Log.md b/200 projects/220 Web Host Migration/Migration Work Log.md index acbc2f9..e8d325a 100644 --- a/200 projects/220 Web Host Migration/Migration Work Log.md +++ b/200 projects/220 Web Host Migration/Migration Work Log.md @@ -80,6 +80,17 @@ id: 1848129023-LOG ## Pending next action +## 2026-09-01 — BACKUP GAP CLOSURE (tasks in progress) +| Phase | What was done | Status | +|---|---|---| +| .13 Borg | `backup.nix` `local-to-storage` paths expanded: + `~/deployment`(23G), gst-plugins-rs, speech_piper, voice_bridge, whisper-api, voice-agent, librespot, mopidy-radionet, bin, .agents, .pi, .config; exclusions + `.cache`, `.npm`, `.local`, nixos_backup.tar.gz | ✅ config applied (rebuild done) | +| .13 Borg | **prefect live DB excluded** (caused recurring `.failed` runs) | ✅ config in, applies on next rebuild | +| .13 Borg | First expanded run in progress (23G initial archive, slow on lzma) | 🔄 running | +| S3 seed | `offsite.nix` + sync of `borg/.13` (STANDARD) + home dirs (DEEP_ARCHIVE) | ✅ config in; auto-syncs tonight 23:00 | +| Restore runbook | to be written | ⏳ Task 3 | + +## Next steps + ## PHASE 2 EMAIL — progress log (2026-08-31) | Item | Status | |---|---| diff --git a/200 projects/220 Web Host Migration/Restore Runbook.md b/200 projects/220 Web Host Migration/Restore Runbook.md new file mode 100644 index 0000000..1f587b7 --- /dev/null +++ b/200 projects/220 Web Host Migration/Restore Runbook.md @@ -0,0 +1,105 @@ +--- +created: 2026-09-01 +modified: 2026-09-01 +type: runbook +client: sam +project: web-host-migration +status: active +priority: 1 +tags: + - backup + - restore + - runbook + - borg + - kopia + - restic + - s3 +aliases: + - restore-runbook + - how-to-restore +id: 1848129028-RR +--- +# Restore Runbook — bring it all back + +> How to restore every machine/data from local backups (`.23`) or from **S3** (house-fire scenario). +> Linked from [[Backup Architecture — Offsite to AWS S3]] and [[Backup Action Plan — Close the Gaps + Restore]]. + +## Restore principle +- **Machine died, .23 alive** → restore from **local repos** (fast, minutes–hours). +- **House fire / .23 gone** → restore from **S3** (photos use DEEP_ARCHIVE → hours to retrieve, then download; Borg repos STANDARD → fast). + +--- + +## 1. Restore .27 desktop (from Borg `borg/.27`) + +**Local (fast):** +```bash +export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase) +borg extract sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27:: +``` +**From S3 (if .13 gone):** +```bash +# on any machine with aws cli + borg +source /home/sam/.aws-offsite-env.sh +aws s3 sync s3://sam-offsite-backup/borg/.27/ ./borg-repo/ +export BORG_PASSPHRASE=$(cat ~/.borg-passphrase) +borg extract ./borg-repo:: +# or borg mount to browse +``` + +## 2. Restore .13 (from Borg `borg/.13`) + +**Local (fast):** +```bash +export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase) +borg list /mnt/ubuntu_storage_3TB/backup/borg/.13 # find archive +borg extract /mnt/ubuntu_storage_3TB/backup/borg/.13:: # run from / +``` +**From S3 (house fire):** +```bash +aws s3 sync s3://sam-offsite-backup/borg/.13/ ./borg-repo/ +borg extract ./borg-repo:: +``` +Contents restored: `~/Docker`, `~/deployment`, voice/audio dirs, dotfiles, `.ssh`, `.agents`, `.pi`, `~/bin`, `.config`, pocketbase, family-home-lab, photo-pipeline. (prefect DB excluded by design — restore a prefect DB dump separately if needed.) + +## 3. Restore .35 containers / Gitea (from Kopia/Restic on .23) + +**Local (repos on .23 via .35):** +- **Backrest (Restic):** open `http://<.35>:9898` (Backrest UI) → restore plan → pick snapshot → restore `/docker-volumes`, `/docker-containers`, `/host_fs/mnt/hd`. +- **Kopia:** `docker exec kopia kopia restore /restore-target` (snapshots of `/host_fs/Docker/Containers`, `/host_fs/var/lib/docker/volumes`, trilium-data). +- **Gitea:** its data is `/Docker/Containers/gitea` → inside the Docker_Containers restic snapshot → restore then `docker compose up -d` in `/Docker/Containers/gitea/`. +**From S3:** the restic/kopia repos are NOT offsited (decision 2026-09-01) — .35 containers restore requires .23. They are rebuildable from configs (which are in borg/.13 + Gitea) + Docker Compose files. + +## 4. Restore photos (by_date master) — from S3 DEEP_ARCHIVE + +**⚠️ DEEP_ARCHIVE restore takes HOURS (12–48h) — plan for it.** +```bash +aws s3 ls s3://sam-offsite-backup/photos/by_date/ --recursive | wc -l # sanity +# Option 1: bulk +aws s3 sync s3://sam-offsite-backup/photos/by_date/ /mnt/ubuntu_storage_3TB/archive/03_photos/by_date/ +# Option 2: single file (one-off, pay per retrieve) +aws s3api restore-object --bucket sam-offsite-backup --key photos/by_date/ --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Bulk"}}' +``` +- Actual file data downloads **only after** the restore job completes. +- **Immich originals** (`photos/immich/`, if added later) same procedure. + +## 5. Restore email / DNS +- DNS: Cloudflare is authoritative — nothing to restore (cloud-managed). +- Email: Cloudflare Email Routing + SES — cloud services, nothing local to restore. + +--- + +## Test-restore cadence (recommended) +- **Monthly:** restore ONE small file from S3 (e.g. a borg chunk or a single photo) → proves read path works. +- **Quarterly:** restore one complete small Borg archive → verify integrity (`borg check`). + +## Key commands (quick reference) +| Task | Command | +|---|---| +| List .27 archives | `borg list sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27` | +| List .13 archives | `export BORG_PASSPHRASE=$(cat ~/.borg-passphrase); borg list /mnt/ubuntu_storage_3TB/backup/borg/.13` | +| Backrest UI | `http://192.168.20.35:9898` | +| Check S3 sizes | `aws s3 ls s3://sam-offsite-backup --summarize` | + +--- +*Part of [[Migration Plan — InMotion to Local + AWS Offsite]]. Track in [[Migration Work Log]].* \ No newline at end of file