obsidian-vault/300 areas/360 Dev-Ops Network Computers/Backup System — Borg, Kopia & Restic.md

created, modified, type, tags, aliases

created	modified	type	tags	aliases
2026-05-20 09:30	2026-05-28	note	dev-ops network backup system-config localhost	borg-backups

Backup System — Borg, Kopia & Restic

Overview

Three-machine backup with dual-copy (local + network) covering all critical data.

Machine	Tool	Target	Schedule
.27 — sam-4screen-desktop	Borg (SSH)	.13:/mnt/ubuntu_storage_3TB/backup/borg/.27	Daily (after boot)
.13 — nixos-desktop	Borg (local)	/mnt/ubuntu_storage_3TB/backup/borg/.13 + rsync→.35/.23	Daily 04:00 / 05:00
.13 — archive	Borg (SSH)	.35:/mnt/nfs_usb_3tb/borg-repos/archive	NEW 2026-05-30
.35 — sam-ubuntu1	Kopia + Backrest	.23 NFS (usb_3tb)	Weekly Sun 22:00

---

.27 — sam-4screen-desktop

Source: /home/sam/ (excl: .cache, .npm, node_modules, .venv, target, .cargo, etc.) Target: sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27 via SSH (key: ~/.ssh/id_ed25519) Schedule: systemd timer — OnCalendar=daily (fires midnight, catches up after boot via Persistent) Prune: 7 daily, 4 weekly, 3 monthly Passphrase: /home/sam/.borg-passphrase Config: /etc/nixos/hosts/sam-4screen-desktop/backup.nix Status: systemctl status borgbackup-job-home-to-13.service Snapshots: borg list sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27

.13 — nixos-desktop

Source: Docker configs (/home/sam/Docker/), PocketBase (/home/sam/pocketbase-data/), dotfiles, .ssh/ Primary: /mnt/ubuntu_storage_3TB/backup/borg/.13 (local ext4, 2.7T, 2.6T free) Secondary: rsync → sam@192.168.20.35:/mnt/nfs_usb_3tb/borg-repos/.13 → .23 USB Schedule: Borg daily 04:00 | rsync daily 05:00 Prune: 7 daily, 4 weekly, 3 monthly Passphrase: /home/sam/.borg-passphrase (same as .27 recommended) Config: /etc/nixos/backup.nix Snapshots: borg list /mnt/ubuntu_storage_3TB/backup/borg/.13

⚠️ Borg runs as root — postHook chowns to sam:users. Known hosts issue: /home/sam/.ssh/known_hosts was a broken directory (fixed).

.35 — sam-ubuntu1

Tools: Kopia + Backrest (Restic) — Docker containers Target: //192.168.20.23/usb_3tb → /mnt/nfs_usb_3tb/

• Kopia repo: /mnt/nfs_usb_3tb/kopia/backups/ (46G)
• Restic repo: /mnt/nfs_usb_3tb/backups/restic/ (11G)

Backup Sources

Source	Tool	Status
/Docker/Containers/	Kopia	✅ Weekly Sun 22:00
/docker-config, /docker-volumes	Backrest (Restic)	✅ Weekly Sun 23:00
/host_fs/root/trilium-data	Backrest (Restic)	✅ Weekly Sun 23:00
/host_fs/mnt/hd ← new	Backrest (Restic)	✅ Added 2026-05-28
/host_fs/var/lib/docker/volumes	Kopia	✅ Weekly Sun 22:00

Retention

• Kopia: 10 latest, 4 weekly, 12 monthly, 3 annual
• Backrest: 10 latest (prune/check daily)

Data Drive (My Passport)

.dev/vda1 — 1.8T NTFS — physical WD My Passport passed through to VM Mount point: /mnt/hd (was incorrectly at /mnt/my_passport — fixed 2026-05-28) Contains: Immich photos (40K), and all Docker service data (audiobookshelf, lidarr, radarr, sonarr, qbittorrent, etc.)

VM-level

Proxmox Backup Server (.48) — full VM backup

.23 — file-server

• 2.7T USB drive shared via CIFS/SMB to .35
• Backup repos: Kopia (46G) + Restic (13G) + Borg archive repo (new)
• 2.0T free
• Also stores: sam/ (personal docs/music/photos) — to be moved to .13 archive

Borg archive repo (new 2026-05-30)

• Location: /mnt/nfs_usb_3tb/borg-repos/archive
• Source: .13 /mnt/ubuntu_storage_3TB/archive/ (648G — photos, old web work, documents)
• Encryption: repokey-blake2
• Compression: auto,lzma
• Run from .13: borg create sam@192.168.20.35:/mnt/nfs_usb_3tb/borg-repos/archive::archive-$(date +%Y%m%d) /mnt/ubuntu_storage_3TB/archive/

Restore Commands

.27 restore

export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase)
borg extract sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27::<archive>

.13 restore

export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase)
borg extract /mnt/ubuntu_storage_3TB/backup/borg/.13::<archive>

Not Covered

• .27: Git-tracked NixOS config (already covered by flakes)
• .13: Git-tracked NixOS config, ollama/open-webui Docker volumes
• .35: .23 file-server has no backup — single point of failure for restic/kopia repos
• .23: Itself has no backup at all
• .13 archive — now backed up to .23 Borg repo ✅

UPDATE 2026-09: The AWS S3 offsite layer now covers the highest-value data (see below): photos, .13 Borg repo, .27 Borg repo, and .13 home configs are offsite in sam-offsite-backup. This mitigates the .23 single-point-of-failure for that data (a fire loses .23 but S3 survives). Full restore procedures in Restore Runbook.

Offsite S3 (AWS) — added 2026-09

Item	Detail
Bucket	sam-offsite-backup (region ap-southeast-2 / Sydney), SSE, versioning, lifecycle S3→Glacier Flexible→Deep Archive
Tool	AWS CLI aws s3 sync (run from .13) — append-only, no delete
Daily job	offsite-sync.service + timer, 23:00–05:00 (Melbourne), 6h window; runs .13 repo → .27 repo → home configs FIRST, then photos/archive
Verifier + alert	offsite-verify.service 05:15 daily — checks object-count growth, ntfy push to phone on no-progress/error; log at /var/www/offsite/verify.log
Monitoring page	https://offsite.lab.audasmedia.com.au (basic-auth sam) — live object counts per prefix, 7-night history, service state, disk. Auto-refresh 15 min
IAM	user backup-offsite (object-only, least privilege on the bucket)
Cost	~$1–5 AUD/mo (photos/archive in Deep Archive; repos in Standard)
Credentials	/home/sam/.aws-credentials.local (600, on .13) + aws-credentials.local (.27)

Backed up to S3: photos/by_date, archive/rest (Deep Archive) · borg/.27, borg/.13 (Standard) · home configs (~/deployment, voice/audio, .agents, .pi, etc., Deep Archive).

Restore: see Restore Runbook (per-machine local + S3 steps; Deep Archive restore = hours).

Garage (local S3) — .13

Item	Detail
Role	Local S3-compatible object store (family-home-lab compose); used by portal/media apps
Version	v2.1.0 (upgraded from v1.0.1, 2026-09-05 — v1 wouldn't work with garage-webui; data auto-migrated on start, backup taken first)
S3 API	[::]:3900 (region garage)
S3 Web	[::]:3902 (vhost .web.lab.audasmedia.com.au, index.html) — ⚠️ requires per-bucket auth; NOT anonymous static hosting
Admin API	[::]:3903 (v2 endpoints /v2/...); token in garage.toml/.env
Web UI	garage-webui container → port 3909, published as https://garage-ui.lab.audasmedia.com.au (basic-auth sam)
Config	/home/sam/Docker/Containers/family-home-lab/garage/garage.toml
Backup	Bucket data = app data, included in .13/.35 container backups per app; data+meta snapshot at /mnt/data/family-home-lab/garage-backup-20260905-111241/

⚠️ Garage S3-web gotcha (learned 2026-09-09)

Garage's S3 Web (:3902) is not a drop-in anonymous static host. It rejects anonymous access (403 Forbidden: Garage does not support anonymous access yet) and needs per-bucket web access + a read key. For serving site images:

• Use the S3 API (boto3/aws CLI with a read-scoped key) for uploads, and
• Serve images via the app / Caddy (e.g. reverse-proxy to Garage with the read key, or a Caddy file_server dir) — do NOT expose :3902 for anonymous browsing.

Recent Changes

• 2026-05-28: Discovered My Passport (/dev/vda1, 1.8T) had all real data but was mounted at /mnt/my_passport instead of /mnt/hd. Remounted to /mnt/hd. Immich photos (40K) and all Docker service data recovered.
• 2026-05-28: Added /host_fs/mnt/hd to Backrest (Restic) backup plan.
• 2026-05-30: Created Borg archive repo on .23 (/mnt/nfs_usb_3tb/borg-repos/archive) to back up .13 /mnt/ubuntu_storage_3TB/archive/ (648G). First backup pending.
• 2026-05-30: Reformatted .13 /mnt/storage/ (932G NTFS → ext4) and /mnt/data/ (1.8T NTFS → ext4). Both now empty with 870G and 1.7T free respectively.