--- created: 2026-05-20 09:30 modified: 2026-05-28 type: note tags: - dev-ops - network - backup - system-config - localhost aliases: - borg-backups --- # [[Backup System — Borg, Kopia & Restic]] ## Overview Three-machine backup with dual-copy (local + network) covering all critical data. | Machine | Tool | Target | Schedule | |---------|------|--------|----------| | **.27** — sam-4screen-desktop | Borg (SSH) | `.13:/mnt/ubuntu_storage_3TB/backup/borg/.27` | Daily (after boot) | | **.13** — nixos-desktop | Borg (local) | `/mnt/ubuntu_storage_3TB/backup/borg/.13` + rsync→.35/.23 | Daily 04:00 / 05:00 | | **.13** — archive | Borg (SSH) | `.35:/mnt/nfs_usb_3tb/borg-repos/archive` | **NEW 2026-05-30** | | **.35** — sam-ubuntu1 | Kopia + Backrest | `.23` NFS (`usb_3tb`) | Weekly Sun 22:00 | --- ## .27 — sam-4screen-desktop **Source**: `/home/sam/` (excl: `.cache`, `.npm`, `node_modules`, `.venv`, `target`, `.cargo`, etc.) **Target**: `sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27` via SSH (key: `~/.ssh/id_ed25519`) **Schedule**: systemd timer — `OnCalendar=daily` (fires midnight, catches up after boot via Persistent) **Prune**: 7 daily, 4 weekly, 3 monthly **Passphrase**: `/home/sam/.borg-passphrase` **Config**: `/etc/nixos/hosts/sam-4screen-desktop/backup.nix` **Status**: `systemctl status borgbackup-job-home-to-13.service` **Snapshots**: `borg list sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27` ## .13 — nixos-desktop **Source**: Docker configs (`/home/sam/Docker/`), PocketBase (`/home/sam/pocketbase-data/`), dotfiles, `.ssh/` **Primary**: `/mnt/ubuntu_storage_3TB/backup/borg/.13` (local ext4, 2.7T, 2.6T free) **Secondary**: rsync → `sam@192.168.20.35:/mnt/nfs_usb_3tb/borg-repos/.13` → `.23` USB **Schedule**: Borg daily 04:00 | rsync daily 05:00 **Prune**: 7 daily, 4 weekly, 3 monthly **Passphrase**: `/home/sam/.borg-passphrase` (same as .27 recommended) **Config**: `/etc/nixos/backup.nix` **Snapshots**: `borg list /mnt/ubuntu_storage_3TB/backup/borg/.13` > ⚠️ Borg runs as root — `postHook` chowns to `sam:users`. Known hosts issue: `/home/sam/.ssh/known_hosts` was a broken directory (fixed). ## .35 — sam-ubuntu1 **Tools**: Kopia + Backrest (Restic) — Docker containers **Target**: `//192.168.20.23/usb_3tb` → `/mnt/nfs_usb_3tb/` - Kopia repo: `/mnt/nfs_usb_3tb/kopia/backups/` (46G) - Restic repo: `/mnt/nfs_usb_3tb/backups/restic/` (11G) ### Backup Sources | Source | Tool | Status | |--------|------|--------| | `/Docker/Containers/` | Kopia | ✅ Weekly Sun 22:00 | | `/docker-config`, `/docker-volumes` | Backrest (Restic) | ✅ Weekly Sun 23:00 | | `/host_fs/root/trilium-data` | Backrest (Restic) | ✅ Weekly Sun 23:00 | | **`/host_fs/mnt/hd`** ← _new_ | **Backrest (Restic)** | ✅ **Added 2026-05-28** | | **`/host_fs/var/lib/docker/volumes`** | Kopia | ✅ Weekly Sun 22:00 | ### Retention - Kopia: 10 latest, 4 weekly, 12 monthly, 3 annual - Backrest: 10 latest (prune/check daily) ### Data Drive (My Passport) .dev/vda1 — 1.8T NTFS — physical WD My Passport passed through to VM **Mount point**: `/mnt/hd` (was incorrectly at `/mnt/my_passport` — fixed 2026-05-28) **Contains**: Immich photos (40K), and all Docker service data (audiobookshelf, lidarr, radarr, sonarr, qbittorrent, etc.) ### VM-level Proxmox Backup Server (.48) — full VM backup ## .23 — file-server - 2.7T USB drive shared via CIFS/SMB to `.35` - Backup repos: Kopia (46G) + Restic (13G) + Borg archive repo (new) - 2.0T free - Also stores: `sam/` (personal docs/music/photos) — to be moved to .13 archive ### Borg archive repo (new 2026-05-30) - **Location**: `/mnt/nfs_usb_3tb/borg-repos/archive` - **Source**: `.13` `/mnt/ubuntu_storage_3TB/archive/` (648G — photos, old web work, documents) - **Encryption**: repokey-blake2 - **Compression**: auto,lzma - **Run from .13**: `borg create sam@192.168.20.35:/mnt/nfs_usb_3tb/borg-repos/archive::archive-$(date +%Y%m%d) /mnt/ubuntu_storage_3TB/archive/` ## Restore Commands ### .27 restore ```bash export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase) borg extract sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27:: ``` ### .13 restore ```bash export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase) borg extract /mnt/ubuntu_storage_3TB/backup/borg/.13:: ``` ## Not Covered - `.27`: Git-tracked NixOS config (already covered by flakes) - `.13`: Git-tracked NixOS config, `ollama`/`open-webui` Docker volumes - `.35`: **`.23` file-server has no backup** — single point of failure for restic/kopia repos - `.23`: Itself has no backup at all - **`.13` archive** — now backed up to .23 Borg repo ✅ > **UPDATE 2026-09:** The **AWS S3 offsite** layer now covers the highest-value data (see below): photos, `.13` Borg repo, `.27` Borg repo, and `.13` home configs are offsite in `sam-offsite-backup`. This **mitigates the `.23` single-point-of-failure for that data** (a fire loses .23 but S3 survives). Full restore procedures in **[[Restore Runbook]]**. ## Offsite S3 (AWS) — added 2026-09 | Item | Detail | |---|---| | **Bucket** | `sam-offsite-backup` (region ap-southeast-2 / Sydney), SSE, versioning, lifecycle S3→Glacier Flexible→Deep Archive | | **Tool** | AWS CLI `aws s3 sync` (run from `.13`) — append-only, no delete | | **Daily job** | `offsite-sync.service` + timer, **23:00–05:00** (Melbourne), 6h window; runs `.13` repo → `.27` repo → home configs FIRST, then photos/archive | | **Verifier + alert** | `offsite-verify.service` 05:15 daily — checks object-count growth, **ntfy push** to phone on no-progress/error; log at `/var/www/offsite/verify.log` | | **Monitoring page** | `https://offsite.lab.audasmedia.com.au` (basic-auth `sam`) — live object counts per prefix, 7-night history, service state, disk. Auto-refresh 15 min | | **IAM** | user `backup-offsite` (object-only, least privilege on the bucket) | | **Cost** | ~$1–5 AUD/mo (photos/archive in Deep Archive; repos in Standard) | | **Credentials** | `/home/sam/.aws-credentials.local` (600, on .13) + `aws-credentials.local` (.27) | **Backed up to S3:** photos/by_date, archive/rest (Deep Archive) · borg/.27, borg/.13 (Standard) · home configs (`~/deployment`, voice/audio, `.agents`, `.pi`, etc., Deep Archive). **Restore:** see **[[Restore Runbook]]** (per-machine local + S3 steps; Deep Archive restore = hours). ## Recent Changes - **2026-05-28**: Discovered My Passport (`/dev/vda1`, 1.8T) had all real data but was mounted at `/mnt/my_passport` instead of `/mnt/hd`. Remounted to `/mnt/hd`. Immich photos (40K) and all Docker service data recovered. - **2026-05-28**: Added `/host_fs/mnt/hd` to Backrest (Restic) backup plan. - **2026-05-30**: Created Borg archive repo on `.23` (`/mnt/nfs_usb_3tb/borg-repos/archive`) to back up `.13` `/mnt/ubuntu_storage_3TB/archive/` (648G). First backup pending. - **2026-05-30**: Reformatted `.13` `/mnt/storage/` (932G NTFS → ext4) and `/mnt/data/` (1.8T NTFS → ext4). Both now empty with 870G and 1.7T free respectively.