166 lines
8.4 KiB
Markdown
166 lines
8.4 KiB
Markdown
---
|
||
created: 2026-05-20 09:30
|
||
modified: 2026-05-28
|
||
type: backup-system
|
||
status: active
|
||
tags:
|
||
- dev-ops
|
||
- network
|
||
- backup
|
||
- system-config
|
||
- localhost
|
||
aliases:
|
||
- borg-backups
|
||
---
|
||
# [[Backup System — Borg, Kopia & Restic]]
|
||
|
||
## Overview
|
||
|
||
Three-machine backup with dual-copy (local + network) covering all critical data.
|
||
|
||
| Machine | Tool | Target | Schedule |
|
||
|---------|------|--------|----------|
|
||
| **.27** — sam-4screen-desktop | Borg (SSH) | `.13:/mnt/ubuntu_storage_3TB/backup/borg/.27` | Daily (after boot) |
|
||
| **.13** — nixos-desktop | Borg (local) | `/mnt/ubuntu_storage_3TB/backup/borg/.13` + rsync→.35/.23 | Daily 04:00 / 05:00 |
|
||
| **.13** — archive | Borg (SSH) | `.35:/mnt/nfs_usb_3tb/borg-repos/archive` | **NEW 2026-05-30** |
|
||
| **.35** — sam-ubuntu1 | Kopia + Backrest | `.23` NFS (`usb_3tb`) | Weekly Sun 22:00 |
|
||
|
||
---
|
||
|
||
## .27 — sam-4screen-desktop
|
||
|
||
**Source**: `/home/sam/` (excl: `.cache`, `.npm`, `node_modules`, `.venv`, `target`, `.cargo`, etc.)
|
||
**Target**: `sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27` via SSH (key: `~/.ssh/id_ed25519`)
|
||
**Schedule**: systemd timer — `OnCalendar=daily` (fires midnight, catches up after boot via Persistent)
|
||
**Prune**: 7 daily, 4 weekly, 3 monthly
|
||
**Passphrase**: `/home/sam/.borg-passphrase`
|
||
**Config**: `/etc/nixos/hosts/sam-4screen-desktop/backup.nix`
|
||
**Status**: `systemctl status borgbackup-job-home-to-13.service`
|
||
**Snapshots**: `borg list sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27`
|
||
|
||
## .13 — nixos-desktop
|
||
|
||
**Source**: Docker configs (`/home/sam/Docker/`), PocketBase (`/home/sam/pocketbase-data/`), dotfiles, `.ssh/`
|
||
**Primary**: `/mnt/ubuntu_storage_3TB/backup/borg/.13` (local ext4, 2.7T, 2.6T free)
|
||
**Secondary**: rsync → `sam@192.168.20.35:/mnt/nfs_usb_3tb/borg-repos/.13` → `.23` USB
|
||
**Schedule**: Borg daily 04:00 | rsync daily 05:00
|
||
**Prune**: 7 daily, 4 weekly, 3 monthly
|
||
**Passphrase**: `/home/sam/.borg-passphrase` (same as .27 recommended)
|
||
**Config**: `/etc/nixos/backup.nix`
|
||
**Snapshots**: `borg list /mnt/ubuntu_storage_3TB/backup/borg/.13`
|
||
|
||
> ⚠️ Borg runs as root — `postHook` chowns to `sam:users`. Known hosts issue: `/home/sam/.ssh/known_hosts` was a broken directory (fixed).
|
||
|
||
## .35 — sam-ubuntu1
|
||
|
||
**Tools**: Kopia + Backrest (Restic) — Docker containers
|
||
**Target**: `//192.168.20.23/usb_3tb` → `/mnt/nfs_usb_3tb/`
|
||
- Kopia repo: `/mnt/nfs_usb_3tb/kopia/backups/` (46G)
|
||
- Restic repo: `/mnt/nfs_usb_3tb/backups/restic/` (11G)
|
||
|
||
### Backup Sources
|
||
|
||
| Source | Tool | Status |
|
||
|--------|------|--------|
|
||
| `/Docker/Containers/` | Kopia | ✅ Weekly Sun 22:00 |
|
||
| `/docker-config`, `/docker-volumes` | Backrest (Restic) | ✅ Weekly Sun 23:00 |
|
||
| `/host_fs/root/trilium-data` | Backrest (Restic) | ✅ Weekly Sun 23:00 |
|
||
| **`/host_fs/mnt/hd`** ← _new_ | **Backrest (Restic)** | ✅ **Added 2026-05-28** |
|
||
| **`/host_fs/var/lib/docker/volumes`** | Kopia | ✅ Weekly Sun 22:00 |
|
||
|
||
### Retention
|
||
|
||
- Kopia: 10 latest, 4 weekly, 12 monthly, 3 annual
|
||
- Backrest: 10 latest (prune/check daily)
|
||
|
||
### Data Drive (My Passport)
|
||
|
||
.dev/vda1 — 1.8T NTFS — physical WD My Passport passed through to VM
|
||
**Mount point**: `/mnt/hd` (was incorrectly at `/mnt/my_passport` — fixed 2026-05-28)
|
||
**Contains**: Immich photos (40K), and all Docker service data (audiobookshelf, lidarr, radarr, sonarr, qbittorrent, etc.)
|
||
|
||
### VM-level
|
||
|
||
Proxmox Backup Server (.48) — full VM backup
|
||
|
||
## .23 — file-server
|
||
|
||
- 2.7T USB drive shared via CIFS/SMB to `.35`
|
||
- Backup repos: Kopia (46G) + Restic (13G) + Borg archive repo (new)
|
||
- 2.0T free
|
||
- Also stores: `sam/` (personal docs/music/photos) — to be moved to .13 archive
|
||
|
||
### Borg archive repo (new 2026-05-30)
|
||
|
||
- **Location**: `/mnt/nfs_usb_3tb/borg-repos/archive`
|
||
- **Source**: `.13` `/mnt/ubuntu_storage_3TB/archive/` (648G — photos, old web work, documents)
|
||
- **Encryption**: repokey-blake2
|
||
- **Compression**: auto,lzma
|
||
- **Run from .13**: `borg create sam@192.168.20.35:/mnt/nfs_usb_3tb/borg-repos/archive::archive-$(date +%Y%m%d) /mnt/ubuntu_storage_3TB/archive/`
|
||
|
||
## Restore Commands
|
||
|
||
### .27 restore
|
||
```bash
|
||
export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase)
|
||
borg extract sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27::<archive>
|
||
```
|
||
|
||
### .13 restore
|
||
```bash
|
||
export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase)
|
||
borg extract /mnt/ubuntu_storage_3TB/backup/borg/.13::<archive>
|
||
```
|
||
|
||
## Not Covered
|
||
|
||
- `.27`: Git-tracked NixOS config (already covered by flakes)
|
||
- `.13`: Git-tracked NixOS config, `ollama`/`open-webui` Docker volumes
|
||
- `.35`: **`.23` file-server has no backup** — single point of failure for restic/kopia repos
|
||
- `.23`: Itself has no backup at all
|
||
- **`.13` archive** — now backed up to .23 Borg repo ✅
|
||
|
||
> **UPDATE 2026-09:** The **AWS S3 offsite** layer now covers the highest-value data (see below): photos, `.13` Borg repo, `.27` Borg repo, and `.13` home configs are offsite in `sam-offsite-backup`. This **mitigates the `.23` single-point-of-failure for that data** (a fire loses .23 but S3 survives). Full restore procedures in **[[Restore Runbook]]**.
|
||
|
||
## Offsite S3 (AWS) — added 2026-09
|
||
|
||
| Item | Detail |
|
||
|---|---|
|
||
| **Bucket** | `sam-offsite-backup` (region ap-southeast-2 / Sydney), SSE, versioning, lifecycle S3→Glacier Flexible→Deep Archive |
|
||
| **Tool** | AWS CLI `aws s3 sync` (run from `.13`) — append-only, no delete |
|
||
| **Daily job** | `offsite-sync.service` + timer, **23:00–05:00** (Melbourne), 6h window; runs `.13` repo → `.27` repo → home configs FIRST, then photos/archive |
|
||
| **Verifier + alert** | `offsite-verify.service` 05:15 daily — checks object-count growth, **ntfy push** to phone on no-progress/error; log at `/var/www/offsite/verify.log` |
|
||
| **Monitoring page** | `https://offsite.lab.audasmedia.com.au` (basic-auth `sam`) — live object counts per prefix, 7-night history, service state, disk. Auto-refresh 15 min |
|
||
| **IAM** | user `backup-offsite` (object-only, least privilege on the bucket) |
|
||
| **Cost** | ~$1–5 AUD/mo (photos/archive in Deep Archive; repos in Standard) |
|
||
| **Credentials** | `/home/sam/.aws-credentials.local` (600, on .13) + `aws-credentials.local` (.27) |
|
||
|
||
**Backed up to S3:** photos/by_date, archive/rest (Deep Archive) · borg/.27, borg/.13 (Standard) · home configs (`~/deployment`, voice/audio, `.agents`, `.pi`, etc., Deep Archive).
|
||
|
||
**Restore:** see **[[Restore Runbook]]** (per-machine local + S3 steps; Deep Archive restore = hours).
|
||
|
||
## Garage (local S3) — .13
|
||
|
||
| Item | Detail |
|
||
|---|---|
|
||
| **Role** | Local S3-compatible object store (`family-home-lab` compose); used by portal/media apps |
|
||
| **Version** | **v2.1.0** (upgraded from v1.0.1, 2026-09-05 — v1 wouldn't work with garage-webui; data auto-migrated on start, backup taken first) |
|
||
| **S3 API** | `[::]:3900` (region `garage`) |
|
||
| **S3 Web** | `[::]:3902` (vhost `.web.lab.audasmedia.com.au`, `index.html`) — ⚠️ **requires per-bucket auth; NOT anonymous static hosting** |
|
||
| **Admin API** | `[::]:3903` (v2 endpoints `/v2/...`); token in `garage.toml`/`.env` |
|
||
| **Web UI** | `garage-webui` container → port **3909**, published as `https://garage-ui.lab.audasmedia.com.au` (basic-auth `sam`) |
|
||
| **Config** | `/home/sam/Docker/Containers/family-home-lab/garage/garage.toml` |
|
||
| **Backup** | Bucket data = app data, included in `.13`/`.35` container backups per app; data+meta snapshot at `/mnt/data/family-home-lab/garage-backup-20260905-111241/` |
|
||
|
||
### ⚠️ Garage S3-web gotcha (learned 2026-09-09)
|
||
Garage's **S3 Web** (`:3902`) is **not** a drop-in anonymous static host. It rejects anonymous access (`403 Forbidden: Garage does not support anonymous access yet`) and needs **per-bucket web access + a read key**. For serving site images:
|
||
- **Use the S3 API** (boto3/aws CLI with a read-scoped key) for uploads, and
|
||
- **Serve images via the app / Caddy** (e.g. reverse-proxy to Garage with the read key, or a Caddy `file_server` dir) — do NOT expose `:3902` for anonymous browsing.
|
||
|
||
## Recent Changes
|
||
|
||
- **2026-05-28**: Discovered My Passport (`/dev/vda1`, 1.8T) had all real data but was mounted at `/mnt/my_passport` instead of `/mnt/hd`. Remounted to `/mnt/hd`. Immich photos (40K) and all Docker service data recovered.
|
||
- **2026-05-28**: Added `/host_fs/mnt/hd` to Backrest (Restic) backup plan.
|
||
- **2026-05-30**: Created Borg archive repo on `.23` (`/mnt/nfs_usb_3tb/borg-repos/archive`) to back up `.13` `/mnt/ubuntu_storage_3TB/archive/` (648G). First backup pending.
|
||
- **2026-05-30**: Reformatted `.13` `/mnt/storage/` (932G NTFS → ext4) and `/mnt/data/` (1.8T NTFS → ext4). Both now empty with 870G and 1.7T free respectively.
|