sam-4screen-desktop 2026-9-5:9:39:8

300 areas/360 Dev-Ops Network Computers/Backup System — Borg, Kopia & Restic.md

@@ -119,6 +119,25 @@ borg extract /mnt/ubuntu_storage_3TB/backup/borg/.13::<archive>
 - `.23`: Itself has no backup at all
 - **`.13` archive** — now backed up to .23 Borg repo ✅
 
+> **UPDATE 2026-09:** The **AWS S3 offsite** layer now covers the highest-value data (see below): photos, `.13` Borg repo, `.27` Borg repo, and `.13` home configs are offsite in `sam-offsite-backup`. This **mitigates the `.23` single-point-of-failure for that data** (a fire loses .23 but S3 survives). Full restore procedures in **[[Restore Runbook]]**.
+
+## Offsite S3 (AWS) — added 2026-09
+
+| Item | Detail |
+|---|---|
+| **Bucket** | `sam-offsite-backup` (region ap-southeast-2 / Sydney), SSE, versioning, lifecycle S3→Glacier Flexible→Deep Archive |
+| **Tool** | AWS CLI `aws s3 sync` (run from `.13`) — append-only, no delete |
+| **Daily job** | `offsite-sync.service` + timer, **23:00–05:00** (Melbourne), 6h window; runs `.13` repo → `.27` repo → home configs FIRST, then photos/archive |
+| **Verifier + alert** | `offsite-verify.service` 05:15 daily — checks object-count growth, **ntfy push** to phone on no-progress/error; log at `/var/www/offsite/verify.log` |
+| **Monitoring page** | `https://offsite.lab.audasmedia.com.au` (basic-auth `sam`) — live object counts per prefix, 7-night history, service state, disk. Auto-refresh 15 min |
+| **IAM** | user `backup-offsite` (object-only, least privilege on the bucket) |
+| **Cost** | ~$1–5 AUD/mo (photos/archive in Deep Archive; repos in Standard) |
+| **Credentials** | `/home/sam/.aws-credentials.local` (600, on .13) + `aws-credentials.local` (.27) |
+
+**Backed up to S3:** photos/by_date, archive/rest (Deep Archive) · borg/.27, borg/.13 (Standard) · home configs (`~/deployment`, voice/audio, `.agents`, `.pi`, etc., Deep Archive).
+
+**Restore:** see **[[Restore Runbook]]** (per-machine local + S3 steps; Deep Archive restore = hours).
+
 ## Recent Changes
 
 - **2026-05-28**: Discovered My Passport (`/dev/vda1`, 1.8T) had all real data but was mounted at `/mnt/my_passport` instead of `/mnt/hd`. Remounted to `/mnt/hd`. Immich photos (40K) and all Docker service data recovered.