sam-4screen-desktop 2026-8-28:19:9:8
This commit is contained in:
@@ -0,0 +1,96 @@
|
||||
---
|
||||
created: 2026-08-28
|
||||
modified: 2026-08-28
|
||||
type: note
|
||||
client: sam
|
||||
project: web-host-migration
|
||||
status: active
|
||||
priority: 1
|
||||
tags:
|
||||
- backup
|
||||
- aws
|
||||
- s3
|
||||
- glacier
|
||||
- obsidian
|
||||
aliases:
|
||||
- backup-architecture
|
||||
- offsite-backup
|
||||
id: 1848129024-BAC
|
||||
---
|
||||
# Backup Architecture — Offsite to AWS S3 (Disaster / "House burns down")
|
||||
|
||||
> Part of [[Migration Plan — InMotion to Local + AWS Offsite]], Phase 1.
|
||||
> Status: **bucket + lifecycle + pipeline built & validated. Seeding pending user go/no-go.**
|
||||
> Source-set refined from the Google-photos migration note (see "Source set" below).
|
||||
|
||||
## Why
|
||||
Protect the "house burns down" set off-machine so a total local loss doesn't destroy photos, site data, or Borg backups. Local copies alone (Borg/rsync/Kopia between .13/.35/.23) are all on-site — a fire takes them all. AWS is the off-site copy.
|
||||
|
||||
## Tech stack (what we actually use)
|
||||
- **AWS S3 bucket** `sam-offsite-backup` — region **ap-southeast-2 (Sydney)**
|
||||
- Block **all public access**, **Versioning Enabled**, **SSE-S3** server-side encryption at rest
|
||||
- **Lifecycle rule** `archive-to-glacier`: S3 Standard (first 90 days, fast restore) → **Glacier Flexible** @ 90d → **Glacier Deep Archive** @ 180d (cold/cheap)
|
||||
- **IAM user** `backup-offsite` — programmatic, **least-privilege**, only object-level S3 on this one bucket (ListBucket/PutObject/GetObject/DeleteObject). No bucket-management, no console, no other AWS services. Root keeps full power.
|
||||
- **Tool:** **AWS CLI** `aws s3 sync` (chosen over rclone — rclone fought S3 region discovery with the scoped IAM; AWS CLI works with object-only perms). Validated: full upload/list/readback/delete test succeeded.
|
||||
- **Local driver:** pending — will be a NixOS systemd timer (won't be in `backup.nix` to avoid disrupting existing Borg; separate service) running from `.13`, reading creds from `/home/sam/.aws-credentials.local` (0600).
|
||||
|
||||
## Source set (what we back up)
|
||||
From **.13** (unless noted):
|
||||
|
||||
| Source | Path | Size |
|
||||
|---|---|---|
|
||||
| Photo master (by_date) | `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` | 373 GB |
|
||||
| Immich phone-only | `.35:/mnt/hd/immich/library` (via SSH to .35) | small |
|
||||
| Projects / notes / docs | `/mnt/ubuntu_storage_3TB/archive/` (`01_family…08_transfer`, minus `03_photos`) | ~177 GB |
|
||||
| `.27` Borg repo | `/mnt/ubuntu_storage_3TB/backup/borg/.27` | 162 GB |
|
||||
| **Total** | | **~712 GB** |
|
||||
|
||||
### Explicitly SKIPPED (duplicates/transient/regenerable)
|
||||
- `by_subject` (69 GB) — duplicate of `by_date` (different folder order)
|
||||
- `/mnt/data/takeout/` (269 GB) — transient Google-photos download, deleted after verification
|
||||
- Immich thumbs / encoded-video — regenerable
|
||||
- 14 zip archives — redundant
|
||||
- Immich library copies of `by_date` photos — duplicates
|
||||
|
||||
## Proposed bucket layout (prefixes)
|
||||
```
|
||||
s3://sam-offsite-backup/
|
||||
├── photos/by_date/ (373G)
|
||||
├── photos/phone-only/ (small, from .35 Immich)
|
||||
├── archive/rest/ (177G)
|
||||
└── borg/.27/ (162G)
|
||||
```
|
||||
|
||||
## Procedure to run a seed / sync
|
||||
On `.13` (after tooling installed):
|
||||
```bash
|
||||
source /home/sam/.aws-offsite-env.sh # sets AWS_* from creds file
|
||||
AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
|
||||
aws s3 sync /mnt/ubuntu_storage_3TB/archive/03_photos/by_date \
|
||||
s3://sam-offsite-backup/photos/by_date/ --storage-class STANDARD
|
||||
```
|
||||
- `s3 sync` is **incremental** — after the initial seed, re-runs upload only new/changed files.
|
||||
- Bandwidth: **~3.3 MB/s** measured → ~62 h for full 712 GB seed.
|
||||
|
||||
## Cost (AUD, honest — revised 2026-08-28)
|
||||
Seeding everything in **S3 Standard** (fast restore) means the first **90 days** cost the most:
|
||||
- **Months 0–3 (all S3 Standard):** ~712 GB × $0.036 ≈ **~$26/mo**
|
||||
- **Months 3–6 (Glacier Flexible):** ~$0.005/GB ≈ **~$3.6/mo**
|
||||
- **After month 6 (Deep Archive):** ~$0.0015/GB ≈ **~$1.1/mo**
|
||||
- One-time upload bytes: **free** (AWS ingress is free) ✓
|
||||
|
||||
### Cheaper option: send cold data straight to Glacier Deep Archive
|
||||
If photos + archive are genuinely cold (only needed in disaster), upload them with `--storage-class DEEP_ARCHIVE` directly → **no 90-day Standard wait**, ~**$1/mo immediately** for the cold set. Only *recent/active* data (Borg .27 repo + phone-only) stays in Standard. Downside: Deep Archive restore takes **hours** (acceptable for disaster recovery). This could reduce peak to **~$5–8/mo** total.
|
||||
|
||||
> The two choices are a small cost-vs-restore-speed tradeoff; **both are well under the 20–50 AUD/mo budget** and ~75%+ cheaper than the retired $100 VPS.
|
||||
|
||||
## Pending / Open
|
||||
- [ ] User confirms go/no-go on seed scope + storage class (Standard vs Deep Archive for cold set)
|
||||
- [ ] Install AWS CLI declaratively on `.13` + systemd timer (managed NixOS service) — but **not** in existing `backup.nix` (protect current Borg jobs); separate config
|
||||
- [ ] Begin first seed (run overnight / across days given ~62 h)
|
||||
- [ ] **Verify a test restore** from S3 after seed (critical)
|
||||
- [ ] Decide optional "delete noncurrent versions" lifecycle to cap version bloat
|
||||
- [ ] Security: rotate access key after seeding (it was shared in chat); root MFA now enabled
|
||||
|
||||
---
|
||||
*See [[Migration Work Log]] for live track record; [[Migration Plan — InMotion to Local + AWS Offsite]] §9 for the master cost model.*
|
||||
@@ -51,6 +51,10 @@ id: 1848129023-LOG
|
||||
| P1 | **Lifecycle rule `archive-to-glacier` created** via console (root): S3 Standard → Glacier Flexible @ **90 days** → Glacier Deep Archive @ **180 days** (current AWS min: 90/180) | user-confirmed in console |
|
||||
| P1 | **Pipeline chosen: AWS CLI `s3 sync`** (NOT rclone). rclone fought S3 region-discovery (needs `ListAllMyBuckets`/`GetBucketLocation` which least-priv reserved user lacks); AWS CLI works with object-only perms. | validated: full UP/LIST/READBACK/DELETE sync test passed on `sam-offsite-backup` |
|
||||
| P1 | Cleaned up temp rclone config + test scripts on `.13`; kept `~/.aws-offsite-env.sh` env helper (0600, reads creds from file, no secrets echoed) | — |
|
||||
| P1 | **Bandwidth probe** on `.13`: ~**3.3 MB/s** upstream to AWS → ~62 h for full ~712 GB seed | measured |
|
||||
| P1 | Wrote **[[Backup Architecture — Offsite to AWS S3]]** note (source set, tech stack, procedure, cost, status) | obsidian |
|
||||
| P1 | Refined source set (~712 GB): `photos/by_date`(373G), `photos/phone-only`(.35 Immich, small), `archive/rest`(177G), `borg/.27`(162G). Skips duplicates/transient/regenerable | per Google-photos agent note |
|
||||
| P1 | **Awaiting user go/no-go** on seed scope + storage-class (Standard vs Deep Archive for cold set) | decision needed |
|
||||
|
||||
## Pending next action
|
||||
- **Build:** create bucket `sam-offsite-backup` + lifecycle (S3→Glacier Flexible@30d→Glacier DA@90d), then wire local Borg/rclone→S3 on `.13`, seed, test restore.
|
||||
|
||||
Reference in New Issue
Block a user