| P0 |
Machine confirmation — assistant runs on .27 (sam-4screen-desktop); .13 reachable via sam@192.168.20.13 |
hostname sam-4screen-desktop = .27 |
| P1 |
AWS account ready; IAM user backup-offsite created (programmatic, least-priv) |
— |
| P1 |
Credentials verified via AWS STS (get-caller-identity) |
Account 648879824522, user backup-offsite, region ap-southeast-2 |
| P1 |
Credentials stored local-only (aws-credentials.local 0600, .27) + copied to .13 (~/.aws-credentials.local.new 0600). Not in Obsidian. |
— |
| P1 |
Verified .13 has no aws-cli/rclone; will use one-off nix-shell awscli2 for imperative steps until backup.nix updated |
verified |
| P1 |
STS key check OK (Account 648879824522, user backup-offsite); created env-helper ~/.aws-offsite-env.sh on .13 (0600, sources creds from file, no secrets echoed) |
verified |
| P1 |
MFA: root MFA enabled in console; Ente Auth as authenticator (phone). Ente recovery keyphrase stored in local aws-credentials.local (0600), NOT Obsidian |
— |
| P1 |
Blocked on expanded IAM policy attach (bucket-mgmt + object actions scoped to sam-offsite-backup) before bucket+lifecycle can be created |
awaiting user |
| P1 |
Bucket sam-offsite-backup created (ap-southeast-2, ACLs disabled, all public-access blocked, versioning enabled) — created via console as root; IAM backup-offsite kept object-only |
created |
| P1 |
Object pipeline verified on .13: PUT/LIST/GET/DELETE probe to s3://sam-offsite-backup/test/ all succeeded via backup-offsite |
probe-write, content readback OK |
| P1 |
TODO: add lifecycle rule (S3→Glacier Flexible@30d→Glacier Deep Archive@90d) — IAM backup-offsite lacks lifecycle perm, so rule will be created via console (root) |
pending |
| P1 |
Lifecycle rule archive-to-glacier created via console (root): S3 Standard → Glacier Flexible @ 90 days → Glacier Deep Archive @ 180 days (current AWS min: 90/180) |
user-confirmed in console |
| P1 |
Pipeline chosen: AWS CLI s3 sync (NOT rclone). rclone fought S3 region-discovery (needs ListAllMyBuckets/GetBucketLocation which least-priv reserved user lacks); AWS CLI works with object-only perms. |
validated: full UP/LIST/READBACK/DELETE sync test passed on sam-offsite-backup |
| P1 |
Cleaned up temp rclone config + test scripts on .13; kept ~/.aws-offsite-env.sh env helper (0600, reads creds from file, no secrets echoed) |
— |
| P1 |
Bandwidth probe on .13: ~3.3 MB/s upstream to AWS → ~62 h for full ~712 GB seed |
measured |
| P1 |
Wrote Backup Architecture — Offsite to AWS S3 note (source set, tech stack, procedure, cost, status) |
obsidian |
| P1 |
Refined source set (~712 GB): photos/by_date(373G), photos/phone-only(.35 Immich, small), archive/rest(177G), borg/.27(162G). Skips duplicates/transient/regenerable |
per Google-photos agent note |
| P1 |
Awaiting user go/no-go on seed scope + storage-class (Standard vs Deep Archive for cold set) |
decision needed |
| P1 |
Seed strategy locked: COLD (photos + archive/rest) → DEEP_ARCHIVE; ACTIVE (.27 borg) → STANDARD. APPEND-ONLY, no --delete (nothing is ever removed from S3). New Google-photos additions auto-picked-up on re-run. |
user choices |
| P1 |
Deep Archive end-to-end probe PASSED on .13: sync with --storage-class DEEP_ARCHIVE → head-object reported DEEP_ARCHIVE; probe cleaned. |
verified |
| P1 |
Seed script drafted (offsite-seed.sh): photos→DA, archive/rest→DA, borg/.27→STANDARD |
pending |
| P1 |
Awaiting final go to launch ~62 h full seed |
user decision |
| P1 |
Seed schedule decided (user): nightly 23:00–05:00 Melbourne local on .13 (off-peak, avoids NBN/ABB peak shaping). Resumable/incremental, append-only. |
user choice |
| P1 |
NixOS module offsite.nix added (separate file; does NOT touch backup.nix): adds awscli2 to systemPackages, offsite-sync.service (Type=exec, RuntimeMaxSec=6h) + offsite-sync.timer (daily 23:00). "Fixed" initial Type=oneshot bug (RuntimeMaxSec ignored) -> Type=exec. |
flake check passed; rebuilt |
| P1 |
Rebuilt .13: AWS CLI v2.35.11 on PATH; service Type=exec RuntimeMaxSec=6h; timer active (next fire 23:00 tonight). |
verified |
| P1 |
First seed scheduled to auto-start 23:00 tonight. |
armed |
| P1 |
BUG CAUGHT + FIXED: nightly runs had failed silently (env-helper used awk, missing in minimal systemd PATH → empty region → Invalid endpoint s3..amazonaws.com; zero objects uploaded). Fixed offsite.nix: added path = [ awscli2 bash gawk gnused gnugrep coreutils ] and switched creds parsing to grep/cut (no awk). |
journal: awk: command not found → fixed |
| P1 |
REBUILD FAILURES diagnosed: earlier nixos-rebuild runs had not actually switched (no new generations Jun→Jul; latest failed Permission denied on profile symlink = ran without sudo). Rebuilt with sudo → generation now system-145-link. |
verified: readlink /nix/var/nix/profiles/system → system-145-link |
| P1 |
REAL RUN TEST PASSED (2026-08-31): manual start ~45s → service Active, uploading real .jpg files; IP out 123.4M; S3 now holds photos/by_date: 95 objects / 120 MB (archive/rest + borg/.27 queue behind). Timer still armed for 23:00. |
journal + aws s3 ls counts |
| P1 |
Monitoring page requested: offsite.lab.audasmedia.com.au behind Caddy basic-auth → .13 status page. |
in progress |
| P1 |
Monitoring page built: gen-status.sh (on .13, /etc/nixos/gen-status.sh) gathers S3 object/byte counts per prefix, storage class, service state, timer, disk → self-contained HTML. systemd offsite-status.service+timer (every 15 min) + offsite-web.service (python http.server :8091). Script validated (shows 95 objects/0.12GB, next timer 23:00). |
working |
| P1 |
Caddy block added on .35: offsite.lab.audasmedia.com.au → import basic-auth → reverse_proxy 192.168.20.13:8091. Validated + reloaded; backup Caddyfile.bak.20260831-094426. Wildcard DNS already resolves. |
live |
| P1 |
Awaiting rebuild on .13 (sudo nixos-rebuild) to activate offsite-status/offsite-web services. |
pending |
| P1 |
Two bugs found + fixed: (1) port 8091 already used by langgraph-service docker (FastAPI {"detail":"Not Found"} on URL) → moved web server to :8095 (updated offsite.nix + Caddy). (2) generator wrote to $HOME/offsite-status.html (OUT env unset) → added environment.OUT=/var/www/offsite/status.html to offsite-status.service. |
verified |
| P1 |
MONITORING PAGE LIVE + VERIFIED: https://offsite.lab.audasmedia.com.au (basic-auth sam) → Caddy → .13:8095 → status.html. Local 200, URL returns 401 auth challenge (backend connected, no more 502). Page shows live data: 95 objects/0.12GB, photos 0.0% of 374.98GB, next timer 23:00, service state, disk. |
end-to-end verified |
| P1 |
502 root cause found + fixed: .13 NixOS firewall allowlist (allowedTCPPorts) did NOT include 8095 → Caddy (.35) connections to .13:8095 silently dropped → 502 in browsers despite local 200. Added 8095 to allowedTCPPorts in configuration.nix, rebuilt, verified .35→.13:8095 = 200 (2ms), URL = 401 gate, 0 timeouts/502s. |
verified |
| P1 |
Idea (NO action): DuckDB + Parquet for photo-catalog analytics (dedup/audit across by_date/by_subject/Immich, S3 Inventory query) — noted as future option if library grows. |
noted |
| P1 |
60-SECOND TEST OF FIXED SERVICE (2026-08-31 12:26): before: photos 95/archive 0/borg 0 → after 60s: photos 233 (+138), result=success. Real .jpg uploads confirmed. |
before/after counts + journal |
| P1 |
NIGHTLY VERIFIER ADDED: offsite-verify.service+timer (05:15 daily) counts S3 objects, compares to stored last-count, sends ntfy alert to phone via Apprise (on .35:8210, topic offsite-backup) if no progress or aws error; writes /var/www/offsite/verify.log + shown on monitoring page. Alert path tested: Apprise→ntfy delivered. |
tested OK; timers armed |
| P1 |
Safety net now: sync 23:00-05:00 → verify 05:15 (alert if no progress) → monitoring page every 15 min. Silent-failure mode is closed. |
armed |
