--- created: 2026-08-28 modified: 2026-08-28 type: log client: sam project: web-host-migration status: active priority: 1 tags: - migration - log aliases: - migration-work-log - work-log id: 1848129023-LOG --- # Migration Work Log > Central log of what has been **done** across the migration phases. > Update this as work completes. Each entry: **date · phase · what · evidence**. ## Phase legend - **P0** Pre-flight · **P1** AWS offsite · **P2** Email · **P3** Local infra · **P4** CMS (separate) · **P5** Decommission --- ## 2026-08-28 | Phase | What was done | Notes/Evidence | |---|---|---| | P0 | Verified no AWS CLI / rclone / creds on `.13` (clean slate) | `command -v aws` empty; `rclone` empty on `.13` | | P0 | Verified existing backup stack on `.13` | Borg 1.4.3 in `backup.nix` (`services.borgbackup.jobs.*`, timers 04:00/05:00/06:00, BORG_RSH wrapper for SSH to `.35`) | | P0 | Verified `.13` containers | `wherewoof-admin`(:3031), `wherewoof-minio`(:9010), `wherewoof-db`(:5434), Garage(:3900/3902)+portal(:8500) | | P0 | Confirmed external exposure | static IP port-forward 80/443 → Caddy `.35`; `where-woof.com`→`192.168.20.13:3020` | | P1 | Decision: AWS-only, S3→Glacier DA, ap-southeast-2 (Sydney), IAM least-privilege | Awaiting AWS account signup to proceed | | — | Created per-phase plan notes (P0, P1, P2, P3, P5) + this log | Obsidian `220 Web Host Migration/` | ## 2026-08-28 (Phase 1 — build begun) | Phase | What was done | Evidence | |---|---|---| | P0 | **Machine confirmation** — assistant runs on `.27` (sam-4screen-desktop); `.13` reachable via `sam@192.168.20.13` | hostname `sam-4screen-desktop` = `.27` | | P1 | AWS account ready; IAM user `backup-offsite` created (programmatic, least-priv) | — | | P1 | **Credentials verified** via AWS STS (`get-caller-identity`) | Account `648879824522`, user `backup-offsite`, region `ap-southeast-2` | | P1 | Credentials stored local-only (`aws-credentials.local` 0600, .27) + copied to `.13` (`~/.aws-credentials.local.new` 0600). **Not in Obsidian.** | — | | P1 | Verified `.13` has no aws-cli/rclone; will use one-off `nix-shell awscli2` for imperative steps until `backup.nix` updated | verified | | P1 | STS key check OK (Account `648879824522`, user `backup-offsite`); created env-helper `~/.aws-offsite-env.sh` on `.13` (0600, sources creds from file, no secrets echoed) | verified | | P1 | **MFA**: root MFA enabled in console; Ente Auth as authenticator (phone). Ente recovery keyphrase stored in local `aws-credentials.local` (0600), NOT Obsidian | — | | P1 | **Blocked on** expanded IAM policy attach (bucket-mgmt + object actions scoped to `sam-offsite-backup`) before bucket+lifecycle can be created | awaiting user | | P1 | **Bucket `sam-offsite-backup` created** (ap-southeast-2, ACLs disabled, all public-access blocked, versioning enabled) — created via console as root; IAM `backup-offsite` kept object-only | created | | P1 | **Object pipeline verified** on `.13`: PUT/LIST/GET/DELETE probe to `s3://sam-offsite-backup/test/` all succeeded via `backup-offsite` | probe-write, content readback OK | | P1 | **TODO:** add lifecycle rule (S3→Glacier Flexible@30d→Glacier Deep Archive@90d) — IAM `backup-offsite` lacks lifecycle perm, so rule will be created via console (root) | pending | | P1 | **Lifecycle rule `archive-to-glacier` created** via console (root): S3 Standard → Glacier Flexible @ **90 days** → Glacier Deep Archive @ **180 days** (current AWS min: 90/180) | user-confirmed in console | | P1 | **Pipeline chosen: AWS CLI `s3 sync`** (NOT rclone). rclone fought S3 region-discovery (needs `ListAllMyBuckets`/`GetBucketLocation` which least-priv reserved user lacks); AWS CLI works with object-only perms. | validated: full UP/LIST/READBACK/DELETE sync test passed on `sam-offsite-backup` | | P1 | Cleaned up temp rclone config + test scripts on `.13`; kept `~/.aws-offsite-env.sh` env helper (0600, reads creds from file, no secrets echoed) | — | | P1 | **Bandwidth probe** on `.13`: ~**3.3 MB/s** upstream to AWS → ~62 h for full ~712 GB seed | measured | | P1 | Wrote **[[Backup Architecture — Offsite to AWS S3]]** note (source set, tech stack, procedure, cost, status) | obsidian | | P1 | Refined source set (~712 GB): `photos/by_date`(373G), `photos/phone-only`(.35 Immich, small), `archive/rest`(177G), `borg/.27`(162G). Skips duplicates/transient/regenerable | per Google-photos agent note | | P1 | **Awaiting user go/no-go** on seed scope + storage-class (Standard vs Deep Archive for cold set) | decision needed | | P1 | **Seed strategy locked:** COLD (photos + archive/rest) → `DEEP_ARCHIVE`; ACTIVE (.27 borg) → `STANDARD`. APPEND-ONLY, no `--delete` (nothing is ever removed from S3). New Google-photos additions auto-picked-up on re-run. | user choices | | P1 | **Deep Archive end-to-end probe PASSED** on `.13`: sync with `--storage-class DEEP_ARCHIVE` → `head-object` reported `DEEP_ARCHIVE`; probe cleaned. | verified | | P1 | Seed script drafted (`offsite-seed.sh`): photos→DA, archive/rest→DA, borg/.27→STANDARD | pending | | P1 | **Awaiting final go** to launch ~62 h full seed | user decision | | P1 | **Seed schedule decided (user):** nightly **23:00–05:00 Melbourne local** on `.13` (off-peak, avoids NBN/ABB peak shaping). Resumable/incremental, append-only. | user choice | | P1 | **NixOS module `offsite.nix` added** (separate file; does NOT touch `backup.nix`): adds `awscli2` to systemPackages, `offsite-sync.service` (Type=exec, RuntimeMaxSec=6h) + `offsite-sync.timer` (daily 23:00). "Fixed" initial Type=oneshot bug (RuntimeMaxSec ignored) -> Type=exec. | flake check passed; rebuilt | | P1 | **Rebuilt `.13`**: AWS CLI v2.35.11 on PATH; service Type=exec RuntimeMaxSec=6h; timer active (next fire 23:00 tonight). | verified | | P1 | **First seed scheduled** to auto-start 23:00 tonight. | armed | | P1 | **BUG CAUGHT + FIXED:** nightly runs had failed silently (env-helper used `awk`, missing in minimal systemd PATH → empty region → `Invalid endpoint s3..amazonaws.com`; zero objects uploaded). Fixed `offsite.nix`: added `path = [ awscli2 bash gawk gnused gnugrep coreutils ]` and switched creds parsing to grep/cut (no awk). | journal: `awk: command not found` → fixed | | P1 | **REBUILD FAILURES diagnosed:** earlier `nixos-rebuild` runs had not actually switched (no new generations Jun→Jul; latest failed `Permission denied` on profile symlink = ran without sudo). Rebuilt with `sudo` → generation now `system-145-link`. | verified: readlink /nix/var/nix/profiles/system → system-145-link | | P1 | **REAL RUN TEST PASSED (2026-08-31):** manual start ~45s → service Active, uploading real `.jpg` files; IP out 123.4M; S3 now holds **photos/by_date: 95 objects / 120 MB** (archive/rest + borg/.27 queue behind). Timer still armed for 23:00. | journal + `aws s3 ls` counts | | P1 | Monitoring page requested: `offsite.lab.audasmedia.com.au` behind Caddy basic-auth → .13 status page. | in progress | | P1 | **Monitoring page built:** `gen-status.sh` (on .13, `/etc/nixos/gen-status.sh`) gathers S3 object/byte counts per prefix, storage class, service state, timer, disk → self-contained HTML. systemd `offsite-status.service`+timer (every 15 min) + `offsite-web.service` (python http.server :8091). Script validated (shows 95 objects/0.12GB, next timer 23:00). | working | | P1 | **Caddy block added on .35**: `offsite.lab.audasmedia.com.au` → `import basic-auth` → `reverse_proxy 192.168.20.13:8091`. Validated + reloaded; backup `Caddyfile.bak.20260831-094426`. Wildcard DNS already resolves. | live | | P1 | **Awaiting rebuild on .13** (sudo nixos-rebuild) to activate offsite-status/offsite-web services. | pending | | P1 | **Two bugs found + fixed:** (1) port 8091 already used by `langgraph-service` docker (FastAPI `{"detail":"Not Found"}` on URL) → moved web server to **:8095** (updated offsite.nix + Caddy). (2) generator wrote to `$HOME/offsite-status.html` (OUT env unset) → added `environment.OUT=/var/www/offsite/status.html` to offsite-status.service. | verified | | P1 | **MONITORING PAGE LIVE + VERIFIED:** `https://offsite.lab.audasmedia.com.au` (basic-auth `sam`) → Caddy → .13:8095 → status.html. Local 200, URL returns 401 auth challenge (backend connected, no more 502). Page shows live data: 95 objects/0.12GB, photos 0.0% of 374.98GB, next timer 23:00, service state, disk. | end-to-end verified | | P1 | **502 root cause found + fixed:** `.13` NixOS firewall allowlist (`allowedTCPPorts`) did NOT include **8095** → Caddy (`.35`) connections to `.13:8095` silently dropped → 502 in browsers despite local 200. Added `8095` to `allowedTCPPorts` in `configuration.nix`, rebuilt, verified `.35→.13:8095` = 200 (2ms), URL = 401 gate, 0 timeouts/502s. | verified | | P1 | Idea (NO action): DuckDB + Parquet for photo-catalog analytics (dedup/audit across by_date/by_subject/Immich, S3 Inventory query) — noted as future option if library grows. | noted | | P1 | **60-SECOND TEST OF FIXED SERVICE (2026-08-31 12:26):** before: photos 95/archive 0/borg 0 → after 60s: photos **233** (+138), result=success. Real .jpg uploads confirmed. | before/after counts + journal | | P1 | **NIGHTLY VERIFIER ADDED:** `offsite-verify.service`+timer (05:15 daily) counts S3 objects, compares to stored last-count, sends **ntfy alert to phone** via Apprise (on .35:8210, topic `offsite-backup`) if no progress or aws error; writes `/var/www/offsite/verify.log` + shown on monitoring page. Alert path tested: Apprise→ntfy delivered. | tested OK; timers armed | | P1 | **Safety net now:** sync 23:00-05:00 → verify 05:15 (alert if no progress) → monitoring page every 15 min. Silent-failure mode is closed. | armed | ## Pending next action ## 2026-09-01 — BACKUP GAP CLOSURE (tasks in progress) | Phase | What was done | Status | |---|---|---| | .13 Borg | `backup.nix` `local-to-storage` paths expanded: + `~/deployment`(23G), gst-plugins-rs, speech_piper, voice_bridge, whisper-api, voice-agent, librespot, mopidy-radionet, bin, .agents, .pi, .config; exclusions + `.cache`, `.npm`, `.local`, nixos_backup.tar.gz | ✅ config applied (rebuild done) | | .13 Borg | **prefect live DB excluded** (caused recurring `.failed` runs) | ✅ config in, applies on next rebuild | | .13 Borg | First expanded run in progress (23G initial archive, slow on lzma) | 🔄 running | | S3 seed | `offsite.nix` + sync of `borg/.13` (STANDARD) + home dirs (DEEP_ARCHIVE) | ✅ config in; auto-syncs tonight 23:00 | | Restore runbook | to be written | ⏳ Task 3 | ## Next steps ## PHASE 2 EMAIL — progress log (2026-08-31) | Item | Status | |---|---| | DNS moved to Cloudflare (audasmedia.com.au) | ✅ NS = jo/osmar.ns.cloudflare.com; A/wwww/cpanel/ftp preserved; **wildcard `*.lab` → 144.6.86.11 re-added (was missing after import — caught it, home sites restored)** | | SES identity `audasmedia.com.au` | ✅ created; 3 DKIM CNAMEs + DMARC added to Cloudflare | | Email Routing (free inbound → Gmail) | ✅ enabled; MX route1/2/3.mx.cloudflare.net live; catch-all → samuelrolfe@gmail.com; **test delivery confirmed (arrived in All Mail)** | | SPF | ✅ `v=spf1 ip4:173.231.229.92 include:_spf.mx.cloudflare.net include:amazonses.com ~all` | | Zoho | ❌ abandoned (SMS OTP never arrives) | | MXroute | ⏸ deferred (no funds today; Cloudflare Email Routing covers inbound for $0) | | SES outbound test | ⏳ next: send test email via SES → verify DKIM/SPF pass | | SES production access | ⏳ later (free, ~hrs) so sites can send to anyone | | SES MAIL FROM | ✅ set to default (banner cleared; DMARC still passes via DKIM) | ## PHASE 2 EMAIL — STATUS: DONE ✅ (2026-08-31) - **Inbound:** Cloudflare Email Routing → Gmail. Test confirmed (All Mail). $0. - **Outbound:** SES identity `audasmedia.com.au` verified; test email delivered; DKIM/SPF pass. - **DNS:** Cloudflare authoritative; all email records in (MX route1/2/3, DKIM×3, DMARC, SPF). - **Zoho:** abandoned (SMS OTP never arrives). MXroute deferred (no funds today; Email Routing covers inbound at $0). - **Remaining (optional/deferred):** SES production access (so sites send to arbitrary addresses — needed when CMS migration wires forms/reset emails); where-woof.com email later. ## Next steps (after email) 1. Request SES production access when ready (free, ~hrs approval). 2. Phase 3 (local infra prep) or the CMS migration project. ## Next steps (email) ## SCOPE NOTE — where-woof.com (2026-08-31) - **User is building where-woof in another AI session.** Do NOT touch its DNS/email from this project. - Current state (unchanged): NS = InMotion (ns.inmotionhosting.com / ns2), A → 144.6.86.11 (static IP, local), no email records. - Revisit later: same Cloudflare + Email Routing + SES pattern as audasmedia.com.au, when user is ready. ## Next steps (post-email) ## PHASE 2 EMAIL — where-woof.com DONE ✅ (2026-08-31) - DNS → Cloudflare (zone live on barbara.ns.cloudflare.com; NS switched at Squarespace). - Records: A apex/www/admin → 144.6.86.11 (DNS-only); InMotion A's + stale TXT deleted; DKIM×3 + DMARC added. - Email Routing active (MX route1/2/3 priorities 4/25/14, SPF include:_spf.mx.cloudflare.net); rule `hello@where-woof.com → samuelrolfe@gmail.com`. - SES identity created; MAIL FROM = default. - Awaiting end-to-end send test (from non-Gmail-same-account address to hello@where-woof.com). - Note: audasmedia MX priorities are 54/92/78; where-woof 4/25/14 — Cloudflare assigns per-zone, both valid. ## PHASE 2 EMAIL — where-woof.com → PURELYMAIL (2026-09-02) - **Switched from Cloudflare Email Routing to PurelyMail** (real domain mailboxes, IMAP/SMTP, ~US$10/yr). - **Cloudflare Email Routing DISABLED** on where-woof.com (frees MX — cannot coexist with external provider). - DNS records added in Cloudflare (all verified live): - MX @ → `mailserver.purelymail.com` (prio 10) - TXT @ → `purelymail_ownership_proof=3ac61cfe...` - TXT @ → `v=spf1 include:_spf.purelymail.com include:amazonses.com ~all` - CNAME `purelymail1/2/3._domainkey` → `key1/2/3.dkimroot.purelymail.com` - CNAME `_dmarc` → `dmarcroot.purelymail.com` - CNAME `autoconfig` → `autoconfig.purelymail.com` - SRV `_autodiscover._tcp` → `autodiscover.purelymail.com` (0 0 443) - Deleted old leftovers: SES DKIM×3 (where-woof), `mail`/`smtp` CNAMEs, old `_dmarc` TXT. - PurelyMail domain: validated (Check DNS Records done); Deliver mail to = PurelyMail; mailbox `hello@where-woof.com` created. - **Pending:** PurelyMail mailbox → Thunderbird or Gmail "Send mail as" (user leaning: Gmail inbox + PurelyMail SMTP send-as; Thunderbird optional). ## SITE SENDING HANDOFF — for the AI building where-woof.com (2026-09-02) Hand this to the where-woof website builder: ``` where-woof.com outgoing (transactional) email: - Provider: Amazon SES, region ap-southeast-2; identity where-woof.com verified - Sender: hello@where-woof.com - SMTP: create user in SES → SMTP settings; host email-smtp.ap-southeast-2.amazonaws.com, port 587 STARTTLS - BEFORE sending to real users: request SES production access (SES → Account dashboard → Request production access; free, ~hours) - DNS for delivery already set (SPF/DKIM/DMARC include SES + PurelyMail) - Do NOT use PurelyMail SMTP for site/transactional mail — use SES ``` ## SES SMTP CREDENTIALS (created 2026-09-03, local file `aws-credentials.local`) - Endpoint: `email-smtp.ap-southeast-2.amazonaws.com` : `587` (STARTTLS) - IAM user: `ses-smtp-user.20260903-152339`; SMTP username/password stored in `aws-credentials.local` (600, not Obsidian) - Permission: `ses:SendRawEmail` only (least privilege) - **Production access requested 2026-09-03** (transactional form, where-woof.com) — awaiting AWS approval (~hours) - Until approval: SES sandbox = can only send to verified addresses (samuelrolfe@gmail.com). ## TODO — audasmedia.com.au SES (2026-09-03) - [ ] **Give audasmedia.com.au the same SES sending access as where-woof**: create/confirm SES identity for audasmedia.com.au (already exists), create SMTP creds for it, and ensure production access covers it. - audasmedia.com.au will also send email (site mail) → same pattern as where-woof. - When audasmedia infra is built on .13 (user to advise). ## NEXT — audasmedia.com.au infra on .13 (2026-09-03) - User will advise details. Apply same hosting/backup/email pattern as where-woof. ## VISUALIZATION — status page upgrade + Garage UI handoff (2026-09-05) - **Status page upgraded** (`offsite.lab.audasmedia.com.au`): - Fixed "failed" label → now shows `OK (finished cleanly)` / `OK (6h window ended 05:00)` in green (the systemd "timeout" at 05:00 is the expected RuntimeMaxSec stop, not an error). - Added **7-night history table** (parsed from verify.log: date, result, objects, delta). - Added "Last verify" line. - **Garage UI handoff written** (`Garage Web UI — handoff.md`): use `khairul169/garage-webui` (Garage v1.x compatible; Noooste/garage-ui needs 2.0+). Deploy on :3909, admin API :3902, token from .env. - **Grafana deferral noted**: Grafana on .35 has NO data sources (empty). Defer consolidated dashboard unless needed after 1+2. - Assessment: Backrest/Kopia UIs visualize only their own local repos (.23), not the AWS S3 offsite. Garage UI is separate (local Garage, not AWS). ## GARAGE v2 UPGRADE (2026-09-05) - **Why:** garage-webui / Noooste UIs didn't list buckets on Garage v1.0.1 (v1 admin API lacks their expected endpoints). Storage itself was fine (S3 v1==v2 for portal). - **Backup first:** `/mnt/data/family-home-lab/garage-backup-20260905-111241/` (data 181M + meta 1.1M, via busybox as root since LMDB files are root-owned). - **Upgraded:** `dxflrs/garage:v1.0.1` → `v2.1.0`; garage.toml rewritten (v2 schema: `replication_factor`, `[s3_web]`, admin on **3903**); compose updated (ports 3900/3901/3902/3903). Tokens preserved exactly (admin/metrics/rpc 64-hex). - **Verified:** v2 auto-migrated the LMDB on start; `GetClusterStatus` shows node up v2.1.0; `ListBuckets` returns existing buckets (outline,...) via `POST /v2/...`. - **garage-webui:** API_BASE_URL → `http://garage:3903` (admin moved to 3903), recreated, 0 refused-erores, HTTP 200. **USER CONFIRMED: all buckets now show in the dashboard. ✅** - Caddy `garage-ui.lab.audasmedia.com.au` → 3909 (basic-auth). Removed a stale 3902 block. - **Rollback:** if needed, stop v2, restore garage-backup-*, revert compose/garage.toml, restart v1.0.1. ## TIER-A DB HARDENING (2026-09-05/06) - **Findings:** All ~21 DB containers across .13/.35 are covered by existing backups (Borg on .13 via `~/Docker`; Backrest/Kopia on .35 via `/docker-containers`+volumes) — **no coverage gap**. Confirmed Vikunja = 437 paths in restic; Outline in Borg. The shared weakness: all are **file-copies of live DBs** (inconsistent-restore risk). - **Plan:** dump critical DBs to an already-backed-up path before the backup runs. - **.13 (done, dumps tested):** `db-dump-13.sh` dumps **outline, wherewoof-db, family-home-lab(portal), langfuse, ai-resume, airflow** Postgres via container-env creds (no secrets in config) → `~/Docker/Containers//backups/db-dumps/` (inside Borg `~/Docker` paths, keep-3). Tested: 201K/31K/6K/278K/824/197K dumps written. - **Pending:** wire .13 timer (03:30 pre-Borg); do .35 Tier-A (vikunja + others) via Backrest pre-hook or dump script. ## TIER-A DB HARDENING — IMPLEMENTATION (2026-09-06) - **.13** (`~/.Docker → Borg`): `db-dump-13.sh` + NixOS module `dbdump.nix` → timer **03:30** pre-Borg. Dumps outline, wherewoof-db, family-home-lab(portal), langfuse, ai-resume, airflow PG → `~/Docker/Containers//backups/db-dumps/` (in Borg paths). **Fixed:** service must run `User=root` (/home/sam/Docker is root-owned) — rebuild applied 2026-09-06. - **.35** (`/Docker/Containers → Backrest`): `db-dump-35.sh` + systemd unit/timer → **03:35** daily. Dumps vikunja(mariadb via env pw), gitea(mysqldump, pw `sam4jo`), affine, immich, paperless (pg via env) → `/Docker/Containers//backups/db-dumps/` (in restic path). Timer installed+enabled; verified affine 2MB dump works. - **Notes:** Backrest image lacks pg/mysql/docker clients → used host systemd timer (root) instead of Backrest hooks. firefly_iii compose not found under /Docker/Containers/firefly_iii (Tier-B anyway; skip). Gitea root pw `sam4jo` stored in root-only script. - **Awaiting:** .13 rebuild (User=root) + first automatic fires tonight/tomorrow 03:30/03:35. - **langfuse + airflow removed (2026-09-06):** stopped (not deleted; compose dirs remain). Removed langfuse/airflow lines from `db-dump-13.sh` (would have errored nightly). pi-langfuse extension already absent from .27. Offsite/backup configs unaffected. 1. SES console → Send test email (audasmedia.com.au → samuelrolfe@gmail.com); check delivery + DKIM/SPF pass. 2. Request SES production access (free). 3. Then return to Phase 3 (local infra) / CMS migration project. - **Build:** create bucket `sam-offsite-backup` + lifecycle (S3→Glacier Flexible@30d→Glacier DA@90d), then wire local Borg/rclone→S3 on `.13`, seed, test restore. - **Security:** enable root MFA; consider rotating access key after seeding (it was pasted in chat). --- ## 2026-09-02 — Backup gap closure: config + root-cause fix | Item | What | Status | |---|---|---| | .13 Borg coverage | `backup.nix` local-to-storage + `~/deployment`, voice/audio/dev dirs, `.agents`, `.pi`, `bin`, `.config`; excludes + `.cache`/`.npm`/`.local` | ✅ applied | | prefect DB | Added to Borg exclude (was causing recurring `.failed` runs) | ✅ applied | | Borg nightly | 05:00 runs succeeding (index written Sep 2 05:01); 23G first archive slow (lzma) | ✅ working | | **S3 ordering bug** | **Found: script ran sequentially; photos consumed the entire 6h nightly window → `borg/.13` + home dirs never reached S3 (0 objects)** | 🔧 **fixed** | | S3 ordering fix | Reordered `offsite.nix`: `.13` repo + `.27` repo + home configs sync **FIRST** (minutes), photos/archive last (fill remaining window) | ✅ config in; rebuild pending | | Restore runbook | `Restore Runbook.md` written + linked from Backup Architecture | ✅ | | Note | `borg list` fails for sam (repo data/ root-owned — known artifact; Borg job runs as root, works) | noted | --- *Linked from [[Migration Plan — InMotion to Local + AWS Offsite]].*