229 lines
22 KiB
Markdown
229 lines
22 KiB
Markdown
---
|
||
created: 2026-08-28
|
||
modified: 2026-08-28
|
||
type: project
|
||
client: sam
|
||
project: web-host-migration
|
||
status: active
|
||
priority: 1
|
||
tags:
|
||
- migration
|
||
- log
|
||
aliases:
|
||
- migration-work-log
|
||
- work-log
|
||
id: 1848129023-LOG
|
||
---
|
||
# Migration Work Log
|
||
|
||
> Central log of what has been **done** across the migration phases.
|
||
> Update this as work completes. Each entry: **date · phase · what · evidence**.
|
||
|
||
## Phase legend
|
||
- **P0** Pre-flight · **P1** AWS offsite · **P2** Email · **P3** Local infra · **P4** CMS (separate) · **P5** Decommission
|
||
|
||
---
|
||
|
||
## 2026-08-28
|
||
| Phase | What was done | Notes/Evidence |
|
||
|---|---|---|
|
||
| P0 | Verified no AWS CLI / rclone / creds on `.13` (clean slate) | `command -v aws` empty; `rclone` empty on `.13` |
|
||
| P0 | Verified existing backup stack on `.13` | Borg 1.4.3 in `backup.nix` (`services.borgbackup.jobs.*`, timers 04:00/05:00/06:00, BORG_RSH wrapper for SSH to `.35`) |
|
||
| P0 | Verified `.13` containers | `wherewoof-admin`(:3031), `wherewoof-minio`(:9010), `wherewoof-db`(:5434), Garage(:3900/3902)+portal(:8500) |
|
||
| P0 | Confirmed external exposure | static IP port-forward 80/443 → Caddy `.35`; `where-woof.com`→`192.168.20.13:3020` |
|
||
| P1 | Decision: AWS-only, S3→Glacier DA, ap-southeast-2 (Sydney), IAM least-privilege | Awaiting AWS account signup to proceed |
|
||
| — | Created per-phase plan notes (P0, P1, P2, P3, P5) + this log | Obsidian `220 Web Host Migration/` |
|
||
|
||
## 2026-08-28 (Phase 1 — build begun)
|
||
| Phase | What was done | Evidence |
|
||
|---|---|---|
|
||
| P0 | **Machine confirmation** — assistant runs on `.27` (sam-4screen-desktop); `.13` reachable via `sam@192.168.20.13` | hostname `sam-4screen-desktop` = `.27` |
|
||
| P1 | AWS account ready; IAM user `backup-offsite` created (programmatic, least-priv) | — |
|
||
| P1 | **Credentials verified** via AWS STS (`get-caller-identity`) | Account `648879824522`, user `backup-offsite`, region `ap-southeast-2` |
|
||
| P1 | Credentials stored local-only (`aws-credentials.local` 0600, .27) + copied to `.13` (`~/.aws-credentials.local.new` 0600). **Not in Obsidian.** | — |
|
||
| P1 | Verified `.13` has no aws-cli/rclone; will use one-off `nix-shell awscli2` for imperative steps until `backup.nix` updated | verified |
|
||
| P1 | STS key check OK (Account `648879824522`, user `backup-offsite`); created env-helper `~/.aws-offsite-env.sh` on `.13` (0600, sources creds from file, no secrets echoed) | verified |
|
||
| P1 | **MFA**: root MFA enabled in console; Ente Auth as authenticator (phone). Ente recovery keyphrase stored in local `aws-credentials.local` (0600), NOT Obsidian | — |
|
||
| P1 | **Blocked on** expanded IAM policy attach (bucket-mgmt + object actions scoped to `sam-offsite-backup`) before bucket+lifecycle can be created | awaiting user |
|
||
| P1 | **Bucket `sam-offsite-backup` created** (ap-southeast-2, ACLs disabled, all public-access blocked, versioning enabled) — created via console as root; IAM `backup-offsite` kept object-only | created |
|
||
| P1 | **Object pipeline verified** on `.13`: PUT/LIST/GET/DELETE probe to `s3://sam-offsite-backup/test/` all succeeded via `backup-offsite` | probe-write, content readback OK |
|
||
| P1 | **TODO:** add lifecycle rule (S3→Glacier Flexible@30d→Glacier Deep Archive@90d) — IAM `backup-offsite` lacks lifecycle perm, so rule will be created via console (root) | pending |
|
||
| P1 | **Lifecycle rule `archive-to-glacier` created** via console (root): S3 Standard → Glacier Flexible @ **90 days** → Glacier Deep Archive @ **180 days** (current AWS min: 90/180) | user-confirmed in console |
|
||
| P1 | **Pipeline chosen: AWS CLI `s3 sync`** (NOT rclone). rclone fought S3 region-discovery (needs `ListAllMyBuckets`/`GetBucketLocation` which least-priv reserved user lacks); AWS CLI works with object-only perms. | validated: full UP/LIST/READBACK/DELETE sync test passed on `sam-offsite-backup` |
|
||
| P1 | Cleaned up temp rclone config + test scripts on `.13`; kept `~/.aws-offsite-env.sh` env helper (0600, reads creds from file, no secrets echoed) | — |
|
||
| P1 | **Bandwidth probe** on `.13`: ~**3.3 MB/s** upstream to AWS → ~62 h for full ~712 GB seed | measured |
|
||
| P1 | Wrote **[[Backup Architecture — Offsite to AWS S3]]** note (source set, tech stack, procedure, cost, status) | obsidian |
|
||
| P1 | Refined source set (~712 GB): `photos/by_date`(373G), `photos/phone-only`(.35 Immich, small), `archive/rest`(177G), `borg/.27`(162G). Skips duplicates/transient/regenerable | per Google-photos agent note |
|
||
| P1 | **Awaiting user go/no-go** on seed scope + storage-class (Standard vs Deep Archive for cold set) | decision needed |
|
||
| P1 | **Seed strategy locked:** COLD (photos + archive/rest) → `DEEP_ARCHIVE`; ACTIVE (.27 borg) → `STANDARD`. APPEND-ONLY, no `--delete` (nothing is ever removed from S3). New Google-photos additions auto-picked-up on re-run. | user choices |
|
||
| P1 | **Deep Archive end-to-end probe PASSED** on `.13`: sync with `--storage-class DEEP_ARCHIVE` → `head-object` reported `DEEP_ARCHIVE`; probe cleaned. | verified |
|
||
| P1 | Seed script drafted (`offsite-seed.sh`): photos→DA, archive/rest→DA, borg/.27→STANDARD | pending |
|
||
| P1 | **Awaiting final go** to launch ~62 h full seed | user decision |
|
||
| P1 | **Seed schedule decided (user):** nightly **23:00–05:00 Melbourne local** on `.13` (off-peak, avoids NBN/ABB peak shaping). Resumable/incremental, append-only. | user choice |
|
||
| P1 | **NixOS module `offsite.nix` added** (separate file; does NOT touch `backup.nix`): adds `awscli2` to systemPackages, `offsite-sync.service` (Type=exec, RuntimeMaxSec=6h) + `offsite-sync.timer` (daily 23:00). "Fixed" initial Type=oneshot bug (RuntimeMaxSec ignored) -> Type=exec. | flake check passed; rebuilt |
|
||
| P1 | **Rebuilt `.13`**: AWS CLI v2.35.11 on PATH; service Type=exec RuntimeMaxSec=6h; timer active (next fire 23:00 tonight). | verified |
|
||
| P1 | **First seed scheduled** to auto-start 23:00 tonight. | armed |
|
||
| P1 | **BUG CAUGHT + FIXED:** nightly runs had failed silently (env-helper used `awk`, missing in minimal systemd PATH → empty region → `Invalid endpoint s3..amazonaws.com`; zero objects uploaded). Fixed `offsite.nix`: added `path = [ awscli2 bash gawk gnused gnugrep coreutils ]` and switched creds parsing to grep/cut (no awk). | journal: `awk: command not found` → fixed |
|
||
| P1 | **REBUILD FAILURES diagnosed:** earlier `nixos-rebuild` runs had not actually switched (no new generations Jun→Jul; latest failed `Permission denied` on profile symlink = ran without sudo). Rebuilt with `sudo` → generation now `system-145-link`. | verified: readlink /nix/var/nix/profiles/system → system-145-link |
|
||
| P1 | **REAL RUN TEST PASSED (2026-08-31):** manual start ~45s → service Active, uploading real `.jpg` files; IP out 123.4M; S3 now holds **photos/by_date: 95 objects / 120 MB** (archive/rest + borg/.27 queue behind). Timer still armed for 23:00. | journal + `aws s3 ls` counts |
|
||
| P1 | Monitoring page requested: `offsite.lab.audasmedia.com.au` behind Caddy basic-auth → .13 status page. | in progress |
|
||
| P1 | **Monitoring page built:** `gen-status.sh` (on .13, `/etc/nixos/gen-status.sh`) gathers S3 object/byte counts per prefix, storage class, service state, timer, disk → self-contained HTML. systemd `offsite-status.service`+timer (every 15 min) + `offsite-web.service` (python http.server :8091). Script validated (shows 95 objects/0.12GB, next timer 23:00). | working |
|
||
| P1 | **Caddy block added on .35**: `offsite.lab.audasmedia.com.au` → `import basic-auth` → `reverse_proxy 192.168.20.13:8091`. Validated + reloaded; backup `Caddyfile.bak.20260831-094426`. Wildcard DNS already resolves. | live |
|
||
| P1 | **Awaiting rebuild on .13** (sudo nixos-rebuild) to activate offsite-status/offsite-web services. | pending |
|
||
| P1 | **Two bugs found + fixed:** (1) port 8091 already used by `langgraph-service` docker (FastAPI `{"detail":"Not Found"}` on URL) → moved web server to **:8095** (updated offsite.nix + Caddy). (2) generator wrote to `$HOME/offsite-status.html` (OUT env unset) → added `environment.OUT=/var/www/offsite/status.html` to offsite-status.service. | verified |
|
||
| P1 | **MONITORING PAGE LIVE + VERIFIED:** `https://offsite.lab.audasmedia.com.au` (basic-auth `sam`) → Caddy → .13:8095 → status.html. Local 200, URL returns 401 auth challenge (backend connected, no more 502). Page shows live data: 95 objects/0.12GB, photos 0.0% of 374.98GB, next timer 23:00, service state, disk. | end-to-end verified |
|
||
| P1 | **502 root cause found + fixed:** `.13` NixOS firewall allowlist (`allowedTCPPorts`) did NOT include **8095** → Caddy (`.35`) connections to `.13:8095` silently dropped → 502 in browsers despite local 200. Added `8095` to `allowedTCPPorts` in `configuration.nix`, rebuilt, verified `.35→.13:8095` = 200 (2ms), URL = 401 gate, 0 timeouts/502s. | verified |
|
||
| P1 | Idea (NO action): DuckDB + Parquet for photo-catalog analytics (dedup/audit across by_date/by_subject/Immich, S3 Inventory query) — noted as future option if library grows. | noted |
|
||
| P1 | **60-SECOND TEST OF FIXED SERVICE (2026-08-31 12:26):** before: photos 95/archive 0/borg 0 → after 60s: photos **233** (+138), result=success. Real .jpg uploads confirmed. | before/after counts + journal |
|
||
| P1 | **NIGHTLY VERIFIER ADDED:** `offsite-verify.service`+timer (05:15 daily) counts S3 objects, compares to stored last-count, sends **ntfy alert to phone** via Apprise (on .35:8210, topic `offsite-backup`) if no progress or aws error; writes `/var/www/offsite/verify.log` + shown on monitoring page. Alert path tested: Apprise→ntfy delivered. | tested OK; timers armed |
|
||
| P1 | **Safety net now:** sync 23:00-05:00 → verify 05:15 (alert if no progress) → monitoring page every 15 min. Silent-failure mode is closed. | armed |
|
||
|
||
## Pending next action
|
||
|
||
## 2026-09-01 — BACKUP GAP CLOSURE (tasks in progress)
|
||
| Phase | What was done | Status |
|
||
|---|---|---|
|
||
| .13 Borg | `backup.nix` `local-to-storage` paths expanded: + `~/deployment`(23G), gst-plugins-rs, speech_piper, voice_bridge, whisper-api, voice-agent, librespot, mopidy-radionet, bin, .agents, .pi, .config; exclusions + `.cache`, `.npm`, `.local`, nixos_backup.tar.gz | ✅ config applied (rebuild done) |
|
||
| .13 Borg | **prefect live DB excluded** (caused recurring `.failed` runs) | ✅ config in, applies on next rebuild |
|
||
| .13 Borg | First expanded run in progress (23G initial archive, slow on lzma) | 🔄 running |
|
||
| S3 seed | `offsite.nix` + sync of `borg/.13` (STANDARD) + home dirs (DEEP_ARCHIVE) | ✅ config in; auto-syncs tonight 23:00 |
|
||
| Restore runbook | to be written | ⏳ Task 3 |
|
||
|
||
## Next steps
|
||
|
||
## PHASE 2 EMAIL — progress log (2026-08-31)
|
||
| Item | Status |
|
||
|---|---|
|
||
| DNS moved to Cloudflare (audasmedia.com.au) | ✅ NS = jo/osmar.ns.cloudflare.com; A/wwww/cpanel/ftp preserved; **wildcard `*.lab` → 144.6.86.11 re-added (was missing after import — caught it, home sites restored)** |
|
||
| SES identity `audasmedia.com.au` | ✅ created; 3 DKIM CNAMEs + DMARC added to Cloudflare |
|
||
| Email Routing (free inbound → Gmail) | ✅ enabled; MX route1/2/3.mx.cloudflare.net live; catch-all → samuelrolfe@gmail.com; **test delivery confirmed (arrived in All Mail)** |
|
||
| SPF | ✅ `v=spf1 ip4:173.231.229.92 include:_spf.mx.cloudflare.net include:amazonses.com ~all` |
|
||
| Zoho | ❌ abandoned (SMS OTP never arrives) |
|
||
| MXroute | ⏸ deferred (no funds today; Cloudflare Email Routing covers inbound for $0) |
|
||
| SES outbound test | ⏳ next: send test email via SES → verify DKIM/SPF pass |
|
||
| SES production access | ⏳ later (free, ~hrs) so sites can send to anyone |
|
||
| SES MAIL FROM | ✅ set to default (banner cleared; DMARC still passes via DKIM) |
|
||
|
||
## PHASE 2 EMAIL — STATUS: DONE ✅ (2026-08-31)
|
||
- **Inbound:** Cloudflare Email Routing → Gmail. Test confirmed (All Mail). $0.
|
||
- **Outbound:** SES identity `audasmedia.com.au` verified; test email delivered; DKIM/SPF pass.
|
||
- **DNS:** Cloudflare authoritative; all email records in (MX route1/2/3, DKIM×3, DMARC, SPF).
|
||
- **Zoho:** abandoned (SMS OTP never arrives). MXroute deferred (no funds today; Email Routing covers inbound at $0).
|
||
- **Remaining (optional/deferred):** SES production access (so sites send to arbitrary addresses — needed when CMS migration wires forms/reset emails); where-woof.com email later.
|
||
|
||
## Next steps (after email)
|
||
1. Request SES production access when ready (free, ~hrs approval).
|
||
2. Phase 3 (local infra prep) or the CMS migration project.
|
||
|
||
## Next steps (email)
|
||
|
||
## SCOPE NOTE — where-woof.com (2026-08-31)
|
||
- **User is building where-woof in another AI session.** Do NOT touch its DNS/email from this project.
|
||
- Current state (unchanged): NS = InMotion (ns.inmotionhosting.com / ns2), A → 144.6.86.11 (static IP, local), no email records.
|
||
- Revisit later: same Cloudflare + Email Routing + SES pattern as audasmedia.com.au, when user is ready.
|
||
|
||
## Next steps (post-email)
|
||
|
||
## PHASE 2 EMAIL — where-woof.com DONE ✅ (2026-08-31)
|
||
- DNS → Cloudflare (zone live on barbara.ns.cloudflare.com; NS switched at Squarespace).
|
||
- Records: A apex/www/admin → 144.6.86.11 (DNS-only); InMotion A's + stale TXT deleted; DKIM×3 + DMARC added.
|
||
- Email Routing active (MX route1/2/3 priorities 4/25/14, SPF include:_spf.mx.cloudflare.net); rule `hello@where-woof.com → samuelrolfe@gmail.com`.
|
||
- SES identity created; MAIL FROM = default.
|
||
- Awaiting end-to-end send test (from non-Gmail-same-account address to hello@where-woof.com).
|
||
- Note: audasmedia MX priorities are 54/92/78; where-woof 4/25/14 — Cloudflare assigns per-zone, both valid.
|
||
|
||
## PHASE 2 EMAIL — where-woof.com → PURELYMAIL (2026-09-02)
|
||
- **Switched from Cloudflare Email Routing to PurelyMail** (real domain mailboxes, IMAP/SMTP, ~US$10/yr).
|
||
- **Cloudflare Email Routing DISABLED** on where-woof.com (frees MX — cannot coexist with external provider).
|
||
- DNS records added in Cloudflare (all verified live):
|
||
- MX @ → `mailserver.purelymail.com` (prio 10)
|
||
- TXT @ → `purelymail_ownership_proof=3ac61cfe...`
|
||
- TXT @ → `v=spf1 include:_spf.purelymail.com include:amazonses.com ~all`
|
||
- CNAME `purelymail1/2/3._domainkey` → `key1/2/3.dkimroot.purelymail.com`
|
||
- CNAME `_dmarc` → `dmarcroot.purelymail.com`
|
||
- CNAME `autoconfig` → `autoconfig.purelymail.com`
|
||
- SRV `_autodiscover._tcp` → `autodiscover.purelymail.com` (0 0 443)
|
||
- Deleted old leftovers: SES DKIM×3 (where-woof), `mail`/`smtp` CNAMEs, old `_dmarc` TXT.
|
||
- PurelyMail domain: validated (Check DNS Records done); Deliver mail to = PurelyMail; mailbox `hello@where-woof.com` created.
|
||
- **Pending:** PurelyMail mailbox → Thunderbird or Gmail "Send mail as" (user leaning: Gmail inbox + PurelyMail SMTP send-as; Thunderbird optional).
|
||
|
||
## SITE SENDING HANDOFF — for the AI building where-woof.com (2026-09-02)
|
||
Hand this to the where-woof website builder:
|
||
```
|
||
where-woof.com outgoing (transactional) email:
|
||
- Provider: Amazon SES, region ap-southeast-2; identity where-woof.com verified
|
||
- Sender: hello@where-woof.com
|
||
- SMTP: create user in SES → SMTP settings; host email-smtp.ap-southeast-2.amazonaws.com, port 587 STARTTLS
|
||
- BEFORE sending to real users: request SES production access (SES → Account dashboard → Request production access; free, ~hours)
|
||
- DNS for delivery already set (SPF/DKIM/DMARC include SES + PurelyMail)
|
||
- Do NOT use PurelyMail SMTP for site/transactional mail — use SES
|
||
```
|
||
|
||
## SES SMTP CREDENTIALS (created 2026-09-03, local file `aws-credentials.local`)
|
||
- Endpoint: `email-smtp.ap-southeast-2.amazonaws.com` : `587` (STARTTLS)
|
||
- IAM user: `ses-smtp-user.20260903-152339`; SMTP username/password stored in `aws-credentials.local` (600, not Obsidian)
|
||
- Permission: `ses:SendRawEmail` only (least privilege)
|
||
- **Production access requested 2026-09-03** (transactional form, where-woof.com) — awaiting AWS approval (~hours)
|
||
- Until approval: SES sandbox = can only send to verified addresses (samuelrolfe@gmail.com).
|
||
|
||
## TODO — audasmedia.com.au SES (2026-09-03)
|
||
- [ ] **Give audasmedia.com.au the same SES sending access as where-woof**: create/confirm SES identity for audasmedia.com.au (already exists), create SMTP creds for it, and ensure production access covers it.
|
||
- audasmedia.com.au will also send email (site mail) → same pattern as where-woof.
|
||
- When audasmedia infra is built on .13 (user to advise).
|
||
|
||
## NEXT — audasmedia.com.au infra on .13 (2026-09-03)
|
||
- User will advise details. Apply same hosting/backup/email pattern as where-woof.
|
||
|
||
## VISUALIZATION — status page upgrade + Garage UI handoff (2026-09-05)
|
||
- **Status page upgraded** (`offsite.lab.audasmedia.com.au`):
|
||
- Fixed "failed" label → now shows `OK (finished cleanly)` / `OK (6h window ended 05:00)` in green (the systemd "timeout" at 05:00 is the expected RuntimeMaxSec stop, not an error).
|
||
- Added **7-night history table** (parsed from verify.log: date, result, objects, delta).
|
||
- Added "Last verify" line.
|
||
- **Garage UI handoff written** (`Garage Web UI — handoff.md`): use `khairul169/garage-webui` (Garage v1.x compatible; Noooste/garage-ui needs 2.0+). Deploy on :3909, admin API :3902, token from .env.
|
||
- **Grafana deferral noted**: Grafana on .35 has NO data sources (empty). Defer consolidated dashboard unless needed after 1+2.
|
||
- Assessment: Backrest/Kopia UIs visualize only their own local repos (.23), not the AWS S3 offsite. Garage UI is separate (local Garage, not AWS).
|
||
|
||
## GARAGE v2 UPGRADE (2026-09-05)
|
||
- **Why:** garage-webui / Noooste UIs didn't list buckets on Garage v1.0.1 (v1 admin API lacks their expected endpoints). Storage itself was fine (S3 v1==v2 for portal).
|
||
- **Backup first:** `/mnt/data/family-home-lab/garage-backup-20260905-111241/` (data 181M + meta 1.1M, via busybox as root since LMDB files are root-owned).
|
||
- **Upgraded:** `dxflrs/garage:v1.0.1` → `v2.1.0`; garage.toml rewritten (v2 schema: `replication_factor`, `[s3_web]`, admin on **3903**); compose updated (ports 3900/3901/3902/3903). Tokens preserved exactly (admin/metrics/rpc 64-hex).
|
||
- **Verified:** v2 auto-migrated the LMDB on start; `GetClusterStatus` shows node up v2.1.0; `ListBuckets` returns existing buckets (outline,...) via `POST /v2/...`.
|
||
- **garage-webui:** API_BASE_URL → `http://garage:3903` (admin moved to 3903), recreated, 0 refused-erores, HTTP 200. **USER CONFIRMED: all buckets now show in the dashboard. ✅**
|
||
- Caddy `garage-ui.lab.audasmedia.com.au` → 3909 (basic-auth). Removed a stale 3902 block.
|
||
- **Rollback:** if needed, stop v2, restore garage-backup-*, revert compose/garage.toml, restart v1.0.1.
|
||
|
||
## TIER-A DB HARDENING (2026-09-05/06)
|
||
- **Findings:** All ~21 DB containers across .13/.35 are covered by existing backups (Borg on .13 via `~/Docker`; Backrest/Kopia on .35 via `/docker-containers`+volumes) — **no coverage gap**. Confirmed Vikunja = 437 paths in restic; Outline in Borg. The shared weakness: all are **file-copies of live DBs** (inconsistent-restore risk).
|
||
- **Plan:** dump critical DBs to an already-backed-up path before the backup runs.
|
||
- **.13 (done, dumps tested):** `db-dump-13.sh` dumps **outline, wherewoof-db, family-home-lab(portal), langfuse, ai-resume, airflow** Postgres via container-env creds (no secrets in config) → `~/Docker/Containers/<app>/backups/db-dumps/` (inside Borg `~/Docker` paths, keep-3). Tested: 201K/31K/6K/278K/824/197K dumps written.
|
||
- **Pending:** wire .13 timer (03:30 pre-Borg); do .35 Tier-A (vikunja + others) via Backrest pre-hook or dump script.
|
||
|
||
## TIER-A DB HARDENING — IMPLEMENTATION (2026-09-06)
|
||
- **.13** (`~/.Docker → Borg`): `db-dump-13.sh` + NixOS module `dbdump.nix` → timer **03:30** pre-Borg. Dumps outline, wherewoof-db, family-home-lab(portal), langfuse, ai-resume, airflow PG → `~/Docker/Containers/<app>/backups/db-dumps/` (in Borg paths). **Fixed:** service must run `User=root` (/home/sam/Docker is root-owned) — rebuild applied 2026-09-06.
|
||
- **.35** (`/Docker/Containers → Backrest`): `db-dump-35.sh` + systemd unit/timer → **03:35** daily. Dumps vikunja(mariadb via env pw), gitea(mysqldump, pw `sam4jo`), affine, immich, paperless (pg via env) → `/Docker/Containers/<app>/backups/db-dumps/` (in restic path). Timer installed+enabled; verified affine 2MB dump works.
|
||
- **Notes:** Backrest image lacks pg/mysql/docker clients → used host systemd timer (root) instead of Backrest hooks. firefly_iii compose not found under /Docker/Containers/firefly_iii (Tier-B anyway; skip). Gitea root pw `sam4jo` stored in root-only script.
|
||
- **Awaiting:** .13 rebuild (User=root) + first automatic fires tonight/tomorrow 03:30/03:35.
|
||
- **langfuse + airflow removed (2026-09-06):** stopped (not deleted; compose dirs remain). Removed langfuse/airflow lines from `db-dump-13.sh` (would have errored nightly). pi-langfuse extension already absent from .27. Offsite/backup configs unaffected.
|
||
1. SES console → Send test email (audasmedia.com.au → samuelrolfe@gmail.com); check delivery + DKIM/SPF pass.
|
||
2. Request SES production access (free).
|
||
3. Then return to Phase 3 (local infra) / CMS migration project.
|
||
- **Build:** create bucket `sam-offsite-backup` + lifecycle (S3→Glacier Flexible@30d→Glacier DA@90d), then wire local Borg/rclone→S3 on `.13`, seed, test restore.
|
||
- **Security:** enable root MFA; consider rotating access key after seeding (it was pasted in chat).
|
||
|
||
---
|
||
|
||
## 2026-09-02 — Backup gap closure: config + root-cause fix
|
||
|
||
| Item | What | Status |
|
||
|---|---|---|
|
||
| .13 Borg coverage | `backup.nix` local-to-storage + `~/deployment`, voice/audio/dev dirs, `.agents`, `.pi`, `bin`, `.config`; excludes + `.cache`/`.npm`/`.local` | ✅ applied |
|
||
| prefect DB | Added to Borg exclude (was causing recurring `.failed` runs) | ✅ applied |
|
||
| Borg nightly | 05:00 runs succeeding (index written Sep 2 05:01); 23G first archive slow (lzma) | ✅ working |
|
||
| **S3 ordering bug** | **Found: script ran sequentially; photos consumed the entire 6h nightly window → `borg/.13` + home dirs never reached S3 (0 objects)** | 🔧 **fixed** |
|
||
| S3 ordering fix | Reordered `offsite.nix`: `.13` repo + `.27` repo + home configs sync **FIRST** (minutes), photos/archive last (fill remaining window) | ✅ config in; rebuild pending |
|
||
| Restore runbook | `Restore Runbook.md` written + linked from Backup Architecture | ✅ |
|
||
| Note | `borg list` fails for sam (repo data/ root-owned — known artifact; Borg job runs as root, works) | noted |
|
||
|
||
---
|
||
|
||
*Linked from [[Migration Plan — InMotion to Local + AWS Offsite]].* |