From 86e8f9e02991c390f884ddcd0e78a08b55c6b0ee Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Fri, 28 Aug 2026 15:09:08 +1000 Subject: [PATCH] sam-4screen-desktop 2026-8-28:15:9:8 --- .obsidian/workspace.json | 2 +- ... Plan — InMotion to Local + AWS Offsite.md | 280 ++++++++++++++++++ 2 files changed, 281 insertions(+), 1 deletion(-) create mode 100644 200 projects/220 Web Host Migration/Migration Plan — InMotion to Local + AWS Offsite.md diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index 3d417cd..9472963 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -211,6 +211,7 @@ }, "active": "7066c9273c6fb936", "lastOpenFiles": [ + "200 projects/220 Web Host Migration/Migration Plan — InMotion to Local + AWS Offsite.md", "300 areas/360 Dev-Ops Network Computers/Filesystem Drive Map.md", "300 areas/360 Dev-Ops Network Computers/Home Network Map Overview.md", "300 areas/360 Dev-Ops Network Computers/Backup Health Check Commands.md", @@ -236,7 +237,6 @@ "000 daily/House Family General.md", "100 inbox/Pi Subagent Integration.md", "100 inbox/Photo Consolidation Plan.md", - "200 projects/210 AI Resume/TencentCloudTencentDB-Agent-Memory TencentDB Agent Memory is a team-level memory hub for AI Agents — turning conversations, docs, and code into four reusable memory assets (Chat Memory, Skill, LLM-Wiki, Code-Graph) that are governed, shared, and.md", "000 daily/ThinkPad Recommendations", "500 archive/510 Daily", "300 areas/305 Ideas Businesses", diff --git a/200 projects/220 Web Host Migration/Migration Plan — InMotion to Local + AWS Offsite.md b/200 projects/220 Web Host Migration/Migration Plan — InMotion to Local + AWS Offsite.md new file mode 100644 index 0000000..ef4ff91 --- /dev/null +++ b/200 projects/220 Web Host Migration/Migration Plan — InMotion to Local + AWS Offsite.md @@ -0,0 +1,280 @@ +--- +created: 2026-08-28 +modified: 2026-08-28 +type: plan +client: sam +project: web-host-migration +status: active +priority: 1 +last_verified: 2026-08-28 +tags: + - migration + - self-hosting + - aws + - email + - backup + - inmotionhosting + - nixos +aliases: + - in-motion-migration-plan +id: 1848129017-MIGR +--- + +# Migration Plan — InMotion VPS → Local (.13) + AWS Offsite + +> **Status: PLAN ONLY.** This document plans infrastructure + offsite backup + email. +> The CMS code / MySQL data migration itself is a **separate project** (not covered here). + +--- + +## 1. Objective & Scope + +Move websites off an expensive, old **InMotion Hosting VPS** to **local hosting on `.13`** +(nixos-desktop, AMD RX 6600 GPU) with **email** (inbound mailboxes + transactional outbound) +and **cheap AWS offsite backup**. Build toward `${30–50}/mo AUD` total infra spend — a **~50–70 AUD/mo net saving** versus the retired VPS — while demonstrating **employable AWS / DevOps skills**. + +### In scope (this plan) +- Target hosting architecture on `.13` + reverse proxy/service stack +- Email architecture (inbound + outbound) +- AWS offsite backup design + cost model +- Networking / external exposure +- Migration phasing & risks + +### Out of scope (separate project) +- Porting the custom PHP/MySQL multimedia CMS code +- Migrating MySQL/Cron tasks for those 5–6 sites +- Updating the CMS itself (noted as needed — handled elsewhere) + +--- + +## 2. Verified Current State (2026-08-28) + +| Item | Detail | +|---|---| +| Public exposure | Static IP (Aussie Broadband) + router port-forward 80/443 → Caddy on `.35` | +| Reverse proxy | Caddy on `.35`; e.g. `where-woof.com` → `192.168.20.13:3020` | +| VM host | Proxmox (hosts `.35` caddy-server + Home Assistant) | +| `.13` host | NixOS, Docker host, 15.5 GB RAM, **AMD RX 6600** (replacing dead GTX 760) | +| `.13` web-adjacent | `wherewoof-admin` (:3031), `wherewoof-minio` (S3 :9010), `wherewoof-db` (postgres :5434), **Garage** S3 (`family-home-lab-garage` :3900/3902) + **portal/console** (:8500), postgres, etc. | +| Current VPS | InMotion smallest plan, **~100 AUD/mo**, cPanel + PHP + MySQL + FFmpeg + ImageMagick + cron, 5–6 CMS sites | +| Email | **No mail stack deployed yet** — clean slate | +| Backup (existing, local) | Borg (`.27`/`.13` → `.13` USB 3TB), rsync → `.35`→`.23`); Kopia + Restic (`.35`→`.23`); large free capacity | +| Backup gap | `.23` file-server has **no backup**; Immich photos (~472G on `.13` archive) and My-Passport/`.35` data at risk in single-drive sites | + +**Note on Garages/minio:** `.13` already runs **Garage** (S3-compatible, family-home-lab) at `:3900/:3902` serving the local console (`console.lab.audasmedia.com.au`). This is the local S3 asset store; it is distinct from AWS offsite backup. + +--- + +## 3. Target Architecture (3 layers) + +``` + ┌─────────────────────────────────────────────┐ + │ Public internet (A-record → static IP) │ + │ Aussie Broadband static IP → router │ + └──────────────┬──────────────────────────────┘ + │ port-forward 80/443 + ┌────────▼─────────┐ + │ Caddy (.35) │ reverse proxy / TLS/ACME + │ existing │ + └────────┬─────────┘ + ┌──────────────▼──────────────────────────────┐ + │ .13 NixOS (Docker) ← migrate CMS here │ + │ + Garage local S3 (media) :3900/3902 │ + │ + NEWMail stack (inbound or relay) │ + └──────────────┬──────────────────────────────┘ + │ rclone/restic (via Borg) over HTTPS + ┌────────▼──────────────────────────────┐ + │ AWS (S3 hot → lifecycle → Glacier DA) │ + │ offsite backup (burn-down safe) │ + └────────────────────────────────────────┘ + +Email path (outbound): App → SES SMTP/API → internet (SPF/DKIM/DMARC) +Email path (inbound): MX → provider (webmail/IMAP) → user apps +``` + +- **Hosting:** migrate to `.13` Docker (+ AMD GPU) behind existing Caddy on `.35`. Replace cPanel/InMotion with NixOS services + Docker + Caddy — no cPanel needed. (cPanel alternatives / FLoci are for emulation/testing, not production hosting — see §8.) +- **Offsite:** AWS S3 Standard for recent site data → **lifecycle** auto-archives old media + Immich photos to **S3 Glacier Deep Archive**. +- **Email:** inbound mailboxes via a managed provider; outbound transactional via **Amazon SES**. See §6. + +--- + +## 4. Decisions & Rationale (decision log) + +| # | Decision | Rationale | +|---|---|---| +| D1 | **AWS-only offsite** (S3→Glacier DA) | Cheapest cold storage + single vendor + employable AWS skills. User confirmed. | +| D2 | Budget **${20–50}/mo** | Retiring ~100/mo VPS leaves huge headroom; aim ~50–70/mo net saving. | +| D3 | Inbound = **real mailboxes + transactional outbound** (both) | Needs inboxes users check + programmatic per-site sending. | +| D4 | **No cPanel** | NixOS + Docker + Caddy + Garage is the control plane; cPanel is a Windows-era admin layer on an old VPS — unnecessary locally. | +| D5 | Amazon SES for outbound | Best deliverability + AWS skill; ~1000s/mo free tier; scales at ~0.10/1000. | +| D6 | Managed inbound provider (not self-hosted mail on residential IP) | Residential IPs are routinely blacklisted for inbound MX; keeps reputation isolated (ref: obsidian note). | + +> Outbound credibility + inbound management: use a paid-free-tier inbound provider for **mailboxes** and SES for **transactional** sending — keeps them on separate reputation islands. + +--- + +## 5. Hosting Stack (target, on `.13`) + +| Layer | Choice | Notes | +|---|---|---| +| OS / services | NixOS + Docker Compose | Existing tooling on `.13` | +| Reverse proxy | **Caddy on `.35`** (keep) | Already fronts all `*.lab.audasmedia.com.au` + `where-woof.com` | +| S3 asset store | **Garage on `.13`** (keep) | Local object storage for media; S3-compatible | +| CMS backend | To be migrated in separate project | PHP/MySQL/FFmpeg/ImageMagick/cron | +| GPU tasks | Cloud (future-proof) | See §7 — move GPU/FFmpeg/ImageMagick jobs to AWS (e.g. Lambda/EC2/Fargate) or local AMD on `.13` for dev/CI, cloud for scale | +| Control panel | **None needed** | NixOS + Docker + Caddy + portainer(opt) | + +**Web forwarding flow (unchanged):** Caddy on `.35` reverse_proxies `*.audasmedia` + `where-woof` → `.13:`. + +--- + +## 6. Email Architecture + +### 6.1 Inbound mailboxes (managed provider — reputation isolation) +Recommend (pick one; both low-volume friendly): +- **Zoho Mail** — free tier ≤5 users, webmail + IMAP/POP. ~1/user/mo to unlock IMAP for Thunderbird/Gmail apps. +- **MXroute** — flat ~30–45/yr, unlimited domains, cPanel-style, no server admin. + +### 6.2 Outbound transactional (SES) +- **Amazon SES** — deliverability + AWS skill. + - Free tier: 62,000/mo if sending from EC2; otherwise ~0.10 USD per 1,000 (low volume here → effectively ~free). + - Configure SPF + DKIM + DMARC TXT records; sandbox exit via limit increase. + +### 6.3 DNS records needed +- `MX` → inbound provider +- `TXT` SPF (include both SES + inbound provider), `DKIM` (SES + provider), `DMARC` +- Keep Caddy ACME TLS (already working via port-forward). + +**Note:** at very low current volume, plan for ~0–6 AUD/mo email. + +--- + +## 7. GPU / Media (FFmpeg, ImageMagick) — Future-Proofing + +- **Local (`AMD RX 6600`):** dev, bursty local jobs, scheduled cron work for small media (thumbnails, transcodes) — free, but only-on-PC. +- **Cloud (AWS) for scale/portability:** + - **S3 event → Lambda → FFmpeg/ImageMagick** for on-demand image/video processing (media-heavy sites already S3-backed via Garage → eventual AWS MediaConvert/or transcoding). + - **AWS MediaConvert** for video; **Lambda** for image ops; **AWS Batch/ECS/Fargate** for heavier episodic work like this CMS's cron jobs. +- **Hugging Face (optional):** only if you add AI/ML media features (face detection, captioning, etc.) — ruled out for now; revisit if CMS upgrade adds AI. + +Recommendation: keep GPU-media **local-first on `.13`** for cost + dev; use **cloud for scale/off-site cron** only when it becomes the bottleneck. Any cloud GPU is only needed at scale — don't commit SaaS GPU spend now. + +--- + +## 8. FLoci / cPanel Alternatives — Clarification + +- **cPanel** is the admin panel on the old VPS. Not needed locally — `docker compose` + NixOS + Caddy replaces it. +- **FLoci** = local emulators of AWS/Azure/GCP for **offline testing** (no cloud, no credentials) on `localhost:4566`. Useful for: + - developing/test the AWS S3/Glacier backup scripts without touching real AWS + - testing SES/API integration offline + Use it as a **dev/CI tool**, not production hosting. (Good for the CMS migration project too.) + +--- + +## 9. Offsite Backup Design (AWS-only) — Cost Model in AUD + +### 9.1 Target layout +- **S3 bucket** `sam-offsite-backup` + - **S3 Standard** (hot, <30 days): recent site data, DB dumps + - **Lifecycle rule** → move to **Glacier Deep Archive** after ~30 days (new cold data: old site media, Immich photos, archives) +- **Tooling:** keep existing **Borg** for local, add **rclone** (or restic) → AWS (S3 API). `rclone` is free, already skill-aligned with your backup pipeline. + +### 9.2 Cost @ approx AUD (2026, incl. ~0.10 USD/AUD) + +| Tier | AUD/GB-mo | 100 GB/mo | 500 GB/mo | +|---|---|---|---| +| S3 Standard (hot) | ~0.036 | **3.60** | 18.00 | +| Glacier Flexible (backup) | ~0.005 | 0.50 | 2.80 | +| **Glacier Deep Archive (cold)** | **~0.0015** | 0.15 | **0.75** | +| Backblaze B2 (ref) | ~0.0095 | 0.95 | 4.75 | +| Hetzner Storage Box (ref, 1 TB flat) | ~6.00 flat | 6.00 | 6.00 | + +**Bottom line:** +- **~472 G Immich photos + old site media → Glacier DA ≈ 0.75–1.2 AUD/mo.** +- Recent site data → S3 Standard ≈ a few AUD/mo. +- **Expected offsite = ~2–6 AUD/mo.** Total infra (email+backup+compute) well under the 20–50 budget. + +### 9.3 Restore/egress note +- Glacier DA restores are slower (hours) but rare — fine for burn-down recovery. +- Get data out costs ~0.14 AUD/GB on AWS; B2/Hetzner cheaper (free to 3×), but since you chose **AWS-first for skill**, restore volume is low → acceptable. + +--- + +## 10. Backup of local systems (disaster / burn-down) + +- Currently: Borg (local USB 3T on `.13`), Kopia/Restic (`.35`→`.23` NFS). **This offsite AWS bucket covers the burn-down risk.** +- **Gap to address in this plan:** the **Immich photos (~472 G on `.13` archive)** and **My-Passport `/mnt/hd`** need to be in the AWS offsite (lifecycle → Glacier DA). +- **Optional:** also add the `.23` file-server no-backup gap, but it's a redundancy target for other repos — lower priority. + +--- + +## 11. Networking / External Exposure (unchanged, verified) + +- A-record → static IP → router port-forward 80/443 → Caddy `.35` → `.13`. +- Keep this. Works for SSH-git (Gitea), Caddy ACME TLS, and public sites. +- **Residential IP port-80/443 works today** for sites; **do not** use residential IP as *mail* MX (blacklist risk) — inbound via provider, outbound via SES. +- Email doesn't require inbound ports on your IP. + +--- + +## 12. Phased Migration Plan (Infra first; CMS elsewhere) + +### Phase 0 — Pre-flight (read-only, no changes) +- [ ] Confirm AWS account exists / create `ap-southeast-2` regional setup +- [ ] Confirm total media size; refine cost model +- [ ] Verify `.13` can reach AWS (no proxy issues) + +### Phase 1 — Offsite backup (quick win) +- [ ] Create initial AWS account + IAM user (least privilege, `s3:PutObject/GetObject` on backup bucket only) +- [ ] Configure `rclone` remote (or restic) against the bucket +- [ ] Enact Borg → rclone → AWS S3 pipeline on `.13` +- [ ] Add S3 lifecycle rule S3→Glacier DA after 30 days +- [ ] **First offsite seed** (photos + site data) — verify restore in test + +### Phase 2 — Email +- [ ] Pick inbound provider (Zoho or MXroute) → set up mailboxes + MX +- [ ] Set up Amazon SES outbound + verify domain, SPF/DKIM/DMARC, exit sandbox +- [ ] Point site forms / notifications → SES + +### Phase 3 — Target infra prep (on `.13`), no CMS migration +- [ ] Confirm AMD RX 6600 installed + `amdgpu` driver on `.13` +- [ ] Prepare Docker-compose volumes / directories for incoming CMS sites +- [ ] Verify Caddy routes; plan reverse_proxy entries for new sites + +### Phase 4 — (separate project) CMS migration +- Not in scope here. + +### Phase 5 — Decommission +- [ ] Retire InMotion VPS (share of ~100/mo) once sites are live locally +- [ ] Point final DNS A-records fully to static IP + +--- + +## 13. Risks & Open Items + +| Risk | Impact | Mitigation | +|---|---|---| +| Residential IP public services | moderate | Keep Caddy ACME working; proxy sites; don't use MAIL MX on residential IP | +| AWS egress on restore | low | Cold-tier restores rare; budget it; test restore path in Phase 1 | +| CMS GPU-media scale | low | Local AMD on `.13` now; AWS MediaConvert/Lambda later only if bottleneck | +| `.23` no backup | low-med | Downgrade constant; add to AWS offsite if desired (Phase 1 optional) | +| Immich/My-Passport single-drive risk | **concern** | MUST be in AWS offsite (Glacier DA) — Phase 1 first seed | +| cPanel dependency | low | Fully replaced by Docker/NixOS/Caddy | + +--- + +## 14. Summary of Spend / Skill + +| Item | Est AUD/mo | +|---|---| +| Retired InMotion VPS | −100 (saving) | +| Email (inbound free-tier + SES low volume) | ~0–6 | +| AWS offsite backup (sites + photos, Glacier DA) | ~2–6 | +| Optional small compute (if GPU moved to cloud) | ~0–5 | +| **Total new infra** | **~5–15 AUD/mo** | +| **Net saving vs today** | **~50–80 AUD/mo** | +| **Skill wins** | AWS S3 + lifecycle + IAM + SES, Terraform/CLI-ready, rclone/Borg, Docker/NixOS/Caddy — all employable | + +--- + +*Created 2026-08-28. Source context: [[Local Web Host]], [[Email Backup off site hosting]], [[AWS Cloud Emulator Floci]], backup/network/drive notes.* \ No newline at end of file