sam-4screen-desktop 2026-8-28:15:9:8

2026-08-28 15:09:08 +10:00
parent 52fb7e6b3b
commit 86e8f9e029
2 changed files with 281 additions and 1 deletions
--- a/.obsidian/workspace.json
+++ b/.obsidian/workspace.json
@@ -211,6 +211,7 @@
  },
  "active": "7066c9273c6fb936",
  "lastOpenFiles": [
    "200 projects/220 Web Host Migration/Migration Plan — InMotion to Local + AWS Offsite.md",
    "300 areas/360 Dev-Ops Network Computers/Filesystem Drive Map.md",
    "300 areas/360 Dev-Ops Network Computers/Home Network Map Overview.md",
    "300 areas/360 Dev-Ops Network Computers/Backup Health Check Commands.md",
@@ -236,7 +237,6 @@
    "000 daily/House Family General.md",
    "100 inbox/Pi Subagent Integration.md",
    "100 inbox/Photo Consolidation Plan.md",
    "200 projects/210 AI Resume/TencentCloudTencentDB-Agent-Memory TencentDB Agent Memory is a team-level memory hub for AI Agents — turning conversations, docs, and code into four reusable memory assets (Chat Memory, Skill, LLM-Wiki, Code-Graph) that are governed, shared, and.md",
    "000 daily/ThinkPad Recommendations",
    "500 archive/510 Daily",
    "300 areas/305 Ideas Businesses",
--- a/Migration/Migration
+++ b/Migration/Migration
@@ -0,0 +1,280 @@
 ---
 created: 2026-08-28
 modified: 2026-08-28
 type: plan
 client: sam
 project: web-host-migration
 status: active
 priority: 1
 last_verified: 2026-08-28
 tags:
  - migration
  - self-hosting
  - aws
  - email
  - backup
  - inmotionhosting
  - nixos
 aliases:
  - in-motion-migration-plan
 id: 1848129017-MIGR
 ---
 # Migration Plan — InMotion VPS → Local (.13) + AWS Offsite
 > **Status: PLAN ONLY.** This document plans infrastructure + offsite backup + email.
 > The CMS code / MySQL data migration itself is a **separate project** (not covered here).
 ---
 ## 1. Objective & Scope
 Move websites off an expensive, old **InMotion Hosting VPS** to **local hosting on `.13`**
 (nixos-desktop, AMD RX 6600 GPU) with **email** (inbound mailboxes + transactional outbound)
 and **cheap AWS offsite backup**. Build toward `${30–50}/mo AUD` total infra spend — a **~50–70 AUD/mo net saving** versus the retired VPS — while demonstrating **employable AWS / DevOps skills**.
 ### In scope (this plan)
 - Target hosting architecture on `.13` + reverse proxy/service stack
 - Email architecture (inbound + outbound)
 - AWS offsite backup design + cost model
 - Networking / external exposure
 - Migration phasing & risks
 ### Out of scope (separate project)
 - Porting the custom PHP/MySQL multimedia CMS code
 - Migrating MySQL/Cron tasks for those 5–6 sites
 - Updating the CMS itself (noted as needed — handled elsewhere)
 ---
 ## 2. Verified Current State (2026-08-28)
 | Item | Detail |
 |---|---|
 | Public exposure | Static IP (Aussie Broadband) + router port-forward 80/443 → Caddy on `.35` |
 | Reverse proxy | Caddy on `.35`; e.g. `where-woof.com` → `192.168.20.13:3020` |
 | VM host | Proxmox (hosts `.35` caddy-server + Home Assistant) |
 | `.13` host | NixOS, Docker host, 15.5 GB RAM, **AMD RX 6600** (replacing dead GTX 760) |
 | `.13` web-adjacent | `wherewoof-admin` (:3031), `wherewoof-minio` (S3 :9010), `wherewoof-db` (postgres :5434), **Garage** S3 (`family-home-lab-garage` :3900/3902) + **portal/console** (:8500), postgres, etc. |
 | Current VPS | InMotion smallest plan, **~100 AUD/mo**, cPanel + PHP + MySQL + FFmpeg + ImageMagick + cron, 5–6 CMS sites |
 | Email | **No mail stack deployed yet** — clean slate |
 | Backup (existing, local) | Borg (`.27`/`.13` → `.13` USB 3TB), rsync → `.35`→`.23`); Kopia + Restic (`.35`→`.23`); large free capacity |
 | Backup gap | `.23` file-server has **no backup**; Immich photos (~472G on `.13` archive) and My-Passport/`.35` data at risk in single-drive sites |
 **Note on Garages/minio:** `.13` already runs **Garage** (S3-compatible, family-home-lab) at `:3900/:3902` serving the local console (`console.lab.audasmedia.com.au`). This is the local S3 asset store; it is distinct from AWS offsite backup.
 ---
 ## 3. Target Architecture (3 layers)
 ```
                        ┌─────────────────────────────────────────────┐
                        │  Public internet  (A-record → static IP)     │
                        │  Aussie Broadband static IP → router         │
                        └──────────────┬──────────────────────────────┘
                                       │ port-forward 80/443
                              ┌────────▼─────────┐
                              │  Caddy (.35)     │  reverse proxy / TLS/ACME
                              │  existing        │
                              └────────┬─────────┘
                        ┌──────────────▼──────────────────────────────┐
                        │  .13  NixOS  (Docker)  ← migrate CMS here   │
                        │  + Garage local S3 (media) :3900/3902        │
                        │  + NEWMail stack (inbound or relay)          │
                        └──────────────┬──────────────────────────────┘
                                       │ rclone/restic (via Borg) over HTTPS
                              ┌────────▼──────────────────────────────┐
                              │ AWS  (S3 hot → lifecycle → Glacier DA) │
                              │  offsite backup (burn-down safe)       │
                              └────────────────────────────────────────┘
 Email path (outbound):  App → SES SMTP/API → internet  (SPF/DKIM/DMARC)
 Email path (inbound):   MX → provider (webmail/IMAP) → user apps
 ```
 - **Hosting:** migrate to `.13` Docker (+ AMD GPU) behind existing Caddy on `.35`. Replace cPanel/InMotion with NixOS services + Docker + Caddy — no cPanel needed. (cPanel alternatives / FLoci are for emulation/testing, not production hosting — see §8.)
 - **Offsite:** AWS S3 Standard for recent site data → **lifecycle** auto-archives old media + Immich photos to **S3 Glacier Deep Archive**.
 - **Email:** inbound mailboxes via a managed provider; outbound transactional via **Amazon SES**. See §6.
 ---
 ## 4. Decisions & Rationale (decision log)
 | # | Decision | Rationale |
 |---|---|---|
 | D1 | **AWS-only offsite** (S3→Glacier DA) | Cheapest cold storage + single vendor + employable AWS skills. User confirmed. |
 | D2 | Budget **${20–50}/mo** | Retiring ~100/mo VPS leaves huge headroom; aim ~50–70/mo net saving. |
 | D3 | Inbound = **real mailboxes + transactional outbound** (both) | Needs inboxes users check + programmatic per-site sending. |
 | D4 | **No cPanel** | NixOS + Docker + Caddy + Garage is the control plane; cPanel is a Windows-era admin layer on an old VPS — unnecessary locally. |
 | D5 | Amazon SES for outbound | Best deliverability + AWS skill; ~1000s/mo free tier; scales at ~0.10/1000. |
 | D6 | Managed inbound provider (not self-hosted mail on residential IP) | Residential IPs are routinely blacklisted for inbound MX; keeps reputation isolated (ref: obsidian note). |
 > Outbound credibility + inbound management: use a paid-free-tier inbound provider for **mailboxes** and SES for **transactional** sending — keeps them on separate reputation islands.
 ---
 ## 5. Hosting Stack (target, on `.13`)
 | Layer | Choice | Notes |
 |---|---|---|
 | OS / services | NixOS + Docker Compose | Existing tooling on `.13` |
 | Reverse proxy | **Caddy on `.35`** (keep) | Already fronts all `*.lab.audasmedia.com.au` + `where-woof.com` |
 | S3 asset store | **Garage on `.13`** (keep) | Local object storage for media; S3-compatible |
 | CMS backend | To be migrated in separate project | PHP/MySQL/FFmpeg/ImageMagick/cron |
 | GPU tasks | Cloud (future-proof) | See §7 — move GPU/FFmpeg/ImageMagick jobs to AWS (e.g. Lambda/EC2/Fargate) or local AMD on `.13` for dev/CI, cloud for scale |
 | Control panel | **None needed** | NixOS + Docker + Caddy + portainer(opt) |
 **Web forwarding flow (unchanged):** Caddy on `.35` reverse_proxies `*.audasmedia` + `where-woof` → `.13:<port>`.
 ---
 ## 6. Email Architecture
 ### 6.1 Inbound mailboxes (managed provider — reputation isolation)
 Recommend (pick one; both low-volume friendly):
 - **Zoho Mail** — free tier ≤5 users, webmail + IMAP/POP. ~1/user/mo to unlock IMAP for Thunderbird/Gmail apps.
 - **MXroute** — flat ~30–45/yr, unlimited domains, cPanel-style, no server admin.
 ### 6.2 Outbound transactional (SES)
 - **Amazon SES** — deliverability + AWS skill.
  - Free tier: 62,000/mo if sending from EC2; otherwise ~0.10 USD per 1,000 (low volume here → effectively ~free).
  - Configure SPF + DKIM + DMARC TXT records; sandbox exit via limit increase.
 ### 6.3 DNS records needed
 - `MX` → inbound provider
 - `TXT` SPF (include both SES + inbound provider), `DKIM` (SES + provider), `DMARC`
 - Keep Caddy ACME TLS (already working via port-forward).
 **Note:** at very low current volume, plan for ~0–6 AUD/mo email.
 ---
 ## 7. GPU / Media (FFmpeg, ImageMagick) — Future-Proofing
 - **Local (`AMD RX 6600`):** dev, bursty local jobs, scheduled cron work for small media (thumbnails, transcodes) — free, but only-on-PC.
 - **Cloud (AWS) for scale/portability:** 
  - **S3 event → Lambda → FFmpeg/ImageMagick** for on-demand image/video processing (media-heavy sites already S3-backed via Garage → eventual AWS MediaConvert/or transcoding).
  - **AWS MediaConvert** for video; **Lambda** for image ops; **AWS Batch/ECS/Fargate** for heavier episodic work like this CMS's cron jobs.
 - **Hugging Face (optional):** only if you add AI/ML media features (face detection, captioning, etc.) — ruled out for now; revisit if CMS upgrade adds AI.
 Recommendation: keep GPU-media **local-first on `.13`** for cost + dev; use **cloud for scale/off-site cron** only when it becomes the bottleneck. Any cloud GPU is only needed at scale — don't commit SaaS GPU spend now.
 ---
 ## 8. FLoci / cPanel Alternatives — Clarification
 - **cPanel** is the admin panel on the old VPS. Not needed locally — `docker compose` + NixOS + Caddy replaces it.
 - **FLoci** = local emulators of AWS/Azure/GCP for **offline testing** (no cloud, no credentials) on `localhost:4566`. Useful for:
  - developing/test the AWS S3/Glacier backup scripts without touching real AWS
  - testing SES/API integration offline
  Use it as a **dev/CI tool**, not production hosting. (Good for the CMS migration project too.)
 ---
 ## 9. Offsite Backup Design (AWS-only) — Cost Model in AUD
 ### 9.1 Target layout
 - **S3 bucket** `sam-offsite-backup`
  - **S3 Standard** (hot, <30 days): recent site data, DB dumps
  - **Lifecycle rule** → move to **Glacier Deep Archive** after ~30 days (new cold data: old site media, Immich photos, archives)
 - **Tooling:** keep existing **Borg** for local, add **rclone** (or restic) → AWS (S3 API). `rclone` is free, already skill-aligned with your backup pipeline.
 ### 9.2 Cost @ approx AUD (2026, incl. ~0.10 USD/AUD)
 | Tier | AUD/GB-mo | 100 GB/mo | 500 GB/mo |
 |---|---|---|---|
 | S3 Standard (hot) | ~0.036 | **3.60** | 18.00 |
 | Glacier Flexible (backup) | ~0.005 | 0.50 | 2.80 |
 | **Glacier Deep Archive (cold)** | **~0.0015** | 0.15 | **0.75** |
 | Backblaze B2 (ref) | ~0.0095 | 0.95 | 4.75 |
 | Hetzner Storage Box (ref, 1 TB flat) | ~6.00 flat | 6.00 | 6.00 |
 **Bottom line:** 
 - **~472 G Immich photos + old site media → Glacier DA ≈ 0.75–1.2 AUD/mo.**
 - Recent site data → S3 Standard ≈ a few AUD/mo.
 - **Expected offsite = ~2–6 AUD/mo.** Total infra (email+backup+compute) well under the 20–50 budget.
 ### 9.3 Restore/egress note
 - Glacier DA restores are slower (hours) but rare — fine for burn-down recovery.
 - Get data out costs ~0.14 AUD/GB on AWS; B2/Hetzner cheaper (free to 3×), but since you chose **AWS-first for skill**, restore volume is low → acceptable.
 ---
 ## 10. Backup of local systems (disaster / burn-down)
 - Currently: Borg (local USB 3T on `.13`), Kopia/Restic (`.35`→`.23` NFS). **This offsite AWS bucket covers the burn-down risk.**
 - **Gap to address in this plan:** the **Immich photos (~472 G on `.13` archive)** and **My-Passport `/mnt/hd`** need to be in the AWS offsite (lifecycle → Glacier DA).
 - **Optional:** also add the `.23` file-server no-backup gap, but it's a redundancy target for other repos — lower priority.
 ---
 ## 11. Networking / External Exposure (unchanged, verified)
 - A-record → static IP → router port-forward 80/443 → Caddy `.35` → `.13`.
 - Keep this. Works for SSH-git (Gitea), Caddy ACME TLS, and public sites.
 - **Residential IP port-80/443 works today** for sites; **do not** use residential IP as *mail* MX (blacklist risk) — inbound via provider, outbound via SES.
 - Email doesn't require inbound ports on your IP.
 ---
 ## 12. Phased Migration Plan (Infra first; CMS elsewhere)
 ### Phase 0 — Pre-flight (read-only, no changes)
 - [ ] Confirm AWS account exists / create `ap-southeast-2` regional setup
 - [ ] Confirm total media size; refine cost model
 - [ ] Verify `.13` can reach AWS (no proxy issues)
 ### Phase 1 — Offsite backup (quick win)
 - [ ] Create initial AWS account + IAM user (least privilege, `s3:PutObject/GetObject` on backup bucket only)
 - [ ] Configure `rclone` remote (or restic) against the bucket
 - [ ] Enact Borg → rclone → AWS S3 pipeline on `.13`
 - [ ] Add S3 lifecycle rule S3→Glacier DA after 30 days
 - [ ] **First offsite seed** (photos + site data) — verify restore in test
 ### Phase 2 — Email
 - [ ] Pick inbound provider (Zoho or MXroute) → set up mailboxes + MX
 - [ ] Set up Amazon SES outbound + verify domain, SPF/DKIM/DMARC, exit sandbox
 - [ ] Point site forms / notifications → SES
 ### Phase 3 — Target infra prep (on `.13`), no CMS migration
 - [ ] Confirm AMD RX 6600 installed + `amdgpu` driver on `.13`
 - [ ] Prepare Docker-compose volumes / directories for incoming CMS sites
 - [ ] Verify Caddy routes; plan reverse_proxy entries for new sites
 ### Phase 4 — (separate project) CMS migration
 - Not in scope here.
 ### Phase 5 — Decommission
 - [ ] Retire InMotion VPS (share of ~100/mo) once sites are live locally
 - [ ] Point final DNS A-records fully to static IP
 ---
 ## 13. Risks & Open Items
 | Risk | Impact | Mitigation |
 |---|---|---|
 | Residential IP public services | moderate | Keep Caddy ACME working; proxy sites; don't use MAIL MX on residential IP |
 | AWS egress on restore | low | Cold-tier restores rare; budget it; test restore path in Phase 1 |
 | CMS GPU-media scale | low | Local AMD on `.13` now; AWS MediaConvert/Lambda later only if bottleneck |
 | `.23` no backup | low-med | Downgrade constant; add to AWS offsite if desired (Phase 1 optional) |
 | Immich/My-Passport single-drive risk | **concern** | MUST be in AWS offsite (Glacier DA) — Phase 1 first seed |
 | cPanel dependency | low | Fully replaced by Docker/NixOS/Caddy |
 ---
 ## 14. Summary of Spend / Skill
 | Item | Est AUD/mo |
 |---|---|
 | Retired InMotion VPS | −100 (saving) |
 | Email (inbound free-tier + SES low volume) | ~0–6 |
 | AWS offsite backup (sites + photos, Glacier DA) | ~2–6 |
 | Optional small compute (if GPU moved to cloud) | ~0–5 |
 | **Total new infra** | **~5–15 AUD/mo** |
 | **Net saving vs today** | **~50–80 AUD/mo** |
 | **Skill wins** | AWS S3 + lifecycle + IAM + SES, Terraform/CLI-ready, rclone/Borg, Docker/NixOS/Caddy — all employable |
 ---
 *Created 2026-08-28. Source context: [[Local Web Host]], [[Email Backup off site hosting]], [[AWS Cloud Emulator Floci]], backup/network/drive notes.*