sam-4screen-desktop 2026-8-28:20:24:8

.obsidian/workspace.json

@@ -211,6 +211,7 @@
   },
   "active": "8018aa52f0591df9",
   "lastOpenFiles": [
+    "300 areas/360 Dev-Ops Network Computers/Photo Project Master Overview.md",
     "300 areas/360 Dev-Ops Network Computers/FAMILY CONSOLE.md",
     "000 daily/FAMILY  S3 Storage Integration & Blueprint.md",
     "000 daily/FAMILY DeepSeek Harness (dsh) Home Lab Setup.md",
@@ -236,7 +237,6 @@
     "300 areas/360 Dev-Ops Network Computers/Websites on Nixos-Dekstop 13.md",
     "300 areas/360 Dev-Ops Network Computers/Docker Containers.md",
     "300 areas/360 Dev-Ops Network Computers/Pi Neovim Coding Harness.md",
-    "300 areas/360 Dev-Ops Network Computers/Pi MCP.md",
     "000 daily/ThinkPad Recommendations",
     "500 archive/510 Daily",
     "300 areas/305 Ideas Businesses",

200 projects/220 Web Host Migration/Migration Work Log.md

@@ -59,6 +59,10 @@ id: 1848129023-LOG
 | P1 | **Deep Archive end-to-end probe PASSED** on `.13`: sync with `--storage-class DEEP_ARCHIVE` → `head-object` reported `DEEP_ARCHIVE`; probe cleaned. | verified |
 | P1 | Seed script drafted (`offsite-seed.sh`): photos→DA, archive/rest→DA, borg/.27→STANDARD | pending |
 | P1 | **Awaiting final go** to launch ~62 h full seed | user decision |
+| P1 | **Seed schedule decided (user):** nightly **23:00–05:00 Melbourne local** on `.13` (off-peak, avoids NBN/ABB peak shaping). Resumable/incremental, append-only. | user choice |
+| P1 | **NixOS module `offsite.nix` added** (separate file; does NOT touch `backup.nix`): adds `awscli2` to systemPackages, `offsite-sync.service` (Type=exec, RuntimeMaxSec=6h) + `offsite-sync.timer` (daily 23:00). "Fixed" initial Type=oneshot bug (RuntimeMaxSec ignored) -> Type=exec. | flake check passed; rebuilt |
+| P1 | **Rebuilt `.13`**: AWS CLI v2.35.11 on PATH; service Type=exec RuntimeMaxSec=6h; timer active (next fire 23:00 tonight). | verified |
+| P1 | **First seed scheduled** to auto-start 23:00 tonight. | armed |
 
 ## Pending next action
 - **Build:** create bucket `sam-offsite-backup` + lifecycle (S3→Glacier Flexible@30d→Glacier DA@90d), then wire local Borg/rclone→S3 on `.13`, seed, test restore.

300 areas/360 Dev-Ops Network Computers/Photo Project Master Overview.md

@@ -0,0 +1,202 @@
+---
+tags: [photos, immich, pipeline, project-overview, google-takeout, backup]
+created: 2026-08-27
+aliases: [photo-pipeline, photo-project]
+---
+
+# 📸 Photo Project — Master Overview
+
+> Single source of truth for the photo consolidation + Google Takeout + Immich + backup effort.
+> Status: **Takeout downloaded & ingested; 11,269 photos awaiting review in dashboard.**
+
+---
+
+## 🗺️ What this project is
+
+Downloading everything from **Google Photos** (via Takeout), merging it with the local archive (**by_date**), importing to **Immich** (self-hosted photo server), and backing it up **offsite to AWS S3**. Then freeing Google storage by deleting (only after verified locally).
+
+```
+Google Photos (Takeout, 134GB/14 zips)
+    ↓ download → .13:/mnt/data/takeout/incoming/
+Extract (36,937 files)
+    ↓
+Ingest (fingerprint sha256+dhash, dedup vs by_date) → 12,326 new
+    ↓
+Quality scan (PIL: blurry/dark) → 11,877 keep / 449 flagged
+    ↓
+DASHBOARD REVIEW (you) → approve / reject
+    ↓
+merge_orphans (approved only) → by_date/YYYY/MM-Mon
+    ↓
+Immich import (checksum dedup = no doubles)
+    ↓
+ONLY NOW: delete from Google → frees storage
+```
+
+---
+
+## 🏗️ How it's built (architecture — resume-facing)
+
+### The stack (honest)
+
+| Layer | Technology | Why |
+|---|---|---|
+| **Orchestration** | **Prefect 3** (flow framework) | Scheduled, retryable, observable pipelines. Flows run on a systemd worker (photo-pool) |
+| **Fingerprinting/dedup** | Python + SQLite (sha256 + dhash) | Content-based dedup — re-importing never doubles. dhash for fast near-dup checks |
+| **Quality analysis** | **PIL/Pillow** (image math) | Blurry (edge variance < 200) + dark (mean luminance < 40) at 300px downscale. Deterministic, no ML |
+| **Review UI** | **FastAPI** + Jinja2 templates | Dashboard at :8092 — batch approve/reject, pagination, lazy thumbnails (200/page) |
+| **Import target** | **Immich** (self-hosted) + its server-bundled CLI | Checksum dedup server-side = safe re-imports |
+| **Data store** | SQLite (`photo_pipeline.db`) | image_hashes, known_paths, takeout_archives, batches |
+| **Storage** | ext4 on USB 3TB + NixOS mounts | Master archive + pipeline staging |
+| **Deployment** | NixOS systemd units (worker, dashboard, borg) | Declarative config in /etc/nixos |
+| **Not used (yet)** | LangChain/LangGraph, ML models | Quality scan is deterministic math by design; AI tagging is a future candidate |
+
+### Flow diagram
+
+```
+Prefect (photo-pool worker)
+├── takeout-fetch     download zips → extract → register
+├── photo-ingest      walk dir → sha256+dhash → dedup vs known_paths
+├── photo-quality-scan  PIL verdicts → status keep/delete_candidate
+├── photo-watch       watches incoming/ every 15 min, triggers chain
+├── merge_orphans     approved → by_date (EXIF-date routed)
+└── immich-import     scp → docker cp → server CLI upload
+```
+
+### Why Prefect + PIL, not LangChain/LLM
+
+- **Deterministic by design**: a pipeline that moves 12K irreplaceable photos must do exactly the same thing every run. Flows don't improvise.
+- **Resume note**: this is a showcase of **Prefect workflow engineering, dedup algorithms, image-quality analysis, and self-hosted infra** — deliberately NOT an LLM app. (LangChain/LangGraph exist in the homelab as the DSH agent platform — separate concern.)
+
+### What this demonstrates (resume bullets)
+
+- Designed a **content-addressed dedup pipeline** (sha256) over 35K+ photos — zero data loss, zero doubles
+- **Prefect 3** deployment: flow orchestration, systemd worker, scheduled deployments, crash recovery
+- **Image quality analysis** with PIL: calibrated thresholds, downscale + edge/luminance metrics
+- **FastAPI review dashboard**: batch operations, lazy loading, DB-backed decisions
+- **Full data-safety discipline**: backup verified before merge, checksum dedup before Immich import, verify before delete
+- **Self-hosted infra**: NixOS, Immich, Caddy reverse-proxy, SSH key management, disk/mount troubleshooting
+
+---
+
+## 🔗 URLs & Services
+
+| Service | URL | IP:Port | Notes |
+|---|---|---|---|
+| **Photo review dashboard** | http://photo-filter.home.lab | 192.168.20.13:8092 | Main tool — review/approve photos |
+| **Dashboard (direct)** | http://192.168.20.13:8092 | — | Works without DNS |
+| **Prefect** (pipeline orchestrator) | http://prefect.home.lab | 192.168.20.13:4200 | Flow runs, schedules |
+| **Immich** (photo server) | http://192.168.20.35:2283 | 192.168.20.35:2283 | The destination library |
+| **Langfuse** | http://langfuse.home.lab | 192.168.20.13:3001 | LLM observability |
+| **Supabase** | http://supabase.home.lab | 192.168.20.27:8001 | Postgres backend |
+| **DSH agents** (Langchain-ish) | dsh-sam/jo/harry/finn.lab.audasmedia.com.au | .13:3081-3084 | DeepSeek Harness per-user agents |
+
+---
+
+## 💾 Where the files live
+
+| Path | Machine | Size | What |
+|---|---|---|---|
+| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` | **.13** | 373 GB | ⭐ **MASTER** — sorted YYYY/MM-Mon |
+| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` | .13 | 69 GB | Mirror by subject (duplicate content) |
+| `/mnt/ubuntu_storage_3TB/archive/03_photos/{temp_transfer,Pictures,phone_dumps}` | .13 | 31 GB | Orphans awaiting review |
+| `/mnt/hd/immich/` | **.35** | ~1.1T | Immich store (library/thumbs/encoded-video) |
+| `/mnt/data/takeout/` | .13 | 269 GB | **TRANSIENT** — downloaded Takeout, delete after verify |
+| `/mnt/data/photo_pipeline.db` | .13 | — | SQLite — image_hashes, known_paths, takeout_archives |
+| `~/photo-pipeline/` | .13 | — | Code repo (gitea: photo-pipeline.git) |
+
+---
+
+## 📊 Current Progress (verified 2026-08-27)
+
+### ✅ Done
+1. **Takeout downloaded** — 14 archives, 134 GB, sha256-verified byte-for-byte
+2. **Extracted** — 36,937 files
+3. **Ingested** — 12,326 new photos (225 dups skipped, 272 HEIC deferred)
+4. **Quality scanned** — verdicts assigned
+5. **DB reconciled** — by_date 35,457 files = DB + known_paths ✓
+
+### ⏳ In progress
+6. **Dashboard review** — **11,269 takeout photos still `keep`** (awaiting approval)
+   - 459 approved, 598 rejected so far
+   - 200 per page → **~57 pages of takeout keep** left
+   - Plus older orphans still unreviewed
+
+### 🔜 Pending
+7. **Merge approved → by_date**
+8. **Import to Immich**
+9. **S3 offsite backup** (~550 GB: by_date + archive/rest + .27 borg)
+10. **Delete from Google** (only after 1-9 verified) → frees 17 GB storage
+
+---
+
+## ⚠️ Unbacked phone photos (known gap)
+
+**Photos uploaded directly from phone → Immich app live only on `.35:/mnt/hd/immich/library`.**
+
+- ❌ NOT in by_date (never went through the pipeline)
+- ❌ NOT in any backup (Kopia skips /mnt/hd; borg only covers .13 archive)
+- **If .35's disk dies, these are lost.**
+- **Decision (2026-08-27): deferred** — not backing up now. Phone still has originals until cleaned.
+
+**If this changes**: back up `library/` folder from .35 to S3 (small), or import them into by_date via the pipeline.
+
+---
+
+## 🛠️ How to use (operations)
+
+### Reviewing photos (the main daily task)
+1. Open **http://photo-filter.home.lab** (or http://192.168.20.13:8092)
+2. Filter: `?source=takeout` (new Google Photos) · `?status=delete_candidate` (flagged dark/blurry) · `?status=keep` (clean)
+3. **Bulk approve**: select-all → **Bulk Keep**. **Reject**: select → Bulk Reject
+4. Decisions save instantly to DB; approved = ready to merge
+
+### Running the merge (after approving a batch)
+```bash
+ssh sam@192.168.20.13
+cd ~/photo-pipeline
+./run-python.sh merge_orphans.py           # dry-run (safe)
+./run-python.sh merge_orphans.py --apply   # real move approved → by_date
+```
+
+### Importing to Immich (after merge)
+```bash
+ssh sam@192.168.20.13
+cd ~/photo-pipeline
+./run-python.sh -c "import immich_import as ii; ii.upload_to_immich.fn('/mnt/ubuntu_storage_3TB/archive/03_photos/by_date')"
+# Immich reports "N new / M duplicates" — dups are skipped, never doubled
+```
+
+### Monitoring
+- **Prefect UI**: http://prefect.home.lab (flow runs, failures)
+- **Worker service**: `systemctl --user status prefect-worker` on .13
+- **Backup health**: daily borg → .23; check with `borg list` (see [[Backup Health Check Commands]])
+
+### New Google Takeout in future
+1. Download zips to `.13:/mnt/data/takeout/incoming/`
+2. photo-watch picks them up every 15 min → extract → ingest → quality → dashboard
+3. Review → merge → Immich → then delete from Google
+
+### Getting the code
+```bash
+git clone ssh://gitea.lab.audasmedia.com.au:2222/sam/photo-pipeline.git
+```
+
+---
+
+## 🔑 Key facts
+
+- **by_date is the single master** — everything else derived/transient
+- **Immich = derived copy** of by_date (re-importable); its DB (albums/faces) IS backed up by Kopia
+- **Dedup is content-based** (sha256) — never doubles
+- **382 HEIC files** lack PIL support — skipped, safe on disk, need pillow-heif later
+- **Backup chain**: borg (.13→.23) daily + Kopia (.35) daily + (planned) S3 offsite
+- **SSH .13→.35** needs `-i ~/.ssh/id_ed25519_rsync`
+- The 3TB archive drive is **USB** — unplugged once (Aug 21-26) caused backup failures
+
+---
+
+## Related notes
+- [[Backup Architecture — Offsite to AWS S3]] — the S3 plan (bucket, IAM, costs)
+- [[Filesystem Drive Map]] — disk layout
+- [[Photo Consolidation Plan]] — superseded (May 2026)