sam-4screen-desktop 2026-8-28:18:54:8

300 areas/360 Dev-Ops Network Computers/Photo Pipeline & Storage Map.md

@@ -0,0 +1,77 @@
+---
+tags: [photos, immich, pipeline, backup, google-takeout]
+created: 2026-08-27
+---
+
+# Photo Pipeline & Storage Map
+
+## What the pipeline does
+
+```
+Google Photos (Takeout export, 134GB/14 zips)
+        │  download → /mnt/data/takeout/incoming/ (.13)
+        ▼
+Extract → /mnt/data/takeout/takeout-*/ (36,937 files)
+        │
+        ▼
+Ingest (photo-ingest flow) → fingerprint sha256+dhash → dedup vs by_date
+        │  12,326 new / 225 dups skipped / 272 HEIC deferred
+        ▼
+Quality scan (PIL verdicts) → keep / delete_candidate (dark/blurry)
+        │  11,877 keep / 449 delete_candidate
+        ▼
+Dashboard review (http://192.168.20.13:8092/review) → approve/reject
+        │
+        ▼
+merge_orphans (approved only) → by_date/sorted_pictures_holder/YYYY/MM-Mon
+        │
+        ▼
+Immich import (checksum dedup — no doubles)
+        │
+        ▼
+ONLY NOW: delete from Google Photos → frees storage
+```
+
+## Storage locations (where the files live)
+
+| Path | Machine | Size | What it is |
+|---|---|---|---|
+| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` | .13 | 373 GB | **MASTER** photo archive, sorted YYYY/MM-Mon |
+| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` | .13 | 69 GB | Backup mirror by subject |
+| `/mnt/ubuntu_storage_3TB/archive/03_photos/{temp_transfer,Pictures,phone_dumps}` | .13 | 31 GB | Orphan folders awaiting review/merge |
+| `/mnt/hd/immich/` | .35 | ~1.1T (60% of disk) | Immich store: library/thumbs/encoded-video/upload |
+| `/mnt/data/takeout/` | .13 | 269 GB | **TRANSIENT** Takeout downloads+extract (do NOT back up) |
+| `/mnt/data/photo_pipeline.db` | .13 | — | SQLite: image_hashes, known_paths, takeout_archives |
+
+## Key facts
+
+- **by_date is the single master.** Everything else is derived or transient.
+- **Immich is a derived copy** of by_date (re-importable). Its thumbs/encoded-video are regenerable.
+- **Takeout downloads (269GB) are transient** — safe to delete after verification.
+- **Dedup is content-based** (sha256) — re-importing by_date never doubles.
+- 382 HEIC files lack PIL support (skipped by quality scan) — safe on disk, need pillow-heif or conversion.
+
+## Backup to S3 (offsite)
+
+**Back up (the "house burns down" set):**
+- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` (373 GB) — master
+- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` (69 GB) — mirror
+- ✅ Optionally: Immich `library/` on .35 (catches phone-only uploads)
+
+**SKIP (transient/regenerable):**
+- ❌ `/mnt/data/takeout/` (269 GB — transient downloads)
+- ❌ Immich `thumbs/`, `encoded-video/` (regenerable)
+- ❌ The 14 zip archives (they're in Google until deleted)
+
+## Machine map
+
+- **.13** nixos-desktop — pipeline, dashboard :8092, Prefect :4200, n8n
+- **.35** — Immich :2283, Caddy (prefect/photo-filter), pi-hole; SSH needs `-i ~/.ssh/id_ed25519_rsync`
+- **.23** — borg backup server (2.7T USB via CIFS to .35)
+- **.27** — this dev box
+
+## Review dashboard
+
+- URL: http://192.168.20.13:8092/review
+- Filters: `?source=takeout`, `?status=delete_candidate` (flagged), `?status=keep` (clean)
+- Bulk: select-all → Bulk Keep (approve) or Bulk Reject