--- type: area status: active tags: [photos, immich, pipeline, project-overview, google-takeout, backup] created: 2026-08-27 modified: 2026-09-06 aliases: [photo-pipeline, photo-project] --- # πŸ“Έ Photo Project β€” Master Overview > Single source of truth for the photo consolidation + Google Takeout + Immich + backup effort. > Status: **Takeout downloaded & ingested; review in progress; merge+import built & tested.** ## πŸ“‹ Project tracking & docs (AI + human) | Tool | What | Link/How to access | |---|---|---| | **Vikunja** | Live kanban β€” tasks/stages/progress | Project **"Photo Pipeline & Archive"** (To Do / In Progress / Done) | | **Outline** | Project docs for AI agents | Collection **"Photo Pipeline & Archive"** (Overview / Architecture / Operations / Storage & Backup) | | **Obsidian** | This note β€” human reading + key configs | 300 areas/360 Dev-Ops | > Tasks, architecture and ops live in Vikunja + Outline. This note is the quick human reference. --- ## πŸ—ΊοΈ What this project is Downloading everything from **Google Photos** (via Takeout), merging it with the local archive (**by_date**), importing to **Immich** (self-hosted photo server), and backing it up **offsite to AWS S3**. Then freeing Google storage by deleting (only after verified locally). ``` Google Photos (Takeout, 134GB/14 zips) ↓ download β†’ .13:/mnt/data/takeout/incoming/ Extract (36,937 files) ↓ Ingest (fingerprint sha256+dhash, dedup vs by_date) β†’ 12,326 new ↓ Quality scan (PIL: blurry/dark) β†’ 11,877 keep / 449 flagged ↓ DASHBOARD REVIEW (you) β†’ approve / reject ↓ merge_orphans (approved only) β†’ by_date/YYYY/MM-Mon ↓ Immich import (checksum dedup = no doubles) ↓ ONLY NOW: delete from Google β†’ frees storage ``` --- ## πŸ—οΈ How it's built (architecture β€” resume-facing) ### The stack (honest) | Layer | Technology | Why | |---|---|---| | **Orchestration** | **Prefect 3** (flow framework) | Scheduled, retryable, observable pipelines. Flows run on a systemd worker (photo-pool) | | **Fingerprinting/dedup** | Python + SQLite (sha256 + dhash) | Content-based dedup β€” re-importing never doubles. dhash for fast near-dup checks | | **Quality analysis** | **PIL/Pillow** (image math) | Blurry (edge variance < 200) + dark (mean luminance < 40) at 300px downscale. Deterministic, no ML | | **Review UI** | **FastAPI** + Jinja2 templates | Dashboard at :8092 β€” batch approve/reject, pagination, lazy thumbnails (200/page) | | **Import target** | **Immich** (self-hosted) + its server-bundled CLI | Checksum dedup server-side = safe re-imports | | **Data store** | SQLite (`photo_pipeline.db`) | image_hashes, known_paths, takeout_archives, batches | | **Storage** | ext4 on USB 3TB + NixOS mounts | Master archive + pipeline staging | | **Deployment** | NixOS systemd units (worker, dashboard, borg) | Declarative config in /etc/nixos | | **Not used (yet)** | LangChain/LangGraph, ML models | Quality scan is deterministic math by design; AI tagging is a future candidate | ### Flow diagram ``` Prefect (photo-pool worker) β”œβ”€β”€ takeout-fetch download zips β†’ extract β†’ register β”œβ”€β”€ photo-ingest walk dir β†’ sha256+dhash β†’ dedup vs known_paths β”œβ”€β”€ photo-quality-scan PIL verdicts β†’ status keep/delete_candidate β”œβ”€β”€ photo-watch watches incoming/ every 15 min, triggers chain β”œβ”€β”€ merge_orphans approved β†’ by_date (EXIF-date routed) └── immich-import scp β†’ docker cp β†’ server CLI upload ``` ### Why Prefect + PIL, not LangChain/LLM - **Deterministic by design**: a pipeline that moves 12K irreplaceable photos must do exactly the same thing every run. Flows don't improvise. - **Resume note**: this is a showcase of **Prefect workflow engineering, dedup algorithms, image-quality analysis, and self-hosted infra** β€” deliberately NOT an LLM app. (LangChain/LangGraph exist in the homelab as the DSH agent platform β€” separate concern.) ### What this demonstrates (resume bullets) - Designed a **content-addressed dedup pipeline** (sha256) over 35K+ photos β€” zero data loss, zero doubles - **Prefect 3** deployment: flow orchestration, systemd worker, scheduled deployments, crash recovery - **Image quality analysis** with PIL: calibrated thresholds, downscale + edge/luminance metrics - **FastAPI review dashboard**: batch operations, lazy loading, DB-backed decisions - **Full data-safety discipline**: backup verified before merge, checksum dedup before Immich import, verify before delete - **Self-hosted infra**: NixOS, Immich, Caddy reverse-proxy, SSH key management, disk/mount troubleshooting --- ## πŸ”— URLs & Services | Service | URL | IP:Port | Notes | |---|---|---|---| | **Photo review dashboard** | http://photo-filter.home.lab | 192.168.20.13:8092 | Main tool β€” review/approve photos | | **Dashboard (direct)** | http://192.168.20.13:8092 | β€” | Works without DNS | | **Prefect** (pipeline orchestrator) | http://prefect.home.lab | 192.168.20.13:4200 | Flow runs, schedules | | **Immich** (photo server) | http://192.168.20.35:2283 | 192.168.20.35:2283 | The destination library | | **Langfuse** | http://langfuse.home.lab | 192.168.20.13:3001 | LLM observability | | **Supabase** | http://supabase.home.lab | 192.168.20.27:8001 | Postgres backend | | **DSH agents** (Langchain-ish) | dsh-sam/jo/harry/finn.lab.audasmedia.com.au | .13:3081-3084 | DeepSeek Harness per-user agents | --- ## πŸ’Ύ Where the files live | Path | Machine | Size | What | |---|---|---|---| | `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` | **.13** | 373 GB | ⭐ **MASTER** β€” sorted YYYY/MM-Mon | | `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` | .13 | 69 GB | Mirror by subject (duplicate content) | | `/mnt/ubuntu_storage_3TB/archive/03_photos/{temp_transfer,Pictures,phone_dumps}` | .13 | 31 GB | Orphans awaiting review | | `/mnt/hd/immich/` | **.35** | ~1.1T | Immich store (library/thumbs/encoded-video) | | `/mnt/data/takeout/` | .13 | 269 GB | **TRANSIENT** β€” downloaded Takeout, delete after verify | | `/mnt/data/photo_pipeline.db` | .13 | β€” | SQLite β€” image_hashes, known_paths, takeout_archives | | `~/photo-pipeline/` | .13 | β€” | Code repo (gitea: photo-pipeline.git) | --- ## πŸ“Š Current Progress (verified 2026-08-27) ### βœ… Done 1. **Takeout downloaded** β€” 14 archives, 134 GB, sha256-verified byte-for-byte 2. **Extracted** β€” 36,937 files 3. **Ingested** β€” 12,326 new photos (225 dups skipped, 272 HEIC deferred) 4. **Quality scanned** β€” verdicts assigned 5. **DB reconciled** β€” by_date 35,457 files = DB + known_paths βœ“ ### ⏳ In progress 6. **Dashboard review** β€” **11,269 takeout photos still `keep`** (awaiting approval) - 459 approved, 598 rejected so far - 200 per page β†’ **~57 pages of takeout keep** left - Plus older orphans still unreviewed ### πŸ”œ Pending 7. **Merge approved β†’ by_date** 8. **Import to Immich** 9. **S3 offsite backup** (~550 GB: by_date + archive/rest + .27 borg) 10. **Delete from Google** (only after 1-9 verified) β†’ frees 17 GB storage --- ## ⚠️ Unbacked phone photos (known gap) **Photos uploaded directly from phone β†’ Immich app live only on `.35:/mnt/hd/immich/library`.** - ❌ NOT in by_date (never went through the pipeline) - ❌ NOT in any backup (Kopia skips /mnt/hd; borg only covers .13 archive) - **If .35's disk dies, these are lost.** - **Decision (2026-08-27): deferred** β€” not backing up now. Phone still has originals until cleaned. **If this changes**: back up `library/` folder from .35 to S3 (small), or import them into by_date via the pipeline. --- ## πŸ› οΈ How to use (operations) ### Reviewing photos (the main daily task) 1. Open **http://photo-filter.home.lab** (or http://192.168.20.13:8092) 2. Filter: `?source=takeout` (new Google Photos) Β· `?status=delete_candidate` (flagged dark/blurry) Β· `?status=keep` (clean) 3. **Bulk approve**: select-all β†’ **Bulk Keep**. **Reject**: select β†’ Bulk Reject 4. Decisions save instantly to DB; approved = ready to merge ### Running the merge (after approving a batch) ```bash ssh sam@192.168.20.13 cd ~/photo-pipeline ./run-python.sh merge_orphans.py # dry-run (safe) ./run-python.sh merge_orphans.py --apply # real move approved β†’ by_date ``` ### Importing to Immich (after merge) ```bash ssh sam@192.168.20.13 cd ~/photo-pipeline ./run-python.sh -c "import immich_import as ii; ii.upload_to_immich.fn('/mnt/ubuntu_storage_3TB/archive/03_photos/by_date')" # Immich reports "N new / M duplicates" β€” dups are skipped, never doubled ``` ### Monitoring - **Prefect UI**: http://prefect.home.lab (flow runs, failures) - **Worker service**: `systemctl --user status prefect-worker` on .13 - **Backup health**: daily borg β†’ .23; check with `borg list` (see [[Backup Health Check Commands]]) ### New Google Takeout in future 1. Download zips to `.13:/mnt/data/takeout/incoming/` 2. photo-watch picks them up every 15 min β†’ extract β†’ ingest β†’ quality β†’ dashboard 3. Review β†’ merge β†’ Immich β†’ then delete from Google ### Getting the code ```bash git clone ssh://gitea.lab.audasmedia.com.au:2222/sam/photo-pipeline.git ``` --- ## πŸ”‘ Key facts - **by_date is the single master** β€” everything else derived/transient - **Immich = derived copy** of by_date (re-importable); its DB (albums/faces) IS backed up by Kopia - **Dedup is content-based** (sha256) β€” never doubles - **382 HEIC files** lack PIL support β€” skipped, safe on disk, need pillow-heif later - **Backup chain**: borg (.13β†’.23) daily + Kopia (.35) daily + (planned) S3 offsite - **SSH .13β†’.35** needs `-i ~/.ssh/id_ed25519_rsync` - The 3TB archive drive is **USB** β€” unplugged once (Aug 21-26) caused backup failures --- ## πŸ› οΈ Tools (employment reference) | Category | Tools used | Role | |---|---|---| | **Orchestration** | **Prefect 3** | Flow pipelines, scheduled deployments, worker lifecycle | | **Backend** | Python, FastAPI, SQLite | Review dashboard, dedup, quality scan | | **Image analysis** | PIL/Pillow | Blurry/dark verdicts (luminance/edge metrics) | | **Frontend** | htmx, Jinja2, HTML/CSS/JS | Dashboard UI with live polling | | **Self-hosted** | NixOS, Docker, Caddy, Immich, Gitea, Vikunja, Outline | Full homelab | | **Storage/backup** | Borg, Kopia, AWS S3, rclone/aws-cli | Backup layers + offsite | | **Databases** | SQLite (pipeline), Postgres (Immich) | Fingerprints + assets | | **Monitoring** | Prefect UI, systemd, journalctl, Prefect logs | Observability | **Demonstrated strengths**: content-addressed dedup, incremental import design, data-safety discipline (backup-verify-before-merge), image-quality analysis at scale, full-stack dashboard with background jobs, cross-machine SSH automation. --- ## Related notes - [[Backup Architecture β€” Offsite to AWS S3]] β€” the S3 plan (bucket, IAM, costs) - [[Filesystem Drive Map]] β€” disk layout - [[Photo Consolidation Plan]] β€” superseded (May 2026)