Single source of truth for the photo consolidation + Google Takeout + Immich + backup effort.
Status: Takeout downloaded & ingested; 11,269 photos awaiting review in dashboard.
🗺️ What this project is
Downloading everything from Google Photos (via Takeout), merging it with the local archive (by_date), importing to Immich (self-hosted photo server), and backing it up offsite to AWS S3. Then freeing Google storage by deleting (only after verified locally).
Google Photos (Takeout, 134GB/14 zips)
↓ download → .13:/mnt/data/takeout/incoming/
Extract (36,937 files)
↓
Ingest (fingerprint sha256+dhash, dedup vs by_date) → 12,326 new
↓
Quality scan (PIL: blurry/dark) → 11,877 keep / 449 flagged
↓
DASHBOARD REVIEW (you) → approve / reject
↓
merge_orphans (approved only) → by_date/YYYY/MM-Mon
↓
Immich import (checksum dedup = no doubles)
↓
ONLY NOW: delete from Google → frees storage
🏗️ How it's built (architecture — resume-facing)
The stack (honest)
Layer
Technology
Why
Orchestration
Prefect 3 (flow framework)
Scheduled, retryable, observable pipelines. Flows run on a systemd worker (photo-pool)
Fingerprinting/dedup
Python + SQLite (sha256 + dhash)
Content-based dedup — re-importing never doubles. dhash for fast near-dup checks
Quality analysis
PIL/Pillow (image math)
Blurry (edge variance < 200) + dark (mean luminance < 40) at 300px downscale. Deterministic, no ML
Review UI
FastAPI + Jinja2 templates
Dashboard at :8092 — batch approve/reject, pagination, lazy thumbnails (200/page)
Quality scan is deterministic math by design; AI tagging is a future candidate
Flow diagram
Prefect (photo-pool worker)
├── takeout-fetch download zips → extract → register
├── photo-ingest walk dir → sha256+dhash → dedup vs known_paths
├── photo-quality-scan PIL verdicts → status keep/delete_candidate
├── photo-watch watches incoming/ every 15 min, triggers chain
├── merge_orphans approved → by_date (EXIF-date routed)
└── immich-import scp → docker cp → server CLI upload
Why Prefect + PIL, not LangChain/LLM
Deterministic by design: a pipeline that moves 12K irreplaceable photos must do exactly the same thing every run. Flows don't improvise.
Resume note: this is a showcase of Prefect workflow engineering, dedup algorithms, image-quality analysis, and self-hosted infra — deliberately NOT an LLM app. (LangChain/LangGraph exist in the homelab as the DSH agent platform — separate concern.)
What this demonstrates (resume bullets)
Designed a content-addressed dedup pipeline (sha256) over 35K+ photos — zero data loss, zero doubles
Decisions save instantly to DB; approved = ready to merge
Running the merge (after approving a batch)
ssh sam@192.168.20.13
cd ~/photo-pipeline
./run-python.sh merge_orphans.py # dry-run (safe)
./run-python.sh merge_orphans.py --apply # real move approved → by_date
Importing to Immich (after merge)
ssh sam@192.168.20.13
cd ~/photo-pipeline
./run-python.sh -c "import immich_import as ii; ii.upload_to_immich.fn('/mnt/ubuntu_storage_3TB/archive/03_photos/by_date')"# Immich reports "N new / M duplicates" — dups are skipped, never doubled
