Google Cloud Images Immich

Photo Pipeline — migrate Google Photos to a private archive, serve via Immich, back up offsite. A Prefect-orchestrated pipeline that downloads Google Takeout exports, fingerprints + dedups every image, applies quality analysis (blurry/dark), flows through a human review dashboard, merges approved photos into a date-sorted master archive (by_date), imports them into Immich, and protects everything with layered backups (Borg, Kopia, AWS S3).

🗺️ The pipeline (Mermaid)

flowchart LR
    TO[Google Takeout<br/>134GB / 14 zips] -->|drop manifest| W[photo-watch<br/>every 15 min]
    W -->|download + extract| I[photo-ingest<br/>sha256 + dhash dedup]
    I -->|fingerprints| Q[quality-scan<br/>PIL blurry/dark]
    Q -->|verdicts| R[Review dashboard<br/>photo-filter.home.lab]
    R -->|approved| M[merge_orphans<br/>EXIF date routing]
    M --> B[(by_date master<br/>35K files)]
    B -->|incremental import| IM[Immich<br/>.35 library]
    R -->|rejected| T[(trash → purge)]
    B -.->|offsite| S3[AWS S3<br/>Glacier lifecycle]

Explore the interactive map: https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/photo-pipeline-map.html

What it is

A self-hosted Google Photos replacement pipeline. Photos leave Google via Takeout, are verified + deduplicated + quality-checked locally, reviewed by a human, then land in a date-sorted master archive (the single source of truth) and are served through Immich (a self-hosted photo server). The goal is full independence from Google + a robust local + offsite backup story.

Why it matters

Content-based dedup by sha256 — re-importing never doubles photos
Quality verdicts are deterministic (PIL image math, not ML) — reliable at 35K+ scale
Human gate — nothing merges until a person approves it
Incremental import — only new photos go to Immich (no 376GB re-uploads)
Safe cleanup — rejected → trash (recoverable) → purge only on confirmation
PHOTOS-TO-DELETE album — remove unwanted photos from Immich + by_date in one action

How to use it

Reviewing photos (main daily task)

Open https://photo-filter.home.lab (basic-auth: sam)
Filter: ?source=takeout&status=keep (new Google photos) or ?status=delete_candidate (blurry/dark)
Bulk Keep (approve) or Bulk Reject — decisions save instantly to the DB
Approved photos are ready for merge; rejected sit in trash (recoverable)

Pipeline actions (dashboard buttons)

Go to the "Pipeline actions" page in the dashboard:

Button	What it does	Safe?
🔍 Dry-run merge	Preview what approved photos WOULD move	✅ read-only
▶ Merge approved → by_date	Move approved files (EXIF-date routed)	Confirmation required
📤 Import by_date → Immich	Upload only NEW photos (dedup-safe)	✅ incremental
🗑 Rejected preview / trash / purge	Move rejected → trash → permanent delete	Trash recoverable
🔄 Check PHOTOS-TO-DELETE album	See what's in the removal album	✅ read-only
🗑 Remove album photos	Trash by_date copies + delete from Immich	Confirmation required

Removing unwanted photos (Immich side)

In Immich, move photos you don't want into the PHOTOS-TO-DELETE album
Dashboard → Pipeline → Check PHOTOS-TO-DELETE
Remove album photos → by_date copies go to trash (recoverable), Immich copies are deleted

Adding NEW Google Takeout in future

Download zips to incoming/ (or drop a manifest of URLs)
photo-watch auto-processes every 15 min → extract → ingest → quality → review
Review → merge → import → then you can delete from Google

🛠️Tools used (resume showcase)

Layer	Technology	Role
Orchestration	Prefect 3	Flows, scheduled deploys, worker lifecycle, crash recovery
Backend	Python, FastAPI, SQLite	Review dashboard, dedup, quality scan, incremental import
Image analysis	PIL/Pillow	Blurry/dark verdicts (luminance + edge variance at 300px)
Frontend	htmx, Jinja2, JS	Dashboard with live polling, batch actions, confirm dialogs
Data stores	SQLite (pipeline), Postgres (Immich)	Fingerprints + assets/albums
Import	Immich server-bundled CLI	Checksum-dedup'd upload
Backup	Borg, Kopia, AWS S3 (+ rclone/aws cli)	Local chains + offsite
Diagrams	Mermaid, Archify (published to maps.lab)	Architecture visualization
Ops/infra	NixOS, Docker, Caddy, Gitea, Vikunja, Outline	Full self-hosted stack

Demonstrated strengths: content-addressed dedup, incremental import design, data-safety discipline (backup-verify-before-merge), image-quality analysis at scale, full-stack dashboard with background jobs, cross-machine SSH automation, backup failure detection + repair.

Where things are

What	Where
Master archive (`by_date`)	`.13:/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` — 35K files, YYYY/MM-Mon
Pipeline code	Gitea: `ssh://gitea.lab.audasmedia.com.au:2222/sam/photo-pipeline.git`
Review dashboard	https://photo-filter.home.lab (`.13:8092`) — basic-auth
Prefect UI	https://prefect.home.lab (`.13:4200`)
Immich	`http://192.168.20.35:2283` — self-hosted photo server
Interactive map	https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/photo-pipeline-map.html
Fingerprint DB	`.13:/mnt/data/photo_pipeline.db` (SQLite)
Takeout staging	`.13:/mnt/data/takeout/` — transient downloads/extract
Removed photos trash	`.13:/mnt/data/takeout/removed/` + `rejected-trash/` — recoverable
Vikunja (tasks)	Project "Photo Pipeline & Archive" — kanban of remaining work
Outline (docs)	Collection "Photo Pipeline & Archive" — Overview / Architecture / Operations / Storage

Current status (Sep 2026)

Item	State
Google Takeout download	✅ Complete (134GB / 14 zips, sha256-verified)
Extract + ingest	✅ 12.3K new photos fingerprinted, dedup-skipped 225
Quality scan	✅ Verdicts applied (PIL)
Review	🔄 2,246 approved · 8,877 keep to review · 1,203 rejected
Merge + import to Immich	✅ Built + tested (incremental, dedup-safe)
Rejected cleanup	✅ Trash → purge flow working
PHOTOS-TO-DELETE removal	✅ Working (album → remove from Immich + by_date)
S3 offsite backup	⏳ Pending seed (by_date + archive/rest ≈ 712GB)
Delete from Google (final)	⏳ After local verification complete

⚙️ Config notes (for future maintenance)

SSH .13 → .35 uses key: -i ~/.ssh/id_ed25519_rsync
Dashboard service (on .13): systemctl --user restart photo-dashboard (users must refresh browser after code changes — stale JS causes "buttons do nothing")
Prefect worker: systemctl --user status prefect-worker (runs photo-pool)
Backup job: borgbackup-job-archive-to-23 (daily 06:00, .13 → .23) — covers archive/ and ~/photo-pipeline (DB)
The 3TB archive drive is USB — got unplugged once (Aug 21-26 2026 → backup failures); replugging resolved. Remount if /mnt/ubuntu_storage_3TB missing: the disk is sde1 (UUID 037a542c-…), sudo mount /mnt/ubuntu_storage_3TB
Immich API key lives at ~/.ssh/../photo-pipeline/.immich-key (0600) — needs user.read + asset.read + asset.upload; skip server.about scope (not granted)
382 HEIC files skip the PIL quality scan (no HEIF plugin) — safe on disk, handle with pillow-heif later
Known gap: phone-only photos uploaded directly to Immich are NOT in by_date and NOT backed up — decision deferred
Gitea repo pushes use SSH port 2222; .13 SSH can be intermittently flaky — retry after 30s if a push times out

Resources

Interactive map: https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/photo-pipeline-map.html
Photo Project Master Overview — deep-dive human reference (storage map, backup S3 plan, tools table)
Backup Architecture — Offsite to AWS S3 — the S3 offsite plan
Filesystem Drive Map — disk layout
Vikunja: project Photo Pipeline & Archive · Outline: collection Photo Pipeline & Archive

8.5 KiB Raw Blame History