How To — photo-pipeline

What this system does

An automated photo ingestion pipeline: it downloads Google Takeout exports, fingerprints every image (deduplication), audits quality (blurry/dark/etc), and imports approved photos into your Immich library — with notifications and a review dashboard at every step.

Core safety rule: nothing is ever deleted automatically. Rejected photos go to a holding folder (03_delete) and stay there until you confirm.

The pipeline at a glance

Step	Flow	What happens
1. Export	— (manual)	You create a Google Takeout export
2. Feed	upload / scp	Drop manifest or archives into `incoming/`
3. Fetch	takeout-fetch	Downloads archives (resumable), tracks them, extracts
4. Ingest	photo-ingest	Hashes every file, flags exact + near duplicates
5. Quality	photo-quality-scan	CleanVision flags blurry/dark/odd images
6. Notify	Apprise	You get a message with counts + dashboard link
7. Review	this dashboard	You Keep or Reject images in the review queue
8. Stage	process-staging	Approved → 01_keep, Rejected → 03_delete
9. Import	immich-import	Approved photos uploaded to Immich

How to add photos — 4 ways

A. From Google Takeout (bulk)

Go to takeout.google.com and create an export (Photos, batched by year so each archive is <10GB). Google has no API for this — it's the one manual step.
When it's ready, copy the download URLs into a text file (one per line) — call it urls.txt.
Drop it in incoming/ — via the Upload page, or from a terminal: scp urls.txt sam@100.114.62.46:/mnt/data/takeout/incoming/ (Tailscale IP).
The watch flow picks it up within 15 minutes and runs the whole pipeline. You get an Apprise notification with results.

B. An already-downloaded archive

If you already have Takeout archives (.zip/.tgz), drop them in incoming/ the same way. The pipeline extracts and processes them.

C. A folder already on .13 (like your archive)

For folders already on the server (e.g. archive/03_photos), run the ingest flow directly — it scans in batches (1000 at a time) with checkpoints, so it's safe to interrupt:

export PREFECT_API_URL=http://localhost:4200/api
~/photo-pipeline/.venv/bin/prefect deployment run "photo-ingest/ingest" \
  --param base_dir=/mnt/ubuntu_storage_3TB/archive/03_photos/Pictures \
  --param source=archive-pictures --param batch_size=1000

D. From your phone

Install the Immich app on your phone and enable auto-backup — photos upload straight to your Immich library, bypassing Google entirely. This is the recommended phone path.

How to review

Open the Review queue. It shows 200 images per page, newest first.
Click a photo to see it full-size (opens in a new tab).
Keep = approved for Immich. Reject = moves to the delete-holding folder. ↺ = undo back to unscanned.
Use the checkbox + Select all + Bulk Keep/Reject for large batches.
Filter by status or source with the controls above the grid.
Track progress on the Pipeline page (auto-refreshes every 30s).

Remember: review decisions are instant DB updates. Files physically move only when the process-staging flow runs. Nothing is deleted without your explicit confirmation.

How to check on work

This dashboard — overview counts, review, pipeline status.
Prefect UI — prefect.home.lab — every flow run, logs, retries, schedules. (username/password: your Caddy basic-auth)
Immich — immich.home.lab — the photo library itself.

Common operations

Run a flow manually

export PREFECT_API_URL=http://localhost:4200/api
~/photo-pipeline/.venv/bin/prefect deployment run "photo-watch/watch"

Replace photo-watch/watch with any deployment name: takeout-fetch/fetch, photo-ingest/ingest, photo-quality-scan/quality, process-staging/staging, immich-import/import.

Restart the dashboard

systemctl --user restart photo-dashboard.service

See recent actions

Open the Pipeline page — the lower table shows your last 20 review decisions.

How it's built (for the curious)

Prefect 3 orchestrates everything on photo-pool; a SQLite database (photo_pipeline.db) is the source of truth for every image's hash, path and status. imagehash detects duplicates (exact sha256 + perceptual phash/dhash), CleanVision audits quality, FastAPI+htmx powers this dashboard, and Apprise sends notifications. Everything runs on your .13 server as Docker + systemd services. Code lives in the photo-pipeline repo on your Gitea.