photo-pipeline — Dashboard & System How-To

The photo-pipeline is a Prefect-orchestrated photo ingestion system on the home server (.13). It downloads Google Takeout exports, fingerprints every image (dedup), audits quality (blurry/dark/etc), and imports approved photos to Immich.

Quick links

Service	URL
Review dashboard (this)	http://192.168.20.13:8092
Prefect UI (flow runs, logs, schedules)	http://192.168.20.13:4200
Immich (photo library)	http://192.168.20.35:2283
Apprise (notifications)	https://apprise.lab.audasmedia.com.au/
Code repo	ssh://gitea.lab.audasmedia.com.au:2222/sam/photo-pipeline.git

The pipeline

flowchart LR
    TO[Google Takeout<br/>134GB / 14 zips] -->|drop manifest| W[photo-watch<br/>every 15 min]
    W -->|download + extract| I[photo-ingest<br/>sha256 + dhash dedup]
    I -->|fingerprints| Q[quality-scan<br/>PIL blurry/dark]
    Q -->|verdicts| R[Review dashboard<br/>photo-filter.home.lab]
    R -->|approved| M[merge_orphans<br/>EXIF date routing]
    M --> B[(by_date master<br/>35K files)]
    B -->|incremental import| IM[Immich<br/>.35 library]
    R -->|rejected| T[(trash → purge)]
    B -.->|offsite| S3[AWS S3<br/>Glacier lifecycle]

The flow explained

1. Takeout — download Google Photos export archives (14 zips, ~134GB)
2. photo-watch — watches incoming/, extracts, fingerprints
3. photo-ingest — sha256 + dhash; content dedup (never doubles)
4. quality-scan — PIL verdicts: blurry/dark flagged
5. Review — approve / reject in the dashboard
6. merge_orphans — approved → by_date (EXIF-date routed, rename-on-collision)
7. Immich import — incremental (only new files, checksum dedup)
8. Rejected → trash → purge · by_date → Borg daily + AWS S3 offsite

Interactive map: see docs/photo-pipeline-map.html Open the interactive map: https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/photo-pipeline-map.html

Deployments (Prefect, pool photo-pool)

Flow	Purpose	Trigger
photo-watch/watch	Watches incoming folder, auto-chains	Every 15 min
takeout-fetch/fetch	Download + track + extract Takeout archives	Manual / watch
photo-ingest/ingest	Fingerprint + dedup a folder	Manual / watch
photo-quality-scan/quality	CleanVision audit → staging	Manual / watch
immich-import/import	Upload approved (01_keep) to Immich	Manual

Triggering from the command line (on .13)

export PREFECT_API_URL=http://localhost:4200/api
~/photo-pipeline/.venv/bin/prefect deployment run "takeout-fetch/fetch" \
  --param manifest=/mnt/data/takeout/incoming/urls.txt --param export_id=photos-2026-08

~/photo-pipeline/.venv/bin/prefect deployment run "photo-ingest/ingest" \
  --param base_dir=/mnt/data/takeout/<export> --param source=takeout

Or simpler: drop the file in incoming/ and let the watch flow do it.

Key paths (on .13)

~/photo-pipeline/                 code (git repo)
  photo_db.py                     fingerprint DB module
  photo_ingest.py                 ingest flow
  takeout_fetch.py                download/track/extract flow
  quality_scan.py                 CleanVision flow
  immich_import.py                Immich upload flow
  photo_watch.py                  watch-folder trigger
  dashboard/                      this FastAPI app
  photo_pipeline.db               SQLite fingerprint DB (WAL)
  .immich-key                     Immich API key (chmod 600, gitignored)
/mnt/data/                        staging root
  takeout/incoming/               drop Takeout manifests/archives here
  takeout/processed/              done items
  01_keep/                        approved, ready for Immich
  02_review/                      flagged, awaiting decision
  03_delete/                      rejected candidates (holding — never auto-deleted)
  .thumbs/                        generated thumbnails

Services (systemd user units on .13)

• prefect-server — Docker container, port 4200
• prefect-worker — process worker on photo-pool
• photo-dashboard — FastAPI on port 8092

systemctl --user status photo-dashboard
systemctl --user restart photo-dashboard

Safety rules

• Nothing is ever auto-deleted. 03_delete is a holding folder; emptying it is a deliberate human act.
• photo_pipeline.db, .immich-key, .venv, node_modules are gitignored — never commit secrets.
• Immich API key lives in .immich-key (chmod 600); needs scopes user.read, asset.read, asset.upload.
• SSH from .13 → .35 uses -i ~/.ssh/id_ed25519_rsync.

Notifications (Apprise)

Self-hosted Apprise server fans out to all configured targets. The flows POST batch summaries (counts + dashboard link). Configure targets in the Apprise UI (https://apprise.lab.audasmedia.com.au/) — no per-machine config needed.