113 lines
4.9 KiB
Markdown
113 lines
4.9 KiB
Markdown
# photo-pipeline — Dashboard & System How-To
|
|
|
|
The photo-pipeline is a Prefect-orchestrated photo ingestion system on the home
|
|
server (.13). It downloads Google Takeout exports, fingerprints every image
|
|
(dedup), audits quality (blurry/dark/etc), and imports approved photos to Immich.
|
|
|
|
## Quick links
|
|
|
|
| Service | URL |
|
|
|---|---|
|
|
| **Review dashboard** (this) | `http://192.168.20.13:8092` |
|
|
| **Prefect UI** (flow runs, logs, schedules) | `http://192.168.20.13:4200` |
|
|
| **Immich** (photo library) | `http://192.168.20.35:2283` |
|
|
| **Apprise** (notifications) | `https://apprise.lab.audasmedia.com.au/` |
|
|
| **Code repo** | `ssh://gitea.lab.audasmedia.com.au:2222/sam/photo-pipeline.git` |
|
|
|
|
## The pipeline
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
TO[Google Takeout<br/>134GB / 14 zips] -->|drop manifest| W[photo-watch<br/>every 15 min]
|
|
W -->|download + extract| I[photo-ingest<br/>sha256 + dhash dedup]
|
|
I -->|fingerprints| Q[quality-scan<br/>PIL blurry/dark]
|
|
Q -->|verdicts| R[Review dashboard<br/>photo-filter.home.lab]
|
|
R -->|approved| M[merge_orphans<br/>EXIF date routing]
|
|
M --> B[(by_date master<br/>35K files)]
|
|
B -->|incremental import| IM[Immich<br/>.35 library]
|
|
R -->|rejected| T[(trash → purge)]
|
|
B -.->|offsite| S3[AWS S3<br/>Glacier lifecycle]
|
|
```
|
|
|
|
### The flow explained
|
|
|
|
1. **Takeout** — download Google Photos export archives (14 zips, ~134GB)
|
|
2. **photo-watch** — watches incoming/, extracts, fingerprints
|
|
3. **photo-ingest** — sha256 + dhash; content dedup (never doubles)
|
|
4. **quality-scan** — PIL verdicts: blurry/dark flagged
|
|
5. **Review** — approve / reject in the dashboard
|
|
6. **merge_orphans** — approved → by_date (EXIF-date routed, rename-on-collision)
|
|
7. **Immich import** — incremental (only new files, checksum dedup)
|
|
8. **Rejected** → trash → purge · **by_date** → Borg daily + AWS S3 offsite
|
|
|
|
> Interactive map: see [`docs/photo-pipeline-map.html`](docs/photo-pipeline-map.html)
|
|
> **Open the interactive map:** https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/photo-pipeline-map.html
|
|
## Deployments (Prefect, pool `photo-pool`)
|
|
|
|
| Flow | Purpose | Trigger |
|
|
|---|---|---|
|
|
| `photo-watch/watch` | Watches incoming folder, auto-chains | Every 15 min |
|
|
| `takeout-fetch/fetch` | Download + track + extract Takeout archives | Manual / watch |
|
|
| `photo-ingest/ingest` | Fingerprint + dedup a folder | Manual / watch |
|
|
| `photo-quality-scan/quality` | CleanVision audit → staging | Manual / watch |
|
|
| `immich-import/import` | Upload approved (01_keep) to Immich | Manual |
|
|
|
|
### Triggering from the command line (on .13)
|
|
|
|
```bash
|
|
export PREFECT_API_URL=http://localhost:4200/api
|
|
~/photo-pipeline/.venv/bin/prefect deployment run "takeout-fetch/fetch" \
|
|
--param manifest=/mnt/data/takeout/incoming/urls.txt --param export_id=photos-2026-08
|
|
|
|
~/photo-pipeline/.venv/bin/prefect deployment run "photo-ingest/ingest" \
|
|
--param base_dir=/mnt/data/takeout/<export> --param source=takeout
|
|
```
|
|
|
|
Or simpler: drop the file in `incoming/` and let the watch flow do it.
|
|
|
|
## Key paths (on .13)
|
|
|
|
```
|
|
~/photo-pipeline/ code (git repo)
|
|
photo_db.py fingerprint DB module
|
|
photo_ingest.py ingest flow
|
|
takeout_fetch.py download/track/extract flow
|
|
quality_scan.py CleanVision flow
|
|
immich_import.py Immich upload flow
|
|
photo_watch.py watch-folder trigger
|
|
dashboard/ this FastAPI app
|
|
photo_pipeline.db SQLite fingerprint DB (WAL)
|
|
.immich-key Immich API key (chmod 600, gitignored)
|
|
/mnt/data/ staging root
|
|
takeout/incoming/ drop Takeout manifests/archives here
|
|
takeout/processed/ done items
|
|
01_keep/ approved, ready for Immich
|
|
02_review/ flagged, awaiting decision
|
|
03_delete/ rejected candidates (holding — never auto-deleted)
|
|
.thumbs/ generated thumbnails
|
|
```
|
|
|
|
## Services (systemd user units on .13)
|
|
|
|
- `prefect-server` — Docker container, port 4200
|
|
- `prefect-worker` — process worker on photo-pool
|
|
- `photo-dashboard` — FastAPI on port 8092
|
|
|
|
```bash
|
|
systemctl --user status photo-dashboard
|
|
systemctl --user restart photo-dashboard
|
|
```
|
|
|
|
## Safety rules
|
|
|
|
- **Nothing is ever auto-deleted.** `03_delete` is a holding folder; emptying it is a deliberate human act.
|
|
- `photo_pipeline.db`, `.immich-key`, `.venv`, `node_modules` are gitignored — never commit secrets.
|
|
- Immich API key lives in `.immich-key` (chmod 600); needs scopes `user.read`, `asset.read`, `asset.upload`.
|
|
- SSH from .13 → .35 uses `-i ~/.ssh/id_ed25519_rsync`.
|
|
|
|
## Notifications (Apprise)
|
|
|
|
Self-hosted Apprise server fans out to all configured targets. The flows POST
|
|
batch summaries (counts + dashboard link). Configure targets in the Apprise UI
|
|
(`https://apprise.lab.audasmedia.com.au/`) — no per-machine config needed.
|