sam/photo-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

readme: Mermaid pipeline diagram + flow explanation (renders in Gitea)

Browse Source
This commit is contained in:
Sam Rolfe
2026-09-06 18:27:43 +10:00
parent 8527bf6434
commit 61a501716e
1 changed files with 24 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
README.md
View File

@@ -16,25 +16,32 @@ server (.13). It downloads Google Takeout exports, fingerprints every image
## The pipeline ## The pipeline
``` ```mermaid
Google Takeout export (manual — Google has no export API) flowchart LR
│ drop manifest of URLs into /mnt/data/takeout/incoming/ TO[Google Takeout<br/>134GB / 14 zips] -->|drop manifest| W[photo-watch<br/>every 15 min]
▼ W -->|download + extract| I[photo-ingest<br/>sha256 + dhash dedup]
photo-watch (Prefect, every 15 min — watches incoming/) I -->|fingerprints| Q[quality-scan<br/>PIL blurry/dark]
│ downloads archives (aria2c, resume), tracks sha256, extracts Q -->|verdicts| R[Review dashboard<br/>photo-filter.home.lab]
▼ R -->|approved| M[merge_orphans<br/>EXIF date routing]
photo-ingest (fingerprint DB: sha256 exact + phash/dhash near-dup detection) M --> B[(by_date master<br/>35K files)]
│ B -->|incremental import| IM[Immich<br/>.35 library]
▼ R -->|rejected| T[(trash → purge)]
photo-quality-scan (CleanVision: blurry, dark, light, grayscale, low-info…) B -.->|offsite| S3[AWS S3<br/>Glacier lifecycle]
│ verdicts → 01_keep / 02_review / 03_delete staging
▼
Apprise notification → you review in the dashboard
│ Keep → moves file to 01_keep · Reject → 03_delete (never auto-deleted)
▼
immich-import (scp → server CLI upload with checksum dedup)
``` ```
### The flow explained
1. **Takeout** — download Google Photos export archives (14 zips, ~134GB)
2. **photo-watch** — watches incoming/, extracts, fingerprints
3. **photo-ingest** — sha256 + dhash; content dedup (never doubles)
4. **quality-scan** — PIL verdicts: blurry/dark flagged
5. **Review** — approve / reject in the dashboard
6. **merge_orphans** — approved → by_date (EXIF-date routed, rename-on-collision)
7. **Immich import** — incremental (only new files, checksum dedup)
8. **Rejected** → trash → purge · **by_date** → Borg daily + AWS S3 offsite
> Interactive map: see [`docs/photo-pipeline-map.html`](docs/photo-pipeline-map.html)
> (published: https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/)
## Deployments (Prefect, pool `photo-pool`) ## Deployments (Prefect, pool `photo-pool`)
| Flow | Purpose | Trigger | | Flow | Purpose | Trigger |