readme: Mermaid pipeline diagram + flow explanation (renders in Gitea)
This commit is contained in:
41
README.md
41
README.md
@@ -16,25 +16,32 @@ server (.13). It downloads Google Takeout exports, fingerprints every image
|
||||
|
||||
## The pipeline
|
||||
|
||||
```
|
||||
Google Takeout export (manual — Google has no export API)
|
||||
│ drop manifest of URLs into /mnt/data/takeout/incoming/
|
||||
▼
|
||||
photo-watch (Prefect, every 15 min — watches incoming/)
|
||||
│ downloads archives (aria2c, resume), tracks sha256, extracts
|
||||
▼
|
||||
photo-ingest (fingerprint DB: sha256 exact + phash/dhash near-dup detection)
|
||||
│
|
||||
▼
|
||||
photo-quality-scan (CleanVision: blurry, dark, light, grayscale, low-info…)
|
||||
│ verdicts → 01_keep / 02_review / 03_delete staging
|
||||
▼
|
||||
Apprise notification → you review in the dashboard
|
||||
│ Keep → moves file to 01_keep · Reject → 03_delete (never auto-deleted)
|
||||
▼
|
||||
immich-import (scp → server CLI upload with checksum dedup)
|
||||
```mermaid
|
||||
flowchart LR
|
||||
TO[Google Takeout<br/>134GB / 14 zips] -->|drop manifest| W[photo-watch<br/>every 15 min]
|
||||
W -->|download + extract| I[photo-ingest<br/>sha256 + dhash dedup]
|
||||
I -->|fingerprints| Q[quality-scan<br/>PIL blurry/dark]
|
||||
Q -->|verdicts| R[Review dashboard<br/>photo-filter.home.lab]
|
||||
R -->|approved| M[merge_orphans<br/>EXIF date routing]
|
||||
M --> B[(by_date master<br/>35K files)]
|
||||
B -->|incremental import| IM[Immich<br/>.35 library]
|
||||
R -->|rejected| T[(trash → purge)]
|
||||
B -.->|offsite| S3[AWS S3<br/>Glacier lifecycle]
|
||||
```
|
||||
|
||||
### The flow explained
|
||||
|
||||
1. **Takeout** — download Google Photos export archives (14 zips, ~134GB)
|
||||
2. **photo-watch** — watches incoming/, extracts, fingerprints
|
||||
3. **photo-ingest** — sha256 + dhash; content dedup (never doubles)
|
||||
4. **quality-scan** — PIL verdicts: blurry/dark flagged
|
||||
5. **Review** — approve / reject in the dashboard
|
||||
6. **merge_orphans** — approved → by_date (EXIF-date routed, rename-on-collision)
|
||||
7. **Immich import** — incremental (only new files, checksum dedup)
|
||||
8. **Rejected** → trash → purge · **by_date** → Borg daily + AWS S3 offsite
|
||||
|
||||
> Interactive map: see [`docs/photo-pipeline-map.html`](docs/photo-pipeline-map.html)
|
||||
> (published: https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/)
|
||||
## Deployments (Prefect, pool `photo-pool`)
|
||||
|
||||
| Flow | Purpose | Trigger |
|
||||
|
||||
Reference in New Issue
Block a user