Add system README/how-to (dashboard, prefect, paths, safety)

2026-08-08 09:52:41 +10:00
parent 4209a1153e
commit e7c0bfc584
1 changed files with 101 additions and 12 deletions
--- a/README.md
+++ b/README.md
@@ -1,16 +1,105 @@
-# photo-pipeline
+# photo-pipeline — Dashboard & System How-To

-Prefect-orchestrated photo ingestion: Takeout download → fingerprint DB (dedup) → quality scan → Immich.
+The photo-pipeline is a Prefect-orchestrated photo ingestion system on the home
+server (.13). It downloads Google Takeout exports, fingerprints every image
+(dedup), audits quality (blurry/dark/etc), and imports approved photos to Immich.

-## Components
- `photo_db.py` — SQLite fingerprint DB (sha256 exact + phash/dhash near dupes; Takeout archive tracking)
- `takeout_fetch.py` — flow: download (aria2c resume) → track sha256 → extract
- `photo_ingest.py` — flow: scan dir → check dups → register new
- `prefect.yaml` — deployments (fetch, ingest) on photo-pool
+## Quick links

-## NixOS note
-Venv python needs LD_LIBRARY_PATH (gcc libstdc++ + zlib) — see run-python.sh / run-prefect.sh.
+| Service | URL |
+|---|---|
+| **Review dashboard** (this) | `http://192.168.20.13:8092` |
+| **Prefect UI** (flow runs, logs, schedules) | `http://192.168.20.13:4200` |
+| **Immich** (photo library) | `http://192.168.20.35:2283` |
+| **Apprise** (notifications) | `https://apprise.lab.audasmedia.com.au/` |
+| **Code repo** | `ssh://gitea.lab.audasmedia.com.au:2222/sam/photo-pipeline.git` |

-## Run
-prefect deployment run "takeout-fetch/fetch" --param manifest=/path/urls.txt --param export_id=photos-2026-08
-prefect deployment run "photo-ingest/ingest" --param base_dir=/mnt/data/takeout/photos-2026-08
+## The pipeline
+
+```
+Google Takeout export (manual — Google has no export API)
+        │  drop manifest of URLs into /mnt/data/takeout/incoming/
+        ▼
+photo-watch   (Prefect, every 15 min — watches incoming/)
+        │  downloads archives (aria2c, resume), tracks sha256, extracts
+        ▼
+photo-ingest  (fingerprint DB: sha256 exact + phash/dhash near-dup detection)
+        │
+        ▼
+photo-quality-scan  (CleanVision: blurry, dark, light, grayscale, low-info…)
+        │  verdicts → 01_keep / 02_review / 03_delete staging
+        ▼
+Apprise notification  →  you review in the dashboard
+        │  Keep → moves file to 01_keep · Reject → 03_delete (never auto-deleted)
+        ▼
+immich-import  (scp → server CLI upload with checksum dedup)
+```
+
+## Deployments (Prefect, pool `photo-pool`)
+
+| Flow | Purpose | Trigger |
+|---|---|---|
+| `photo-watch/watch` | Watches incoming folder, auto-chains | Every 15 min |
+| `takeout-fetch/fetch` | Download + track + extract Takeout archives | Manual / watch |
+| `photo-ingest/ingest` | Fingerprint + dedup a folder | Manual / watch |
+| `photo-quality-scan/quality` | CleanVision audit → staging | Manual / watch |
+| `immich-import/import` | Upload approved (01_keep) to Immich | Manual |
+
+### Triggering from the command line (on .13)
+
+```bash
+export PREFECT_API_URL=http://localhost:4200/api
+~/photo-pipeline/.venv/bin/prefect deployment run "takeout-fetch/fetch" \
+  --param manifest=/mnt/data/takeout/incoming/urls.txt --param export_id=photos-2026-08
+
+~/photo-pipeline/.venv/bin/prefect deployment run "photo-ingest/ingest" \
+  --param base_dir=/mnt/data/takeout/<export> --param source=takeout
+```
+
+Or simpler: drop the file in `incoming/` and let the watch flow do it.
+
+## Key paths (on .13)
+
+```
+~/photo-pipeline/                 code (git repo)
+  photo_db.py                     fingerprint DB module
+  photo_ingest.py                 ingest flow
+  takeout_fetch.py                download/track/extract flow
+  quality_scan.py                 CleanVision flow
+  immich_import.py                Immich upload flow
+  photo_watch.py                  watch-folder trigger
+  dashboard/                      this FastAPI app
+  photo_pipeline.db               SQLite fingerprint DB (WAL)
+  .immich-key                     Immich API key (chmod 600, gitignored)
+/mnt/data/                        staging root
+  takeout/incoming/               drop Takeout manifests/archives here
+  takeout/processed/              done items
+  01_keep/                        approved, ready for Immich
+  02_review/                      flagged, awaiting decision
+  03_delete/                      rejected candidates (holding — never auto-deleted)
+  .thumbs/                        generated thumbnails
+```
+
+## Services (systemd user units on .13)
+
+- `prefect-server` — Docker container, port 4200
+- `prefect-worker` — process worker on photo-pool
+- `photo-dashboard` — FastAPI on port 8092
+
+```bash
+systemctl --user status photo-dashboard
+systemctl --user restart photo-dashboard
+```
+
+## Safety rules
+
+- **Nothing is ever auto-deleted.** `03_delete` is a holding folder; emptying it is a deliberate human act.
+- `photo_pipeline.db`, `.immich-key`, `.venv`, `node_modules` are gitignored — never commit secrets.
+- Immich API key lives in `.immich-key` (chmod 600); needs scopes `user.read`, `asset.read`, `asset.upload`.
+- SSH from .13 → .35 uses `-i ~/.ssh/id_ed25519_rsync`.
+
+## Notifications (Apprise)
+
+Self-hosted Apprise server fans out to all configured targets. The flows POST
+batch summaries (counts + dashboard link). Configure targets in the Apprise UI
+(`https://apprise.lab.audasmedia.com.au/`) — no per-machine config needed.