85 lines
3.1 KiB
Markdown
85 lines
3.1 KiB
Markdown
---
|
|
type: area
|
|
status: active
|
|
tags:
|
|
- photos
|
|
- immich
|
|
- pipeline
|
|
- backup
|
|
- google-takeout
|
|
created: 2026-08-27
|
|
---
|
|
|
|
# Photo Pipeline & Storage Map
|
|
|
|
## What the pipeline does
|
|
|
|
```
|
|
Google Photos (Takeout export, 134GB/14 zips)
|
|
│ download → /mnt/data/takeout/incoming/ (.13)
|
|
▼
|
|
Extract → /mnt/data/takeout/takeout-*/ (36,937 files)
|
|
│
|
|
▼
|
|
Ingest (photo-ingest flow) → fingerprint sha256+dhash → dedup vs by_date
|
|
│ 12,326 new / 225 dups skipped / 272 HEIC deferred
|
|
▼
|
|
Quality scan (PIL verdicts) → keep / delete_candidate (dark/blurry)
|
|
│ 11,877 keep / 449 delete_candidate
|
|
▼
|
|
Dashboard review (http://192.168.20.13:8092/review) → approve/reject
|
|
│
|
|
▼
|
|
merge_orphans (approved only) → by_date/sorted_pictures_holder/YYYY/MM-Mon
|
|
│
|
|
▼
|
|
Immich import (checksum dedup — no doubles)
|
|
│
|
|
▼
|
|
ONLY NOW: delete from Google Photos → frees storage
|
|
```
|
|
|
|
## Storage locations (where the files live)
|
|
|
|
| Path | Machine | Size | What it is |
|
|
|---|---|---|---|
|
|
| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` | .13 | 373 GB | **MASTER** photo archive, sorted YYYY/MM-Mon |
|
|
| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` | .13 | 69 GB | Backup mirror by subject |
|
|
| `/mnt/ubuntu_storage_3TB/archive/03_photos/{temp_transfer,Pictures,phone_dumps}` | .13 | 31 GB | Orphan folders awaiting review/merge |
|
|
| `/mnt/hd/immich/` | .35 | ~1.1T (60% of disk) | Immich store: library/thumbs/encoded-video/upload |
|
|
| `/mnt/data/takeout/` | .13 | 269 GB | **TRANSIENT** Takeout downloads+extract (do NOT back up) |
|
|
| `/mnt/data/photo_pipeline.db` | .13 | — | SQLite: image_hashes, known_paths, takeout_archives |
|
|
|
|
## Key facts
|
|
|
|
- **by_date is the single master.** Everything else is derived or transient.
|
|
- **Immich is a derived copy** of by_date (re-importable). Its thumbs/encoded-video are regenerable.
|
|
- **Takeout downloads (269GB) are transient** — safe to delete after verification.
|
|
- **Dedup is content-based** (sha256) — re-importing by_date never doubles.
|
|
- 382 HEIC files lack PIL support (skipped by quality scan) — safe on disk, need pillow-heif or conversion.
|
|
|
|
## Backup to S3 (offsite)
|
|
|
|
**Back up (the "house burns down" set):**
|
|
- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` (373 GB) — master
|
|
- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` (69 GB) — mirror
|
|
- ✅ Optionally: Immich `library/` on .35 (catches phone-only uploads)
|
|
|
|
**SKIP (transient/regenerable):**
|
|
- ❌ `/mnt/data/takeout/` (269 GB — transient downloads)
|
|
- ❌ Immich `thumbs/`, `encoded-video/` (regenerable)
|
|
- ❌ The 14 zip archives (they're in Google until deleted)
|
|
|
|
## Machine map
|
|
|
|
- **.13** nixos-desktop — pipeline, dashboard :8092, Prefect :4200, n8n
|
|
- **.35** — Immich :2283, Caddy (prefect/photo-filter), pi-hole; SSH needs `-i ~/.ssh/id_ed25519_rsync`
|
|
- **.23** — borg backup server (2.7T USB via CIFS to .35)
|
|
- **.27** — this dev box
|
|
|
|
## Review dashboard
|
|
|
|
- URL: http://192.168.20.13:8092/review
|
|
- Filters: `?source=takeout`, `?status=delete_candidate` (flagged), `?status=keep` (clean)
|
|
- Bulk: select-all → Bulk Keep (approve) or Bulk Reject
|