---
tags: [photos, immich, pipeline, backup, google-takeout]
created: 2026-08-27
---

# Photo Pipeline & Storage Map

## What the pipeline does

```
Google Photos (Takeout export, 134GB/14 zips)
        │  download → /mnt/data/takeout/incoming/ (.13)
        ▼
Extract → /mnt/data/takeout/takeout-*/ (36,937 files)
        │
        ▼
Ingest (photo-ingest flow) → fingerprint sha256+dhash → dedup vs by_date
        │  12,326 new / 225 dups skipped / 272 HEIC deferred
        ▼
Quality scan (PIL verdicts) → keep / delete_candidate (dark/blurry)
        │  11,877 keep / 449 delete_candidate
        ▼
Dashboard review (http://192.168.20.13:8092/review) → approve/reject
        │
        ▼
merge_orphans (approved only) → by_date/sorted_pictures_holder/YYYY/MM-Mon
        │
        ▼
Immich import (checksum dedup — no doubles)
        │
        ▼
ONLY NOW: delete from Google Photos → frees storage
```

## Storage locations (where the files live)

| Path | Machine | Size | What it is |
|---|---|---|---|
| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` | .13 | 373 GB | **MASTER** photo archive, sorted YYYY/MM-Mon |
| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` | .13 | 69 GB | Backup mirror by subject |
| `/mnt/ubuntu_storage_3TB/archive/03_photos/{temp_transfer,Pictures,phone_dumps}` | .13 | 31 GB | Orphan folders awaiting review/merge |
| `/mnt/hd/immich/` | .35 | ~1.1T (60% of disk) | Immich store: library/thumbs/encoded-video/upload |
| `/mnt/data/takeout/` | .13 | 269 GB | **TRANSIENT** Takeout downloads+extract (do NOT back up) |
| `/mnt/data/photo_pipeline.db` | .13 | — | SQLite: image_hashes, known_paths, takeout_archives |

## Key facts

- **by_date is the single master.** Everything else is derived or transient.
- **Immich is a derived copy** of by_date (re-importable). Its thumbs/encoded-video are regenerable.
- **Takeout downloads (269GB) are transient** — safe to delete after verification.
- **Dedup is content-based** (sha256) — re-importing by_date never doubles.
- 382 HEIC files lack PIL support (skipped by quality scan) — safe on disk, need pillow-heif or conversion.

## Backup to S3 (offsite)

**Back up (the "house burns down" set):**
- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` (373 GB) — master
- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` (69 GB) — mirror
- ✅ Optionally: Immich `library/` on .35 (catches phone-only uploads)

**SKIP (transient/regenerable):**
- ❌ `/mnt/data/takeout/` (269 GB — transient downloads)
- ❌ Immich `thumbs/`, `encoded-video/` (regenerable)
- ❌ The 14 zip archives (they're in Google until deleted)

## Machine map

- **.13** nixos-desktop — pipeline, dashboard :8092, Prefect :4200, n8n
- **.35** — Immich :2283, Caddy (prefect/photo-filter), pi-hole; SSH needs `-i ~/.ssh/id_ed25519_rsync`
- **.23** — borg backup server (2.7T USB via CIFS to .35)
- **.27** — this dev box

## Review dashboard

- URL: http://192.168.20.13:8092/review
- Filters: `?source=takeout`, `?status=delete_candidate` (flagged), `?status=keep` (clean)
- Bulk: select-all → Bulk Keep (approve) or Bulk Reject