obsidian-vault/300 areas/360 Dev-Ops Network Computers/Photo Pipeline & Storage Map.md

type, status, tags, created

type	status	tags	created
subject	active	photos immich pipeline backup google-takeout	2026-08-27

Photo Pipeline & Storage Map

What the pipeline does

Google Photos (Takeout export, 134GB/14 zips)
        │  download → /mnt/data/takeout/incoming/ (.13)
        ▼
Extract → /mnt/data/takeout/takeout-*/ (36,937 files)
        │
        ▼
Ingest (photo-ingest flow) → fingerprint sha256+dhash → dedup vs by_date
        │  12,326 new / 225 dups skipped / 272 HEIC deferred
        ▼
Quality scan (PIL verdicts) → keep / delete_candidate (dark/blurry)
        │  11,877 keep / 449 delete_candidate
        ▼
Dashboard review (http://192.168.20.13:8092/review) → approve/reject
        │
        ▼
merge_orphans (approved only) → by_date/sorted_pictures_holder/YYYY/MM-Mon
        │
        ▼
Immich import (checksum dedup — no doubles)
        │
        ▼
ONLY NOW: delete from Google Photos → frees storage

Storage locations (where the files live)

Path	Machine	Size	What it is
/mnt/ubuntu_storage_3TB/archive/03_photos/by_date	.13	373 GB	MASTER photo archive, sorted YYYY/MM-Mon
/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject	.13	69 GB	Backup mirror by subject
/mnt/ubuntu_storage_3TB/archive/03_photos/{temp_transfer,Pictures,phone_dumps}	.13	31 GB	Orphan folders awaiting review/merge
/mnt/hd/immich/	.35	~1.1T (60% of disk)	Immich store: library/thumbs/encoded-video/upload
/mnt/data/takeout/	.13	269 GB	TRANSIENT Takeout downloads+extract (do NOT back up)
/mnt/data/photo_pipeline.db	.13	—	SQLite: image_hashes, known_paths, takeout_archives

Key facts

• by_date is the single master. Everything else is derived or transient.
• Immich is a derived copy of by_date (re-importable). Its thumbs/encoded-video are regenerable.
• Takeout downloads (269GB) are transient — safe to delete after verification.
• Dedup is content-based (sha256) — re-importing by_date never doubles.
• 382 HEIC files lack PIL support (skipped by quality scan) — safe on disk, need pillow-heif or conversion.

Backup to S3 (offsite)

Back up (the "house burns down" set):

• ✅ /mnt/ubuntu_storage_3TB/archive/03_photos/by_date (373 GB) — master
• ✅ /mnt/ubuntu_storage_3TB/archive/03_photos/by_subject (69 GB) — mirror
• ✅ Optionally: Immich library/ on .35 (catches phone-only uploads)

SKIP (transient/regenerable):

• ❌ /mnt/data/takeout/ (269 GB — transient downloads)
• ❌ Immich thumbs/, encoded-video/ (regenerable)
• ❌ The 14 zip archives (they're in Google until deleted)

Machine map

• .13 nixos-desktop — pipeline, dashboard :8092, Prefect :4200, n8n
• .35 — Immich :2283, Caddy (prefect/photo-filter), pi-hole; SSH needs -i ~/.ssh/id_ed25519_rsync
• .23 — borg backup server (2.7T USB via CIFS to .35)
• .27 — this dev box

Review dashboard

• URL: http://192.168.20.13:8092/review
• Filters: ?source=takeout, ?status=delete_candidate (flagged), ?status=keep (clean)
• Bulk: select-all → Bulk Keep (approve) or Bulk Reject