233 lines
11 KiB
Markdown
233 lines
11 KiB
Markdown
---
|
|
tags: [photos, immich, pipeline, project-overview, google-takeout, backup]
|
|
created: 2026-08-27
|
|
modified: 2026-09-06
|
|
aliases: [photo-pipeline, photo-project]
|
|
---
|
|
|
|
# 📸 Photo Project — Master Overview
|
|
|
|
> Single source of truth for the photo consolidation + Google Takeout + Immich + backup effort.
|
|
> Status: **Takeout downloaded & ingested; review in progress; merge+import built & tested.**
|
|
|
|
## 📋 Project tracking & docs (AI + human)
|
|
|
|
| Tool | What | Link/How to access |
|
|
|---|---|---|
|
|
| **Vikunja** | Live kanban — tasks/stages/progress | Project **"Photo Pipeline & Archive"** (To Do / In Progress / Done) |
|
|
| **Outline** | Project docs for AI agents | Collection **"Photo Pipeline & Archive"** (Overview / Architecture / Operations / Storage & Backup) |
|
|
| **Obsidian** | This note — human reading + key configs | 300 areas/360 Dev-Ops |
|
|
|
|
> Tasks, architecture and ops live in Vikunja + Outline. This note is the quick human reference.
|
|
|
|
---
|
|
|
|
## 🗺️ What this project is
|
|
|
|
Downloading everything from **Google Photos** (via Takeout), merging it with the local archive (**by_date**), importing to **Immich** (self-hosted photo server), and backing it up **offsite to AWS S3**. Then freeing Google storage by deleting (only after verified locally).
|
|
|
|
```
|
|
Google Photos (Takeout, 134GB/14 zips)
|
|
↓ download → .13:/mnt/data/takeout/incoming/
|
|
Extract (36,937 files)
|
|
↓
|
|
Ingest (fingerprint sha256+dhash, dedup vs by_date) → 12,326 new
|
|
↓
|
|
Quality scan (PIL: blurry/dark) → 11,877 keep / 449 flagged
|
|
↓
|
|
DASHBOARD REVIEW (you) → approve / reject
|
|
↓
|
|
merge_orphans (approved only) → by_date/YYYY/MM-Mon
|
|
↓
|
|
Immich import (checksum dedup = no doubles)
|
|
↓
|
|
ONLY NOW: delete from Google → frees storage
|
|
```
|
|
|
|
---
|
|
|
|
## 🏗️ How it's built (architecture — resume-facing)
|
|
|
|
### The stack (honest)
|
|
|
|
| Layer | Technology | Why |
|
|
|---|---|---|
|
|
| **Orchestration** | **Prefect 3** (flow framework) | Scheduled, retryable, observable pipelines. Flows run on a systemd worker (photo-pool) |
|
|
| **Fingerprinting/dedup** | Python + SQLite (sha256 + dhash) | Content-based dedup — re-importing never doubles. dhash for fast near-dup checks |
|
|
| **Quality analysis** | **PIL/Pillow** (image math) | Blurry (edge variance < 200) + dark (mean luminance < 40) at 300px downscale. Deterministic, no ML |
|
|
| **Review UI** | **FastAPI** + Jinja2 templates | Dashboard at :8092 — batch approve/reject, pagination, lazy thumbnails (200/page) |
|
|
| **Import target** | **Immich** (self-hosted) + its server-bundled CLI | Checksum dedup server-side = safe re-imports |
|
|
| **Data store** | SQLite (`photo_pipeline.db`) | image_hashes, known_paths, takeout_archives, batches |
|
|
| **Storage** | ext4 on USB 3TB + NixOS mounts | Master archive + pipeline staging |
|
|
| **Deployment** | NixOS systemd units (worker, dashboard, borg) | Declarative config in /etc/nixos |
|
|
| **Not used (yet)** | LangChain/LangGraph, ML models | Quality scan is deterministic math by design; AI tagging is a future candidate |
|
|
|
|
### Flow diagram
|
|
|
|
```
|
|
Prefect (photo-pool worker)
|
|
├── takeout-fetch download zips → extract → register
|
|
├── photo-ingest walk dir → sha256+dhash → dedup vs known_paths
|
|
├── photo-quality-scan PIL verdicts → status keep/delete_candidate
|
|
├── photo-watch watches incoming/ every 15 min, triggers chain
|
|
├── merge_orphans approved → by_date (EXIF-date routed)
|
|
└── immich-import scp → docker cp → server CLI upload
|
|
```
|
|
|
|
### Why Prefect + PIL, not LangChain/LLM
|
|
|
|
- **Deterministic by design**: a pipeline that moves 12K irreplaceable photos must do exactly the same thing every run. Flows don't improvise.
|
|
- **Resume note**: this is a showcase of **Prefect workflow engineering, dedup algorithms, image-quality analysis, and self-hosted infra** — deliberately NOT an LLM app. (LangChain/LangGraph exist in the homelab as the DSH agent platform — separate concern.)
|
|
|
|
### What this demonstrates (resume bullets)
|
|
|
|
- Designed a **content-addressed dedup pipeline** (sha256) over 35K+ photos — zero data loss, zero doubles
|
|
- **Prefect 3** deployment: flow orchestration, systemd worker, scheduled deployments, crash recovery
|
|
- **Image quality analysis** with PIL: calibrated thresholds, downscale + edge/luminance metrics
|
|
- **FastAPI review dashboard**: batch operations, lazy loading, DB-backed decisions
|
|
- **Full data-safety discipline**: backup verified before merge, checksum dedup before Immich import, verify before delete
|
|
- **Self-hosted infra**: NixOS, Immich, Caddy reverse-proxy, SSH key management, disk/mount troubleshooting
|
|
|
|
---
|
|
|
|
## 🔗 URLs & Services
|
|
|
|
| Service | URL | IP:Port | Notes |
|
|
|---|---|---|---|
|
|
| **Photo review dashboard** | http://photo-filter.home.lab | 192.168.20.13:8092 | Main tool — review/approve photos |
|
|
| **Dashboard (direct)** | http://192.168.20.13:8092 | — | Works without DNS |
|
|
| **Prefect** (pipeline orchestrator) | http://prefect.home.lab | 192.168.20.13:4200 | Flow runs, schedules |
|
|
| **Immich** (photo server) | http://192.168.20.35:2283 | 192.168.20.35:2283 | The destination library |
|
|
| **Langfuse** | http://langfuse.home.lab | 192.168.20.13:3001 | LLM observability |
|
|
| **Supabase** | http://supabase.home.lab | 192.168.20.27:8001 | Postgres backend |
|
|
| **DSH agents** (Langchain-ish) | dsh-sam/jo/harry/finn.lab.audasmedia.com.au | .13:3081-3084 | DeepSeek Harness per-user agents |
|
|
|
|
---
|
|
|
|
## 💾 Where the files live
|
|
|
|
| Path | Machine | Size | What |
|
|
|---|---|---|---|
|
|
| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` | **.13** | 373 GB | ⭐ **MASTER** — sorted YYYY/MM-Mon |
|
|
| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` | .13 | 69 GB | Mirror by subject (duplicate content) |
|
|
| `/mnt/ubuntu_storage_3TB/archive/03_photos/{temp_transfer,Pictures,phone_dumps}` | .13 | 31 GB | Orphans awaiting review |
|
|
| `/mnt/hd/immich/` | **.35** | ~1.1T | Immich store (library/thumbs/encoded-video) |
|
|
| `/mnt/data/takeout/` | .13 | 269 GB | **TRANSIENT** — downloaded Takeout, delete after verify |
|
|
| `/mnt/data/photo_pipeline.db` | .13 | — | SQLite — image_hashes, known_paths, takeout_archives |
|
|
| `~/photo-pipeline/` | .13 | — | Code repo (gitea: photo-pipeline.git) |
|
|
|
|
---
|
|
|
|
## 📊 Current Progress (verified 2026-08-27)
|
|
|
|
### ✅ Done
|
|
1. **Takeout downloaded** — 14 archives, 134 GB, sha256-verified byte-for-byte
|
|
2. **Extracted** — 36,937 files
|
|
3. **Ingested** — 12,326 new photos (225 dups skipped, 272 HEIC deferred)
|
|
4. **Quality scanned** — verdicts assigned
|
|
5. **DB reconciled** — by_date 35,457 files = DB + known_paths ✓
|
|
|
|
### ⏳ In progress
|
|
6. **Dashboard review** — **11,269 takeout photos still `keep`** (awaiting approval)
|
|
- 459 approved, 598 rejected so far
|
|
- 200 per page → **~57 pages of takeout keep** left
|
|
- Plus older orphans still unreviewed
|
|
|
|
### 🔜 Pending
|
|
7. **Merge approved → by_date**
|
|
8. **Import to Immich**
|
|
9. **S3 offsite backup** (~550 GB: by_date + archive/rest + .27 borg)
|
|
10. **Delete from Google** (only after 1-9 verified) → frees 17 GB storage
|
|
|
|
---
|
|
|
|
## ⚠️ Unbacked phone photos (known gap)
|
|
|
|
**Photos uploaded directly from phone → Immich app live only on `.35:/mnt/hd/immich/library`.**
|
|
|
|
- ❌ NOT in by_date (never went through the pipeline)
|
|
- ❌ NOT in any backup (Kopia skips /mnt/hd; borg only covers .13 archive)
|
|
- **If .35's disk dies, these are lost.**
|
|
- **Decision (2026-08-27): deferred** — not backing up now. Phone still has originals until cleaned.
|
|
|
|
**If this changes**: back up `library/` folder from .35 to S3 (small), or import them into by_date via the pipeline.
|
|
|
|
---
|
|
|
|
## 🛠️ How to use (operations)
|
|
|
|
### Reviewing photos (the main daily task)
|
|
1. Open **http://photo-filter.home.lab** (or http://192.168.20.13:8092)
|
|
2. Filter: `?source=takeout` (new Google Photos) · `?status=delete_candidate` (flagged dark/blurry) · `?status=keep` (clean)
|
|
3. **Bulk approve**: select-all → **Bulk Keep**. **Reject**: select → Bulk Reject
|
|
4. Decisions save instantly to DB; approved = ready to merge
|
|
|
|
### Running the merge (after approving a batch)
|
|
```bash
|
|
ssh sam@192.168.20.13
|
|
cd ~/photo-pipeline
|
|
./run-python.sh merge_orphans.py # dry-run (safe)
|
|
./run-python.sh merge_orphans.py --apply # real move approved → by_date
|
|
```
|
|
|
|
### Importing to Immich (after merge)
|
|
```bash
|
|
ssh sam@192.168.20.13
|
|
cd ~/photo-pipeline
|
|
./run-python.sh -c "import immich_import as ii; ii.upload_to_immich.fn('/mnt/ubuntu_storage_3TB/archive/03_photos/by_date')"
|
|
# Immich reports "N new / M duplicates" — dups are skipped, never doubled
|
|
```
|
|
|
|
### Monitoring
|
|
- **Prefect UI**: http://prefect.home.lab (flow runs, failures)
|
|
- **Worker service**: `systemctl --user status prefect-worker` on .13
|
|
- **Backup health**: daily borg → .23; check with `borg list` (see [[Backup Health Check Commands]])
|
|
|
|
### New Google Takeout in future
|
|
1. Download zips to `.13:/mnt/data/takeout/incoming/`
|
|
2. photo-watch picks them up every 15 min → extract → ingest → quality → dashboard
|
|
3. Review → merge → Immich → then delete from Google
|
|
|
|
### Getting the code
|
|
```bash
|
|
git clone ssh://gitea.lab.audasmedia.com.au:2222/sam/photo-pipeline.git
|
|
```
|
|
|
|
---
|
|
|
|
## 🔑 Key facts
|
|
|
|
- **by_date is the single master** — everything else derived/transient
|
|
- **Immich = derived copy** of by_date (re-importable); its DB (albums/faces) IS backed up by Kopia
|
|
- **Dedup is content-based** (sha256) — never doubles
|
|
- **382 HEIC files** lack PIL support — skipped, safe on disk, need pillow-heif later
|
|
- **Backup chain**: borg (.13→.23) daily + Kopia (.35) daily + (planned) S3 offsite
|
|
- **SSH .13→.35** needs `-i ~/.ssh/id_ed25519_rsync`
|
|
- The 3TB archive drive is **USB** — unplugged once (Aug 21-26) caused backup failures
|
|
|
|
---
|
|
|
|
## 🛠️ Tools (employment reference)
|
|
|
|
| Category | Tools used | Role |
|
|
|---|---|---|
|
|
| **Orchestration** | **Prefect 3** | Flow pipelines, scheduled deployments, worker lifecycle |
|
|
| **Backend** | Python, FastAPI, SQLite | Review dashboard, dedup, quality scan |
|
|
| **Image analysis** | PIL/Pillow | Blurry/dark verdicts (luminance/edge metrics) |
|
|
| **Frontend** | htmx, Jinja2, HTML/CSS/JS | Dashboard UI with live polling |
|
|
| **Self-hosted** | NixOS, Docker, Caddy, Immich, Gitea, Vikunja, Outline | Full homelab |
|
|
| **Storage/backup** | Borg, Kopia, AWS S3, rclone/aws-cli | Backup layers + offsite |
|
|
| **Databases** | SQLite (pipeline), Postgres (Immich) | Fingerprints + assets |
|
|
| **Monitoring** | Prefect UI, systemd, journalctl, Prefect logs | Observability |
|
|
|
|
**Demonstrated strengths**: content-addressed dedup, incremental import design,
|
|
data-safety discipline (backup-verify-before-merge), image-quality analysis at scale,
|
|
full-stack dashboard with background jobs, cross-machine SSH automation.
|
|
|
|
---
|
|
|
|
## Related notes
|
|
- [[Backup Architecture — Offsite to AWS S3]] — the S3 plan (bucket, IAM, costs)
|
|
- [[Filesystem Drive Map]] — disk layout
|
|
- [[Photo Consolidation Plan]] — superseded (May 2026)
|