160 lines
8.5 KiB
Markdown
160 lines
8.5 KiB
Markdown
---
|
||
created: 2026-09-09 11:57
|
||
modified: 2026-09-09
|
||
type: project
|
||
status: active
|
||
tags:
|
||
- tool
|
||
- tools
|
||
- photos
|
||
- ai
|
||
- immich
|
||
- pipeline
|
||
- google
|
||
- prefect
|
||
aliases: [photo-pipeline, google-photos-migration]
|
||
---
|
||
# [[Google Cloud Images Immich]]
|
||
|
||
> **Photo Pipeline — migrate Google Photos to a private archive, serve via Immich, back up offsite.**
|
||
> A Prefect-orchestrated pipeline that downloads Google Takeout exports, fingerprints + dedups every image, applies quality analysis (blurry/dark), flows through a human review dashboard, merges approved photos into a date-sorted master archive (`by_date`), imports them into Immich, and protects everything with layered backups (Borg, Kopia, AWS S3).
|
||
|
||
---
|
||
|
||
## 🗺️ The pipeline (Mermaid)
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
TO[Google Takeout<br/>134GB / 14 zips] -->|drop manifest| W[photo-watch<br/>every 15 min]
|
||
W -->|download + extract| I[photo-ingest<br/>sha256 + dhash dedup]
|
||
I -->|fingerprints| Q[quality-scan<br/>PIL blurry/dark]
|
||
Q -->|verdicts| R[Review dashboard<br/>photo-filter.home.lab]
|
||
R -->|approved| M[merge_orphans<br/>EXIF date routing]
|
||
M --> B[(by_date master<br/>35K files)]
|
||
B -->|incremental import| IM[Immich<br/>.35 library]
|
||
R -->|rejected| T[(trash → purge)]
|
||
B -.->|offsite| S3[AWS S3<br/>Glacier lifecycle]
|
||
```
|
||
|
||
**Explore the interactive map:** https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/photo-pipeline-map.html
|
||
|
||
---
|
||
|
||
## What it is
|
||
|
||
A **self-hosted Google Photos replacement pipeline**. Photos leave Google via Takeout, are verified + deduplicated + quality-checked locally, reviewed by a human, then land in a **date-sorted master archive** (the single source of truth) and are served through **Immich** (a self-hosted photo server). The goal is full independence from Google + a robust local + offsite backup story.
|
||
|
||
### Why it matters
|
||
- Content-based dedup by **sha256** — re-importing never doubles photos
|
||
- Quality **verdicts are deterministic** (PIL image math, not ML) — reliable at 35K+ scale
|
||
- **Human gate** — nothing merges until a person approves it
|
||
- **Incremental import** — only new photos go to Immich (no 376GB re-uploads)
|
||
- **Safe cleanup** — rejected → trash (recoverable) → purge only on confirmation
|
||
- **PHOTOS-TO-DELETE album** — remove unwanted photos from Immich + by_date in one action
|
||
|
||
---
|
||
|
||
## How to use it
|
||
|
||
### Reviewing photos (main daily task)
|
||
1. Open **https://photo-filter.home.lab** (basic-auth: `sam`)
|
||
2. Filter: `?source=takeout&status=keep` (new Google photos) or `?status=delete_candidate` (blurry/dark)
|
||
3. **Bulk Keep** (approve) or **Bulk Reject** — decisions save instantly to the DB
|
||
4. Approved photos are ready for merge; rejected sit in trash (recoverable)
|
||
|
||
### Pipeline actions (dashboard buttons)
|
||
Go to the **"Pipeline actions"** page in the dashboard:
|
||
|
||
| Button | What it does | Safe? |
|
||
|---|---|---|
|
||
| 🔍 Dry-run merge | Preview what approved photos WOULD move | ✅ read-only |
|
||
| ▶ Merge approved → by_date | Move approved files (EXIF-date routed) | Confirmation required |
|
||
| 📤 Import by_date → Immich | Upload only NEW photos (dedup-safe) | ✅ incremental |
|
||
| 🗑 Rejected preview / trash / purge | Move rejected → trash → permanent delete | Trash recoverable |
|
||
| 🔄 Check PHOTOS-TO-DELETE album | See what's in the removal album | ✅ read-only |
|
||
| 🗑 Remove album photos | Trash by_date copies + delete from Immich | Confirmation required |
|
||
|
||
### Removing unwanted photos (Immich side)
|
||
1. In **Immich**, move photos you don't want into the **PHOTOS-TO-DELETE** album
|
||
2. Dashboard → Pipeline → **Check PHOTOS-TO-DELETE**
|
||
3. **Remove album photos** → by_date copies go to trash (recoverable), Immich copies are deleted
|
||
|
||
### Adding NEW Google Takeout in future
|
||
1. Download zips to `incoming/` (or drop a manifest of URLs)
|
||
2. `photo-watch` auto-processes every 15 min → extract → ingest → quality → review
|
||
3. Review → merge → import → **then** you can delete from Google
|
||
|
||
---
|
||
|
||
## 🛠️Tools used (resume showcase)
|
||
|
||
| Layer | Technology | Role |
|
||
|---|---|---|
|
||
| **Orchestration** | **Prefect 3** | Flows, scheduled deploys, worker lifecycle, crash recovery |
|
||
| **Backend** | Python, **FastAPI**, SQLite | Review dashboard, dedup, quality scan, incremental import |
|
||
| **Image analysis** | **PIL/Pillow** | Blurry/dark verdicts (luminance + edge variance at 300px) |
|
||
| **Frontend** | htmx, Jinja2, JS | Dashboard with live polling, batch actions, confirm dialogs |
|
||
| **Data stores** | SQLite (pipeline), Postgres (Immich) | Fingerprints + assets/albums |
|
||
| **Import** | Immich server-bundled CLI | Checksum-dedup'd upload |
|
||
| **Backup** | Borg, Kopia, AWS S3 (+ rclone/aws cli) | Local chains + offsite |
|
||
| **Diagrams** | Mermaid, Archify (published to maps.lab) | Architecture visualization |
|
||
| **Ops/infra** | NixOS, Docker, Caddy, Gitea, Vikunja, Outline | Full self-hosted stack |
|
||
|
||
**Demonstrated strengths:** content-addressed dedup, incremental import design, data-safety discipline (backup-verify-before-merge), image-quality analysis at scale, full-stack dashboard with background jobs, cross-machine SSH automation, backup failure detection + repair.
|
||
|
||
---
|
||
|
||
## Where things are
|
||
|
||
| What | Where |
|
||
|---|---|
|
||
| **Master archive (`by_date`)** | `.13:/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` — 35K files, YYYY/MM-Mon |
|
||
| **Pipeline code** | Gitea: `ssh://gitea.lab.audasmedia.com.au:2222/sam/photo-pipeline.git` |
|
||
| **Review dashboard** | https://photo-filter.home.lab (`.13:8092`) — basic-auth |
|
||
| **Prefect UI** | https://prefect.home.lab (`.13:4200`) |
|
||
| **Immich** | `http://192.168.20.35:2283` — self-hosted photo server |
|
||
| **Interactive map** | https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/photo-pipeline-map.html |
|
||
| **Fingerprint DB** | `.13:/mnt/data/photo_pipeline.db` (SQLite) |
|
||
| **Takeout staging** | `.13:/mnt/data/takeout/` — transient downloads/extract |
|
||
| **Removed photos trash** | `.13:/mnt/data/takeout/removed/` + `rejected-trash/` — recoverable |
|
||
| **Vikunja (tasks)** | Project **"Photo Pipeline & Archive"** — kanban of remaining work |
|
||
| **Outline (docs)** | Collection **"Photo Pipeline & Archive"** — Overview / Architecture / Operations / Storage |
|
||
|
||
---
|
||
|
||
## Current status (Sep 2026)
|
||
|
||
| Item | State |
|
||
|---|---|
|
||
| Google Takeout download | ✅ Complete (134GB / 14 zips, sha256-verified) |
|
||
| Extract + ingest | ✅ 12.3K new photos fingerprinted, dedup-skipped 225 |
|
||
| Quality scan | ✅ Verdicts applied (PIL) |
|
||
| **Review** | 🔄 **2,246 approved · 8,877 keep to review · 1,203 rejected** |
|
||
| Merge + import to Immich | ✅ Built + tested (incremental, dedup-safe) |
|
||
| Rejected cleanup | ✅ Trash → purge flow working |
|
||
| PHOTOS-TO-DELETE removal | ✅ Working (album → remove from Immich + by_date) |
|
||
| S3 offsite backup | ⏳ Pending seed (by_date + archive/rest ≈ 712GB) |
|
||
| Delete from Google (final) | ⏳ After local verification complete |
|
||
|
||
---
|
||
|
||
## ⚙️ Config notes (for future maintenance)
|
||
|
||
- **SSH .13 → .35** uses key: `-i ~/.ssh/id_ed25519_rsync`
|
||
- **Dashboard service** (on .13): `systemctl --user restart photo-dashboard` (users must **refresh browser** after code changes — stale JS causes "buttons do nothing")
|
||
- **Prefect worker**: `systemctl --user status prefect-worker` (runs photo-pool)
|
||
- **Backup job**: `borgbackup-job-archive-to-23` (daily 06:00, .13 → .23) — covers `archive/` **and** `~/photo-pipeline` (DB)
|
||
- The **3TB archive drive is USB** — got unplugged once (Aug 21-26 2026 → backup failures); replugging resolved. Remount if `/mnt/ubuntu_storage_3TB` missing: the disk is `sde1` (UUID `037a542c-…`), `sudo mount /mnt/ubuntu_storage_3TB`
|
||
- **Immich API key** lives at `~/.ssh/../photo-pipeline/.immich-key` (0600) — needs user.read + asset.read + asset.upload; skip server.about scope (not granted)
|
||
- **382 HEIC files** skip the PIL quality scan (no HEIF plugin) — safe on disk, handle with pillow-heif later
|
||
- **Known gap**: phone-only photos uploaded directly to Immich are NOT in by_date and NOT backed up — decision deferred
|
||
- Gitea repo pushes use SSH port **2222**; .13 SSH can be intermittently flaky — retry after 30s if a push times out
|
||
|
||
---
|
||
|
||
## Resources
|
||
- Interactive map: https://maps.lab.audasmedia.com.au/google_cloud_and_images/docs/photo-pipeline-map.html
|
||
- [[Photo Project Master Overview]] — deep-dive human reference (storage map, backup S3 plan, tools table)
|
||
- [[Backup Architecture — Offsite to AWS S3]] — the S3 offsite plan
|
||
- [[Filesystem Drive Map]] — disk layout
|
||
- Vikunja: project **Photo Pipeline & Archive** · Outline: collection **Photo Pipeline & Archive** |