From 550acd01dfc9fc5bda939f8f55adf354957ba650 Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Fri, 28 Aug 2026 18:54:08 +1000 Subject: [PATCH] sam-4screen-desktop 2026-8-28:18:54:8 --- .obsidian/workspace.json | 4 +- .../Filesystem Drive Map.md | 23 +++--- .../Photo Pipeline & Storage Map.md | 77 +++++++++++++++++++ 3 files changed, 92 insertions(+), 12 deletions(-) create mode 100644 300 areas/360 Dev-Ops Network Computers/Photo Pipeline & Storage Map.md diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index f43d1a6..49e02ab 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -209,8 +209,9 @@ "templater-obsidian:Templater": false } }, - "active": "7066c9273c6fb936", + "active": "8018aa52f0591df9", "lastOpenFiles": [ + "300 areas/360 Dev-Ops Network Computers/Photo Pipeline & Storage Map.md", "200 projects/220 Web Host Migration/Migration Work Log.md", "200 projects/220 Web Host Migration/Phase 5 — Decommission InMotion.md", "200 projects/220 Web Host Migration/Phase 3 — Local Infra Prep.md", @@ -236,7 +237,6 @@ "000 daily/FAMILY S3 Storage Integration & Blueprint.md", "400 resources/450 Recipes/Easy Yogurt flat bread.md", "400 resources/450 Recipes/Chilli Sauce Hack.md", - "000 daily/basketball training.md", "000 daily/ThinkPad Recommendations", "500 archive/510 Daily", "300 areas/305 Ideas Businesses", diff --git a/300 areas/360 Dev-Ops Network Computers/Filesystem Drive Map.md b/300 areas/360 Dev-Ops Network Computers/Filesystem Drive Map.md index 0253144..63ba388 100644 --- a/300 areas/360 Dev-Ops Network Computers/Filesystem Drive Map.md +++ b/300 areas/360 Dev-Ops Network Computers/Filesystem Drive Map.md @@ -121,19 +121,22 @@ aliases: [drive-map, filesystem-map] | **Boot** | `sda1` | 1G | — | — | `/boot` | — | `4D80-F99E` | EFI boot | | **Storage** | `sdc2` | 931G | 2.1M | **870G (1%)** | `/mnt/storage` | archive_storage | — | **EMPTY** — reformatted ext4 2026-05-30 (was NTFS Windows drive) | | **Data** | `sdb2` | 1.8T | 2.1M | **1.7T (1%)** | `/mnt/data` | Data | — | **EMPTY** — reformatted ext4 2026-05-30. **Planned Takeout landing zone** | -| **Ubuntu Storage** | `sdd1` | 2.7T | 810G | **1.9T (30%)** | `/mnt/ubuntu_storage_3TB` | ubuntu_storage_3 | `037a542c-...` | Holds `archive/` (649G), `timeshift` (64G), `backup/` (35G). Backup target for .27 + .13 Borg repos | -| **MaxtorBackup** | `sde1` | 1.4T | ~0 | — | `/mnt/maxtor_backup` | MaxtorBackup | `b0fa7768-...` | Now mounted — EMPTY (4KB). Old backup drive, content unknown/clear | +| **Ubuntu Storage** | `sde1` | 2.7T | 874G | **1.9T (33%)** | `/mnt/ubuntu_storage_3TB` | ubuntu_storage_3 | `037a542c-...` | **USB drive** — holds `archive/` (photo master), `timeshift`, `backup/`. Backup target for .27 + .13 Borg repos. ⚠️ USB — unplugged Aug 21-26 2026 caused backup failures; replugged, letter shifted `sdd1→sde1` | +| **MaxtorBackup** | `sdd1` | 1.4T | ~0 | — | `/mnt/maxtor_backup` | MaxtorBackup | `b0fa7768-...` | Old backup drive — only NixOS migration tarballs, no photos | -### Key Photo Location (NEW — moved 2026-05-30) +### Key Photo Location (moved 2026-05-30; reconciled 2026-08-27) ``` -/mnt/ubuntu_storage_3TB/archive/03_photos/ ← 472G, 58,948 files -├── by_date/ ← year→month organized -├── by_subject/ ← subject-organized -├── phone_dumps/ ← phone exports -├── Pictures/ ← old Pictures folder -├── temp_transfer/ ← event photos -└── lily.jpg, lily2jpg.jpg +/mnt/ubuntu_storage_3TB/archive/03_photos/ ← 473G total +├── by_date/ 373G ← MASTER, 35,457 files, YYYY/MM-Mon +├── by_subject/ 69G ← backup mirror (in Immich) +├── phone_dumps/ 2G ← phone exports (awaiting review) +├── Pictures/ 1G ← old Pictures (awaiting review) +├── temp_transfer/ 28G ← event photos (awaiting review) +└── lily2jpg.jpg + +Takeout downloads (TRANSIENT — do NOT back up): /mnt/data/takeout/ = 269G +Immich store: /mnt/hd/immich on .35 (library/thumbs/encoded-video) archive/ also contains: 01_family, 02_projects, 04_education, 05_documents, 06_creative, 07_notes, 08_transfer diff --git a/300 areas/360 Dev-Ops Network Computers/Photo Pipeline & Storage Map.md b/300 areas/360 Dev-Ops Network Computers/Photo Pipeline & Storage Map.md new file mode 100644 index 0000000..e19c0cb --- /dev/null +++ b/300 areas/360 Dev-Ops Network Computers/Photo Pipeline & Storage Map.md @@ -0,0 +1,77 @@ +--- +tags: [photos, immich, pipeline, backup, google-takeout] +created: 2026-08-27 +--- + +# Photo Pipeline & Storage Map + +## What the pipeline does + +``` +Google Photos (Takeout export, 134GB/14 zips) + │ download → /mnt/data/takeout/incoming/ (.13) + ▼ +Extract → /mnt/data/takeout/takeout-*/ (36,937 files) + │ + ▼ +Ingest (photo-ingest flow) → fingerprint sha256+dhash → dedup vs by_date + │ 12,326 new / 225 dups skipped / 272 HEIC deferred + ▼ +Quality scan (PIL verdicts) → keep / delete_candidate (dark/blurry) + │ 11,877 keep / 449 delete_candidate + ▼ +Dashboard review (http://192.168.20.13:8092/review) → approve/reject + │ + ▼ +merge_orphans (approved only) → by_date/sorted_pictures_holder/YYYY/MM-Mon + │ + ▼ +Immich import (checksum dedup — no doubles) + │ + ▼ +ONLY NOW: delete from Google Photos → frees storage +``` + +## Storage locations (where the files live) + +| Path | Machine | Size | What it is | +|---|---|---|---| +| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` | .13 | 373 GB | **MASTER** photo archive, sorted YYYY/MM-Mon | +| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` | .13 | 69 GB | Backup mirror by subject | +| `/mnt/ubuntu_storage_3TB/archive/03_photos/{temp_transfer,Pictures,phone_dumps}` | .13 | 31 GB | Orphan folders awaiting review/merge | +| `/mnt/hd/immich/` | .35 | ~1.1T (60% of disk) | Immich store: library/thumbs/encoded-video/upload | +| `/mnt/data/takeout/` | .13 | 269 GB | **TRANSIENT** Takeout downloads+extract (do NOT back up) | +| `/mnt/data/photo_pipeline.db` | .13 | — | SQLite: image_hashes, known_paths, takeout_archives | + +## Key facts + +- **by_date is the single master.** Everything else is derived or transient. +- **Immich is a derived copy** of by_date (re-importable). Its thumbs/encoded-video are regenerable. +- **Takeout downloads (269GB) are transient** — safe to delete after verification. +- **Dedup is content-based** (sha256) — re-importing by_date never doubles. +- 382 HEIC files lack PIL support (skipped by quality scan) — safe on disk, need pillow-heif or conversion. + +## Backup to S3 (offsite) + +**Back up (the "house burns down" set):** +- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` (373 GB) — master +- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` (69 GB) — mirror +- ✅ Optionally: Immich `library/` on .35 (catches phone-only uploads) + +**SKIP (transient/regenerable):** +- ❌ `/mnt/data/takeout/` (269 GB — transient downloads) +- ❌ Immich `thumbs/`, `encoded-video/` (regenerable) +- ❌ The 14 zip archives (they're in Google until deleted) + +## Machine map + +- **.13** nixos-desktop — pipeline, dashboard :8092, Prefect :4200, n8n +- **.35** — Immich :2283, Caddy (prefect/photo-filter), pi-hole; SSH needs `-i ~/.ssh/id_ed25519_rsync` +- **.23** — borg backup server (2.7T USB via CIFS to .35) +- **.27** — this dev box + +## Review dashboard + +- URL: http://192.168.20.13:8092/review +- Filters: `?source=takeout`, `?status=delete_candidate` (flagged), `?status=keep` (clean) +- Bulk: select-all → Bulk Keep (approve) or Bulk Reject