sam-4screen-desktop 2026-8-28:18:54:8
This commit is contained in:
4
.obsidian/workspace.json
vendored
4
.obsidian/workspace.json
vendored
@@ -209,8 +209,9 @@
|
||||
"templater-obsidian:Templater": false
|
||||
}
|
||||
},
|
||||
"active": "7066c9273c6fb936",
|
||||
"active": "8018aa52f0591df9",
|
||||
"lastOpenFiles": [
|
||||
"300 areas/360 Dev-Ops Network Computers/Photo Pipeline & Storage Map.md",
|
||||
"200 projects/220 Web Host Migration/Migration Work Log.md",
|
||||
"200 projects/220 Web Host Migration/Phase 5 — Decommission InMotion.md",
|
||||
"200 projects/220 Web Host Migration/Phase 3 — Local Infra Prep.md",
|
||||
@@ -236,7 +237,6 @@
|
||||
"000 daily/FAMILY S3 Storage Integration & Blueprint.md",
|
||||
"400 resources/450 Recipes/Easy Yogurt flat bread.md",
|
||||
"400 resources/450 Recipes/Chilli Sauce Hack.md",
|
||||
"000 daily/basketball training.md",
|
||||
"000 daily/ThinkPad Recommendations",
|
||||
"500 archive/510 Daily",
|
||||
"300 areas/305 Ideas Businesses",
|
||||
|
||||
@@ -121,19 +121,22 @@ aliases: [drive-map, filesystem-map]
|
||||
| **Boot** | `sda1` | 1G | — | — | `/boot` | — | `4D80-F99E` | EFI boot |
|
||||
| **Storage** | `sdc2` | 931G | 2.1M | **870G (1%)** | `/mnt/storage` | archive_storage | — | **EMPTY** — reformatted ext4 2026-05-30 (was NTFS Windows drive) |
|
||||
| **Data** | `sdb2` | 1.8T | 2.1M | **1.7T (1%)** | `/mnt/data` | Data | — | **EMPTY** — reformatted ext4 2026-05-30. **Planned Takeout landing zone** |
|
||||
| **Ubuntu Storage** | `sdd1` | 2.7T | 810G | **1.9T (30%)** | `/mnt/ubuntu_storage_3TB` | ubuntu_storage_3 | `037a542c-...` | Holds `archive/` (649G), `timeshift` (64G), `backup/` (35G). Backup target for .27 + .13 Borg repos |
|
||||
| **MaxtorBackup** | `sde1` | 1.4T | ~0 | — | `/mnt/maxtor_backup` | MaxtorBackup | `b0fa7768-...` | Now mounted — EMPTY (4KB). Old backup drive, content unknown/clear |
|
||||
| **Ubuntu Storage** | `sde1` | 2.7T | 874G | **1.9T (33%)** | `/mnt/ubuntu_storage_3TB` | ubuntu_storage_3 | `037a542c-...` | **USB drive** — holds `archive/` (photo master), `timeshift`, `backup/`. Backup target for .27 + .13 Borg repos. ⚠️ USB — unplugged Aug 21-26 2026 caused backup failures; replugged, letter shifted `sdd1→sde1` |
|
||||
| **MaxtorBackup** | `sdd1` | 1.4T | ~0 | — | `/mnt/maxtor_backup` | MaxtorBackup | `b0fa7768-...` | Old backup drive — only NixOS migration tarballs, no photos |
|
||||
|
||||
### Key Photo Location (NEW — moved 2026-05-30)
|
||||
### Key Photo Location (moved 2026-05-30; reconciled 2026-08-27)
|
||||
|
||||
```
|
||||
/mnt/ubuntu_storage_3TB/archive/03_photos/ ← 472G, 58,948 files
|
||||
├── by_date/ ← year→month organized
|
||||
├── by_subject/ ← subject-organized
|
||||
├── phone_dumps/ ← phone exports
|
||||
├── Pictures/ ← old Pictures folder
|
||||
├── temp_transfer/ ← event photos
|
||||
└── lily.jpg, lily2jpg.jpg
|
||||
/mnt/ubuntu_storage_3TB/archive/03_photos/ ← 473G total
|
||||
├── by_date/ 373G ← MASTER, 35,457 files, YYYY/MM-Mon
|
||||
├── by_subject/ 69G ← backup mirror (in Immich)
|
||||
├── phone_dumps/ 2G ← phone exports (awaiting review)
|
||||
├── Pictures/ 1G ← old Pictures (awaiting review)
|
||||
├── temp_transfer/ 28G ← event photos (awaiting review)
|
||||
└── lily2jpg.jpg
|
||||
|
||||
Takeout downloads (TRANSIENT — do NOT back up): /mnt/data/takeout/ = 269G
|
||||
Immich store: /mnt/hd/immich on .35 (library/thumbs/encoded-video)
|
||||
|
||||
archive/ also contains: 01_family, 02_projects, 04_education, 05_documents,
|
||||
06_creative, 07_notes, 08_transfer
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
tags: [photos, immich, pipeline, backup, google-takeout]
|
||||
created: 2026-08-27
|
||||
---
|
||||
|
||||
# Photo Pipeline & Storage Map
|
||||
|
||||
## What the pipeline does
|
||||
|
||||
```
|
||||
Google Photos (Takeout export, 134GB/14 zips)
|
||||
│ download → /mnt/data/takeout/incoming/ (.13)
|
||||
▼
|
||||
Extract → /mnt/data/takeout/takeout-*/ (36,937 files)
|
||||
│
|
||||
▼
|
||||
Ingest (photo-ingest flow) → fingerprint sha256+dhash → dedup vs by_date
|
||||
│ 12,326 new / 225 dups skipped / 272 HEIC deferred
|
||||
▼
|
||||
Quality scan (PIL verdicts) → keep / delete_candidate (dark/blurry)
|
||||
│ 11,877 keep / 449 delete_candidate
|
||||
▼
|
||||
Dashboard review (http://192.168.20.13:8092/review) → approve/reject
|
||||
│
|
||||
▼
|
||||
merge_orphans (approved only) → by_date/sorted_pictures_holder/YYYY/MM-Mon
|
||||
│
|
||||
▼
|
||||
Immich import (checksum dedup — no doubles)
|
||||
│
|
||||
▼
|
||||
ONLY NOW: delete from Google Photos → frees storage
|
||||
```
|
||||
|
||||
## Storage locations (where the files live)
|
||||
|
||||
| Path | Machine | Size | What it is |
|
||||
|---|---|---|---|
|
||||
| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` | .13 | 373 GB | **MASTER** photo archive, sorted YYYY/MM-Mon |
|
||||
| `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` | .13 | 69 GB | Backup mirror by subject |
|
||||
| `/mnt/ubuntu_storage_3TB/archive/03_photos/{temp_transfer,Pictures,phone_dumps}` | .13 | 31 GB | Orphan folders awaiting review/merge |
|
||||
| `/mnt/hd/immich/` | .35 | ~1.1T (60% of disk) | Immich store: library/thumbs/encoded-video/upload |
|
||||
| `/mnt/data/takeout/` | .13 | 269 GB | **TRANSIENT** Takeout downloads+extract (do NOT back up) |
|
||||
| `/mnt/data/photo_pipeline.db` | .13 | — | SQLite: image_hashes, known_paths, takeout_archives |
|
||||
|
||||
## Key facts
|
||||
|
||||
- **by_date is the single master.** Everything else is derived or transient.
|
||||
- **Immich is a derived copy** of by_date (re-importable). Its thumbs/encoded-video are regenerable.
|
||||
- **Takeout downloads (269GB) are transient** — safe to delete after verification.
|
||||
- **Dedup is content-based** (sha256) — re-importing by_date never doubles.
|
||||
- 382 HEIC files lack PIL support (skipped by quality scan) — safe on disk, need pillow-heif or conversion.
|
||||
|
||||
## Backup to S3 (offsite)
|
||||
|
||||
**Back up (the "house burns down" set):**
|
||||
- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_date` (373 GB) — master
|
||||
- ✅ `/mnt/ubuntu_storage_3TB/archive/03_photos/by_subject` (69 GB) — mirror
|
||||
- ✅ Optionally: Immich `library/` on .35 (catches phone-only uploads)
|
||||
|
||||
**SKIP (transient/regenerable):**
|
||||
- ❌ `/mnt/data/takeout/` (269 GB — transient downloads)
|
||||
- ❌ Immich `thumbs/`, `encoded-video/` (regenerable)
|
||||
- ❌ The 14 zip archives (they're in Google until deleted)
|
||||
|
||||
## Machine map
|
||||
|
||||
- **.13** nixos-desktop — pipeline, dashboard :8092, Prefect :4200, n8n
|
||||
- **.35** — Immich :2283, Caddy (prefect/photo-filter), pi-hole; SSH needs `-i ~/.ssh/id_ed25519_rsync`
|
||||
- **.23** — borg backup server (2.7T USB via CIFS to .35)
|
||||
- **.27** — this dev box
|
||||
|
||||
## Review dashboard
|
||||
|
||||
- URL: http://192.168.20.13:8092/review
|
||||
- Filters: `?source=takeout`, `?status=delete_candidate` (flagged), `?status=keep` (clean)
|
||||
- Bulk: select-all → Bulk Keep (approve) or Bulk Reject
|
||||
Reference in New Issue
Block a user