sam/obsidian-vault
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sam-4screen-desktop 2026-9-15:14:33:7

Browse Source
This commit is contained in:
Sam Rolfe
2026-09-15 14:33:07 +10:00
parent 8e70d10a22
commit 04e9e1e233
3 changed files with 22 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

109
300 areas/360 Dev-Ops Network Computers/Backup Health Check Commands.md
View File

@@ -1,109 +0,0 @@
---
created: 2026-05-20 12:00
modified: 2026-05-20 12:00
type: note
tags: [dev-ops, backup, system-config, network]
aliases: [backup-status-checks]
---
# [[Backup Health Check Commands]]
Quick reference for verifying backups across all machines.
---
## .27 — sam-4screen-desktop (Borg → .13)
```bash
# Service status
systemctl status borgbackup-job-home-to-13.service
# Timer status
systemctl status borgbackup-job-home-to-13.timer
# List snapshots
export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase)
borg list sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27
# Recent logs
sudo journalctl -u borgbackup-job-home-to-13.service --no-pager -n 10
# Check passphrase exists
ls /home/sam/.borg-passphrase
```
## .13 — nixos-desktop (Borg local + rsync → .35)
```bash
# Service status
systemctl status borgbackup-job-local-to-storage.service
# Timer status
systemctl status borgbackup-job-local-to-storage.timer
# Rsync timer
systemctl status borg-rsync-to-35.timer
# List snapshots
export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase)
borg list /mnt/ubuntu_storage_3TB/backup/borg/.13
# Also check .27 repo (stored locally on .13)
borg list /mnt/ubuntu_storage_3TB/backup/borg/.27
# Mount status
df -h /mnt/ubuntu_storage_3TB
# Recent logs
sudo journalctl -u borgbackup-job-local-to-storage.service --no-pager -n 10
# Fix permissions if sam can't read (borg runs as root)
sudo chown -R sam:users /mnt/ubuntu_storage_3TB/backup/borg/.13
```
## .35 — sam-ubuntu1 (Kopia + Backrest → .23 NFS)
```bash
# Kopia snapshots
docker exec kopia kopia snapshot list --all
# Kopia policy
docker exec kopia kopia policy show --global
# Backrest / Restic snapshots
docker exec -e RESTIC_PASSWORD=sam4jo backrest restic -r /repos snapshots
# NFS mount check
df -h /mnt/nfs_usb_3tb
# NFS share usage
du -sh /mnt/nfs_usb_3tb/kopia/ /mnt/nfs_usb_3tb/backups/
```
---
## Storage Stats
| Location | Machine | Size | Free | Repo |
|----------|---------|------|------|------|
| `/mnt/ubuntu_storage_3TB` | .13 | 2.7T | 2.6T | `.13/` + `.27/` |
| `/mnt/nfs_usb_3tb` | .35→.23 | 2.7T | 2.5T | Kopia (`46G`) + Restic (`11G`) |
---
## Manual Ad-Hoc Backup
```bash
# .27 — trigger backup immediately
sudo systemctl start borgbackup-job-home-to-13.service
# .13 — trigger backup immediately
sudo systemctl start borgbackup-job-local-to-storage.service
```
## Vorta GUI (.27 only)
- Repo: `sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27`
- Encryption: `repokey-blake2`
- SSH key: auto-detect
- Compression: LZMA
- Use for browsing/restoring snapshots — leave scheduler **disabled** (systemd timer handles it)

164
300 areas/360 Dev-Ops Network Computers/Backup System — Borg, Kopia & Restic.md
View File

@@ -1,164 +0,0 @@
---
created: 2026-05-20 09:30
modified: 2026-05-28
type: note
tags:
- dev-ops
- network
- backup
- system-config
- localhost
aliases:
- borg-backups
---
# [[Backup System — Borg, Kopia & Restic]]
## Overview
Three-machine backup with dual-copy (local + network) covering all critical data.
| Machine | Tool | Target | Schedule |
|---------|------|--------|----------|
| **.27** — sam-4screen-desktop | Borg (SSH) | `.13:/mnt/ubuntu_storage_3TB/backup/borg/.27` | Daily (after boot) |
| **.13** — nixos-desktop | Borg (local) | `/mnt/ubuntu_storage_3TB/backup/borg/.13` + rsync→.35/.23 | Daily 04:00 / 05:00 |
| **.13** — archive | Borg (SSH) | `.35:/mnt/nfs_usb_3tb/borg-repos/archive` | **NEW 2026-05-30** |
| **.35** — sam-ubuntu1 | Kopia + Backrest | `.23` NFS (`usb_3tb`) | Weekly Sun 22:00 |
---
## .27 — sam-4screen-desktop
**Source**: `/home/sam/` (excl: `.cache`, `.npm`, `node_modules`, `.venv`, `target`, `.cargo`, etc.)
**Target**: `sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27` via SSH (key: `~/.ssh/id_ed25519`)
**Schedule**: systemd timer — `OnCalendar=daily` (fires midnight, catches up after boot via Persistent)
**Prune**: 7 daily, 4 weekly, 3 monthly
**Passphrase**: `/home/sam/.borg-passphrase`
**Config**: `/etc/nixos/hosts/sam-4screen-desktop/backup.nix`
**Status**: `systemctl status borgbackup-job-home-to-13.service`
**Snapshots**: `borg list sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27`
## .13 — nixos-desktop
**Source**: Docker configs (`/home/sam/Docker/`), PocketBase (`/home/sam/pocketbase-data/`), dotfiles, `.ssh/`
**Primary**: `/mnt/ubuntu_storage_3TB/backup/borg/.13` (local ext4, 2.7T, 2.6T free)
**Secondary**: rsync → `sam@192.168.20.35:/mnt/nfs_usb_3tb/borg-repos/.13` → `.23` USB
**Schedule**: Borg daily 04:00 | rsync daily 05:00
**Prune**: 7 daily, 4 weekly, 3 monthly
**Passphrase**: `/home/sam/.borg-passphrase` (same as .27 recommended)
**Config**: `/etc/nixos/backup.nix`
**Snapshots**: `borg list /mnt/ubuntu_storage_3TB/backup/borg/.13`
> ⚠️ Borg runs as root — `postHook` chowns to `sam:users`. Known hosts issue: `/home/sam/.ssh/known_hosts` was a broken directory (fixed).
## .35 — sam-ubuntu1
**Tools**: Kopia + Backrest (Restic) — Docker containers
**Target**: `//192.168.20.23/usb_3tb` → `/mnt/nfs_usb_3tb/`
- Kopia repo: `/mnt/nfs_usb_3tb/kopia/backups/` (46G)
- Restic repo: `/mnt/nfs_usb_3tb/backups/restic/` (11G)
### Backup Sources
| Source | Tool | Status |
|--------|------|--------|
| `/Docker/Containers/` | Kopia | ✅ Weekly Sun 22:00 |
| `/docker-config`, `/docker-volumes` | Backrest (Restic) | ✅ Weekly Sun 23:00 |
| `/host_fs/root/trilium-data` | Backrest (Restic) | ✅ Weekly Sun 23:00 |
| **`/host_fs/mnt/hd`** ← _new_ | **Backrest (Restic)** | ✅ **Added 2026-05-28** |
| **`/host_fs/var/lib/docker/volumes`** | Kopia | ✅ Weekly Sun 22:00 |
### Retention
- Kopia: 10 latest, 4 weekly, 12 monthly, 3 annual
- Backrest: 10 latest (prune/check daily)
### Data Drive (My Passport)
.dev/vda1 — 1.8T NTFS — physical WD My Passport passed through to VM
**Mount point**: `/mnt/hd` (was incorrectly at `/mnt/my_passport` — fixed 2026-05-28)
**Contains**: Immich photos (40K), and all Docker service data (audiobookshelf, lidarr, radarr, sonarr, qbittorrent, etc.)
### VM-level
Proxmox Backup Server (.48) — full VM backup
## .23 — file-server
- 2.7T USB drive shared via CIFS/SMB to `.35`
- Backup repos: Kopia (46G) + Restic (13G) + Borg archive repo (new)
- 2.0T free
- Also stores: `sam/` (personal docs/music/photos) — to be moved to .13 archive
### Borg archive repo (new 2026-05-30)
- **Location**: `/mnt/nfs_usb_3tb/borg-repos/archive`
- **Source**: `.13` `/mnt/ubuntu_storage_3TB/archive/` (648G — photos, old web work, documents)
- **Encryption**: repokey-blake2
- **Compression**: auto,lzma
- **Run from .13**: `borg create sam@192.168.20.35:/mnt/nfs_usb_3tb/borg-repos/archive::archive-$(date +%Y%m%d) /mnt/ubuntu_storage_3TB/archive/`
## Restore Commands
### .27 restore
```bash
export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase)
borg extract sam@192.168.20.13:/mnt/ubuntu_storage_3TB/backup/borg/.27::<archive>
```
### .13 restore
```bash
export BORG_PASSPHRASE=$(cat /home/sam/.borg-passphrase)
borg extract /mnt/ubuntu_storage_3TB/backup/borg/.13::<archive>
```
## Not Covered
- `.27`: Git-tracked NixOS config (already covered by flakes)
- `.13`: Git-tracked NixOS config, `ollama`/`open-webui` Docker volumes
- `.35`: **`.23` file-server has no backup** — single point of failure for restic/kopia repos
- `.23`: Itself has no backup at all
- **`.13` archive** — now backed up to .23 Borg repo ✅
> **UPDATE 2026-09:** The **AWS S3 offsite** layer now covers the highest-value data (see below): photos, `.13` Borg repo, `.27` Borg repo, and `.13` home configs are offsite in `sam-offsite-backup`. This **mitigates the `.23` single-point-of-failure for that data** (a fire loses .23 but S3 survives). Full restore procedures in **[[Restore Runbook]]**.
## Offsite S3 (AWS) — added 2026-09
| Item | Detail |
|---|---|
| **Bucket** | `sam-offsite-backup` (region ap-southeast-2 / Sydney), SSE, versioning, lifecycle S3→Glacier Flexible→Deep Archive |
| **Tool** | AWS CLI `aws s3 sync` (run from `.13`) — append-only, no delete |
| **Daily job** | `offsite-sync.service` + timer, **23:00–05:00** (Melbourne), 6h window; runs `.13` repo → `.27` repo → home configs FIRST, then photos/archive |
| **Verifier + alert** | `offsite-verify.service` 05:15 daily — checks object-count growth, **ntfy push** to phone on no-progress/error; log at `/var/www/offsite/verify.log` |
| **Monitoring page** | `https://offsite.lab.audasmedia.com.au` (basic-auth `sam`) — live object counts per prefix, 7-night history, service state, disk. Auto-refresh 15 min |
| **IAM** | user `backup-offsite` (object-only, least privilege on the bucket) |
| **Cost** | ~$1–5 AUD/mo (photos/archive in Deep Archive; repos in Standard) |
| **Credentials** | `/home/sam/.aws-credentials.local` (600, on .13) + `aws-credentials.local` (.27) |
**Backed up to S3:** photos/by_date, archive/rest (Deep Archive) · borg/.27, borg/.13 (Standard) · home configs (`~/deployment`, voice/audio, `.agents`, `.pi`, etc., Deep Archive).
**Restore:** see **[[Restore Runbook]]** (per-machine local + S3 steps; Deep Archive restore = hours).
## Garage (local S3) — .13
| Item | Detail |
|---|---|
| **Role** | Local S3-compatible object store (`family-home-lab` compose); used by portal/media apps |
| **Version** | **v2.1.0** (upgraded from v1.0.1, 2026-09-05 — v1 wouldn't work with garage-webui; data auto-migrated on start, backup taken first) |
| **S3 API** | `[::]:3900` (region `garage`) |
| **S3 Web** | `[::]:3902` (vhost `.web.lab.audasmedia.com.au`, `index.html`) — ⚠️ **requires per-bucket auth; NOT anonymous static hosting** |
| **Admin API** | `[::]:3903` (v2 endpoints `/v2/...`); token in `garage.toml`/`.env` |
| **Web UI** | `garage-webui` container → port **3909**, published as `https://garage-ui.lab.audasmedia.com.au` (basic-auth `sam`) |
| **Config** | `/home/sam/Docker/Containers/family-home-lab/garage/garage.toml` |
| **Backup** | Bucket data = app data, included in `.13`/`.35` container backups per app; data+meta snapshot at `/mnt/data/family-home-lab/garage-backup-20260905-111241/` |
### ⚠️ Garage S3-web gotcha (learned 2026-09-09)
Garage's **S3 Web** (`:3902`) is **not** a drop-in anonymous static host. It rejects anonymous access (`403 Forbidden: Garage does not support anonymous access yet`) and needs **per-bucket web access + a read key**. For serving site images:
- **Use the S3 API** (boto3/aws CLI with a read-scoped key) for uploads, and
- **Serve images via the app / Caddy** (e.g. reverse-proxy to Garage with the read key, or a Caddy `file_server` dir) — do NOT expose `:3902` for anonymous browsing.
## Recent Changes
- **2026-05-28**: Discovered My Passport (`/dev/vda1`, 1.8T) had all real data but was mounted at `/mnt/my_passport` instead of `/mnt/hd`. Remounted to `/mnt/hd`. Immich photos (40K) and all Docker service data recovered.
- **2026-05-28**: Added `/host_fs/mnt/hd` to Backrest (Restic) backup plan.
- **2026-05-30**: Created Borg archive repo on `.23` (`/mnt/nfs_usb_3tb/borg-repos/archive`) to back up `.13` `/mnt/ubuntu_storage_3TB/archive/` (648G). First backup pending.
- **2026-05-30**: Reformatted `.13` `/mnt/storage/` (932G NTFS → ext4) and `/mnt/data/` (1.8T NTFS → ext4). Both now empty with 870G and 1.7T free respectively.