sam/family_home_lab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add comprehensive technical architecture specs for DSH, Portal, and Host Ports

Browse Source
This commit is contained in:
Sam Rolfe
2026-10-10 10:18:00 +11:00
parent a639824fd0
commit c9d09ac859
5 changed files with 363 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@@ -94,7 +94,12 @@ ssh sam@192.168.20.13 'cd /home/sam/Docker/Containers/dsh && docker compose buil
pi host (see `deploy/pi-dash-sync.sh`; run it on a cron to keep it fresh).
- Kids' accounts get a reduced view (no AI/chat, no admin-only items).
See `docs/` for the website inventory, the audio plan, and the dsh plugin plan.
See `docs/` for complete technical specifications:
- **[DeepSeek Harness (DSH) Architecture](docs/dsh-architecture.md)**
- **[Console Portal Architecture](docs/portal-architecture.md)**
- **[Host Network & Port Registry](docs/host-network-ports.md)**
- **[Websites & Services Inventory](docs/websites.md)**
- **[Audio Plan](docs/audio-plan.md)** & **[DSH Plugin Plan](docs/dsh-plugin-plan.md)**
## How to get files into the tools (e.g. GIMP)
Upload from your browser via the console → **Files** (top nav). Files appear in
GIMP (and video/audio) under `/media/<your-username>/`. GIMP runs on `.13`, so

28
docs/README.md
View File

@@ -1,8 +1,24 @@
# Docs
# Family Home Lab Technical Documentation
Future phase — document/library tools (e.g. paperless-ngx, BookStack, or an internal wiki).
This directory contains architectural specifications, component manuals, integration tables, and network registries for the Family Home Lab ecosystem.
- Category colour assigned: teal `#2a9d99`.
- No service ships from this folder yet; design decisions belong in plan.md/AGENT.md
when a tool is chosen.
- User permission flag (`can_docs`) already exists in the portal user model, default off.
## Core Technical Reference Specs
- **[DeepSeek Harness (DSH) Architecture](dsh-architecture.md)**:
Multi-instance chat engine, per-user container allocation (`sam`, `jo`, `harry`, `finn`), ports (`3081–3084`), JSON session persistence, multi-model routing table (OmniRoute, OpenRouter, Whisper, Jervis Voice Agent), CSP framing, and API endpoints.
- **[Console Portal Architecture](portal-architecture.md)**:
FastAPI application spec, user database schema, access control entitlement flags (`can_chat`, `can_image`, `can_audio`, `can_docs`), tool catalogue structure (`tools.py`), host-side `service-controller` bridge, ON/OFF RAM management, and Garage S3 file integration.
- **[Host Network & Port Registry](host-network-ports.md)**:
Single source of truth network topology and port allocations across hosts `.13` (containers/models), `.35` (Caddy/Gitea/Outline), `.27` (dev workstation), and `.51`.
---
## Component Plans & Features
- **[Audio Plan](audio-plan.md)**: MIDI transcription (Basic Pitch) and sheet music generation (MuScriptor).
- **[DSH Plugins Shortlist](dsh-plugins.md)**: Curated index of DeepSeek Harness extension plugins.
- **[DSH Plugin Plan](dsh-plugin-plan.md)**: Roadmap for integrating summary and media plugins into DSH.
- **[Websites & Services Inventory](websites.md)**: Catalogue of public domains and upstream destinations.
- **[Interactive Architecture Map](family-home-lab-map.html)**: Visual HTML component map.

121
docs/dsh-architecture.md Normal file
View File

@@ -0,0 +1,121 @@
# DeepSeek Harness (DSH) — Architecture & Integration Spec
## 1. System Overview
**DeepSeek Harness (`dsh`)** is a single-user, multi-tenant streaming chat web application that serves as the AI chat interface for Family Home Lab members. Instead of running a single shared application with multi-user authentication, DSH is deployed as **isolated, per-user container instances** on `.13`.
The portal (`console.lab.audasmedia.com.au`) embeds each user's dedicated DSH instance inline via iframe or links directly to their subdomains (`https://dsh-{username}.lab.audasmedia.com.au`).
---
## 2. Multi-Tenancy & Port Allocation
Each family member gets a dedicated DSH container instance running on host `.13`:
| User | Subdomain | Host Port | Container Name | Host Workspace Directory |
| :--- | :--- | :--- | :--- | :--- |
| **Sam** | `dsh-sam.lab.audasmedia.com.au` | `3081` | `dsh-sam` | `/mnt/data/family-home-lab/dsh/sam` |
| **Jo** | `dsh-jo.lab.audasmedia.com.au` | `3082` | `dsh-jo` | `/mnt/data/family-home-lab/dsh/jo` |
| **Harry** | `dsh-harry.lab.audasmedia.com.au` | `3083` | `dsh-harry` | `/mnt/data/family-home-lab/dsh/harry` |
| **Finn** | `dsh-finn.lab.audasmedia.com.au` | `3084` | `dsh-finn` | `/mnt/data/family-home-lab/dsh/finn` |
### Reverse Proxy & Routing
- Caddy on `.35` terminates HTTPS for `dsh-*.lab.audasmedia.com.au` and proxies traffic to `.13:{port}`.
- Portal `tools.py` dynamically resolves the user's DSH URL via `_dsh_for_user(user)` based on their authenticated portal session.
---
## 3. Storage & Session Persistence
Chat history and sessions are stored directly as JSON files inside each container's `/workspace/sessions/` directory (mapped to host `/mnt/data/family-home-lab/dsh/{user}`):
### Session Directory Structure
```
/workspace/
├── sessions/
│ ├── main.json
│ ├── session-172840192.json
│ └── ...
└── uploads/
```
### Session File Schema (`sessions/{id}.json`)
```json
{
"id": "session-172840192",
"name": "Project discussion",
"updated": "2026-10-08T19:30:00Z",
"messages": [
{
"role": "user",
"content": "Summarise the home lab setup",
"timestamp": 172840191
},
{
"role": "assistant",
"content": "The family home lab consists of...",
"timestamp": 172840192
}
]
}
```
---
## 4. Multi-Model Routing & External Integration Table
DSH routes user requests across specialized AI services and local house infrastructure:
| Function | Backend Endpoint | Model / Engine | Environment Variable |
| :--- | :--- | :--- | :--- |
| **Text Chat (SSE Stream)** | `http://192.168.20.13:20129/v1` | `auto/best-chat` (OmniRoute) | `DSH_LLM_BASE`, `DSH_LLM_MODEL` |
| **Vision (Image Ingest)** | `https://openrouter.ai/api/v1` | `openai/gpt-5` | `DSH_VISION_BASE`, `DSH_VISION_MODEL` |
| **Speech-to-Text (STT)** | `http://192.168.20.13:5000/transcribe` | `faster-whisper` (`voice_whisper`) | `DSH_WHISPER_URL` |
| **Voice Agent (Action/TTS)** | `http://192.168.20.13:8501/voice` | Jervis Voice Agent | `DSH_VOICE_AGENT_URL` |
| **Image Generation** | `https://openrouter.ai/api/v1` | `openai/gpt-image-1` | `DSH_IMAGE_MODEL` |
| **Video Generation** | `https://openrouter.ai/api/v1` | `z-ai/glm-5v-turbo` | `DSH_VIDEO_MODEL` |
---
## 5. API Endpoints & Request Flow
### Primary API Routes (`dsh/app.py`)
| Method | Route | Description |
| :--- | :--- | :--- |
| `GET` | `/` | Serves main single-page chat UI (`templates/chat.html`). |
| `POST` | `/chat/stream` | Primary chat endpoint. Receives `{message, session_id}`,Streams response via Server-Sent Events (SSE). |
| `POST` | `/transcribe` | Uploads `.wav`/`.mp3`/`.m4a` file; forwards to `voice_whisper` (`:5000`); returns transcribed text. |
| `POST` | `/upload-image` | Uploads image; forwards to Vision model (`openai/gpt-5`) with user prompt; returns image description. |
| `POST` | `/voice-agent` | Forwards prompt to Jervis Voice Agent (`:8501`); triggers house audio/action. |
| `POST` | `/generate-image` | Generates image via OpenRouter API and saves output to workspace. |
| `POST` | `/generate-video` | Generates video via OpenRouter API and saves output to workspace. |
| `GET` | `/sessions` | Returns list of all session summaries for the current user. |
| `POST` | `/sessions/new` | Creates a new conversation session. |
| `POST` | `/sessions/{id}/delete` | Deletes a conversation session file. |
---
## 6. Embedding & Security Headers
To support seamless inline framing inside the Family Home Lab Console (`/tool/dsh`):
- **Content Security Policy (CSP)**: `frame-ancestors https://console.lab.audasmedia.com.au`
- **X-Frame-Options**: Explicitly omitted (not set to `DENY` or `SAMEORIGIN`), allowing Caddy and Portal to frame DSH cleanly.
---
## 7. Environment Variables Reference
| Variable | Default Value | Purpose |
| :--- | :--- | :--- |
| `DSH_USER` | `user` | Name of the user (e.g., `sam`, `jo`). |
| `DSH_SLOT` | `Welcome to DeepSeek Harness` | Subtitle/greeting shown in the header. |
| `DSH_LLM_BASE` | `http://192.168.20.13:20129/v1` | OmniRoute OpenAI-compatible base URL. |
| `DSH_LLM_MODEL` | `auto/best-chat` | Default model passed to OmniRoute. |
| `DSH_LLM_KEY` | `""` | Optional API key for OmniRoute. |
| `DSH_SYSTEM_PROMPT` | `"You are {USER_NAME}'s helpful DeepSeek assistant..."` | Custom system prompt per user instance. |
| `DSH_VISION_BASE` | `https://openrouter.ai/api/v1` | OpenRouter base URL for multimodal vision. |
| `DSH_VISION_MODEL` | `openai/gpt-5` | Multimodal model for image analysis. |
| `DSH_WHISPER_URL` | `http://192.168.20.13:5000/transcribe` | Local faster-whisper STT service. |
| `DSH_VOICE_AGENT_URL` | `http://192.168.20.13:8501/voice` | Jervis Voice Agent execution endpoint. |
| `DSH_HISTORY_DIR` | `/workspace` | Mount point for history and session storage. |

75
docs/host-network-ports.md Normal file
View File

@@ -0,0 +1,75 @@
# Host Network & Port Registry — Family Home Lab
## 1. Multi-Machine Network Topology
| Host Alias | IP Address | Machine Purpose & Role |
| :--- | :--- | :--- |
| **`.13`** (`nixos-desktop`) | `192.168.20.13` | **Primary Application Host**: Docker containers, Webtop applications, DSH instances, AI models, PostgreSQL, Garage S3. |
| **`.35`** (`nixos-server`) | `192.168.20.35` | **Edge & Infrastructure Host**: Caddy reverse proxy, Gitea (`:3001`), Outline Wiki (`:3000`), Vikunja Tasks (`:3456`). |
| **`.27`** | `192.168.20.27` | **Development Workstation**: Local coding, testing, git pushes. |
| **`.51`** | `192.168.20.51` | **Secondary Node**: Configured via Gitea assets. |
---
## 2. Port Registry on Primary Host (`.13`)
### Core Console Stack
| Port | Service Name | Protocol / Interface | Access / Description |
| :--- | :--- | :--- | :--- |
| **8500** | Portal (`family-home-lab-portal-1`) | HTTP (FastAPI) | Main family dashboard (`console.lab...`). |
| **8443** | `service-controller` | HTTP (FastAPI + `X-Controller-Token`) | Host-side systemd user unit on `.13` for container/service toggles. |
| **3900** | Garage S3 API | S3 Protocol | S3 bucket storage endpoint. |
| **3902** | Garage Admin UI | HTTP | Object storage administration interface. |
| **5432** | PostgreSQL (`fhl`) | PostgreSQL | Dedicated container database (`pgvector`). |
| **6379** | Redis | Redis | Cache and broker backend. |
| **5672 / 15672** | RabbitMQ | AMQP / Management | Task queue broker. |
### Per-User DeepSeek Harness (DSH) Instances
| Port | Container Name | Target User | Workspace Path |
| :--- | :--- | :--- | :--- |
| **3081** | `dsh-sam` | Sam | `/mnt/data/family-home-lab/dsh/sam` |
| **3082** | `dsh-jo` | Jo | `/mnt/data/family-home-lab/dsh/jo` |
| **3083** | `dsh-harry` | Harry | `/mnt/data/family-home-lab/dsh/harry` |
| **3084** | `dsh-finn` | Finn | `/mnt/data/family-home-lab/dsh/finn` |
### Media & Webtop GUI Applications
| Port | Application | Technology | Notes |
| :--- | :--- | :--- | :--- |
| **8083** | Video Editor (Kdenlive) | LinuxServer Webtop | Togglable (~237 MB idle). |
| **8084** | Audio Editor (Audacity) | LinuxServer Webtop | Togglable (~201 MB idle). |
| **8085** | GIMP | LinuxServer Webtop | Togglable (~233 MB idle). |
| **8086** | LMMS Music Studio | LinuxServer Webtop | Togglable (~798 MB idle; Xvfb clamped to 2560x1440). |
| **8487** | Photopea | Container (`texasbullphotography`) | In-browser Photoshop alternative. |
### AI, Speech & House Automation Infrastructure
| Port | Service | Description |
| :--- | :--- | :--- |
| **20129** | OmniRoute API Gateway | OpenAI-compatible LLM router (`auto/best-chat`). |
| **5000** | `voice_whisper` | Local `faster-whisper` speech-to-text service. |
| **8501** | Jervis Voice Agent | Voice execution engine with Snapcast audio output. |
---
## 3. Port Registry on Edge Host (`.35`)
| Port | Service | Public Domain |
| :--- | :--- | :--- |
| **80 / 443** | Caddy Reverse Proxy | Handles HTTPS certs for `*.lab.audasmedia.com.au`. |
| **3001** | Gitea Web | `gitea.lab.audasmedia.com.au` |
| **2222** | Gitea SSH | Git SSH remote port. |
| **3000** | Outline Wiki | `outline.lab.audasmedia.com.au` |
| **3456** | Vikunja Tasks | `tasks.lab.audasmedia.com.au` |
---
## 4. Reverse Proxy Mapping (`*.lab.audasmedia.com.au`)
| External Domain | Caddy Upstream Destination |
| :--- | :--- |
| `console.lab.audasmedia.com.au` | `http://192.168.20.13:8500` |
| `dsh-sam.lab.audasmedia.com.au` | `http://192.168.20.13:3081` |
| `dsh-jo.lab.audasmedia.com.au` | `http://192.168.20.13:3082` |
| `dsh-harry.lab.audasmedia.com.au` | `http://192.168.20.13:3083` |
| `dsh-finn.lab.audasmedia.com.au` | `http://192.168.20.13:3084` |
| `gitea.lab.audasmedia.com.au` | `http://192.168.20.35:3001` |
| `outline.lab.audasmedia.com.au` | `http://192.168.20.35:3000` |

139
docs/portal-architecture.md Normal file
View File

@@ -0,0 +1,139 @@
# Console Portal Architecture & Integration Spec
## 1. System Overview
The **Family Home Lab Console Portal** (`portal/`) serves as the central unified dashboard, authentication gateway, file manager, and service control plane for the home lab (`console.lab.audasmedia.com.au`).
It is built with **FastAPI**, **Jinja2 templates**, **HTMX**, and **PostgreSQL**, running as container `family-home-lab-portal-1` on host `.13:8500`.
---
## 2. Technology Stack & Architecture
```
Browser (HTMX / Vanilla JS)
│
▼
Caddy Reverse Proxy (.35:443) ──[HTTPS]──► Portal Container (.13:8500)
│
┌───────────────────┬───────────────────────┼───────────────────┐
▼ ▼ ▼ ▼
PostgreSQL (pg16) Redis / RabbitMQ Garage (S3) service-controller
(portal DB) (task queue / cache) (.13:3900) (.13:8443)
```
- **Framework**: FastAPI (async Python 3.12)
- **Frontend**: Jinja2 HTML templates + HTMX (interactivity) + CSS custom tokens (`DESIGN.md` sticker system)
- **Database**: PostgreSQL (`pgvector/pgvector:pg16`), SQLAlchemy async ORM (`asyncpg`)
- **Queue / Async Tasks**: Celery + Redis + RabbitMQ
- **Object Storage**: Garage S3 (`:3900`)
- **Service Control**: Host-side `service-controller` (`:8443`)
---
## 3. Database Schema & User Entitlements
The user model (`portal/database.py`) defines authentication credentials, administrative roles, and feature capability flags:
### `User` Entity Schema
```python
class User(Base):
__tablename__ = "users"
id: Mapped[int] = mapped_column(primary_key=True)
username: Mapped[str] = mapped_column(String(50), unique=True)
password_hash: Mapped[str] = mapped_column(String(255))
full_name: Mapped[str] = mapped_column(String(100))
is_admin: Mapped[bool] = mapped_column(Boolean, default=False)
# Per-user capability flags
can_chat: Mapped[bool] = mapped_column(Boolean, default=True)
can_image: Mapped[bool] = mapped_column(Boolean, default=True)
can_video: Mapped[bool] = mapped_column(Boolean, default=True)
can_audio: Mapped[bool] = mapped_column(Boolean, default=True)
can_docs: Mapped[bool] = mapped_column(Boolean, default=False)
```
### Access Control Rules (`tools_for_user(user)`)
- **Admins (`is_admin=True`)**: Unrestricted access to all tools, categories, and service toggles.
- **Non-Admin Family Members**:
- Open categories: `media` (Media & Fun), `data`, `storage`, `home`, `home-mgmt`, `dev`, `portfolio`.
- Feature-gated categories: `chat`/`ai` (`can_chat`), `image` (`can_image`), `video` (`can_video`), `audio` (`can_audio`), `docs` (`can_docs`).
- Gated tools: Infrastructure (`infra`) and Network (`network`) tools marked `admin=True` are filtered out.
---
## 4. Tool Catalogue (`portal/tools.py`)
Tools are registered in `TOOLS = _cat()` using the `Tool` dataclass:
```python
@dataclass(frozen=True)
class Tool:
id: str
name: str
category: str
description: str
url: str
mode: str = "tab" # "embed" or "tab"
status: str = "online" # initial visual default
external: bool = True
private: bool = False
lan: bool = False # LAN-only reachable badge
login: bool = False # Has separate login badge
admin: bool = False # Admins only
admin_service: bool = False # Service control admin only
service_id: str | None = None # Controller allowlist ID
ram_mb: int | None = None # Estimated idle RAM footprint
togglable: bool = False # Displays ON/OFF service strip
```
---
## 5. ON/OFF Service Controller Integration
To allow family members to free up RAM on `.13` without exposing `/var/run/docker.sock` to the web portal, all container/systemd actions pass through a **token-gated, host-side controller process**.
### Architecture
1. **Host Process**: `service-controller` runs natively on host `.13` as a systemd user unit (`family-service-controller.service`) under user `sam` on `127.0.0.1:8443`.
2. **Security Boundary**: The controller enforces an allowlist (`deploy/service-controller/services.toml`). Requests for unlisted services return `404`.
3. **Portal Proxy Endpoints**:
- `GET /api/services`: Fetches live status for allowlisted services from `http://host.docker.internal:8443/services`.
- `POST /service/{id}/{action}`: Sends `start` or `stop` command with `X-Controller-Token` header.
### Togglable Services Allowlist
| Service ID | Target / Kind | Idle RAM | Group | Allowed Users |
| :--- | :--- | :--- | :--- | :--- |
| `gimp` | `family-home-lab-gimp-1` (Container) | ~233 MB | `media` | `sam`, `finn`, `harry`, `jo` |
| `video-editor` | `family-home-lab-video-editor-1` (Container) | ~237 MB | `media` | `sam`, `finn`, `harry`, `jo` |
| `audio-editor` | `family-home-lab-audio-editor-1` (Container) | ~201 MB | `media` | `sam`, `finn`, `harry`, `jo` |
| `lmms` | `lmms` (Container) | ~798 MB | `media` | `sam`, `finn`, `harry`, `jo` |
| `paperclip` | `paperclipai` (systemd user) | ~495 MB | `admin` | `sam` |
| `prefect-worker` | `prefect-worker` (systemd user) | ~129 MB | `pipeline` | `sam` |
| `photo-dashboard` | `photo-dashboard` (systemd user) | ~112 MB | `pipeline` | `sam` |
| `worldmonitor` | 4-container group (Container) | ~120 MB | `admin` | `sam` |
---
## 6. Garage S3 & File Management
The portal includes an integrated S3 file manager (`portal/s3.py`, `portal/templates/files.html`):
- Connects to internal Garage S3 endpoint (`http://garage:3900`).
- Each user gets a personal bucket (`sam`, `jo`, `harry`, `finn`) plus access to `shared-media`.
- File actions: Upload, Download, Delete, Preview, Create Directory.
---
## 7. Operational & Deployment Commands
Deployment directory on host `.13`: `/home/sam/Docker/Containers/family-home-lab`
```bash
# Sync local changes from .27 to .13
rsync -avz --exclude='.git' --exclude='.venv' \
/home/sam/home_network/custom_tools/family_home_lab/ \
sam@192.168.20.13:/home/sam/Docker/Containers/family-home-lab/
# Rebuild and restart portal container on .13
ssh sam@192.168.20.13 "cd /home/sam/Docker/Containers/family-home-lab && docker compose build portal && docker compose up -d portal"
```