From c9d09ac859eef9fde60d68a1221f8f75cccf344f Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Sat, 10 Oct 2026 10:18:00 +1100 Subject: [PATCH] docs: add comprehensive technical architecture specs for DSH, Portal, and Host Ports --- README.md | 7 +- docs/README.md | 28 ++++++-- docs/dsh-architecture.md | 121 +++++++++++++++++++++++++++++++ docs/host-network-ports.md | 75 +++++++++++++++++++ docs/portal-architecture.md | 139 ++++++++++++++++++++++++++++++++++++ 5 files changed, 363 insertions(+), 7 deletions(-) create mode 100644 docs/dsh-architecture.md create mode 100644 docs/host-network-ports.md create mode 100644 docs/portal-architecture.md diff --git a/README.md b/README.md index d0f3f58..a33f1f8 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,12 @@ ssh sam@192.168.20.13 'cd /home/sam/Docker/Containers/dsh && docker compose buil pi host (see `deploy/pi-dash-sync.sh`; run it on a cron to keep it fresh). - Kids' accounts get a reduced view (no AI/chat, no admin-only items). -See `docs/` for the website inventory, the audio plan, and the dsh plugin plan. +See `docs/` for complete technical specifications: +- **[DeepSeek Harness (DSH) Architecture](docs/dsh-architecture.md)** +- **[Console Portal Architecture](docs/portal-architecture.md)** +- **[Host Network & Port Registry](docs/host-network-ports.md)** +- **[Websites & Services Inventory](docs/websites.md)** +- **[Audio Plan](docs/audio-plan.md)** & **[DSH Plugin Plan](docs/dsh-plugin-plan.md)** ## How to get files into the tools (e.g. GIMP) Upload from your browser via the console → **Files** (top nav). Files appear in GIMP (and video/audio) under `/media//`. GIMP runs on `.13`, so diff --git a/docs/README.md b/docs/README.md index 0c87810..2af15d7 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,8 +1,24 @@ -# Docs +# Family Home Lab Technical Documentation -Future phase — document/library tools (e.g. paperless-ngx, BookStack, or an internal wiki). +This directory contains architectural specifications, component manuals, integration tables, and network registries for the Family Home Lab ecosystem. -- Category colour assigned: teal `#2a9d99`. -- No service ships from this folder yet; design decisions belong in plan.md/AGENT.md - when a tool is chosen. -- User permission flag (`can_docs`) already exists in the portal user model, default off. \ No newline at end of file +## Core Technical Reference Specs + +- **[DeepSeek Harness (DSH) Architecture](dsh-architecture.md)**: + Multi-instance chat engine, per-user container allocation (`sam`, `jo`, `harry`, `finn`), ports (`3081–3084`), JSON session persistence, multi-model routing table (OmniRoute, OpenRouter, Whisper, Jervis Voice Agent), CSP framing, and API endpoints. + +- **[Console Portal Architecture](portal-architecture.md)**: + FastAPI application spec, user database schema, access control entitlement flags (`can_chat`, `can_image`, `can_audio`, `can_docs`), tool catalogue structure (`tools.py`), host-side `service-controller` bridge, ON/OFF RAM management, and Garage S3 file integration. + +- **[Host Network & Port Registry](host-network-ports.md)**: + Single source of truth network topology and port allocations across hosts `.13` (containers/models), `.35` (Caddy/Gitea/Outline), `.27` (dev workstation), and `.51`. + +--- + +## Component Plans & Features + +- **[Audio Plan](audio-plan.md)**: MIDI transcription (Basic Pitch) and sheet music generation (MuScriptor). +- **[DSH Plugins Shortlist](dsh-plugins.md)**: Curated index of DeepSeek Harness extension plugins. +- **[DSH Plugin Plan](dsh-plugin-plan.md)**: Roadmap for integrating summary and media plugins into DSH. +- **[Websites & Services Inventory](websites.md)**: Catalogue of public domains and upstream destinations. +- **[Interactive Architecture Map](family-home-lab-map.html)**: Visual HTML component map. diff --git a/docs/dsh-architecture.md b/docs/dsh-architecture.md new file mode 100644 index 0000000..97e227e --- /dev/null +++ b/docs/dsh-architecture.md @@ -0,0 +1,121 @@ +# DeepSeek Harness (DSH) — Architecture & Integration Spec + +## 1. System Overview + +**DeepSeek Harness (`dsh`)** is a single-user, multi-tenant streaming chat web application that serves as the AI chat interface for Family Home Lab members. Instead of running a single shared application with multi-user authentication, DSH is deployed as **isolated, per-user container instances** on `.13`. + +The portal (`console.lab.audasmedia.com.au`) embeds each user's dedicated DSH instance inline via iframe or links directly to their subdomains (`https://dsh-{username}.lab.audasmedia.com.au`). + +--- + +## 2. Multi-Tenancy & Port Allocation + +Each family member gets a dedicated DSH container instance running on host `.13`: + +| User | Subdomain | Host Port | Container Name | Host Workspace Directory | +| :--- | :--- | :--- | :--- | :--- | +| **Sam** | `dsh-sam.lab.audasmedia.com.au` | `3081` | `dsh-sam` | `/mnt/data/family-home-lab/dsh/sam` | +| **Jo** | `dsh-jo.lab.audasmedia.com.au` | `3082` | `dsh-jo` | `/mnt/data/family-home-lab/dsh/jo` | +| **Harry** | `dsh-harry.lab.audasmedia.com.au` | `3083` | `dsh-harry` | `/mnt/data/family-home-lab/dsh/harry` | +| **Finn** | `dsh-finn.lab.audasmedia.com.au` | `3084` | `dsh-finn` | `/mnt/data/family-home-lab/dsh/finn` | + +### Reverse Proxy & Routing +- Caddy on `.35` terminates HTTPS for `dsh-*.lab.audasmedia.com.au` and proxies traffic to `.13:{port}`. +- Portal `tools.py` dynamically resolves the user's DSH URL via `_dsh_for_user(user)` based on their authenticated portal session. + +--- + +## 3. Storage & Session Persistence + +Chat history and sessions are stored directly as JSON files inside each container's `/workspace/sessions/` directory (mapped to host `/mnt/data/family-home-lab/dsh/{user}`): + +### Session Directory Structure +``` +/workspace/ +├── sessions/ +│ ├── main.json +│ ├── session-172840192.json +│ └── ... +└── uploads/ +``` + +### Session File Schema (`sessions/{id}.json`) +```json +{ + "id": "session-172840192", + "name": "Project discussion", + "updated": "2026-10-08T19:30:00Z", + "messages": [ + { + "role": "user", + "content": "Summarise the home lab setup", + "timestamp": 172840191 + }, + { + "role": "assistant", + "content": "The family home lab consists of...", + "timestamp": 172840192 + } + ] +} +``` + +--- + +## 4. Multi-Model Routing & External Integration Table + +DSH routes user requests across specialized AI services and local house infrastructure: + +| Function | Backend Endpoint | Model / Engine | Environment Variable | +| :--- | :--- | :--- | :--- | +| **Text Chat (SSE Stream)** | `http://192.168.20.13:20129/v1` | `auto/best-chat` (OmniRoute) | `DSH_LLM_BASE`, `DSH_LLM_MODEL` | +| **Vision (Image Ingest)** | `https://openrouter.ai/api/v1` | `openai/gpt-5` | `DSH_VISION_BASE`, `DSH_VISION_MODEL` | +| **Speech-to-Text (STT)** | `http://192.168.20.13:5000/transcribe` | `faster-whisper` (`voice_whisper`) | `DSH_WHISPER_URL` | +| **Voice Agent (Action/TTS)** | `http://192.168.20.13:8501/voice` | Jervis Voice Agent | `DSH_VOICE_AGENT_URL` | +| **Image Generation** | `https://openrouter.ai/api/v1` | `openai/gpt-image-1` | `DSH_IMAGE_MODEL` | +| **Video Generation** | `https://openrouter.ai/api/v1` | `z-ai/glm-5v-turbo` | `DSH_VIDEO_MODEL` | + +--- + +## 5. API Endpoints & Request Flow + +### Primary API Routes (`dsh/app.py`) + +| Method | Route | Description | +| :--- | :--- | :--- | +| `GET` | `/` | Serves main single-page chat UI (`templates/chat.html`). | +| `POST` | `/chat/stream` | Primary chat endpoint. Receives `{message, session_id}`,Streams response via Server-Sent Events (SSE). | +| `POST` | `/transcribe` | Uploads `.wav`/`.mp3`/`.m4a` file; forwards to `voice_whisper` (`:5000`); returns transcribed text. | +| `POST` | `/upload-image` | Uploads image; forwards to Vision model (`openai/gpt-5`) with user prompt; returns image description. | +| `POST` | `/voice-agent` | Forwards prompt to Jervis Voice Agent (`:8501`); triggers house audio/action. | +| `POST` | `/generate-image` | Generates image via OpenRouter API and saves output to workspace. | +| `POST` | `/generate-video` | Generates video via OpenRouter API and saves output to workspace. | +| `GET` | `/sessions` | Returns list of all session summaries for the current user. | +| `POST` | `/sessions/new` | Creates a new conversation session. | +| `POST` | `/sessions/{id}/delete` | Deletes a conversation session file. | + +--- + +## 6. Embedding & Security Headers + +To support seamless inline framing inside the Family Home Lab Console (`/tool/dsh`): +- **Content Security Policy (CSP)**: `frame-ancestors https://console.lab.audasmedia.com.au` +- **X-Frame-Options**: Explicitly omitted (not set to `DENY` or `SAMEORIGIN`), allowing Caddy and Portal to frame DSH cleanly. + +--- + +## 7. Environment Variables Reference + +| Variable | Default Value | Purpose | +| :--- | :--- | :--- | +| `DSH_USER` | `user` | Name of the user (e.g., `sam`, `jo`). | +| `DSH_SLOT` | `Welcome to DeepSeek Harness` | Subtitle/greeting shown in the header. | +| `DSH_LLM_BASE` | `http://192.168.20.13:20129/v1` | OmniRoute OpenAI-compatible base URL. | +| `DSH_LLM_MODEL` | `auto/best-chat` | Default model passed to OmniRoute. | +| `DSH_LLM_KEY` | `""` | Optional API key for OmniRoute. | +| `DSH_SYSTEM_PROMPT` | `"You are {USER_NAME}'s helpful DeepSeek assistant..."` | Custom system prompt per user instance. | +| `DSH_VISION_BASE` | `https://openrouter.ai/api/v1` | OpenRouter base URL for multimodal vision. | +| `DSH_VISION_MODEL` | `openai/gpt-5` | Multimodal model for image analysis. | +| `DSH_WHISPER_URL` | `http://192.168.20.13:5000/transcribe` | Local faster-whisper STT service. | +| `DSH_VOICE_AGENT_URL` | `http://192.168.20.13:8501/voice` | Jervis Voice Agent execution endpoint. | +| `DSH_HISTORY_DIR` | `/workspace` | Mount point for history and session storage. | diff --git a/docs/host-network-ports.md b/docs/host-network-ports.md new file mode 100644 index 0000000..8c3fae5 --- /dev/null +++ b/docs/host-network-ports.md @@ -0,0 +1,75 @@ +# Host Network & Port Registry — Family Home Lab + +## 1. Multi-Machine Network Topology + +| Host Alias | IP Address | Machine Purpose & Role | +| :--- | :--- | :--- | +| **`.13`** (`nixos-desktop`) | `192.168.20.13` | **Primary Application Host**: Docker containers, Webtop applications, DSH instances, AI models, PostgreSQL, Garage S3. | +| **`.35`** (`nixos-server`) | `192.168.20.35` | **Edge & Infrastructure Host**: Caddy reverse proxy, Gitea (`:3001`), Outline Wiki (`:3000`), Vikunja Tasks (`:3456`). | +| **`.27`** | `192.168.20.27` | **Development Workstation**: Local coding, testing, git pushes. | +| **`.51`** | `192.168.20.51` | **Secondary Node**: Configured via Gitea assets. | + +--- + +## 2. Port Registry on Primary Host (`.13`) + +### Core Console Stack +| Port | Service Name | Protocol / Interface | Access / Description | +| :--- | :--- | :--- | :--- | +| **8500** | Portal (`family-home-lab-portal-1`) | HTTP (FastAPI) | Main family dashboard (`console.lab...`). | +| **8443** | `service-controller` | HTTP (FastAPI + `X-Controller-Token`) | Host-side systemd user unit on `.13` for container/service toggles. | +| **3900** | Garage S3 API | S3 Protocol | S3 bucket storage endpoint. | +| **3902** | Garage Admin UI | HTTP | Object storage administration interface. | +| **5432** | PostgreSQL (`fhl`) | PostgreSQL | Dedicated container database (`pgvector`). | +| **6379** | Redis | Redis | Cache and broker backend. | +| **5672 / 15672** | RabbitMQ | AMQP / Management | Task queue broker. | + +### Per-User DeepSeek Harness (DSH) Instances +| Port | Container Name | Target User | Workspace Path | +| :--- | :--- | :--- | :--- | +| **3081** | `dsh-sam` | Sam | `/mnt/data/family-home-lab/dsh/sam` | +| **3082** | `dsh-jo` | Jo | `/mnt/data/family-home-lab/dsh/jo` | +| **3083** | `dsh-harry` | Harry | `/mnt/data/family-home-lab/dsh/harry` | +| **3084** | `dsh-finn` | Finn | `/mnt/data/family-home-lab/dsh/finn` | + +### Media & Webtop GUI Applications +| Port | Application | Technology | Notes | +| :--- | :--- | :--- | :--- | +| **8083** | Video Editor (Kdenlive) | LinuxServer Webtop | Togglable (~237 MB idle). | +| **8084** | Audio Editor (Audacity) | LinuxServer Webtop | Togglable (~201 MB idle). | +| **8085** | GIMP | LinuxServer Webtop | Togglable (~233 MB idle). | +| **8086** | LMMS Music Studio | LinuxServer Webtop | Togglable (~798 MB idle; Xvfb clamped to 2560x1440). | +| **8487** | Photopea | Container (`texasbullphotography`) | In-browser Photoshop alternative. | + +### AI, Speech & House Automation Infrastructure +| Port | Service | Description | +| :--- | :--- | :--- | +| **20129** | OmniRoute API Gateway | OpenAI-compatible LLM router (`auto/best-chat`). | +| **5000** | `voice_whisper` | Local `faster-whisper` speech-to-text service. | +| **8501** | Jervis Voice Agent | Voice execution engine with Snapcast audio output. | + +--- + +## 3. Port Registry on Edge Host (`.35`) + +| Port | Service | Public Domain | +| :--- | :--- | :--- | +| **80 / 443** | Caddy Reverse Proxy | Handles HTTPS certs for `*.lab.audasmedia.com.au`. | +| **3001** | Gitea Web | `gitea.lab.audasmedia.com.au` | +| **2222** | Gitea SSH | Git SSH remote port. | +| **3000** | Outline Wiki | `outline.lab.audasmedia.com.au` | +| **3456** | Vikunja Tasks | `tasks.lab.audasmedia.com.au` | + +--- + +## 4. Reverse Proxy Mapping (`*.lab.audasmedia.com.au`) + +| External Domain | Caddy Upstream Destination | +| :--- | :--- | +| `console.lab.audasmedia.com.au` | `http://192.168.20.13:8500` | +| `dsh-sam.lab.audasmedia.com.au` | `http://192.168.20.13:3081` | +| `dsh-jo.lab.audasmedia.com.au` | `http://192.168.20.13:3082` | +| `dsh-harry.lab.audasmedia.com.au` | `http://192.168.20.13:3083` | +| `dsh-finn.lab.audasmedia.com.au` | `http://192.168.20.13:3084` | +| `gitea.lab.audasmedia.com.au` | `http://192.168.20.35:3001` | +| `outline.lab.audasmedia.com.au` | `http://192.168.20.35:3000` | diff --git a/docs/portal-architecture.md b/docs/portal-architecture.md new file mode 100644 index 0000000..eff586e --- /dev/null +++ b/docs/portal-architecture.md @@ -0,0 +1,139 @@ +# Console Portal Architecture & Integration Spec + +## 1. System Overview + +The **Family Home Lab Console Portal** (`portal/`) serves as the central unified dashboard, authentication gateway, file manager, and service control plane for the home lab (`console.lab.audasmedia.com.au`). + +It is built with **FastAPI**, **Jinja2 templates**, **HTMX**, and **PostgreSQL**, running as container `family-home-lab-portal-1` on host `.13:8500`. + +--- + +## 2. Technology Stack & Architecture + +``` + Browser (HTMX / Vanilla JS) + │ + ▼ + Caddy Reverse Proxy (.35:443) ──[HTTPS]──► Portal Container (.13:8500) + │ + ┌───────────────────┬───────────────────────┼───────────────────┐ + ▼ ▼ ▼ ▼ + PostgreSQL (pg16) Redis / RabbitMQ Garage (S3) service-controller + (portal DB) (task queue / cache) (.13:3900) (.13:8443) +``` + +- **Framework**: FastAPI (async Python 3.12) +- **Frontend**: Jinja2 HTML templates + HTMX (interactivity) + CSS custom tokens (`DESIGN.md` sticker system) +- **Database**: PostgreSQL (`pgvector/pgvector:pg16`), SQLAlchemy async ORM (`asyncpg`) +- **Queue / Async Tasks**: Celery + Redis + RabbitMQ +- **Object Storage**: Garage S3 (`:3900`) +- **Service Control**: Host-side `service-controller` (`:8443`) + +--- + +## 3. Database Schema & User Entitlements + +The user model (`portal/database.py`) defines authentication credentials, administrative roles, and feature capability flags: + +### `User` Entity Schema +```python +class User(Base): + __tablename__ = "users" + + id: Mapped[int] = mapped_column(primary_key=True) + username: Mapped[str] = mapped_column(String(50), unique=True) + password_hash: Mapped[str] = mapped_column(String(255)) + full_name: Mapped[str] = mapped_column(String(100)) + is_admin: Mapped[bool] = mapped_column(Boolean, default=False) + + # Per-user capability flags + can_chat: Mapped[bool] = mapped_column(Boolean, default=True) + can_image: Mapped[bool] = mapped_column(Boolean, default=True) + can_video: Mapped[bool] = mapped_column(Boolean, default=True) + can_audio: Mapped[bool] = mapped_column(Boolean, default=True) + can_docs: Mapped[bool] = mapped_column(Boolean, default=False) +``` + +### Access Control Rules (`tools_for_user(user)`) +- **Admins (`is_admin=True`)**: Unrestricted access to all tools, categories, and service toggles. +- **Non-Admin Family Members**: + - Open categories: `media` (Media & Fun), `data`, `storage`, `home`, `home-mgmt`, `dev`, `portfolio`. + - Feature-gated categories: `chat`/`ai` (`can_chat`), `image` (`can_image`), `video` (`can_video`), `audio` (`can_audio`), `docs` (`can_docs`). + - Gated tools: Infrastructure (`infra`) and Network (`network`) tools marked `admin=True` are filtered out. + +--- + +## 4. Tool Catalogue (`portal/tools.py`) + +Tools are registered in `TOOLS = _cat()` using the `Tool` dataclass: + +```python +@dataclass(frozen=True) +class Tool: + id: str + name: str + category: str + description: str + url: str + mode: str = "tab" # "embed" or "tab" + status: str = "online" # initial visual default + external: bool = True + private: bool = False + lan: bool = False # LAN-only reachable badge + login: bool = False # Has separate login badge + admin: bool = False # Admins only + admin_service: bool = False # Service control admin only + service_id: str | None = None # Controller allowlist ID + ram_mb: int | None = None # Estimated idle RAM footprint + togglable: bool = False # Displays ON/OFF service strip +``` + +--- + +## 5. ON/OFF Service Controller Integration + +To allow family members to free up RAM on `.13` without exposing `/var/run/docker.sock` to the web portal, all container/systemd actions pass through a **token-gated, host-side controller process**. + +### Architecture +1. **Host Process**: `service-controller` runs natively on host `.13` as a systemd user unit (`family-service-controller.service`) under user `sam` on `127.0.0.1:8443`. +2. **Security Boundary**: The controller enforces an allowlist (`deploy/service-controller/services.toml`). Requests for unlisted services return `404`. +3. **Portal Proxy Endpoints**: + - `GET /api/services`: Fetches live status for allowlisted services from `http://host.docker.internal:8443/services`. + - `POST /service/{id}/{action}`: Sends `start` or `stop` command with `X-Controller-Token` header. + +### Togglable Services Allowlist +| Service ID | Target / Kind | Idle RAM | Group | Allowed Users | +| :--- | :--- | :--- | :--- | :--- | +| `gimp` | `family-home-lab-gimp-1` (Container) | ~233 MB | `media` | `sam`, `finn`, `harry`, `jo` | +| `video-editor` | `family-home-lab-video-editor-1` (Container) | ~237 MB | `media` | `sam`, `finn`, `harry`, `jo` | +| `audio-editor` | `family-home-lab-audio-editor-1` (Container) | ~201 MB | `media` | `sam`, `finn`, `harry`, `jo` | +| `lmms` | `lmms` (Container) | ~798 MB | `media` | `sam`, `finn`, `harry`, `jo` | +| `paperclip` | `paperclipai` (systemd user) | ~495 MB | `admin` | `sam` | +| `prefect-worker` | `prefect-worker` (systemd user) | ~129 MB | `pipeline` | `sam` | +| `photo-dashboard` | `photo-dashboard` (systemd user) | ~112 MB | `pipeline` | `sam` | +| `worldmonitor` | 4-container group (Container) | ~120 MB | `admin` | `sam` | + +--- + +## 6. Garage S3 & File Management + +The portal includes an integrated S3 file manager (`portal/s3.py`, `portal/templates/files.html`): +- Connects to internal Garage S3 endpoint (`http://garage:3900`). +- Each user gets a personal bucket (`sam`, `jo`, `harry`, `finn`) plus access to `shared-media`. +- File actions: Upload, Download, Delete, Preview, Create Directory. + +--- + +## 7. Operational & Deployment Commands + +Deployment directory on host `.13`: `/home/sam/Docker/Containers/family-home-lab` + +```bash +# Sync local changes from .27 to .13 +rsync -avz --exclude='.git' --exclude='.venv' \ + /home/sam/home_network/custom_tools/family_home_lab/ \ + sam@192.168.20.13:/home/sam/Docker/Containers/family-home-lab/ + +# Rebuild and restart portal container on .13 +ssh sam@192.168.20.13 "cd /home/sam/Docker/Containers/family-home-lab && docker compose build portal && docker compose up -d portal" +```