family_home_lab/docs/portal-architecture.md

Console Portal Architecture & Integration Spec

1. System Overview

The Family Home Lab Console Portal (portal/) serves as the central unified dashboard, authentication gateway, file manager, and service control plane for the home lab (console.lab.audasmedia.com.au).

It is built with FastAPI, Jinja2 templates, HTMX, and PostgreSQL, running as container family-home-lab-portal-1 on host .13:8500.

---

2. Technology Stack & Architecture

 Browser (HTMX / Vanilla JS)
      │
      ▼
 Caddy Reverse Proxy (.35:443)  ──[HTTPS]──►  Portal Container (.13:8500)
                                                    │
        ┌───────────────────┬───────────────────────┼───────────────────┐
        ▼                   ▼                       ▼                   ▼
 PostgreSQL (pg16)    Redis / RabbitMQ         Garage (S3)       service-controller
 (portal DB)          (task queue / cache)     (.13:3900)        (.13:8443)

• Framework: FastAPI (async Python 3.12)
• Frontend: Jinja2 HTML templates + HTMX (interactivity) + CSS custom tokens (DESIGN.md sticker system)
• Database: PostgreSQL (pgvector/pgvector:pg16), SQLAlchemy async ORM (asyncpg)
• Queue / Async Tasks: Celery + Redis + RabbitMQ
• Object Storage: Garage S3 (:3900)
• Service Control: Host-side service-controller (:8443)

---

3. Database Schema & User Entitlements

The user model (portal/database.py) defines authentication credentials, administrative roles, and feature capability flags:

User Entity Schema

class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(primary_key=True)
    username: Mapped[str] = mapped_column(String(50), unique=True)
    password_hash: Mapped[str] = mapped_column(String(255))
    full_name: Mapped[str] = mapped_column(String(100))
    is_admin: Mapped[bool] = mapped_column(Boolean, default=False)

    # Per-user capability flags
    can_chat: Mapped[bool] = mapped_column(Boolean, default=True)
    can_image: Mapped[bool] = mapped_column(Boolean, default=True)
    can_video: Mapped[bool] = mapped_column(Boolean, default=True)
    can_audio: Mapped[bool] = mapped_column(Boolean, default=True)
    can_docs: Mapped[bool] = mapped_column(Boolean, default=False)

Access Control Rules (tools_for_user(user))

• Admins (is_admin=True): Unrestricted access to all tools, categories, and service toggles.
• Non-Admin Family Members:
  • Open categories: media (Media & Fun), data, storage, home, home-mgmt, dev, portfolio.
  • Feature-gated categories: chat/ai (can_chat), image (can_image), video (can_video), audio (can_audio), docs (can_docs).
  • Gated tools: Infrastructure (infra) and Network (network) tools marked admin=True are filtered out.

---

4. Tool Catalogue (portal/tools.py)

Tools are registered in TOOLS = _cat() using the Tool dataclass:

@dataclass(frozen=True)
class Tool:
    id: str
    name: str
    category: str
    description: str
    url: str
    mode: str = "tab"             # "embed" or "tab"
    status: str = "online"         # initial visual default
    external: bool = True
    private: bool = False
    lan: bool = False              # LAN-only reachable badge
    login: bool = False            # Has separate login badge
    admin: bool = False            # Admins only
    admin_service: bool = False    # Service control admin only
    service_id: str | None = None  # Controller allowlist ID
    ram_mb: int | None = None      # Estimated idle RAM footprint
    togglable: bool = False        # Displays ON/OFF service strip

---

5. ON/OFF Service Controller Integration

To allow family members to free up RAM on .13 without exposing /var/run/docker.sock to the web portal, all container/systemd actions pass through a token-gated, host-side controller process.

Architecture

1. Host Process: service-controller runs natively on host .13 as a systemd user unit (family-service-controller.service) under user sam on 127.0.0.1:8443.
2. Security Boundary: The controller enforces an allowlist (deploy/service-controller/services.toml). Requests for unlisted services return 404.
3. Portal Proxy Endpoints:
   • GET /api/services: Fetches live status for allowlisted services from http://host.docker.internal:8443/services.
   • POST /service/{id}/{action}: Sends start or stop command with X-Controller-Token header.

Togglable Services Allowlist

Service ID	Target / Kind	Idle RAM	Group	Allowed Users
gimp	family-home-lab-gimp-1 (Container)	~233 MB	media	sam, finn, harry, jo
video-editor	family-home-lab-video-editor-1 (Container)	~237 MB	media	sam, finn, harry, jo
audio-editor	family-home-lab-audio-editor-1 (Container)	~201 MB	media	sam, finn, harry, jo
lmms	lmms (Container)	~798 MB	media	sam, finn, harry, jo
paperclip	paperclipai (systemd user)	~495 MB	admin	sam
prefect-worker	prefect-worker (systemd user)	~129 MB	pipeline	sam
photo-dashboard	photo-dashboard (systemd user)	~112 MB	pipeline	sam
worldmonitor	4-container group (Container)	~120 MB	admin	sam

Controller kinds (kind in services.toml)

kind	How it acts	Notes
container	docker start/stop <target>	Same host (.13); sam is in the docker group.
systemd-user	systemctl --user <action> <target>	Local .13 user units (paperclip, prefect-worker, photo-dashboard).
systemd-system	sudo -n <canonical systemctl> <action> <target>	System units, local or on a remote host over ssh. Currently unused — kept as a tested capability.

systemd-system gotchas (learned while it was in use; apply if it is ever wired up again):

1. It needs a matching NOPASSWD sudoers rule on the target host, and the target must match the rule verbatim — foo, never foo.service.
2. On NixOS, systemctl on PATH is a symlink (/run/current-system/sw/bin/systemctl) while the sudoers rule names the store path, and sudo matches the literal resolved path. A bare sudo systemctl … therefore does not match. The controller resolves systemctl to its canonical store path first, which also survives nixpkgs bumps that change the systemd store hash.
3. _remote() quotes the remote command with shlex.join before handing it to ssh, because ssh joins its trailing argv with spaces before the remote shell parses it.

---

6. Garage S3 & File Management

The portal includes an integrated S3 file manager (portal/s3.py, portal/templates/files.html):

• Connects to internal Garage S3 endpoint (http://garage:3900).
• Each user gets a personal bucket (sam, jo, harry, finn) plus access to shared-media.
• File actions: Upload, Download, Delete, Preview, Create Directory.

---

7. Operational & Deployment Commands

Deployment directory on host .13: /home/sam/Docker/Containers/family-home-lab

# Sync local changes from .27 to .13
rsync -avz --exclude='.git' --exclude='.venv' \
    /home/sam/home_network/custom_tools/family_home_lab/ \
    sam@192.168.20.13:/home/sam/Docker/Containers/family-home-lab/

# Rebuild and restart portal container on .13
ssh sam@192.168.20.13 "cd /home/sam/Docker/Containers/family-home-lab && docker compose build portal && docker compose up -d portal"