family_home_lab/docs/portal-architecture.md

# Console Portal Architecture & Integration Spec

## 1. System Overview

The **Family Home Lab Console Portal** (`portal/`) serves as the central unified dashboard, authentication gateway, file manager, and service control plane for the home lab (`console.lab.audasmedia.com.au`).

It is built with **FastAPI**, **Jinja2 templates**, **HTMX**, and **PostgreSQL**, running as container `family-home-lab-portal-1` on host `.13:8500`.

---

## 2. Technology Stack & Architecture

```
 Browser (HTMX / Vanilla JS)
      │
      ▼
 Caddy Reverse Proxy (.35:443)  ──[HTTPS]──►  Portal Container (.13:8500)
                                                    │
        ┌───────────────────┬───────────────────────┼───────────────────┐
        ▼                   ▼                       ▼                   ▼
 PostgreSQL (pg16)    Redis / RabbitMQ         Garage (S3)       service-controller
 (portal DB)          (task queue / cache)     (.13:3900)        (.13:8443)
```

- **Framework**: FastAPI (async Python 3.12)
- **Frontend**: Jinja2 HTML templates + HTMX (interactivity) + CSS custom tokens (`DESIGN.md` sticker system)
- **Database**: PostgreSQL (`pgvector/pgvector:pg16`), SQLAlchemy async ORM (`asyncpg`)
- **Queue / Async Tasks**: Celery + Redis + RabbitMQ
- **Object Storage**: Garage S3 (`:3900`)
- **Service Control**: Host-side `service-controller` (`:8443`)

---

## 3. Database Schema & User Entitlements

The user model (`portal/database.py`) defines authentication credentials, administrative roles, and feature capability flags:

### `User` Entity Schema
```python
class User(Base):
    __tablename__ = "users"

    id: Mapped[int] = mapped_column(primary_key=True)
    username: Mapped[str] = mapped_column(String(50), unique=True)
    password_hash: Mapped[str] = mapped_column(String(255))
    full_name: Mapped[str] = mapped_column(String(100))
    is_admin: Mapped[bool] = mapped_column(Boolean, default=False)

    # Per-user capability flags
    can_chat: Mapped[bool] = mapped_column(Boolean, default=True)
    can_image: Mapped[bool] = mapped_column(Boolean, default=True)
    can_video: Mapped[bool] = mapped_column(Boolean, default=True)
    can_audio: Mapped[bool] = mapped_column(Boolean, default=True)
    can_docs: Mapped[bool] = mapped_column(Boolean, default=False)
```

### Access Control Rules (`tools_for_user(user)`)
- **Admins (`is_admin=True`)**: Unrestricted access to all tools, categories, and service toggles.
- **Non-Admin Family Members**:
  - Open categories: `media` (Media & Fun), `data`, `storage`, `home`, `home-mgmt`, `dev`, `portfolio`.
  - Feature-gated categories: `chat`/`ai` (`can_chat`), `image` (`can_image`), `video` (`can_video`), `audio` (`can_audio`), `docs` (`can_docs`).
  - Gated tools: Infrastructure (`infra`) and Network (`network`) tools marked `admin=True` are filtered out.

---

## 4. Tool Catalogue (`portal/tools.py`)

Tools are registered in `TOOLS = _cat()` using the `Tool` dataclass:

```python
@dataclass(frozen=True)
class Tool:
    id: str
    name: str
    category: str
    description: str
    url: str
    mode: str = "tab"             # "embed" or "tab"
    status: str = "online"         # initial visual default
    external: bool = True
    private: bool = False
    lan: bool = False              # LAN-only reachable badge
    login: bool = False            # Has separate login badge
    admin: bool = False            # Admins only
    admin_service: bool = False    # Service control admin only
    service_id: str | None = None  # Controller allowlist ID
    ram_mb: int | None = None      # Estimated idle RAM footprint
    togglable: bool = False        # Displays ON/OFF service strip
```

---

## 5. ON/OFF Service Controller Integration

To allow family members to free up RAM on `.13` without exposing `/var/run/docker.sock` to the web portal, all container/systemd actions pass through a **token-gated, host-side controller process**.

### Architecture
1. **Host Process**: `service-controller` runs natively on host `.13` as a systemd user unit (`family-service-controller.service`) under user `sam` on `127.0.0.1:8443`.
2. **Security Boundary**: The controller enforces an allowlist (`deploy/service-controller/services.toml`). Requests for unlisted services return `404`.
3. **Portal Proxy Endpoints**:
   - `GET /api/services`: Fetches live status for allowlisted services from `http://host.docker.internal:8443/services`.
   - `POST /service/{id}/{action}`: Sends `start` or `stop` command with `X-Controller-Token` header.

### Togglable Services Allowlist
| Service ID | Target / Kind | Idle RAM | Group | Allowed Users |
| :--- | :--- | :--- | :--- | :--- |
| `gimp` | `family-home-lab-gimp-1` (Container) | ~233 MB | `media` | `sam`, `finn`, `harry`, `jo` |
| `video-editor` | `family-home-lab-video-editor-1` (Container) | ~237 MB | `media` | `sam`, `finn`, `harry`, `jo` |
| `audio-editor` | `family-home-lab-audio-editor-1` (Container) | ~201 MB | `media` | `sam`, `finn`, `harry`, `jo` |
| `lmms` | `lmms` (Container) | ~798 MB | `media` | `sam`, `finn`, `harry`, `jo` |
| `paperclip` | `paperclipai` (systemd user) | ~495 MB | `admin` | `sam` |
| `prefect-worker` | `prefect-worker` (systemd user) | ~129 MB | `pipeline` | `sam` |
| `photo-dashboard` | `photo-dashboard` (systemd user) | ~112 MB | `pipeline` | `sam` |
| `worldmonitor` | 4-container group (Container) | ~120 MB | `admin` | `sam` |
| `ollama` | `ollama` on **`.27`** (`systemd-system`, via ssh) | ~9000 MB (loaded model) | `ai` | `sam`, `jo`, `harry`, `finn` |

### Controller kinds (`kind` in `services.toml`)

| `kind` | How it acts | Notes |
| :--- | :--- | :--- |
| `container` | `docker start/stop <target>` | Same host (`.13`); `sam` is in the `docker` group. |
| `systemd-user` | `systemctl --user <action> <target>` | Local `.13` user units (paperclip, prefect-worker, photo-dashboard). |
| `systemd-system` | `sudo -n <canonical systemctl> <action> <target>` | System units, local or on a `remote` host over ssh. |

**`systemd-system` gotchas** (both were hit while wiring `ollama`):

1. It needs a matching **NOPASSWD** sudoers rule on the target host, and the
   `target` must match the rule verbatim — `ollama`, never `ollama.service`.
2. On NixOS, `systemctl` on `PATH` is a symlink (`/run/current-system/sw/bin/systemctl`)
   while the sudoers rule names the store path, and sudo matches the *literal*
   resolved path. A bare `sudo systemctl …` therefore does **not** match. The
   controller resolves `systemctl` to its canonical store path first, which also
   survives nixpkgs bumps that change the systemd store hash.
3. `_remote()` quotes the remote command with `shlex.join` before handing it to
   ssh, because ssh joins its trailing argv with spaces before the remote shell
   parses it.

---

## 6. Garage S3 & File Management

The portal includes an integrated S3 file manager (`portal/s3.py`, `portal/templates/files.html`):
- Connects to internal Garage S3 endpoint (`http://garage:3900`).
- Each user gets a personal bucket (`sam`, `jo`, `harry`, `finn`) plus access to `shared-media`.
- File actions: Upload, Download, Delete, Preview, Create Directory.

---

## 7. Operational & Deployment Commands

Deployment directory on host `.13`: `/home/sam/Docker/Containers/family-home-lab`

```bash
# Sync local changes from .27 to .13
rsync -avz --exclude='.git' --exclude='.venv' \
    /home/sam/home_network/custom_tools/family_home_lab/ \
    sam@192.168.20.13:/home/sam/Docker/Containers/family-home-lab/

# Rebuild and restart portal container on .13
ssh sam@192.168.20.13 "cd /home/sam/Docker/Containers/family-home-lab && docker compose build portal && docker compose up -d portal"
```