Local LLM Infrastructure & Qwen2.5 Deployment

📌 Executive Summary

This project runs an on-demand local AI model server on workstation sam-4screen-desktop (192.168.20.27). It uses the machine's 64 GB of RAM and its CPU to run 14-billion parameter Qwen2.5 models, so AI work is possible without cloud cost or cloud dependency. Other machines and services reach it over the home network.

🖥️ System Architecture

flowchart LR
    subgraph Clients["Clients"]
        NV["Neovim / Avante / CodeCompanion (.27, .13, .51)"]
        DASH["Family Console Dashboard (start / stop)"]
    end
    subgraph Host27["sam-4screen-desktop (192.168.20.27)"]
        SYS["NixOS system service: ollama.service"]
        ENG["Ollama Engine :11434 - CPU + AVX2"]
        MEM["64 GB DDR4 System RAM"]
    end
    subgraph Router13["nixos-desktop (192.168.20.13)"]
        OR["OmniRoute Gateway :20128"]
        CLOUD["Cloud fallback: DeepSeek, Claude"]
    end
    DASH -->|start / stop| SYS
    SYS --> ENG
    ENG --> MEM
    NV -->|direct :11434| ENG
    NV -.->|ollama-local/qwen2.5-coder:14b| OR
    OR -->|primary route| ENG
    OR -.->|when .27 is off| CLOUD

🖧 Machine Roles

.27 (sam-4screen-desktop) — AI host. Intel Core i7-7700K (4 cores, AVX2) and 64 GB DDR4 RAM, with about 32 GB free for models. The NVIDIA Quadro P620 drives the displays only; it is not used for inference.
.13 (nixos-desktop) — network gateway. Runs OmniRoute, Caddy, and DNS. By design it runs no model inference, because it has little spare memory.

⚙️ How the Service Works

The model server is declared in NixOS, which makes it reproducible and easy to audit. It is off at boot so it only uses resources when wanted.

Start / stop: from the family console dashboard.
Manual control: sudo systemctl start ollama and sudo systemctl stop ollama.
Listen address: 0.0.0.0:11434, so .13, .51, and other LAN clients can connect.
Warm models: OLLAMA_KEEP_ALIVE = -1 keeps the model in RAM while the service runs, so responses are immediate. Stopping the service releases all of that RAM.

🤖 Installed Models

Model	Size	Quantization	Primary Purpose
`qwen2.5-coder:14b`	9.0 GB	Q4_K_M GGUF	Code generation, review, and refactoring
`qwen2.5:14b`	9.0 GB	Q4_K_M GGUF	Drafting, reasoning, and general analysis

Models live in /var/lib/ollama/models.

🌐 Routing Through OmniRoute (verified working)

OmniRoute on .13 exposes the local models alongside cloud models, with automatic fallback.

Entry	Value
Base URL	`http://192.168.20.13:20128/v1`
Model name	`ollama-local/qwen2.5-coder:14b`
Direct to `.27`	`http://192.168.20.27:11434` with model `qwen2.5-coder:14b`

Verified: chat requests through OmniRoute return HTTP 200 for both models, and the connection reports test_status: active.

💼 Skills & Technical Concepts Demonstrated

Linux system administration — systemd service design, boot policy, sandboxing trade-offs (DynamicUser, ProtectHome), firewall rules, network binding.
Fault diagnosis — traced a failed system rebuild to two independent causes: stale config symlinks and two services competing for one port.
Local AI operations — CPU inference, GGUF quantization, RAM budgeting, model warm-up and idle policy.
Hybrid cloud architecture — local-first inference with automatic cloud fallback through an API gateway.

Interactive architecture map: Archify map
AI context (Outline): docs/outline/00-index.md

3.9 KiB Raw Blame History