From e30b8b0d9ecd62b704c0d1b7df7adc0063b7e7a8 Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Mon, 12 Oct 2026 17:21:28 +1100 Subject: [PATCH] sam-4screen-desktop 2026-10-12:17:21:28 --- ...LLM Infrastructure & Qwen2.5 Deployment.md | 106 +++++++++++++----- 1 file changed, 78 insertions(+), 28 deletions(-) diff --git a/100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md b/100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md index 5cf5a56..9880dfc 100644 --- a/100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md +++ b/100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md @@ -1,54 +1,104 @@ --- created: 2026-10-12 14:07 -modified: 2026-10-12 14:07 +modified: 2026-10-12 18:05 type: note -tags: [projects/local-llm, infra, ollama, qwen2.5] +tags: [projects/local-llm, infra, ollama, qwen2.5, omniroute] aliases: [Local LLM Infrastructure, Qwen2.5 Deployment] --- # [[Local LLM Infrastructure & Qwen2.5 Deployment]] ## 📌 Executive Summary -This project establishes an **on-demand local LLM inference node** on workstation **`sam-4screen-desktop` (`192.168.20.27`)**. It leverages the machine's 64 GB DDR4 RAM and 4-core Intel i7 CPU to run 14-billion parameter models locally (**Qwen2.5-Coder-14B** and **Qwen2.5-14B-Instruct**) without consuming idle background resources or requiring dedicated server GPUs. +This project runs an **on-demand local AI model server** on workstation **`sam-4screen-desktop` (`192.168.20.27`)**. +It uses the machine's 64 GB of RAM and its CPU to run 14-billion parameter Qwen2.5 models, so AI work is possible without cloud cost or cloud dependency. +Other machines and services reach it over the home network. --- -## 🖥️ System Architecture & Machine Roles +## 🖥️ System Architecture -* **Workstation `.27` (`sam-4screen-desktop`):** - * **Role:** Primary Local LLM Host. - * **Hardware:** Intel i7-7700K CPU + **64 GB DDR4 RAM** (~32 GB available for models). - * **Inference Strategy:** **Native CPU execution** using AVX2 instructions (bypasses Docker virtualization overhead for maximum CPU performance). - * **Auto-Unload:** Idle models automatically unload from RAM after 5 minutes (`OLLAMA_KEEP_ALIVE=5m`). - -* **Gateway Node `.13` (`nixos-desktop`):** - * **Role:** Network Gateway & OmniRoute Router. - * **Rule:** **Strictly no LLM inference runs on `.13`** due to memory constraints. +```mermaid +flowchart LR + subgraph Clients["Clients"] + NV["Neovim / Avante / CodeCompanion (.27, .13, .51)"] + DASH["Family Console Dashboard (start / stop)"] + end + subgraph Host27["sam-4screen-desktop (192.168.20.27)"] + SYS["NixOS system service: ollama.service"] + ENG["Ollama Engine :11434 - CPU + AVX2"] + MEM["64 GB DDR4 System RAM"] + end + subgraph Router13["nixos-desktop (192.168.20.13)"] + OR["OmniRoute Gateway :20128"] + CLOUD["Cloud fallback: DeepSeek, Claude"] + end + DASH -->|start / stop| SYS + SYS --> ENG + ENG --> MEM + NV -->|direct :11434| ENG + NV -.->|ollama-local/qwen2.5-coder:14b| OR + OR -->|primary route| ENG + OR -.->|when .27 is off| CLOUD +``` --- -## 🛠️ On-Demand Management Commands +## 🖧 Machine Roles -Simple terminal commands manage the Ollama service on `.27`: - -| Command | Action | Output / Behavior | -| :--- | :--- | :--- | -| `ai-on` | Starts local Ollama service | Binds to `0.0.0.0:11434` for local & LAN access. | -| `ai-off` | Stops local Ollama service | Immediately frees RAM and CPU cycles. | -| `ai-status` | Displays service status | Reports systemd unit state and port listeners. | +* **`.27` (`sam-4screen-desktop`) — AI host.** + Intel Core i7-7700K (4 cores, AVX2) and 64 GB DDR4 RAM, with about 32 GB free for models. + The NVIDIA Quadro P620 drives the displays only; it is not used for inference. +* **`.13` (`nixos-desktop`) — network gateway.** + Runs OmniRoute, Caddy, and DNS. By design it runs **no** model inference, because it has little spare memory. --- -## 🤖 Installed Local Models +## ⚙️ How the Service Works -| Model | Size | Parameter Quantization | Primary Purpose | +The model server is declared in NixOS, which makes it reproducible and easy to audit. +It is **off at boot** so it only uses resources when wanted. + +* **Start / stop:** from the family console dashboard. +* **Manual control:** `sudo systemctl start ollama` and `sudo systemctl stop ollama`. +* **Listen address:** `0.0.0.0:11434`, so `.13`, `.51`, and other LAN clients can connect. +* **Warm models:** `OLLAMA_KEEP_ALIVE = -1` keeps the model in RAM while the service runs, so responses are immediate. Stopping the service releases all of that RAM. + +--- + +## 🤖 Installed Models + +| Model | Size | Quantization | Primary Purpose | | :--- | :--- | :--- | :--- | -| `qwen2.5-coder:14b` | 9.0 GB | Q4_K_M GGUF | Offline code generation, refactoring, and code review. | -| `qwen2.5:14b` | 9.0 GB | Q4_K_M GGUF | General drafting, reasoning, and context analysis. | +| `qwen2.5-coder:14b` | 9.0 GB | Q4_K_M GGUF | Code generation, review, and refactoring | +| `qwen2.5:14b` | 9.0 GB | Q4_K_M GGUF | Drafting, reasoning, and general analysis | + +Models live in `/var/lib/ollama/models`. + +--- + +## 🌐 Routing Through OmniRoute (verified working) + +OmniRoute on `.13` exposes the local models alongside cloud models, with automatic fallback. + +| Entry | Value | +| :--- | :--- | +| Base URL | `http://192.168.20.13:20128/v1` | +| Model name | `ollama-local/qwen2.5-coder:14b` | +| Direct to `.27` | `http://192.168.20.27:11434` with model `qwen2.5-coder:14b` | + +Verified: chat requests through OmniRoute return HTTP 200 for both models, and the connection reports `test_status: active`. --- ## 💼 Skills & Technical Concepts Demonstrated -* **Linux System Administration:** Systemd user unit management, environment variables, network binding (`0.0.0.0`), process lifecycle control. -* **Local AI & GGUF Inference:** CPU-optimized LLM deployment, RAM budgeting, model quantization trade-offs. -* **Edge & Hybrid Architecture:** On-demand local processing paired with cloud API fallback mechanisms. +* **Linux system administration** — systemd service design, boot policy, sandboxing trade-offs (`DynamicUser`, `ProtectHome`), firewall rules, network binding. +* **Fault diagnosis** — traced a failed system rebuild to two independent causes: stale config symlinks and two services competing for one port. +* **Local AI operations** — CPU inference, GGUF quantization, RAM budgeting, model warm-up and idle policy. +* **Hybrid cloud architecture** — local-first inference with automatic cloud fallback through an API gateway. + +--- + +## 🔗 Related + +* Interactive architecture map: [Archify map](https://maps.lab.audasmedia.com.au/ollama_local_llms/docs/) +* AI context (Outline): `docs/outline/00-index.md`