diff --git a/100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md b/100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md deleted file mode 100644 index 9880dfc..0000000 --- a/100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md +++ /dev/null @@ -1,104 +0,0 @@ ---- -created: 2026-10-12 14:07 -modified: 2026-10-12 18:05 -type: note -tags: [projects/local-llm, infra, ollama, qwen2.5, omniroute] -aliases: [Local LLM Infrastructure, Qwen2.5 Deployment] ---- -# [[Local LLM Infrastructure & Qwen2.5 Deployment]] - -## 📌 Executive Summary -This project runs an **on-demand local AI model server** on workstation **`sam-4screen-desktop` (`192.168.20.27`)**. -It uses the machine's 64 GB of RAM and its CPU to run 14-billion parameter Qwen2.5 models, so AI work is possible without cloud cost or cloud dependency. -Other machines and services reach it over the home network. - ---- - -## 🖥️ System Architecture - -```mermaid -flowchart LR - subgraph Clients["Clients"] - NV["Neovim / Avante / CodeCompanion (.27, .13, .51)"] - DASH["Family Console Dashboard (start / stop)"] - end - subgraph Host27["sam-4screen-desktop (192.168.20.27)"] - SYS["NixOS system service: ollama.service"] - ENG["Ollama Engine :11434 - CPU + AVX2"] - MEM["64 GB DDR4 System RAM"] - end - subgraph Router13["nixos-desktop (192.168.20.13)"] - OR["OmniRoute Gateway :20128"] - CLOUD["Cloud fallback: DeepSeek, Claude"] - end - DASH -->|start / stop| SYS - SYS --> ENG - ENG --> MEM - NV -->|direct :11434| ENG - NV -.->|ollama-local/qwen2.5-coder:14b| OR - OR -->|primary route| ENG - OR -.->|when .27 is off| CLOUD -``` - ---- - -## 🖧 Machine Roles - -* **`.27` (`sam-4screen-desktop`) — AI host.** - Intel Core i7-7700K (4 cores, AVX2) and 64 GB DDR4 RAM, with about 32 GB free for models. - The NVIDIA Quadro P620 drives the displays only; it is not used for inference. -* **`.13` (`nixos-desktop`) — network gateway.** - Runs OmniRoute, Caddy, and DNS. By design it runs **no** model inference, because it has little spare memory. - ---- - -## ⚙️ How the Service Works - -The model server is declared in NixOS, which makes it reproducible and easy to audit. -It is **off at boot** so it only uses resources when wanted. - -* **Start / stop:** from the family console dashboard. -* **Manual control:** `sudo systemctl start ollama` and `sudo systemctl stop ollama`. -* **Listen address:** `0.0.0.0:11434`, so `.13`, `.51`, and other LAN clients can connect. -* **Warm models:** `OLLAMA_KEEP_ALIVE = -1` keeps the model in RAM while the service runs, so responses are immediate. Stopping the service releases all of that RAM. - ---- - -## 🤖 Installed Models - -| Model | Size | Quantization | Primary Purpose | -| :--- | :--- | :--- | :--- | -| `qwen2.5-coder:14b` | 9.0 GB | Q4_K_M GGUF | Code generation, review, and refactoring | -| `qwen2.5:14b` | 9.0 GB | Q4_K_M GGUF | Drafting, reasoning, and general analysis | - -Models live in `/var/lib/ollama/models`. - ---- - -## 🌐 Routing Through OmniRoute (verified working) - -OmniRoute on `.13` exposes the local models alongside cloud models, with automatic fallback. - -| Entry | Value | -| :--- | :--- | -| Base URL | `http://192.168.20.13:20128/v1` | -| Model name | `ollama-local/qwen2.5-coder:14b` | -| Direct to `.27` | `http://192.168.20.27:11434` with model `qwen2.5-coder:14b` | - -Verified: chat requests through OmniRoute return HTTP 200 for both models, and the connection reports `test_status: active`. - ---- - -## 💼 Skills & Technical Concepts Demonstrated - -* **Linux system administration** — systemd service design, boot policy, sandboxing trade-offs (`DynamicUser`, `ProtectHome`), firewall rules, network binding. -* **Fault diagnosis** — traced a failed system rebuild to two independent causes: stale config symlinks and two services competing for one port. -* **Local AI operations** — CPU inference, GGUF quantization, RAM budgeting, model warm-up and idle policy. -* **Hybrid cloud architecture** — local-first inference with automatic cloud fallback through an API gateway. - ---- - -## 🔗 Related - -* Interactive architecture map: [Archify map](https://maps.lab.audasmedia.com.au/ollama_local_llms/docs/) -* AI context (Outline): `docs/outline/00-index.md`