sam/obsidian-vault
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove local LLM (Qwen2.5/Ollama) project note — failed experiment

Browse Source
This commit is contained in:
Sam Rolfe
2026-10-12 20:18:45 +11:00
parent e30b8b0d9e
commit 4f66700a14
1 changed files with 0 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md
View File

@@ -1,104 +0,0 @@
---
created: 2026-10-12 14:07
modified: 2026-10-12 18:05
type: note
tags: [projects/local-llm, infra, ollama, qwen2.5, omniroute]
aliases: [Local LLM Infrastructure, Qwen2.5 Deployment]
---
# [[Local LLM Infrastructure & Qwen2.5 Deployment]]
## 📌 Executive Summary
This project runs an **on-demand local AI model server** on workstation **`sam-4screen-desktop` (`192.168.20.27`)**.
It uses the machine's 64 GB of RAM and its CPU to run 14-billion parameter Qwen2.5 models, so AI work is possible without cloud cost or cloud dependency.
Other machines and services reach it over the home network.
---
## 🖥️ System Architecture
```mermaid
flowchart LR
subgraph Clients["Clients"]
NV["Neovim / Avante / CodeCompanion (.27, .13, .51)"]
DASH["Family Console Dashboard (start / stop)"]
end
subgraph Host27["sam-4screen-desktop (192.168.20.27)"]
SYS["NixOS system service: ollama.service"]
ENG["Ollama Engine :11434 - CPU + AVX2"]
MEM["64 GB DDR4 System RAM"]
end
subgraph Router13["nixos-desktop (192.168.20.13)"]
OR["OmniRoute Gateway :20128"]
CLOUD["Cloud fallback: DeepSeek, Claude"]
end
DASH -->|start / stop| SYS
SYS --> ENG
ENG --> MEM
NV -->|direct :11434| ENG
NV -.->|ollama-local/qwen2.5-coder:14b| OR
OR -->|primary route| ENG
OR -.->|when .27 is off| CLOUD
```
---
## 🖧 Machine Roles
* **`.27` (`sam-4screen-desktop`) — AI host.**
Intel Core i7-7700K (4 cores, AVX2) and 64 GB DDR4 RAM, with about 32 GB free for models.
The NVIDIA Quadro P620 drives the displays only; it is not used for inference.
* **`.13` (`nixos-desktop`) — network gateway.**
Runs OmniRoute, Caddy, and DNS. By design it runs **no** model inference, because it has little spare memory.
---
## ⚙️ How the Service Works
The model server is declared in NixOS, which makes it reproducible and easy to audit.
It is **off at boot** so it only uses resources when wanted.
* **Start / stop:** from the family console dashboard.
* **Manual control:** `sudo systemctl start ollama` and `sudo systemctl stop ollama`.
* **Listen address:** `0.0.0.0:11434`, so `.13`, `.51`, and other LAN clients can connect.
* **Warm models:** `OLLAMA_KEEP_ALIVE = -1` keeps the model in RAM while the service runs, so responses are immediate. Stopping the service releases all of that RAM.
---
## 🤖 Installed Models
| Model | Size | Quantization | Primary Purpose |
| :--- | :--- | :--- | :--- |
| `qwen2.5-coder:14b` | 9.0 GB | Q4_K_M GGUF | Code generation, review, and refactoring |
| `qwen2.5:14b` | 9.0 GB | Q4_K_M GGUF | Drafting, reasoning, and general analysis |
Models live in `/var/lib/ollama/models`.
---
## 🌐 Routing Through OmniRoute (verified working)
OmniRoute on `.13` exposes the local models alongside cloud models, with automatic fallback.
| Entry | Value |
| :--- | :--- |
| Base URL | `http://192.168.20.13:20128/v1` |
| Model name | `ollama-local/qwen2.5-coder:14b` |
| Direct to `.27` | `http://192.168.20.27:11434` with model `qwen2.5-coder:14b` |
Verified: chat requests through OmniRoute return HTTP 200 for both models, and the connection reports `test_status: active`.
---
## 💼 Skills & Technical Concepts Demonstrated
* **Linux system administration** — systemd service design, boot policy, sandboxing trade-offs (`DynamicUser`, `ProtectHome`), firewall rules, network binding.
* **Fault diagnosis** — traced a failed system rebuild to two independent causes: stale config symlinks and two services competing for one port.
* **Local AI operations** — CPU inference, GGUF quantization, RAM budgeting, model warm-up and idle policy.
* **Hybrid cloud architecture** — local-first inference with automatic cloud fallback through an API gateway.
---
## 🔗 Related
* Interactive architecture map: [Archify map](https://maps.lab.audasmedia.com.au/ollama_local_llms/docs/)
* AI context (Outline): `docs/outline/00-index.md`