Local LLM Infrastructure & Qwen2.5 Deployment

📌 Executive Summary

This project establishes an on-demand local LLM inference node on workstation sam-4screen-desktop (192.168.20.27). It leverages the machine's 64 GB DDR4 RAM and 4-core Intel i7 CPU to run 14-billion parameter models locally (Qwen2.5-Coder-14B and Qwen2.5-14B-Instruct) without consuming idle background resources or requiring dedicated server GPUs.

🖥️ System Architecture & Machine Roles

Workstation .27 (sam-4screen-desktop):
- Role: Primary Local LLM Host.
- Hardware: Intel i7-7700K CPU + 64 GB DDR4 RAM (~32 GB available for models).
- Inference Strategy: Native CPU execution using AVX2 instructions (bypasses Docker virtualization overhead for maximum CPU performance).
- Auto-Unload: Idle models automatically unload from RAM after 5 minutes (OLLAMA_KEEP_ALIVE=5m).
Gateway Node .13 (nixos-desktop):
- Role: Network Gateway & OmniRoute Router.
- Rule: Strictly no LLM inference runs on .13 due to memory constraints.

🛠️ On-Demand Management Commands

Simple terminal commands manage the Ollama service on .27:

Command	Action	Output / Behavior
`ai-on`	Starts local Ollama service	Binds to `0.0.0.0:11434` for local & LAN access.
`ai-off`	Stops local Ollama service	Immediately frees RAM and CPU cycles.
`ai-status`	Displays service status	Reports systemd unit state and port listeners.

🤖 Installed Local Models

Model	Size	Parameter Quantization	Primary Purpose
`qwen2.5-coder:14b`	9.0 GB	Q4_K_M GGUF	Offline code generation, refactoring, and code review.
`qwen2.5:14b`	9.0 GB	Q4_K_M GGUF	General drafting, reasoning, and context analysis.

💼 Skills & Technical Concepts Demonstrated

Linux System Administration: Systemd user unit management, environment variables, network binding (0.0.0.0), process lifecycle control.
Local AI & GGUF Inference: CPU-optimized LLM deployment, RAM budgeting, model quantization trade-offs.
Edge & Hybrid Architecture: On-demand local processing paired with cloud API fallback mechanisms.

2.4 KiB Raw Blame History