diff --git a/.graphifyignore b/.graphifyignore index 59eb27f..0717aa3 100644 --- a/.graphifyignore +++ b/.graphifyignore @@ -3,6 +3,7 @@ # hard-skipped by graphify, but .obsidian/ is NOT — dot-dirs are allowed) .obsidian/ +.pi/ graphify-out/ graphify-out-temp/ .trash/ diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index 2ee6760..ec9d101 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -212,19 +212,20 @@ }, "active": "26813e16e829b798", "lastOpenFiles": [ + "010 inbox/Ai Resume RAG Architecture Ideas.md", + "200 projects/210 AI Resume/Ai Resume RAG Architecture Ideas.md", + "000 daily/Daily Notes and To Do.md", "010 inbox/Graphify.md", "010 inbox/Internet Use Control — Kids' Screen Time Limits.md", "010 inbox/Email Agent — Family Inbox Triage.md", "100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md", "010 inbox/Email & DNS Setup — Domains, Mailboxes & Records.md", "300 areas/360 Dev-Ops Network Computers/Email & DNS Setup — Domains, Mailboxes & Records.md", - "000 daily/Daily Notes and To Do.md", "010 inbox/Central AI Hub.md", "010 inbox/Family Holiday Camping Wilsons Prom.md", "400 Personal Family/470 Holidays Travel/Family Holiday Camping Wilsons Prom.md", "010 inbox/MiniMax Voice — Cloud STT and TTS with Local Fallback.md", "200 projects/Tools Software WebUI/MiniMax Voice — Cloud STT and TTS with Local Fallback.md", - "010 inbox/Untitled.md", "300 areas/355 AI Research and Techniques/Ai memory management.md", "200 projects/Tools Software WebUI/Family Console.md", "200 projects/Tools Software WebUI/Family Home Lab.md", @@ -239,7 +240,6 @@ "010 inbox/Hindsight AI Resume Tool.md", "200 projects/210 AI Resume/Hindsight AI Resume Tool.md", "010 inbox/Alibaba Code Review AI Tool.md", - "300 areas/395 AI Tools to try/Alibaba Code Review AI Tool.md", "010 inbox/_WATCHER", "300 areas/355 AI Research and Techniques", "300 areas/355 Ai Research and Techniques", diff --git a/010 inbox/_sort log.md b/010 inbox/_sort log.md index 35a8ccd..69dad5f 100644 --- a/010 inbox/_sort log.md +++ b/010 inbox/_sort log.md @@ -20069,3 +20069,28 @@ What the watcher did, newest last. ### 2026-10-14 13:23:07 - **proposed** `Graphify.md` → `200 projects/Tools Software WebUI` (confidence 0.36) - - Low JEV confidence — recommend human confirmation + +### 2026-10-14 15:59:02 +- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content. + +### 2026-10-14 15:59:12 +- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content. + +### 2026-10-14 15:59:22 +- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content. + +### 2026-10-14 15:59:32 +- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content. + +### 2026-10-14 15:59:42 +- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content. + +### 2026-10-14 15:59:52 +- **proposed** `Ai Resume RAG Architecture Ideas.md` → `200 projects/210 AI Resume` (confidence 0.61) + +### 2026-10-14 16:00:13 +- **re-proposing** `Ai Resume RAG Architecture Ideas.md` — you edited it +- **proposed** `Ai Resume RAG Architecture Ideas.md` → `200 projects/210 AI Resume` (folder already set — kept) (confidence 0.91) + +### 2026-10-14 16:00:53 +- **filed** `Ai Resume RAG Architecture Ideas.md` → `200 projects/210 AI Resume/Ai Resume RAG Architecture Ideas.md` (type project) diff --git a/100 Table of Contents/Projects.md b/100 Table of Contents/Projects.md index c2b12eb..3b6f8aa 100644 --- a/100 Table of Contents/Projects.md +++ b/100 Table of Contents/Projects.md @@ -40,6 +40,7 @@ SORT title ASC - [[AI Resume - Content Organization Plan]] - [[Local Hybrid Vector + Graph RAG Setup]] - [[Hindsight AI Resume Tool]] +- [[Ai Resume RAG Architecture Ideas]] ### Web Host Migration - [[Migration Plan — InMotion to Local + AWS Offsite]] diff --git a/200 projects/210 AI Resume/Ai Resume RAG Architecture Ideas.md b/200 projects/210 AI Resume/Ai Resume RAG Architecture Ideas.md new file mode 100644 index 0000000..a1ed731 --- /dev/null +++ b/200 projects/210 AI Resume/Ai Resume RAG Architecture Ideas.md @@ -0,0 +1,140 @@ +--- +title: Ai Resume RAG Architecture Ideas +type: project +status: active +created: "2026-10-14 15:58" +updated: "2026-09-26" +tags: + - architecture + - pipeline + - ideas +modified: "2026-10-14 15:58" +--- + +# [[Ai Resume RAG Architecture Ideas]] + +# Enterprise Homelab AI Knowledge Engine: Architecture & Workflows + +This document outlines the blueprint for a professional-grade, resource-efficient, local-to-cloud hybrid RAG (Retrieval-Augmented Generation) infrastructure [3.2]. Built on a local Docker network, it orchestrates data pipelines from self-hosted tools (Gitea, Outline, Obsidian, Wiki.js, IoT systems) and routes semantic context to low-cost cloud LLMs. + +--- + +## 1. Architectural Overview & Component Responsibilities + +The system decouples data extraction, stateful scheduling, data persistence, and processing layers [3.2]. This guarantees low memory overhead on the host machine while matching industry-standard data design patterns. + +* **Extraction Layer (n8n & Python Scrapers):** Captures change events and extracts raw content via API parsing, web scraping, or webhook listeners. +* **Orchestration Layer (Prefect):** Manages data pipelines, handling fault tolerance, rate limiting, document chunking, and batch schema execution. +* **Semantic Router Layer (LangGraph / LangChain):** Coordinates state machines to receive user queries, perform pre-filtered database retrieval, and manage cloud LLM context boundaries. +* **Storage Layer (Qdrant Vector Database):** Persists mathematical representations (embeddings) of text blocks along with explicit schema definitions. +* **Inference Layer (Cloud APIs via OpenRouter, DeepSeek, Gemini):** Executes logic, token generation, and codebase debugging tasks statelessly over HTTPS. + +--- + +## 2. Component Workflow Implementations + +### A. Data Ingestion & Extraction (n8n & Python) +Instead of executing heavy brute-force loops, data extraction runs in two structural forms: +1. **Event-Driven (n8n):** Listens to live inbound HTTP payloads (e.g., Gitea webhooks on code push events, Outline page revisions). n8n captures the operational change event payload and acts as a trigger to offload heavy lifting to the ETL pipeline. +2. **Scheduled Batch Jobs (Python Scrapers):** Custom headless scripts built with modern parsers (such as `Crawl4AI` or `BeautifulSoup4` coupled with `html2text`) to sweep legacy local web portals, hobby forums, or IoT data repositories. HTML is stripped of CSS stylesheets, tracking components, and sidebars, condensing raw input into text files. + +### B. Enterprise ETL Orchestration (Prefect) +Prefect takes raw string text payloads from the extraction layer and runs them through deterministic python validation tasks: +* **Retry & Error Budgets:** Implements graceful back-offs when hitting remote token pools or local database limits. +* **Token-Aware Chunking:** Uses text splitters (like `RecursiveCharacterTextSplitter`) to split long files into target dimensions (e.g., 500-1000 characters) with an intentional safety overlap (e.g., 10%–15%) to avoid cutting sentences or code definitions in half. +* **Batch Pipelining:** Aggregates multiple document updates into single payloads to lower total HTTP requests across network connections. + +### C. The Metadata Layer & Vector Storage (Qdrant) +To guarantee quick queries across massive local document footprints, data must be structurally categorized before it is saved into Qdrant. Every text block pushed to the database is accompanied by a structured metadata schema. + +#### The Metadata Schema Format: +```json +{ + "source": "gitea | outline | obsidian | iot | hobby-web", + "project_name": "home-automation", + "file_path_or_url": "https://gitea.local", + "last_updated": "2026-10-14T15:55:00Z", + "content_type": "source_code | documentation | log_summary" +} +``` + +#### Why Qdrant is Selected: +Qdrant acts as the system's indexing anchor. Written in Rust, it requires trivial idle resources (~200MB RAM), supports native Docker setups, and exposes comprehensive payload filtering capabilities. This allows the system to combine text math (vectors) with standard database query filters. + +### D. The Vector Embedding Lifecycle +Raw text chunks cannot be parsed directly by Qdrant. They must go through a vector tokenization loop: +1. **Text Transform:** Prefect submits a block of text to an external embedding endpoint (e.g., Gemini's free tier embedding endpoint or OpenRouter's text-embedding variants). +2. **Numerical Array Return:** The endpoint converts the text into a fixed-length numerical array (e.g., 1536 floating-point numbers reflecting semantic context). +3. **Qdrant Upsert:** Prefect packages the numerical vector array alongside the source text string and its corresponding metadata object, writing it to a specific storage collection block inside Qdrant. + +### E. Semantic Processing & Retrieval State Machines (LangGraph) +When a user asks a question, LangGraph acts as the cognitive director, controlling how data travels between the vector store and the cloud LLM. + +1. **Query Extraction Node:** Evaluates the user's intent. If a user asks: *"Fix the bug in my home-automation repository,"* LangGraph flags that the request is specifically tied to code infrastructure. +2. **Metadata-Filtered Retrieval Node:** Instead of executing a blind vector scan against millions of lines of text (which risks pulling in unrelated data from Obsidian notes or IoT metrics), LangGraph passes a specific metadata condition directly to Qdrant: + * *Condition:* Find nearest vector coordinates *WHERE* `source == "gitea"` *AND* `project_name == "home-automation"`. +3. **Context Construction Node:** Qdrant returns only the most relevant, targeted code snippets within milliseconds. LangGraph packages these exact blocks into a unified system context window payload. +4. **Cloud LLM Call Node:** The finalized packet is sent statelessly over a secure outbound HTTPS connection to low-cost APIs (DeepSeek-V3/R1, OpenRouter, or Gemini). The LLM processes the contextualized prompt and fires back the perfect answer instantly. + +--- + +## 3. Production-Ready Deployment: Docker Compose Topology + +This composition maps your local storage environment seamlessly into your running n8n, Prefect, and LangGraph network spaces. + +```yaml +version: '3.8' + +services: + # The central vector memory engine + qdrant: + image: qdrant/qdrant:latest + container_name: local-ai-qdrant + ports: + - "6333:6333" # Main REST/HTTP API endpoint + - "6334:6334" # High-performance gRPC endpoint + volumes: + - qdrant_data_storage:/qdrant/storage + environment: + - QDRANT__SERVICE__ANALYTICS_TRACKING=false + restart: unless-stopped + networks: + - homelab-ai-mesh + + # Your custom LangGraph running node/worker environment + langgraph-agent-worker: + build: + context: ./infra-agent-code + dockerfile: Dockerfile + container_name: langgraph-agent-worker + depends_on: + - qdrant + environment: + - QDRANT_HOST=http://qdrant:6333 + - OPENROUTER_API_KEY=your_key_here + - DEEPSEEK_API_KEY=your_key_here + - GEMINI_API_KEY=your_key_here + volumes: + - /mnt/shared_local_data/obsidian_vault:/data/obsidian:ro # Read-only mount for safety + restart: unless-stopped + networks: + - homelab-ai-mesh + +volumes: + qdrant_data_storage: + +networks: + homelab-ai-mesh: + external: true # Bridges directly with your running n8n and Prefect network interfaces +``` + +--- + +## 4. Architectural Summary for Professional Interviews + +When presenting this architecture to industry engineering leads, highlight these technical choices to demonstrate production maturity: + +1. **Strict Resource Decoupling:** *"Instead of spinning up heavy local GPU servers or bloated monolithic RAG frameworks, I isolated the platform layers [3.2]. Compute costs are completely externalized to commodity cloud APIs via OpenRouter/DeepSeek, while data hygiene, scheduling, and storage remain entirely local inside Docker [3.2]."* +2. **Hybrid Metadata Optimization:** *"To ensure sub-second response speeds and completely prevent model hallucinations across distinct applications, I built custom Python ETL workflows in Prefect that tag all ingested data from Gitea, Outline, and local sites with strict metadata schemas. This allows my LangGraph pipelines to execute pre-filtered vector queries before the context hits the LLM."* +3. **Environment Agnostic Design:** *"The entire infrastructure interacts via standard OpenAI-compatible API schemas and container definitions. The codebase remains fully portable, meaning the system can scale from a single local workstation node to an enterprise Kubernetes tier simply by updating network configuration keys."* +