sam/obsidian-vault
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sam-4screen-desktop 2026-10-14:16:6:28

Browse Source
This commit is contained in:
Sam Rolfe
2026-10-14 16:06:28 +11:00
parent 1990144a69
commit 7635c874d2
5 changed files with 170 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.graphifyignore
View File

@@ -3,6 +3,7 @@
# hard-skipped by graphify, but .obsidian/ is NOT — dot-dirs are allowed)
.obsidian/
.pi/
graphify-out/
graphify-out-temp/
.trash/

6
.obsidian/workspace.json vendored
View File

@@ -212,19 +212,20 @@
},
"active": "26813e16e829b798",
"lastOpenFiles": [
"010 inbox/Ai Resume RAG Architecture Ideas.md",
"200 projects/210 AI Resume/Ai Resume RAG Architecture Ideas.md",
"000 daily/Daily Notes and To Do.md",
"010 inbox/Graphify.md",
"010 inbox/Internet Use Control — Kids' Screen Time Limits.md",
"010 inbox/Email Agent — Family Inbox Triage.md",
"100 inbox/Local LLM Infrastructure & Qwen2.5 Deployment.md",
"010 inbox/Email & DNS Setup — Domains, Mailboxes & Records.md",
"300 areas/360 Dev-Ops Network Computers/Email & DNS Setup — Domains, Mailboxes & Records.md",
"000 daily/Daily Notes and To Do.md",
"010 inbox/Central AI Hub.md",
"010 inbox/Family Holiday Camping Wilsons Prom.md",
"400 Personal Family/470 Holidays Travel/Family Holiday Camping Wilsons Prom.md",
"010 inbox/MiniMax Voice — Cloud STT and TTS with Local Fallback.md",
"200 projects/Tools Software WebUI/MiniMax Voice — Cloud STT and TTS with Local Fallback.md",
"010 inbox/Untitled.md",
"300 areas/355 AI Research and Techniques/Ai memory management.md",
"200 projects/Tools Software WebUI/Family Console.md",
"200 projects/Tools Software WebUI/Family Home Lab.md",
@@ -239,7 +240,6 @@
"010 inbox/Hindsight AI Resume Tool.md",
"200 projects/210 AI Resume/Hindsight AI Resume Tool.md",
"010 inbox/Alibaba Code Review AI Tool.md",
"300 areas/395 AI Tools to try/Alibaba Code Review AI Tool.md",
"010 inbox/_WATCHER",
"300 areas/355 AI Research and Techniques",
"300 areas/355 Ai Research and Techniques",

25
010 inbox/_sort log.md
View File

@@ -20069,3 +20069,28 @@ What the watcher did, newest last.
### 2026-10-14 13:23:07
- **proposed** `Graphify.md` → `200 projects/Tools Software WebUI` (confidence 0.36)
- - Low JEV confidence — recommend human confirmation
### 2026-10-14 15:59:02
- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content.
### 2026-10-14 15:59:12
- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content.
### 2026-10-14 15:59:22
- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content.
### 2026-10-14 15:59:32
- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content.
### 2026-10-14 15:59:42
- **held** `Untitled.md` — BLOCKED: untitled notes are never moved automatically. Rename it first if it is real content.
### 2026-10-14 15:59:52
- **proposed** `Ai Resume RAG Architecture Ideas.md` → `200 projects/210 AI Resume` (confidence 0.61)
### 2026-10-14 16:00:13
- **re-proposing** `Ai Resume RAG Architecture Ideas.md` — you edited it
- **proposed** `Ai Resume RAG Architecture Ideas.md` → `200 projects/210 AI Resume` (folder already set — kept) (confidence 0.91)
### 2026-10-14 16:00:53
- **filed** `Ai Resume RAG Architecture Ideas.md` → `200 projects/210 AI Resume/Ai Resume RAG Architecture Ideas.md` (type project)

1
100 Table of Contents/Projects.md
View File

@@ -40,6 +40,7 @@ SORT title ASC
- [[AI Resume - Content Organization Plan]]
- [[Local Hybrid Vector + Graph RAG Setup]]
- [[Hindsight AI Resume Tool]]
- [[Ai Resume RAG Architecture Ideas]]
### Web Host Migration
- [[Migration Plan — InMotion to Local + AWS Offsite]]

140
200 projects/210 AI Resume/Ai Resume RAG Architecture Ideas.md Normal file
View File

@@ -0,0 +1,140 @@
---
title: Ai Resume RAG Architecture Ideas
type: project
status: active
created: "2026-10-14 15:58"
updated: "2026-09-26"
tags:
- architecture
- pipeline
- ideas
modified: "2026-10-14 15:58"
---
# [[Ai Resume RAG Architecture Ideas]]
# Enterprise Homelab AI Knowledge Engine: Architecture & Workflows
This document outlines the blueprint for a professional-grade, resource-efficient, local-to-cloud hybrid RAG (Retrieval-Augmented Generation) infrastructure [3.2]. Built on a local Docker network, it orchestrates data pipelines from self-hosted tools (Gitea, Outline, Obsidian, Wiki.js, IoT systems) and routes semantic context to low-cost cloud LLMs.
---
## 1. Architectural Overview & Component Responsibilities
The system decouples data extraction, stateful scheduling, data persistence, and processing layers [3.2]. This guarantees low memory overhead on the host machine while matching industry-standard data design patterns.
* **Extraction Layer (n8n & Python Scrapers):** Captures change events and extracts raw content via API parsing, web scraping, or webhook listeners.
* **Orchestration Layer (Prefect):** Manages data pipelines, handling fault tolerance, rate limiting, document chunking, and batch schema execution.
* **Semantic Router Layer (LangGraph / LangChain):** Coordinates state machines to receive user queries, perform pre-filtered database retrieval, and manage cloud LLM context boundaries.
* **Storage Layer (Qdrant Vector Database):** Persists mathematical representations (embeddings) of text blocks along with explicit schema definitions.
* **Inference Layer (Cloud APIs via OpenRouter, DeepSeek, Gemini):** Executes logic, token generation, and codebase debugging tasks statelessly over HTTPS.
---
## 2. Component Workflow Implementations
### A. Data Ingestion & Extraction (n8n & Python)
Instead of executing heavy brute-force loops, data extraction runs in two structural forms:
1. **Event-Driven (n8n):** Listens to live inbound HTTP payloads (e.g., Gitea webhooks on code push events, Outline page revisions). n8n captures the operational change event payload and acts as a trigger to offload heavy lifting to the ETL pipeline.
2. **Scheduled Batch Jobs (Python Scrapers):** Custom headless scripts built with modern parsers (such as `Crawl4AI` or `BeautifulSoup4` coupled with `html2text`) to sweep legacy local web portals, hobby forums, or IoT data repositories. HTML is stripped of CSS stylesheets, tracking components, and sidebars, condensing raw input into text files.
### B. Enterprise ETL Orchestration (Prefect)
Prefect takes raw string text payloads from the extraction layer and runs them through deterministic python validation tasks:
* **Retry & Error Budgets:** Implements graceful back-offs when hitting remote token pools or local database limits.
* **Token-Aware Chunking:** Uses text splitters (like `RecursiveCharacterTextSplitter`) to split long files into target dimensions (e.g., 500-1000 characters) with an intentional safety overlap (e.g., 10%–15%) to avoid cutting sentences or code definitions in half.
* **Batch Pipelining:** Aggregates multiple document updates into single payloads to lower total HTTP requests across network connections.
### C. The Metadata Layer & Vector Storage (Qdrant)
To guarantee quick queries across massive local document footprints, data must be structurally categorized before it is saved into Qdrant. Every text block pushed to the database is accompanied by a structured metadata schema.
#### The Metadata Schema Format:
```json
{
"source": "gitea | outline | obsidian | iot | hobby-web",
"project_name": "home-automation",
"file_path_or_url": "https://gitea.local",
"last_updated": "2026-10-14T15:55:00Z",
"content_type": "source_code | documentation | log_summary"
}
```
#### Why Qdrant is Selected:
Qdrant acts as the system's indexing anchor. Written in Rust, it requires trivial idle resources (~200MB RAM), supports native Docker setups, and exposes comprehensive payload filtering capabilities. This allows the system to combine text math (vectors) with standard database query filters.
### D. The Vector Embedding Lifecycle
Raw text chunks cannot be parsed directly by Qdrant. They must go through a vector tokenization loop:
1. **Text Transform:** Prefect submits a block of text to an external embedding endpoint (e.g., Gemini's free tier embedding endpoint or OpenRouter's text-embedding variants).
2. **Numerical Array Return:** The endpoint converts the text into a fixed-length numerical array (e.g., 1536 floating-point numbers reflecting semantic context).
3. **Qdrant Upsert:** Prefect packages the numerical vector array alongside the source text string and its corresponding metadata object, writing it to a specific storage collection block inside Qdrant.
### E. Semantic Processing & Retrieval State Machines (LangGraph)
When a user asks a question, LangGraph acts as the cognitive director, controlling how data travels between the vector store and the cloud LLM.
1. **Query Extraction Node:** Evaluates the user's intent. If a user asks: *"Fix the bug in my home-automation repository,"* LangGraph flags that the request is specifically tied to code infrastructure.
2. **Metadata-Filtered Retrieval Node:** Instead of executing a blind vector scan against millions of lines of text (which risks pulling in unrelated data from Obsidian notes or IoT metrics), LangGraph passes a specific metadata condition directly to Qdrant:
* *Condition:* Find nearest vector coordinates *WHERE* `source == "gitea"` *AND* `project_name == "home-automation"`.
3. **Context Construction Node:** Qdrant returns only the most relevant, targeted code snippets within milliseconds. LangGraph packages these exact blocks into a unified system context window payload.
4. **Cloud LLM Call Node:** The finalized packet is sent statelessly over a secure outbound HTTPS connection to low-cost APIs (DeepSeek-V3/R1, OpenRouter, or Gemini). The LLM processes the contextualized prompt and fires back the perfect answer instantly.
---
## 3. Production-Ready Deployment: Docker Compose Topology
This composition maps your local storage environment seamlessly into your running n8n, Prefect, and LangGraph network spaces.
```yaml
version: '3.8'
services:
# The central vector memory engine
qdrant:
image: qdrant/qdrant:latest
container_name: local-ai-qdrant
ports:
- "6333:6333" # Main REST/HTTP API endpoint
- "6334:6334" # High-performance gRPC endpoint
volumes:
- qdrant_data_storage:/qdrant/storage
environment:
- QDRANT__SERVICE__ANALYTICS_TRACKING=false
restart: unless-stopped
networks:
- homelab-ai-mesh
# Your custom LangGraph running node/worker environment
langgraph-agent-worker:
build:
context: ./infra-agent-code
dockerfile: Dockerfile
container_name: langgraph-agent-worker
depends_on:
- qdrant
environment:
- QDRANT_HOST=http://qdrant:6333
- OPENROUTER_API_KEY=your_key_here
- DEEPSEEK_API_KEY=your_key_here
- GEMINI_API_KEY=your_key_here
volumes:
- /mnt/shared_local_data/obsidian_vault:/data/obsidian:ro # Read-only mount for safety
restart: unless-stopped
networks:
- homelab-ai-mesh
volumes:
qdrant_data_storage:
networks:
homelab-ai-mesh:
external: true # Bridges directly with your running n8n and Prefect network interfaces
```
---
## 4. Architectural Summary for Professional Interviews
When presenting this architecture to industry engineering leads, highlight these technical choices to demonstrate production maturity:
1. **Strict Resource Decoupling:** *"Instead of spinning up heavy local GPU servers or bloated monolithic RAG frameworks, I isolated the platform layers [3.2]. Compute costs are completely externalized to commodity cloud APIs via OpenRouter/DeepSeek, while data hygiene, scheduling, and storage remain entirely local inside Docker [3.2]."*
2. **Hybrid Metadata Optimization:** *"To ensure sub-second response speeds and completely prevent model hallucinations across distinct applications, I built custom Python ETL workflows in Prefect that tag all ingested data from Gitea, Outline, and local sites with strict metadata schemas. This allows my LangGraph pipelines to execute pre-filtered vector queries before the context hits the LLM."*
3. **Environment Agnostic Design:** *"The entire infrastructure interacts via standard OpenAI-compatible API schemas and container definitions. The codebase remains fully portable, meaning the system can scale from a single local workstation node to an enterprise Kubernetes tier simply by updating network configuration keys."*