aboutme_chat/plan.md

# Project Plan: aboutme_chat_demo

## Goal
Build a comprehensive AI agent system that ingests data from self-hosted services (Gitea, notes, wikis), stores it in a vector database, and provides intelligent responses through a multi-agent orchestration layer. The system emphasizes modular containerized architecture, industry-standard tools, and employment-relevant skills.

---

## Phase 1: Foundation & Core Infrastructure (COMPLETED)

### Phase 1.1: Frontend Application
**Location:** `/home/sam/development/aboutme_chat_demo/frontend/`

**Stack & Tools:**
- **Framework:** Vite 6.2.0 + React 19.0.0 + TypeScript
- **Styling:** Tailwind CSS 4.0.0
- **State Management:** TanStack Query (React Query) 5.67.0
- **Build Tool:** Vite with React plugin
- **Linting:** ESLint 9.21.0 + typescript-eslint 8.24.0

**Components Implemented:**
- `ChatInterface.tsx` - Auto-expanding text input with scrolling message list
- `App.tsx` - Main application container
- Real-time chat UI with message history
- HTTP client integration to backend gateway

**Docker Configuration:**
- Hot-reload development setup
- Volume mounting for instant code changes
- Node modules isolation (`/app/node_modules`)

### Phase 1.2: Chat Gateway (Orchestration Entry Point)
**Location:** `/home/sam/development/aboutme_chat_demo/backend/`

**Stack & Tools:**
- **Framework:** FastAPI (Python 3.11)
- **HTTP Client:** httpx 0.28.1
- **CORS:** Configured for all origins (development)

**Architecture Changes:**
- **OLD:** Hardcoded keyword matching (`["sam", "hobby", "music", "guitar", "skiing", "experience"]`) to trigger knowledge lookup
- **NEW:** Thin routing layer - all queries passed to LangGraph Supervisor for intelligent agent selection
- Removed direct Brain (LLM) integration
- Removed direct Knowledge Service calls
- Now acts as stateless entry point to LangGraph orchestration layer

**Endpoints:**
- `POST /chat` - Routes queries to LangGraph Supervisor
- `GET /health` - Service health check
- `GET /agents` - Lists available agents from LangGraph

### Phase 1.3: Knowledge Service (Librarian Agent)
**Location:** `/home/sam/development/knowledge_service/`

**Stack & Tools:**
- **Framework:** FastAPI + Uvicorn
- **Vector Database:** ChromaDB 1.5.1
- **Embeddings:** OpenAI via OpenRouter API (text-embedding-3-small)
- **LLM Framework:** LangChain ecosystem
  - langchain 1.2.10
  - langchain-community 0.4.1
  - langchain-core 1.2.15
  - langchain-text-splitters 1.1.1
  - langchain-openai
- **Document Processing:** RecursiveCharacterTextSplitter

**Key Files:**
- `main.py` - FastAPI endpoints for /query and /health
- `gitea_scraper.py` - Gitea API integration module (NEW)
- `data/hobbies.md` - Sample knowledge base content
- `chroma_db/` - Persistent vector storage

**Docker Architecture (Optimized):**
- **Pattern:** Separate `/app/packages` (cached) from `/app/code` (volume-mounted)
- **Benefits:** 
  - Code changes apply instantly without rebuild
  - Package installation happens once during image build
  - PYTHONPATH=/app/packages ensures imports work
- **Volumes:**
  - `./data:/app/code/data` - Knowledge documents
  - `./chroma_db:/app/code/chroma_db` - Vector database persistence
  - `./main.py:/app/code/main.py:ro` - Read-only code mount

### Phase 1.4: LangGraph Supervisor Service (NEW)
**Location:** `/home/sam/development/langgraph_service/`

**Stack & Tools:**
- **Framework:** FastAPI + Uvicorn
- **Orchestration:** LangGraph 1.0.9
  - langgraph-checkpoint 4.0.0
  - langgraph-prebuilt 1.0.8
  - langgraph-sdk 0.3.9
- **State Management:** TypedDict with Annotated operators
- **Message Types:** LangChain Core Messages (HumanMessage, AIMessage)

**Architecture:**
- **Supervisor Node:** Analyzes queries and routes to specialist agents
- **Agent Graph:** StateGraph with conditional edges
- **Three Specialist Agents:**
  1. **Librarian Agent** - Queries ChromaDB via knowledge-service:8080
  2. **Opencode Agent** - Placeholder for coding tasks (MCP integration ready)
  3. **Brain Agent** - Fallback to OpenCode Brain LLM (opencode-brain:5000)

**Routing Logic:**
```
Query → Supervisor → [Librarian | Opencode | Brain]
- "repo/code/git/project" → Librarian (RAG)
- "write/edit/create/fix" → Opencode (Coding)
- "sam/hobby/music/about" → Librarian (RAG)
- Default → Brain (General LLM)
```

**Docker Configuration:**
- Self-contained with own `/app/packages`
- No package sharing with other services (modular)
- Port 8090 exposed

### Phase 1.5: Apache Airflow (Scheduled Ingestion)
**Location:** `/home/sam/development/airflow/`

**Stack & Tools:**
- **Orchestration:** Apache Airflow 2.8.1
- **Executor:** CeleryExecutor (distributed task processing)
- **Database:** PostgreSQL 13 (metadata)
- **Message Queue:** Redis (Celery broker)
- **Services:**
  - airflow-webserver (UI + API)
  - airflow-scheduler (DAG scheduling)
  - airflow-worker (task execution)
  - airflow-triggerer (deferrable operators)

**DAG: gitea_daily_ingestion**
- **Schedule:** Daily
- **Tasks:**
  1. `fetch_repos` - Get all user repos from Gitea API
  2. `fetch_readmes` - Download README files
  3. `ingest_to_chroma` - Store in Knowledge Service

**Integration:**
- Mounts `knowledge_service/gitea_scraper.py` into DAGs folder
- Environment variables for Gitea API token
- Network: ai-mesh (communicates with knowledge-service)

### Phase 1.6: Gitea Scraper Module
**Location:** `/home/sam/development/knowledge_service/gitea_scraper.py`

**Functionality:**
- **API Integration:** Gitea REST API v1
- **Authentication:** Token-based (Authorization header)
- **Methods:**
  - `get_user_repos()` - Paginated repo listing
  - `get_readme(repo_name)` - README content with fallback names
  - `get_repo_files(repo_name, path)` - Directory listing
  - `get_file_content(repo_name, filepath)` - File download

**Data Model:**
- `RepoMetadata` dataclass (name, description, url, branch, updated_at, language)

### Phase 1.7: Docker Infrastructure

**Network:**
- `ai-mesh` (external) - Shared bridge network for all services

**Services Overview:**
| Service | Port | Purpose | Dependencies |
|---------|------|---------|--------------|
| frontend | 5173 | React UI | backend |
| backend | 8000 | Chat Gateway | langgraph-service, db |
| db | 5432 | PostgreSQL (chat history) | - |
| knowledge-service | 8080 | RAG / Vector DB | - |
| langgraph-service | 8090 | Agent Orchestration | knowledge-service |
| airflow-webserver | 8081 | Workflow UI | postgres, redis |
| airflow-scheduler | - | DAG scheduling | postgres, redis |
| airflow-worker | - | Task execution | postgres, redis |
| redis | 6379 | Message broker | - |
| postgres (airflow) | - | Airflow metadata | - |

**Container Patterns:**
- All Python services use `/app/packages` + `/app/code` separation
- Node.js services use volume mounting for hot reload
- PostgreSQL uses named volumes for persistence
- External network (`ai-mesh`) for cross-service communication

---

## Phase 2: Multi-Source Knowledge Ingestion (IN PROGRESS)

### Goal
Expand beyond Gitea to ingest data from all self-hosted knowledge sources.

### Data Sources to Integrate:
1. **Notes & Documentation**
   - **Trilium Next** - Hierarchical note-taking (tree structure)
   - **Obsidian** - Markdown vault with backlinks
   - **Flatnotes** - Flat file markdown notes
   - **HedgeDoc** - Collaborative markdown editor

2. **Wiki**
   - **DokuWiki** - Structured wiki content

3. **Project Management**
   - **Vikunja** - Task lists and project tracking

4. **Media & Assets**
   - **Immich** - Photo/video metadata + Gemini Vision API for content description
   - **HomeBox** - Physical inventory with images

### Technical Approach:
- **Crawling:** Selenium/Playwright for JavaScript-heavy UIs
- **Extraction:** Firecrawl or LangChain loaders for structured content
- **Vision:** Gemini Vision API for image-to-text conversion
- **Storage:** ChromaDB (vectors) + PostgreSQL (metadata, hashes for deduplication)
- **Scheduling:** Additional Airflow DAGs per source

---

## Phase 3: Advanced Agent Capabilities

### Goal
Integrate external AI tools and expand agent capabilities.

### Agent Tooling:
1. **MCP (Model Context Protocol) Servers**
   - Git MCP - Local repository operations
   - Filesystem MCP - Secure file access
   - Memory MCP - Knowledge graph persistence
   - Custom Gitea MCP (if/when available)

2. **External Agents**
   - **Goose** - CLI-based agent for local task execution
   - **Aider** - AI pair programming
   - **Opencode** - Already integrated (Brain Agent)
   - **Automaker** - Workflow automation
   - **Autocoder** - Code generation

3. **Orchestration Tools**
   - **CAO CLI** - Agent orchestrator
   - **Agent Pipe** - Pipeline management

### Integration Pattern:
- Each external tool wrapped as LangGraph node
- Supervisor routes to appropriate specialist
- State management for multi-turn interactions

---

## Phase 4: Production Hardening

### Goal
Prepare system for production deployment.

### Authentication & Security:
- **Laravel** - User authentication service (Phase 4 original plan)
- **JWT tokens** - Session management
- **API key management** - Secure credential storage
- **Network policies** - Inter-service communication restrictions

### Monitoring & Observability:
- **LangSmith** - LLM tracing and debugging
- **Langfuse** - LLM observability (note: currently in per-project install list)
- **Prometheus/Grafana** - Metrics and dashboards
- **Airflow monitoring** - DAG success/failure alerting

### Scaling:
- **ChromaDB** - Migration to server mode for concurrent access
- **Airflow** - Multiple Celery workers
- **Load balancing** - Nginx reverse proxy
- **Backup strategies** - Vector DB snapshots, PostgreSQL dumps

---

## Phase 5: Workflow Automation & Visual Tools

### Goal
Add visual prototyping and automation capabilities.

### Tools to Integrate:
1. **Flowise** - Visual LangChain builder
   - Prototype agent flows without coding
   - Export to Python code
   - Debug RAG pipelines visually

2. **Windmill** - Turn scripts into workflows
   - Schedule Python/LangChain scripts
   - Reactive triggers (e.g., on-commit)
   - Low-code workflow builder

3. **Activepieces** - Event-driven automation
   - Webhook triggers from Gitea
   - Integration with external APIs
   - Visual workflow designer

4. **N8N** - Alternative workflow automation
   - Consider if Activepieces doesn't meet needs

### Use Cases:
- **On-commit triggers:** Gitea push → immediate re-scan → notification
- **Scheduled reports:** Weekly summary of new/updated projects
- **Reactive workflows:** New photo uploaded → Gemini Vision → update knowledge base

---

## Phase 6: Knowledge Library Options & RAG Enhancement

### Goal
Advanced retrieval and knowledge organization.

### RAG Pipeline Improvements:
1. **Hybrid Search**
   - Semantic search (ChromaDB) + Keyword search (PostgreSQL)
   - Re-ranking with cross-encoders
   - Query expansion and decomposition

2. **Multi-Modal RAG**
   - Image retrieval (Immich + CLIP embeddings)
   - Document parsing (PDFs, code files)
   - Structured data (tables, lists)

3. **Knowledge Organization**
   - Entity extraction and linking
   - Knowledge graph construction
   - Hierarchical chunking strategies

### Alternative Vector Stores (Evaluation):
- **pgvector** - PostgreSQL native (if ChromaDB limitations hit)
- **Weaviate** - GraphQL interface, hybrid search
- **Qdrant** - Rust-based, high performance
- **Milvus** - Enterprise-grade, distributed

---

## Phase 7: User Experience & Interface

### Goal
Enhanced frontend and interaction patterns.

### Frontend Enhancements:
1. **Chat Interface Improvements**
   - Streaming responses (Server-Sent Events)
   - Message threading and context
   - File upload for document ingestion
   - Image display (for Immich integration)

2. **Knowledge Browser**
   - View ingested documents
   - Search knowledge base directly
   - See confidence scores and sources
   - Manual document upload/ingestion trigger

3. **Agent Management**
   - View active agents
   - Configure agent behavior
   - Monitor agent performance
   - Override routing decisions

### Mobile & Accessibility:
- Responsive design improvements
- Mobile app (React Native or PWA)
- Accessibility compliance (WCAG)

---

## Technology Stack Summary

### Core Frameworks:
- **Backend:** FastAPI (Python 3.11)
- **Frontend:** Vite + React 19 + TypeScript
- **Styling:** Tailwind CSS
- **Database:** PostgreSQL 15
- **Vector DB:** ChromaDB 1.5.1

### AI/ML Stack:
- **LLM Orchestration:** LangGraph 1.0.9 + LangChain
- **Embeddings:** OpenAI via OpenRouter (text-embedding-3-small)
- **LLM:** OpenCode Brain (opencode-brain:5000)
- **Vision:** Gemini Vision API (Phase 2)

### Workflow & Scheduling:
- **Orchestration:** Apache Airflow 2.8.1 (CeleryExecutor)
- **Message Queue:** Redis
- **External Tools:** Flowise, Windmill, Activepieces

### Development Tools:
- **Containers:** Docker + Docker Compose
- **Networking:** Bridge network (ai-mesh)
- **Testing:** curl/httpx for API testing
- **Version Control:** Gitea (self-hosted)

### Skills Demonstrated:
- Containerized microservices architecture
- Multi-agent AI orchestration (LangGraph)
- Vector database implementation (RAG)
- ETL pipeline development (Airflow)
- API integration and web scraping
- Modular, maintainable code organization
- Industry-standard AI tooling (LangChain ecosystem)
- Workflow automation and scheduling