From 5c0622c1b1fb8f4faf5d0dcab6b4b444fa57326b Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Mon, 26 Oct 2026 22:16:33 +1100 Subject: [PATCH] Ingestion redesigned: harvest from the estate, not just content/ - docs/phase-4-corpus.md rewritten: Gitea (all repos), vault via Gitea, Graphify graph, curated content as tier 1. Platform guides are config and public copy. - docs/phase-5-ai-overlay.md: citations to originals, tier preference, live source readout. - Board: AUD-144 (index content/ only) retired; AUD-147 replaces it, blocked by AUD-141 and AUD-142. AUD-145 relinked to AUD-143 and AUD-147. --- docs/phase-4-corpus.md | 174 ++++++++++++++++++++++++++----------- docs/phase-5-ai-overlay.md | 40 +++++++++ 2 files changed, 165 insertions(+), 49 deletions(-) diff --git a/docs/phase-4-corpus.md b/docs/phase-4-corpus.md index 5d40066..2a4a41f 100644 --- a/docs/phase-4-corpus.md +++ b/docs/phase-4-corpus.md @@ -1,81 +1,157 @@ -# Phase 4 — Corpus and retrieval +# Phase 4 — Ingestion from sources -**Goal:** `content/` becomes a retrievable corpus with filters that work before the vector search, and a freshness audit that cannot silently rot. +**Goal:** a fireable tool that reads the estate — Gitea, the notes vault, the knowledge graph — and keeps a searchable pool current. The pool is not built from `content/` alone; `content/` is the top tier of it. -**Depends on:** Phase 1 (content), Phase 2 (FTS5 index exists). +**Depends on:** Phase 1 (content and schema), Phase 2 (FTS5 index). + +**Why this changed.** An earlier version of this document said the corpus would be built from `content/` only. That was wrong. A hand-curated set cannot answer "how did the photo pipeline deduplicate" or "what did the garden controller do before the rewrite". The estate can. Curated claims answer questions about Sam; the estate answers questions about the work. --- -## Work +## 1. Sources -### 4.1 Pipeline +| Source | Reached by | What to take | +|---|---|---| +| **Gitea**, all repositories | API over HTTPS | `README.md`, file tree (paths only), selected source files, recent commit messages, language statistics, repo description and topics | +| **`sam/obsidian-vault`** | Same Gitea API — the vault is a repo | Markdown body, front-matter title and tags, folder path | +| **Graphify output** | `graph.json` committed to the vault repo | Nodes (label, source_file, community), edges (relation, confidence) | +| **`content/`** | Local repo | The curated claims. **Tier 1** — outranks everything above on conflict | -``` -content/ ──► validate ──► chunk ──► enrich ──► embed ──► upsert Qdrant - │ - SQLite FTS5 (from Phase 2) ┘ +Gitea is the only network dependency. The vault and the graph arrive through it, so there is no cross-machine copy and no SSH from `.13` to `.27`. + +### What not to read + +Media never becomes text. Images, video and audio are recorded as `image` node references, tagged with their path, and nothing more. + +--- + +## 2. Platform guides + +One file per platform in `platforms/.yml`. They are configuration **and** public copy — the site shows them so a visitor can see what the assistant reads. + +```yaml +name: gitea +enabled: true +what: "Self-hosted Git. Source for every current project." +fits: [ai, platform, edge] +base: https://gitea.lab.audasmedia.com.au +auth: GITEA_TOKEN +include: + repos: all +exclude: + paths: ["**/node_modules/**", "**/.venv/**", "**/.git/**", "**/*.env*", + "**/.silverbullet.auth.json", "**/*.pem", "**/*.key"] + repos: [] +read: [readme, tree, selected_source, commits, languages] +limits: { max_documents: 4000, max_file_kb: 200 } ``` -- **Chunking:** 400–800 tokens, ~12% overlap, **code-aware** — never split inside a fenced block, table or front-matter. -- **Deterministic IDs:** `sha256(source_path + chunk_index + content_hash)`. Re-running must update, not duplicate. -- **Incremental:** hash the source set; only changed documents are re-chunked and re-embedded. -- **One command** rebuilds from scratch. Deleting the Qdrant volume is always recoverable. +```yaml +name: obsidian +enabled: true +what: "Personal knowledge base. High-level notes on every system, in Sam's words." +fits: [about, ai, platform, edge] +base: sam/obsidian-vault +include: + paths: ["200 projects/**", "300 areas/**", "400 Personal Family/**", + "100 Table of Contents/**", "010 inbox/**"] +exclude: + paths: [".obsidian/**", ".trash/**", ".venv/**", "templates/**", + "**/*.excalidraw", "**/*.png", "**/*.jpg"] +limits: { max_documents: 4000, max_file_kb: 200 } +``` -### 4.2 Payload schema (the retrieval filter) +```yaml +name: graphify +enabled: true +what: "Knowledge graph over the vault. Answers 'what connects X to Y' rather than 'what mentions X'." +fits: [ai] +base: sam/obsidian-vault +read: [graph_json] +``` + +`fits` drives the payload `area`, which is what lets the query filter before it searches. + +--- + +## 3. Metadata on every chunk ```json { - "area": "ai|platform|edge|products|history|about", - "level": 1, - "source_type": "doc|code|adr|metric", - "confidence": "delivered|running|in-progress|abandoned|historic", - "period": "2023–", - "artefacts": ["..."], - "last_verified": "2026-10-20", + "platform": "gitea | obsidian | graphify | curated", + "tier": 1, + "source_type": "doc | code | commit | graph_node | claim", + "area": "ai | platform | edge | products | history | about", + "origin_url": "https://gitea.../sam/photo-pipeline/src/x.py", + "path": "src/x.py", + "title": "photo-pipeline / dedup", + "confidence": "delivered | running | in-progress | abandoned | historic", + "last_modified": "2026-10-14T05:00:00Z", + "content_hash": "sha256:...", + "public": true, "text": "..." } ``` -`area` and `confidence` are the two that matter: they let the query path filter *before* searching, and they let the answer qualify unfinished work. +`tier: 1` is curated `content/`. Everything harvested is `tier: 2`. The query layer uses this to prefer the curated claim when the two disagree. -### 4.3 Embeddings -- Batch, low volume, incremental. -- **Record the embedding model and dimension per collection.** A provider change is a governed migration, not a silent corruption. -- Provider decision pending Sam — see `reference/stack.md`. Build provider-agnostic behind one interface. - -### 4.4 Scheduling -- Prefect deployment on the existing Prefect server; **worker only**, do not deploy a second server. -- Trigger: git push to `content/`, plus a nightly audit. -- Nightly audit reports documents past their `last_verified` horizon and alerts via the existing Apprise service. +`origin_url` is mandatory for harvested chunks. A chunk with no origin is not admitted — an answer that cannot be traced is worse than no answer. --- -## Definition of Done +## 4. Pipeline -- [ ] One command rebuilds the corpus from an empty volume. -- [ ] Running it twice produces no duplicate vectors (count is stable). -- [ ] No chunk boundary lands inside a code fence, table or front-matter block. -- [ ] Every chunk carries all payload fields; a missing field fails the run rather than defaulting. -- [ ] Payload filtering demonstrably narrows results — demonstrate the same query with and without an `area` filter. -- [ ] Incremental mode re-embeds only changed documents; prove it with the provider call count. -- [ ] Freshness audit runs and reports stale documents. -- [ ] Qdrant memory measured at 500 / 5,000 / 20,000 chunks and recorded in `reference/stack.md`, within its cap. +``` +guides → harvest (per platform) → normalise to text → chunk → enrich → embed → upsert + │ + └─ reject: excluded path, no origin, binary, oversize +``` + +- **Chunking** is code-aware: 400–800 tokens, ~12% overlap, never split inside a fenced block, table, front-matter block or function. Use the language grammar where available; fall back to paragraph splitting. +- **Deterministic IDs:** `sha256(origin_url + chunk_index + content_hash)`. Re-running updates, never duplicates. +- **Incremental:** compare `content_hash` against what the pool already holds; embed only what changed. A full rebuild must always be possible and is one command. +- **Graph** is not chunked. Nodes and edges go into their own collection, joined to note chunks by `path`, so a graph answer can name the note it came from. --- -## Out of scope +## 5. Scheduling -- Answering questions. Phase 4 ends at "retrieval returns the right chunks". -- Query graphs, prompts, guardrails — Phase 5 and 6. -- Ingesting anything outside `content/`. Four sources maximum, and this phase uses one: the repo. +- On demand: one command. +- Scheduled: a Prefect deployment on the existing server. Worker only; do not deploy a second Prefect server. +- Reports what changed each run: documents added, updated, removed, and any excluded-path rejections. +- A source that has not changed in its expected window raises an alert through the existing Apprise service. Silent staleness is the failure mode that matters. --- -## Risks +## 6. Definition of Done + +- [ ] One command rebuilds the pool from empty. +- [ ] A second run produces no duplicate vectors. +- [ ] A probe document placed in an excluded path never appears in the pool, and the run reports the rejection. +- [ ] A document with no resolvable origin is refused, not defaulted. +- [ ] Every chunk carries all fields in section 3. +- [ ] Incremental mode re-embeds only changed documents, proven by embedding call count. +- [ ] Graph nodes and edges are queryable and each resolves to a source note. +- [ ] The scheduled run reports deltas and alerts on staleness. +- [ ] Qdrant stays inside its `mem_limit`; footprint recorded at 1k, 10k and 40k chunks. +- [ ] No credential, `.env`, key file or family-personal binary is present in the pool — verified by search, not by assertion. + +--- + +## 7. Out of scope + +- Answering questions. This phase ends at "retrieval returns the right chunks with their origins". +- The query graph, prompts and guardrails — Phase 5. +- Building the Graphify graph. It already exists and is rebuilt on `.27` with its own skill. This phase consumes its output. + +--- + +## 8. Risks | Risk | Mitigation | |---|---| -| Corpus grows beyond RAM | Incremental by default; measure at three corpus sizes; cap Qdrant with `mem_limit` | -| Duplicate vectors on re-run | Deterministic IDs are mandatory, not optional | -| Silent staleness | Nightly audit + Apprise alert; a stale index is otherwise an invisible quality failure | -| Provider rate limits during bulk embed | Batch with retry/backoff; ingest is low-volume and incremental | +| Volume — 50 repos plus 280 notes exceeds the memory cap | Cap per platform, skip large files, measure at three corpus sizes | +| Scraped text contradicts a curated claim | `tier` field; the query prefers tier 1 and says so | +| A secret is harvested | Exclusion list plus a post-run scan of the pool for credential patterns, failing the run if found | +| Staleness is invisible | Scheduled run plus alert | +| The estate is unavailable when a recruiter asks | The pool is local; only ingestion needs Gitea. Serving never does | diff --git a/docs/phase-5-ai-overlay.md b/docs/phase-5-ai-overlay.md index db5bb6d..21196aa 100644 --- a/docs/phase-5-ai-overlay.md +++ b/docs/phase-5-ai-overlay.md @@ -80,3 +80,43 @@ One hand-written TypeScript island, ES modules, no framework, no bundler. | Panel refactor needed | Phase 2 §2.4 contract; if it fails, fix the panel contract, do not fork the UI | | Streaming stalls behind Caddy | Explicit flush settings; p90 measurement, not a spot check | | Answers drift from the site | Citation verification is a node, not a hope | + +--- + +## 5.6 Citations point at the original + +An answer cites where it came from, not a curated page standing in for it. + +| Source | Cited as | +|---|---| +| Gitea repo file | full URL to the file in Gitea | +| Vault note | full URL to the note in `sam/obsidian-vault` | +| Graph answer | the note title the node came from, plus its URL | +| Curated claim | the site page | + +A citation whose URL cannot be opened is dropped, not rendered. A claim with no +traceable origin is not returned at all. + +## 5.7 Curated outranks harvested + +`content/` is tier 1; everything harvested is tier 2. When the two disagree, the +tier 1 claim wins and the answer says so plainly: the curated description is +current; the older text is background. + +This is the rule that stops a stale README from quietly overwriting Sam's own +description of his work. + +## 5.8 The live source readout + +While the answer streams, a panel shows what is being read, named as the visitor +would recognise it: + +``` +reading · gitea/photo-pipeline/README.md + · obsidian/210 AI Resume/Graphify.md + · graph · community 42 · vault infrastructure +``` + +This is a feature, not a debug view. It answers the question every recruiter has +and does not ask: is this making things up? Show the working — it is also the most +convincing demonstration of the system on the page.