Ingestion redesigned: harvest from the estate, not just content/
- docs/phase-4-corpus.md rewritten: Gitea (all repos), vault via Gitea, Graphify graph, curated content as tier 1. Platform guides are config and public copy. - docs/phase-5-ai-overlay.md: citations to originals, tier preference, live source readout. - Board: AUD-144 (index content/ only) retired; AUD-147 replaces it, blocked by AUD-141 and AUD-142. AUD-145 relinked to AUD-143 and AUD-147.
This commit is contained in:
Sam Rolfe
@@ -1,81 +1,157 @@
|
||||
# Phase 4 — Corpus and retrieval
|
||||
# Phase 4 — Ingestion from sources
|
||||
|
||||
**Goal:** `content/` becomes a retrievable corpus with filters that work before the vector search, and a freshness audit that cannot silently rot.
|
||||
**Goal:** a fireable tool that reads the estate — Gitea, the notes vault, the knowledge graph — and keeps a searchable pool current. The pool is not built from `content/` alone; `content/` is the top tier of it.
|
||||
|
||||
**Depends on:** Phase 1 (content), Phase 2 (FTS5 index exists).
|
||||
**Depends on:** Phase 1 (content and schema), Phase 2 (FTS5 index).
|
||||
|
||||
**Why this changed.** An earlier version of this document said the corpus would be built from `content/` only. That was wrong. A hand-curated set cannot answer "how did the photo pipeline deduplicate" or "what did the garden controller do before the rewrite". The estate can. Curated claims answer questions about Sam; the estate answers questions about the work.
|
||||
|
||||
---
|
||||
|
||||
## Work
|
||||
## 1. Sources
|
||||
|
||||
### 4.1 Pipeline
|
||||
| Source | Reached by | What to take |
|
||||
|---|---|---|
|
||||
| **Gitea**, all repositories | API over HTTPS | `README.md`, file tree (paths only), selected source files, recent commit messages, language statistics, repo description and topics |
|
||||
| **`sam/obsidian-vault`** | Same Gitea API — the vault is a repo | Markdown body, front-matter title and tags, folder path |
|
||||
| **Graphify output** | `graph.json` committed to the vault repo | Nodes (label, source_file, community), edges (relation, confidence) |
|
||||
| **`content/`** | Local repo | The curated claims. **Tier 1** — outranks everything above on conflict |
|
||||
|
||||
```
|
||||
content/ ──► validate ──► chunk ──► enrich ──► embed ──► upsert Qdrant
|
||||
│
|
||||
SQLite FTS5 (from Phase 2) ┘
|
||||
Gitea is the only network dependency. The vault and the graph arrive through it, so there is no cross-machine copy and no SSH from `.13` to `.27`.
|
||||
|
||||
### What not to read
|
||||
|
||||
Media never becomes text. Images, video and audio are recorded as `image` node references, tagged with their path, and nothing more.
|
||||
|
||||
---
|
||||
|
||||
## 2. Platform guides
|
||||
|
||||
One file per platform in `platforms/<name>.yml`. They are configuration **and** public copy — the site shows them so a visitor can see what the assistant reads.
|
||||
|
||||
```yaml
|
||||
name: gitea
|
||||
enabled: true
|
||||
what: "Self-hosted Git. Source for every current project."
|
||||
fits: [ai, platform, edge]
|
||||
base: https://gitea.lab.audasmedia.com.au
|
||||
auth: GITEA_TOKEN
|
||||
include:
|
||||
repos: all
|
||||
exclude:
|
||||
paths: ["**/node_modules/**", "**/.venv/**", "**/.git/**", "**/*.env*",
|
||||
"**/.silverbullet.auth.json", "**/*.pem", "**/*.key"]
|
||||
repos: []
|
||||
read: [readme, tree, selected_source, commits, languages]
|
||||
limits: { max_documents: 4000, max_file_kb: 200 }
|
||||
```
|
||||
|
||||
- **Chunking:** 400–800 tokens, ~12% overlap, **code-aware** — never split inside a fenced block, table or front-matter.
|
||||
- **Deterministic IDs:** `sha256(source_path + chunk_index + content_hash)`. Re-running must update, not duplicate.
|
||||
- **Incremental:** hash the source set; only changed documents are re-chunked and re-embedded.
|
||||
- **One command** rebuilds from scratch. Deleting the Qdrant volume is always recoverable.
|
||||
```yaml
|
||||
name: obsidian
|
||||
enabled: true
|
||||
what: "Personal knowledge base. High-level notes on every system, in Sam's words."
|
||||
fits: [about, ai, platform, edge]
|
||||
base: sam/obsidian-vault
|
||||
include:
|
||||
paths: ["200 projects/**", "300 areas/**", "400 Personal Family/**",
|
||||
"100 Table of Contents/**", "010 inbox/**"]
|
||||
exclude:
|
||||
paths: [".obsidian/**", ".trash/**", ".venv/**", "templates/**",
|
||||
"**/*.excalidraw", "**/*.png", "**/*.jpg"]
|
||||
limits: { max_documents: 4000, max_file_kb: 200 }
|
||||
```
|
||||
|
||||
### 4.2 Payload schema (the retrieval filter)
|
||||
```yaml
|
||||
name: graphify
|
||||
enabled: true
|
||||
what: "Knowledge graph over the vault. Answers 'what connects X to Y' rather than 'what mentions X'."
|
||||
fits: [ai]
|
||||
base: sam/obsidian-vault
|
||||
read: [graph_json]
|
||||
```
|
||||
|
||||
`fits` drives the payload `area`, which is what lets the query filter before it searches.
|
||||
|
||||
---
|
||||
|
||||
## 3. Metadata on every chunk
|
||||
|
||||
```json
|
||||
{
|
||||
"area": "ai|platform|edge|products|history|about",
|
||||
"level": 1,
|
||||
"source_type": "doc|code|adr|metric",
|
||||
"confidence": "delivered|running|in-progress|abandoned|historic",
|
||||
"period": "2023–",
|
||||
"artefacts": ["..."],
|
||||
"last_verified": "2026-10-20",
|
||||
"platform": "gitea | obsidian | graphify | curated",
|
||||
"tier": 1,
|
||||
"source_type": "doc | code | commit | graph_node | claim",
|
||||
"area": "ai | platform | edge | products | history | about",
|
||||
"origin_url": "https://gitea.../sam/photo-pipeline/src/x.py",
|
||||
"path": "src/x.py",
|
||||
"title": "photo-pipeline / dedup",
|
||||
"confidence": "delivered | running | in-progress | abandoned | historic",
|
||||
"last_modified": "2026-10-14T05:00:00Z",
|
||||
"content_hash": "sha256:...",
|
||||
"public": true,
|
||||
"text": "..."
|
||||
}
|
||||
```
|
||||
|
||||
`area` and `confidence` are the two that matter: they let the query path filter *before* searching, and they let the answer qualify unfinished work.
|
||||
`tier: 1` is curated `content/`. Everything harvested is `tier: 2`. The query layer uses this to prefer the curated claim when the two disagree.
|
||||
|
||||
### 4.3 Embeddings
|
||||
- Batch, low volume, incremental.
|
||||
- **Record the embedding model and dimension per collection.** A provider change is a governed migration, not a silent corruption.
|
||||
- Provider decision pending Sam — see `reference/stack.md`. Build provider-agnostic behind one interface.
|
||||
|
||||
### 4.4 Scheduling
|
||||
- Prefect deployment on the existing Prefect server; **worker only**, do not deploy a second server.
|
||||
- Trigger: git push to `content/`, plus a nightly audit.
|
||||
- Nightly audit reports documents past their `last_verified` horizon and alerts via the existing Apprise service.
|
||||
`origin_url` is mandatory for harvested chunks. A chunk with no origin is not admitted — an answer that cannot be traced is worse than no answer.
|
||||
|
||||
---
|
||||
|
||||
## Definition of Done
|
||||
## 4. Pipeline
|
||||
|
||||
- [ ] One command rebuilds the corpus from an empty volume.
|
||||
- [ ] Running it twice produces no duplicate vectors (count is stable).
|
||||
- [ ] No chunk boundary lands inside a code fence, table or front-matter block.
|
||||
- [ ] Every chunk carries all payload fields; a missing field fails the run rather than defaulting.
|
||||
- [ ] Payload filtering demonstrably narrows results — demonstrate the same query with and without an `area` filter.
|
||||
- [ ] Incremental mode re-embeds only changed documents; prove it with the provider call count.
|
||||
- [ ] Freshness audit runs and reports stale documents.
|
||||
- [ ] Qdrant memory measured at 500 / 5,000 / 20,000 chunks and recorded in `reference/stack.md`, within its cap.
|
||||
```
|
||||
guides → harvest (per platform) → normalise to text → chunk → enrich → embed → upsert
|
||||
│
|
||||
└─ reject: excluded path, no origin, binary, oversize
|
||||
```
|
||||
|
||||
- **Chunking** is code-aware: 400–800 tokens, ~12% overlap, never split inside a fenced block, table, front-matter block or function. Use the language grammar where available; fall back to paragraph splitting.
|
||||
- **Deterministic IDs:** `sha256(origin_url + chunk_index + content_hash)`. Re-running updates, never duplicates.
|
||||
- **Incremental:** compare `content_hash` against what the pool already holds; embed only what changed. A full rebuild must always be possible and is one command.
|
||||
- **Graph** is not chunked. Nodes and edges go into their own collection, joined to note chunks by `path`, so a graph answer can name the note it came from.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
## 5. Scheduling
|
||||
|
||||
- Answering questions. Phase 4 ends at "retrieval returns the right chunks".
|
||||
- Query graphs, prompts, guardrails — Phase 5 and 6.
|
||||
- Ingesting anything outside `content/`. Four sources maximum, and this phase uses one: the repo.
|
||||
- On demand: one command.
|
||||
- Scheduled: a Prefect deployment on the existing server. Worker only; do not deploy a second Prefect server.
|
||||
- Reports what changed each run: documents added, updated, removed, and any excluded-path rejections.
|
||||
- A source that has not changed in its expected window raises an alert through the existing Apprise service. Silent staleness is the failure mode that matters.
|
||||
|
||||
---
|
||||
|
||||
## Risks
|
||||
## 6. Definition of Done
|
||||
|
||||
- [ ] One command rebuilds the pool from empty.
|
||||
- [ ] A second run produces no duplicate vectors.
|
||||
- [ ] A probe document placed in an excluded path never appears in the pool, and the run reports the rejection.
|
||||
- [ ] A document with no resolvable origin is refused, not defaulted.
|
||||
- [ ] Every chunk carries all fields in section 3.
|
||||
- [ ] Incremental mode re-embeds only changed documents, proven by embedding call count.
|
||||
- [ ] Graph nodes and edges are queryable and each resolves to a source note.
|
||||
- [ ] The scheduled run reports deltas and alerts on staleness.
|
||||
- [ ] Qdrant stays inside its `mem_limit`; footprint recorded at 1k, 10k and 40k chunks.
|
||||
- [ ] No credential, `.env`, key file or family-personal binary is present in the pool — verified by search, not by assertion.
|
||||
|
||||
---
|
||||
|
||||
## 7. Out of scope
|
||||
|
||||
- Answering questions. This phase ends at "retrieval returns the right chunks with their origins".
|
||||
- The query graph, prompts and guardrails — Phase 5.
|
||||
- Building the Graphify graph. It already exists and is rebuilt on `.27` with its own skill. This phase consumes its output.
|
||||
|
||||
---
|
||||
|
||||
## 8. Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Corpus grows beyond RAM | Incremental by default; measure at three corpus sizes; cap Qdrant with `mem_limit` |
|
||||
| Duplicate vectors on re-run | Deterministic IDs are mandatory, not optional |
|
||||
| Silent staleness | Nightly audit + Apprise alert; a stale index is otherwise an invisible quality failure |
|
||||
| Provider rate limits during bulk embed | Batch with retry/backoff; ingest is low-volume and incremental |
|
||||
| Volume — 50 repos plus 280 notes exceeds the memory cap | Cap per platform, skip large files, measure at three corpus sizes |
|
||||
| Scraped text contradicts a curated claim | `tier` field; the query prefers tier 1 and says so |
|
||||
| A secret is harvested | Exclusion list plus a post-run scan of the pool for credential patterns, failing the run if found |
|
||||
| Staleness is invisible | Scheduled run plus alert |
|
||||
| The estate is unavailable when a recruiter asks | The pool is local; only ingestion needs Gitea. Serving never does |
|
||||
|
||||
@@ -80,3 +80,43 @@ One hand-written TypeScript island, ES modules, no framework, no bundler.
|
||||
| Panel refactor needed | Phase 2 §2.4 contract; if it fails, fix the panel contract, do not fork the UI |
|
||||
| Streaming stalls behind Caddy | Explicit flush settings; p90 measurement, not a spot check |
|
||||
| Answers drift from the site | Citation verification is a node, not a hope |
|
||||
|
||||
---
|
||||
|
||||
## 5.6 Citations point at the original
|
||||
|
||||
An answer cites where it came from, not a curated page standing in for it.
|
||||
|
||||
| Source | Cited as |
|
||||
|---|---|
|
||||
| Gitea repo file | full URL to the file in Gitea |
|
||||
| Vault note | full URL to the note in `sam/obsidian-vault` |
|
||||
| Graph answer | the note title the node came from, plus its URL |
|
||||
| Curated claim | the site page |
|
||||
|
||||
A citation whose URL cannot be opened is dropped, not rendered. A claim with no
|
||||
traceable origin is not returned at all.
|
||||
|
||||
## 5.7 Curated outranks harvested
|
||||
|
||||
`content/` is tier 1; everything harvested is tier 2. When the two disagree, the
|
||||
tier 1 claim wins and the answer says so plainly: the curated description is
|
||||
current; the older text is background.
|
||||
|
||||
This is the rule that stops a stale README from quietly overwriting Sam's own
|
||||
description of his work.
|
||||
|
||||
## 5.8 The live source readout
|
||||
|
||||
While the answer streams, a panel shows what is being read, named as the visitor
|
||||
would recognise it:
|
||||
|
||||
```
|
||||
reading · gitea/photo-pipeline/README.md
|
||||
· obsidian/210 AI Resume/Graphify.md
|
||||
· graph · community 42 · vault infrastructure
|
||||
```
|
||||
|
||||
This is a feature, not a debug view. It answers the question every recruiter has
|
||||
and does not ask: is this making things up? Show the working — it is also the most
|
||||
convincing demonstration of the system on the page.
|
||||
|
||||
Reference in New Issue
Block a user