ai-resume/docs/phase-4-corpus.md

Phase 4 — Ingestion from sources

Goal: a fireable tool that reads the estate — Gitea, the notes vault, the knowledge graph — and keeps a searchable pool current. The pool is not built from content/ alone; content/ is the top tier of it.

Depends on: Phase 1 (content and schema), Phase 2 (FTS5 index).

Why this changed. An earlier version of this document said the corpus would be built from content/ only. That was wrong. A hand-curated set cannot answer "how did the photo pipeline deduplicate" or "what did the garden controller do before the rewrite". The estate can. Curated claims answer questions about Sam; the estate answers questions about the work.

---

1. Sources

Source	Reached by	What to take
Gitea, all repositories	API over HTTPS	README.md, file tree (paths only), selected source files, recent commit messages, language statistics, repo description and topics
sam/obsidian-vault	Same Gitea API — the vault is a repo	Markdown body, front-matter title and tags, folder path
Graphify output	graph.json committed to the vault repo	Nodes (label, source_file, community), edges (relation, confidence)
content/	Local repo	The curated claims. Tier 1 — outranks everything above on conflict

Gitea is the only network dependency. The vault and the graph arrive through it, so there is no cross-machine copy and no SSH from .13 to .27.

What not to read

Media never becomes text. Images, video and audio are recorded as image node references, tagged with their path, and nothing more.

---

2. Platform guides

One file per platform in platforms/<name>.yml. They are configuration and public copy — the site shows them so a visitor can see what the assistant reads.

name: gitea
enabled: true
what: "Self-hosted Git. Source for every current project."
fits: [ai, platform, edge]
base: https://gitea.lab.audasmedia.com.au
auth: GITEA_TOKEN
include:
  repos: all
exclude:
  paths: ["**/node_modules/**", "**/.venv/**", "**/.git/**", "**/*.env*",
          "**/.silverbullet.auth.json", "**/*.pem", "**/*.key"]
  repos: []
read: [readme, tree, selected_source, commits, languages]
limits: { max_documents: 4000, max_file_kb: 200 }

name: obsidian
enabled: true
what: "Personal knowledge base. High-level notes on every system, in Sam's words."
fits: [about, ai, platform, edge]
base: sam/obsidian-vault
include:
  paths: ["200 projects/**", "300 areas/**", "400 Personal Family/**",
          "100 Table of Contents/**", "010 inbox/**"]
exclude:
  paths: [".obsidian/**", ".trash/**", ".venv/**", "templates/**",
          "**/*.excalidraw", "**/*.png", "**/*.jpg"]
limits: { max_documents: 4000, max_file_kb: 200 }

name: graphify
enabled: true
what: "Knowledge graph over the vault. Answers 'what connects X to Y' rather than 'what mentions X'."
fits: [ai]
base: sam/obsidian-vault
read: [graph_json]

fits drives the payload area, which is what lets the query filter before it searches.

---

3. Metadata on every chunk

{
  "platform": "gitea | obsidian | graphify | curated",
  "tier": 1,
  "source_type": "doc | code | commit | graph_node | claim",
  "area": "ai | platform | edge | products | history | about",
  "origin_url": "https://gitea.../sam/photo-pipeline/src/x.py",
  "path": "src/x.py",
  "title": "photo-pipeline / dedup",
  "confidence": "delivered | running | in-progress | abandoned | historic",
  "last_modified": "2026-10-14T05:00:00Z",
  "content_hash": "sha256:...",
  "public": true,
  "text": "..."
}

tier: 1 is curated content/. Everything harvested is tier: 2. The query layer uses this to prefer the curated claim when the two disagree.

origin_url is mandatory for harvested chunks. A chunk with no origin is not admitted — an answer that cannot be traced is worse than no answer.

---

4. Pipeline

guides  →  harvest (per platform)  →  normalise to text  →  chunk  →  enrich  →  embed  →  upsert
                                        │
                                        └─ reject: excluded path, no origin, binary, oversize

• Chunking is code-aware: 400–800 tokens, ~12% overlap, never split inside a fenced block, table, front-matter block or function. Use the language grammar where available; fall back to paragraph splitting.
• Deterministic IDs: sha256(origin_url + chunk_index + content_hash). Re-running updates, never duplicates.
• Incremental: compare content_hash against what the pool already holds; embed only what changed. A full rebuild must always be possible and is one command.
• Graph is not chunked. Nodes and edges go into their own collection, joined to note chunks by path, so a graph answer can name the note it came from.

---

5. Scheduling

• On demand: one command.
• Scheduled: a Prefect deployment on the existing server. Worker only; do not deploy a second Prefect server.
• Reports what changed each run: documents added, updated, removed, and any excluded-path rejections.
• A source that has not changed in its expected window raises an alert through the existing Apprise service. Silent staleness is the failure mode that matters.

---

6. Definition of Done

• One command rebuilds the pool from empty.
• A second run produces no duplicate vectors.
• A probe document placed in an excluded path never appears in the pool, and the run reports the rejection.
• A document with no resolvable origin is refused, not defaulted.
• Every chunk carries all fields in section 3.
• Incremental mode re-embeds only changed documents, proven by embedding call count.
• Graph nodes and edges are queryable and each resolves to a source note.
• The scheduled run reports deltas and alerts on staleness.
• Qdrant stays inside its mem_limit; footprint recorded at 1k, 10k and 40k chunks.
• No credential, .env, key file or family-personal binary is present in the pool — verified by search, not by assertion.

---

7. Out of scope

• Answering questions. This phase ends at "retrieval returns the right chunks with their origins".
• The query graph, prompts and guardrails — Phase 5.
• Building the Graphify graph. It already exists and is rebuilt on .27 with its own skill. This phase consumes its output.

---

8. Risks

Risk	Mitigation
Volume — 50 repos plus 280 notes exceeds the memory cap	Cap per platform, skip large files, measure at three corpus sizes
Scraped text contradicts a curated claim	tier field; the query prefers tier 1 and says so
A secret is harvested	Exclusion list plus a post-run scan of the pool for credential patterns, failing the run if found
Staleness is invisible	Scheduled run plus alert
The estate is unavailable when a recruiter asks	The pool is local; only ingestion needs Gitea. Serving never does