5c0622c1b1
- docs/phase-4-corpus.md rewritten: Gitea (all repos), vault via Gitea, Graphify graph, curated content as tier 1. Platform guides are config and public copy. - docs/phase-5-ai-overlay.md: citations to originals, tier preference, live source readout. - Board: AUD-144 (index content/ only) retired; AUD-147 replaces it, blocked by AUD-141 and AUD-142. AUD-145 relinked to AUD-143 and AUD-147.
158 lines
6.8 KiB
Markdown
158 lines
6.8 KiB
Markdown
# Phase 4 — Ingestion from sources
|
||
|
||
**Goal:** a fireable tool that reads the estate — Gitea, the notes vault, the knowledge graph — and keeps a searchable pool current. The pool is not built from `content/` alone; `content/` is the top tier of it.
|
||
|
||
**Depends on:** Phase 1 (content and schema), Phase 2 (FTS5 index).
|
||
|
||
**Why this changed.** An earlier version of this document said the corpus would be built from `content/` only. That was wrong. A hand-curated set cannot answer "how did the photo pipeline deduplicate" or "what did the garden controller do before the rewrite". The estate can. Curated claims answer questions about Sam; the estate answers questions about the work.
|
||
|
||
---
|
||
|
||
## 1. Sources
|
||
|
||
| Source | Reached by | What to take |
|
||
|---|---|---|
|
||
| **Gitea**, all repositories | API over HTTPS | `README.md`, file tree (paths only), selected source files, recent commit messages, language statistics, repo description and topics |
|
||
| **`sam/obsidian-vault`** | Same Gitea API — the vault is a repo | Markdown body, front-matter title and tags, folder path |
|
||
| **Graphify output** | `graph.json` committed to the vault repo | Nodes (label, source_file, community), edges (relation, confidence) |
|
||
| **`content/`** | Local repo | The curated claims. **Tier 1** — outranks everything above on conflict |
|
||
|
||
Gitea is the only network dependency. The vault and the graph arrive through it, so there is no cross-machine copy and no SSH from `.13` to `.27`.
|
||
|
||
### What not to read
|
||
|
||
Media never becomes text. Images, video and audio are recorded as `image` node references, tagged with their path, and nothing more.
|
||
|
||
---
|
||
|
||
## 2. Platform guides
|
||
|
||
One file per platform in `platforms/<name>.yml`. They are configuration **and** public copy — the site shows them so a visitor can see what the assistant reads.
|
||
|
||
```yaml
|
||
name: gitea
|
||
enabled: true
|
||
what: "Self-hosted Git. Source for every current project."
|
||
fits: [ai, platform, edge]
|
||
base: https://gitea.lab.audasmedia.com.au
|
||
auth: GITEA_TOKEN
|
||
include:
|
||
repos: all
|
||
exclude:
|
||
paths: ["**/node_modules/**", "**/.venv/**", "**/.git/**", "**/*.env*",
|
||
"**/.silverbullet.auth.json", "**/*.pem", "**/*.key"]
|
||
repos: []
|
||
read: [readme, tree, selected_source, commits, languages]
|
||
limits: { max_documents: 4000, max_file_kb: 200 }
|
||
```
|
||
|
||
```yaml
|
||
name: obsidian
|
||
enabled: true
|
||
what: "Personal knowledge base. High-level notes on every system, in Sam's words."
|
||
fits: [about, ai, platform, edge]
|
||
base: sam/obsidian-vault
|
||
include:
|
||
paths: ["200 projects/**", "300 areas/**", "400 Personal Family/**",
|
||
"100 Table of Contents/**", "010 inbox/**"]
|
||
exclude:
|
||
paths: [".obsidian/**", ".trash/**", ".venv/**", "templates/**",
|
||
"**/*.excalidraw", "**/*.png", "**/*.jpg"]
|
||
limits: { max_documents: 4000, max_file_kb: 200 }
|
||
```
|
||
|
||
```yaml
|
||
name: graphify
|
||
enabled: true
|
||
what: "Knowledge graph over the vault. Answers 'what connects X to Y' rather than 'what mentions X'."
|
||
fits: [ai]
|
||
base: sam/obsidian-vault
|
||
read: [graph_json]
|
||
```
|
||
|
||
`fits` drives the payload `area`, which is what lets the query filter before it searches.
|
||
|
||
---
|
||
|
||
## 3. Metadata on every chunk
|
||
|
||
```json
|
||
{
|
||
"platform": "gitea | obsidian | graphify | curated",
|
||
"tier": 1,
|
||
"source_type": "doc | code | commit | graph_node | claim",
|
||
"area": "ai | platform | edge | products | history | about",
|
||
"origin_url": "https://gitea.../sam/photo-pipeline/src/x.py",
|
||
"path": "src/x.py",
|
||
"title": "photo-pipeline / dedup",
|
||
"confidence": "delivered | running | in-progress | abandoned | historic",
|
||
"last_modified": "2026-10-14T05:00:00Z",
|
||
"content_hash": "sha256:...",
|
||
"public": true,
|
||
"text": "..."
|
||
}
|
||
```
|
||
|
||
`tier: 1` is curated `content/`. Everything harvested is `tier: 2`. The query layer uses this to prefer the curated claim when the two disagree.
|
||
|
||
`origin_url` is mandatory for harvested chunks. A chunk with no origin is not admitted — an answer that cannot be traced is worse than no answer.
|
||
|
||
---
|
||
|
||
## 4. Pipeline
|
||
|
||
```
|
||
guides → harvest (per platform) → normalise to text → chunk → enrich → embed → upsert
|
||
│
|
||
└─ reject: excluded path, no origin, binary, oversize
|
||
```
|
||
|
||
- **Chunking** is code-aware: 400–800 tokens, ~12% overlap, never split inside a fenced block, table, front-matter block or function. Use the language grammar where available; fall back to paragraph splitting.
|
||
- **Deterministic IDs:** `sha256(origin_url + chunk_index + content_hash)`. Re-running updates, never duplicates.
|
||
- **Incremental:** compare `content_hash` against what the pool already holds; embed only what changed. A full rebuild must always be possible and is one command.
|
||
- **Graph** is not chunked. Nodes and edges go into their own collection, joined to note chunks by `path`, so a graph answer can name the note it came from.
|
||
|
||
---
|
||
|
||
## 5. Scheduling
|
||
|
||
- On demand: one command.
|
||
- Scheduled: a Prefect deployment on the existing server. Worker only; do not deploy a second Prefect server.
|
||
- Reports what changed each run: documents added, updated, removed, and any excluded-path rejections.
|
||
- A source that has not changed in its expected window raises an alert through the existing Apprise service. Silent staleness is the failure mode that matters.
|
||
|
||
---
|
||
|
||
## 6. Definition of Done
|
||
|
||
- [ ] One command rebuilds the pool from empty.
|
||
- [ ] A second run produces no duplicate vectors.
|
||
- [ ] A probe document placed in an excluded path never appears in the pool, and the run reports the rejection.
|
||
- [ ] A document with no resolvable origin is refused, not defaulted.
|
||
- [ ] Every chunk carries all fields in section 3.
|
||
- [ ] Incremental mode re-embeds only changed documents, proven by embedding call count.
|
||
- [ ] Graph nodes and edges are queryable and each resolves to a source note.
|
||
- [ ] The scheduled run reports deltas and alerts on staleness.
|
||
- [ ] Qdrant stays inside its `mem_limit`; footprint recorded at 1k, 10k and 40k chunks.
|
||
- [ ] No credential, `.env`, key file or family-personal binary is present in the pool — verified by search, not by assertion.
|
||
|
||
---
|
||
|
||
## 7. Out of scope
|
||
|
||
- Answering questions. This phase ends at "retrieval returns the right chunks with their origins".
|
||
- The query graph, prompts and guardrails — Phase 5.
|
||
- Building the Graphify graph. It already exists and is rebuilt on `.27` with its own skill. This phase consumes its output.
|
||
|
||
---
|
||
|
||
## 8. Risks
|
||
|
||
| Risk | Mitigation |
|
||
|---|---|
|
||
| Volume — 50 repos plus 280 notes exceeds the memory cap | Cap per platform, skip large files, measure at three corpus sizes |
|
||
| Scraped text contradicts a curated claim | `tier` field; the query prefers tier 1 and says so |
|
||
| A secret is harvested | Exclusion list plus a post-run scan of the pool for credential patterns, failing the run if found |
|
||
| Staleness is invisible | Scheduled run plus alert |
|
||
| The estate is unavailable when a recruiter asks | The pool is local; only ingestion needs Gitea. Serving never does |
|