# Phase 4 — Corpus and retrieval

**Goal:** `content/` becomes a retrievable corpus with filters that work before the vector search, and a freshness audit that cannot silently rot.

**Depends on:** Phase 1 (content), Phase 2 (FTS5 index exists).

---

## Work

### 4.1 Pipeline

```
content/  ──► validate ──► chunk ──► enrich ──► embed ──► upsert Qdrant
                                                              │
                                    SQLite FTS5 (from Phase 2) ┘
```

- **Chunking:** 400–800 tokens, ~12% overlap, **code-aware** — never split inside a fenced block, table or front-matter.
- **Deterministic IDs:** `sha256(source_path + chunk_index + content_hash)`. Re-running must update, not duplicate.
- **Incremental:** hash the source set; only changed documents are re-chunked and re-embedded.
- **One command** rebuilds from scratch. Deleting the Qdrant volume is always recoverable.

### 4.2 Payload schema (the retrieval filter)

```json
{
  "area": "ai|platform|edge|products|history|about",
  "level": 1,
  "source_type": "doc|code|adr|metric",
  "confidence": "delivered|running|in-progress|abandoned|historic",
  "period": "2023–",
  "artefacts": ["..."],
  "last_verified": "2026-10-20",
  "text": "..."
}
```

`area` and `confidence` are the two that matter: they let the query path filter *before* searching, and they let the answer qualify unfinished work.

### 4.3 Embeddings
- Batch, low volume, incremental.
- **Record the embedding model and dimension per collection.** A provider change is a governed migration, not a silent corruption.
- Provider decision pending Sam — see `reference/stack.md`. Build provider-agnostic behind one interface.

### 4.4 Scheduling
- Prefect deployment on the existing Prefect server; **worker only**, do not deploy a second server.
- Trigger: git push to `content/`, plus a nightly audit.
- Nightly audit reports documents past their `last_verified` horizon and alerts via the existing Apprise service.

---

## Definition of Done

- [ ] One command rebuilds the corpus from an empty volume.
- [ ] Running it twice produces no duplicate vectors (count is stable).
- [ ] No chunk boundary lands inside a code fence, table or front-matter block.
- [ ] Every chunk carries all payload fields; a missing field fails the run rather than defaulting.
- [ ] Payload filtering demonstrably narrows results — demonstrate the same query with and without an `area` filter.
- [ ] Incremental mode re-embeds only changed documents; prove it with the provider call count.
- [ ] Freshness audit runs and reports stale documents.
- [ ] Qdrant memory measured at 500 / 5,000 / 20,000 chunks and recorded in `reference/stack.md`, within its cap.

---

## Out of scope

- Answering questions. Phase 4 ends at "retrieval returns the right chunks".
- Query graphs, prompts, guardrails — Phase 5 and 6.
- Ingesting anything outside `content/`. Four sources maximum, and this phase uses one: the repo.

---

## Risks

| Risk | Mitigation |
|---|---|
| Corpus grows beyond RAM | Incremental by default; measure at three corpus sizes; cap Qdrant with `mem_limit` |
| Duplicate vectors on re-run | Deterministic IDs are mandatory, not optional |
| Silent staleness | Nightly audit + Apprise alert; a stale index is otherwise an invisible quality failure |
| Provider rate limits during bulk embed | Batch with retry/backoff; ingest is low-volume and incremental |