b267e55f8c
- AGENTS.md: build spec for Paperclip (constraints, conventions, 7 phases) - docs/phase-1..7: per-phase work, DoD, out-of-scope, risks - reference/: frozen decisions, stack+ports+budgets, content map, acceptance tests - agents/: persona and security policy drafts - Retires the React scaffold; history retained as record of the false start
82 lines
3.4 KiB
Markdown
82 lines
3.4 KiB
Markdown
# Phase 4 — Corpus and retrieval
|
||
|
||
**Goal:** `content/` becomes a retrievable corpus with filters that work before the vector search, and a freshness audit that cannot silently rot.
|
||
|
||
**Depends on:** Phase 1 (content), Phase 2 (FTS5 index exists).
|
||
|
||
---
|
||
|
||
## Work
|
||
|
||
### 4.1 Pipeline
|
||
|
||
```
|
||
content/ ──► validate ──► chunk ──► enrich ──► embed ──► upsert Qdrant
|
||
│
|
||
SQLite FTS5 (from Phase 2) ┘
|
||
```
|
||
|
||
- **Chunking:** 400–800 tokens, ~12% overlap, **code-aware** — never split inside a fenced block, table or front-matter.
|
||
- **Deterministic IDs:** `sha256(source_path + chunk_index + content_hash)`. Re-running must update, not duplicate.
|
||
- **Incremental:** hash the source set; only changed documents are re-chunked and re-embedded.
|
||
- **One command** rebuilds from scratch. Deleting the Qdrant volume is always recoverable.
|
||
|
||
### 4.2 Payload schema (the retrieval filter)
|
||
|
||
```json
|
||
{
|
||
"area": "ai|platform|edge|products|history|about",
|
||
"level": 1,
|
||
"source_type": "doc|code|adr|metric",
|
||
"confidence": "delivered|running|in-progress|abandoned|historic",
|
||
"period": "2023–",
|
||
"artefacts": ["..."],
|
||
"last_verified": "2026-10-20",
|
||
"text": "..."
|
||
}
|
||
```
|
||
|
||
`area` and `confidence` are the two that matter: they let the query path filter *before* searching, and they let the answer qualify unfinished work.
|
||
|
||
### 4.3 Embeddings
|
||
- Batch, low volume, incremental.
|
||
- **Record the embedding model and dimension per collection.** A provider change is a governed migration, not a silent corruption.
|
||
- Provider decision pending Sam — see `reference/stack.md`. Build provider-agnostic behind one interface.
|
||
|
||
### 4.4 Scheduling
|
||
- Prefect deployment on the existing Prefect server; **worker only**, do not deploy a second server.
|
||
- Trigger: git push to `content/`, plus a nightly audit.
|
||
- Nightly audit reports documents past their `last_verified` horizon and alerts via the existing Apprise service.
|
||
|
||
---
|
||
|
||
## Definition of Done
|
||
|
||
- [ ] One command rebuilds the corpus from an empty volume.
|
||
- [ ] Running it twice produces no duplicate vectors (count is stable).
|
||
- [ ] No chunk boundary lands inside a code fence, table or front-matter block.
|
||
- [ ] Every chunk carries all payload fields; a missing field fails the run rather than defaulting.
|
||
- [ ] Payload filtering demonstrably narrows results — demonstrate the same query with and without an `area` filter.
|
||
- [ ] Incremental mode re-embeds only changed documents; prove it with the provider call count.
|
||
- [ ] Freshness audit runs and reports stale documents.
|
||
- [ ] Qdrant memory measured at 500 / 5,000 / 20,000 chunks and recorded in `reference/stack.md`, within its cap.
|
||
|
||
---
|
||
|
||
## Out of scope
|
||
|
||
- Answering questions. Phase 4 ends at "retrieval returns the right chunks".
|
||
- Query graphs, prompts, guardrails — Phase 5 and 6.
|
||
- Ingesting anything outside `content/`. Four sources maximum, and this phase uses one: the repo.
|
||
|
||
---
|
||
|
||
## Risks
|
||
|
||
| Risk | Mitigation |
|
||
|---|---|
|
||
| Corpus grows beyond RAM | Incremental by default; measure at three corpus sizes; cap Qdrant with `mem_limit` |
|
||
| Duplicate vectors on re-run | Deterministic IDs are mandatory, not optional |
|
||
| Silent staleness | Nightly audit + Apprise alert; a stale index is otherwise an invisible quality failure |
|
||
| Provider rate limits during bulk embed | Batch with retry/backoff; ingest is low-volume and incremental |
|