Files

Sam Rolfe b267e55f8c Replace Vite/React scaffold with Go build spec

- AGENTS.md: build spec for Paperclip (constraints, conventions, 7 phases)
- docs/phase-1..7: per-phase work, DoD, out-of-scope, risks
- reference/: frozen decisions, stack+ports+budgets, content map, acceptance tests
- agents/: persona and security policy drafts
- Retires the React scaffold; history retained as record of the false start

2026-10-26 16:02:35 +11:00

3.4 KiB

Raw Permalink Blame History

Phase 4 — Corpus and retrieval

Goal: content/ becomes a retrievable corpus with filters that work before the vector search, and a freshness audit that cannot silently rot.

Depends on: Phase 1 (content), Phase 2 (FTS5 index exists).

Work

4.1 Pipeline

content/  ──► validate ──► chunk ──► enrich ──► embed ──► upsert Qdrant
                                                              │
                                    SQLite FTS5 (from Phase 2) ┘

Chunking: 400–800 tokens, ~12% overlap, code-aware — never split inside a fenced block, table or front-matter.
Deterministic IDs: sha256(source_path + chunk_index + content_hash). Re-running must update, not duplicate.
Incremental: hash the source set; only changed documents are re-chunked and re-embedded.
One command rebuilds from scratch. Deleting the Qdrant volume is always recoverable.

4.2 Payload schema (the retrieval filter)

{
  "area": "ai|platform|edge|products|history|about",
  "level": 1,
  "source_type": "doc|code|adr|metric",
  "confidence": "delivered|running|in-progress|abandoned|historic",
  "period": "2023–",
  "artefacts": ["..."],
  "last_verified": "2026-10-20",
  "text": "..."
}

area and confidence are the two that matter: they let the query path filter before searching, and they let the answer qualify unfinished work.

4.3 Embeddings

Batch, low volume, incremental.
Record the embedding model and dimension per collection. A provider change is a governed migration, not a silent corruption.
Provider decision pending Sam — see reference/stack.md. Build provider-agnostic behind one interface.

4.4 Scheduling

Prefect deployment on the existing Prefect server; worker only, do not deploy a second server.
Trigger: git push to content/, plus a nightly audit.
Nightly audit reports documents past their last_verified horizon and alerts via the existing Apprise service.

Definition of Done

One command rebuilds the corpus from an empty volume.
Running it twice produces no duplicate vectors (count is stable).
No chunk boundary lands inside a code fence, table or front-matter block.
Every chunk carries all payload fields; a missing field fails the run rather than defaulting.
Payload filtering demonstrably narrows results — demonstrate the same query with and without an area filter.
Incremental mode re-embeds only changed documents; prove it with the provider call count.
Freshness audit runs and reports stale documents.
Qdrant memory measured at 500 / 5,000 / 20,000 chunks and recorded in reference/stack.md, within its cap.

Out of scope

Answering questions. Phase 4 ends at "retrieval returns the right chunks".
Query graphs, prompts, guardrails — Phase 5 and 6.
Ingesting anything outside content/. Four sources maximum, and this phase uses one: the repo.

Risks

Risk	Mitigation
Corpus grows beyond RAM	Incremental by default; measure at three corpus sizes; cap Qdrant with `mem_limit`
Duplicate vectors on re-run	Deterministic IDs are mandatory, not optional
Silent staleness	Nightly audit + Apprise alert; a stale index is otherwise an invisible quality failure
Provider rate limits during bulk embed	Batch with retry/backoff; ingest is low-volume and incremental

3.4 KiB Raw Permalink Blame History Unescape Escape