sam/ai-resume
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 4: corpus and retrieval — chunks, Qdrant, FTS5, freshness audit

Browse Source 
Satisfies docs/phase-4-corpus.md. content/ stays the single source of truth:
one pass over it writes both indexes, so the static pages and the retrieval
corpus cannot drift.

- corpus/: stdlib-only pipeline. validate -> chunk -> enrich -> embed -> upsert.
  Chunking is block-based, so a boundary cannot fall inside a fenced block, a
  table or front-matter (verified structurally, not by inspection).
- Deterministic ids: sha256(source_path + chunk_index + content_hash), rendered
  as a UUID. Re-running updates points; re-chunking deletes the old ones by
  source filter, so duplicates and orphans are both impossible.
- Incremental by content hash against a SQLite manifest. A provider, dimension
  or chunker change is refused instead of mixed: it needs a rebuild.
- Payloads per section 4.2 plus provenance, validated before any vector is
  written; a missing field fails the run and names the document and chunk.
- SQLite FTS5 over chunks (the keyword half of hybrid retrieval, decision D10)
  with area/confidence filters and a `terms` column so front-matter tech and
  tags are findable.
- Freshness audit: stale documents, due soon, index drift, contract drift;
  optional Apprise alert; exit code for Prefect.
- Embeddings behind one interface: omniroute (OpenAI-compatible, retries, key
  from the environment only) and local-hashing (offline placeholder). The
  provider decision is still Sam's; the pipeline does not depend on it.
- deploy/qdrant.yml: mem_limit = memswap_limit = 400m, loopback only.
- docs/phase-4-notes.md: operator manual, DoD evidence, measured memory.
- 76 unit tests, no container and no network.

Measured, Qdrant 1.20.0 under a 400 m cap at 1024 dimensions: 500 / 5,000 /
20,000 chunks settle at 128 / 138 / 151 MiB, peaking at 149 / 305 / 292 MiB
during ingest. Qdrant's default settings are OOM-killed between 5,000 and
20,000, so the collection is created with HNSW and payloads on disk; recorded in
reference/stack.md with the ceiling and the escalation order.

Note: the `make corpus*` targets were swept into the Phase 3 deploy commit
24590cc by concurrent work in this tree; the Makefile content is correct, and
this commit contains the pipeline they call.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Sam Rolfe
2026-10-26 21:04:52 +11:00
parent 24590cc351
commit f9dc574172
28 changed files with 4372 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

304
docs/phase-4-notes.md Normal file
View File

@@ -0,0 +1,304 @@
# Phase 4 notes — corpus and retrieval
Operator manual for the corpus pipeline, and the evidence for the Phase 4
Definition of Done in `docs/phase-4-corpus.md`. Every number below was measured
on 2026-10-26 against a live Qdrant 1.20.0 container with a 400 MiB cap.
---
## 1. What exists now
| Path | What it is |
|---|---|
| `corpus/` | The pipeline: one Python package, standard library only |
| `corpus/cli.py` | `python3 -m corpus.cli {rebuild,sync,audit,verify,demo-filter,hybrid,measure,info}` |
| `corpus/chunker.py` | Code-aware chunking (fenced block, table and front-matter are atomic) |
| `corpus/embeddings.py` | The only place a vector is made; two providers behind one interface |
| `corpus/qdrant.py` | Qdrant REST client (no `qdrant-client` dependency) |
| `corpus/store.py` | SQLite: FTS5 over chunks, the incremental manifest, the collection contract |
| `corpus/pipeline.py` | validate → chunk → enrich → embed → upsert, in that order |
| `corpus/freshness.py` | Nightly audit: stale documents, index drift, contract drift |
| `corpus/measure.py` | Memory measurement at 500 / 5,000 / 20,000 chunks |
| `corpus/tests/` | 76 unit tests, no container, no network |
| `flows/corpus.py` | Prefect flows: `corpus-sync`, `corpus-rebuild`, `corpus-audit` |
| `deploy/qdrant.yml` | Qdrant container: `mem_limit` = `memswap_limit` = 400 m, loopback only |
Two indexes are written from one pass over `content/`, so they cannot drift:
* **vectors** in Qdrant (`resume_corpus`), one point per chunk;
* **keywords** in SQLite (`data/corpus/corpus.db`), one FTS5 row per chunk.
Retrieval uses both (`corpus.cli hybrid`, reciprocal rank fusion). Vectors miss
exact tokens such as repo names, `LoRaWAN`, `MQTT` and dates; FTS5 alone misses
paraphrase. Decision D10 requires the pair.
---
## 2. Commands
```sh
make corpus # rebuild from content/ (drops the collection first)
make corpus-sync # incremental: only changed documents are re-embedded
make corpus-audit # freshness + drift audit; exit 1 on findings
make corpus-verify # index vs content/ vs Qdrant, consistency report
make corpus-demo # the same query with and without an area filter
make corpus-test # 76 unit tests
```
Direct form, when you need flags:
```sh
python3 -m corpus.cli rebuild --qdrant-url http://127.0.0.1:6333 --dim 1024
python3 -m corpus.cli audit --today 2027-01-20 --json --fail-on-findings
python3 -m corpus.cli measure --sizes 500,5000,20000 --container ai-resume-qdrant
```
Configuration is environment-first. Nothing is read from a file in this repo.
| Variable | Default | Meaning |
|---|---|---|
| `RESUME_QDRANT_URL` | `http://127.0.0.1:6333` | Qdrant REST endpoint |
| `RESUME_COLLECTION` | `resume_corpus` | collection name |
| `RESUME_EMBED_PROVIDER` | `local-hashing` | `local-hashing` or `omniroute` |
| `RESUME_EMBED_MODEL` | provider default | recorded per collection |
| `RESUME_EMBED_DIM` | provider default | recorded per collection |
| `RESUME_CORPUS_DB` | `data/corpus/corpus.db` | FTS5 + manifest |
| `RESUME_MAX_AGE_DAYS` | `180` | freshness horizon |
| `RESUME_WARN_DAYS` | `30` | how early "due soon" starts |
| `RESUME_APPRISE_URL` | unset | existing Apprise endpoint for audit alerts |
| `OMNIROUTE_API_KEY` | unset | only for the `omniroute` provider; never in the repo |
---
## 3. Pipeline
```
content/ ─► make validate ─► chunk ─► enrich ─► embed ─► upsert Qdrant
│
└─► SQLite FTS5 + manifest
```
* **Validate first.** The pipeline runs the Phase 1 validator
(`tools/validate_content.py`). A broken content tree stops the run before a
single vector is written.
* **Chunking.** Target 600 tokens, hard ceiling 800, ~12 % overlap. The body is
parsed into atomic blocks (fenced code, table, heading, list item, quote,
paragraph, rule) and chunks are packed from whole blocks, so a boundary
cannot fall inside a fence, a table or front-matter. A list item or a table
that exceeds 800 tokens becomes one oversize chunk and is flagged, never cut.
Tokens are counted as words plus punctuation, which is a conservative
under-estimate of BPE tokens.
* **Payload before vectors.** Every chunk payload is built and validated for
every changed document *before* one point is written. A missing or invalid
field aborts the run with the document and chunk named.
* **Deterministic ids.** `sha256(source_path + chunk_index + content_hash)`,
rendered as a UUID because Qdrant accepts only integers or UUIDs. Re-running
updates points; a document whose chunk count shrinks has its old points
deleted by `source_path` filter, so orphans cannot accumulate.
* **Incremental.** One hash comparison against the manifest per document. The
document is re-embedded when its bytes changed, or when the embedding model,
dimension or chunker version changed.
* **Consistency check.** At the end of every run the point count in Qdrant must
equal the chunk count in SQLite, or the run fails.
Corpus today: **40 documents → 51 chunks**, token range 100–707 (median 225),
8 protected regions (fenced blocks and tables), 0 boundary violations.
---
## 4. Payload
`docs/phase-4-corpus.md` §4.2 fields, plus provenance so an answer can cite:
```json
{
"area": "edge", "level": 3, "source_type": "doc", "confidence": "running",
"period": "2020–", "artefacts": [], "last_verified": "2026-10-26",
"text": "# Artefact — Distributed audio …",
"id": "014d792f-28d9-e0a0-610e-7ffc04eea3ad",
"source_path": "content/edge/l3/edge-snapcast.md", "source_id": "edge-snapcast",
"chunk_index": 0, "title": "…", "claim": "…", "url": "/edge/a/edge-snapcast/",
"content_hash": "12fc4fa9…", "embedding_model": "local-hashing-v1",
"embedding_dim": 1024, "chunker_version": "blocks-v1", "chunk_tokens": 191,
"public": true, "tech": ["Snapcast", "Mopidy", "…"], "tags": ["edge", "audio"]
}
```
`area` and `confidence` are the two that matter. The query path filters on them
*before* it searches, and an answer can qualify unfinished work instead of
presenting it as delivered.
---
## 5. Freshness audit
```sh
make corpus-audit # human table, exit 1 on findings
python3 -m corpus.cli audit --json # machine-readable, for Prefect
```
Findings, in order of seriousness:
| Finding | Meaning |
|---|---|
| `stale` | `last_verified` older than `RESUME_MAX_AGE_DAYS` (180). The claim may no longer be true |
| `due_soon` | inside the warning window (30 days by default). A warning, not a failure |
| `index_drift` | the source changed, disappeared or was never indexed since it was embedded |
| `contract_drift` | the collection was built with a different model, dimension or chunker |
`ok` is false when there is a `stale`, `index_drift` or `contract_drift`
finding. `due_soon` alone keeps the audit green.
`--alert` posts the report to `RESUME_APPRISE_URL` when the audit is not ok.
The Prefect flow does this by default.
---
## 6. Scheduling (Prefect, existing server, worker only)
`flows/corpus.py` defines three flows. It imports Prefect when present and runs
as plain Python when it is not, so the CLI never depends on the orchestrator.
On `.13`, which already runs the Prefect server — **do not start a second one**:
```sh
# once, per deployment
export RESUME_QDRANT_URL=http://127.0.0.1:6333
export RESUME_EMBED_PROVIDER=omniroute RESUME_EMBED_MODEL=<approved model> RESUME_EMBED_DIM=<dim>
prefect deploy flows/corpus.py:sync_corpus --name corpus-sync
prefect deploy flows/corpus.py:freshness_audit --name corpus-audit --cron "0 4 * * *"
```
Triggers required by `docs/phase-4-corpus.md` §4.4:
* **on push touching `content/`** — a deployment trigger on the repository, or
`make corpus-sync` from the same post-receive hook that runs `deploy.sh`;
* **nightly audit** — cron `0 4 * * *`, which alerts through Apprise.
This deployment is **not applied yet**: it needs the runtime host and the
provider decision in §8. Both are Sam's. The flows themselves are ready.
---
## 7. Qdrant memory at three corpus sizes
Measured with `python3 -m corpus.cli measure`, Qdrant 1.20.0 in a container
capped at 400 MiB (`mem_limit` = `memswap_limit`), 1024-dimensional vectors,
real payloads from `content/`, throttled ingest in batches of 250.
| Chunks | Points | Settled RSS | Peak during ingest | Growth | Bytes/chunk | Ingest time |
|---|---|---|---|---|---|---|
| 0 | 0 | 88.6 MiB | — | — | — | — |
| 500 | 500 | 128.3 MiB | 148.5 MiB | +39.7 MiB | 83 KiB | 1.1 s |
| 5,000 | 5,000 | 137.7 MiB | 305.0 MiB | +49.1 MiB | 10 KiB | 10.9 s |
| 20,000 | 20,000 | 150.9 MiB | 292.3 MiB | +62.3 MiB | 3 KiB | 43.7 s |
All three sizes are **within the 400 MiB cap**. The peak is the number that
matters: the cgroup limit is tripped during a bulk write, not after it.
### The finding that changed the design
With Qdrant's **default** settings the container was OOM-killed at exactly the
cap between 5,000 and 20,000 chunks (`anon-rss: 404204kB`, `exitCode 137`,
`constraint=CONSTRAINT_MEMCG`). The default keeps the HNSW graph and the payload
in RAM and holds several segments live during ingest.
The collection is now created memory-frugal, and the pipeline is the only code
that creates it (`corpus/qdrant.py`):
```json
{
"vectors": {"size": 1024, "distance": "Cosine", "on_disk": false},
"hnsw_config": {"m": 8, "ef_construct": 64, "on_disk": true},
"on_disk_payload": true,
"optimizers_config": {"default_segment_number": 2, "memmap_threshold": 5000, "indexing_threshold": 20000}
}
```
Vectors stay in RAM because search reads them on every query. The HNSW graph and
the payloads go to disk, where the kernel can reclaim those pages instead of
killing the process. `memmap_threshold` 5,000 makes segments that size mmap;
`indexing_threshold` 20,000 stops a bulk ingest from building the graph while it
is still writing. The cost is build time and a little search latency, both
irrelevant for a corpus of this size and a public site with no query burst.
Practical ceiling: at 1024 dimensions the cap holds 20,000 chunks. If the corpus
grows past that, set `RESUME_QDRANT_VECTORS_ON_DISK=1`, or reduce the dimension,
or raise the cap — in that order of preference. Vector memory scales linearly
with dimension: 1,536-dimension vectors need 50 % more than recorded here.
---
## 8. Embedding provider — still Sam's decision
`reference/stack.md` lists the open item: Gemini free tier, OpenRouter, or
local. Phase 4 does not wait for it:
* **`omniroute`** is implemented and tested. OpenAI-compatible
`/v1/embeddings`, batched, retry with backoff on 429 and 5xx, key from the
environment only, no provider SDK, no direct vendor call (AGENTS.md §4.5).
* **`local-hashing`** is the default for tests, for offline rebuilds and for the
memory measurement. It is deterministic feature hashing, **not a semantic
model**: it is a placeholder that lets the whole pipeline run without a
provider decision, and it must not be used for answers.
The provider, model, dimension and chunker version are stored in the collection
contract (`data/corpus/collection.json` and the `meta` table). A run whose
configuration differs is refused:
```
the collection contract does not match this configuration — refusing to mix
vectors in one collection (docs/phase-4-corpus.md §4.3).
```
### Changing the provider (governed migration)
1. Get Sam's decision and record it in `reference/decisions.md`.
2. Set `RESUME_EMBED_PROVIDER`, `RESUME_EMBED_MODEL`, `RESUME_EMBED_DIM`, and
`OMNIROUTE_API_KEY` in `~/.config/environment.d/`.
3. `make corpus` — a rebuild, because a provider change invalidates every
vector. It is one command and takes seconds at this corpus size.
4. `make corpus-verify` and `make corpus-audit`.
---
## 9. Recovery
Deleting the Qdrant volume or `data/corpus/` is **always recoverable**:
```sh
docker compose -f deploy/qdrant.yml down -v
docker compose -f deploy/qdrant.yml up -d
make corpus # re-derives every vector and FTS row from content/
make corpus-verify
```
There is no state that exists only in the index. `content/` is the source of
truth; the corpus is a derived artefact and lives in `data/`, which is
gitignored.
---
## 10. Definition of Done — evidence
| DoD item | Evidence |
|---|---|
| One command rebuilds from an empty volume | `make corpus` on an empty Qdrant volume: 40 documents → 51 chunks, 51 points, 1.0 s |
| Running it twice produces no duplicate vectors | rebuild ×2 → 51 points both times; `sync` with no changes → 0 embedding calls, 0 points sent |
| No chunk boundary inside a fence, table or front-matter | `corpus/tests/test_chunker.py`; 51/51 stored chunks reproduce from `content/`, 8 protected regions, 0 violations |
| Every chunk carries all payload fields; missing field fails the run | `corpus/tests/test_payloads.py` (each field removed in turn); 51/51 stored payloads validate; a broken payload aborts before any write (`test_invalid_payload_aborts_before_any_write`) |
| Payload filtering demonstrably narrows results | `make corpus-demo` — query "distributed audio endpoints": unfiltered returns edge *and* ai documents; `--area edge` returns edge only, `--area ai` returns ai only |
| Incremental re-embeds only changed documents | edit one of 40 documents → `updated=1 unchanged=39 embedding calls=1 texts=1`; re-run → `calls=0` |
| Freshness audit runs and reports stale documents | `python3 -m corpus.cli audit` today: 0 findings. `--today 2027-10-05`: 40 stale documents, one row per document with age in days. `--today 2026-11-01`: 0 findings |
| Qdrant memory at 500 / 5,000 / 20,000 chunks, within cap | §7 table; 128 / 138 / 151 MiB settled, 149 / 305 / 292 MiB peak, cap 400 MiB |
---
## 11. Not done here
* Qdrant is **not deployed to `.13`** and the Prefect deployment is not applied.
`deploy/qdrant.yml` is ready; the runtime host is Phase 3's (`AUD-143`) and
Sam's to apply.
* No provider is chosen (§8).
* Answering questions, the query graph, the streaming console and guardrails are
Phase 5 and Phase 6. Phase 4 ends at "retrieval returns the right chunks".