ai-resume/docs/phase-4-notes.md

Phase 4 notes — corpus and retrieval

Operator manual for the corpus pipeline, and the evidence for the Phase 4 Definition of Done in docs/phase-4-corpus.md. Every number below was measured on 2026-10-26 against a live Qdrant 1.20.0 container with a 400 MiB cap.

---

1. What exists now

Path	What it is
corpus/	The pipeline: one Python package, standard library only
corpus/cli.py	python3 -m corpus.cli {rebuild,sync,audit,verify,demo-filter,hybrid,measure,info}
corpus/chunker.py	Code-aware chunking (fenced block, table and front-matter are atomic)
corpus/embeddings.py	The only place a vector is made; two providers behind one interface
corpus/qdrant.py	Qdrant REST client (no qdrant-client dependency)
corpus/store.py	SQLite: FTS5 over chunks, the incremental manifest, the collection contract
corpus/pipeline.py	validate → chunk → enrich → embed → upsert, in that order
corpus/freshness.py	Nightly audit: stale documents, index drift, contract drift
corpus/measure.py	Memory measurement at 500 / 5,000 / 20,000 chunks
corpus/tests/	76 unit tests, no container, no network
flows/corpus.py	Prefect flows: corpus-sync, corpus-rebuild, corpus-audit
deploy/qdrant.yml	Qdrant container: mem_limit = memswap_limit = 400 m, loopback only

Two indexes are written from one pass over content/, so they cannot drift:

• vectors in Qdrant (resume_corpus), one point per chunk;
• keywords in SQLite (data/corpus/corpus.db), one FTS5 row per chunk.

Retrieval uses both (corpus.cli hybrid, reciprocal rank fusion). Vectors miss exact tokens such as repo names, LoRaWAN, MQTT and dates; FTS5 alone misses paraphrase. Decision D10 requires the pair.

---

2. Commands

make corpus          # rebuild from content/ (drops the collection first)
make corpus-sync     # incremental: only changed documents are re-embedded
make corpus-audit    # freshness + drift audit; exit 1 on findings
make corpus-verify   # index vs content/ vs Qdrant, consistency report
make corpus-demo     # the same query with and without an area filter
make corpus-test     # 76 unit tests

Direct form, when you need flags:

python3 -m corpus.cli rebuild --qdrant-url http://127.0.0.1:6333 --dim 1024
python3 -m corpus.cli audit --today 2027-01-20 --json --fail-on-findings
python3 -m corpus.cli measure --sizes 500,5000,20000 --container ai-resume-qdrant

Configuration is environment-first. Nothing is read from a file in this repo.

Variable	Default	Meaning
RESUME_QDRANT_URL	http://127.0.0.1:6333	Qdrant REST endpoint
RESUME_COLLECTION	resume_corpus	collection name
RESUME_EMBED_PROVIDER	local-hashing	local-hashing or omniroute
RESUME_EMBED_MODEL	provider default	recorded per collection
RESUME_EMBED_DIM	provider default	recorded per collection
RESUME_CORPUS_DB	data/corpus/corpus.db	FTS5 + manifest
RESUME_MAX_AGE_DAYS	180	freshness horizon
RESUME_WARN_DAYS	30	how early "due soon" starts
RESUME_APPRISE_URL	unset	existing Apprise endpoint for audit alerts
OMNIROUTE_API_KEY	unset	only for the omniroute provider; never in the repo

---

3. Pipeline

content/ ─► make validate ─► chunk ─► enrich ─► embed ─► upsert Qdrant
                                        │
                                        └─► SQLite FTS5 + manifest

• Validate first. The pipeline runs the Phase 1 validator (tools/validate_content.py). A broken content tree stops the run before a single vector is written.
• Chunking. Target 600 tokens, hard ceiling 800, ~12 % overlap. The body is parsed into atomic blocks (fenced code, table, heading, list item, quote, paragraph, rule) and chunks are packed from whole blocks, so a boundary cannot fall inside a fence, a table or front-matter. A list item or a table that exceeds 800 tokens becomes one oversize chunk and is flagged, never cut. Tokens are counted as words plus punctuation, which is a conservative under-estimate of BPE tokens.
• Payload before vectors. Every chunk payload is built and validated for every changed document before one point is written. A missing or invalid field aborts the run with the document and chunk named.
• Deterministic ids. sha256(source_path + chunk_index + content_hash), rendered as a UUID because Qdrant accepts only integers or UUIDs. Re-running updates points; a document whose chunk count shrinks has its old points deleted by source_path filter, so orphans cannot accumulate.
• Incremental. One hash comparison against the manifest per document. The document is re-embedded when its bytes changed, or when the embedding model, dimension or chunker version changed.
• Consistency check. At the end of every run the point count in Qdrant must equal the chunk count in SQLite, or the run fails.

Corpus today: 40 documents → 51 chunks, token range 100–707 (median 225), 8 protected regions (fenced blocks and tables), 0 boundary violations.

---

4. Payload

docs/phase-4-corpus.md §4.2 fields, plus provenance so an answer can cite:

{
  "area": "edge", "level": 3, "source_type": "doc", "confidence": "running",
  "period": "2020–", "artefacts": [], "last_verified": "2026-10-26",
  "text": "# Artefact — Distributed audio …",
  "id": "014d792f-28d9-e0a0-610e-7ffc04eea3ad",
  "source_path": "content/edge/l3/edge-snapcast.md", "source_id": "edge-snapcast",
  "chunk_index": 0, "title": "…", "claim": "…", "url": "/edge/a/edge-snapcast/",
  "content_hash": "12fc4fa9…", "embedding_model": "local-hashing-v1",
  "embedding_dim": 1024, "chunker_version": "blocks-v1", "chunk_tokens": 191,
  "public": true, "tech": ["Snapcast", "Mopidy", "…"], "tags": ["edge", "audio"]
}

area and confidence are the two that matter. The query path filters on them before it searches, and an answer can qualify unfinished work instead of presenting it as delivered.

---

5. Freshness audit

make corpus-audit                  # human table, exit 1 on findings
python3 -m corpus.cli audit --json # machine-readable, for Prefect

Findings, in order of seriousness:

Finding	Meaning
stale	last_verified older than RESUME_MAX_AGE_DAYS (180). The claim may no longer be true
due_soon	inside the warning window (30 days by default). A warning, not a failure
index_drift	the source changed, disappeared or was never indexed since it was embedded
contract_drift	the collection was built with a different model, dimension or chunker

ok is false when there is a stale, index_drift or contract_drift finding. due_soon alone keeps the audit green.

--alert posts the report to RESUME_APPRISE_URL when the audit is not ok. The Prefect flow does this by default.

---

6. Scheduling (Prefect, existing server, worker only)

flows/corpus.py defines three flows. It imports Prefect when present and runs as plain Python when it is not, so the CLI never depends on the orchestrator.

On .13, which already runs the Prefect server — do not start a second one:

# once, per deployment
export RESUME_QDRANT_URL=http://127.0.0.1:6333
export RESUME_EMBED_PROVIDER=omniroute RESUME_EMBED_MODEL=<approved model> RESUME_EMBED_DIM=<dim>
prefect deploy flows/corpus.py:sync_corpus    --name corpus-sync
prefect deploy flows/corpus.py:freshness_audit --name corpus-audit --cron "0 4 * * *"

Triggers required by docs/phase-4-corpus.md §4.4:

• on push touching content/ — a deployment trigger on the repository, or make corpus-sync from the same post-receive hook that runs deploy.sh;
• nightly audit — cron 0 4 * * *, which alerts through Apprise.

This deployment is not applied yet: it needs the runtime host and the provider decision in §8. Both are Sam's. The flows themselves are ready.

---

7. Qdrant memory at three corpus sizes

Measured with python3 -m corpus.cli measure, Qdrant 1.20.0 in a container capped at 400 MiB (mem_limit = memswap_limit), 1024-dimensional vectors, real payloads from content/, throttled ingest in batches of 250.

Chunks	Points	Settled RSS	Peak during ingest	Growth	Bytes/chunk	Ingest time
0	0	88.6 MiB	—	—	—	—
500	500	128.3 MiB	148.5 MiB	+39.7 MiB	83 KiB	1.1 s
5,000	5,000	137.7 MiB	305.0 MiB	+49.1 MiB	10 KiB	10.9 s
20,000	20,000	150.9 MiB	292.3 MiB	+62.3 MiB	3 KiB	43.7 s

All three sizes are within the 400 MiB cap. The peak is the number that matters: the cgroup limit is tripped during a bulk write, not after it.

The finding that changed the design

With Qdrant's default settings the container was OOM-killed at exactly the cap between 5,000 and 20,000 chunks (anon-rss: 404204kB, exitCode 137, constraint=CONSTRAINT_MEMCG). The default keeps the HNSW graph and the payload in RAM and holds several segments live during ingest.

The collection is now created memory-frugal, and the pipeline is the only code that creates it (corpus/qdrant.py):

{
  "vectors": {"size": 1024, "distance": "Cosine", "on_disk": false},
  "hnsw_config": {"m": 8, "ef_construct": 64, "on_disk": true},
  "on_disk_payload": true,
  "optimizers_config": {"default_segment_number": 2, "memmap_threshold": 5000, "indexing_threshold": 20000}
}

Vectors stay in RAM because search reads them on every query. The HNSW graph and the payloads go to disk, where the kernel can reclaim those pages instead of killing the process. memmap_threshold 5,000 makes segments that size mmap; indexing_threshold 20,000 stops a bulk ingest from building the graph while it is still writing. The cost is build time and a little search latency, both irrelevant for a corpus of this size and a public site with no query burst.

Practical ceiling: at 1024 dimensions the cap holds 20,000 chunks. If the corpus grows past that, set RESUME_QDRANT_VECTORS_ON_DISK=1, or reduce the dimension, or raise the cap — in that order of preference. Vector memory scales linearly with dimension: 1,536-dimension vectors need 50 % more than recorded here.

---

8. Embedding provider — still Sam's decision

reference/stack.md lists the open item: Gemini free tier, OpenRouter, or local. Phase 4 does not wait for it:

• omniroute is implemented and tested. OpenAI-compatible /v1/embeddings, batched, retry with backoff on 429 and 5xx, key from the environment only, no provider SDK, no direct vendor call (AGENTS.md §4.5).
• local-hashing is the default for tests, for offline rebuilds and for the memory measurement. It is deterministic feature hashing, not a semantic model: it is a placeholder that lets the whole pipeline run without a provider decision, and it must not be used for answers.

The provider, model, dimension and chunker version are stored in the collection contract (data/corpus/collection.json and the meta table). A run whose configuration differs is refused:

the collection contract does not match this configuration — refusing to mix
vectors in one collection (docs/phase-4-corpus.md §4.3).

Changing the provider (governed migration)

1. Get Sam's decision and record it in reference/decisions.md.
2. Set RESUME_EMBED_PROVIDER, RESUME_EMBED_MODEL, RESUME_EMBED_DIM, and OMNIROUTE_API_KEY in ~/.config/environment.d/.
3. make corpus — a rebuild, because a provider change invalidates every vector. It is one command and takes seconds at this corpus size.
4. make corpus-verify and make corpus-audit.

---

9. Recovery

Deleting the Qdrant volume or data/corpus/ is always recoverable:

docker compose -f deploy/qdrant.yml down -v
docker compose -f deploy/qdrant.yml up -d
make corpus          # re-derives every vector and FTS row from content/
make corpus-verify

There is no state that exists only in the index. content/ is the source of truth; the corpus is a derived artefact and lives in data/, which is gitignored.

---

10. Definition of Done — evidence

DoD item	Evidence
One command rebuilds from an empty volume	make corpus on an empty Qdrant volume: 40 documents → 51 chunks, 51 points, 1.0 s
Running it twice produces no duplicate vectors	rebuild ×2 → 51 points both times; sync with no changes → 0 embedding calls, 0 points sent
No chunk boundary inside a fence, table or front-matter	corpus/tests/test_chunker.py; 51/51 stored chunks reproduce from content/, 8 protected regions, 0 violations
Every chunk carries all payload fields; missing field fails the run	corpus/tests/test_payloads.py (each field removed in turn); 51/51 stored payloads validate; a broken payload aborts before any write (test_invalid_payload_aborts_before_any_write)
Payload filtering demonstrably narrows results	make corpus-demo — query "distributed audio endpoints": unfiltered returns edge and ai documents; --area edge returns edge only, --area ai returns ai only
Incremental re-embeds only changed documents	edit one of 40 documents → updated=1 unchanged=39 embedding calls=1 texts=1; re-run → calls=0
Freshness audit runs and reports stale documents	python3 -m corpus.cli audit today: 0 findings. --today 2027-10-05: 40 stale documents, one row per document with age in days. --today 2026-11-01: 0 findings
Qdrant memory at 500 / 5,000 / 20,000 chunks, within cap	§7 table; 128 / 138 / 151 MiB settled, 149 / 305 / 292 MiB peak, cap 400 MiB

---

11. Not done here

• Qdrant is not deployed to .13 and the Prefect deployment is not applied. deploy/qdrant.yml is ready; the runtime host is Phase 3's (AUD-143) and Sam's to apply.
• No provider is chosen (§8).
• Answering questions, the query graph, the streaming console and guardrails are Phase 5 and Phase 6. Phase 4 ends at "retrieval returns the right chunks".