Phase 4: corpus and retrieval — chunks, Qdrant, FTS5, freshness audit

Satisfies docs/phase-4-corpus.md. content/ stays the single source of truth:
one pass over it writes both indexes, so the static pages and the retrieval
corpus cannot drift.

- corpus/: stdlib-only pipeline. validate -> chunk -> enrich -> embed -> upsert.
  Chunking is block-based, so a boundary cannot fall inside a fenced block, a
  table or front-matter (verified structurally, not by inspection).
- Deterministic ids: sha256(source_path + chunk_index + content_hash), rendered
  as a UUID. Re-running updates points; re-chunking deletes the old ones by
  source filter, so duplicates and orphans are both impossible.
- Incremental by content hash against a SQLite manifest. A provider, dimension
  or chunker change is refused instead of mixed: it needs a rebuild.
- Payloads per section 4.2 plus provenance, validated before any vector is
  written; a missing field fails the run and names the document and chunk.
- SQLite FTS5 over chunks (the keyword half of hybrid retrieval, decision D10)
  with area/confidence filters and a `terms` column so front-matter tech and
  tags are findable.
- Freshness audit: stale documents, due soon, index drift, contract drift;
  optional Apprise alert; exit code for Prefect.
- Embeddings behind one interface: omniroute (OpenAI-compatible, retries, key
  from the environment only) and local-hashing (offline placeholder). The
  provider decision is still Sam's; the pipeline does not depend on it.
- deploy/qdrant.yml: mem_limit = memswap_limit = 400m, loopback only.
- docs/phase-4-notes.md: operator manual, DoD evidence, measured memory.
- 76 unit tests, no container and no network.

Measured, Qdrant 1.20.0 under a 400 m cap at 1024 dimensions: 500 / 5,000 /
20,000 chunks settle at 128 / 138 / 151 MiB, peaking at 149 / 305 / 292 MiB
during ingest. Qdrant's default settings are OOM-killed between 5,000 and
20,000, so the collection is created with HNSW and payloads on disk; recorded in
reference/stack.md with the ceiling and the escalation order.

Note: the `make corpus*` targets were swept into the Phase 3 deploy commit
24590cc by concurrent work in this tree; the Makefile content is correct, and
this commit contains the pipeline they call.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

reference/stack.md

@@ -36,6 +36,25 @@
 | Guardrail (if local) | 300 m |
 | **Total ceiling** | **1.0 GiB / 4 containers** |
 
+### Qdrant memory, measured (2026-10-26, v1.20.0, 400 m cap, 1024 dims)
+
+| Chunks | Settled RSS | Peak during ingest |
+|---|---|---|
+| 500 | 128 MiB | 149 MiB |
+| 5,000 | 138 MiB | 305 MiB |
+| 20,000 | 151 MiB | 292 MiB |
+
+Within the cap at all three sizes, but only with the memory-frugal collection
+settings in `corpus/qdrant.py`: HNSW graph and payloads on disk
+(`hnsw_config.on_disk`, `on_disk_payload`), `m=8`, `ef_construct=64`,
+`memmap_threshold=5000`, `indexing_threshold=20000`. With Qdrant's defaults the
+container is OOM-killed between 5,000 and 20,000 chunks. The pipeline is the only
+code that creates the collection, so those settings cannot drift.
+
+Ceiling: 20,000 chunks at 1024 dimensions. Beyond that, set
+`RESUME_QDRANT_VECTORS_ON_DISK=1`, reduce the dimension, or raise the cap — in
+that order. Detail: `docs/phase-4-notes.md` §7.
+
 Every service sets `mem_limit` **and** `memswap_limit` to the same value so it cannot consume host swap. Bind to `127.0.0.1`; Caddy is the only public listener.
 
 ## Paths
@@ -56,6 +75,6 @@ Every service sets `mem_limit` **and** `memswap_limit` to the same value so it c
 
 | Item | Owner |
 |---|---|
-| Embedding provider (Gemini free tier vs OpenRouter vs local) | Sam — decide before Phase 4 |
+| Embedding provider (Gemini free tier vs OpenRouter vs local) | Sam — Phase 4 shipped provider-agnostic (`corpus/embeddings.py`); the `omniroute` adapter is ready, `local-hashing` is a placeholder, so the decision is still Sam's before Phase 5 answers questions |
 | Repo public/private after launch | Sam — decide after Phase 3 |
 | Phone number on the contact page | Sam |