Sam Rolfe
f9dc574172
Satisfies docs/phase-4-corpus.md. content/ stays the single source of truth:
one pass over it writes both indexes, so the static pages and the retrieval
corpus cannot drift.
- corpus/: stdlib-only pipeline. validate -> chunk -> enrich -> embed -> upsert.
Chunking is block-based, so a boundary cannot fall inside a fenced block, a
table or front-matter (verified structurally, not by inspection).
- Deterministic ids: sha256(source_path + chunk_index + content_hash), rendered
as a UUID. Re-running updates points; re-chunking deletes the old ones by
source filter, so duplicates and orphans are both impossible.
- Incremental by content hash against a SQLite manifest. A provider, dimension
or chunker change is refused instead of mixed: it needs a rebuild.
- Payloads per section 4.2 plus provenance, validated before any vector is
written; a missing field fails the run and names the document and chunk.
- SQLite FTS5 over chunks (the keyword half of hybrid retrieval, decision D10)
with area/confidence filters and a `terms` column so front-matter tech and
tags are findable.
- Freshness audit: stale documents, due soon, index drift, contract drift;
optional Apprise alert; exit code for Prefect.
- Embeddings behind one interface: omniroute (OpenAI-compatible, retries, key
from the environment only) and local-hashing (offline placeholder). The
provider decision is still Sam's; the pipeline does not depend on it.
- deploy/qdrant.yml: mem_limit = memswap_limit = 400m, loopback only.
- docs/phase-4-notes.md: operator manual, DoD evidence, measured memory.
- 76 unit tests, no container and no network.
Measured, Qdrant 1.20.0 under a 400 m cap at 1024 dimensions: 500 / 5,000 /
20,000 chunks settle at 128 / 138 / 151 MiB, peaking at 149 / 305 / 292 MiB
during ingest. Qdrant's default settings are OOM-killed between 5,000 and
20,000, so the collection is created with HNSW and payloads on disk; recorded in
reference/stack.md with the ceiling and the escalation order.
Note: the `make corpus*` targets were swept into the Phase 3 deploy commit
24590cc by concurrent work in this tree; the Makefile content is correct, and
this commit contains the pipeline they call.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
56 lines
1.9 KiB
YAML
56 lines
1.9 KiB
YAML
# Qdrant for the corpus (docs/phase-4-corpus.md, reference/stack.md).
|
|
#
|
|
# mem_limit == memswap_limit == 400m the cap in the memory budget, and the
|
|
# container cannot reach host swap
|
|
# ports on 127.0.0.1 only Caddy on .35 is the only public listener
|
|
# its own compose project stopping or recreating it cannot disturb
|
|
# the Mode A site (test A1 still passes:
|
|
# the site never calls Qdrant)
|
|
#
|
|
# Run on the runtime host (.13):
|
|
#
|
|
# docker compose -f deploy/qdrant.yml up -d
|
|
# curl -s http://127.0.0.1:6333/ # must answer with a version
|
|
#
|
|
# Then, in the corpus environment (RESUME_QDRANT_URL is already the default):
|
|
#
|
|
# python3 -m corpus.cli rebuild
|
|
#
|
|
# Deleting the volume is always recoverable: `rebuild` re-derives every vector
|
|
# from content/.
|
|
|
|
name: ai-resume-index
|
|
|
|
services:
|
|
qdrant:
|
|
image: qdrant/qdrant:v1.20.0
|
|
container_name: ai-resume-qdrant
|
|
restart: unless-stopped
|
|
mem_limit: 400m
|
|
memswap_limit: 400m
|
|
ports:
|
|
- "127.0.0.1:6333:6333" # REST — the pipeline and the query service use this
|
|
- "127.0.0.1:6334:6334" # gRPC — reserved for the Phase 5 query service
|
|
volumes:
|
|
- qdrant_storage:/qdrant/storage
|
|
environment:
|
|
# The corpus is ~1k chunks today and the measurement in
|
|
# reference/stack.md records the ceiling. Keep the HNSW defaults; do not
|
|
# tune for speed on a memory-bound host.
|
|
QDRANT__TELEMETRY_DISABLED: "true"
|
|
QDRANT__LOG_LEVEL: "WARN"
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: 10m
|
|
max-file: "3"
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "bash -c '</dev/tcp/127.0.0.1/6333' || exit 1"]
|
|
interval: 30s
|
|
timeout: 3s
|
|
retries: 3
|
|
start_period: 10s
|
|
|
|
volumes:
|
|
qdrant_storage:
|