Sam Rolfe
f9dc574172
Satisfies docs/phase-4-corpus.md. content/ stays the single source of truth:
one pass over it writes both indexes, so the static pages and the retrieval
corpus cannot drift.
- corpus/: stdlib-only pipeline. validate -> chunk -> enrich -> embed -> upsert.
Chunking is block-based, so a boundary cannot fall inside a fenced block, a
table or front-matter (verified structurally, not by inspection).
- Deterministic ids: sha256(source_path + chunk_index + content_hash), rendered
as a UUID. Re-running updates points; re-chunking deletes the old ones by
source filter, so duplicates and orphans are both impossible.
- Incremental by content hash against a SQLite manifest. A provider, dimension
or chunker change is refused instead of mixed: it needs a rebuild.
- Payloads per section 4.2 plus provenance, validated before any vector is
written; a missing field fails the run and names the document and chunk.
- SQLite FTS5 over chunks (the keyword half of hybrid retrieval, decision D10)
with area/confidence filters and a `terms` column so front-matter tech and
tags are findable.
- Freshness audit: stale documents, due soon, index drift, contract drift;
optional Apprise alert; exit code for Prefect.
- Embeddings behind one interface: omniroute (OpenAI-compatible, retries, key
from the environment only) and local-hashing (offline placeholder). The
provider decision is still Sam's; the pipeline does not depend on it.
- deploy/qdrant.yml: mem_limit = memswap_limit = 400m, loopback only.
- docs/phase-4-notes.md: operator manual, DoD evidence, measured memory.
- 76 unit tests, no container and no network.
Measured, Qdrant 1.20.0 under a 400 m cap at 1024 dimensions: 500 / 5,000 /
20,000 chunks settle at 128 / 138 / 151 MiB, peaking at 149 / 305 / 292 MiB
during ingest. Qdrant's default settings are OOM-killed between 5,000 and
20,000, so the collection is created with HNSW and payloads on disk; recorded in
reference/stack.md with the ceiling and the escalation order.
Note: the `make corpus*` targets were swept into the Phase 3 deploy commit
24590cc by concurrent work in this tree; the Makefile content is correct, and
this commit contains the pipeline they call.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
3.6 KiB
Stack, hosts, ports, budgets
Hosts (measured 2026-10-17)
| Host | Name | Role | RAM total / free | Swap used | CPUs |
|---|---|---|---|---|---|
.13 |
nixos-desktop |
Runtime — app, Qdrant, Prefect worker | 15 GiB / 2.3 GiB | 8.1 GiB | 12 |
.35 |
sam-Ubuntu1 |
Public edge — Caddy terminates 443; also Gitea, Immich, Nextcloud, Penpot, Wiki.js | 14 GiB / 4.1 GiB | 4.0 GiB | 2 |
.27 |
sam-4screen-desktop |
Build / dev — sources, Obsidian vault, agent work | — | — | — |
.13 is memory-bound, not CPU-bound. The budget is real: exceeding it destabilises services Sam relies on.
Ports
| Service | Host | Port | State |
|---|---|---|---|
| Caddy | .35 |
443 (+ LAN 8000) | existing — Sam applies any change |
| Go app | .13 |
8096 | new — verify free (8095 = offsite.lab) |
| Qdrant REST | .13 |
6333 | new, 127.0.0.1 only |
| Qdrant gRPC | .13 |
6334 | new, 127.0.0.1 only |
| Prefect server | .13 |
existing | reuse — do not deploy a second |
| OmniRoute | .13 |
existing | reuse — the only model gateway |
| Postgres | .13 |
5432 | reuse if needed |
| Whisper / Piper | .13 |
existing | reuse (Phase 7 voice only) |
| LangGraph service | .13 |
8096+1 or existing | Phase 5 — see note below |
Known occupied: 3000, 3020, 5000, 5002, 5432, 8000, 8001, 8080, 8082, 8090, 8091, 8092, 8095, 8230, 8231, 8500, 8501, 8788–8792, 3081–3084, 1883.
Memory budget
| Component | Cap |
|---|---|
| Go app | 120 m |
| Qdrant | 400 m |
| Prefect worker | 150 m |
| Guardrail (if local) | 300 m |
| Total ceiling | 1.0 GiB / 4 containers |
Qdrant memory, measured (2026-10-26, v1.20.0, 400 m cap, 1024 dims)
| Chunks | Settled RSS | Peak during ingest |
|---|---|---|
| 500 | 128 MiB | 149 MiB |
| 5,000 | 138 MiB | 305 MiB |
| 20,000 | 151 MiB | 292 MiB |
Within the cap at all three sizes, but only with the memory-frugal collection
settings in corpus/qdrant.py: HNSW graph and payloads on disk
(hnsw_config.on_disk, on_disk_payload), m=8, ef_construct=64,
memmap_threshold=5000, indexing_threshold=20000. With Qdrant's defaults the
container is OOM-killed between 5,000 and 20,000 chunks. The pipeline is the only
code that creates the collection, so those settings cannot drift.
Ceiling: 20,000 chunks at 1024 dimensions. Beyond that, set
RESUME_QDRANT_VECTORS_ON_DISK=1, reduce the dimension, or raise the cap — in
that order. Detail: docs/phase-4-notes.md §7.
Every service sets mem_limit and memswap_limit to the same value so it cannot consume host swap. Bind to 127.0.0.1; Caddy is the only public listener.
Paths
| Item | Path |
|---|---|
| Repo working copy | /home/sam/paseo/projects/resume_ai_system_paperclip_build |
| Release root | /home/sam/deployment/resume/ |
| Live symlink | /home/sam/deployment/resume/current |
| Secrets | ~/.config/environment.d/10-secrets.conf |
| Planning reference | /home/sam/paseo/projects/resume_ai_system/plan/ (read-only) |
Public entry
resume.lab.audasmedia.com.au — wildcard DNS already resolves to the public edge on .35. No DNS work is required. Caddy block is specified in Phase 3 §3.3.
Open items
| Item | Owner |
|---|---|
| Embedding provider (Gemini free tier vs OpenRouter vs local) | Sam — Phase 4 shipped provider-agnostic (corpus/embeddings.py); the omniroute adapter is ready, local-hashing is a placeholder, so the decision is still Sam's before Phase 5 answers questions |
| Repo public/private after launch | Sam — decide after Phase 3 |
| Phone number on the contact page | Sam |