Sam Rolfe
f9dc574172
Satisfies docs/phase-4-corpus.md. content/ stays the single source of truth:
one pass over it writes both indexes, so the static pages and the retrieval
corpus cannot drift.
- corpus/: stdlib-only pipeline. validate -> chunk -> enrich -> embed -> upsert.
Chunking is block-based, so a boundary cannot fall inside a fenced block, a
table or front-matter (verified structurally, not by inspection).
- Deterministic ids: sha256(source_path + chunk_index + content_hash), rendered
as a UUID. Re-running updates points; re-chunking deletes the old ones by
source filter, so duplicates and orphans are both impossible.
- Incremental by content hash against a SQLite manifest. A provider, dimension
or chunker change is refused instead of mixed: it needs a rebuild.
- Payloads per section 4.2 plus provenance, validated before any vector is
written; a missing field fails the run and names the document and chunk.
- SQLite FTS5 over chunks (the keyword half of hybrid retrieval, decision D10)
with area/confidence filters and a `terms` column so front-matter tech and
tags are findable.
- Freshness audit: stale documents, due soon, index drift, contract drift;
optional Apprise alert; exit code for Prefect.
- Embeddings behind one interface: omniroute (OpenAI-compatible, retries, key
from the environment only) and local-hashing (offline placeholder). The
provider decision is still Sam's; the pipeline does not depend on it.
- deploy/qdrant.yml: mem_limit = memswap_limit = 400m, loopback only.
- docs/phase-4-notes.md: operator manual, DoD evidence, measured memory.
- 76 unit tests, no container and no network.
Measured, Qdrant 1.20.0 under a 400 m cap at 1024 dimensions: 500 / 5,000 /
20,000 chunks settle at 128 / 138 / 151 MiB, peaking at 149 / 305 / 292 MiB
during ingest. Qdrant's default settings are OOM-killed between 5,000 and
20,000, so the collection is created with HNSW and payloads on disk; recorded in
reference/stack.md with the ceiling and the escalation order.
Note: the `make corpus*` targets were swept into the Phase 3 deploy commit
24590cc by concurrent work in this tree; the Makefile content is correct, and
this commit contains the pipeline they call.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
94 lines
4.6 KiB
Markdown
94 lines
4.6 KiB
Markdown
# Ai Resume — Paperclip build directory
|
|
|
|
This is the **working directory for the Paperclip project `Resume Ai`**. Paperclip builds here. Everything needed to build is in this folder; nothing outside it is required.
|
|
|
|
## Relationship to the planning folder
|
|
|
|
| Folder | Holds | Status |
|
|
|---|---|---|
|
|
| `/home/sam/paseo/projects/resume_ai_system/` | The *why* — research, cull decisions, stack rationale, evidence notes, content drafts | **Planning. Not the build.** Sam works here |
|
|
| `/home/sam/paseo/projects/resume_ai_system_paperclip_build/` | The *what* — build spec, phase documents, reference material, and the site itself | **This folder. Paperclip works here** |
|
|
|
|
The planning folder is read-only reference. Do not edit it from here, and do not make the build depend on it — anything needed must be copied in.
|
|
|
|
## Start here
|
|
|
|
1. **`AGENTS.md`** — the build spec. Constraints, conventions, and the phase list. Read it before anything else.
|
|
2. **`docs/phase-N-*.md`** — one document per phase: work, Definition of Done, out of scope, risks.
|
|
3. **`reference/`** — frozen decisions, stack and budgets, content map, acceptance tests.
|
|
|
|
## Phases
|
|
|
|
| # | Phase | Ships |
|
|
|---|---|---|
|
|
| 1 | Content source of truth | `content/` + validator |
|
|
| 2 | Mode A static site | Browsable pre-rendered site, no AI |
|
|
| 3 | Public deployment | Live URL behind Caddy |
|
|
| 4 | Corpus and retrieval | `content/` → Qdrant |
|
|
| 5 | Mode B console | Streaming, cited answers |
|
|
| 6 | Guardrails, cost, kill switch | Safe to leave public |
|
|
| 7 | Polish | Logos, diagrams, code exhibits |
|
|
|
|
Order is deliberate. Content first (both renderers need it). **Deploy at Phase 3, not last** — a live half-built URL is worth more than a perfect site that never ships. AI after deployment so it can never block launch.
|
|
|
|
## Repository
|
|
|
|
This folder **is** the git working copy of `sam/ai-resume`. One repo, one product.
|
|
|
|
Previously held an abandoned Vite/React scaffold. It was replaced by this build spec; the history is retained because it honestly records the false start, and it is referenced in the site's "alternatives considered" content.
|
|
|
|
## Build and run
|
|
|
|
The site is one Go binary serving pre-rendered HTML. There is no Node, no bundler and no
|
|
`node_modules`; `go.sum` is the only dependency lock.
|
|
|
|
```sh
|
|
nix develop # go, templ, sqlite3, make, curl
|
|
make validate # content gate: front-matter against the frozen schema
|
|
make test # go test ./... plus the validator self-test
|
|
make dist # templ generate → bin/ → pre-rendered dist/
|
|
make dev # build, then serve on http://127.0.0.1:8096
|
|
```
|
|
|
|
`bin/resume-build` reads `content/` and writes `dist/` (pages, stage fragments, the fragment
|
|
manifest, the SQLite FTS5 index, assets). `bin/resume-server` serves `dist/` and never reads
|
|
`content/`, never calls a model, and never opens a socket to another service — that is what
|
|
keeps acceptance test A1 true.
|
|
|
|
The two things that are not files are `/search` (FTS5 over page titles and body text) and
|
|
`/api/stage/stream` (the presentation panel driven by an event stream instead of a route).
|
|
|
|
The retrieval corpus (Phase 4) is built separately, because it writes to a service rather
|
|
than to `dist/`:
|
|
|
|
```sh
|
|
docker compose -f deploy/qdrant.yml up -d # on the runtime host, loopback only
|
|
make corpus # content/ -> chunks -> Qdrant + SQLite FTS5
|
|
make corpus-verify # does the index agree with content/ and with Qdrant?
|
|
make corpus-audit # stale documents, index drift, contract drift
|
|
make corpus-test # 76 unit tests, no container and no network
|
|
```
|
|
|
|
The site never calls Qdrant, so acceptance test A1 is unaffected: stopping the corpus
|
|
services leaves every page working. Operator manual and measurements:
|
|
`docs/phase-4-notes.md`.
|
|
|
|
## Ground rules
|
|
|
|
- **Do not touch other systems.** No stopping containers, no editing other projects, no `nixos-rebuild`, no Caddy changes except the single block specified in Phase 3 — and that one Sam applies.
|
|
- **Mode A must work with every AI service stopped.** Hard gate.
|
|
- **Never invent facts about Sam's career.** If it is not in `reference/` or `content/`, ask.
|
|
- **Secrets** live in `~/.config/environment.d/` only.
|
|
- **All models** go through OmniRoute.
|
|
|
|
## Current state
|
|
|
|
- [x] Repo prepared, scaffold retired, build spec written
|
|
- [x] Phase 1 — content source of truth (40 documents across six areas, `make validate`)
|
|
- [x] Phase 2 — static site (`make dist`, 45 pages, 86 stage fragments, FTS5 search)
|
|
- [ ] Phase 3 — public deployment
|
|
- [x] Phase 4 — corpus and retrieval (51 chunks, FTS5 + Qdrant, freshness audit)
|
|
- [ ] Phase 5 — AI overlay
|
|
- [ ] Phase 6 — guardrails
|
|
- [ ] Phase 7 — polish
|