Compare commits
25 Commits
feat/phase
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6900101506 | ||
|
|
396eeb175e | ||
|
|
36d8e514d4 | ||
| 2ceb0c147c | |||
| cb45c462be | |||
| 6ac0e1ff47 | |||
| a76882de67 | |||
|
|
59580b6e45 | ||
| d793ab0871 | |||
|
|
286aa12bb0 | ||
|
|
f39f560c7f | ||
|
|
df864cc29e | ||
|
|
5c0622c1b1 | ||
| 9fae1c7dd8 | |||
| f9dc574172 | |||
| 24590cc351 | |||
| c95e5361e3 | |||
| 2906683f44 | |||
| c2b4e33897 | |||
| 84fc585c91 | |||
| fd9974d07e | |||
| ea2c722816 | |||
| 72edfec59c | |||
|
|
00fa15f98f | ||
| 0eb151117e |
5
.gitignore
vendored
@@ -1,4 +1,9 @@
|
||||
# python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# build output
|
||||
/bin/
|
||||
/dist/
|
||||
/current/
|
||||
*.test
|
||||
|
||||
106
Makefile
Normal file
@@ -0,0 +1,106 @@
|
||||
# Ai Resume - build entry points.
|
||||
#
|
||||
# `make validate` is the content gate: it is what CI and deploy.sh call.
|
||||
# It exits 0 only on a clean content tree.
|
||||
#
|
||||
# `make build` is the site gate: it generates the templ components, compiles the
|
||||
# two small binaries and writes the pre-rendered site into dist/. One binary
|
||||
# (bin/resume-server) serves the whole site; there is no bundler and no Node.
|
||||
|
||||
PYTHON ?= python3
|
||||
VALIDATOR := $(PYTHON) tools/validate_content.py
|
||||
GO ?= go
|
||||
TEMPL ?= templ
|
||||
|
||||
# The runtime image is distroless/static (deploy/Dockerfile): no libc, no
|
||||
# dynamic loader. Everything the server needs is pure Go (modernc.org/sqlite),
|
||||
# so a static binary is buildable and is what the container can actually exec.
|
||||
# Without this the build host's cgo links the nix glibc loader and the container
|
||||
# fails with "exec /usr/bin/resume-server: no such file or directory".
|
||||
CGO_ENABLED ?= 0
|
||||
|
||||
BIN := bin
|
||||
DIST := dist
|
||||
|
||||
.PHONY: build dist generate dev test validate validate-self-test fmt vet clean help \
|
||||
corpus corpus-sync corpus-rebuild corpus-audit corpus-verify corpus-demo corpus-test
|
||||
|
||||
help:
|
||||
@echo "make build templ generate, compile bin/, no dist"
|
||||
@echo "make dist build, then pre-render content into dist/"
|
||||
@echo "make dev pre-render into dist/ and serve it on 127.0.0.1:8096"
|
||||
@echo "make generate run templ generate (generated files are not committed)"
|
||||
@echo "make test go test ./... plus the validator self-test"
|
||||
@echo "make validate validate content/** front-matter against the frozen schema"
|
||||
@echo "make fmt vet gofmt / go vet over cmd and internal"
|
||||
@echo "make clean remove bin/ and dist/"
|
||||
@echo ""
|
||||
@echo "corpus (Phase 4, python3 -m corpus.cli):"
|
||||
@echo "make corpus rebuild the corpus from content/ (drops the collection)"
|
||||
@echo "make corpus-sync incremental update; only changed documents are re-embedded"
|
||||
@echo "make corpus-audit freshness + drift audit (exit 1 on findings)"
|
||||
@echo "make corpus-verify check the index against content/ and Qdrant"
|
||||
@echo "make corpus-demo same query with and without an area filter"
|
||||
@echo "make corpus-test corpus unit tests (no container, no network)"
|
||||
|
||||
build: generate
|
||||
@mkdir -p $(BIN)
|
||||
CGO_ENABLED=$(CGO_ENABLED) $(GO) build -trimpath -buildvcs=false -o $(BIN)/resume-build ./cmd/build
|
||||
CGO_ENABLED=$(CGO_ENABLED) $(GO) build -trimpath -buildvcs=false -o $(BIN)/resume-server ./cmd/server
|
||||
|
||||
dist: build
|
||||
@$(BIN)/resume-build --dist $(DIST)
|
||||
|
||||
dev: dist
|
||||
@RESUME_DIST=$(DIST) $(BIN)/resume-server
|
||||
|
||||
generate:
|
||||
@command -v $(TEMPL) >/dev/null || { echo "templ is not on PATH; run this inside 'nix develop'"; exit 1; }
|
||||
@$(TEMPL) generate
|
||||
|
||||
test: generate validate-self-test
|
||||
@$(GO) test ./...
|
||||
|
||||
validate:
|
||||
@$(VALIDATOR)
|
||||
|
||||
validate-self-test:
|
||||
@$(VALIDATOR) --self-test
|
||||
|
||||
fmt:
|
||||
@gofmt -l -w cmd internal
|
||||
|
||||
vet:
|
||||
@$(GO) vet ./...
|
||||
|
||||
# -- corpus (Phase 4, docs/phase-4-corpus.md) --------------------------------
|
||||
#
|
||||
# The corpus is data, not a build artefact: it lives in data/corpus/ (gitignored)
|
||||
# and in Qdrant. RESUME_QDRANT_URL defaults to http://127.0.0.1:6333 and
|
||||
# RESUME_EMBED_PROVIDER defaults to local-hashing; production sets both in the
|
||||
# environment, never in this file.
|
||||
|
||||
CORPUS := $(PYTHON) -m corpus.cli
|
||||
|
||||
corpus: corpus-rebuild
|
||||
|
||||
corpus-rebuild:
|
||||
@$(CORPUS) rebuild
|
||||
|
||||
corpus-sync:
|
||||
@$(CORPUS) sync
|
||||
|
||||
corpus-audit:
|
||||
@$(CORPUS) audit --fail-on-findings
|
||||
|
||||
corpus-verify:
|
||||
@$(CORPUS) verify
|
||||
|
||||
corpus-demo:
|
||||
@$(CORPUS) demo-filter --query "distributed audio endpoints" --area edge
|
||||
|
||||
corpus-test:
|
||||
@$(PYTHON) -m unittest discover -s corpus/tests -t .
|
||||
|
||||
clean:
|
||||
@rm -rf $(BIN) $(DIST)
|
||||
42
README.md
@@ -37,6 +37,42 @@ This folder **is** the git working copy of `sam/ai-resume`. One repo, one produc
|
||||
|
||||
Previously held an abandoned Vite/React scaffold. It was replaced by this build spec; the history is retained because it honestly records the false start, and it is referenced in the site's "alternatives considered" content.
|
||||
|
||||
## Build and run
|
||||
|
||||
The site is one Go binary serving pre-rendered HTML. There is no Node, no bundler and no
|
||||
`node_modules`; `go.sum` is the only dependency lock.
|
||||
|
||||
```sh
|
||||
nix develop # go, templ, sqlite3, make, curl
|
||||
make validate # content gate: front-matter against the frozen schema
|
||||
make test # go test ./... plus the validator self-test
|
||||
make dist # templ generate → bin/ → pre-rendered dist/
|
||||
make dev # build, then serve on http://127.0.0.1:8096
|
||||
```
|
||||
|
||||
`bin/resume-build` reads `content/` and writes `dist/` (pages, stage fragments, the fragment
|
||||
manifest, the SQLite FTS5 index, assets). `bin/resume-server` serves `dist/` and never reads
|
||||
`content/`, never calls a model, and never opens a socket to another service — that is what
|
||||
keeps acceptance test A1 true.
|
||||
|
||||
The two things that are not files are `/search` (FTS5 over page titles and body text) and
|
||||
`/api/stage/stream` (the presentation panel driven by an event stream instead of a route).
|
||||
|
||||
The retrieval corpus (Phase 4) is built separately, because it writes to a service rather
|
||||
than to `dist/`:
|
||||
|
||||
```sh
|
||||
docker compose -f deploy/qdrant.yml up -d # on the runtime host, loopback only
|
||||
make corpus # content/ -> chunks -> Qdrant + SQLite FTS5
|
||||
make corpus-verify # does the index agree with content/ and with Qdrant?
|
||||
make corpus-audit # stale documents, index drift, contract drift
|
||||
make corpus-test # 76 unit tests, no container and no network
|
||||
```
|
||||
|
||||
The site never calls Qdrant, so acceptance test A1 is unaffected: stopping the corpus
|
||||
services leaves every page working. Operator manual and measurements:
|
||||
`docs/phase-4-notes.md`.
|
||||
|
||||
## Ground rules
|
||||
|
||||
- **Do not touch other systems.** No stopping containers, no editing other projects, no `nixos-rebuild`, no Caddy changes except the single block specified in Phase 3 — and that one Sam applies.
|
||||
@@ -48,10 +84,10 @@ Previously held an abandoned Vite/React scaffold. It was replaced by this build
|
||||
## Current state
|
||||
|
||||
- [x] Repo prepared, scaffold retired, build spec written
|
||||
- [ ] Phase 1 — content source of truth
|
||||
- [ ] Phase 2 — static site
|
||||
- [x] Phase 1 — content source of truth (40 documents across six areas, `make validate`)
|
||||
- [x] Phase 2 — static site (`make dist`, 45 pages, 86 stage fragments, FTS5 search)
|
||||
- [ ] Phase 3 — public deployment
|
||||
- [ ] Phase 4 — corpus
|
||||
- [x] Phase 4 — corpus and retrieval (51 chunks, FTS5 + Qdrant, freshness audit)
|
||||
- [ ] Phase 5 — AI overlay
|
||||
- [ ] Phase 6 — guardrails
|
||||
- [ ] Phase 7 — polish
|
||||
|
||||
58
cmd/build/main.go
Normal file
@@ -0,0 +1,58 @@
|
||||
// Command build pre-renders the whole site into a directory.
|
||||
//
|
||||
// It reads content/ and writes dist/: every page as HTML, one file per stage
|
||||
// fragment, the fragment manifest, the SQLite FTS5 index, the assets and the
|
||||
// static files. No network call is made, and the output is what the server
|
||||
// serves — Mode A never renders a page per request.
|
||||
//
|
||||
// Configuration is from the environment, so the same binary is used by
|
||||
// `make build`, by CI and by the container image:
|
||||
//
|
||||
// RESUME_ROOT repository root (default ".")
|
||||
// RESUME_CONTENT content directory (default $RESUME_ROOT/content)
|
||||
// RESUME_DIST output directory (default $RESUME_ROOT/dist)
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"gitea.lab.audasmedia.com.au/sam/ai-resume/internal/site"
|
||||
)
|
||||
|
||||
func main() {
|
||||
flag.Usage = func() {
|
||||
fmt.Fprintf(flag.CommandLine.Output(), "usage: build [--root DIR] [--content DIR] [--dist DIR]\n")
|
||||
fmt.Fprintf(flag.CommandLine.Output(), "environment: RESUME_ROOT, RESUME_CONTENT, RESUME_DIST\n")
|
||||
}
|
||||
root := flag.String("root", env("RESUME_ROOT", "."), "repository root")
|
||||
contentDir := flag.String("content", env("RESUME_CONTENT", ""), "content directory")
|
||||
distDir := flag.String("dist", env("RESUME_DIST", ""), "output directory")
|
||||
flag.Parse()
|
||||
|
||||
cfg := site.Config{Root: *root, ContentDir: *contentDir, OutDir: *distDir}
|
||||
s, err := site.New(cfg)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "build: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := s.Build(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "build: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
info, err := site.LoadBuildInfo(s.Config.OutDir)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "build: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("built %d pages, %d stage fragments, %d indexed documents into %s\n",
|
||||
info.Pages, info.Fragments, info.Indexed, s.Config.OutDir)
|
||||
}
|
||||
|
||||
func env(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
155
cmd/server/main.go
Normal file
@@ -0,0 +1,155 @@
|
||||
// Command server serves the site the build wrote.
|
||||
//
|
||||
// It reads a directory produced by cmd/build and serves it on loopback. It does
|
||||
// not read content/, does not need Qdrant, OmniRoute or any other service, and
|
||||
// makes no outbound connection, so every page it serves works with the AI stack
|
||||
// stopped (acceptance test A1).
|
||||
//
|
||||
// Configuration comes from the environment so the same binary works under
|
||||
// `make dev`, systemd and compose:
|
||||
//
|
||||
// RESUME_ADDR listen address (default 127.0.0.1:8096)
|
||||
// RESUME_DIST build directory (default "dist")
|
||||
// RESUME_STREAM_DELAY pacing for the streamed panel, e.g. 350ms
|
||||
//
|
||||
// Flags with the same names override the environment for local work.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.lab.audasmedia.com.au/sam/ai-resume/internal/server"
|
||||
)
|
||||
|
||||
func main() {
|
||||
addr := flag.String("addr", env("RESUME_ADDR", server.DefaultAddr), "listen address")
|
||||
dist := flag.String("dist", env("RESUME_DIST", server.DefaultDist), "build directory")
|
||||
streamDelay := flag.Duration("stream-delay", envDuration("RESUME_STREAM_DELAY", server.DefaultStreamDelay), "delay between streamed panel frames")
|
||||
healthCheck := flag.Bool("health-check", false, "ask a running server for /healthz and exit 0 only if static is ready")
|
||||
flag.Usage = func() {
|
||||
fmt.Fprintf(flag.CommandLine.Output(), "usage: server [--addr HOST:PORT] [--dist DIR] [--stream-delay DURATION] [--health-check]\n")
|
||||
fmt.Fprintf(flag.CommandLine.Output(), "environment: RESUME_ADDR, RESUME_DIST, RESUME_STREAM_DELAY\n")
|
||||
}
|
||||
flag.Parse()
|
||||
|
||||
// The container image has no shell and no curl, so the health probe is this
|
||||
// binary talking to itself (deploy/compose.yml healthcheck).
|
||||
if *healthCheck {
|
||||
if err := probe(*addr); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "health check failed: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Println("static ready")
|
||||
return
|
||||
}
|
||||
|
||||
log := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
|
||||
|
||||
site, err := server.New(server.Config{
|
||||
Addr: *addr,
|
||||
DistDir: *dist,
|
||||
StreamDelay: *streamDelay,
|
||||
Logger: log,
|
||||
})
|
||||
if err != nil {
|
||||
log.Error("start failed", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer site.Close()
|
||||
|
||||
// No WriteTimeout on purpose: the stage stream keeps the response open for
|
||||
// as long as it is producing frames. Reads are bounded.
|
||||
srv := &http.Server{
|
||||
Addr: *addr,
|
||||
Handler: site.Handler(),
|
||||
ReadHeaderTimeout: 5 * time.Second,
|
||||
IdleTimeout: 120 * time.Second,
|
||||
MaxHeaderBytes: 64 << 10,
|
||||
ErrorLog: slog.NewLogLogger(log.Handler(), slog.LevelWarn),
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
go func() {
|
||||
<-ctx.Done()
|
||||
log.Info("shutting down")
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if err := srv.Shutdown(shutdownCtx); err != nil {
|
||||
log.Warn("shutdown", "error", err)
|
||||
}
|
||||
}()
|
||||
|
||||
log.Info("listening", "addr", *addr, "dist", *dist)
|
||||
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
log.Error("serve failed", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// probe asks a running server whether the static site is ready. Only the static
|
||||
// field decides the exit code: Mode A must report ready with every AI service
|
||||
// stopped (acceptance test A1).
|
||||
func probe(addr string) error {
|
||||
host, port, err := net.SplitHostPort(addr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if host == "" || host == "0.0.0.0" || host == "::" {
|
||||
host = "127.0.0.1"
|
||||
}
|
||||
client := &http.Client{Timeout: 3 * time.Second}
|
||||
res, err := client.Get("http://" + net.JoinHostPort(host, port) + "/healthz")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer res.Body.Close()
|
||||
if res.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("/healthz returned %s", res.Status)
|
||||
}
|
||||
var body struct {
|
||||
Static struct {
|
||||
Ready bool `json:"ready"`
|
||||
} `json:"static"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(res.Body, 64<<10)).Decode(&body); err != nil {
|
||||
return fmt.Errorf("decode /healthz: %w", err)
|
||||
}
|
||||
if !body.Static.Ready {
|
||||
return errors.New("static is not ready")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func env(key, fallback string) string {
|
||||
if v, ok := os.LookupEnv(key); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func envDuration(key string, fallback time.Duration) time.Duration {
|
||||
v, ok := os.LookupEnv(key)
|
||||
if !ok || v == "" {
|
||||
return fallback
|
||||
}
|
||||
d, err := time.ParseDuration(v)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "%s=%q is not a duration: %v\n", key, v, err)
|
||||
os.Exit(2)
|
||||
}
|
||||
return d
|
||||
}
|
||||
62
content/about/about-narrative.md
Normal file
@@ -0,0 +1,62 @@
|
||||
---
|
||||
id: about-narrative
|
||||
area: about
|
||||
level: 2
|
||||
title: "The narrative — including the years that need explaining"
|
||||
claim: "A self-taught career across six technology eras — interrupted on purpose by a non-IT working life and five years overseas — with 2020 to 2026 spent raising two children while building a business."
|
||||
confidence: running
|
||||
period: "1999–"
|
||||
tech: [NixOS, Neovim, Python, JavaScript, React, Go, HTMX]
|
||||
metrics: [{label: "years shipping software", value: 25}, {label: "years spent overseas", value: 5}, {label: "non-IT roles worked", value: 6}]
|
||||
artefacts: [{type: doc, label: "Non-IT working history", url: "content/about/l3/about-non-it-work.md"}, {type: doc, label: "Education", url: "content/about/l3/about-education.md"}, {type: doc, label: "Interests — one honest line each", url: "content/about/l3/about-interests.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [about, narrative, 2020-2026]
|
||||
---
|
||||
|
||||
# The narrative — including the years that need explaining
|
||||
|
||||
**Capability statement.** Someone who learned by doing, in public, on deadlines, and who has kept learning after the deadlines stopped being someone else's. The range is real and the gaps are stated.
|
||||
|
||||
## The beginning
|
||||
|
||||
Programming started as a childhood interest in the 1980s, self-taught alongside art, music, philosophy and sport. The career proper started in the late 1990s building websites for an Australian car marketplace. When interactive front ends arrived — a creative layer that needed a technical spine — there was no course to take, so it was learned directly and immediately put into commercial work. That combination, creative front end with engineering underneath, has been the shape of the work ever since.
|
||||
|
||||
## The agency years
|
||||
|
||||
Freelance in Melbourne, then full-time roles in the United Kingdom during the emergence of online gaming and interactive advertising, then back to freelance where the work was with the largest agencies and their clients — banks, broadcasters, drinks brands, publishers, a global energy company, a design software vendor. Most of it was delivered under deadlines that could not move, and a meaningful share of it was rescue work: rewriting someone else's stalled project and shipping it on their date.
|
||||
|
||||
## The other working life
|
||||
|
||||
Not all of it was software, and that is worth stating rather than hiding. Six roles outside technology — a ski chalet kitchen in the French Alps, wine service, guided horse tours in the Victorian high country, building a stable complex, state government administration, and thoroughbred stable work. They are the reason the engineering reads as practical: a kitchen at service and a production incident have the same shape.
|
||||
|
||||
## The education
|
||||
|
||||
A fine arts degree with philosophy, a master's in international relations and contemporary political theory, and study in music composition and economics. None of it is computer science, and all of it is used: systems thinking, argument, aesthetics, and the habit of asking what a thing is actually for.
|
||||
|
||||
## 2020 to 2026 — stated plainly
|
||||
|
||||
This period is on the page because a résumé that hides it invites the wrong question. It was spent as a full-time parent to two children while building in parallel:
|
||||
|
||||
- A digital signage business taken from idea to customers — the product, the deployments, the support.
|
||||
- A site builder and CMS that carried the marketing for it.
|
||||
- A lost-property tag system, deployed.
|
||||
- A homelab that became the platform for everything else: declarative configuration, containers, backups, networking, public ingress.
|
||||
- An AI estate — pipelines, routing, retrieval, voice, guardrails — built and operated on the same hardware.
|
||||
- A deliberate move from PHP and front-end scripting into Go, systems work and platform engineering, and from hand-installed servers into reproducible configuration.
|
||||
|
||||
The honest framing is in the evidence notes and is repeated here: *full-time parent, building in parallel*. The output of those years is on the rest of this site and can be inspected.
|
||||
|
||||
## What next
|
||||
|
||||
The same three areas at a larger scale: [AI and agentic systems](../ai/ai-systems.md), the [platform engineering underneath them](../platform/platform-estate.md), and the [constrained devices at the edge](../edge/edge-fleet.md). Melbourne based and flexible about location and format — remote, hybrid, office, or travel.
|
||||
|
||||
## Contact policy
|
||||
|
||||
This site publishes no address, no phone number and no family details. Contact is by email, with the address supplied directly rather than scraped from a page. If a phone number belongs on the site, that is Sam's decision to make and it is not made here.
|
||||
|
||||
## Trade-offs and limits
|
||||
|
||||
- **No computer science degree.** The trade is a longer, broader practical education; the pages on this site are the argument, not the certificate.
|
||||
- **A period of self-employment is described as such.** It is not dressed up as a sabbatical or as consulting.
|
||||
- **Interests are one honest sentence each.** The previous version of this site invented venues and results. That has been deleted rather than corrected in place.
|
||||
16
content/about/about-overview.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
id: about-overview
|
||||
area: about
|
||||
level: 1
|
||||
title: "About"
|
||||
claim: "Sam has shipped software for twenty-five years — self-taught, through a non-IT working life and five years overseas — and now builds AI systems, the platforms under them and the devices at the edge."
|
||||
confidence: running
|
||||
period: "1999–"
|
||||
tech: [NixOS, Neovim, Python, JavaScript, React, Go, HTMX]
|
||||
metrics: [{label: "years shipping software", value: 25}, {label: "years spent overseas", value: 5}, {label: "non-IT roles worked", value: 6}]
|
||||
artefacts: [{type: doc, label: "Non-IT working history", url: "content/about/l3/about-non-it-work.md"}, {type: doc, label: "Education", url: "content/about/l3/about-education.md"}, {type: doc, label: "Interests — one honest line each", url: "content/about/l3/about-interests.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [about, narrative, background]
|
||||
---
|
||||
The other five areas prove what can be built; this one explains where the judgement came from. The narrative, including 2020 to 2026, is on the [narrative page](about-narrative.md).
|
||||
35
content/about/l3/about-education.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
id: about-education
|
||||
area: about
|
||||
level: 3
|
||||
title: "Artefact — Education"
|
||||
claim: "A fine arts degree with philosophy · a master's in international relations and contemporary political theory · and study in music composition and economics."
|
||||
confidence: historic
|
||||
period: "1997–"
|
||||
tech: []
|
||||
metrics: [{label: "tertiary qualifications completed", value: 2}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [about, education, artefact]
|
||||
---
|
||||
|
||||
Formal presentation lands in Phase 7. No certificates, transcripts or student identifiers are published.
|
||||
|
||||
## The record
|
||||
|
||||
- **Master of International Relations and Contemporary Political Theory** — University of Westminster, Centre for the Study of Democracy.
|
||||
- **Bachelor of Visual Communication and Fine Arts** — RMIT (Melbourne) and the University of South Australia, 1997–1999.
|
||||
- **Music composition and economics** — studied at La Trobe University alongside the composition work described under interests.
|
||||
|
||||
## What it proves
|
||||
|
||||
Two things the technical record cannot: formal training in argument and political economy, and formal training in visual judgement. Domain-switching is a habit rather than a claim — the degrees are in different fields from each other and from the career.
|
||||
|
||||
## Why no computer science degree
|
||||
|
||||
Because there is not one, and saying so is cheaper than having it discovered. The self-taught path is documented across this site: the History pages show the eras, the Platform and Edge pages show what came of them.
|
||||
|
||||
## Coverage limit
|
||||
|
||||
Qualification names, awarding institutions and the years the résumé states are published. Grades, student numbers and the master's completion year are not in the sources and are therefore not stated.
|
||||
70
content/about/l3/about-interests.md
Normal file
@@ -0,0 +1,70 @@
|
||||
---
|
||||
id: about-interests
|
||||
area: about
|
||||
level: 3
|
||||
title: "Artefact — Interests · one honest line each"
|
||||
claim: "Fourteen pursuits with one true sentence each — and the invented venues · results and trips from the previous version of this site deleted rather than restated."
|
||||
confidence: running
|
||||
period: "1980–"
|
||||
tech: []
|
||||
metrics: [{label: "pursuits listed", value: 15}, {label: "bands played in", value: 2}, {label: "bee hives kept", value: 2}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [about, interests, artefact]
|
||||
---
|
||||
|
||||
|
||||
|
||||
## Music
|
||||
Blues and jazz piano and blues harp; played in two bands. Studied music composition at university and kept the piano as the thing that resets the week.
|
||||
|
||||
## Fine art
|
||||
A fine arts degree and a continuing practice. Drawing and painting taught patience with iteration, which is most of debugging.
|
||||
|
||||
## Philosophy
|
||||
A degree's worth of it, and it is where the habit of asking what a thing is actually for came from.
|
||||
|
||||
## Skiing
|
||||
Italy, France, Japan and Australia. Six months based in Meribel in the French Alps as a chalet chef, including the Vallée Blanche glacier run.
|
||||
|
||||
## Camping
|
||||
Hiking, camping and canoeing, mostly in Victoria. Most recent: a two-day hike through Wilsons Promontory with a group of teenagers in tow.
|
||||
|
||||
## Canoeing
|
||||
The Yarra, the Otways and the Murray.
|
||||
|
||||
## Horse riding
|
||||
Grew up riding in the Otways; broke horses from track work back to dressage; guided tours in the Victorian high country; competed in dressage for two years.
|
||||
|
||||
## Surfing
|
||||
Grew up between Lorne and the Otways, on waves and horses.
|
||||
|
||||
## Hiking
|
||||
The walking version of the camping answer: long days, small pack, no phone signal.
|
||||
|
||||
## Global politics
|
||||
The master's was in it. Reading it is a habit, not a credential to display.
|
||||
|
||||
## Reading
|
||||
History, philosophy, politics, economics — the same four subjects as the degrees, which is either consistency or a rut.
|
||||
|
||||
## Cooking
|
||||
Worked in many restaurants and as chef in a five-star chalet in Meribel, France.
|
||||
|
||||
## Gardening
|
||||
Permaculture and a wood-chip, low-input approach to self-sufficiency, with sensors and automation doing the boring parts.
|
||||
|
||||
## Food production
|
||||
Bees, salami and cheese. Two hives, and salami made annually from four pigs with Italian friends in Benalla.
|
||||
|
||||
## Italy and travel
|
||||
Five years spent overseas, travelling extensively, with Italy as the standing preference.
|
||||
|
||||
## What was deleted, and why
|
||||
|
||||
The previous site claimed venues, competition results, swell sizes, distances and named restaurants. None of it could be traced. It has been cut rather than corrected: a single true sentence is worth more than ten specific ones that cannot be substantiated, and an interviewer who asks a follow-up question is entitled to an answer that survives it.
|
||||
|
||||
## Coverage limit
|
||||
|
||||
Everything above traces to Sam's own notes. Anything not in those notes is absent, not paraphrased.
|
||||
40
content/about/l3/about-non-it-work.md
Normal file
@@ -0,0 +1,40 @@
|
||||
---
|
||||
id: about-non-it-work
|
||||
area: about
|
||||
level: 3
|
||||
title: "Artefact — Non-IT working history"
|
||||
claim: "Six roles outside technology — a chalet kitchen in the French Alps · wine service · guided horse tours · a stable build · government administration · thoroughbred stable work."
|
||||
confidence: historic
|
||||
period: "1990–2010"
|
||||
tech: []
|
||||
metrics: [{label: "non-IT roles worked", value: 6}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [about, history, work, artefact]
|
||||
---
|
||||
|
||||
The roles are final; the presentation lands in Phase 7.
|
||||
|
||||
## The roles
|
||||
|
||||
| Role | Where | What it involved |
|
||||
|---|---|---|
|
||||
| Chalet chef | Merribel, France | Meals for chalet guests, kitchen operations, menu planning, dietary requirements, in a high-pressure alpine season |
|
||||
| Wine waiter | South Australia and Victoria | Premium wine regions, varietals, food pairing |
|
||||
| Horseback tour guide | Victorian high country | Guided tours, rider safety, horse welfare, trail navigation |
|
||||
| Stable construction | Mount Beauty, Victoria | A 24-bay stable build — project management, budgeting, hands-on construction |
|
||||
| Civil servant | Victorian State Government, Melbourne | Administrative and operational support to state-level programs |
|
||||
| Horse trainer and stable hand | Sunbury, Victoria | Care and training of thoroughbreds, exercise, grooming, race-day preparation |
|
||||
|
||||
## What it proves
|
||||
|
||||
That the engineering sits on a wider working life. Kitchen service and production incidents share a shape; guiding strangers on horses in the high country is risk management with consequences; a stable build is a budget and a schedule held by hand. These are the reasons the technical pages read as practical rather than academic.
|
||||
|
||||
## Why this is on a technical site
|
||||
|
||||
Because leaving it out would make the career look narrower than it is, and because the years it occupies are otherwise unexplained.
|
||||
|
||||
## Coverage limit
|
||||
|
||||
Descriptions are limited to what the résumé and the evidence notes corroborate. Roles, employers' towns and the type of work are stated; no employer is named beyond the region, and no claim is made about scale, revenue or headcount that the sources do not carry.
|
||||
49
content/about/l3/about-sources.md
Normal file
@@ -0,0 +1,49 @@
|
||||
---
|
||||
id: about-sources
|
||||
area: about
|
||||
level: 3
|
||||
title: "What the assistant reads"
|
||||
claim: "Four sources, one pool, and a published rule for every one of them."
|
||||
confidence: running
|
||||
period: "2026–"
|
||||
tech: [Gitea, Obsidian, Graphify, Qdrant, SQLite]
|
||||
metrics: []
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, sources, corpus, retrieval]
|
||||
---
|
||||
|
||||
# What the assistant reads
|
||||
|
||||
The assistant answers from a pool built out of four sources, not from a single
|
||||
folder of copy.
|
||||
|
||||
| Source | Tier | What it contributes |
|
||||
|---|---|---|
|
||||
| This site's content | 1 | The curated claims: what Sam says is true about the work |
|
||||
| The Gitea estate | 2 | README files, file trees, selected source files, commit messages and language statistics |
|
||||
| The notes vault | 2 | Project and area notes, in Sam's own words |
|
||||
| The knowledge graph | 2 | Nodes and edges over the vault, each joined back to the note it came from |
|
||||
|
||||
Tier 1 wins when the two disagree. A harvested README can be out of date; the
|
||||
curated claim is the one that is maintained.
|
||||
|
||||
Every chunk records the platform it came from, the URL or repository path it can
|
||||
be opened at, the area it feeds, its tier, its honest confidence and when it last
|
||||
changed. A chunk without an origin is refused, not defaulted.
|
||||
|
||||
The rules are not hidden in code. Each platform has a guide that says what it is,
|
||||
where it lives, what is included, what is excluded and whether it is enabled.
|
||||
Those guides are shown at [the sources page](/platforms/) and are the same files
|
||||
the ingestion pipeline reads.
|
||||
|
||||
## What is excluded
|
||||
|
||||
`node_modules`, virtualenvs, git internals, trash, editor state, the SilverBullet
|
||||
auth file, credential-shaped files, binaries and media. Media is recorded as an
|
||||
image reference and never turned into text. Personal and family notes are not
|
||||
read.
|
||||
|
||||
The finished pool is scanned for credential-shaped content before a run can
|
||||
succeed, so the exclusion list is checked rather than trusted.
|
||||
16
content/ai/ai-overview.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
id: ai-overview
|
||||
area: ai
|
||||
level: 1
|
||||
title: "AI & Agentic Systems"
|
||||
claim: "Sam builds and operates sixteen or more AI pipelines: agent harnesses, model routing and cost control, retrieval over a tagged corpus, and a voice front-end that runs a house."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [LangGraph, Prefect, n8n, Qdrant, OmniRoute, Whisper, Piper, MiniMax]
|
||||
metrics: [{label: "pipelines maintained", value: 16}, {label: "model providers behind one gateway", value: 6}, {label: "audio endpoints for spoken output", value: 8}]
|
||||
artefacts: [{type: doc, label: "House AI hub — service topology", url: "content/ai/l3/ai-hub-architecture.md"}, {type: doc, label: "Model routing and cost control", url: "content/ai/l3/ai-omniroute-routing.md"}, {type: doc, label: "Voice pipeline — speech in and speech out", url: "content/ai/l3/ai-voice-pipeline.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, agents, routing, retrieval, voice]
|
||||
---
|
||||
The pipelines run against real obligations — school portals, calendars, email, household audio — and they are operated rather than demonstrated. What actually runs, and what it cost to get there, is on the [capability page](ai-systems.md).
|
||||
56
content/ai/ai-systems.md
Normal file
@@ -0,0 +1,56 @@
|
||||
---
|
||||
id: ai-systems
|
||||
area: ai
|
||||
level: 2
|
||||
title: "Agent systems that are operated rather than demonstrated"
|
||||
claim: "Maintains sixteen or more production AI pipelines — agent harnesses · prompt-analysis routing across six providers · retrieval filtered by metadata before it searches · and a guarded voice front-end."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [LangGraph, Prefect, n8n, Paperclip, Qdrant, OmniRoute, Whisper, Piper, MiniMax, TUIOS]
|
||||
metrics: [{label: "pipelines maintained", value: 16}, {label: "model providers behind one gateway", value: 6}, {label: "audio endpoints for spoken output", value: 8}, {label: "hosts under standardised agent operations", value: 4}]
|
||||
artefacts: [{type: doc, label: "House AI hub — service topology", url: "content/ai/l3/ai-hub-architecture.md"}, {type: doc, label: "Model routing and cost control", url: "content/ai/l3/ai-omniroute-routing.md"}, {type: doc, label: "Paperclip and n8n pipelines in production", url: "content/ai/l3/ai-paperclip-pipelines.md"}, {type: doc, label: "Retrieval corpus and query graph", url: "content/ai/l3/ai-rag-corpus.md"}, {type: doc, label: "Voice pipeline — speech in and speech out", url: "content/ai/l3/ai-voice-pipeline.md"}, {type: doc, label: "Guardrails — hooks in the model path", url: "content/ai/l3/ai-guardrails.md"}, {type: doc, label: "Memory in two layers", url: "content/ai/l3/ai-memory-layers.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, agents, routing, retrieval, voice, guardrails]
|
||||
---
|
||||
|
||||
**Capability statement.** This is a working estate, not a prototype: more than sixteen pipelines in production, six providers behind a single gateway, retrieval that filters on metadata before it searches, and a voice front-end used daily by people who did not build it.
|
||||
|
||||
## What is actually run
|
||||
|
||||
**Routing.** Every model call goes through one gateway — OmniRoute. Six providers sit behind it: DeepSeek, MiniMax, OpenRouter, OpenCode-Go, Gemini and OpenAI. Requests are classified before they are sent, so the cheapest model that can do the job gets the job. No project in the estate calls a provider directly; that is a rule, not a preference.
|
||||
|
||||
**Pipelines.** Ten or more are maintained on n8n, Prefect and LangGraph — email scanners, school-portal scanners, calendar scanners, project agents — plus six Paperclip pipelines that run work through a bench of narrow specialist agents behind human approval gates. The estate is registered in a dashboard built to keep track of it.
|
||||
|
||||
**Harnesses.** JEV is a custom harness with its own routing, tool set and memory rather than a wrapper around a vendor SDK. Work is executed as sequences of narrow agents instead of one large prompt; capability is loaded per task through a skill-selection step, so the agent gets what this job needs and not an enormous system prompt.
|
||||
|
||||
**Memory.** Two deliberate layers. Engram keeps what should persist — curated summaries, decisions, prompts and session handoffs in local SQLite, integrated with JEV for fast compaction. Headroom keeps what should not persist out of the window, stripping and compressing prompt context, tool output and logs before they reach the model. Storage and token control are separate problems and are solved separately.
|
||||
|
||||
**Retrieval.** ChromaDB first, then replaced with Qdrant for payload-filtered retrieval and a smaller footprint. LangGraph query graphs on top; Prefect and n8n for ingestion. The corpus is metadata-tagged at ingest, which means retrieval filters before it searches — the difference between a device answer and a work-history answer is a filter, not a better prompt.
|
||||
|
||||
**Voice.** Local Whisper for speech to text with a fallback, Piper for text to speech, MiniMax for a cloned voice with emotion and non-verbal cues. A custom wake word on an ESP32 with LED confirmation. Voice is a system front-end: it controls lighting, music, calendars, shopping, email, SMS and alerts, and it answers through eight Snapcast endpoints with music ducked underneath.
|
||||
|
||||
**Guardrails.** Custom hooks live in the model path and block injection and constrain tool use, built because the house voice channel is shared and unauthenticated and there was no vendor control to borrow. Consequential actions sit behind approval gates, and no model sits in the path of a fast or dangerous command. The public site applies the same reasoning from the other end: the assistant it exposes has no tools at all, so the attack surface is a conversation rather than a shell.
|
||||
|
||||
**Agent operations.** Agent sessions are standardised on TUIOS across four hosts — sessions survive a disconnect, agents report their own state, and approvals land in one inbox. TUIOS is a third-party tool: the contribution is operations and packaging, not authorship. It is pinned at 0.10.1 in a hand-written Nix derivation because nixpkgs lagged at 0.7.0, and its daemon is declared as a systemd user service so the fleet is reproducible rather than hand-installed.
|
||||
|
||||
## Evidence
|
||||
|
||||
| Claim | Evidence | Artefact |
|
||||
|---|---|---|
|
||||
| Sixteen or more pipelines in production | Ten or more on n8n, Prefect and LangGraph plus six on Paperclip | [Paperclip and n8n pipelines](l3/ai-paperclip-pipelines.md) |
|
||||
| Six providers behind one gateway | DeepSeek, MiniMax, OpenRouter, OpenCode-Go, Gemini, OpenAI | [Model routing](l3/ai-omniroute-routing.md) |
|
||||
| Retrieval filters before it searches | Corpus tagged at ingest; Qdrant payload filters; LangGraph query nodes | [Corpus and query graph](l3/ai-rag-corpus.md) |
|
||||
| Spoken output reaches eight endpoints | Eight Snapcast clients with music ducking | [Voice pipeline](l3/ai-voice-pipeline.md) |
|
||||
| Guardrails are code in the path | Injection-blocking hooks and constrained tool use | [Guardrails](l3/ai-guardrails.md) |
|
||||
| Memory is two layers by design | Engram for persistence, Headroom for context control | [Memory layers](l3/ai-memory-layers.md) |
|
||||
| Agent operations are standardised | TUIOS across four hosts with one approval inbox | [House AI hub](l3/ai-hub-architecture.md) |
|
||||
|
||||
## Trade-offs and limits
|
||||
|
||||
- **Local inference was measured, then demoted.** The estate started local on Ollama. Hardware constraints made local inference both slower and more expensive than partitioned remote calls, so heavy flows moved remote while sensitive and low-latency work stayed local. That is a documented comparison, not a taste in hardware.
|
||||
- **The retired stack is in the record.** Airflow DAGs were abandoned for RAM, ChromaDB was replaced by Qdrant, and a visual-builder layer was dropped. Each replacement is stated rather than hidden, because the reasoning is the transferable part.
|
||||
- **Not everything is automated.** Parts of the house are automated end to end and parts still need a person to press the button. The estate is described as it is run, not as it was imagined.
|
||||
- **Cost is bounded by routing, not by hope.** Caps and a refusal path exist so a runaway session cannot become a bill; the routing layer keeps the median request on a cheap model.
|
||||
|
||||
The period starts at the homelab year in the evidence notes: the AI estate and the homelab have the same origin.
|
||||
29
content/ai/l3/ai-guardrails.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: ai-guardrails
|
||||
area: ai
|
||||
level: 3
|
||||
title: "Artefact — Guardrails · hooks in the model path"
|
||||
claim: "Injection blocking and constrained tool use are code in the model path — built because the house voice channel is shared and unauthenticated."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [OmniRoute, Python]
|
||||
metrics: [{label: "tools exposed by the public assistant", value: 0}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, guardrails, security, artefact]
|
||||
---
|
||||
|
||||
A redacted injection-test result lands in Phase 6, when the guardrail library is wired into this site's own assistant.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The hooks that run before a request reaches a provider and before a response reaches an actuator: injection patterns are blocked, tool use is constrained to a declared set, and consequential actions stop at an approval gate. Behind them, one rule that outranks every prompt — no model sits in the path of a fast or dangerous command.
|
||||
|
||||
## What it proves
|
||||
|
||||
Safety enforced by mechanism. A prompt instruction is a suggestion; a hook is a control.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
The test result in Phase 6, using the same library the house runs. The policy that the public assistant has no tools at all is a Phase 5 and 6 decision and is not negotiable at runtime.
|
||||
29
content/ai/l3/ai-hub-architecture.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: ai-hub-architecture
|
||||
area: ai
|
||||
level: 3
|
||||
title: "Artefact — House AI hub service topology"
|
||||
claim: "One hub fronts the AI estate — intake · routing · pipelines · retrieval · voice output · and a single set of network tools namespaced per household member."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [n8n, Prefect, LangGraph, OmniRoute, Qdrant, MQTT, Snapcast]
|
||||
metrics: [{label: "provider gateway entry points", value: 1}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, architecture, artefact]
|
||||
---
|
||||
|
||||
The diagram and the generated topology dump land in Phase 7 (polish). This stub exists so the level 2 page has no dangling reference.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
A one-page diagram of the hub: request intake (voice · chat · scheduled job), the gateway hop, the pipeline runners, the retrieval path, and the output paths (house speakers, phone alerts, dashboards). Every box is a service that runs; nothing on the diagram is aspirational.
|
||||
|
||||
## What it proves
|
||||
|
||||
That the estate is a designed system with a single gateway and a single set of namespaced tools — not a collection of scripts that happen to share a machine.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. The source facts are in the evidence notes (sections 2 and 4). Capture must be from the live estate, so no diagram is drawn from memory in Phase 1.
|
||||
29
content/ai/l3/ai-memory-layers.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: ai-memory-layers
|
||||
area: ai
|
||||
level: 3
|
||||
title: "Artefact — Memory in two layers · Engram and Headroom"
|
||||
claim: "Memory is split by purpose — Engram stores what should persist and Headroom keeps what should not persist out of the context window."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [Engram, Headroom, SQLite, JEV]
|
||||
metrics: [{label: "memory layers", value: 2}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, memory, context, artefact]
|
||||
---
|
||||
|
||||
A redacted session handoff and a before-and-after context measurement land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
Engram is local-first storage for curated summaries, decisions, prompts and session handoffs in SQLite, so a conversation can be resumed in any project; it is integrated with JEV for fast compaction. Headroom is token control: it strips, cleanses and analyses code and logs before they reach the model, compressing rather than truncating.
|
||||
|
||||
## What it proves
|
||||
|
||||
Two distinct problems recognised as two — persistence and window economy — instead of one tool asked to do both.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Both tools are part of Sam's own workflow; they are deliberately not part of the public assistant, which is stateless by decision (see `reference/decisions.md` D11).
|
||||
29
content/ai/l3/ai-omniroute-routing.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: ai-omniroute-routing
|
||||
area: ai
|
||||
level: 3
|
||||
title: "Artefact — Model routing and cost control"
|
||||
claim: "Every model call is classified first and sent to the cheapest capable provider — six providers behind one gateway with no direct SDK calls anywhere in the estate."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [OmniRoute, JEV, DeepSeek, MiniMax, OpenRouter, Gemini, OpenAI]
|
||||
metrics: [{label: "providers in rotation", value: 6}, {label: "gateways", value: 1}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, routing, cost, artefact]
|
||||
---
|
||||
|
||||
The routing decision path and a redacted cost table land in Phase 7. No keys or account identifiers are ever published.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The decision path: request arrives, a prompt-analysis step classifies it, the gateway selects a provider and model, and the response returns with its cost recorded. Plus the comparison that demoted local inference for heavy flows — local Ollama measured slower and more expensive under the available hardware than partitioned remote calls.
|
||||
|
||||
## What it proves
|
||||
|
||||
Cost control implemented as routing and measurement rather than as a monthly promise, and a single chokepoint where provider access can be audited.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. The evidence for the provider list and the local-versus-remote decision is in the evidence notes, section 2 and section 4.
|
||||
29
content/ai/l3/ai-paperclip-pipelines.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: ai-paperclip-pipelines
|
||||
area: ai
|
||||
level: 3
|
||||
title: "Artefact — Paperclip and n8n pipelines in production"
|
||||
claim: "Sixteen or more pipelines run real work — scanners · schedulers · project agents — with specialist agents dispatched by task and human approval gates on anything consequential."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [Paperclip, n8n, Prefect, LangGraph]
|
||||
metrics: [{label: "pipelines on n8n and Prefect and LangGraph", value: 10}, {label: "Paperclip pipelines", value: 6}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, pipelines, agents, artefact]
|
||||
---
|
||||
|
||||
A redacted run log of one pipeline lands in Phase 7, alongside the pipeline inventory.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
One pipeline shown end to end: trigger, steps, the agent or model each step uses, the approval gate, and the run log with identifiers removed. Alongside it, the inventory of what the sixteen or more pipelines do — email scanners, school-portal scanners, calendar scanners, project oversight, and the household automations.
|
||||
|
||||
## What it proves
|
||||
|
||||
Volume with operations behind it: these are scheduled, logged and monitored, not run once for a demonstration.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Counts come from the evidence notes, section 3.
|
||||
35
content/ai/l3/ai-rag-corpus.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
id: ai-rag-corpus
|
||||
area: ai
|
||||
level: 3
|
||||
title: "Artefact — Retrieval corpus and query graph"
|
||||
claim: "Retrieval is filtered by metadata before it searches — a tagged corpus · a four-node LangGraph query graph · and a deliberate replacement of ChromaDB with Qdrant."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [Qdrant, LangGraph, Prefect, n8n, ChromaDB]
|
||||
metrics: [{label: "retrieval stores used over time", value: 2}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, retrieval, rag, artefact]
|
||||
---
|
||||
|
||||
Ingest and index are live; the diagram below shows both sides, and the query path is labelled as the design it is until Phase 5 ships.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The ingest and query path: documents are chunked and tagged with metadata at ingest, stored in Qdrant, and filtered on that metadata before a similarity search runs. The LangGraph graph has four nodes; the fourth decides whether the answer can be given from what was retrieved.
|
||||
|
||||
## What it proves
|
||||
|
||||
Retrieval designed around payload filtering and a hard stop, plus the reasoning for moving off ChromaDB — heavier footprint and weaker filtering — recorded as a decision rather than a migration.
|
||||
|
||||
## The pipeline
|
||||
|
||||

|
||||
|
||||
The separating line is the design decision: metadata is written at ingest, so the query can filter by area, level and source before it pays for a similarity search. The fourth node is the hard stop — an answer that cannot be supported by what was retrieved is refused rather than improvised.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Ingest is delivered and measured (this site's own corpus, Phase 4). The query path is delivered in Phase 5; the streaming console is the visitor-facing half of it.
|
||||
29
content/ai/l3/ai-voice-pipeline.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: ai-voice-pipeline
|
||||
area: ai
|
||||
level: 3
|
||||
title: "Artefact — Voice pipeline · speech in and speech out"
|
||||
claim: "Speech in on a custom wake word · local transcription · a cloned voice out · and music ducked under every announcement across eight endpoints."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [Whisper, Piper, MiniMax, ESP32, Snapcast, Mopidy, Home Assistant]
|
||||
metrics: [{label: "audio endpoints", value: 8}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, voice, audio, artefact]
|
||||
---
|
||||
|
||||
A short recording of music ducking under a spoken announcement lands in the optional voice phase. Text describes the chain until then.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The chain: an ESP32 device with a microphone detects the wake word and confirms with an LED, audio is transcribed locally with Whisper, the request goes through the gateway, and the reply is spoken through Piper or the MiniMax cloned voice over all eight Snapcast endpoints, with music lowered for the announcement and restored afterwards.
|
||||
|
||||
## What it proves
|
||||
|
||||
Voice treated as a system front-end with latency and reliability requirements — not a party trick bolted onto a chatbot.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Recording in the optional voice phase (Phase 7). The endpoint count and the ducking behaviour are from the evidence notes, sections 1 and 2.
|
||||
0
content/assets/.gitkeep
Normal file
@@ -1,68 +0,0 @@
|
||||
# Phase 7.4 — asset weight budget
|
||||
|
||||
Measured 2026-10-26 on `.13` with `stat -c%s` and `gzip -9`. Every number below
|
||||
is a measurement of a file that is in this repository, not an estimate.
|
||||
|
||||
## What is in the assets directory
|
||||
|
||||
| Group | Files | Raw | Gzip (`-9`) |
|
||||
|---|---|---|---|
|
||||
| Diagrams (`diagrams/*.svg`) | 8 | 51,729 B | 8,943 B |
|
||||
| Code exhibits (`code/*.as`, `.js`, `.php`, `.ts`, `.hpp`, `.nix`) | 6 | 8,161 B | 2,989 B |
|
||||
| Client wall data (`logos/clients.yaml`) | 1 | 3,912 B | 1,288 B |
|
||||
| **Served payload total** | **15** | **63,802 B** | **13,220 B** |
|
||||
| Manifests and READMEs (build-time only, never served) | 5 | 15,955 B | — |
|
||||
|
||||
## Per-asset
|
||||
|
||||
| File | Raw | Gzip |
|
||||
|---|---|---|
|
||||
| `diagrams/01-homelab-topology.svg` | 7,079 | 2,169 |
|
||||
| `diagrams/02-agent-architecture.svg` | 6,027 | 1,971 |
|
||||
| `diagrams/03-rag-pipeline.svg` | 7,010 | 2,043 |
|
||||
| `diagrams/04-edge-architecture.svg` | 6,896 | 2,163 |
|
||||
| `diagrams/05-distributed-audio.svg` | 6,833 | 1,997 |
|
||||
| `diagrams/06-media-pipeline.svg` | 5,694 | 1,718 |
|
||||
| `diagrams/07-cost-resource-engineering.svg` | 6,151 | 2,023 |
|
||||
| `diagrams/08-era-timeline.svg` | 6,039 | 1,930 |
|
||||
| `code/01-actionscript-class.as` | 958 | 495 |
|
||||
| `code/02-js-easing-core.js` | 1,848 | 584 |
|
||||
| `code/03-php-encode-op.php` | 2,295 | 765 |
|
||||
| `code/04-typescript-service-layer.ts` | 1,078 | 482 |
|
||||
| `code/05-embedded-cpp-interface.hpp` | 788 | 419 |
|
||||
| `code/06-nix-derivation.nix` | 1,194 | 724 |
|
||||
|
||||
Rasters: **zero.** There are no PNG, JPEG, WebP or font files in this phase.
|
||||
That is a design outcome, not an oversight — every diagram is text-based SVG and
|
||||
every code exhibit is text, so the "optimise every image" requirement has no
|
||||
image to optimise yet. Phase 7.4 screenshots are deferred (see below).
|
||||
|
||||
## The budget
|
||||
|
||||
| Rule | Limit | Worst case today |
|
||||
|---|---|---|
|
||||
| Any single diagram | ≤ 7.5 KB raw / ≤ 2.5 KB gzip | 7,079 B / 2,169 B |
|
||||
| Any single code exhibit | ≤ 3 KB raw / ≤ 1 KB gzip | 2,295 B / 765 B |
|
||||
| Diagrams on one page | ≤ 2 | 2 (14.1 KB raw) |
|
||||
| Asset payload per page, gzipped | ≤ 12 KB | 4.3 KB (two diagrams + one exhibit) |
|
||||
| Web fonts | 0 | 0 — system font stack only |
|
||||
| JS required by any asset | 0 | 0 — SVG carries its own palette |
|
||||
|
||||
Worst realistic page: an L2 page with two diagrams and one code exhibit is
|
||||
14,089 B raw, about **4.3 KB over the wire** after gzip and before the HTML, CSS
|
||||
and template overhead. The whole 15-file asset set is 13.2 KB gzipped, so even a
|
||||
page that loaded every one of them would stay inside a single small image's
|
||||
budget.
|
||||
|
||||
## What is not yet measured, and why
|
||||
|
||||
| Item | Status | Why |
|
||||
|---|---|---|
|
||||
| Full page weight (HTML + CSS + assets) | **not measured** | No page exists yet: Phase 2 owns the templates |
|
||||
| Lighthouse ≥ 95 / 95 | **not measured** | Needs a rendered page (Phase 2) served behind the edge (Phase 3). Nothing in this phase can stand in for it |
|
||||
| Screenshots and recordings (7.4) | **deferred** | The agent console is Phase 5 and the TUIOS fleet session has to be real work, not staged. Recording a console that does not exist would be a fabrication |
|
||||
| Home Assistant / Snapcast stills | **deferred** | Same reason: needs a live system and a judgement about whether a still adds what diagram 4 and diagram 5 already show |
|
||||
|
||||
These are recorded as open rather than asserted. DoD items 4 and 5 cannot be
|
||||
closed until Phase 2 and Phase 3 produce a page to measure, and the measurement
|
||||
must be taken by whoever holds that page.
|
||||
@@ -1,73 +0,0 @@
|
||||
# content/assets — authored exhibits
|
||||
|
||||
Everything the site shows that is not Markdown prose. Authored in Phase 7; wired
|
||||
into templates by Phase 2's builder.
|
||||
|
||||
```
|
||||
content/assets/
|
||||
├── BUDGET.md measured weights for every asset, and the budget
|
||||
├── VERIFY.md what was checked, with the output pasted in
|
||||
├── diagrams/ 8 SVGs, light and dark in one file each
|
||||
│ ├── README.md the visual language and the presentation contract
|
||||
│ └── manifest.yaml id, area, level, title, alt text per diagram
|
||||
├── code/ 6 verbatim excerpts, 212 lines, 5 eras
|
||||
│ ├── README.md the rules these follow
|
||||
│ └── manifest.yaml era, language, shows, rationale, honest note
|
||||
└── logos/
|
||||
└── clients.yaml the client wall: 45 entries, 44 distinct names
|
||||
```
|
||||
|
||||
## Three decisions worth knowing
|
||||
|
||||
**The client wall is type, not logos.** Forty-five names in one typeface at one
|
||||
size on an even grid. Nothing is an image, so nothing can break, and no client is
|
||||
visually ranked above another. Ten client marks do exist in Simple Icons; none is
|
||||
enabled, because using a company's mark on a public portfolio is a permission
|
||||
question for Sam, not a design choice for an agent.
|
||||
|
||||
**Diagrams carry both themes in one file.** Not two files that can drift: one
|
||||
`<style>` block with the light palette and the same selectors repeated under
|
||||
`prefers-color-scheme: dark`. The eight files therefore share one palette by
|
||||
construction.
|
||||
|
||||
**Code excerpts are copied, never retyped.** Including the defects. An excerpt
|
||||
that has been tidied is a claim, not evidence — and this site's whole argument is
|
||||
that the claims are checkable.
|
||||
|
||||
## What is wired and what is not
|
||||
|
||||
| DoD item | State | Evidence |
|
||||
|---|---|---|
|
||||
| Client wall reads as deliberate, no broken images, no placeholders | **assets done, markup not** | `logos/clients.yaml` — 45 entries, zero image references by design. The grid is CSS and belongs to Builder |
|
||||
| Eight diagrams, both themes, readable at 1× and on mobile | **done** | `VERIFY.md` §1–§3: 0 geometry findings, 8/8 resolve two palettes, rendered widths measured |
|
||||
| Six code exhibits with a two-sentence rationale each | **done** | `code/manifest.yaml`, `VERIFY.md` §5 |
|
||||
| Every image optimised; total weight budgeted and recorded | **assets done, page weight not** | `BUDGET.md`. Zero rasters. A full page cannot be weighed until a page exists |
|
||||
| Lighthouse ≥ 95 / 95 after assets | **not measurable yet** | Needs a served page (Phase 3) |
|
||||
| Archive never linked or exposed | **done** | `VERIFY.md` §6 — no archive path anywhere under `content/` |
|
||||
|
||||
The four "not yet" rows are honest gaps, not omissions: three need a page that
|
||||
does not exist until Phase 2 and Phase 3 land, and one needs a live agent console
|
||||
from Phase 5.
|
||||
|
||||
## Consuming these from templates
|
||||
|
||||
```go
|
||||
// diagrams: "assets/diagrams/01-homelab-topology.svg"
|
||||
<img src="/assets/diagrams/01-homelab-topology.svg"
|
||||
alt="{{ .Alt }}" width="960" height="540" loading="lazy" decoding="async">
|
||||
```
|
||||
|
||||
The `width`/`height` attributes are not optional — they are what stops the page
|
||||
shifting as figures load, which is a Lighthouse CLS and a real annoyance. `alt`
|
||||
comes from `manifest.yaml`; if it is set to `""` the SVG's own `<title>`/`<desc>`
|
||||
are not read out.
|
||||
|
||||
Figures go inside a horizontally scrollable wrapper (`diagrams/README.md`), or a
|
||||
960 px schematic becomes 5 px type on a phone.
|
||||
|
||||
## Adding an asset
|
||||
|
||||
1. Put the file in the right folder; no external references of any kind.
|
||||
2. Add an entry to the folder's `manifest.yaml` with `alt` and, for code,
|
||||
`rationale` and `honest_note`.
|
||||
3. Re-measure and update `BUDGET.md`. A budget nobody updates is a wish.
|
||||
@@ -1,124 +0,0 @@
|
||||
# Phase 7 — verification record
|
||||
|
||||
What was checked, how, and what it returned. Every block below is pasted output,
|
||||
not a description of output.
|
||||
|
||||
Run 2026-10-26 on `.13`, headless Chrome for measurement.
|
||||
|
||||
## 1. Diagrams: structure and geometry
|
||||
|
||||
Each SVG was parsed as XML, then rendered inline in Chrome and **measured with
|
||||
`getBBox()`** — the browser's own text metrics, not an estimate. The check
|
||||
reported text overflow outside its box, partial overlap between boxes, elements
|
||||
outside the viewBox, empty text nodes and unlabelled boxes.
|
||||
|
||||
```
|
||||
01-homelab-topology.svg OK
|
||||
02-agent-architecture.svg OK
|
||||
03-rag-pipeline.svg OK
|
||||
04-edge-architecture.svg OK
|
||||
05-distributed-audio.svg OK
|
||||
06-media-pipeline.svg OK
|
||||
07-cost-resource-engineering.svg OK
|
||||
08-era-timeline.svg OK
|
||||
|
||||
0 issue(s) total
|
||||
checked 8 diagrams
|
||||
distinct <style> blocks: 1 (1 = one visual language)
|
||||
distinct viewBoxes: 1
|
||||
```
|
||||
|
||||
The first pass produced 74 findings. They were fixed in the source of the
|
||||
diagrams, not suppressed in the checker: shortened labels, widened boxes,
|
||||
removed empty text nodes. The final run is clean.
|
||||
|
||||
**"One visual language" is one number: `distinct <style> blocks: 1`.** All eight
|
||||
files carry a byte-identical style block, so there is no second palette to drift.
|
||||
|
||||
## 2. Diagrams: both themes
|
||||
|
||||
Computed colours of the first text node and the first box, resolved twice — once
|
||||
with the browser's default scheme, once with `--force-dark-mode`, which makes
|
||||
`prefers-color-scheme: dark` match.
|
||||
|
||||
```
|
||||
01-homelab-topology.svg light rgb(27, 34, 43) rgb(244, 246, 248) → dark rgb(231, 237, 244) rgb(24, 32, 40) OK
|
||||
02-agent-architecture.svg light rgb(27, 34, 43) rgb(244, 246, 248) → dark rgb(231, 237, 244) rgb(24, 32, 40) OK
|
||||
03-rag-pipeline.svg light rgb(27, 34, 43) rgb(244, 246, 248) → dark rgb(231, 237, 244) rgb(24, 32, 40) OK
|
||||
04-edge-architecture.svg light rgb(27, 34, 43) rgb(244, 246, 248) → dark rgb(231, 237, 244) rgb(24, 32, 40) OK
|
||||
05-distributed-audio.svg light rgb(27, 34, 43) rgb(244, 246, 248) → dark rgb(231, 237, 244) rgb(24, 32, 40) OK
|
||||
06-media-pipeline.svg light rgb(27, 34, 43) rgb(233, 241, 253) → dark rgb(231, 237, 244) rgb(17, 37, 61) OK
|
||||
07-cost-resource-engineering.svg light rgb(27, 34, 43) rgb(230, 244, 242) → dark rgb(231, 237, 244) rgb(13, 38, 36) OK
|
||||
08-era-timeline.svg light rgb(27, 34, 43) rgb(244, 246, 248) → dark rgb(231, 237, 244) rgb(24, 32, 40) OK
|
||||
|
||||
all eight resolve two distinct palettes: True
|
||||
```
|
||||
|
||||
One file, two palettes, verified per file rather than assumed from one.
|
||||
|
||||
## 3. Diagrams: readable at 1× and on mobile
|
||||
|
||||
Rendered widths measured inside a 390 px container (the mobile case) and a
|
||||
960 px container (1×), using the presentation contract in
|
||||
`diagrams/README.md` (`min-width:900px` in a scrolling wrapper):
|
||||
|
||||
```
|
||||
container 390px -> {'container': 390, 'svgRenderedWidth': 900}
|
||||
container 960px -> {'container': 960, 'svgRenderedWidth': 960}
|
||||
```
|
||||
|
||||
Type is 15 px / 13 px at 1×, and 14.1 px / 12.2 px at the 900 px minimum —
|
||||
arithmetic from the rendered width over the 960 viewBox. Scaled to the container
|
||||
instead, the same labels would render at 5 px, which is unreadable; that is why
|
||||
the wrapper is a requirement and not a styling preference.
|
||||
|
||||
## 4. Accessibility
|
||||
|
||||
Every file carries `role="img"`, `<title>` and `<desc>`. `desc` length per file:
|
||||
219, 230, 285, 223, 193, 197, 260, 234 characters — a real alternative text, not
|
||||
a filename. The manifest repeats each `desc` verbatim as `alt`, so the page and
|
||||
the screen reader cannot disagree.
|
||||
|
||||
## 5. Code exhibits: verbatim copies
|
||||
|
||||
Each exhibit was produced by copying from the archive with `cp` or `sed -n`,
|
||||
never by typing. Line counts as reported by `wc -l`:
|
||||
|
||||
```
|
||||
36 01-actionscript-class.as
|
||||
42 02-js-easing-core.js
|
||||
27 03-php-encode-op.php
|
||||
37 04-typescript-service-layer.ts
|
||||
28 05-embedded-cpp-interface.hpp
|
||||
33 06-nix-derivation.nix
|
||||
212 total
|
||||
```
|
||||
|
||||
`03-php-encode-op.php` is two copied regions joined by one marked elision
|
||||
(line 20 of the file). Nothing else was altered.
|
||||
|
||||
## 6. Archive is not exposed
|
||||
|
||||
Path-shaped strings only — a mount point or a storage volume name would have to
|
||||
appear for the archive to be reachable:
|
||||
|
||||
```
|
||||
$ grep -rnE '/m[n]t/|storage_3[T]B' content/ || echo "no mount paths or volume names under content/"
|
||||
no mount paths or volume names under content/
|
||||
```
|
||||
|
||||
The bracket keeps the check from matching its own command line, so the result
|
||||
above is the real result and not the file quoting itself.
|
||||
|
||||
No matches. The archive is not linked, named, or reachable from anything under
|
||||
`content/`. Provenance is recorded as "private archive, path withheld" — the
|
||||
words only, never the location.
|
||||
|
||||
## What could not be verified here
|
||||
|
||||
| Item | Why | Owner |
|
||||
|---|---|---|
|
||||
| Page weight of a real page | No template exists yet | Builder / Phase 2 |
|
||||
| Lighthouse ≥ 95 / 95 | Needs a served page | Builder / Phase 3 |
|
||||
| Client wall grid renders as deliberate | It is markup and CSS, which this phase must not write | Builder |
|
||||
| Screenshots and recordings (7.4) | The console is Phase 5; staging it would be fabrication | Phase 5, then this phase |
|
||||
0
content/assets/code/01-actionscript-class.as
Executable file → Normal file
@@ -1,3 +1,7 @@
|
||||
// Excerpt from js/utils/text_effects.js (2,156 lines) in the WebsiteBuilder
|
||||
// source. Two regions are reproduced verbatim: the timeline loop that derives
|
||||
// each letter's delay from an easing function (lines 1111-1147), and the
|
||||
// hand-written easing functions themselves (lines 1340-1349).
|
||||
$(mySplitTextTarg).each(function(i){
|
||||
|
||||
tweenData.letter=$(this);
|
||||
@@ -5,8 +9,6 @@
|
||||
tweenData.text_text_to.letter=$(this);
|
||||
tweenData.text_text_from.letter=$(this);
|
||||
|
||||
|
||||
|
||||
if(text_effect_type_animation=="standard"){
|
||||
if(tweenData.ignoreItems=="on" && Math.random()<tweenData.ignore_amount)return;
|
||||
|
||||
@@ -37,6 +39,15 @@
|
||||
|
||||
setCssTweens(stime,text_effect_type_animation,tweenData,tweenData.text_static,tweenData.text_text_to,tweenData.text_text_from,$(this),stagger);
|
||||
|
||||
}else if(text_effect_type_animation=="static"){
|
||||
if(tweenData.text_static.ignoreItems=="on" && Math.random()<tweenData.text_static.ignore_amount)return;
|
||||
|
||||
|
||||
// --- easing functions, lines 1340-1349 ---
|
||||
//t,b,c,d
|
||||
//currenttime,start,change,duration
|
||||
//t and do can be time or frames
|
||||
function positionsCubic(t, b, c, d){
|
||||
t /= d;
|
||||
return c*t*t*t + b;
|
||||
}
|
||||
function positionSine(t, b, c, d) {
|
||||
return -c * Math.cos(t/d * (Math.PI/2)) + c + b;
|
||||
}
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
// Excerpt from php/setMediaEncoder.php (3,402 lines) in the WebsiteBuilder
|
||||
// source: the create_gif encode operation, lines 614-665, reproduced verbatim.
|
||||
if($output_options->encode_media=="create_gif") {
|
||||
$output_gif= ".." . $settings->directory . $output_file .$output_options->format .".gif";
|
||||
|
||||
@@ -17,11 +19,36 @@
|
||||
// $cmd = "ffmpeg -ss ".$first_media->add_media_start." -t ".$first_media->add_media_duration." -i ".$input_file." -i palette.png -y -filter_complex \"fps=10[x];[x][1:v]paletteuse\" ".$output_gif." ";
|
||||
$cmd = "ffmpeg -ss ".$first_media->add_media_start." -t ".$first_media->add_media_duration." -i ".$input_file." -i palette.png -y -filter_complex \"fps=".$settings->settings->movie_frame_rate.",scale=".$settings->settings->video_width.":".$settings->settings->video_height.":flags=lanczos[x];[x][1:v]paletteuse\" ".$output_gif." ";
|
||||
ffmpegMsg("ffmpeg","Creating Gif ", 0,$cmd);
|
||||
/* [...] 34 lines elided: ImageMagick mogrify frame-optimisation
|
||||
options, left disabled because they delayed output */
|
||||
|
||||
// $cmd_2='mogrify -delay '.$_POST['compress_gif_delay'].' -alpha remove '.$matte.' -depth '.$_POST['compress_gif_depth'].' '.$deconstruct.' '.$dither.' -colors '.$_POST['compress_gif_colors'].' -fuzz '.$_POST['compress_gif_fuzz'].'% '.$loop.' '.$optimize.' '.$path.' 2>&1';
|
||||
|
||||
$_POST['loop_gif']="true";
|
||||
$_POST['compress_gif_matte']="false";
|
||||
$_POST['compress_gif_deconstruct']="false";
|
||||
$_POST['compress_gif_dither']="true";
|
||||
$_POST['compress_gif_optimize']="true";
|
||||
|
||||
$loop=($_POST['loop_gif']=="false")?"-loop 1":"";
|
||||
$matte=($_POST['compress_gif_matte']=="true")?"-matte":"";
|
||||
$deconstruct=($_POST['compress_gif_deconstruct']=="true")?"-deconstruct":"";
|
||||
$dither=($_POST['compress_gif_dither']=="true")?"-dither none ":"";
|
||||
$optimize=($_POST['compress_gif_optimize']=="true")?"-layers optimize ":"";
|
||||
// $optimize=($_POST['compress_gif_optimize']=="true")?" -coalesce -layers RemoveDups ":"";
|
||||
|
||||
$_POST['compress_gif_depth']=16;
|
||||
$_POST['compress_gif_delay']=0;
|
||||
$_POST['compress_gif_fuzz']="4";
|
||||
$_POST['compress_gif_colors']="16";
|
||||
|
||||
$cmd_2='mogrify -delay '.$_POST['compress_gif_delay'].' -alpha remove '.$matte.' -depth '.$_POST['compress_gif_depth'].' '.$deconstruct.' '.$dither.' -colors '.$_POST['compress_gif_colors'].' -fuzz '.$_POST['compress_gif_fuzz'].'% '.$loop.' '.$optimize.' '.$output_gif.' 2>&1';
|
||||
$result=shell_exec($cmd_2);
|
||||
|
||||
|
||||
$output_options->edited_url= $output_options->options->directory.$output_file .$output_options->format . '.gif';
|
||||
$output_options->edited_name= $output_file .$output_options->format . '.gif';
|
||||
$output_options->edited_type = "image/gif";
|
||||
$output_options->edited_filetype = explode ( "/", "image/gif");
|
||||
insert_document($output_options->edited_url,"image/gif",$output_options->edited_name,"image","edited");
|
||||
$returnFiles = returnFiles ( $output_options, $output_file .$output_options->format . '.mp4', "video/mp4", '', null );
|
||||
return $returnFiles;
|
||||
}
|
||||
|
||||
0
content/assets/code/04-typescript-service-layer.ts
Executable file → Normal file
@@ -1,3 +1,5 @@
|
||||
// Two files from firmware/src/hw, reproduced verbatim: WiFiMgr.hpp (the
|
||||
// interface) and WiFiMgr.cpp (the implementation). The boundary is marked.
|
||||
// WiFiMgr.hpp — the interface the rest of the firmware depends on
|
||||
// WiFiMgr.hpp
|
||||
#pragma once
|
||||
|
||||
@@ -1,48 +0,0 @@
|
||||
# Phase 7.3 — code exhibits
|
||||
|
||||
Six excerpts, **212 lines**, spanning five eras. Evidence that the range behind
|
||||
the claims is real, in a form an engineer can read in two minutes.
|
||||
|
||||
## The rules these follow
|
||||
|
||||
1. **Verbatim.** Every file is a byte-for-byte copy from the private archive.
|
||||
Nothing is retyped, reformatted or tidied. A retyped excerpt proves nothing.
|
||||
2. **Never linked.** The archive is not referenced, linked or served from the
|
||||
public site. Provenance is recorded as "private archive" and the path is
|
||||
withheld from this repository so it cannot leak into a page or a build.
|
||||
3. **Two sentences on why it is good code.** `rationale` in `manifest.yaml`.
|
||||
Not what the code does — that is what reading it is for.
|
||||
4. **Honest notes.** Where the code is not good, `honest_note` says so. A 2012
|
||||
procedural PHP script presented as elegant would undermine every other claim
|
||||
on the site.
|
||||
|
||||
## The six
|
||||
|
||||
| # | Era | Language | Lines | What it demonstrates |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Flash / ActionScript | AS3 | 36 | class, imports, encapsulation |
|
||||
| 2 | jQuery + PHP | JavaScript | 42 | easing + timeline core |
|
||||
| 3 | jQuery + PHP | PHP | 27 | one encode op + the FFmpeg piping |
|
||||
| 4 | React + TypeScript | TypeScript | 37 | types, interfaces, service layer |
|
||||
| 5 | Embedded C++ | C++ | 28 | interface split from implementation |
|
||||
| 6 | Declarative infrastructure | Nix | 33 | a patch with its own removal condition |
|
||||
|
||||
Era 6 is the most modern and the least common. A short, correct, self-documenting
|
||||
derivation that says when to delete itself is a better infrastructure signal than
|
||||
a thousand lines of configuration.
|
||||
|
||||
## Renderer requirements
|
||||
|
||||
- Syntax highlighting happens **server-side** (chroma), on the server, at build
|
||||
time. The client receives highlighted HTML. No browser-side highlighter.
|
||||
- Line numbers are generated by the template, not embedded in the files.
|
||||
- Files stay in `content/assets/code/` as plain text. They are not inlined into
|
||||
a Go string, so the archive copy and the served copy cannot diverge silently.
|
||||
- Total weight: 7 KB of source, which compresses well because it is code.
|
||||
|
||||
## Honest notes worth reading twice
|
||||
|
||||
- Exhibit 1 contains a latent typo, kept.
|
||||
- Exhibit 3 is 27 lines out of a 3,400-line file, with one marked elision.
|
||||
- Exhibit 4 shows the router rather than the schema it types against.
|
||||
- Exhibit 5 is two files presented as one, because the point is the split.
|
||||
@@ -1,76 +0,0 @@
|
||||
version: 1
|
||||
authored: 2026-10-26
|
||||
total_lines: 212
|
||||
provenance: "private archive, copied 2026-10-26, path withheld"
|
||||
exhibits:
|
||||
- id: history-as3-class
|
||||
order: 1
|
||||
era: "Flash / ActionScript"
|
||||
language: "ActionScript 3"
|
||||
file: 01-actionscript-class.as
|
||||
lines: 36
|
||||
shows: "class, imports, encapsulation"
|
||||
area: history
|
||||
title: "A small class that hides its own state"
|
||||
rationale: "Every field is private and explicitly typed, and the whole public surface is two methods, so a caller cannot push the object into a state it does not understand. The constructor takes what it needs as parameters instead of reaching for it, which is why the same class can drive every siren on the screen without knowing where any of them are."
|
||||
honest_note: "Verbatim, including a latent typo on one private field (this.high rather than this.height). Left as found: the excerpt is evidence, not a portfolio piece."
|
||||
|
||||
- id: history-js-easing-engine
|
||||
order: 2
|
||||
era: "jQuery + PHP"
|
||||
language: JavaScript
|
||||
file: 02-js-easing-core.js
|
||||
lines: 42
|
||||
shows: "easing + timeline core"
|
||||
area: history
|
||||
title: "One curve, used twice"
|
||||
rationale: "A named easing function is evaluated to give each letter its delay, so the spacing of an animation follows the same curve as its motion instead of a guessed linear stagger. Both directions build from one transform object, which is what stops the two halves of the same effect drifting apart over years of edits."
|
||||
honest_note: "Excerpt from a 2,195-line file in which easing curves are used as a spacing distribution rather than only a timing function."
|
||||
|
||||
- id: history-php-media-pipeline
|
||||
order: 3
|
||||
era: "jQuery + PHP"
|
||||
language: PHP
|
||||
file: 03-php-encode-op.php
|
||||
lines: 27
|
||||
shows: "one encode op + the FFmpeg piping"
|
||||
area: history
|
||||
title: "Check the input, then encode in two passes"
|
||||
rationale: "The real MIME type of the input is checked before any command is built, so a wrong file type returns a message instead of a half-written artefact. The encode takes two passes, generating a palette from the clip and then mapping frames onto it, which is the difference between a GIF that holds its colour and one that does not."
|
||||
honest_note: "Row 20 is an elision marker: 34 lines of ImageMagick frame-optimisation options are omitted because they are disabled in the live code. This is 3,400 lines of procedural PHP with global state and interpolated shell commands; what is worth reading is the operational judgement, not the structure."
|
||||
|
||||
- id: history-typescript-service-layer
|
||||
order: 4
|
||||
era: "React + TypeScript"
|
||||
language: TypeScript
|
||||
file: 04-typescript-service-layer.ts
|
||||
lines: 37
|
||||
shows: "types, interfaces, service layer"
|
||||
area: history
|
||||
title: "Validate at the edge, then do one thing"
|
||||
rationale: "Input is checked by a schema at the boundary of each procedure, so the data layer receives a shape it can trust rather than one it hopes for. Each procedure is a thin call on the database context, which keeps the router a description of the API rather than a second copy of the queries."
|
||||
honest_note: "The types come from a Prisma schema in the same project (Device, Telemetry and Command models). The router is shown because it is where the two meet."
|
||||
|
||||
- id: edge-embedded-cpp-interface
|
||||
order: 5
|
||||
era: "Embedded C++"
|
||||
language: C++
|
||||
file: 05-embedded-cpp-interface.hpp
|
||||
lines: 28
|
||||
shows: "interface vs implementation"
|
||||
area: edge
|
||||
title: "The hardware stops at the header"
|
||||
rationale: "The header declares one function and one constant and includes nothing but Arduino.h, so a caller compiles without pulling the ESP32 WiFi stack into its translation unit. Only the implementation file knows about station mode, polling and millisecond timeouts, and it returns a boolean so the caller decides what a failed join means."
|
||||
honest_note: "Header and implementation are shown together, with a comment marking the boundary, because the point of the exhibit is the split rather than either file alone."
|
||||
|
||||
- id: platform-nix-derivation
|
||||
order: 6
|
||||
era: "Declarative infrastructure"
|
||||
language: Nix
|
||||
file: 06-nix-derivation.nix
|
||||
lines: 33
|
||||
shows: "fetchurl + hash, override rationale, removal condition"
|
||||
area: platform
|
||||
title: "A patch that says when to delete itself"
|
||||
rationale: "The file records its own removal condition, delete it once nixpkgs ships 0.10, which is the part of an override that usually goes missing and leaves the next reader guessing. Pinning by fetchurl with a verified hash keeps the build reproducible while the reason for the pin lives in the file rather than in a commit message nobody will read."
|
||||
honest_note: "The whole file, unedited. It is the newest exhibit and one of the shortest, and it is the one that best predicts how this engineer works now."
|
||||
@@ -1,111 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 540" width="960" height="540" role="img" aria-labelledby="ttl dsc">
|
||||
<title id="ttl">Homelab topology</title>
|
||||
<desc id="dsc">Three hosts with their roles and the trust boundaries between them: public internet reaches Caddy on .35, which forwards over the LAN to services bound to 127.0.0.1 on .13. .27 is build and dev with no public listeners.</desc>
|
||||
<style>
|
||||
text{font-family:ui-sans-serif,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;fill:#1b222b;font-size:15px}
|
||||
.hd{font-size:15px;font-weight:600}
|
||||
.sub{font-size:13px;fill:#5b6673}
|
||||
.cap{font-size:12px;fill:#5b6673;letter-spacing:.06em;text-transform:uppercase;font-weight:600}
|
||||
.big{font-size:19px;font-weight:600}
|
||||
.node{fill:#f4f6f8;stroke:#c3ccd6;stroke-width:1.4}
|
||||
.node-a{fill:#e9f1fd;stroke:#1a6bc4;stroke-width:1.4}
|
||||
.node-t{fill:#e6f4f2;stroke:#0f766e;stroke-width:1.4}
|
||||
.node-w{fill:#fdf1e3;stroke:#b45309;stroke-width:1.4}
|
||||
.zone{fill:none;stroke:#aab4c0;stroke-width:1.4;stroke-dasharray:7 6}
|
||||
.rule{stroke:#d0d7df;stroke-width:1}
|
||||
.ln{stroke:#6b7684;stroke-width:1.6;fill:none}
|
||||
.ln.d{stroke-dasharray:6 5}
|
||||
.ln.a{stroke:#1a6bc4}
|
||||
.ln.t{stroke:#0f766e}
|
||||
.ln.w{stroke:#b45309}
|
||||
.mN{fill:#6b7684}
|
||||
.mA{fill:#1a6bc4}
|
||||
.mT{fill:#0f766e}
|
||||
.mW{fill:#b45309}
|
||||
.dot{fill:#6b7684}
|
||||
.tick{stroke:#aab4c0;stroke-width:1.2}
|
||||
@media (prefers-color-scheme: dark){
|
||||
text{fill:#e7edf4}
|
||||
.sub,.cap{fill:#98a4b3}
|
||||
.node{fill:#182028;stroke:#35424f}
|
||||
.node-a{fill:#11253d;stroke:#5aa2f0}
|
||||
.node-t{fill:#0d2624;stroke:#38b3a6}
|
||||
.node-w{fill:#2a1d10;stroke:#d98b3f}
|
||||
.zone{stroke:#43525f}
|
||||
.rule{stroke:#2c3743}
|
||||
.ln{stroke:#8496a6}
|
||||
.ln.a{stroke:#5aa2f0}
|
||||
.ln.t{stroke:#38b3a6}
|
||||
.ln.w{stroke:#d98b3f}
|
||||
.mN{fill:#8496a6}
|
||||
.mA{fill:#5aa2f0}
|
||||
.mT{fill:#38b3a6}
|
||||
.mW{fill:#d98b3f}
|
||||
.dot{fill:#8496a6}
|
||||
.tick{stroke:#43525f}
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="aN" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mN" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aA" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mA" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aT" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mT" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aW" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mW" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
</defs>
|
||||
<text x="24" y="40" class="big">Homelab topology — hosts, roles, trust boundaries</text>
|
||||
<rect x="24" y="74" width="150" height="152" rx="10" class="zone"/>
|
||||
<text x="36" y="94" class="cap">public internet</text>
|
||||
<rect x="38" y="110" width="122" height="62" rx="7" class="node "/>
|
||||
<text x="99.0" y="139.0" class="hd" text-anchor="middle">Visitors</text>
|
||||
<text x="99.0" y="157.0" class="sub" text-anchor="middle">no host access</text>
|
||||
<path d="M174,148 L200,148" class="ln a" marker-end="url(#aA)"/>
|
||||
<rect x="206" y="74" width="228" height="300" rx="7" class="node "/>
|
||||
<text x="220" y="100" class="hd">.35 sam-Ubuntu1</text>
|
||||
<text x="220" y="120" class="sub">public edge · 2 CPUs · 14 GiB</text>
|
||||
<path d="M206,132 L434,132" class="rule"/>
|
||||
<rect x="220" y="146" width="200" height="48" rx="7" class="node node-a"/>
|
||||
<text x="320.0" y="168.0" class="hd" text-anchor="middle">Caddy :443</text>
|
||||
<text x="320.0" y="186.0" class="sub" text-anchor="middle">the only public listener</text>
|
||||
<rect x="220" y="208" width="200" height="44" rx="7" class="node "/>
|
||||
<text x="320.0" y="235.0" class="hd" text-anchor="middle">Gitea · Immich · Wiki.js</text>
|
||||
<rect x="220" y="262" width="200" height="44" rx="7" class="node "/>
|
||||
<text x="320.0" y="289.0" class="hd" text-anchor="middle">Nextcloud · Penpot</text>
|
||||
<text x="220" y="330" class="sub">terminates 443 for</text>
|
||||
<text x="220" y="348" class="sub">*.lab.audasmedia.com.au</text>
|
||||
<path d="M434,170 L462,170" class="ln a" marker-end="url(#aA)"/>
|
||||
<rect x="468" y="74" width="268" height="366" rx="7" class="node "/>
|
||||
<text x="482" y="100" class="hd">.13 nixos-desktop</text>
|
||||
<text x="482" y="120" class="sub">runtime · 12 CPUs · 2.3 GiB free</text>
|
||||
<path d="M468,132 L736,132" class="rule"/>
|
||||
<rect x="482" y="146" width="240" height="46" rx="7" class="node node-a"/>
|
||||
<text x="602.0" y="167.0" class="hd" text-anchor="middle">Go app :8096</text>
|
||||
<text x="602.0" y="185.0" class="sub" text-anchor="middle">templ · HTMX · SSE · SQLite</text>
|
||||
<rect x="482" y="200" width="240" height="42" rx="7" class="node "/>
|
||||
<text x="602.0" y="226.0" class="hd" text-anchor="middle">Qdrant :6333 / :6334</text>
|
||||
<rect x="482" y="250" width="240" height="42" rx="7" class="node "/>
|
||||
<text x="602.0" y="269.0" class="hd" text-anchor="middle">Prefect worker</text>
|
||||
<text x="602.0" y="287.0" class="sub" text-anchor="middle">content/ → chunk → embed</text>
|
||||
<rect x="482" y="300" width="240" height="42" rx="7" class="node node-t"/>
|
||||
<text x="602.0" y="319.0" class="hd" text-anchor="middle">OmniRoute</text>
|
||||
<text x="602.0" y="337.0" class="sub" text-anchor="middle">existing · the only model gateway</text>
|
||||
<rect x="482" y="350" width="240" height="42" rx="7" class="node "/>
|
||||
<text x="602.0" y="369.0" class="hd" text-anchor="middle">Whisper · Piper</text>
|
||||
<text x="602.0" y="387.0" class="sub" text-anchor="middle">speech in, speech out</text>
|
||||
<text x="482" y="414" class="sub">every listener on 127.0.0.1</text>
|
||||
<path d="M736,200 L764,200" class="ln d " marker-end="url(#aN)"/>
|
||||
<text x="750.0" y="193.0" class="sub" text-anchor="middle">LAN</text>
|
||||
<rect x="766" y="74" width="170" height="220" rx="7" class="node "/>
|
||||
<text x="780" y="100" class="hd">.27</text>
|
||||
<text x="780" y="120" class="sub">sam-4screen-desktop</text>
|
||||
<path d="M766,132 L936,132" class="rule"/>
|
||||
<rect x="780" y="146" width="142" height="44" rx="7" class="node "/>
|
||||
<text x="851.0" y="173.0" class="hd" text-anchor="middle">content/ + source</text>
|
||||
<rect x="780" y="200" width="142" height="44" rx="7" class="node "/>
|
||||
<text x="851.0" y="227.0" class="hd" text-anchor="middle">agent fleet</text>
|
||||
<text x="780" y="268" class="sub">build / dev · no public</text>
|
||||
<text x="780" y="286" class="sub">listeners</text>
|
||||
<rect x="206" y="396" width="730" height="62" rx="10" class="zone"/>
|
||||
<text x="218" y="416" class="cap">trust boundary — outside this boundary nothing is reachable from the internet</text>
|
||||
<text x="220" y="438" class="sub">Public → Caddy :443 on .35 → LAN → 127.0.0.1 services on .13. New footprint ≤ 1.0 GiB · ≤ 4 containers.</text>
|
||||
<text x="24" y="486" class="sub">Solid = network path. Dashed = trust boundary. Ports are those the app actually binds.</text>
|
||||
<text x="936" y="486" class="sub" text-anchor="end">measured 2026-10-17 · reference/stack.md</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 6.9 KiB |
@@ -1,98 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 540" width="960" height="540" role="img" aria-labelledby="ttl dsc">
|
||||
<title id="ttl">Agent architecture</title>
|
||||
<desc id="dsc">The operator reaches a fleet of agent sessions through one inbox. Paperclip is the control plane and delegates to peer projects, which answer the Hub read-only. Every model call in the estate passes through one gateway, OmniRoute.</desc>
|
||||
<style>
|
||||
text{font-family:ui-sans-serif,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;fill:#1b222b;font-size:15px}
|
||||
.hd{font-size:15px;font-weight:600}
|
||||
.sub{font-size:13px;fill:#5b6673}
|
||||
.cap{font-size:12px;fill:#5b6673;letter-spacing:.06em;text-transform:uppercase;font-weight:600}
|
||||
.big{font-size:19px;font-weight:600}
|
||||
.node{fill:#f4f6f8;stroke:#c3ccd6;stroke-width:1.4}
|
||||
.node-a{fill:#e9f1fd;stroke:#1a6bc4;stroke-width:1.4}
|
||||
.node-t{fill:#e6f4f2;stroke:#0f766e;stroke-width:1.4}
|
||||
.node-w{fill:#fdf1e3;stroke:#b45309;stroke-width:1.4}
|
||||
.zone{fill:none;stroke:#aab4c0;stroke-width:1.4;stroke-dasharray:7 6}
|
||||
.rule{stroke:#d0d7df;stroke-width:1}
|
||||
.ln{stroke:#6b7684;stroke-width:1.6;fill:none}
|
||||
.ln.d{stroke-dasharray:6 5}
|
||||
.ln.a{stroke:#1a6bc4}
|
||||
.ln.t{stroke:#0f766e}
|
||||
.ln.w{stroke:#b45309}
|
||||
.mN{fill:#6b7684}
|
||||
.mA{fill:#1a6bc4}
|
||||
.mT{fill:#0f766e}
|
||||
.mW{fill:#b45309}
|
||||
.dot{fill:#6b7684}
|
||||
.tick{stroke:#aab4c0;stroke-width:1.2}
|
||||
@media (prefers-color-scheme: dark){
|
||||
text{fill:#e7edf4}
|
||||
.sub,.cap{fill:#98a4b3}
|
||||
.node{fill:#182028;stroke:#35424f}
|
||||
.node-a{fill:#11253d;stroke:#5aa2f0}
|
||||
.node-t{fill:#0d2624;stroke:#38b3a6}
|
||||
.node-w{fill:#2a1d10;stroke:#d98b3f}
|
||||
.zone{stroke:#43525f}
|
||||
.rule{stroke:#2c3743}
|
||||
.ln{stroke:#8496a6}
|
||||
.ln.a{stroke:#5aa2f0}
|
||||
.ln.t{stroke:#38b3a6}
|
||||
.ln.w{stroke:#d98b3f}
|
||||
.mN{fill:#8496a6}
|
||||
.mA{fill:#5aa2f0}
|
||||
.mT{fill:#38b3a6}
|
||||
.mW{fill:#d98b3f}
|
||||
.dot{fill:#8496a6}
|
||||
.tick{stroke:#43525f}
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="aN" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mN" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aA" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mA" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aT" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mT" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aW" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mW" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
</defs>
|
||||
<text x="24" y="40" class="big">Agent architecture — operator, control plane, peers, one gateway</text>
|
||||
<rect x="360" y="62" width="240" height="56" rx="7" class="node "/>
|
||||
<text x="480.0" y="88.0" class="hd" text-anchor="middle">Operator</text>
|
||||
<text x="480.0" y="106.0" class="sub" text-anchor="middle">one inbox for runs and approvals</text>
|
||||
<path d="M480,118 L480,146" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="330" y="146" width="300" height="62" rx="7" class="node "/>
|
||||
<text x="480.0" y="175.0" class="hd" text-anchor="middle">TUIOS fleet</text>
|
||||
<text x="480.0" y="193.0" class="sub" text-anchor="middle">four hosts · survive disconnect</text>
|
||||
<path d="M400,208 L300,236" class="ln " marker-end="url(#aN)"/>
|
||||
<path d="M480,208 L480,236" class="ln " marker-end="url(#aN)"/>
|
||||
<path d="M560,208 L680,236" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="40" y="236" width="250" height="132" rx="7" class="node "/>
|
||||
<text x="54" y="262" class="hd">Hub</text>
|
||||
<text x="54" y="282" class="sub">house entry point for every project</text>
|
||||
<path d="M40,292 L290,292" class="rule"/>
|
||||
<text x="54" y="314" class="sub">/capabilities · /ask (read-only)</text>
|
||||
<text x="54" y="334" class="sub">/notify · Apprise delivery</text>
|
||||
<text x="54" y="356" class="sub">no write path to peers</text>
|
||||
<rect x="355" y="236" width="250" height="132" rx="7" class="node "/>
|
||||
<text x="369" y="262" class="hd">Paperclip</text>
|
||||
<text x="369" y="282" class="sub">control plane · the system of record</text>
|
||||
<path d="M355,292 L605,292" class="rule"/>
|
||||
<text x="369" y="314" class="sub">issues · runs · approvals</text>
|
||||
<text x="369" y="334" class="sub">agent identities · budgets</text>
|
||||
<text x="369" y="356" class="sub">routines · blocked-by edges</text>
|
||||
<rect x="670" y="236" width="250" height="132" rx="7" class="node "/>
|
||||
<text x="684" y="262" class="hd">Peer data projects</text>
|
||||
<text x="684" y="282" class="sub">own service, own port, own folder</text>
|
||||
<path d="M670,292 L920,292" class="rule"/>
|
||||
<text x="684" y="314" class="sub">resume (this site)</text>
|
||||
<text x="684" y="334" class="sub">compass_nhs · email_agent</text>
|
||||
<text x="684" y="356" class="sub">IoT estate · signage</text>
|
||||
<path d="M605,302 L670,302" class="ln " marker-end="url(#aN)"/>
|
||||
<text x="637.5" y="295.0" class="sub" text-anchor="middle">delegates</text>
|
||||
<path d="M290,320 L355,320" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="200" y="404" width="560" height="76" rx="7" class="node node-t"/>
|
||||
<text x="480" y="432" class="hd" text-anchor="middle">OmniRoute — the only model gateway</text>
|
||||
<text x="480" y="454" class="sub" text-anchor="middle">fast · mid · escalation tier | token caps | monthly ceiling | trace log</text>
|
||||
<path d="M480,368 L480,404" class="ln t" marker-end="url(#aT)"/>
|
||||
<path d="M120,368 L260,404" class="ln t" marker-end="url(#aT)"/>
|
||||
<path d="M795,368 L700,404" class="ln t" marker-end="url(#aT)"/>
|
||||
<text x="480" y="502" class="sub" text-anchor="middle">OpenRouter · DeepSeek · Gemini · MiniMax · Qwen</text>
|
||||
<text x="480" y="520" class="sub" text-anchor="middle">reachable only through the gateway</text>
|
||||
<text x="24" y="498" class="sub">Enforced by configuration, not convention: no service holds a provider key, so none can bypass the gateway.</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 5.9 KiB |
@@ -1,106 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 540" width="960" height="540" role="img" aria-labelledby="ttl dsc">
|
||||
<title id="ttl">Retrieval pipeline</title>
|
||||
<desc id="dsc">Left, offline ingest: content files are validated, chunked with metadata, embedded and upserted into Qdrant, while the same files render to static pages with no AI in the path. Right, per-visitor query: guardrail, query graph, filtered retrieval, then a streamed answer with citations.</desc>
|
||||
<style>
|
||||
text{font-family:ui-sans-serif,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;fill:#1b222b;font-size:15px}
|
||||
.hd{font-size:15px;font-weight:600}
|
||||
.sub{font-size:13px;fill:#5b6673}
|
||||
.cap{font-size:12px;fill:#5b6673;letter-spacing:.06em;text-transform:uppercase;font-weight:600}
|
||||
.big{font-size:19px;font-weight:600}
|
||||
.node{fill:#f4f6f8;stroke:#c3ccd6;stroke-width:1.4}
|
||||
.node-a{fill:#e9f1fd;stroke:#1a6bc4;stroke-width:1.4}
|
||||
.node-t{fill:#e6f4f2;stroke:#0f766e;stroke-width:1.4}
|
||||
.node-w{fill:#fdf1e3;stroke:#b45309;stroke-width:1.4}
|
||||
.zone{fill:none;stroke:#aab4c0;stroke-width:1.4;stroke-dasharray:7 6}
|
||||
.rule{stroke:#d0d7df;stroke-width:1}
|
||||
.ln{stroke:#6b7684;stroke-width:1.6;fill:none}
|
||||
.ln.d{stroke-dasharray:6 5}
|
||||
.ln.a{stroke:#1a6bc4}
|
||||
.ln.t{stroke:#0f766e}
|
||||
.ln.w{stroke:#b45309}
|
||||
.mN{fill:#6b7684}
|
||||
.mA{fill:#1a6bc4}
|
||||
.mT{fill:#0f766e}
|
||||
.mW{fill:#b45309}
|
||||
.dot{fill:#6b7684}
|
||||
.tick{stroke:#aab4c0;stroke-width:1.2}
|
||||
@media (prefers-color-scheme: dark){
|
||||
text{fill:#e7edf4}
|
||||
.sub,.cap{fill:#98a4b3}
|
||||
.node{fill:#182028;stroke:#35424f}
|
||||
.node-a{fill:#11253d;stroke:#5aa2f0}
|
||||
.node-t{fill:#0d2624;stroke:#38b3a6}
|
||||
.node-w{fill:#2a1d10;stroke:#d98b3f}
|
||||
.zone{stroke:#43525f}
|
||||
.rule{stroke:#2c3743}
|
||||
.ln{stroke:#8496a6}
|
||||
.ln.a{stroke:#5aa2f0}
|
||||
.ln.t{stroke:#38b3a6}
|
||||
.ln.w{stroke:#d98b3f}
|
||||
.mN{fill:#8496a6}
|
||||
.mA{fill:#5aa2f0}
|
||||
.mT{fill:#38b3a6}
|
||||
.mW{fill:#d98b3f}
|
||||
.dot{fill:#8496a6}
|
||||
.tick{stroke:#43525f}
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="aN" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mN" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aA" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mA" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aT" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mT" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aW" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mW" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
</defs>
|
||||
<text x="24" y="40" class="big">Retrieval pipeline — ingest and query, side by side</text>
|
||||
<path d="M480,64 L480,470" class="zone"/>
|
||||
<text x="40" y="76" class="cap">ingest — runs offline</text>
|
||||
<text x="504" y="76" class="cap">query — runs per visitor</text>
|
||||
<rect x="40" y="92" width="250" height="48" rx="7" class="node "/>
|
||||
<text x="165.0" y="114.0" class="hd" text-anchor="middle">content/ in git</text>
|
||||
<text x="165.0" y="132.0" class="sub" text-anchor="middle">Markdown + front-matter + assets</text>
|
||||
<path d="M165.0,140 L165.0,164" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="40" y="164" width="250" height="48" rx="7" class="node "/>
|
||||
<text x="165.0" y="186.0" class="hd" text-anchor="middle">validate front-matter</text>
|
||||
<text x="165.0" y="204.0" class="sub" text-anchor="middle">schema is frozen; build fails closed</text>
|
||||
<path d="M165.0,212 L165.0,236" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="40" y="236" width="250" height="48" rx="7" class="node "/>
|
||||
<text x="165.0" y="258.0" class="hd" text-anchor="middle">chunk + tag metadata</text>
|
||||
<text x="165.0" y="276.0" class="sub" text-anchor="middle">area · level · confidence · period</text>
|
||||
<path d="M165.0,284 L165.0,308" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="40" y="308" width="250" height="48" rx="7" class="node node-t"/>
|
||||
<text x="165.0" y="330.0" class="hd" text-anchor="middle">embed via OmniRoute</text>
|
||||
<text x="165.0" y="348.0" class="sub" text-anchor="middle">one gateway, one trace log</text>
|
||||
<path d="M165.0,356 L165.0,380" class="ln t" marker-end="url(#aT)"/>
|
||||
<rect x="180" y="380" width="300" height="56" rx="7" class="node node-t"/>
|
||||
<text x="330.0" y="406.0" class="hd" text-anchor="middle">Qdrant — payload-filtered store</text>
|
||||
<text x="330.0" y="424.0" class="sub" text-anchor="middle">one point per chunk</text>
|
||||
<rect x="40" y="400" width="116" height="64" rx="7" class="node node-a"/>
|
||||
<text x="98.0" y="430.0" class="hd" text-anchor="middle">templ build</text>
|
||||
<text x="98.0" y="448.0" class="sub" text-anchor="middle">Mode A pages</text>
|
||||
<path d="M98,212 L98,400" class="ln a" marker-end="url(#aA)"/>
|
||||
<text x="98.0" y="299.0" class="sub" text-anchor="middle">same file</text>
|
||||
<text x="106" y="300" class="sub">no AI in this path</text>
|
||||
<text x="106" y="318" class="sub">(acceptance test A1)</text>
|
||||
<rect x="504" y="92" width="250" height="48" rx="7" class="node "/>
|
||||
<text x="629.0" y="114.0" class="hd" text-anchor="middle">visitor prompt</text>
|
||||
<text x="629.0" y="132.0" class="sub" text-anchor="middle">plain English, untrusted</text>
|
||||
<path d="M629.0,140 L629.0,164" class="ln w" marker-end="url(#aW)"/>
|
||||
<rect x="504" y="164" width="250" height="48" rx="7" class="node node-w"/>
|
||||
<text x="629.0" y="186.0" class="hd" text-anchor="middle">guardrail: classify → allow / deny</text>
|
||||
<text x="629.0" y="204.0" class="sub" text-anchor="middle">fails closed, no tools, no write path</text>
|
||||
<path d="M629.0,212 L629.0,236" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="504" y="236" width="250" height="48" rx="7" class="node "/>
|
||||
<text x="629.0" y="258.0" class="hd" text-anchor="middle">LangGraph query graph</text>
|
||||
<text x="629.0" y="276.0" class="sub" text-anchor="middle">route → retrieve → cite → compose</text>
|
||||
<path d="M629.0,284 L629.0,308" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="504" y="308" width="250" height="48" rx="7" class="node "/>
|
||||
<text x="629.0" y="330.0" class="hd" text-anchor="middle">retrieve: top-k, filtered by area</text>
|
||||
<text x="629.0" y="348.0" class="sub" text-anchor="middle">filtered before it is searched</text>
|
||||
<path d="M629.0,356 L629.0,380" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="504" y="380" width="250" height="56" rx="7" class="node node-a"/>
|
||||
<text x="629.0" y="406.0" class="hd" text-anchor="middle">SSE stream + citations</text>
|
||||
<text x="629.0" y="424.0" class="sub" text-anchor="middle">first token <4 s · every claim cited</text>
|
||||
<path d="M480,408 L504,408" class="ln " marker-end="url(#aN)"/>
|
||||
<text x="24" y="492" class="sub">Ingest and query share one store and one schema.</text>
|
||||
<text x="24" y="512" class="sub">Static pages and corpus are built from the same files, so the two modes cannot drift.</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 6.8 KiB |
@@ -1,111 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 540" width="960" height="540" role="img" aria-labelledby="ttl dsc">
|
||||
<title id="ttl">Edge architecture</title>
|
||||
<desc id="dsc">Constrained devices publish over LoRa, Zigbee or WiFi to a single MQTT broker, which feeds one automation layer that drives actuation. Retained configuration, telemetry and last-will presence let devices recover unattended.</desc>
|
||||
<style>
|
||||
text{font-family:ui-sans-serif,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;fill:#1b222b;font-size:15px}
|
||||
.hd{font-size:15px;font-weight:600}
|
||||
.sub{font-size:13px;fill:#5b6673}
|
||||
.cap{font-size:12px;fill:#5b6673;letter-spacing:.06em;text-transform:uppercase;font-weight:600}
|
||||
.big{font-size:19px;font-weight:600}
|
||||
.node{fill:#f4f6f8;stroke:#c3ccd6;stroke-width:1.4}
|
||||
.node-a{fill:#e9f1fd;stroke:#1a6bc4;stroke-width:1.4}
|
||||
.node-t{fill:#e6f4f2;stroke:#0f766e;stroke-width:1.4}
|
||||
.node-w{fill:#fdf1e3;stroke:#b45309;stroke-width:1.4}
|
||||
.zone{fill:none;stroke:#aab4c0;stroke-width:1.4;stroke-dasharray:7 6}
|
||||
.rule{stroke:#d0d7df;stroke-width:1}
|
||||
.ln{stroke:#6b7684;stroke-width:1.6;fill:none}
|
||||
.ln.d{stroke-dasharray:6 5}
|
||||
.ln.a{stroke:#1a6bc4}
|
||||
.ln.t{stroke:#0f766e}
|
||||
.ln.w{stroke:#b45309}
|
||||
.mN{fill:#6b7684}
|
||||
.mA{fill:#1a6bc4}
|
||||
.mT{fill:#0f766e}
|
||||
.mW{fill:#b45309}
|
||||
.dot{fill:#6b7684}
|
||||
.tick{stroke:#aab4c0;stroke-width:1.2}
|
||||
@media (prefers-color-scheme: dark){
|
||||
text{fill:#e7edf4}
|
||||
.sub,.cap{fill:#98a4b3}
|
||||
.node{fill:#182028;stroke:#35424f}
|
||||
.node-a{fill:#11253d;stroke:#5aa2f0}
|
||||
.node-t{fill:#0d2624;stroke:#38b3a6}
|
||||
.node-w{fill:#2a1d10;stroke:#d98b3f}
|
||||
.zone{stroke:#43525f}
|
||||
.rule{stroke:#2c3743}
|
||||
.ln{stroke:#8496a6}
|
||||
.ln.a{stroke:#5aa2f0}
|
||||
.ln.t{stroke:#38b3a6}
|
||||
.ln.w{stroke:#d98b3f}
|
||||
.mN{fill:#8496a6}
|
||||
.mA{fill:#5aa2f0}
|
||||
.mT{fill:#38b3a6}
|
||||
.mW{fill:#d98b3f}
|
||||
.dot{fill:#8496a6}
|
||||
.tick{stroke:#43525f}
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="aN" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mN" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aA" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mA" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aT" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mT" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aW" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mW" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
</defs>
|
||||
<text x="24" y="40" class="big">Edge architecture — devices, one broker, one automation layer</text>
|
||||
<rect x="24" y="74" width="220" height="236" rx="7" class="node "/>
|
||||
<text x="38" y="100" class="hd">Devices</text>
|
||||
<text x="38" y="120" class="sub">20+ nodes · 9 zones</text>
|
||||
<path d="M24,132 L244,132" class="rule"/>
|
||||
<rect x="38" y="146" width="192" height="40" rx="7" class="node "/>
|
||||
<text x="134.0" y="164.0" class="hd" text-anchor="middle">10 × ESP32</text>
|
||||
<text x="134.0" y="182.0" class="sub" text-anchor="middle">sensors, relays, buttons</text>
|
||||
<rect x="38" y="194" width="192" height="40" rx="7" class="node "/>
|
||||
<text x="134.0" y="212.0" class="hd" text-anchor="middle">2 × Zigbee routers</text>
|
||||
<text x="134.0" y="230.0" class="sub" text-anchor="middle">hand-built</text>
|
||||
<rect x="38" y="242" width="192" height="40" rx="7" class="node "/>
|
||||
<text x="134.0" y="260.0" class="hd" text-anchor="middle">8 × Pi audio endpoints</text>
|
||||
<text x="134.0" y="278.0" class="sub" text-anchor="middle">Snapcast clients</text>
|
||||
<text x="38" y="300" class="sub">radio: LoRa · Zigbee · WiFi</text>
|
||||
<path d="M244,190 L288,190" class="ln a" marker-end="url(#aA)"/>
|
||||
<rect x="288" y="74" width="210" height="236" rx="7" class="node "/>
|
||||
<text x="302" y="100" class="hd">MQTT broker</text>
|
||||
<text x="302" y="120" class="sub">the only shared state</text>
|
||||
<path d="M288,132 L498,132" class="rule"/>
|
||||
<rect x="302" y="146" width="182" height="40" rx="7" class="node "/>
|
||||
<text x="393.0" y="171.0" class="hd" text-anchor="middle">retained config</text>
|
||||
<rect x="302" y="194" width="182" height="40" rx="7" class="node "/>
|
||||
<text x="393.0" y="219.0" class="hd" text-anchor="middle">status · telemetry</text>
|
||||
<rect x="302" y="242" width="182" height="40" rx="7" class="node "/>
|
||||
<text x="393.0" y="260.0" class="hd" text-anchor="middle">LWT presence</text>
|
||||
<text x="393.0" y="278.0" class="sub" text-anchor="middle">last will = offline</text>
|
||||
<text x="302" y="300" class="sub">reconnect + re-subscribe</text>
|
||||
<path d="M498,190 L542,190" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="542" y="74" width="194" height="236" rx="7" class="node "/>
|
||||
<text x="556" y="100" class="hd">Automation</text>
|
||||
<text x="556" y="120" class="sub">20 automations · 9 zones</text>
|
||||
<path d="M542,132 L736,132" class="rule"/>
|
||||
<text x="556" y="156" class="sub">trigger → rule → action</text>
|
||||
<text x="556" y="182" class="sub">9 lighting zones</text>
|
||||
<text x="556" y="208" class="sub">auditable in one place</text>
|
||||
<text x="556" y="240" class="sub">rules live in config,</text>
|
||||
<text x="556" y="258" class="sub">not in firmware</text>
|
||||
<path d="M736,190 L780,190" class="ln t" marker-end="url(#aT)"/>
|
||||
<rect x="780" y="74" width="156" height="236" rx="7" class="node "/>
|
||||
<text x="794" y="100" class="hd">Actuation</text>
|
||||
<path d="M780,132 L936,132" class="rule"/>
|
||||
<text x="794" y="158" class="sub">lights · relays</text>
|
||||
<text x="794" y="184" class="sub">speech out</text>
|
||||
<text x="794" y="210" class="sub">displays</text>
|
||||
<text x="794" y="236" class="sub">doors · gates</text>
|
||||
<rect x="24" y="332" width="442" height="84" rx="7" class="node node-t"/>
|
||||
<text x="38" y="358" class="hd">Power and duty cycle</text>
|
||||
<text x="38" y="380" class="sub">solar where there is no mains · deep sleep between reports</text>
|
||||
<text x="38" y="400" class="sub">the budget that gets designed first, not last</text>
|
||||
<rect x="494" y="332" width="442" height="84" rx="7" class="node node-w"/>
|
||||
<text x="508" y="358" class="hd">Failures kept in the record</text>
|
||||
<text x="508" y="380" class="sub">lizard feeder destroyed by the lizard · cameras postponed</text>
|
||||
<text x="508" y="400" class="sub">doorbell being rebuilt with a ToF sensor</text>
|
||||
<text x="24" y="452" class="sub">Same shape at every size: device → one broker → one automation layer → actuation.</text>
|
||||
<text x="24" y="472" class="sub">Devices never talk to each other, so a node that returns from a power cut re-announces itself and nothing else changes.</text>
|
||||
<text x="936" y="500" class="sub" text-anchor="end">20+ nodes = 10 ESP32 · 2 Zigbee routers · 8 Pi · reference/content-map.md</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 6.7 KiB |
@@ -1,111 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 540" width="960" height="540" role="img" aria-labelledby="ttl dsc">
|
||||
<title id="ttl">Distributed audio</title>
|
||||
<desc id="dsc">Music and speech sources feed one Snapcast server, which streams one clock-synchronised mix to eight Pi endpoints. Voice announcements arrive as a priority lane: duck the music, speak, restore.</desc>
|
||||
<style>
|
||||
text{font-family:ui-sans-serif,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;fill:#1b222b;font-size:15px}
|
||||
.hd{font-size:15px;font-weight:600}
|
||||
.sub{font-size:13px;fill:#5b6673}
|
||||
.cap{font-size:12px;fill:#5b6673;letter-spacing:.06em;text-transform:uppercase;font-weight:600}
|
||||
.big{font-size:19px;font-weight:600}
|
||||
.node{fill:#f4f6f8;stroke:#c3ccd6;stroke-width:1.4}
|
||||
.node-a{fill:#e9f1fd;stroke:#1a6bc4;stroke-width:1.4}
|
||||
.node-t{fill:#e6f4f2;stroke:#0f766e;stroke-width:1.4}
|
||||
.node-w{fill:#fdf1e3;stroke:#b45309;stroke-width:1.4}
|
||||
.zone{fill:none;stroke:#aab4c0;stroke-width:1.4;stroke-dasharray:7 6}
|
||||
.rule{stroke:#d0d7df;stroke-width:1}
|
||||
.ln{stroke:#6b7684;stroke-width:1.6;fill:none}
|
||||
.ln.d{stroke-dasharray:6 5}
|
||||
.ln.a{stroke:#1a6bc4}
|
||||
.ln.t{stroke:#0f766e}
|
||||
.ln.w{stroke:#b45309}
|
||||
.mN{fill:#6b7684}
|
||||
.mA{fill:#1a6bc4}
|
||||
.mT{fill:#0f766e}
|
||||
.mW{fill:#b45309}
|
||||
.dot{fill:#6b7684}
|
||||
.tick{stroke:#aab4c0;stroke-width:1.2}
|
||||
@media (prefers-color-scheme: dark){
|
||||
text{fill:#e7edf4}
|
||||
.sub,.cap{fill:#98a4b3}
|
||||
.node{fill:#182028;stroke:#35424f}
|
||||
.node-a{fill:#11253d;stroke:#5aa2f0}
|
||||
.node-t{fill:#0d2624;stroke:#38b3a6}
|
||||
.node-w{fill:#2a1d10;stroke:#d98b3f}
|
||||
.zone{stroke:#43525f}
|
||||
.rule{stroke:#2c3743}
|
||||
.ln{stroke:#8496a6}
|
||||
.ln.a{stroke:#5aa2f0}
|
||||
.ln.t{stroke:#38b3a6}
|
||||
.ln.w{stroke:#d98b3f}
|
||||
.mN{fill:#8496a6}
|
||||
.mA{fill:#5aa2f0}
|
||||
.mT{fill:#38b3a6}
|
||||
.mW{fill:#d98b3f}
|
||||
.dot{fill:#8496a6}
|
||||
.tick{stroke:#43525f}
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="aN" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mN" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aA" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mA" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aT" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mT" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aW" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mW" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
</defs>
|
||||
<text x="24" y="40" class="big">Distributed audio — sources, synchronous endpoints, voice injection</text>
|
||||
<rect x="24" y="92" width="200" height="120" rx="7" class="node "/>
|
||||
<text x="38" y="118" class="hd">Sources</text>
|
||||
<path d="M24,130 L224,130" class="rule"/>
|
||||
<text x="38" y="156" class="sub">Spotify → Mopidy</text>
|
||||
<text x="38" y="180" class="sub">Piper / MiniMax TTS</text>
|
||||
<text x="38" y="204" class="sub">announcements</text>
|
||||
<path d="M224,152 L268,152" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="268" y="92" width="210" height="120" rx="7" class="node "/>
|
||||
<text x="282" y="118" class="hd">Snapcast server</text>
|
||||
<text x="282" y="138" class="sub">one clock, one stream</text>
|
||||
<path d="M268,150 L478,150" class="rule"/>
|
||||
<text x="282" y="176" class="sub">per-client latency offset</text>
|
||||
<text x="282" y="198" class="sub">new clients join mid-stream</text>
|
||||
<path d="M478,152 L522,152" class="ln t" marker-end="url(#aT)"/>
|
||||
<rect x="522" y="74" width="414" height="156" rx="7" class="node "/>
|
||||
<text x="536" y="100" class="hd">8 endpoints — Pi 2W</text>
|
||||
<text x="536" y="120" class="sub">one stream, eight speakers, in sync</text>
|
||||
<path d="M522,132 L936,132" class="rule"/>
|
||||
<rect x="536" y="146" width="42" height="42" rx="7" class="node "/>
|
||||
<text x="557.0" y="172.0" class="hd" text-anchor="middle">1</text>
|
||||
<rect x="586" y="146" width="42" height="42" rx="7" class="node "/>
|
||||
<text x="607.0" y="172.0" class="hd" text-anchor="middle">2</text>
|
||||
<rect x="636" y="146" width="42" height="42" rx="7" class="node "/>
|
||||
<text x="657.0" y="172.0" class="hd" text-anchor="middle">3</text>
|
||||
<rect x="686" y="146" width="42" height="42" rx="7" class="node "/>
|
||||
<text x="707.0" y="172.0" class="hd" text-anchor="middle">4</text>
|
||||
<rect x="736" y="146" width="42" height="42" rx="7" class="node "/>
|
||||
<text x="757.0" y="172.0" class="hd" text-anchor="middle">5</text>
|
||||
<rect x="786" y="146" width="42" height="42" rx="7" class="node "/>
|
||||
<text x="807.0" y="172.0" class="hd" text-anchor="middle">6</text>
|
||||
<rect x="836" y="146" width="42" height="42" rx="7" class="node "/>
|
||||
<text x="857.0" y="172.0" class="hd" text-anchor="middle">7</text>
|
||||
<rect x="886" y="146" width="42" height="42" rx="7" class="node "/>
|
||||
<text x="907.0" y="172.0" class="hd" text-anchor="middle">8</text>
|
||||
<text x="536" y="214" class="sub">music in one room does not arrive in another room late</text>
|
||||
<rect x="24" y="262" width="470" height="152" rx="7" class="node node-t"/>
|
||||
<text x="38" y="288" class="hd">Voice injection — the part that is not just streaming</text>
|
||||
<rect x="38" y="304" width="136" height="38" rx="7" class="node "/>
|
||||
<text x="106.0" y="328.0" class="hd" text-anchor="middle">TTS audio in</text>
|
||||
<path d="M174,323 L200,323" class="ln t" marker-end="url(#aT)"/>
|
||||
<rect x="200" y="304" width="126" height="38" rx="7" class="node "/>
|
||||
<text x="263.0" y="328.0" class="hd" text-anchor="middle">duck music</text>
|
||||
<path d="M326,323 L352,323" class="ln t" marker-end="url(#aT)"/>
|
||||
<rect x="352" y="304" width="120" height="38" rx="7" class="node "/>
|
||||
<text x="412.0" y="328.0" class="hd" text-anchor="middle">announce</text>
|
||||
<text x="38" y="372" class="sub">then restore the music to its previous level</text>
|
||||
<text x="38" y="392" class="sub">an announcement is a priority lane over the music</text>
|
||||
<rect x="516" y="262" width="420" height="152" rx="7" class="node "/>
|
||||
<text x="530" y="288" class="hd">Why it is built this way</text>
|
||||
<text x="530" y="312" class="sub">· one source of time, not one source of audio</text>
|
||||
<text x="530" y="336" class="sub">· endpoints are disposable; the server holds the state</text>
|
||||
<text x="530" y="360" class="sub">· voice reuses the same endpoint fleet</text>
|
||||
<text x="530" y="384" class="sub">· ducking is a mix decision, not a volume hack</text>
|
||||
<text x="24" y="452" class="sub">Eight synchronised endpoints with music ducking for voice.</text>
|
||||
<text x="24" y="472" class="sub">The endpoints are the cheapest part of the system, which is the point.</text>
|
||||
<text x="936" y="500" class="sub" text-anchor="end">confidence: delivered · 8 endpoints · Pi 2W</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 6.7 KiB |
@@ -1,92 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 540" width="960" height="540" role="img" aria-labelledby="ttl dsc">
|
||||
<title id="ttl">Media pipeline</title>
|
||||
<desc id="dsc">Five stages from download to library: fingerprinting gives every file a natural key so re-runs converge, audit verifies the copy, and review is a deliberate human gate before anything is published.</desc>
|
||||
<style>
|
||||
text{font-family:ui-sans-serif,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;fill:#1b222b;font-size:15px}
|
||||
.hd{font-size:15px;font-weight:600}
|
||||
.sub{font-size:13px;fill:#5b6673}
|
||||
.cap{font-size:12px;fill:#5b6673;letter-spacing:.06em;text-transform:uppercase;font-weight:600}
|
||||
.big{font-size:19px;font-weight:600}
|
||||
.node{fill:#f4f6f8;stroke:#c3ccd6;stroke-width:1.4}
|
||||
.node-a{fill:#e9f1fd;stroke:#1a6bc4;stroke-width:1.4}
|
||||
.node-t{fill:#e6f4f2;stroke:#0f766e;stroke-width:1.4}
|
||||
.node-w{fill:#fdf1e3;stroke:#b45309;stroke-width:1.4}
|
||||
.zone{fill:none;stroke:#aab4c0;stroke-width:1.4;stroke-dasharray:7 6}
|
||||
.rule{stroke:#d0d7df;stroke-width:1}
|
||||
.ln{stroke:#6b7684;stroke-width:1.6;fill:none}
|
||||
.ln.d{stroke-dasharray:6 5}
|
||||
.ln.a{stroke:#1a6bc4}
|
||||
.ln.t{stroke:#0f766e}
|
||||
.ln.w{stroke:#b45309}
|
||||
.mN{fill:#6b7684}
|
||||
.mA{fill:#1a6bc4}
|
||||
.mT{fill:#0f766e}
|
||||
.mW{fill:#b45309}
|
||||
.dot{fill:#6b7684}
|
||||
.tick{stroke:#aab4c0;stroke-width:1.2}
|
||||
@media (prefers-color-scheme: dark){
|
||||
text{fill:#e7edf4}
|
||||
.sub,.cap{fill:#98a4b3}
|
||||
.node{fill:#182028;stroke:#35424f}
|
||||
.node-a{fill:#11253d;stroke:#5aa2f0}
|
||||
.node-t{fill:#0d2624;stroke:#38b3a6}
|
||||
.node-w{fill:#2a1d10;stroke:#d98b3f}
|
||||
.zone{stroke:#43525f}
|
||||
.rule{stroke:#2c3743}
|
||||
.ln{stroke:#8496a6}
|
||||
.ln.a{stroke:#5aa2f0}
|
||||
.ln.t{stroke:#38b3a6}
|
||||
.ln.w{stroke:#d98b3f}
|
||||
.mN{fill:#8496a6}
|
||||
.mA{fill:#5aa2f0}
|
||||
.mT{fill:#38b3a6}
|
||||
.mW{fill:#d98b3f}
|
||||
.dot{fill:#8496a6}
|
||||
.tick{stroke:#43525f}
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="aN" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mN" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aA" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mA" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aT" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mT" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aW" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mW" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
</defs>
|
||||
<text x="24" y="40" class="big">Media pipeline — download, identify, verify, review, publish</text>
|
||||
<rect x="24" y="96" width="154" height="84" rx="7" class="node node-a"/>
|
||||
<text x="101.0" y="136.0" class="hd" text-anchor="middle">download</text>
|
||||
<text x="101.0" y="154.0" class="sub" text-anchor="middle">acquire the file</text>
|
||||
<text x="101" y="168" class="cap" text-anchor="middle">stage 1</text>
|
||||
<path d="M178,138 L214,138" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="214" y="96" width="154" height="84" rx="7" class="node "/>
|
||||
<text x="291.0" y="136.0" class="hd" text-anchor="middle">fingerprint</text>
|
||||
<text x="291.0" y="154.0" class="sub" text-anchor="middle">identify what it is</text>
|
||||
<text x="291" y="168" class="cap" text-anchor="middle">stage 2</text>
|
||||
<path d="M368,138 L404,138" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="404" y="96" width="154" height="84" rx="7" class="node "/>
|
||||
<text x="481.0" y="136.0" class="hd" text-anchor="middle">audit</text>
|
||||
<text x="481.0" y="154.0" class="sub" text-anchor="middle">verify the copy</text>
|
||||
<text x="481" y="168" class="cap" text-anchor="middle">stage 3</text>
|
||||
<path d="M558,138 L594,138" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="594" y="96" width="154" height="84" rx="7" class="node node-w"/>
|
||||
<text x="671.0" y="136.0" class="hd" text-anchor="middle">review</text>
|
||||
<text x="671.0" y="154.0" class="sub" text-anchor="middle">human gate</text>
|
||||
<text x="671" y="168" class="cap" text-anchor="middle">stage 4</text>
|
||||
<path d="M748,138 L784,138" class="ln " marker-end="url(#aN)"/>
|
||||
<rect x="784" y="96" width="154" height="84" rx="7" class="node node-a"/>
|
||||
<text x="861.0" y="136.0" class="hd" text-anchor="middle">library</text>
|
||||
<text x="861.0" y="154.0" class="sub" text-anchor="middle">publish + serve</text>
|
||||
<text x="861" y="168" class="cap" text-anchor="middle">stage 5</text>
|
||||
<rect x="214" y="236" width="342" height="96" rx="7" class="node node-w"/>
|
||||
<text x="228" y="262" class="hd">Gate, not a queue</text>
|
||||
<text x="228" y="284" class="sub">a file that fails fingerprint or audit</text>
|
||||
<text x="228" y="304" class="sub">stops where a person can see it</text>
|
||||
<rect x="590" y="236" width="346" height="96" rx="7" class="node "/>
|
||||
<text x="604" y="262" class="hd">Idempotent by identity</text>
|
||||
<text x="604" y="284" class="sub">the fingerprint is the natural key, so a</text>
|
||||
<text x="604" y="304" class="sub">re-run converges instead of duplicating</text>
|
||||
<rect x="24" y="356" width="912" height="92" rx="7" class="node "/>
|
||||
<text x="38" y="382" class="hd">The same shape at 10 files and at 10,000</text>
|
||||
<text x="38" y="404" class="sub">every stage is a separate step with its own failure mode, so a bad stage can be re-run alone</text>
|
||||
<text x="38" y="426" class="sub">the review stage is deliberately manual: automating it would move the risk, not remove it</text>
|
||||
<text x="24" y="480" class="sub">Encoding, thumbnailing and fingerprinting at scale, with a human gate before anything is published.</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 5.6 KiB |
@@ -1,95 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 540" width="960" height="540" role="img" aria-labelledby="ttl dsc">
|
||||
<title id="ttl">Cost and resource engineering</title>
|
||||
<desc id="dsc">A bar chart of the per-service memory caps drawn to scale against the 1024 MiB ceiling, totalling 970 MiB across four containers, beside the rules that hold the ceiling: equal memory and swap limits, loopback binding, a hard container count and a cost ceiling.</desc>
|
||||
<style>
|
||||
text{font-family:ui-sans-serif,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;fill:#1b222b;font-size:15px}
|
||||
.hd{font-size:15px;font-weight:600}
|
||||
.sub{font-size:13px;fill:#5b6673}
|
||||
.cap{font-size:12px;fill:#5b6673;letter-spacing:.06em;text-transform:uppercase;font-weight:600}
|
||||
.big{font-size:19px;font-weight:600}
|
||||
.node{fill:#f4f6f8;stroke:#c3ccd6;stroke-width:1.4}
|
||||
.node-a{fill:#e9f1fd;stroke:#1a6bc4;stroke-width:1.4}
|
||||
.node-t{fill:#e6f4f2;stroke:#0f766e;stroke-width:1.4}
|
||||
.node-w{fill:#fdf1e3;stroke:#b45309;stroke-width:1.4}
|
||||
.zone{fill:none;stroke:#aab4c0;stroke-width:1.4;stroke-dasharray:7 6}
|
||||
.rule{stroke:#d0d7df;stroke-width:1}
|
||||
.ln{stroke:#6b7684;stroke-width:1.6;fill:none}
|
||||
.ln.d{stroke-dasharray:6 5}
|
||||
.ln.a{stroke:#1a6bc4}
|
||||
.ln.t{stroke:#0f766e}
|
||||
.ln.w{stroke:#b45309}
|
||||
.mN{fill:#6b7684}
|
||||
.mA{fill:#1a6bc4}
|
||||
.mT{fill:#0f766e}
|
||||
.mW{fill:#b45309}
|
||||
.dot{fill:#6b7684}
|
||||
.tick{stroke:#aab4c0;stroke-width:1.2}
|
||||
@media (prefers-color-scheme: dark){
|
||||
text{fill:#e7edf4}
|
||||
.sub,.cap{fill:#98a4b3}
|
||||
.node{fill:#182028;stroke:#35424f}
|
||||
.node-a{fill:#11253d;stroke:#5aa2f0}
|
||||
.node-t{fill:#0d2624;stroke:#38b3a6}
|
||||
.node-w{fill:#2a1d10;stroke:#d98b3f}
|
||||
.zone{stroke:#43525f}
|
||||
.rule{stroke:#2c3743}
|
||||
.ln{stroke:#8496a6}
|
||||
.ln.a{stroke:#5aa2f0}
|
||||
.ln.t{stroke:#38b3a6}
|
||||
.ln.w{stroke:#d98b3f}
|
||||
.mN{fill:#8496a6}
|
||||
.mA{fill:#5aa2f0}
|
||||
.mT{fill:#38b3a6}
|
||||
.mW{fill:#d98b3f}
|
||||
.dot{fill:#8496a6}
|
||||
.tick{stroke:#43525f}
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="aN" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mN" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aA" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mA" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aT" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mT" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aW" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mW" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
</defs>
|
||||
<text x="24" y="40" class="big">Cost and resource engineering — the budget, measured</text>
|
||||
<text x="24" y="68" class="sub">Every service declares a ceiling. The ceiling is a measurement, not an estimate: .13 already runs on 2.3 GiB free with 8.1 GiB of swap in use.</text>
|
||||
<text x="40" y="104" class="cap">per-service cap (MiB)</text>
|
||||
<text x="40" y="136" class="sub">Go app</text>
|
||||
<rect x="200" y="112" width="33.6" height="30" rx="5" class="bar node-a"/>
|
||||
<text x="243.6" y="133" class="sub">120 MiB</text>
|
||||
<text x="40" y="182" class="sub">Qdrant</text>
|
||||
<rect x="200" y="158" width="112.00000000000001" height="30" rx="5" class="bar node-t"/>
|
||||
<text x="322.0" y="179" class="sub">400 MiB</text>
|
||||
<text x="40" y="228" class="sub">Prefect worker</text>
|
||||
<rect x="200" y="204" width="42.00000000000001" height="30" rx="5" class="bar "/>
|
||||
<text x="252.0" y="225" class="sub">150 MiB</text>
|
||||
<text x="40" y="274" class="sub">Guardrail (if local)</text>
|
||||
<rect x="200" y="250" width="84.00000000000001" height="30" rx="5" class="bar "/>
|
||||
<text x="294.0" y="271" class="sub">300 MiB</text>
|
||||
<path d="M200,352 L486.72,352" class="tick"/>
|
||||
<text x="200" y="374" class="sub">0</text>
|
||||
<text x="486.72" y="374" class="sub" text-anchor="end">1024 MiB = the whole new footprint</text>
|
||||
<path d="M200,340 L200,362" class="tick"/>
|
||||
<rect x="200" y="372" width="287" height="56" rx="7" class="node node-t"/>
|
||||
<text x="343.5" y="398.0" class="hd" text-anchor="middle">970 MiB of 1024 MiB</text>
|
||||
<text x="343.5" y="416.0" class="sub" text-anchor="middle">4 containers · measured, not asserted</text>
|
||||
<rect x="520" y="92" width="416" height="336" rx="10" class="zone"/>
|
||||
<text x="532" y="112" class="cap">rules that hold the ceiling</text>
|
||||
<rect x="536" y="118" width="384" height="52" rx="7" class="node "/>
|
||||
<text x="728.0" y="142.0" class="hd" text-anchor="middle">mem_limit == memswap_limit</text>
|
||||
<text x="728.0" y="160.0" class="sub" text-anchor="middle">a service cannot buy headroom from host swap</text>
|
||||
<rect x="536" y="180" width="384" height="52" rx="7" class="node "/>
|
||||
<text x="728.0" y="204.0" class="hd" text-anchor="middle">bind 127.0.0.1</text>
|
||||
<text x="728.0" y="222.0" class="sub" text-anchor="middle">Caddy is the only public listener</text>
|
||||
<rect x="536" y="242" width="384" height="52" rx="7" class="node "/>
|
||||
<text x="728.0" y="266.0" class="hd" text-anchor="middle">4 containers, not 14</text>
|
||||
<text x="728.0" y="284.0" class="sub" text-anchor="middle">one process per job, no sidecars</text>
|
||||
<rect x="536" y="304" width="384" height="52" rx="7" class="node "/>
|
||||
<text x="728.0" y="328.0" class="hd" text-anchor="middle">cost has a ceiling too</text>
|
||||
<text x="728.0" y="346.0" class="sub" text-anchor="middle">median session ≤ US$0.01 · monthly refusal</text>
|
||||
<text x="536" y="384" class="sub">verified with docker inspect, not with intent</text>
|
||||
<rect x="24" y="446" width="912" height="56" rx="7" class="node node-w"/>
|
||||
<text x="480.0" y="472.0" class="hd" text-anchor="middle">Host reality on .13: 15 GiB total · 2.3 GiB free · 8.1 GiB swap in use</text>
|
||||
<text x="480.0" y="490.0" class="sub" text-anchor="middle">exceeding the budget destabilises services people rely on</text>
|
||||
<text x="24" y="524" class="sub">Source: reference/stack.md, hosts measured 2026-10-17. The caps are the contract; the bar chart is the contract drawn to scale.</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 6.0 KiB |
@@ -1,106 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 540" width="960" height="540" role="img" aria-labelledby="ttl dsc">
|
||||
<title id="ttl">Era timeline</title>
|
||||
<desc id="dsc">Five consecutive platform eras from Flash and ActionScript through jQuery and PHP, React and TypeScript, server-driven Go with HTMX, to agent systems, on an axis from 2000 to 2026. Era boundaries are approximate; the bands show order.</desc>
|
||||
<style>
|
||||
text{font-family:ui-sans-serif,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;fill:#1b222b;font-size:15px}
|
||||
.hd{font-size:15px;font-weight:600}
|
||||
.sub{font-size:13px;fill:#5b6673}
|
||||
.cap{font-size:12px;fill:#5b6673;letter-spacing:.06em;text-transform:uppercase;font-weight:600}
|
||||
.big{font-size:19px;font-weight:600}
|
||||
.node{fill:#f4f6f8;stroke:#c3ccd6;stroke-width:1.4}
|
||||
.node-a{fill:#e9f1fd;stroke:#1a6bc4;stroke-width:1.4}
|
||||
.node-t{fill:#e6f4f2;stroke:#0f766e;stroke-width:1.4}
|
||||
.node-w{fill:#fdf1e3;stroke:#b45309;stroke-width:1.4}
|
||||
.zone{fill:none;stroke:#aab4c0;stroke-width:1.4;stroke-dasharray:7 6}
|
||||
.rule{stroke:#d0d7df;stroke-width:1}
|
||||
.ln{stroke:#6b7684;stroke-width:1.6;fill:none}
|
||||
.ln.d{stroke-dasharray:6 5}
|
||||
.ln.a{stroke:#1a6bc4}
|
||||
.ln.t{stroke:#0f766e}
|
||||
.ln.w{stroke:#b45309}
|
||||
.mN{fill:#6b7684}
|
||||
.mA{fill:#1a6bc4}
|
||||
.mT{fill:#0f766e}
|
||||
.mW{fill:#b45309}
|
||||
.dot{fill:#6b7684}
|
||||
.tick{stroke:#aab4c0;stroke-width:1.2}
|
||||
@media (prefers-color-scheme: dark){
|
||||
text{fill:#e7edf4}
|
||||
.sub,.cap{fill:#98a4b3}
|
||||
.node{fill:#182028;stroke:#35424f}
|
||||
.node-a{fill:#11253d;stroke:#5aa2f0}
|
||||
.node-t{fill:#0d2624;stroke:#38b3a6}
|
||||
.node-w{fill:#2a1d10;stroke:#d98b3f}
|
||||
.zone{stroke:#43525f}
|
||||
.rule{stroke:#2c3743}
|
||||
.ln{stroke:#8496a6}
|
||||
.ln.a{stroke:#5aa2f0}
|
||||
.ln.t{stroke:#38b3a6}
|
||||
.ln.w{stroke:#d98b3f}
|
||||
.mN{fill:#8496a6}
|
||||
.mA{fill:#5aa2f0}
|
||||
.mT{fill:#38b3a6}
|
||||
.mW{fill:#d98b3f}
|
||||
.dot{fill:#8496a6}
|
||||
.tick{stroke:#43525f}
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="aN" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mN" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aA" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mA" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aT" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mT" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
<marker id="aW" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="mW" d="M0,0 L10,5 L0,10 z"/></marker>
|
||||
</defs>
|
||||
<text x="24" y="40" class="big">Twenty-five years, five platform shifts</text>
|
||||
<rect x="28" y="96" width="176" height="118" rx="7" class="node"/>
|
||||
<text x="42" y="126" class="cap">01</text>
|
||||
<text x="42" y="144" class="hd">Flash / ActionScript</text>
|
||||
<text x="42" y="168" class="sub">classes, imports,</text>
|
||||
<text x="42" y="186" class="sub">encapsulation</text>
|
||||
<text x="42" y="212" class="sub">client wall</text>
|
||||
<path d="M205,155 L211,155" class="ln"/>
|
||||
<rect x="212" y="96" width="176" height="118" rx="7" class="node"/>
|
||||
<text x="226" y="126" class="cap">02</text>
|
||||
<text x="226" y="144" class="hd">jQuery + PHP</text>
|
||||
<text x="226" y="168" class="sub">media pipelines,</text>
|
||||
<text x="226" y="186" class="sub">FFmpeg, ImageMagick</text>
|
||||
<text x="226" y="212" class="sub">delivery at scale</text>
|
||||
<path d="M389,155 L395,155" class="ln"/>
|
||||
<rect x="396" y="96" width="176" height="118" rx="7" class="node"/>
|
||||
<text x="410" y="126" class="cap">03</text>
|
||||
<text x="410" y="144" class="hd">React + TypeScript</text>
|
||||
<text x="410" y="168" class="sub">typed service</text>
|
||||
<text x="410" y="186" class="sub">layers, tRPC</text>
|
||||
<text x="410" y="212" class="sub">typed front ends</text>
|
||||
<path d="M573,155 L579,155" class="ln"/>
|
||||
<rect x="580" y="96" width="176" height="118" rx="7" class="node node-a"/>
|
||||
<text x="594" y="126" class="cap">04</text>
|
||||
<text x="594" y="144" class="hd">Server-driven</text>
|
||||
<text x="594" y="168" class="sub">Go + templ + HTMX,</text>
|
||||
<text x="594" y="186" class="sub">SQLite</text>
|
||||
<text x="594" y="212" class="sub">this site</text>
|
||||
<path d="M757,155 L763,155" class="ln"/>
|
||||
<rect x="764" y="96" width="176" height="118" rx="7" class="node node-t"/>
|
||||
<text x="778" y="126" class="cap">05</text>
|
||||
<text x="778" y="144" class="hd">Agent systems</text>
|
||||
<text x="778" y="168" class="sub">harnesses, routing,</text>
|
||||
<text x="778" y="186" class="sub">retrieval</text>
|
||||
<text x="778" y="212" class="sub">AI estate</text>
|
||||
<path d="M28,268 L932,268" class="ln"/>
|
||||
<path d="M28,262 L28,274" class="tick"/>
|
||||
<text x="28" y="292" class="hd" text-anchor="start">2000</text>
|
||||
<path d="M546,262 L546,274" class="tick"/>
|
||||
<text x="546" y="292" class="hd" text-anchor="middle">2020</text>
|
||||
<path d="M932,262 L932,274" class="tick"/>
|
||||
<text x="932" y="292" class="hd" text-anchor="end">2026</text>
|
||||
<text x="28" y="314" class="sub">employed delivery</text>
|
||||
<text x="546" y="314" class="sub">independent engineering practice</text>
|
||||
<text x="932" y="314" class="sub" text-anchor="end">now</text>
|
||||
<rect x="28" y="348" width="904" height="118" rx="7" class="node "/>
|
||||
<text x="42" y="376" class="hd">What the five eras have in common</text>
|
||||
<text x="42" y="400" class="sub">· every era was learned by shipping with it, not by reading about it</text>
|
||||
<text x="42" y="424" class="sub">· the artefacts survive: a Flash class, an easing engine, a media pipeline, a typed service layer, a 33-line derivation</text>
|
||||
<text x="42" y="448" class="sub">· the current era is the first where platform, agents and edge hardware are one system</text>
|
||||
<text x="28" y="500" class="sub">Era boundaries are approximate: employed 2000–2020 per the résumé, independent practice 2020–.</text>
|
||||
<text x="932" y="500" class="sub" text-anchor="end">six code exhibits span eras 1–5</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 5.9 KiB |
@@ -1,80 +0,0 @@
|
||||
# Phase 7.2 — diagrams
|
||||
|
||||
Eight diagrams, one visual language, light and dark in one file each.
|
||||
|
||||
Authored 2026-10-26. Source of truth is the SVG files themselves: each one is
|
||||
self-contained, with no external font, image or script reference.
|
||||
|
||||
## The language
|
||||
|
||||
| Part | Rule |
|
||||
|---|---|
|
||||
| Canvas | `viewBox="0 0 960 540"`, `width`/`height` set so a bare file view is 1× |
|
||||
| Type | 19 px section title · 15 px heading · 13 px label · 12 px caption, all `ui-sans-serif` system stack |
|
||||
| Boxes | `rx=7`, 1.4 px stroke, four semantic tones: neutral, accent (the thing being described), teal (store / gateway), amber (guard, failure, limit) |
|
||||
| Arrows | solid = data or control flow, dashed = trust boundary or approximate relationship |
|
||||
| Colour | greys carry structure; one blue accent, one teal, one amber, nothing else |
|
||||
| Text | every `<text>` is real text in the SVG, so it scales, selects and reads to a screen reader |
|
||||
|
||||
Every file carries a **byte-identical `<style>` block**. That is the whole of the
|
||||
consistency guarantee: there is one palette, defined once, repeated verbatim.
|
||||
A check that compares the eight style blocks and finds one distinct value is the
|
||||
evidence that the eight diagrams share a language.
|
||||
|
||||
## Both themes in one file
|
||||
|
||||
The `<style>` block defines the light palette, then repeats the same selectors
|
||||
inside `@media (prefers-color-scheme: dark)`. No duplicated files, no theme
|
||||
toggle needed in the page, no chance of a light and a dark copy drifting apart.
|
||||
|
||||
If the site later grows a manual theme toggle, the SVG must be inlined into the
|
||||
page (not `<img>`) so page CSS can override the custom properties. Until then
|
||||
`<img>` is correct: the file follows the reader's operating system.
|
||||
|
||||
| Role | Light | Dark |
|
||||
|---|---|---|
|
||||
| ink (text) | `#1b222b` | `#e7edf4` |
|
||||
| panel (box fill) | `#f4f6f8` | `#182028` |
|
||||
| muted (labels) | `#5b6673` | `#98a4b3` |
|
||||
| accent | `#1a6bc4` | `#5aa2f0` |
|
||||
| teal | `#0f766e` | `#38b3a6` |
|
||||
| amber | `#b45309` | `#d98b3f` |
|
||||
| box stroke | `#c3ccd6` | `#35424f` |
|
||||
|
||||
## Mobile
|
||||
|
||||
A 960 px schematic cannot be read at 360 px. The presentation contract is:
|
||||
|
||||
- wrap the figure in a horizontally scrollable container
|
||||
- `svg { display:block; width:100%; min-width:900px; height:auto }`
|
||||
|
||||
At 1× (container ≥ 960 px) the type renders at 15 px / 13 px. At the 900 px
|
||||
minimum inside a 390 px container it renders at 14.1 px / 12.2 px — the same
|
||||
order as body text, and the figure scrolls rather than shrinking into
|
||||
illegibility. A figure that is scaled to 360 px instead reaches 5 px type and is
|
||||
not readable; that is why the contract is a requirement and not a suggestion.
|
||||
|
||||
## Accessibility
|
||||
|
||||
Each file carries `role="img"` and both `<title>` and `<desc>`. The `<desc>` is
|
||||
the alternative text — 190–285 characters, describing what the diagram asserts
|
||||
rather than naming its parts. If a template sets `alt=""` on the `<img>`, the
|
||||
`<title>`/`<desc>` are not read out; the template must supply `alt` from the
|
||||
manifest's `alt:` field.
|
||||
|
||||
## Files
|
||||
|
||||
| # | File | Area | Answers |
|
||||
|---|---|---|---|
|
||||
| 1 | `01-homelab-topology.svg` | platform | where it runs, and where the trust boundaries are |
|
||||
| 2 | `02-agent-architecture.svg` | ai | how agents are organised and why one gateway |
|
||||
| 3 | `03-rag-pipeline.svg` | ai | which side of the pipeline is AI and which side is not |
|
||||
| 4 | `04-edge-architecture.svg` | edge | why devices never talk to each other |
|
||||
| 5 | `05-distributed-audio.svg` | edge | how eight speakers stay in sync, and how voice gets in |
|
||||
| 6 | `06-media-pipeline.svg` | products | where the human gate sits in a media pipeline |
|
||||
| 7 | `07-cost-resource-engineering.svg` | platform | the memory ceiling, drawn to scale |
|
||||
| 8 | `08-era-timeline.svg` | history | five platform shifts and what survived them |
|
||||
|
||||
Facts behind them come from `reference/stack.md`, `reference/content-map.md` and
|
||||
`docs/phase-3-deploy.md`. Diagram 7 is numbers from a measurement, not an
|
||||
illustration: if the caps change, the bars change.
|
||||
109
content/assets/diagrams/homelab-topology.svg
Normal file
@@ -0,0 +1,109 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1110" height="600" viewBox="0 0 1110 600" role="img" aria-labelledby="topoTitle topoDesc" font-family="ui-sans-serif, system-ui, -apple-system, 'Segoe UI', sans-serif">
|
||||
<title id="topoTitle">Homelab topology: build host, runtime host and public edge</title>
|
||||
<desc id="topoDesc">Three hosts. The build host produces a release; the runtime host runs the site, the vector store, a workflow worker and the model gateway, all bound to loopback; the public edge terminates TLS and is the only host reachable from the internet. All three join one private mesh, and backups run in three tiers.</desc>
|
||||
|
||||
<style>
|
||||
/* On paper the diagram is ink on white, not a dark panel. Presentation
|
||||
attributes are the weakest source, so these rules win without editing
|
||||
the drawing. */
|
||||
@media print {
|
||||
rect { fill: #fff; }
|
||||
text { fill: #111; }
|
||||
text[fill="#6ee7c6"] { fill: #0b5c48; }
|
||||
text[fill="#8b98a8"] { fill: #444; }
|
||||
line, path { stroke: #555; }
|
||||
marker path { fill: #555; }
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||||
<path d="M0 0 10 5 0 10z" fill="#6ee7c6"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<g fill="#12161b" stroke="#33404d" stroke-width="1.5">
|
||||
<rect x="30" y="104" width="310" height="268" rx="10"/>
|
||||
<rect x="400" y="104" width="310" height="268" rx="10"/>
|
||||
<rect x="770" y="104" width="310" height="268" rx="10"/>
|
||||
</g>
|
||||
|
||||
<!-- internet -->
|
||||
<g>
|
||||
<rect x="810" y="24" width="230" height="46" rx="23" fill="#171d24" stroke="#33404d" stroke-width="1.5"/>
|
||||
<text x="925" y="52" text-anchor="middle" font-size="13" fill="#b3bcc8">public internet · https :443</text>
|
||||
<line x1="925" y1="70" x2="925" y2="100" stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)"/>
|
||||
</g>
|
||||
|
||||
<!-- host headers -->
|
||||
<g fill="#e6edf3" font-size="15" font-weight="600">
|
||||
<text x="54" y="136">Build and development</text>
|
||||
<text x="424" y="136">Runtime host</text>
|
||||
<text x="794" y="136">Public edge</text>
|
||||
</g>
|
||||
<g fill="#8b98a8" font-size="12" font-family="ui-monospace, SFMono-Regular, Menlo, monospace">
|
||||
<text x="54" y="156">.27 · 4-screen desktop</text>
|
||||
<text x="424" y="156">.13 · 15 GiB · memory-bound</text>
|
||||
<text x="794" y="156">.35 · Ubuntu · 2 vCPU</text>
|
||||
</g>
|
||||
|
||||
<!-- build host items -->
|
||||
<g fill="#b3bcc8" font-size="12.5">
|
||||
<text x="54" y="192">· repository working copy</text>
|
||||
<text x="54" y="219">· agent fleet, panes and approvals</text>
|
||||
<text x="54" y="246">· content, diagrams and exhibits</text>
|
||||
<text x="54" y="273">· release bundles and hashes</text>
|
||||
</g>
|
||||
|
||||
<!-- runtime: loopback trust boundary -->
|
||||
<rect x="416" y="172" width="278" height="150" rx="8" fill="none" stroke="#6ee7c6" stroke-width="1.2" stroke-dasharray="5 4"/>
|
||||
<text x="430" y="192" font-size="11" fill="#6ee7c6" font-family="ui-monospace, SFMono-Regular, Menlo, monospace">bound to 127.0.0.1</text>
|
||||
<g fill="#b3bcc8" font-size="12.5">
|
||||
<text x="430" y="216">· site server :8096</text>
|
||||
<text x="430" y="241">· Qdrant :6333 / :6334</text>
|
||||
<text x="430" y="266">· Prefect worker</text>
|
||||
<text x="430" y="291">· model gateway (shared)</text>
|
||||
</g>
|
||||
<text x="424" y="345" font-size="11.5" fill="#8b98a8">Caps equal swap; nothing listens publicly.</text>
|
||||
|
||||
<!-- edge host items -->
|
||||
<g fill="#b3bcc8" font-size="12.5">
|
||||
<text x="794" y="192">· Caddy terminates TLS</text>
|
||||
<text x="794" y="219">· proxies to the runtime host</text>
|
||||
<text x="794" y="246">· Gitea · Immich · Nextcloud</text>
|
||||
<text x="794" y="273">· one public listener, many sites</text>
|
||||
</g>
|
||||
|
||||
<!-- hand-off arrows -->
|
||||
<line x1="340" y1="230" x2="396" y2="230" stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)"/>
|
||||
<text x="368" y="218" text-anchor="middle" font-size="10.5" fill="#8b98a8">deploy</text>
|
||||
<line x1="710" y1="230" x2="766" y2="230" stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)"/>
|
||||
<text x="738" y="218" text-anchor="middle" font-size="10.5" fill="#8b98a8">proxy</text>
|
||||
|
||||
<!-- mesh band -->
|
||||
<rect x="30" y="410" width="1050" height="56" rx="10" fill="#12161b" stroke="#33404d" stroke-width="1.2" stroke-dasharray="6 5"/>
|
||||
<text x="555" y="436" text-anchor="middle" font-size="13" fill="#b3bcc8">private mesh · SSH on every host · remote access without an exposed port</text>
|
||||
<text x="555" y="454" text-anchor="middle" font-size="11" fill="#8b98a8">no service is reachable because it happened to bind to every interface</text>
|
||||
<g stroke="#33404d" stroke-width="1" stroke-dasharray="3 4">
|
||||
<line x1="185" y1="372" x2="185" y2="410"/>
|
||||
<line x1="555" y1="372" x2="555" y2="410"/>
|
||||
<line x1="925" y1="372" x2="925" y2="410"/>
|
||||
</g>
|
||||
|
||||
<!-- backup tiers -->
|
||||
<text x="30" y="506" font-size="13" font-weight="600" fill="#e6edf3">Three backup tiers — each covers a different failure</text>
|
||||
<g fill="#12161b" stroke="#33404d" stroke-width="1.2">
|
||||
<rect x="30" y="522" width="330" height="52" rx="8"/>
|
||||
<rect x="390" y="522" width="330" height="52" rx="8"/>
|
||||
<rect x="750" y="522" width="330" height="52" rx="8"/>
|
||||
</g>
|
||||
<g fill="#b3bcc8" font-size="12.5">
|
||||
<text x="50" y="545">local snapshots, on the machine</text>
|
||||
<text x="410" y="545">a copy on another machine</text>
|
||||
<text x="770" y="545">encrypted object storage, offsite</text>
|
||||
</g>
|
||||
<g fill="#8b98a8" font-size="11">
|
||||
<text x="50" y="562">restore tested, runbook beside it</text>
|
||||
<text x="410" y="562">over the private mesh</text>
|
||||
<text x="770" y="562">retention stated in the job</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.6 KiB |
@@ -1,82 +0,0 @@
|
||||
# Phase 7.2 — diagram manifest
|
||||
#
|
||||
# Machine-readable index for the templates. `alt` is copied verbatim from each
|
||||
# file's <desc>, so the text a screen reader gets and the text in this manifest
|
||||
# are the same string.
|
||||
version: 1
|
||||
authored: 2026-10-26
|
||||
viewbox: "0 0 960 540"
|
||||
themes: [light, dark] # both resolved from one file via prefers-color-scheme
|
||||
format: svg-text # real text nodes, not glyph outlines
|
||||
external_refs: none # no fonts, images or scripts referenced
|
||||
presentation:
|
||||
desktop: {css_width: "100%", rendered_type_px: [15, 13]}
|
||||
mobile:
|
||||
wrap: "overflow-x:auto"
|
||||
css: "svg{display:block;width:100%;min-width:900px;height:auto}"
|
||||
rendered_type_px: [14.1, 12.2]
|
||||
diagrams:
|
||||
- id: platform-homelab-topology
|
||||
file: 01-homelab-topology.svg
|
||||
area: platform
|
||||
level: 2
|
||||
title: "Homelab topology"
|
||||
answers: "where it runs, and where the trust boundaries are"
|
||||
alt: "Three hosts with their roles and the trust boundaries between them: public internet reaches Caddy on .35, which forwards over the LAN to services bound to 127.0.0.1 on .13. .27 is build and dev with no public listeners."
|
||||
bytes: 7079
|
||||
- id: ai-agent-architecture
|
||||
file: 02-agent-architecture.svg
|
||||
area: ai
|
||||
level: 2
|
||||
title: "Agent architecture"
|
||||
answers: "how the agent estate is organised, and why there is one gateway"
|
||||
alt: "The operator reaches a fleet of agent sessions through one inbox. Paperclip is the control plane and delegates to peer projects, which answer the Hub read-only. Every model call in the estate passes through one gateway, OmniRoute."
|
||||
bytes: 6027
|
||||
- id: ai-rag-pipeline
|
||||
file: 03-rag-pipeline.svg
|
||||
area: ai
|
||||
level: 2
|
||||
title: "Retrieval pipeline"
|
||||
answers: "which side of the pipeline is AI and which side is not"
|
||||
alt: "Left, offline ingest: content files are validated, chunked with metadata, embedded and upserted into Qdrant, while the same files render to static pages with no AI in the path. Right, per-visitor query: guardrail, query graph, filtered retrieval, then a streamed answer with citations."
|
||||
bytes: 7010
|
||||
- id: edge-architecture
|
||||
file: 04-edge-architecture.svg
|
||||
area: edge
|
||||
level: 2
|
||||
title: "Edge architecture"
|
||||
answers: "why devices never talk to each other directly"
|
||||
alt: "Constrained devices publish over LoRa, Zigbee or WiFi to a single MQTT broker, which feeds one automation layer that drives actuation. Retained configuration, telemetry and last-will presence let devices recover unattended."
|
||||
bytes: 6896
|
||||
- id: edge-distributed-audio
|
||||
file: 05-distributed-audio.svg
|
||||
area: edge
|
||||
level: 2
|
||||
title: "Distributed audio"
|
||||
answers: "how eight endpoints stay in sync, and how voice gets in"
|
||||
alt: "Music and speech sources feed one Snapcast server, which streams one clock-synchronised mix to eight Pi endpoints. Voice announcements arrive as a priority lane: duck the music, speak, restore."
|
||||
bytes: 6833
|
||||
- id: products-media-pipeline
|
||||
file: 06-media-pipeline.svg
|
||||
area: products
|
||||
level: 2
|
||||
title: "Media pipeline"
|
||||
answers: "where the human gate sits in a media pipeline"
|
||||
alt: "Five stages from download to library: fingerprinting gives every file a natural key so re-runs converge, audit verifies the copy, and review is a deliberate human gate before anything is published."
|
||||
bytes: 5694
|
||||
- id: platform-cost-resource-engineering
|
||||
file: 07-cost-resource-engineering.svg
|
||||
area: platform
|
||||
level: 2
|
||||
title: "Cost and resource engineering"
|
||||
answers: "the memory ceiling, drawn to scale from measured caps"
|
||||
alt: "A bar chart of the per-service memory caps drawn to scale against the 1024 MiB ceiling, totalling 970 MiB across four containers, beside the rules that hold the ceiling: equal memory and swap limits, loopback binding, a hard container count and a cost ceiling."
|
||||
bytes: 6151
|
||||
- id: history-era-timeline
|
||||
file: 08-era-timeline.svg
|
||||
area: history
|
||||
level: 2
|
||||
title: "Era timeline"
|
||||
answers: "five platform shifts and what survived them"
|
||||
alt: "Five consecutive platform eras from Flash and ActionScript through jQuery and PHP, React and TypeScript, server-driven Go with HTMX, to agent systems, on an axis from 2000 to 2026. Era boundaries are approximate; the bands show order."
|
||||
bytes: 6039
|
||||
119
content/assets/diagrams/rag-pipeline.svg
Normal file
@@ -0,0 +1,119 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1120" height="580" viewBox="0 0 1120 580" role="img" aria-labelledby="ragTitle ragDesc" font-family="ui-sans-serif, system-ui, -apple-system, 'Segoe UI', sans-serif">
|
||||
<title id="ragTitle">Retrieval pipeline: ingest on top, four-node query path below</title>
|
||||
<desc id="ragDesc">Ingest reads content and estate notes, excludes secrets, chunks and tags each block, embeds through the model gateway and upserts into two Qdrant collections on loopback. The query path classifies the question, routes it to an area, retrieves with a payload filter, composes an answer from the retrieved context only, and verifies that every citation resolves to a page.</desc>
|
||||
|
||||
<style>
|
||||
/* On paper the diagram is ink on white, not a dark panel. Presentation
|
||||
attributes are the weakest source, so these rules win without editing
|
||||
the drawing. */
|
||||
@media print {
|
||||
rect { fill: #fff; }
|
||||
text { fill: #111; }
|
||||
text[fill="#6ee7c6"] { fill: #0b5c48; }
|
||||
text[fill="#8b98a8"] { fill: #444; }
|
||||
line, path { stroke: #555; }
|
||||
marker path { fill: #555; }
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||||
<path d="M0 0 10 5 0 10z" fill="#6ee7c6"/>
|
||||
</marker>
|
||||
<marker id="arrowDim" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||||
<path d="M0 0 10 5 0 10z" fill="#8b98a8"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- ingest lane -->
|
||||
<text x="30" y="42" font-size="14" font-weight="600" fill="#e6edf3">Ingest — one source of truth, one index</text>
|
||||
|
||||
<g fill="#12161b" stroke="#33404d" stroke-width="1.4">
|
||||
<rect x="30" y="60" width="244" height="92" rx="9"/>
|
||||
<rect x="302" y="60" width="244" height="92" rx="9"/>
|
||||
<rect x="574" y="60" width="244" height="92" rx="9"/>
|
||||
<rect x="846" y="60" width="244" height="92" rx="9"/>
|
||||
</g>
|
||||
<g fill="#e6edf3" font-size="13.5" font-weight="600">
|
||||
<text x="48" y="86">Sources</text>
|
||||
<text x="320" y="86">Normalise</text>
|
||||
<text x="592" y="86">Chunk and tag</text>
|
||||
<text x="864" y="86">Embed and upsert</text>
|
||||
</g>
|
||||
<g fill="#b3bcc8" font-size="12">
|
||||
<text x="48" y="110">content/ Markdown</text>
|
||||
<text x="48" y="129">estate notes</text>
|
||||
<text x="48" y="148">platform guides</text>
|
||||
<text x="320" y="110">front-matter parsed</text>
|
||||
<text x="320" y="129">secrets and drafts</text>
|
||||
<text x="320" y="148">excluded by rule</text>
|
||||
<text x="592" y="110">blocks, not lines</text>
|
||||
<text x="592" y="129">area · level · source</text>
|
||||
<text x="592" y="148">title · tags</text>
|
||||
<text x="864" y="110">model gateway only</text>
|
||||
<text x="864" y="129">deterministic ids</text>
|
||||
<text x="864" y="148">batched, resumable</text>
|
||||
</g>
|
||||
<g stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)">
|
||||
<line x1="277" y1="106" x2="298" y2="106"/>
|
||||
<line x1="549" y1="106" x2="570" y2="106"/>
|
||||
<line x1="821" y1="106" x2="842" y2="106"/>
|
||||
</g>
|
||||
|
||||
<line x1="968" y1="152" x2="968" y2="184" stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="30" y="186" width="1060" height="62" rx="9" fill="#12161b" stroke="#6ee7c6" stroke-width="1.2" stroke-dasharray="5 4"/>
|
||||
<text x="50" y="210" font-size="13" font-weight="600" fill="#e6edf3">Qdrant on 127.0.0.1</text>
|
||||
<text x="50" y="231" font-size="12" fill="#b3bcc8">resume_corpus · one point per chunk resume_graph · one point per node and edge HNSW and payloads on disk · within a 400 MiB cap</text>
|
||||
<text x="50" y="272" font-size="11.5" fill="#8b98a8">A sync run re-embeds only the documents whose content, model or chunker version changed.</text>
|
||||
|
||||
<!-- query lane -->
|
||||
<text x="30" y="316" font-size="14" font-weight="600" fill="#e6edf3">Query</text>
|
||||
<text x="108" y="316" font-size="12" fill="#8b98a8">— four graph nodes. It grows past four only with a written reason.</text>
|
||||
|
||||
<line x1="776" y1="248" x2="776" y2="352" stroke="#8b98a8" stroke-width="1.3" stroke-dasharray="4 4" marker-end="url(#arrowDim)"/>
|
||||
<text x="788" y="330" font-size="11" fill="#8b98a8">payload-filtered search</text>
|
||||
|
||||
<g fill="#12161b" stroke="#33404d" stroke-width="1.4">
|
||||
<rect x="30" y="360" width="196" height="112" rx="9"/>
|
||||
<rect x="246" y="360" width="196" height="112" rx="9"/>
|
||||
<rect x="462" y="360" width="196" height="112" rx="9"/>
|
||||
<rect x="678" y="360" width="196" height="112" rx="9"/>
|
||||
<rect x="894" y="360" width="196" height="112" rx="9"/>
|
||||
</g>
|
||||
<g font-size="10.5" fill="#6ee7c6" font-family="ui-monospace, SFMono-Regular, Menlo, monospace">
|
||||
<text x="46" y="382">CLASSIFY</text>
|
||||
<text x="262" y="382">ROUTE</text>
|
||||
<text x="478" y="382">RETRIEVE</text>
|
||||
<text x="694" y="382">COMPOSE</text>
|
||||
<text x="910" y="382">VERIFY</text>
|
||||
</g>
|
||||
<g fill="#e6edf3" font-size="13" font-weight="600">
|
||||
<text x="46" y="406">Read the question</text>
|
||||
<text x="262" y="406">Pick the areas</text>
|
||||
<text x="478" y="406">Find the evidence</text>
|
||||
<text x="694" y="406">Answer from it</text>
|
||||
<text x="910" y="406">Check citations</text>
|
||||
</g>
|
||||
<g fill="#b3bcc8" font-size="11.5">
|
||||
<text x="46" y="430">intent · injection</text>
|
||||
<text x="46" y="448">off-topic · PII</text>
|
||||
<text x="262" y="430">which pages can</text>
|
||||
<text x="262" y="448">answer this</text>
|
||||
<text x="478" y="430">vector + keyword</text>
|
||||
<text x="478" y="448">ranked, top five</text>
|
||||
<text x="694" y="430">context only</text>
|
||||
<text x="694" y="448">no outside claims</text>
|
||||
<text x="910" y="430">every claim maps</text>
|
||||
<text x="910" y="448">to a real page</text>
|
||||
</g>
|
||||
<g stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)">
|
||||
<line x1="227" y1="416" x2="242" y2="416"/>
|
||||
<line x1="443" y1="416" x2="458" y2="416"/>
|
||||
<line x1="659" y1="416" x2="674" y2="416"/>
|
||||
<line x1="875" y1="416" x2="890" y2="416"/>
|
||||
</g>
|
||||
|
||||
<text x="30" y="500" font-size="12" fill="#b3bcc8">A refusal is a real outcome: an off-topic question, an injection attempt or a request over the cost cap stops at the first node with zero model spend.</text>
|
||||
<text x="30" y="526" font-size="11.5" fill="#8b98a8">Ingest runs today. The query path is the Phase 5 design; until it ships, every page is pre-rendered and needs no model.</text>
|
||||
<text x="30" y="550" font-size="11.5" fill="#8b98a8">The public AI has no tools, no filesystem, no network fetch and no memory between visitors — architectural, not a prompt instruction.</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 6.5 KiB |
BIN
content/assets/images/sim7080g-module.jpg
Normal file
|
After Width: | Height: | Size: 29 KiB |
@@ -1,125 +0,0 @@
|
||||
# Phase 7.1 — the client wall
|
||||
#
|
||||
# Decision: a typographic wall, not a logo collage. 45 entries, 44 distinct names, one
|
||||
# typeface, one size, an even grid. Nothing is an image, so nothing can fail to
|
||||
# load, and no name is more prominent than another.
|
||||
#
|
||||
# The Simple Icons audit below was performed for Phase 7. Ten marks exist. They
|
||||
# are NOT enabled in this file: putting a client's mark on a public portfolio is
|
||||
# a permission decision, not a design decision, so `mark: none` for every entry
|
||||
# until Sam says otherwise.
|
||||
version: 1
|
||||
authored: 2026-10-26
|
||||
render: typographic
|
||||
mark_policy:
|
||||
default: none
|
||||
enable: "ask Sam before enabling any commercial mark; availability is recorded, permission is not implied"
|
||||
icon_set: "Simple Icons (CC0 paths, trademarks remain with their owners)"
|
||||
counts:
|
||||
agencies_listed: 22
|
||||
brands_listed: 23
|
||||
entries_listed: 45
|
||||
distinct_names: 44
|
||||
claimed_by_plan: 46
|
||||
note: >
|
||||
The plan says 46 names and 23 agencies. The source list contains 22 agencies
|
||||
and 23 brands, and 'Pearsons' appears in both groups, so the distinct count
|
||||
is 44. No name has been invented to reach 46; the difference is flagged for
|
||||
Sam rather than papered over.
|
||||
corrections_applied:
|
||||
- "Diajio -> Diageo (reference/content-map.md: factual correction, do not undo)"
|
||||
- "AutoCAD listed as the product; the mark that exists is Autodesk's"
|
||||
client_wall:
|
||||
agencies:
|
||||
- name: "Clemenger"
|
||||
mark: none
|
||||
- name: "Agency Republic"
|
||||
mark: none
|
||||
- name: "Brand Union"
|
||||
mark: none
|
||||
- name: "Jackpot Joy"
|
||||
mark: none
|
||||
- name: "Bryce Ford"
|
||||
mark: none
|
||||
- name: "RMG Connect"
|
||||
mark: none
|
||||
- name: "Sporting Index"
|
||||
mark: none
|
||||
- name: "Adventis Digital"
|
||||
mark: none
|
||||
- name: "Trinity St"
|
||||
mark: none
|
||||
- name: "Inferno"
|
||||
mark: none
|
||||
- name: "SAS Design"
|
||||
mark: none
|
||||
- name: "Ogilvy"
|
||||
mark: none
|
||||
- name: "Wunderman"
|
||||
mark: none
|
||||
- name: "Illumina"
|
||||
mark: none
|
||||
- name: "Barkers"
|
||||
mark: none
|
||||
- name: "Mook"
|
||||
mark: none
|
||||
- name: "Pearsons"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "Penguin"
|
||||
mark: none
|
||||
- name: "Draft London"
|
||||
mark: none
|
||||
- name: "Behan Consulting"
|
||||
mark: none
|
||||
- name: "MCM Entertainment"
|
||||
mark: none
|
||||
- name: "Carsales"
|
||||
mark: none
|
||||
brands:
|
||||
- name: "Carlton Draught"
|
||||
mark: none
|
||||
- name: "Diageo"
|
||||
mark: none
|
||||
- name: "KPMG"
|
||||
mark: none
|
||||
- name: "McDonald's"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "Accenture"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "BBC National Archive"
|
||||
mark: none
|
||||
- name: "Bank of America"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "War of the Worlds"
|
||||
mark: none
|
||||
- name: "Madonna (Hard Candy)"
|
||||
mark: none
|
||||
- name: "PlayStation"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "HSBC"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "DHL"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "Reuters"
|
||||
mark: none
|
||||
- name: "Volvic"
|
||||
mark: none
|
||||
- name: "AutoCAD"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
attribution: "Autodesk" if n == "AutoCAD"
|
||||
- name: "Channel 4"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "Pearsons"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "Shell"
|
||||
mark: none # available in Simple Icons; not enabled
|
||||
- name: "Stella Artois"
|
||||
mark: none
|
||||
- name: "Baileys Irish Cream"
|
||||
mark: none
|
||||
- name: "Parkgate Homes"
|
||||
mark: none
|
||||
- name: "Carlton Dry"
|
||||
mark: none
|
||||
- name: "Cottees Cordial"
|
||||
mark: none
|
||||
55
content/edge/edge-fleet.md
Normal file
@@ -0,0 +1,55 @@
|
||||
---
|
||||
id: edge-fleet
|
||||
area: edge
|
||||
level: 2
|
||||
title: "Constrained devices · custom radio · and a record of what broke"
|
||||
claim: "Twenty or more edge nodes across nine zones and twenty automations — solar power · deep sleep · LoRa and hand-built Zigbee — reporting into one automation layer with voice control on top."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [ESP32, LoRa, Zigbee, MQTT, Home Assistant, Snapcast, Pi, PIR, mmWave]
|
||||
metrics: [{label: "edge nodes", value: 20}, {label: "lighting zones", value: 9}, {label: "automations", value: 20}, {label: "hand-built Zigbee routers", value: 2}]
|
||||
artefacts: [{type: doc, label: "Edge architecture — power · radio · deep sleep", url: "content/edge/l3/edge-architecture.md"}, {type: doc, label: "LoRa irrigation under solar power", url: "content/edge/l3/edge-lora-sprinkler.md"}, {type: doc, label: "Hand-built Zigbee routers and lighting", url: "content/edge/l3/edge-zigbee.md"}, {type: doc, label: "Distributed audio — eight endpoints with ducking", url: "content/edge/l3/edge-snapcast.md"}, {type: doc, label: "Front door — vision · voice · and a rebuild", url: "content/edge/l3/edge-front-door.md"}, {type: doc, label: "Sensors and actuation", url: "content/edge/l3/edge-sensors.md"}, {type: doc, label: "The failures, kept in the record", url: "content/edge/l3/edge-failures.md"}, {type: repo, label: "Embedded C++ — the Wi-Fi manager interface and implementation", url: "content/assets/code/05-embedded-cpp-interface.hpp"}]
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
tags: [edge, iot, esp32, lora, zigbee, audio]
|
||||
---
|
||||
|
||||
**Capability statement.** Devices that must run for months on a small battery, talk over radio that was chosen for range rather than convenience, and fail in ways that are visible. Twenty or more nodes, nine zones, twenty automations, and a voice layer on top of all of it.
|
||||
|
||||
## What is actually run
|
||||
|
||||
**Edge architecture.** Each node is designed around its power budget: a LiFePO battery with a solar controller where there is sun, deep sleep between wakes, and a wake source chosen per job. The sprinkler and the garden lights wake on radio rather than on a timer, so a command can arrive while the device is asleep.
|
||||
|
||||
**Radio.** Three links, each for a reason. LoRa for range and penetration on outdoor nodes — long enough to cover the garden and the shed without a repeater. Zigbee for the indoor lighting mesh, including two routers built by hand where the commercial mesh did not reach. Wi-Fi where a device is mains-powered and needs bandwidth, such as the audio endpoints and the camera path.
|
||||
|
||||
**Sensors and actuation.** PIR for movement, radar and mmWave for presence that a PIR cannot see — including a presence sensor built for a bathroom and a child's bedroom because the off-the-shelf options were wrong for the space. Relays and step-down converters for 240V loads, servos and a 16-channel PWM driver for mechanical jobs, solenoids for water, load cells for weight.
|
||||
|
||||
**Audio distribution.** Eight Snapcast endpoints on Pi hardware, fed by a Snapserver, with Spotify and a music server as sources. Every spoken announcement — automation notices, assistant replies, doorbell messages — plays across all eight endpoints with the music ducked underneath and restored afterwards.
|
||||
|
||||
**The automation layer.** Home Assistant holds the state and the routines; MQTT carries the messages. Nine lighting zones and twenty automations, with voice control on top, so a request spoken in one room is answered through the speakers in whichever room a person is standing.
|
||||
|
||||
**Cameras and vision.** The front door takes an image, logs it, has the image described, and speaks the result into the house before deciding what to do next. RFID tags on boxes and on trees resolve to records — inventory with prices for the boxes, care notes for the trees — written by an automation rather than typed by hand.
|
||||
|
||||
## Why this exhibit is good code
|
||||
|
||||
The header declares the interface and includes only `Arduino.h`, so a caller compiles without pulling the ESP32 Wi-Fi stack into its translation unit. Only the implementation file knows about station mode, polling and millisecond timeouts, and it returns a boolean so the caller decides what a failed join means. The two files are shown together because the point is the split, not either file alone.
|
||||
|
||||
## Evidence
|
||||
|
||||
| Claim | Evidence | Artefact |
|
||||
|---|---|---|
|
||||
| Twenty or more nodes | Ten ESP32 devices · two hand-built Zigbee routers · eight Pi audio endpoints | [Edge architecture](l3/edge-architecture.md) |
|
||||
| Radio chosen per job | LoRa outdoors with radio wake from deep sleep; Zigbee indoors with hand-built routers | [Irrigation](l3/edge-lora-sprinkler.md) · [Zigbee](l3/edge-zigbee.md) |
|
||||
| Nine zones and twenty automations | Lighting zones and routines in the automation layer | [Zigbee and lighting](l3/edge-zigbee.md) |
|
||||
| Eight endpoints with ducking | Snapcast clients on Pi hardware with music lowered under speech | [Distributed audio](l3/edge-snapcast.md) |
|
||||
| Vision and voice at the door | Image capture · description · spoken response · two-way audio | [Front door](l3/edge-front-door.md) |
|
||||
| Failures are stated | Lizard feeder destroyed · garden cameras postponed · doorbell being rebuilt | [Failures](l3/edge-failures.md) |
|
||||
|
||||
## Trade-offs and limits
|
||||
|
||||
- **Power decides the design.** Deep sleep means a device is unreachable for most of its life, so every routine has to tolerate a node that is asleep. That constraint is why radio wake exists at all.
|
||||
- **Range over bandwidth.** LoRa is slow and payload-limited. It is chosen where a hundred bytes a day is enough and the alternative is a Wi-Fi access point in a garden bed.
|
||||
- **Some things are still manual.** The garden automation is partly manual — some watering and lighting is scheduled, some is a person turning something on. The page says which is which instead of implying full coverage.
|
||||
- **The failures cost money and are kept.** A lizard destroyed a feeder built for it. The garden cameras were postponed with the parts already bought. The doorbell is being rebuilt rather than declared finished. Those are the most honest evidence of production experience on this site.
|
||||
|
||||
Individual nodes carry their own label, and the ones being rebuilt or postponed are labelled that way on their pages.
|
||||
16
content/edge/edge-overview.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
id: edge-overview
|
||||
area: edge
|
||||
level: 1
|
||||
title: "Edge & Devices"
|
||||
claim: "Sam builds and maintains a twenty-node edge fleet — constrained devices on solar power and deep sleep, with custom radio and sensors — that reports into a single automation layer."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [ESP32, LoRa, Zigbee, MQTT, Home Assistant, Snapcast, Raspberry Pi]
|
||||
metrics: [{label: "edge nodes", value: 20}, {label: "lighting zones", value: 9}, {label: "automations", value: 20}]
|
||||
artefacts: [{type: doc, label: "Edge architecture — power · radio · deep sleep", url: "content/edge/l3/edge-architecture.md"}, {type: doc, label: "The failures, kept in the record", url: "content/edge/l3/edge-failures.md"}, {type: doc, label: "Distributed audio — eight endpoints with ducking", url: "content/edge/l3/edge-snapcast.md"}, {type: doc, label: "Vehicle tracker on LTE Cat-M", url: "content/edge/l3/edge-car-tracker.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, iot, esp32, lora, zigbee]
|
||||
---
|
||||
The devices are outdoors on solar and asleep most of their lives, and they have to keep working when nobody is looking; what failed is written down. The architecture, the radio and the failures are on the [fleet page](edge-fleet.md).
|
||||
29
content/edge/l3/edge-architecture.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: edge-architecture
|
||||
area: edge
|
||||
level: 3
|
||||
title: "Artefact — Edge architecture · power · radio · deep sleep"
|
||||
claim: "Each node is designed around its power budget — battery and solar · deep sleep between wakes · and a deliberate choice of wake source per job."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [ESP32, LoRa, Zigbee, LiFePO, MQTT, Home Assistant]
|
||||
metrics: [{label: "edge nodes", value: 20}, {label: "ESP32 devices", value: 10}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, architecture, power, artefact]
|
||||
---
|
||||
|
||||
One annotated diagram of a representative node and the radio path lands in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The per-node pattern: 240V step-down or solar with a LiFePO battery and charge controller, an ESP32 in deep sleep, a wake source (timer, radio, or a physical input), a relay or driver for the load, and an MQTT message into the automation layer. Around it, the reason each node uses LoRa, Zigbee or Wi-Fi.
|
||||
|
||||
## What it proves
|
||||
|
||||
Power budgeting and remote wake as design decisions, not hobby wiring — and an architecture that survived being repeated across twenty nodes.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Source: evidence notes section 1.
|
||||
51
content/edge/l3/edge-car-tracker.md
Normal file
@@ -0,0 +1,51 @@
|
||||
---
|
||||
id: edge-car-tracker
|
||||
area: edge
|
||||
level: 3
|
||||
title: "Artefact — Vehicle tracker on LTE Cat-M"
|
||||
claim: "A tracker that sleeps between reports · wakes on a schedule or when the car moves · and reports when it leaves a boundary set from the web interface."
|
||||
confidence: in-progress
|
||||
period: "2024–"
|
||||
tech: [ESP32, LTE Cat-M, NB-IoT, SIM7080G, GPS, LiFePO4]
|
||||
metrics: [{label: "battery packs", value: 2}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
tags: [edge, tracker, lte, low-power, artefact]
|
||||
---
|
||||
|
||||
The build is on the bench: two LiFePO batteries, a buck converter, an ESP32 with GPS, and a
|
||||
cellular module for the uplink. The car keeps the packs charged while it is running.
|
||||
|
||||

|
||||
|
||||
## What it does
|
||||
|
||||
Two behaviours, chosen by the settings rather than hard-coded. When the car is parked the
|
||||
ESP32 sits in deep sleep, wakes on a timer, checks its own configuration, and reports only
|
||||
what the settings ask for. When the car is moving it stays awake and reports on a shorter
|
||||
interval. Either way the interval and the boundary can be changed from the web interface,
|
||||
and a change can force an immediate update rather than waiting for the next wake.
|
||||
|
||||
The geofence is the part that matters. A boundary is drawn from the web interface, and the
|
||||
device is responsible for deciding whether it has left it — which is why the settings have
|
||||
to travel to the device rather than live only in the server.
|
||||
|
||||
## Why LTE Cat-M rather than WiFi
|
||||
|
||||
The tracker lives in a car. WiFi coverage is a driveway, not a road. Cat-M and NB-IoT were
|
||||
chosen because they are designed for exactly this shape of traffic — small payloads, long
|
||||
sleeps, coverage beyond what 4G handsets need — and because the modules are cheap enough to
|
||||
leave installed.
|
||||
|
||||
## What it proves
|
||||
|
||||
A complete path rather than a demo: constrained hardware, a power budget that dictates the
|
||||
schedule, a device that holds enough state to act on its own, and a web application that
|
||||
configures it and shows where it has been.
|
||||
|
||||
## Still open
|
||||
|
||||
Field testing across the full battery cycle. The sleep interval that is right for a parked
|
||||
car and the interval that is right for a moving one are guesses until the discharge curve
|
||||
has been measured over a week rather than a day.
|
||||
37
content/edge/l3/edge-failures.md
Normal file
@@ -0,0 +1,37 @@
|
||||
---
|
||||
id: edge-failures
|
||||
area: edge
|
||||
level: 3
|
||||
title: "Artefact — The failures · kept in the record"
|
||||
claim: "Three things did not go to plan — a feeder the animal destroyed · cameras postponed with the parts already bought · and a doorbell being rebuilt instead of declared finished."
|
||||
confidence: in-progress
|
||||
period: "2020–"
|
||||
tech: [ESP32, Servo, PWM, Camera]
|
||||
metrics: [{label: "crickets released per day by the feeder", value: 10}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, failures, artefact]
|
||||
---
|
||||
|
||||
Two items are being rebuilt, one is postponed. The page is not finished because the work is not finished.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
A short, unpolished register of what did not work, in the order it broke.
|
||||
|
||||
**The lizard feeder.** Servo motors on a 16-channel PWM driver, rubber-band spring-loaded boxes, an ESP32 — one box opened per day to release ten crickets for the bearded dragon it was built for. The lizard attacked the boxes and destroyed the mechanism. Version one is abandoned; it is being rebuilt with plastic containers.
|
||||
|
||||
**The garden cameras.** Postponed. The parts have arrived and are sitting in a box. No photographs and no claim until they are installed.
|
||||
|
||||
**The front door.** Built, then found to greet people who were not at the door. It is being rebuilt with a time-of-flight ranging sensor rather than being described as complete.
|
||||
|
||||
**The garden automation.** Partly manual. Some of it runs on schedule and some of it is a person. Both are true and the page says so.
|
||||
|
||||
## What it proves
|
||||
|
||||
That this is a working estate and not a rendering: things break, budgets get spent on parts that are not installed yet, and hardware has to be designed around an animal's behaviour. Production experience is measured in rebuilds as much as in launches.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7, with photographs of the rebuilt feeder and the installed cameras if they exist by then. If they do not, the entry stays as written — labelled rather than deleted.
|
||||
33
content/edge/l3/edge-front-door.md
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: edge-front-door
|
||||
area: edge
|
||||
level: 3
|
||||
title: "Artefact — Front door · vision · voice · and a rebuild"
|
||||
claim: "A doorbell that sees a person · describes the image · speaks to the house and to the visitor · and is currently being rebuilt with a time-of-flight sensor."
|
||||
confidence: in-progress
|
||||
period: "2020–"
|
||||
tech: [ESP32, ESP32-CAM, PIR, mmWave, VL53L0X, MiniMax, Piper, Whisper, Apprise]
|
||||
metrics: [{label: "voice engines in use", value: 2}, {label: "presence sensing inputs", value: "PIR · mmWave · time-of-flight"}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, doorbell, vision, voice, artefact]
|
||||
---
|
||||
|
||||
The device was built and is being rebuilt. The page states this rather than describing it as finished.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
A door unit with microphone, speaker, camera, button, PIR and mmWave presence. It detects a person, captures an image, logs it, has the image described, and announces to the house who is at the door before offering a spoken greeting; the visitor's reply is captured and played over the house speakers, with an alert pushed to a phone. Voice output ran on one provider and now runs on a cloned voice with a fallback to a local synthesiser.
|
||||
|
||||
## What it proves
|
||||
|
||||
Vision, voice and actuation in one device under real constraints — latency that a person at a door will tolerate, and a failure mode that must be polite.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
The measurement it is being rebuilt around: presence detection with a time-of-flight ranging sensor, to stop the unit greeting people who are walking past rather than standing at the door. Until the rebuild is finished the honest label is `in-progress`.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7 for photos and the sensor change. Source: evidence notes section 1.
|
||||
33
content/edge/l3/edge-lora-sprinkler.md
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: edge-lora-sprinkler
|
||||
area: edge
|
||||
level: 3
|
||||
title: "Artefact — LoRa irrigation under solar power"
|
||||
claim: "A solar-powered controller that sleeps · wakes on radio · and drives a solenoid — with the first version polling over a message queue instead."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [ESP32, LoRa, LiFePO, Solenoid, Home Assistant, RabbitMQ]
|
||||
metrics: [{label: "major revisions", value: 2}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, lora, solar, irrigation, artefact]
|
||||
---
|
||||
|
||||
The bill of materials and the wake-on-radio change land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The build: solar panel, charge controller, LiFePO battery, ESP32, LoRa module, relay and solenoid valve. Version one woke on a schedule and polled through a message queue to learn whether it should run. Version two is better: a LoRa packet can wake the ESP32 from deep sleep and switch the valve directly, so on-time is short and the battery is not spent asking whether there is work.
|
||||
|
||||
## What it proves
|
||||
|
||||
Iteration on a real constraint. The first version worked; the second version was designed around the power cost of asking, which is the kind of decision you only make if you are watching the battery.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
Watering is partly scheduled and partly a person. The routine lives in the automation layer and the voice assistant can trigger it, but the garden is not fully automatic and the page does not pretend otherwise.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Source: evidence notes section 1.
|
||||
29
content/edge/l3/edge-sensors.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: edge-sensors
|
||||
area: edge
|
||||
level: 3
|
||||
title: "Artefact — Sensors and actuation"
|
||||
claim: "Movement · presence · weight · temperature · water and mechanical actuation — chosen per job and integrated into one automation layer."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [ESP32, PIR, Radar, mmWave, Load cell, Servo, Solenoid, MQTT]
|
||||
metrics: [{label: "automations", value: 20}, {label: "PWM channels on the feeder driver", value: 16}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, sensors, actuation, artefact]
|
||||
---
|
||||
|
||||
A sensor and actuator table lands in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The parts list of the fleet in the way an engineer would want it: what is sensed, which device senses it, what is actuated, and how the decision reaches a physical action — a relay, a solenoid, a servo, a step-down converter switching 240V, a wheel that flicks food into a tank, an auger that dispenses it, a load cell that decides when to top up water.
|
||||
|
||||
## What it proves
|
||||
|
||||
Range across the awkward middle of embedded work: not just reading a sensor but choosing an actuator, sizing its driver, and making the whole loop recoverable when the network is absent.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Source: evidence notes section 1.
|
||||
29
content/edge/l3/edge-snapcast.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: edge-snapcast
|
||||
area: edge
|
||||
level: 3
|
||||
title: "Artefact — Distributed audio · eight endpoints with ducking"
|
||||
claim: "Eight synchronised endpoints with music ducked under every spoken announcement and restored afterwards — driven by Home Assistant and the voice assistant."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [Snapcast, Mopidy, Spotify, Raspberry Pi, Home Assistant]
|
||||
metrics: [{label: "audio endpoints", value: 8}, {label: "client hardware", value: "Pi 2W"}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, audio, snapcast, artefact]
|
||||
---
|
||||
|
||||
The topology diagram and a short recording of ducking land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
A Snapserver with eight Pi clients, Spotify and a local music server as sources, and one behaviour that matters more than the rest: when the house needs to speak, the music drops and the announcement plays everywhere, then the music returns to its previous level. Requests arrive by voice, so the same system that plays music is also the output for every automation and every answer.
|
||||
|
||||
## What it proves
|
||||
|
||||
Multi-room audio treated as infrastructure with a predictable interruption behaviour — and the difference between "the speakers work" and "the speakers can be interrupted safely". Mains-powered nodes, bandwidth-heavy work, and a state machine for volume: the opposite design constraints to the battery nodes.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Source: evidence notes section 1.
|
||||
29
content/edge/l3/edge-zigbee.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: edge-zigbee
|
||||
area: edge
|
||||
level: 3
|
||||
title: "Artefact — Hand-built Zigbee routers and lighting"
|
||||
claim: "Two routers built by hand to extend an indoor mesh · feeding nine lighting zones with presence sensing and voice control."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [ESP32, Zigbee, Relay, PIR, Radar, Home Assistant]
|
||||
metrics: [{label: "hand-built Zigbee routers", value: 2}, {label: "lighting zones", value: 9}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, zigbee, lighting, artefact]
|
||||
---
|
||||
|
||||
The router build and a zone map land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
Two Zigbee routers on ESP32 hardware, built because the commercial mesh did not cover the house and a repeater would have been a black box. Behind them, nine lighting zones: custom presence sensors with PIR and radar for the bathroom and a child's bedroom, a central light controller with a readout and a potentiometer, and a three-button remote with a dial — click on, double click off, dial for level.
|
||||
|
||||
## What it proves
|
||||
|
||||
Mesh networking understood at the level where you can add a node instead of buying one, and lighting built for how a house is used rather than for a product category.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Source: evidence notes section 1.
|
||||
53
content/history/history-engagements.md
Normal file
@@ -0,0 +1,53 @@
|
||||
---
|
||||
id: history-engagements
|
||||
area: history
|
||||
level: 2
|
||||
title: "Fifteen years of delivery for agencies and their clients"
|
||||
claim: "Twenty-two agencies and twenty-three brands · national campaigns · bank and broadcaster platforms · betting front-ends · and a habit of being called in to rescue work someone else could not finish."
|
||||
confidence: historic
|
||||
period: "1999–2014"
|
||||
tech: [ActionScript, PHP, JavaScript, MySQL, XML, SOAP, React, Python]
|
||||
metrics: [{label: "agencies on the client wall", value: 22}, {label: "brands delivered for", value: 23}, {label: "code exhibits kept", value: 6}]
|
||||
artefacts: [{type: doc, label: "The client wall", url: "content/history/l3/history-client-wall.md"}, {type: doc, label: "Six code exhibits across five eras", url: "content/history/l3/history-code-exhibits.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [history, agencies, brands, exhibits]
|
||||
---
|
||||
|
||||
**Capability statement.** Range under deadline. The work in this period ran from car classifieds and online betting to a bank's pitch piece, a broadcaster's on-demand banners and a publisher's education platform — usually with a date attached that could not move.
|
||||
|
||||
## The eras, briefly
|
||||
|
||||
**1999–2002 — the start.** Web design and development for a high-volume car marketplace, then freelance work building sites, applications, interfaces and database back ends for small agencies.
|
||||
|
||||
**2002–2005 — London, full time.** Online gaming: lead front-end developer on a synchronous multi-game platform handling a thousand simultaneous clients, integrating against Java back ends and XML sockets. Then the emerging interactive advertising industry, where a creative front end with a technical spine was scarce.
|
||||
|
||||
**2006–2009 — the agency run.** The bulk of the client wall: bank pitch work delivered in weeks, broadcaster promotions, a global oil company's re-brand site, premium drinks campaigns, a news organisation's media presentation layer, a publisher's education platform, and a global computer-aided design vendor's localised site. Several of these were rescues — brought in on a short deadline to rewrite work another team could not land.
|
||||
|
||||
**2009–2012 — product and campaign.** A standalone encrypted presentation builder with its own media library and remote update path. A national beer campaign with video submissions, cross-brand promotion, in-store mechanics and authorisation over SOAP. Interactive launches with 3D content in the browser.
|
||||
|
||||
**2012–2014 — small products for small business.** Online book building, a reactive design system, and art applications for a motion-sensing controller, transitioned toward mobile.
|
||||
|
||||
**2014–2020 — the workshop years.** The site builder and the signage product (see [Products](../products/products-business.md)), which is where the work moved from delivering campaigns for other people's brands to owning software and supporting customers.
|
||||
|
||||
## Languages across the eras
|
||||
|
||||
The point of the exhibit set is that the tooling changed repeatedly and the delivery discipline did not: ActionScript 3 with class-based architecture, hand-written JavaScript engines, PHP media pipelines, TypeScript with typed service layers and a relational schema, embedded C++ for firmware, and now declarative infrastructure as its own kind of code.
|
||||
|
||||
## Evidence
|
||||
|
||||
| Claim | Evidence | Artefact |
|
||||
|---|---|---|
|
||||
| Twenty-two agencies and twenty-three brands | The client wall, carried from the résumé and the recovered site content | [Client wall](l3/history-client-wall.md) |
|
||||
| Work across six technology eras | Six exhibits from OO ActionScript to a hand-written Nix package | [Code exhibits](l3/history-code-exhibits.md) |
|
||||
| Rescue work under immovable deadlines | Multiple engagements recorded as rewrites of stalled projects | [Client wall](l3/history-client-wall.md) |
|
||||
| Sponsorship of the later product era | 2014–2020 product work described in the résumé | [Products](../products/products-business.md) |
|
||||
|
||||
## Trade-offs and limits
|
||||
|
||||
- **Historic work is labelled historic.** Nothing in this section is presented as current capability; where a technique is still used today, the page links to where it is used.
|
||||
- **One engagement has no screenshot.** The national archive work was an internal tool. It is described and not illustrated, because there is nothing legitimately publishable to show.
|
||||
- **No client has been named beyond a brand.** Brands are named because they were public campaigns. Individuals, internal contacts and unpublished commercial detail are excluded.
|
||||
- **The exhibits are excerpts.** Around forty lines each, chosen to show judgement — structure, error handling, an exit plan — not to demonstrate volume.
|
||||
|
||||
This work is finished, and it is presented in the past tense throughout.
|
||||
16
content/history/history-overview.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
id: history-overview
|
||||
area: history
|
||||
level: 1
|
||||
title: "History"
|
||||
claim: "Sam delivered interactive work for twenty-two agencies and twenty-three brands between 1999 and 2014 — the commercial apprenticeship that produced the range."
|
||||
confidence: historic
|
||||
period: "1999–2014"
|
||||
tech: [ActionScript, PHP, JavaScript, MySQL, XML, React, Python, Go, HTMX]
|
||||
metrics: [{label: "agencies on the client wall", value: 22}, {label: "brands delivered for", value: 23}, {label: "code exhibits kept", value: 6}]
|
||||
artefacts: [{type: doc, label: "The client wall", url: "content/history/l3/history-client-wall.md"}, {type: doc, label: "Six code exhibits across five eras", url: "content/history/l3/history-code-exhibits.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [history, agencies, brands, career]
|
||||
---
|
||||
The languages changed six times across the period — ActionScript, PHP, JavaScript, React, Python, and now Go with HTMX — and the exhibits are chosen to show that rather than to show a decade. The era-by-era record is on the [engagement record](history-engagements.md).
|
||||
90
content/history/l3/history-client-wall.md
Normal file
@@ -0,0 +1,90 @@
|
||||
---
|
||||
id: history-client-wall
|
||||
area: history
|
||||
level: 3
|
||||
title: "Artefact — The client wall"
|
||||
claim: "Twenty-two agencies and twenty-three brands — the delivery record carried from the résumé rather than reconstructed from memory."
|
||||
confidence: historic
|
||||
period: "1999–2014"
|
||||
tech: [ActionScript, PHP, JavaScript, MySQL, XML]
|
||||
metrics: [{label: "agencies on the client wall", value: 22}, {label: "brands on the client wall", value: 23}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [history, clients, artefact]
|
||||
---
|
||||
|
||||
The names are final. The treatment is deliberate: one typeface, one size, one grid, and no logo collage — see *Why there are no logos here* below.
|
||||
|
||||
## The agencies
|
||||
|
||||
<div class="client-wall" role="list" aria-label="Agencies">
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Clemenger</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Agency Republic</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Brand Union</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Jackpot Joy</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Bryce Ford</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">RMG Connect</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Sporting Index</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Adventis Digital</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Trinity Street</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Inferno</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">SAS Design</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Ogilvy</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Wunderman</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Illumina</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Barkers</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Mook</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Pearsons</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Penguin</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Draft London</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Behan Consulting</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">MCM Entertainment</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Carsales</span></div>
|
||||
</div>
|
||||
|
||||
## The brands
|
||||
|
||||
<div class="client-wall" role="list" aria-label="Brands">
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Carlton Draught</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Diageo</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">KPMG</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">McDonald's</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Accenture</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">BBC National Archive</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Bank of America</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">War of the Worlds</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Madonna (Hard Candy)</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">PlayStation</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">HSBC</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">DHL</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Reuters</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Volvic</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">AutoCAD</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Channel 4</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Pearsons</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Shell</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Stella Artois</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Baileys Irish Cream</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Parkgate Homes</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Carlton Dry</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Cottees Cordial</span></div>
|
||||
</div>
|
||||
|
||||
## Why there are no logos here
|
||||
|
||||
A wall of mixed marks — some full colour, some monochrome, some missing — reads as broken, and the eight marks that exist for these names are not the eight the work is best remembered by. One typeface, one size and one grid says the same thing more clearly, and it sits beside the technical pages without competing with them. A mark is added only when it is the client's official mark and Sam has approved its use; none has been invented.
|
||||
|
||||
## Corrections carried into this list
|
||||
|
||||
- **Diageo**, not the "Diajio" spelling that appeared in the recovered site content.
|
||||
- **Baileys Irish Cream**, stated in full and spelled correctly.
|
||||
- The **BBC National Archive** engagement was an internal tool. It is described, and it is not illustrated with a screenshot, because no legitimately publishable image exists.
|
||||
|
||||
## Count discrepancy — recorded, not smoothed over
|
||||
|
||||
The summary material said twenty-three agencies. The recovered, verified list holds **twenty-two** names, and the count published above is the count of the list. If a name is missing, the number moves to twenty-three and this note is deleted; until then the smaller, checkable number is the one on the page.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Delivered as a body of names and descriptions, and presented as a typographic wall from Phase 7. Names and one-line descriptions are approved as recovered and corrected.
|
||||
46
content/history/l3/history-code-exhibits.md
Normal file
@@ -0,0 +1,46 @@
|
||||
---
|
||||
id: history-code-exhibits
|
||||
area: history
|
||||
level: 3
|
||||
title: "Artefact — Six code exhibits across five eras"
|
||||
claim: "Six short excerpts spanning object-oriented ActionScript · a hand-written JavaScript animation engine · a PHP media pipeline · typed TypeScript · embedded C++ · and declarative Nix."
|
||||
confidence: historic
|
||||
period: "2006–2026"
|
||||
tech: [ActionScript, JavaScript, PHP, TypeScript, C++, Nix]
|
||||
metrics: [{label: "code exhibits", value: 6}, {label: "technology eras spanned", value: 5}]
|
||||
artefacts: [{type: repo, label: "ActionScript 3 — the Siren class", url: "content/assets/code/01-actionscript-class.as"}]
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
tags: [history, code, exhibits, artefact]
|
||||
---
|
||||
|
||||
**Status: published.** This page shows the ActionScript class and indexes the other five, each of which is exhibited on the page where it belongs.
|
||||
|
||||
## The six
|
||||
|
||||
| # | Era | Exhibit | Where it is shown |
|
||||
|---|---|---|---|
|
||||
| 1 | Object-oriented ActionScript | `01-actionscript-class.as` — class, imports, encapsulation | this page |
|
||||
| 2 | Hand-written JavaScript engine | `02-js-easing-core.js` — easing functions and the timeline loop | [the site builder](../../products/l3/products-websitebuilder.md) |
|
||||
| 3 | PHP media pipeline | `03-php-encode-op.php` — one GIF encode operation and its FFmpeg commands | [the site builder](../../products/l3/products-websitebuilder.md) |
|
||||
| 4 | TypeScript | `04-typescript-service-layer.ts` — a typed tRPC boundary | [the estate](../../platform/platform-estate.md) |
|
||||
| 5 | Embedded C++ | `05-embedded-cpp-interface.hpp` — interface split from implementation | [the edge fleet](../../edge/edge-fleet.md) |
|
||||
| 6 | Declarative infrastructure | `06-nix-derivation.nix` — pinned version, verified hash, exit condition | [the Nix package](../../platform/l3/platform-tuios-package.md) |
|
||||
|
||||
Total is 243 lines across twenty years, which is the point: each excerpt is chosen to show judgement rather than volume.
|
||||
|
||||
## Why this exhibit is good code
|
||||
|
||||
Every field is private and explicitly typed, and the public surface is two methods, so a caller cannot push the object into a state it does not understand. The constructor takes what it needs as parameters instead of reaching for it, which is why the same class can drive every siren on the screen without knowing where any of them are.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
The excerpt is verbatim, including a latent typo (`this.high` rather than `this.height`). It is left as found: the excerpt is evidence, not a portfolio piece.
|
||||
|
||||
## What it proves
|
||||
|
||||
That the stack changed six times and the discipline did not. Exhibit 6 is the least common of the six and the most current — a short, correct, self-documenting derivation with an exit plan is a stronger infrastructure signal than a long configuration file.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Published. The six files are in this repository under `content/assets/code/`; the archive path is withheld. The other five are linked above from the pages that carry their claims.
|
||||
33
content/platform/l3/platform-backups.md
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: platform-backups
|
||||
area: platform
|
||||
level: 3
|
||||
title: "Artefact — Backups and restore runbook"
|
||||
claim: "Three tiers — local snapshots · a copy on another machine · an offsite copy — with retention stated in the job and a restore runbook kept beside them."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [Borg, Kopia, Backrest, AWS S3, systemd, NFS]
|
||||
metrics: [{label: "backup tiers", value: 3}, {label: "retention windows in the primary job", value: 3}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [platform, backups, restore, artefact]
|
||||
---
|
||||
|
||||
Redacted job definitions and the restore runbook land in Phase 7. Passphrase locations and remote paths are removed before publication.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The backup design: Borg jobs on the two desktop hosts to local storage with a network copy, Kopia and Backrest on the edge host to network storage, an offsite copy in object storage, and retention expressed in the job (seven daily, four weekly, three monthly, plus annual on the edge). Beside it, the restore runbook and the list of what is deliberately not covered.
|
||||
|
||||
## What it proves
|
||||
|
||||
That failure was designed for: three tiers cover disk loss, machine loss and site loss, and the restore path is written down rather than recovered from memory at the worst possible moment.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
Not everything is covered, and the page says so — one storage host is itself a single point of failure for some repositories, and the file server has no backup of its own. Those are known gaps with owners, not oversights to be hidden.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Source: the infrastructure reference and the backup system notes, read 2026-10-26.
|
||||
29
content/platform/l3/platform-containers.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: platform-containers
|
||||
area: platform
|
||||
level: 3
|
||||
title: "Artefact — Container inventory"
|
||||
claim: "Fifty or more containers across the estate — each with its own port · its own folder · and a memory cap equal to its swap cap."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [Docker, Compose, Caddy, Proxmox]
|
||||
metrics: [{label: "containers in the estate", value: 50}, {label: "memory cap rule", value: "mem_limit equals memswap_limit"}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [platform, docker, artefact]
|
||||
---
|
||||
|
||||
A digest of the inventory — grouped by purpose, with addresses removed — lands in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The inventory generated from the hosts, grouped by purpose: media and file services, home automation, the AI services, the public edge, and the build tooling. For each service the same three facts are recorded — port, configuration folder, memory cap.
|
||||
|
||||
## What it proves
|
||||
|
||||
That the estate is inventoried rather than accumulated, and that the resource rules are applied per service instead of being a policy on a wiki page.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7, generated from the live hosts. Counts follow the approved metric set; the exact number is regenerated at capture time.
|
||||
35
content/platform/l3/platform-network.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
id: platform-network
|
||||
area: platform
|
||||
level: 3
|
||||
title: "Artefact — Network layout and remote access"
|
||||
claim: "A private mesh and SSH everywhere · services bound to loopback · and one public listener at the edge terminating TLS for the sites that are allowed to be seen."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [Tailscale, SSH, Caddy, DNS, Termux]
|
||||
metrics: [{label: "public listeners per service", value: 1}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [platform, networking, security, artefact]
|
||||
---
|
||||
|
||||
The diagram below shows three hosts and one trust boundary; internal addresses, hostnames and public DNS names are generalised, not published.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The layout: hosts on a private mesh for remote access, SSH on every machine, a phone used as a terminal, and an edge host that terminates TLS and proxies to loopback-bound services. New services bind to loopback by default; making something public is an explicit edge change, not a side effect of starting a container.
|
||||
|
||||
## What it proves
|
||||
|
||||
Security posture chosen rather than inherited: no service is reachable because it happened to bind to all interfaces, and there is exactly one place where the outside world meets the estate.
|
||||
|
||||
## The layout
|
||||
|
||||

|
||||
|
||||
Read it left to right: work is built on one machine, runs on a memory-bound machine with every service bound to loopback, and is published by a single edge that terminates TLS. The dashed boundary is the point of the whole arrangement — crossing it is a deliberate change to the edge configuration, not a side effect of starting a container. Below both, the mesh carries remote access and the second backup tier, so neither depends on a port being open.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Delivered. Internal addressing, hostnames and public DNS names are excluded from publication by the redaction pass.
|
||||
33
content/platform/l3/platform-nixos.md
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: platform-nixos
|
||||
area: platform
|
||||
level: 3
|
||||
title: "Artefact — Declarative multi-host configuration"
|
||||
claim: "Machines are described rather than assembled — a shared NixOS base · a replicated desktop and terminal · and Proxmox underneath the virtualised side."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [NixOS, Niri, Proxmox, Neovim, Zsh, Ghostty]
|
||||
metrics: [{label: "hosts on the declarative base", value: 4}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [platform, nixos, artefact]
|
||||
---
|
||||
|
||||
A redacted excerpt of the configuration tree lands in Phase 7. No hostnames, addresses or keys are published.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The shape of the configuration: one shared base, per-host modules, and the parts that make the desktop and the terminal identical everywhere — window manager, editor, shell and the plugins that carry the work. Proxmox hosts the virtualised side, including the file server and the home automation appliance.
|
||||
|
||||
## What it proves
|
||||
|
||||
That a machine can be rebuilt from a description, and that the same environment follows the person between machines instead of being re-created by memory.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
The terminal-first setup is deliberate: a heavier desktop was downgraded to the terminal on one machine for memory headroom. A phone-as-server experiment is on hold, not shipped, and is described as on hold.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Source facts: evidence notes section 3.
|
||||
29
content/platform/l3/platform-signage.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
id: platform-signage
|
||||
area: platform
|
||||
level: 3
|
||||
title: "Artefact — Signage infrastructure · screens as a managed fleet"
|
||||
claim: "HDMI-CEC control of power · rotation · mirroring and multi-screen layout — so a display fleet is driven by the platform rather than by a remote control."
|
||||
confidence: delivered
|
||||
period: "2014–"
|
||||
tech: [HDMI-CEC, Raspberry Pi, Python, JavaScript, FFmpeg]
|
||||
metrics: [{label: "commercial deployments evidenced", value: "salon · financial services business"}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [platform, signage, hdmi-cec, artefact]
|
||||
---
|
||||
|
||||
The CEC command set and a layout diagram land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The platform side of the signage product: the codes and control path that switch screens on and off, rotate and mirror output, and lay out multi-screen displays, including synchronised playback across screens in one site.
|
||||
|
||||
## What it proves
|
||||
|
||||
Device-level control combined with fleet management. This is the same skill the edge pages show on small devices, applied to physical displays that other people depend on.
|
||||
|
||||
## Confidence
|
||||
|
||||
`delivered` — this ran in commercial deployments. The product's business status is stated separately on the Products pages, where the honest position is that the software is being rebuilt and three businesses still run it.
|
||||
33
content/platform/l3/platform-tuios-package.md
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: platform-tuios-package
|
||||
area: platform
|
||||
level: 3
|
||||
title: "Artefact — A hand-written Nix package with an exit plan"
|
||||
claim: "Twenty-five lines of Nix that pin a tool at 0.10.1 over nixpkgs' 0.7.0 — with a verified hash · a written reason and a written condition for deleting the file."
|
||||
confidence: running
|
||||
period: "2026–"
|
||||
tech: [Nix, NixOS]
|
||||
metrics: [{label: "pinned version", value: "0.10.1"}, {label: "nixpkgs version it overrides", value: "0.7.0"}]
|
||||
artefacts: [{type: repo, label: "Nix derivation — tuios 0.10.1 override with a removal condition", url: "content/assets/code/06-nix-derivation.nix"}]
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
tags: [platform, nix, packaging, artefact]
|
||||
---
|
||||
|
||||
**Status: published.** The whole file is shown verbatim.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
A package definition for the agent operations console: `fetchurl` against a released artifact with a verified hash, a documented reason for overriding nixpkgs, an `installPhase` that stages the binary, and a comment stating the condition under which the file should be deleted.
|
||||
|
||||
## What it proves
|
||||
|
||||
More than knowing Nix exists. The signals are that someone noticed a version gap, fixed it locally instead of waiting, pinned it so the fleet is reproducible, and wrote down when to remove their own patch. Knowing when to delete your own override is the senior part.
|
||||
|
||||
## Why this exhibit is good code
|
||||
|
||||
The override records why it exists and the exact condition under which it should be deleted, which is the part of a local patch that usually goes missing. Pinning a released binary by `fetchurl` with a verified hash keeps the result reproducible without waiting for nixpkgs.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Published. The file is in this repository at `content/assets/code/06-nix-derivation.nix`; the infrastructure repository remains the source. Source fact: content spine section 4.2.
|
||||
55
content/platform/platform-estate.md
Normal file
@@ -0,0 +1,55 @@
|
||||
---
|
||||
id: platform-estate
|
||||
area: platform
|
||||
level: 2
|
||||
title: "A small estate run to a documented standard"
|
||||
claim: "Fifty or more containers on a four-host estate, all on declarative configuration — with packaging discipline · encrypted backups and a restore runbook · loopback-bound services · and one public listener."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [NixOS, Proxmox, Docker, Caddy, Tailscale, Borg, Kopia, AWS S3, HDMI-CEC]
|
||||
metrics: [{label: "containers in the estate", value: 50}, {label: "hosts", value: 4}, {label: "backup tiers", value: "Borg · Kopia · AWS"}, {label: "public listeners", value: 1}]
|
||||
artefacts: [{type: doc, label: "Declarative multi-host configuration", url: "content/platform/l3/platform-nixos.md"}, {type: doc, label: "A hand-written Nix package with an exit plan", url: "content/platform/l3/platform-tuios-package.md"}, {type: doc, label: "Container inventory", url: "content/platform/l3/platform-containers.md"}, {type: doc, label: "Backups and restore runbook", url: "content/platform/l3/platform-backups.md"}, {type: doc, label: "Network layout and remote access", url: "content/platform/l3/platform-network.md"}, {type: doc, label: "Signage infrastructure — screens as a managed fleet", url: "content/platform/l3/platform-signage.md"}, {type: repo, label: "TypeScript — a typed tRPC service boundary", url: "content/assets/code/04-typescript-service-layer.ts"}]
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
tags: [platform, devops, nixos, docker, backups, networking]
|
||||
---
|
||||
|
||||
**Capability statement.** Four hosts, fifty or more containers, configuration in version control instead of in a shell history, and one place where traffic enters. The estate is deliberately small because it is run rather than demonstrated.
|
||||
|
||||
## What is actually run
|
||||
|
||||
**Declarative configuration.** NixOS is the base on the machines that matter, with the desktop and the terminal replicated across them so an environment can be rebuilt from a description. Proxmox runs the virtualised side — Ubuntu file server and a Home Assistant appliance alongside the desktop hosts. The point of the arrangement is repeatability: a machine is a configuration, not a pet.
|
||||
|
||||
**Packaging.** When a tool is missing or stale in nixpkgs, it gets a derivation rather than a manual install. The TUIOS package pins 0.10.1 over nixpkgs' 0.7.0 with `fetchurl` and a verified hash, documents why the override exists, and states the condition for deleting it. A patch with an exit plan is maintenance; a patch without one is debt.
|
||||
|
||||
**Container estate.** Fifty or more containers run across the hosts — the file server and media services, the automation layer, the AI services, traffic for public ingress, and the tools used to build all of it. Each new service is given its own port, its own folder and a memory cap equal to its swap cap, so it cannot take the host's swap.
|
||||
|
||||
**Networking.** Remote access is a private mesh rather than an exposed SSH port: Tailscale across the machines, SSH on every host, and a phone used as a terminal. Services bind to loopback; Caddy is the only public listener, terminating TLS at the edge host and proxying the sites that are allowed to be seen.
|
||||
|
||||
**Storage and backups.** Three tiers, each covering a different failure: local snapshots on the machine, a copy on another machine over the network, and an offsite copy in object storage. Retention is stated in the job (daily, weekly, monthly, annual) and a restore runbook sits next to the jobs, because a backup nobody has restored from is an assumption.
|
||||
|
||||
**Observability and operations.** Resource use is watched at the host level, because the runtime host is memory-bound and a new service can destabilise the ones people rely on. The agent fleet is observed as a fleet through TUIOS — sessions, state and approvals in one console — rather than by tailing logs on four machines.
|
||||
|
||||
**Signage infrastructure.** The display work is treated as a fleet to be managed: HDMI-CEC control for power, rotation, mirroring and multi-screen layout, so screens are driven by the platform rather than by a person with a remote control.
|
||||
|
||||
## Why this exhibit is good code
|
||||
|
||||
The Prisma schema defines the stored types, and the tRPC router validates input with Zod at the boundary, so the data layer receives a shape it can trust rather than one it hopes for. Each procedure is a thin call on the database context, which keeps the router a description of the API rather than a second copy of the queries.
|
||||
|
||||
## Evidence
|
||||
|
||||
| Claim | Evidence | Artefact |
|
||||
|---|---|---|
|
||||
| Configuration is declarative and replicated | NixOS hosts with shared configuration; Proxmox for the virtualised side | [Declarative configuration](l3/platform-nixos.md) |
|
||||
| Tooling is packaged rather than hand-installed | `pkgs/tuios.nix` pinning 0.10.1 with a documented removal condition | [Nix package](l3/platform-tuios-package.md) |
|
||||
| Fifty or more containers are operated | Inventory by host and by purpose | [Container inventory](l3/platform-containers.md) |
|
||||
| Backups are three-tier with a restore runbook | Borg on two hosts, Kopia and Backrest on the edge host, offsite object storage | [Backups](l3/platform-backups.md) |
|
||||
| One public listener | Loopback-bound services behind Caddy at the edge | [Network](l3/platform-network.md) |
|
||||
| Screens are managed as a fleet | HDMI-CEC control of power, rotation and layout | [Signage infrastructure](l3/platform-signage.md) |
|
||||
|
||||
## Trade-offs and limits
|
||||
|
||||
- **Memory over hardware.** The runtime host runs 15 GiB with little free and swap in use. The response was caps, fewer and smaller services, and moving heavy work to the gateway rather than buying another machine. Every new service is sized before it is deployed, and the ceiling for this site is one gigabyte and four containers.
|
||||
- **Self-hosting is a choice with a cost.** Monitoring, backups, ingress and the AI gateway are all operated here, which means the failure modes are also operated here. The mitigation is documentation and measurement, not optimism.
|
||||
- **Rejected on purpose.** A long list of tools was evaluated and refused — Airflow for RAM, self-hosted Langfuse for six containers it could not afford, LiteLLM because it would duplicate the existing gateway, and a pile of documentation platforms. "Alternatives considered" is kept because the reasoning is the senior artifact.
|
||||
- **Not everything is automated.** Some operations are still a documented command run by a person. The runbook exists precisely because the automations do not cover everything.
|
||||
16
content/platform/platform-overview.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
id: platform-overview
|
||||
area: platform
|
||||
level: 1
|
||||
title: "Platform & Infrastructure"
|
||||
claim: "Sam runs a multi-machine NixOS and Proxmox estate sized to the hardware rather than the budget: fifty-plus containers, declarative configuration, encrypted backups with a written restore runbook, and public ingress."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [NixOS, Proxmox, Docker, Caddy, Tailscale, Borg, Kopia, AWS S3]
|
||||
metrics: [{label: "containers in the estate", value: 50}, {label: "hosts", value: 4}, {label: "backup tiers", value: "Borg · Kopia · AWS"}]
|
||||
artefacts: [{type: doc, label: "Declarative multi-host configuration", url: "content/platform/l3/platform-nixos.md"}, {type: doc, label: "Backups and restore runbook", url: "content/platform/l3/platform-backups.md"}, {type: doc, label: "A hand-written Nix package with an exit plan", url: "content/platform/l3/platform-tuios-package.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [platform, devops, nixos, docker, backups]
|
||||
---
|
||||
The runtime host is memory-bound rather than CPU-bound, so every service carries a memory cap and is bound to loopback. What runs, where, and what was rejected is on the [estate page](platform-estate.md).
|
||||
33
content/products/l3/products-signage.md
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: products-signage
|
||||
area: products
|
||||
level: 3
|
||||
title: "Artefact — Commercial signage on commodity hardware"
|
||||
claim: "A digital signage platform built as market disruption — any HDMI screen · a small player · fleet control of power · rotation · layout and playback · sold to twelve businesses over time."
|
||||
confidence: delivered
|
||||
period: "2014–"
|
||||
tech: [Raspberry Pi, ESP32, HDMI-CEC, JavaScript, three.js, FFmpeg, Python]
|
||||
metrics: [{label: "businesses currently running the product", value: 3}, {label: "adoptions over time", value: 12}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [products, signage, artefact]
|
||||
---
|
||||
|
||||
The product shipped and customers still run it. The rebuild on the current stack is not finished and is labelled as unfinished everywhere it is mentioned.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The product in use: a player on commodity hardware — a small Raspberry Pi or a microcontroller class device — driving any HDMI screen, with fleet control from the platform: switch screens on and off, rotate, mirror, lay out multiple screens, coordinate boards, and synchronise playback across a site. Content is built by the customer from complex animation down to a simple image rotation, and the system adapts to the board it finds rather than demanding a specification.
|
||||
|
||||
## What it proves
|
||||
|
||||
The commercial read: a market was overpriced, the substitute was built on commodity hardware, and it was sold and operated for years. Twelve adoptions over time, three businesses still running. That is a product, not a project.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
Twelve is cumulative adoption and three is the current count — the distinction is kept because inflating it would be the easiest lie on this site. The rebuild is in progress. No revenue figures or customer names beyond type of business are published.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7 for screenshots and a demonstration recording, captured from the running product. The one-liners for reference customers are approved for the salon and the financial services business only.
|
||||
39
content/products/l3/products-websitebuilder.md
Normal file
@@ -0,0 +1,39 @@
|
||||
---
|
||||
id: products-websitebuilder
|
||||
area: products
|
||||
level: 3
|
||||
title: "Artefact — The site builder behind the signage"
|
||||
claim: "A templated site builder and CMS with media editors — drawing · image · video · text animation · 3D import — that also produced the marketing sites and the social output."
|
||||
confidence: delivered
|
||||
period: "2014–"
|
||||
tech: [PHP, JavaScript, three.js, FFmpeg, ImageMagick, MySQL]
|
||||
metrics: [{label: "media editors in the product", value: "drawing · image · video · text animation · transitions · 3D import"}]
|
||||
artefacts: [{type: repo, label: "Text-effects engine — easing functions and the timeline loop", url: "content/assets/code/02-js-easing-core.js"}, {type: repo, label: "Media encoder — one GIF encode operation and its FFmpeg commands", url: "content/assets/code/03-php-encode-op.php"}]
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
tags: [products, cms, media, artefact]
|
||||
---
|
||||
|
||||
**Status: delivered, retained.** Retired as a general-purpose product; kept as the content engine for the signage.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
The templated builder: pick a template, build the site, publish it — with editors for drawing, images, video and text animation, video effects and transitions, 3D object import and text extrusion, plus document and office-format handling. Image animation is composed in JavaScript and HTML and rendered on Raspberry Pi hardware through Python. Around the site itself sat the commercial scaffolding a small business actually needs: shop, invoicing, room and event management, presentations, document management, client records and bulk email and SMS.
|
||||
|
||||
## What it proves
|
||||
|
||||
Two things at once — depth in media pipelines (FFmpeg, ImageMagick, PDF and office formats, three.js) and the discipline of shipping a general product that strangers could operate without help.
|
||||
|
||||
## Why these exhibits are good code
|
||||
|
||||
The animation engine evaluates a named easing function to give each letter its delay, so the spacing of an animation follows the same curve as its motion instead of a guessed linear stagger. Both directions build from one transform object, which is what stops the two halves of the same effect drifting apart over years of edits.
|
||||
|
||||
The media pipeline checks the real MIME type before it builds a command, so a wrong input returns a message instead of a half-written file. The GIF encode takes two passes — a palette from the clip, then frames mapped onto it — which is the difference between a GIF that holds its colour and one that does not.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
The PHP is 2014-era procedural code: global state, interpolated shell commands, and long functions. The exhibit is here for the operational judgement, not as a model of structure. The surface was too large for one person to support indefinitely; the plan recorded here is to strip it back to signage plus the content engine and stop selling the rest, which is a decision about focus rather than a claim of failure.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Both code exhibits are published at `content/assets/code/`; a screen recording of the editors is still to come. Source: résumé (2014–2020 freelance) and evidence notes section 3.
|
||||
33
content/products/l3/products-where-woof.md
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
id: products-where-woof
|
||||
area: products
|
||||
level: 3
|
||||
title: "Artefact — Where-Woof · deployed and pre-market"
|
||||
claim: "A QR and RFID tag system for lost property that is deployed and in use — and has not gone to market."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [QR, RFID, PHP, JavaScript]
|
||||
metrics: [{label: "market launch", value: 0}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [products, where-woof, artefact]
|
||||
---
|
||||
|
||||
Deployed is true, commercial is not — that is the exact wording the site uses, deliberately.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
A tag system for lost property. A tag carries a QR code and an RFID identity; whoever finds the item resolves the tag and the owner is contacted. The system is running and being deployed now.
|
||||
|
||||
## What it proves
|
||||
|
||||
That a product can be taken into the field before it is monetised, and that the honest label for that state exists. The distinction matters on a CV because the alternative — calling a pre-launch product commercial — is the kind of claim a technical interviewer unpicks in one question.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
No customers, no revenue and no launch date are claimed. The metric recorded here is a zero on purpose.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
The tag and resolve-flow photographs land in Phase 7 if Sam chooses to publish them. Product name and description are approved; commercial framing is not.
|
||||
58
content/products/products-business.md
Normal file
@@ -0,0 +1,58 @@
|
||||
---
|
||||
id: products-business
|
||||
area: products
|
||||
level: 2
|
||||
title: "Software sold and operated — with the status stated"
|
||||
claim: "Three product lines from one workshop — signage on commodity hardware · the site builder that produced the marketing · and a lost-property tag system — with three businesses still running the signage today."
|
||||
confidence: running
|
||||
period: "2014–"
|
||||
tech: [PHP, JavaScript, three.js, FFmpeg, ImageMagick, HDMI-CEC, Raspberry Pi, ESP32, QR, RFID]
|
||||
metrics: [{label: "businesses currently running the product", value: 3}, {label: "adoptions over time", value: 12}, {label: "product lines", value: 3}]
|
||||
artefacts: [{type: doc, label: "Commercial signage on commodity hardware", url: "content/products/l3/products-signage.md"}, {type: doc, label: "The site builder behind it", url: "content/products/l3/products-websitebuilder.md"}, {type: doc, label: "Where-Woof — deployed and pre-market", url: "content/products/l3/products-where-woof.md"}, {type: doc, label: "Signage infrastructure — screens as a managed fleet", url: "content/platform/l3/platform-signage.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [products, business, signage, where-woof, positioning]
|
||||
---
|
||||
|
||||
**Capability statement.** Building software for a client and building software as a product are different jobs. This page is the second one: a market that was overpriced, a substitute built on hardware that costs a fraction of the incumbent systems, customers who adopted it, and the operational cost of keeping it running.
|
||||
|
||||
## The commercial story
|
||||
|
||||
The digital signage market sells walls: thousands of dollars of hardware with fans, cooling and glass, locked to monopoly software. The product was built as a deliberate inversion of that — a system that runs on a small Raspberry Pi or a microcontroller, drives any HDMI screen including an ordinary monitor, and adapts to the board rather than requiring the board to adapt to it.
|
||||
|
||||
It manages screens as a fleet: power on and off, rotation, mirroring, multi-screen layout, coordinated boards, and synchronised playback across screens in one site or across sites. Users build complex animations or simple image rotations, and the system presents whatever suits the display it finds.
|
||||
|
||||
Demonstrations were rolled out in a hair salon and a financial services business. Around twelve businesses adopted it over time and three still run it today. That number is deliberately specific: twelve is cumulative adoption, three is current, and the difference is the honest measure of a product that was sold but not scaled.
|
||||
|
||||
The same workshop produced the CMS behind the marketing — a templated site builder with editors for drawing, image, video and text animation, video effects and transitions, 3D object import and text extrusion — built on JavaScript and three.js with FFmpeg, ImageMagick and PHP tools for documents, and rendering image animation on Raspberry Pi hardware.
|
||||
|
||||
The third product is different in kind: Where-Woof is a QR and RFID tag system for lost property. It is deployed and being deployed now. It has **not** gone to market, and this page will not call it commercial until it has.
|
||||
|
||||
## Evidence
|
||||
|
||||
| Claim | Evidence | Artefact |
|
||||
|---|---|---|
|
||||
| Signage replaced thousand-dollar wall systems with commodity hardware | Any HDMI screen · Raspberry Pi or ESP32 class player · fleet control | [Signage product](l3/products-signage.md) |
|
||||
| Twelve adoptions over time, three businesses running now | Adoption history and current customers | [Signage product](l3/products-signage.md) |
|
||||
| A templated builder produced the marketing sites | Editors for drawing · video · image and animation; FFmpeg and ImageMagick pipeline | [WebsiteBuilder](l3/products-websitebuilder.md) |
|
||||
| Where-Woof is deployed and pre-market | Device and tag system in use; no market launch | [Where-Woof](l3/products-where-woof.md) |
|
||||
| Screens are managed as a fleet | HDMI-CEC control of power · rotation · layout | [Signage infrastructure](../platform/l3/platform-signage.md) |
|
||||
|
||||
## Product status — stated plainly
|
||||
|
||||
| Product | What it is | Status | Next |
|
||||
|---|---|---|---|
|
||||
| Display screen media (Audas Media) | Digital signage on commodity hardware with multi-screen fleet control | Delivered · three businesses running · being rebuilt with AI | Strip back to signage only and rebuild on the current stack |
|
||||
| WebsiteBuilder | Templated site builder and CMS with media editors and social output | Delivered · retained | Retired as a general product; kept as the content engine |
|
||||
| Where-Woof | QR and RFID tags for lost property | Deployed · pre-market | Take to market — not done yet |
|
||||
|
||||
## Trade-offs and limits
|
||||
|
||||
- **Scope discipline came late.** The builder carried a shop, invoicing, room management, calendars, presentations, documents, bulk messaging and client management. Product surface grew faster than a one-person support model could carry, which is why the plan now is to keep the signage and the content engine and retire the rest.
|
||||
- **The rebuild is unfinished.** "Rebuilding with AI" is a statement of work in progress, not a launch. It is labelled as such on every page that mentions it.
|
||||
- **One product is pre-revenue by design.** Where-Woof is deployed before it is sold. Describing it as commercial would be a lie that a careful reader would find in a minute.
|
||||
- **Support is the hidden cost.** Customers running the product for years is the strongest evidence of value and the strongest argument for keeping the surface small.
|
||||
|
||||
Signage infrastructure is also platform work: [screens as a managed fleet](../platform/l3/platform-signage.md).
|
||||
|
||||
Each product carries its own status, and the two caveats — a rebuild in progress and a launch not yet made — are in the table above rather than in a footnote.
|
||||
16
content/products/products-overview.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
id: products-overview
|
||||
area: products
|
||||
level: 1
|
||||
title: "Products"
|
||||
claim: "Sam identified an overpriced market and shipped a cheaper substitute on commodity hardware: twelve adoptions over time and three businesses still running it."
|
||||
confidence: running
|
||||
period: "2014–"
|
||||
tech: [PHP, JavaScript, three.js, FFmpeg, ImageMagick, HDMI-CEC, Raspberry Pi, ESP32]
|
||||
metrics: [{label: "businesses currently running the product", value: 3}, {label: "adoptions over time", value: 12}, {label: "product lines", value: 3}]
|
||||
artefacts: [{type: doc, label: "Commercial signage on commodity hardware", url: "content/products/l3/products-signage.md"}, {type: doc, label: "The site builder behind it", url: "content/products/l3/products-websitebuilder.md"}, {type: doc, label: "Where-Woof — deployed and pre-market", url: "content/products/l3/products-where-woof.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [products, signage, business, where-woof]
|
||||
---
|
||||
Building, selling, deploying and still supporting software for other businesses is a different discipline from writing it, and the caveats on these three products are stated rather than hidden. The commercial story is on the [products page](products-business.md).
|
||||
22
corpus/__init__.py
Normal file
@@ -0,0 +1,22 @@
|
||||
"""Phase 4 — the retrievable corpus.
|
||||
|
||||
`content/` is the source of truth. This package turns it into the two indexes
|
||||
the AI console reads:
|
||||
|
||||
* **vectors** in Qdrant, one point per chunk, with the payload that lets the
|
||||
query path filter *before* searching (`area`, `confidence`);
|
||||
* **keywords** in SQLite FTS5, one row per chunk, so exact tokens (repo names,
|
||||
`LoRaWAN`, `MQTT`, dates) are not lost the way they are with vectors alone
|
||||
(decision D10, hybrid retrieval).
|
||||
|
||||
Everything here is driven by `python3 -m corpus.cli`; `docs/phase-4-notes.md`
|
||||
is the operator manual and records what was measured. The pipeline is
|
||||
Provider-agnostic by design: `corpus/embeddings.py` is the only place that knows
|
||||
how a vector is made, and the provider + model + dimension are recorded per
|
||||
collection so a provider change is a governed migration, not silent corruption.
|
||||
|
||||
Standard library only, plus the repository's own validator (`tools/`). No new
|
||||
dependency, no Node, no framework (AGENTS.md section 4.1).
|
||||
"""
|
||||
|
||||
__all__ = ["config", "loader", "chunker", "payloads", "embeddings", "qdrant", "store", "pipeline", "freshness"]
|
||||
8
corpus/__main__.py
Normal file
@@ -0,0 +1,8 @@
|
||||
"""`python3 -m corpus` is the same as `python3 -m corpus.cli`."""
|
||||
|
||||
import sys
|
||||
|
||||
from .cli import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
287
corpus/chunker.py
Normal file
@@ -0,0 +1,287 @@
|
||||
"""Code-aware chunking.
|
||||
|
||||
The rules come from `docs/phase-4-corpus.md` §4.1:
|
||||
|
||||
* 400–800 tokens per chunk, about 12% overlap;
|
||||
* **never split inside a fenced block, a table or a front-matter block.**
|
||||
|
||||
The last rule is structural here, not a post-hoc check: the body is parsed into
|
||||
*blocks* first and chunks are packed out of whole blocks, so a boundary can only
|
||||
ever fall between blocks. Fenced code, tables, headings, list items and
|
||||
paragraphs are all blocks; the large ones cannot be subdivided.
|
||||
|
||||
Front-matter is removed by `corpus.loader` before this module sees the body, and
|
||||
parsing a body that still has front-matter is an error, so it cannot be split
|
||||
either.
|
||||
|
||||
`tokens` are estimated with a word-and-punctuation regex. A real BPE tokenizer
|
||||
counts 15–30% more for prose and more again for code, so a chunk that measures
|
||||
800 here is a conservative upper bound.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
# Packing targets (docs/phase-4-corpus.md §4.1).
|
||||
TOKEN_TARGET = 600
|
||||
TOKEN_MAX = 800
|
||||
TOKEN_MIN = 400
|
||||
OVERLAP_RATIO = 0.12
|
||||
|
||||
FENCE_RE = re.compile(r"^ {0,3}(`{3,}|~{3,})")
|
||||
HEADING_RE = re.compile(r"^ {0,3}#{1,6}\s")
|
||||
LIST_RE = re.compile(r"^(\s*)([-*+]|\d{1,3}[.)])\s+")
|
||||
RULE_RE = re.compile(r"^ {0,3}([-*_])(\s*\1){2,}\s*$")
|
||||
TOKEN_RE = re.compile(r"\w+|[^\w\s]", re.UNICODE)
|
||||
|
||||
PROTECTED_KINDS = ("fence", "table")
|
||||
|
||||
|
||||
class ChunkError(RuntimeError):
|
||||
"""The body could not be chunked without breaking the rules."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Block:
|
||||
"""An indivisible unit of Markdown. Never split, only packed."""
|
||||
|
||||
kind: str # fence | table | heading | list | quote | para | rule
|
||||
text: str
|
||||
tokens: int
|
||||
line_start: int # 1-based, relative to the body this block came from
|
||||
line_end: int
|
||||
|
||||
@property
|
||||
def protected(self) -> bool:
|
||||
return self.kind in PROTECTED_KINDS
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Chunk:
|
||||
index: int
|
||||
text: str
|
||||
tokens: int
|
||||
line_start: int
|
||||
line_end: int
|
||||
blocks: tuple
|
||||
oversize: bool = False
|
||||
|
||||
@property
|
||||
def kinds(self) -> tuple:
|
||||
return tuple(block.kind for block in self.blocks)
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""Word-plus-punctuation token estimate. Deterministic and cheap."""
|
||||
return len(TOKEN_RE.findall(text))
|
||||
|
||||
|
||||
def parse_blocks(body: str) -> list:
|
||||
"""Split a Markdown body into atomic blocks.
|
||||
|
||||
Raises ChunkError if the body still carries a front-matter block: that is a
|
||||
loader bug, and silently chunking it would split something that must not be
|
||||
split.
|
||||
"""
|
||||
if body.lstrip().startswith("---"):
|
||||
raise ChunkError(
|
||||
"body still starts with a front-matter block; strip front-matter "
|
||||
"before chunking (corpus.loader does this)"
|
||||
)
|
||||
|
||||
lines = body.split("\n")
|
||||
blocks = []
|
||||
i = 0
|
||||
total = len(lines)
|
||||
while i < total:
|
||||
raw = lines[i]
|
||||
stripped = raw.strip()
|
||||
if not stripped:
|
||||
i += 1
|
||||
continue
|
||||
start = i
|
||||
fence = FENCE_RE.match(raw)
|
||||
if fence:
|
||||
i = _read_fence(lines, i, fence.group(1))
|
||||
kind = "fence"
|
||||
elif stripped.startswith("|"):
|
||||
while i < total and lines[i].strip().startswith("|"):
|
||||
i += 1
|
||||
kind = "table"
|
||||
elif HEADING_RE.match(raw):
|
||||
i += 1
|
||||
kind = "heading"
|
||||
elif RULE_RE.match(raw):
|
||||
i += 1
|
||||
kind = "rule"
|
||||
elif LIST_RE.match(raw):
|
||||
i = _read_list_item(lines, i)
|
||||
kind = "list"
|
||||
elif stripped.startswith(">"):
|
||||
while i < total and lines[i].strip().startswith(">"):
|
||||
i += 1
|
||||
kind = "quote"
|
||||
else:
|
||||
i = _read_paragraph(lines, i)
|
||||
kind = "para"
|
||||
text = "\n".join(lines[start:i])
|
||||
blocks.append(
|
||||
Block(kind=kind, text=text, tokens=estimate_tokens(text), line_start=start + 1, line_end=i)
|
||||
)
|
||||
return blocks
|
||||
|
||||
|
||||
def _read_fence(lines: list, i: int, marker: str) -> int:
|
||||
"""Return the index just past a fenced block. Unterminated runs to EOF."""
|
||||
char = marker[0]
|
||||
width = len(marker)
|
||||
i += 1
|
||||
while i < len(lines):
|
||||
candidate = FENCE_RE.match(lines[i])
|
||||
if candidate and candidate.group(1)[0] == char and len(candidate.group(1)) >= width:
|
||||
return i + 1
|
||||
i += 1
|
||||
return len(lines)
|
||||
|
||||
|
||||
def _read_list_item(lines: list, i: int) -> int:
|
||||
"""Return the index just past one top-level list item, continuations included."""
|
||||
base_indent = len(LIST_RE.match(lines[i]).group(1))
|
||||
i += 1
|
||||
while i < len(lines):
|
||||
raw = lines[i]
|
||||
if not raw.strip():
|
||||
# A blank line continues the item only if an indented line follows.
|
||||
if i + 1 < len(lines) and lines[i + 1].startswith(" " * (base_indent + 1)):
|
||||
i += 1
|
||||
continue
|
||||
return i
|
||||
indent = len(raw) - len(raw.lstrip())
|
||||
if indent > base_indent or LIST_RE.match(raw) and len(LIST_RE.match(raw).group(1)) > base_indent:
|
||||
i += 1
|
||||
continue
|
||||
if LIST_RE.match(raw):
|
||||
return i
|
||||
if indent <= base_indent:
|
||||
return i
|
||||
i += 1
|
||||
return i
|
||||
|
||||
|
||||
def _read_paragraph(lines: list, i: int) -> int:
|
||||
i += 1
|
||||
while i < len(lines):
|
||||
raw = lines[i]
|
||||
stripped = raw.strip()
|
||||
if not stripped:
|
||||
return i
|
||||
if FENCE_RE.match(raw) or HEADING_RE.match(raw) or RULE_RE.match(raw):
|
||||
return i
|
||||
if stripped.startswith("|") or stripped.startswith(">") or LIST_RE.match(raw):
|
||||
return i
|
||||
i += 1
|
||||
return i
|
||||
|
||||
|
||||
def chunk_blocks(blocks: list) -> list:
|
||||
"""Pack blocks into windows of at most TOKEN_MAX, with ~12% overlap."""
|
||||
chunks = []
|
||||
total = len(blocks)
|
||||
if total == 0:
|
||||
return chunks
|
||||
i = 0
|
||||
while i < total:
|
||||
start = i
|
||||
tokens = 0
|
||||
end = i
|
||||
while end < total:
|
||||
block = blocks[end]
|
||||
if end > start:
|
||||
if tokens + block.tokens > TOKEN_MAX:
|
||||
break
|
||||
# Prefer breaking in front of a heading.
|
||||
if block.kind == "heading" and tokens >= int(TOKEN_TARGET * 0.75):
|
||||
break
|
||||
tokens += block.tokens
|
||||
end += 1
|
||||
if tokens >= TOKEN_TARGET and end < total and blocks[end].kind == "heading":
|
||||
break
|
||||
window = blocks[start:end]
|
||||
chunks.append(_make_chunk(len(chunks), window))
|
||||
if end >= total:
|
||||
break
|
||||
# Back up over trailing blocks until roughly OVERLAP_RATIO of the window
|
||||
# repeats, leaving at least one block behind so the walk always advances.
|
||||
budget = max(1, int(round(TOKEN_TARGET * OVERLAP_RATIO)))
|
||||
carried = 0
|
||||
back = end
|
||||
while back - 1 > start and carried + blocks[back - 1].tokens <= budget * 2:
|
||||
carried += blocks[back - 1].tokens
|
||||
back -= 1
|
||||
if carried >= budget:
|
||||
break
|
||||
i = max(back, start + 1)
|
||||
return chunks
|
||||
|
||||
|
||||
def _make_chunk(index: int, window: list) -> Chunk:
|
||||
text = "\n\n".join(block.text for block in window)
|
||||
return Chunk(
|
||||
index=index,
|
||||
text=text,
|
||||
tokens=sum(block.tokens for block in window),
|
||||
line_start=window[0].line_start,
|
||||
line_end=window[-1].line_end,
|
||||
blocks=tuple(window),
|
||||
oversize=len(window) == 1 and window[0].tokens > TOKEN_MAX,
|
||||
)
|
||||
|
||||
|
||||
def chunk_document(body: str) -> list:
|
||||
"""Chunk one document body. The pipeline calls this and nothing else."""
|
||||
chunks = chunk_blocks(parse_blocks(body))
|
||||
verify_no_split(chunks, protected_regions(body))
|
||||
return chunks
|
||||
|
||||
|
||||
def protected_regions(body: str) -> list:
|
||||
"""Line ranges (1-based, inclusive) that a chunk boundary must not cross.
|
||||
|
||||
Recomputed from the body text alone, independently of the packing code, so
|
||||
the check is not the implementation grading its own homework.
|
||||
"""
|
||||
regions = []
|
||||
lines = body.split("\n")
|
||||
i = 0
|
||||
while i < len(lines):
|
||||
raw = lines[i]
|
||||
fence = FENCE_RE.match(raw)
|
||||
if fence:
|
||||
end = _read_fence(lines, i, fence.group(1))
|
||||
regions.append((i + 1, end))
|
||||
i = end
|
||||
continue
|
||||
if raw.strip().startswith("|"):
|
||||
start = i
|
||||
while i < len(lines) and lines[i].strip().startswith("|"):
|
||||
i += 1
|
||||
regions.append((start + 1, i))
|
||||
continue
|
||||
i += 1
|
||||
return regions
|
||||
|
||||
|
||||
def verify_no_split(chunks: list, regions: list) -> None:
|
||||
"""Raise ChunkError if any chunk boundary cuts a protected region."""
|
||||
for chunk in chunks:
|
||||
for first, last in regions:
|
||||
spans_start = chunk.line_start <= first <= chunk.line_end
|
||||
spans_end = chunk.line_start <= last <= chunk.line_end
|
||||
covers = chunk.line_start <= first and chunk.line_end >= last
|
||||
if (spans_start or spans_end) and not covers:
|
||||
raise ChunkError(
|
||||
f"chunk {chunk.index} splits protected lines {first}-{last} "
|
||||
f"(chunk covers {chunk.line_start}-{chunk.line_end})"
|
||||
)
|
||||
257
corpus/cli.py
Normal file
@@ -0,0 +1,257 @@
|
||||
"""Command line entry point: `python3 -m corpus.cli <command>`.
|
||||
|
||||
Commands
|
||||
|
||||
sync incremental update of the corpus (the normal command)
|
||||
rebuild drop the collection and rebuild it from content/ (recovery)
|
||||
audit freshness audit; reports stale documents, index drift
|
||||
verify does the index agree with content/ and with Qdrant?
|
||||
demo-filter the same query with and without an `area` filter
|
||||
hybrid vectors + FTS5 fused with reciprocal rank fusion
|
||||
measure Qdrant memory at 500 / 5,000 / 20,000 chunks
|
||||
info what this build thinks it is configured to do
|
||||
|
||||
Exit codes: 0 success, 1 findings or verification problems, 2 configuration or
|
||||
usage error. Every command accepts `--json` where a machine-readable result is
|
||||
useful (Prefect reads JSON, a human reads the table).
|
||||
|
||||
Configuration is environment-first (`corpus/config.py`); the flags exist so a
|
||||
one-off run can point at a scratch Qdrant without exporting variables.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
from . import config as config_module
|
||||
from . import freshness, pipeline, store as store_module
|
||||
from .config import Config, ConfigError
|
||||
from .qdrant import Qdrant, QdrantError
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
parser = _build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
if args.command is None:
|
||||
parser.print_help()
|
||||
return 2
|
||||
|
||||
overrides = {
|
||||
"root": args.root,
|
||||
"qdrant_url": args.qdrant_url,
|
||||
"collection": args.collection,
|
||||
"provider": args.provider,
|
||||
"model": args.model,
|
||||
"dim": args.dim,
|
||||
"db_path": args.db,
|
||||
"data_dir": args.data,
|
||||
}
|
||||
try:
|
||||
cfg = Config.from_env(**overrides)
|
||||
except ConfigError as exc:
|
||||
print(f"configuration error: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
log = (lambda _msg: None) if args.quiet else (lambda msg: print(msg, file=sys.stderr))
|
||||
|
||||
try:
|
||||
handler = {
|
||||
"sync": _cmd_sync,
|
||||
"rebuild": _cmd_rebuild,
|
||||
"audit": _cmd_audit,
|
||||
"verify": _cmd_verify,
|
||||
"demo-filter": _cmd_demo_filter,
|
||||
"hybrid": _cmd_hybrid,
|
||||
"measure": _cmd_measure,
|
||||
"info": _cmd_info,
|
||||
}[args.command]
|
||||
return handler(cfg, args, log)
|
||||
except (pipeline.PipelineError, ConfigError) as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
except QdrantError as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="python3 -m corpus.cli",
|
||||
description="Build and audit the Ai Resume retrieval corpus (docs/phase-4-corpus.md).",
|
||||
)
|
||||
parser.add_argument("--root", help="repository root (default: parent of corpus/)")
|
||||
parser.add_argument("--qdrant-url", help="Qdrant REST URL (default http://127.0.0.1:6333)")
|
||||
parser.add_argument("--collection", help="Qdrant collection name (default resume_corpus)")
|
||||
parser.add_argument("--provider", choices=sorted(config_module.PROVIDERS), help="embedding provider")
|
||||
parser.add_argument("--model", help="embedding model name")
|
||||
parser.add_argument("--dim", type=int, help="embedding dimension")
|
||||
parser.add_argument("--db", help="SQLite corpus database path")
|
||||
parser.add_argument("--data", help="corpus data directory")
|
||||
parser.add_argument("--quiet", action="store_true", help="progress on stderr only becomes silent")
|
||||
|
||||
sub = parser.add_subparsers(dest="command")
|
||||
|
||||
sync = sub.add_parser("sync", help="incremental update")
|
||||
sync.add_argument("--json", action="store_true")
|
||||
|
||||
rebuild = sub.add_parser("rebuild", help="drop the collection and rebuild from content/")
|
||||
rebuild.add_argument("--json", action="store_true")
|
||||
|
||||
audit = sub.add_parser("audit", help="freshness and drift audit")
|
||||
audit.add_argument("--json", action="store_true")
|
||||
audit.add_argument("--today", help="audit as of this date (YYYY-MM-DD), for tests")
|
||||
audit.add_argument("--fail-on-findings", action="store_true", help="exit 1 when anything is found")
|
||||
audit.add_argument("--alert", action="store_true", help="POST the report to RESUME_APPRISE_URL")
|
||||
|
||||
verify = sub.add_parser("verify", help="check the index against content/ and Qdrant")
|
||||
verify.add_argument("--json", action="store_true")
|
||||
|
||||
demo = sub.add_parser("demo-filter", help="same query with and without an area filter")
|
||||
demo.add_argument("--query", required=True)
|
||||
demo.add_argument("--area", help="area to filter on (ai, platform, edge, products, history, about)")
|
||||
demo.add_argument("--limit", type=int, default=5)
|
||||
demo.add_argument("--json", action="store_true")
|
||||
|
||||
hybrid = sub.add_parser("hybrid", help="vectors + FTS5 fused with RRF")
|
||||
hybrid.add_argument("--query", required=True)
|
||||
hybrid.add_argument("--area")
|
||||
hybrid.add_argument("--limit", type=int, default=5)
|
||||
hybrid.add_argument("--json", action="store_true")
|
||||
|
||||
measure = sub.add_parser("measure", help="Qdrant memory at three corpus sizes")
|
||||
measure.add_argument("--sizes", default="500,5000,20000")
|
||||
measure.add_argument("--container", default="", help="Qdrant container name for docker stats")
|
||||
measure.add_argument("--keep", action="store_true", help="keep the measurement collection")
|
||||
measure.add_argument("--json", action="store_true")
|
||||
|
||||
sub.add_parser("info", help="print the resolved configuration")
|
||||
return parser
|
||||
|
||||
|
||||
# -- commands --------------------------------------------------------------
|
||||
|
||||
|
||||
def _cmd_sync(cfg, args, log) -> int:
|
||||
stats = pipeline.run(cfg, mode="sync", log=log)
|
||||
if args.json:
|
||||
print(json.dumps(stats.to_dict(), indent=2, sort_keys=True))
|
||||
else:
|
||||
print(stats.summary())
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_rebuild(cfg, args, log) -> int:
|
||||
stats = pipeline.run(cfg, mode="rebuild", log=log)
|
||||
if args.json:
|
||||
print(json.dumps(stats.to_dict(), indent=2, sort_keys=True))
|
||||
else:
|
||||
print(stats.summary())
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_audit(cfg, args, log) -> int:
|
||||
today = datetime.date.fromisoformat(args.today) if args.today else None
|
||||
report = freshness.audit(cfg, today=today, log=log)
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, sort_keys=True))
|
||||
else:
|
||||
print(freshness.render_text(report))
|
||||
if args.alert:
|
||||
try:
|
||||
sent = freshness.alert(report, cfg.apprise_url)
|
||||
if not sent:
|
||||
log("alert: RESUME_APPRISE_URL is not set; nothing sent")
|
||||
except Exception as exc: # noqa: BLE001 - an alert failure must not hide the audit
|
||||
print(f"warning: could not send the Apprise alert: {exc}", file=sys.stderr)
|
||||
if args.fail_on_findings and not report["ok"]:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_verify(cfg, args, log) -> int:
|
||||
problems = pipeline.verify(cfg, log=log)
|
||||
if args.json:
|
||||
print(json.dumps({"ok": not problems, "problems": problems}, indent=2))
|
||||
else:
|
||||
if problems:
|
||||
print(f"FAIL: {len(problems)} problem(s)")
|
||||
for problem in problems:
|
||||
print(f" - {problem}")
|
||||
else:
|
||||
print("OK: the index agrees with content/ and Qdrant")
|
||||
return 1 if problems else 0
|
||||
|
||||
|
||||
def _cmd_demo_filter(cfg, args, log) -> int:
|
||||
result = pipeline.demo_area_filter(cfg, args.query, area=args.area, limit=args.limit)
|
||||
if args.json:
|
||||
print(json.dumps(result, indent=2, sort_keys=True))
|
||||
return 0
|
||||
print(f"query: {result['query']!r}")
|
||||
print("")
|
||||
print(f"without a filter ({len(result['unfiltered'])} result(s)):")
|
||||
for row in result["unfiltered"]:
|
||||
print(f" {row['score']:.3f} [{row['area']}/{row['confidence']}] {row['title']} {row['url']}")
|
||||
print("")
|
||||
if not args.area:
|
||||
print("no --area given, so the filtered set is empty; re-run with e.g. --area edge")
|
||||
return 0
|
||||
print(f"with area={args.area!r} ({len(result['filtered'])} result(s)):")
|
||||
for row in result["filtered"]:
|
||||
print(f" {row['score']:.3f} [{row['area']}/{row['confidence']}] {row['title']} {row['url']}")
|
||||
wrong = [row for row in result["filtered"] if row["area"] != args.area]
|
||||
if wrong:
|
||||
print(f"FAIL: {len(wrong)} filtered result(s) are not in area {args.area!r}")
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_hybrid(cfg, args, log) -> int:
|
||||
rows = pipeline.hybrid_search(cfg, args.query, area=args.area, limit=args.limit)
|
||||
if args.json:
|
||||
print(json.dumps(rows, indent=2, sort_keys=True, default=str))
|
||||
return 0
|
||||
print(f"hybrid (RRF) query={args.query!r} area={args.area!r}: {len(rows)} result(s)")
|
||||
for row in rows:
|
||||
print(f" {row['score']:.4f} {row.get('title')} {row.get('url')} [{row['id'][:8]}]")
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_measure(cfg, args, log) -> int:
|
||||
from . import measure as measure_module
|
||||
|
||||
sizes = [int(part) for part in args.sizes.split(",") if part.strip()]
|
||||
report = measure_module.measure(
|
||||
cfg, sizes=sizes, container=args.container, keep=args.keep, log=log
|
||||
)
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, sort_keys=True))
|
||||
return 0
|
||||
print(measure_module.render_text(report))
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_info(cfg, args, log) -> int:
|
||||
print(cfg.describe())
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
print("")
|
||||
print(f"qdrant: {'answering, version ' + qdrant.version() if qdrant.health() else 'NOT answering'}")
|
||||
if qdrant.collection_exists(cfg.collection):
|
||||
print(f"collection {cfg.collection}: {qdrant.point_count(cfg.collection)} point(s)")
|
||||
else:
|
||||
print(f"collection {cfg.collection}: does not exist")
|
||||
if os.path.exists(cfg.db_path):
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
print(f"store: {store.chunk_count()} chunk(s) across {store.source_count()} source(s)")
|
||||
print(f"contract: {json.dumps(store.get_meta('collection'), sort_keys=True)}")
|
||||
else:
|
||||
print(f"store: {cfg.db_path} does not exist")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
160
corpus/config.py
Normal file
@@ -0,0 +1,160 @@
|
||||
"""Configuration for the corpus pipeline. Environment only.
|
||||
|
||||
Nothing in this file holds a credential. The OmniRoute key is read from the
|
||||
process environment, which systemd populates from
|
||||
`~/.config/environment.d/10-secrets.conf` (AGENTS.md section 4.4). `describe()`
|
||||
is the only way the pipeline prints its configuration, and it redacts the key.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
|
||||
# Bumped whenever chunk boundaries or payload shape change. A stored corpus
|
||||
# carrying a different version is stale by definition and gets rebuilt.
|
||||
CHUNKER_VERSION = "blocks-v1"
|
||||
|
||||
# Provider defaults. `local-hashing` is a deterministic, offline provider used
|
||||
# by the tests, by `make corpus-test`, and by any rebuild on a host with no
|
||||
# model gateway. It is NOT a semantic embedding model: it exists so the whole
|
||||
# pipeline can be exercised without a provider decision, and so a rebuild never
|
||||
# depends on the network. The production provider is `omniroute`.
|
||||
PROVIDERS = {
|
||||
"local-hashing": ("local-hashing-v1", 256),
|
||||
"omniroute": ("text-embedding-3-small", 1536),
|
||||
}
|
||||
|
||||
DEFAULT_QDRANT_URL = "http://127.0.0.1:6333"
|
||||
DEFAULT_COLLECTION = "resume_corpus"
|
||||
DEFAULT_PROVIDER = "local-hashing"
|
||||
DEFAULT_OMNIROUTE_URL = "https://omniroute.lab.audasmedia.com.au"
|
||||
|
||||
# Freshness horizon: a document whose `last_verified` is older than this is
|
||||
# reported stale by the nightly audit. docs/phase-4-notes.md section 5.
|
||||
DEFAULT_MAX_AGE_DAYS = 180
|
||||
DEFAULT_WARN_DAYS = 30
|
||||
|
||||
|
||||
class ConfigError(RuntimeError):
|
||||
"""A configuration value is missing or unusable."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Config:
|
||||
"""Everything the pipeline needs, resolved once."""
|
||||
|
||||
root: str
|
||||
content_dir: str
|
||||
data_dir: str
|
||||
db_path: str
|
||||
qdrant_url: str
|
||||
collection: str
|
||||
provider: str
|
||||
model: str
|
||||
dim: int
|
||||
api_url: str
|
||||
api_key: str
|
||||
batch_size: int
|
||||
timeout: float
|
||||
max_age_days: int
|
||||
warn_days: int
|
||||
apprise_url: str
|
||||
vectors_on_disk: bool = False
|
||||
|
||||
# -- construction -----------------------------------------------------
|
||||
|
||||
@classmethod
|
||||
def from_env(cls, root: str | None = None, **overrides) -> "Config":
|
||||
"""Build a Config from the environment, then apply explicit overrides.
|
||||
|
||||
Overrides exist so the CLI can accept flags without a second code path;
|
||||
they are applied last so a flag always wins over the environment.
|
||||
"""
|
||||
root = os.path.abspath(root or os.environ.get("RESUME_ROOT") or _default_root())
|
||||
content_dir = os.environ.get("RESUME_CONTENT") or os.path.join(root, "content")
|
||||
data_dir = os.environ.get("RESUME_CORPUS_DATA") or os.path.join(root, "data", "corpus")
|
||||
provider = os.environ.get("RESUME_EMBED_PROVIDER") or DEFAULT_PROVIDER
|
||||
if provider not in PROVIDERS:
|
||||
raise ConfigError(
|
||||
f"RESUME_EMBED_PROVIDER={provider!r} is not one of {sorted(PROVIDERS)}"
|
||||
)
|
||||
default_model, default_dim = PROVIDERS[provider]
|
||||
|
||||
values = dict(
|
||||
root=root,
|
||||
content_dir=os.path.abspath(content_dir),
|
||||
data_dir=os.path.abspath(data_dir),
|
||||
db_path=os.path.abspath(
|
||||
os.environ.get("RESUME_CORPUS_DB") or os.path.join(data_dir, "corpus.db")
|
||||
),
|
||||
qdrant_url=(os.environ.get("RESUME_QDRANT_URL") or DEFAULT_QDRANT_URL).rstrip("/"),
|
||||
collection=os.environ.get("RESUME_COLLECTION") or DEFAULT_COLLECTION,
|
||||
provider=provider,
|
||||
model=os.environ.get("RESUME_EMBED_MODEL") or default_model,
|
||||
dim=int(os.environ.get("RESUME_EMBED_DIM") or default_dim),
|
||||
api_url=(os.environ.get("OMNIROUTE_URL") or DEFAULT_OMNIROUTE_URL).rstrip("/"),
|
||||
api_key=os.environ.get("OMNIROUTE_API_KEY") or "",
|
||||
batch_size=int(os.environ.get("RESUME_EMBED_BATCH") or 32),
|
||||
timeout=float(os.environ.get("RESUME_QDRANT_TIMEOUT") or 30.0),
|
||||
max_age_days=int(
|
||||
os.environ.get("RESUME_MAX_AGE_DAYS") or DEFAULT_MAX_AGE_DAYS
|
||||
),
|
||||
warn_days=int(os.environ.get("RESUME_WARN_DAYS") or DEFAULT_WARN_DAYS),
|
||||
apprise_url=os.environ.get("RESUME_APPRISE_URL") or "",
|
||||
vectors_on_disk=(os.environ.get("RESUME_QDRANT_VECTORS_ON_DISK") or "").lower()
|
||||
in ("1", "true", "yes"),
|
||||
)
|
||||
values.update({k: v for k, v in overrides.items() if v not in (None, "")})
|
||||
cfg = cls(**values)
|
||||
cfg.check()
|
||||
return cfg
|
||||
|
||||
# -- validation -------------------------------------------------------
|
||||
|
||||
def check(self) -> None:
|
||||
if self.dim <= 0:
|
||||
raise ConfigError(f"embedding dimension must be positive, got {self.dim}")
|
||||
if self.batch_size <= 0:
|
||||
raise ConfigError(f"batch size must be positive, got {self.batch_size}")
|
||||
if self.provider == "omniroute" and not self.api_key:
|
||||
raise ConfigError(
|
||||
"provider 'omniroute' needs OMNIROUTE_API_KEY in the environment "
|
||||
"(~/.config/environment.d/10-secrets.conf). It is never read from this repo."
|
||||
)
|
||||
if not os.path.isdir(self.content_dir):
|
||||
raise ConfigError(f"content directory not found: {self.content_dir}")
|
||||
|
||||
# -- presentation -----------------------------------------------------
|
||||
|
||||
def describe(self) -> str:
|
||||
"""Human-readable configuration. Never includes the API key value."""
|
||||
key = "set" if self.api_key else "unset"
|
||||
return (
|
||||
f"root={self.root}\n"
|
||||
f"content={self.content_dir}\n"
|
||||
f"data={self.data_dir}\n"
|
||||
f"db={self.db_path}\n"
|
||||
f"qdrant={self.qdrant_url} collection={self.collection}\n"
|
||||
f"provider={self.provider} model={self.model} dim={self.dim} "
|
||||
f"batch={self.batch_size}\n"
|
||||
f"api_url={self.api_url} api_key={key}\n"
|
||||
f"freshness=max_age_days:{self.max_age_days} warn_days:{self.warn_days}\n"
|
||||
f"qdrant vectors_on_disk={self.vectors_on_disk}\n"
|
||||
f"chunker={CHUNKER_VERSION}"
|
||||
)
|
||||
|
||||
# -- derived paths ----------------------------------------------------
|
||||
|
||||
@property
|
||||
def chunker_version(self) -> str:
|
||||
return CHUNKER_VERSION
|
||||
|
||||
@property
|
||||
def meta_path(self) -> str:
|
||||
return os.path.join(self.data_dir, "collection.json")
|
||||
|
||||
|
||||
def _default_root() -> str:
|
||||
"""The repository root: the parent of this package."""
|
||||
return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
241
corpus/embeddings.py
Normal file
@@ -0,0 +1,241 @@
|
||||
"""The one place a vector is made.
|
||||
|
||||
`docs/phase-4-corpus.md` §4.3: *"Record the embedding model and dimension per
|
||||
collection. A provider change is a governed migration, not a silent corruption.
|
||||
Provider decision pending Sam — build provider-agnostic behind one interface."*
|
||||
|
||||
So: one interface (`Provider`), two implementations, and the provider name,
|
||||
model and dimension are written to the collection metadata by the pipeline. A
|
||||
later run with a different model refuses to mix vectors into the same
|
||||
collection; it reports a governed migration instead.
|
||||
|
||||
**local-hashing** is deterministic, offline, and *not semantic*. It hashes word
|
||||
tokens into a fixed-width vector. It exists so that the pipeline, its tests and
|
||||
a from-scratch rebuild never depend on a network or on an unmade decision. It is
|
||||
the default for `make corpus-test` and for the Qdrant memory measurement.
|
||||
|
||||
**omniroute** is the production path: an OpenAI-compatible `/v1/embeddings`
|
||||
call through OmniRoute, which is the only permitted model gateway
|
||||
(AGENTS.md §4.5). No provider SDK, no direct vendor call, key from the
|
||||
environment only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
|
||||
class EmbeddingError(RuntimeError):
|
||||
"""Embedding failed. The pipeline aborts rather than writing partial vectors."""
|
||||
|
||||
|
||||
class Provider:
|
||||
"""Interface every embedding provider implements."""
|
||||
|
||||
name = "provider"
|
||||
model = "model"
|
||||
dim = 0
|
||||
|
||||
def embed(self, texts: list) -> list:
|
||||
raise NotImplementedError
|
||||
|
||||
def stats(self) -> dict:
|
||||
"""Call counters. The DoD proves incremental mode with these numbers."""
|
||||
return {"calls": 0, "texts": 0}
|
||||
|
||||
def reset_stats(self) -> None:
|
||||
pass
|
||||
|
||||
def describe(self) -> dict:
|
||||
return {"provider": self.name, "model": self.model, "dim": self.dim}
|
||||
|
||||
|
||||
class LocalHashingProvider(Provider):
|
||||
"""Deterministic feature hashing. Offline, fast, not semantic.
|
||||
|
||||
Every token is hashed with sha256 (never Python's salted `hash()`), so the
|
||||
same text produces the same vector on every host and every run. Vectors are
|
||||
L2-normalised, which suits cosine distance.
|
||||
"""
|
||||
|
||||
name = "local-hashing"
|
||||
|
||||
def __init__(self, model: str = "local-hashing-v1", dim: int = 256) -> None:
|
||||
if dim <= 0:
|
||||
raise EmbeddingError(f"local-hashing needs a positive dimension, got {dim}")
|
||||
self.model = model
|
||||
self.dim = dim
|
||||
self._calls = 0
|
||||
self._texts = 0
|
||||
|
||||
def embed(self, texts: list) -> list:
|
||||
self._calls += 1
|
||||
self._texts += len(texts)
|
||||
return [self._vector(text) for text in texts]
|
||||
|
||||
def _vector(self, text: str) -> list:
|
||||
vector = [0.0] * self.dim
|
||||
for token in _tokens(text):
|
||||
digest = hashlib.sha256(token.encode("utf-8")).digest()
|
||||
index = int.from_bytes(digest[:8], "big") % self.dim
|
||||
sign = 1.0 if digest[8] & 1 else -1.0
|
||||
vector[index] += sign
|
||||
norm = math.sqrt(sum(value * value for value in vector))
|
||||
if norm == 0.0:
|
||||
# An empty text still needs a unit vector of the right width.
|
||||
vector[0] = 1.0
|
||||
return vector
|
||||
return [value / norm for value in vector]
|
||||
|
||||
def stats(self) -> dict:
|
||||
return {"calls": self._calls, "texts": self._texts}
|
||||
|
||||
def reset_stats(self) -> None:
|
||||
self._calls = 0
|
||||
self._texts = 0
|
||||
|
||||
|
||||
class OmniRouteProvider(Provider):
|
||||
"""OpenAI-compatible `/v1/embeddings` through OmniRoute.
|
||||
|
||||
Retries on 429 and 5xx with exponential backoff (the risk table in
|
||||
`docs/phase-4-corpus.md` calls this out for bulk ingest). The key is never
|
||||
logged, never put in an error message, and never read from this repository.
|
||||
"""
|
||||
|
||||
name = "omniroute"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
api_url: str,
|
||||
api_key: str,
|
||||
model: str,
|
||||
dim: int,
|
||||
batch_size: int = 32,
|
||||
timeout: float = 30.0,
|
||||
attempts: int = 4,
|
||||
backoff: float = 1.5,
|
||||
) -> None:
|
||||
if not api_key:
|
||||
raise EmbeddingError(
|
||||
"OmniRoute needs OMNIROUTE_API_KEY in the environment "
|
||||
"(~/.config/environment.d/10-secrets.conf)"
|
||||
)
|
||||
self.api_url = api_url.rstrip("/")
|
||||
self._api_key = api_key
|
||||
self.model = model
|
||||
self.dim = dim
|
||||
self.batch_size = max(1, batch_size)
|
||||
self.timeout = timeout
|
||||
self.attempts = max(1, attempts)
|
||||
self.backoff = backoff
|
||||
self._calls = 0
|
||||
self._texts = 0
|
||||
|
||||
def embed(self, texts: list) -> list:
|
||||
out = []
|
||||
for start in range(0, len(texts), self.batch_size):
|
||||
batch = texts[start : start + self.batch_size]
|
||||
out.extend(self._embed_batch(batch))
|
||||
return out
|
||||
|
||||
def _embed_batch(self, batch: list) -> list:
|
||||
body = json.dumps({"model": self.model, "input": batch}).encode("utf-8")
|
||||
last_error = None
|
||||
for attempt in range(self.attempts):
|
||||
request = urllib.request.Request(
|
||||
f"{self.api_url}/v1/embeddings",
|
||||
data=body,
|
||||
method="POST",
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {self._api_key}",
|
||||
},
|
||||
)
|
||||
self._calls += 1
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=self.timeout) as response:
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
vectors = _vectors_from_response(payload, self.model, self.dim)
|
||||
self._texts += len(batch)
|
||||
return vectors
|
||||
except urllib.error.HTTPError as exc:
|
||||
last_error = f"HTTP {exc.code} from {self.api_url}/v1/embeddings"
|
||||
exc.close()
|
||||
# 4xx other than 429 is our fault: do not hammer the gateway.
|
||||
if exc.code != 429 and exc.code < 500:
|
||||
raise EmbeddingError(f"{last_error} (not retryable)") from None
|
||||
except (urllib.error.URLError, TimeoutError) as exc:
|
||||
last_error = f"cannot reach {self.api_url}: {exc.reason}"
|
||||
except EmbeddingError:
|
||||
raise
|
||||
if attempt + 1 < self.attempts:
|
||||
time.sleep(self.backoff ** attempt)
|
||||
raise EmbeddingError(f"{last_error} after {self.attempts} attempts")
|
||||
|
||||
def stats(self) -> dict:
|
||||
return {"calls": self._calls, "texts": self._texts}
|
||||
|
||||
def reset_stats(self) -> None:
|
||||
self._calls = 0
|
||||
self._texts = 0
|
||||
|
||||
|
||||
def _vectors_from_response(payload: dict, model: str, dim: int) -> list:
|
||||
data = payload.get("data")
|
||||
if not isinstance(data, list) or not data:
|
||||
raise EmbeddingError("embedding response has no 'data' array")
|
||||
ordered = sorted(data, key=lambda item: item.get("index", 0))
|
||||
vectors = []
|
||||
for item in ordered:
|
||||
vector = item.get("embedding")
|
||||
if not isinstance(vector, list) or not vector:
|
||||
raise EmbeddingError("embedding response item has no 'embedding' array")
|
||||
if dim and len(vector) != dim:
|
||||
raise EmbeddingError(
|
||||
f"provider returned dimension {len(vector)} but the collection is "
|
||||
f"configured for {dim}; refusing to mix dimensions"
|
||||
)
|
||||
vectors.append([float(value) for value in vector])
|
||||
returned_model = payload.get("model")
|
||||
if returned_model and model and returned_model != model:
|
||||
raise EmbeddingError(
|
||||
f"provider answered as model {returned_model!r}, not {model!r}; "
|
||||
"a model change must be a governed migration"
|
||||
)
|
||||
return vectors
|
||||
|
||||
|
||||
def build_provider(cfg) -> Provider:
|
||||
"""Construct the provider named by the configuration."""
|
||||
if cfg.provider == "local-hashing":
|
||||
return LocalHashingProvider(model=cfg.model, dim=cfg.dim)
|
||||
if cfg.provider == "omniroute":
|
||||
return OmniRouteProvider(
|
||||
api_url=cfg.api_url,
|
||||
api_key=cfg.api_key,
|
||||
model=cfg.model,
|
||||
dim=cfg.dim,
|
||||
batch_size=cfg.batch_size,
|
||||
timeout=cfg.timeout,
|
||||
)
|
||||
raise EmbeddingError(f"unknown embedding provider {cfg.provider!r}")
|
||||
|
||||
|
||||
def _tokens(text: str) -> list:
|
||||
out = []
|
||||
current = []
|
||||
for char in text.lower():
|
||||
if char.isalnum():
|
||||
current.append(char)
|
||||
elif current:
|
||||
out.append("".join(current))
|
||||
current = []
|
||||
if current:
|
||||
out.append("".join(current))
|
||||
return out
|
||||
197
corpus/freshness.py
Normal file
@@ -0,0 +1,197 @@
|
||||
"""Freshness audit — the thing that stops the corpus rotting silently.
|
||||
|
||||
The risk table in `docs/phase-4-corpus.md` names it: *"Silent staleness… a
|
||||
stale index is otherwise an invisible quality failure."* So the audit is a first
|
||||
class command, it runs nightly from Prefect, and it alerts through the existing
|
||||
Apprise service when it finds something.
|
||||
|
||||
It reports four kinds of finding, and they are different problems:
|
||||
|
||||
* **stale** — a document whose `last_verified` is older than the horizon. The
|
||||
claim on the page may no longer be true.
|
||||
* **due soon** — inside the warning window. Re-verify before it becomes stale.
|
||||
* **index drift** — the source changed (or disappeared, or was never indexed)
|
||||
since it was embedded. Retrieval is answering from an older world.
|
||||
* **contract drift** — the collection was built with a different model,
|
||||
dimension or chunker than the running configuration.
|
||||
|
||||
Exit code is 0 for a clean audit and 1 when findings exist and `--fail-on-findings`
|
||||
was passed, so the Prefect deployment can alert on the result without parsing
|
||||
text.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import json
|
||||
import urllib.request
|
||||
|
||||
from . import loader, store as store_module
|
||||
from .config import CHUNKER_VERSION
|
||||
from .pipeline import _contract
|
||||
from .qdrant import Qdrant
|
||||
|
||||
|
||||
def audit(cfg, today: datetime.date = None, log=None) -> dict:
|
||||
"""Return the audit report as a plain dict (JSON-serialisable)."""
|
||||
log = log or (lambda _msg: None)
|
||||
today = today or datetime.date.today()
|
||||
documents = loader.load_documents(cfg)
|
||||
|
||||
findings = {"stale": [], "due_soon": [], "index_drift": [], "contract_drift": []}
|
||||
|
||||
for doc in documents:
|
||||
try:
|
||||
verified = datetime.date.fromisoformat(doc.last_verified)
|
||||
except ValueError:
|
||||
findings["index_drift"].append(
|
||||
{"source_path": doc.path, "problem": f"invalid last_verified {doc.last_verified!r}"}
|
||||
)
|
||||
continue
|
||||
age = (today - verified).days
|
||||
entry = {
|
||||
"source_path": doc.path,
|
||||
"title": doc.title,
|
||||
"area": doc.area,
|
||||
"confidence": doc.confidence,
|
||||
"last_verified": doc.last_verified,
|
||||
"age_days": age,
|
||||
}
|
||||
if age > cfg.max_age_days:
|
||||
findings["stale"].append(entry)
|
||||
elif age > cfg.max_age_days - cfg.warn_days:
|
||||
findings["due_soon"].append(entry)
|
||||
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
manifest = store.manifest()
|
||||
on_disk = {doc.path: doc for doc in documents}
|
||||
|
||||
for doc in documents:
|
||||
row = manifest.get(doc.path)
|
||||
if row is None:
|
||||
findings["index_drift"].append(
|
||||
{"source_path": doc.path, "problem": "not indexed (run sync)"}
|
||||
)
|
||||
continue
|
||||
if row["content_hash"] != doc.digest:
|
||||
findings["index_drift"].append(
|
||||
{"source_path": doc.path, "problem": "source changed since it was embedded"}
|
||||
)
|
||||
if row["chunker_version"] != CHUNKER_VERSION:
|
||||
findings["index_drift"].append(
|
||||
{
|
||||
"source_path": doc.path,
|
||||
"problem": f"chunked with {row['chunker_version']}, current {CHUNKER_VERSION}",
|
||||
}
|
||||
)
|
||||
|
||||
for source in sorted(set(manifest) - set(on_disk)):
|
||||
findings["index_drift"].append(
|
||||
{"source_path": source, "problem": "indexed but no longer in content/"}
|
||||
)
|
||||
|
||||
stored_contract = store.get_meta("collection")
|
||||
if stored_contract != _contract(cfg):
|
||||
findings["contract_drift"].append(
|
||||
{"problem": "collection contract does not match this configuration",
|
||||
"stored": stored_contract, "configured": _contract(cfg)}
|
||||
)
|
||||
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
reachable = qdrant.health()
|
||||
points = None
|
||||
if reachable and qdrant.collection_exists(cfg.collection):
|
||||
points = qdrant.point_count(cfg.collection)
|
||||
elif not reachable:
|
||||
findings["contract_drift"].append(
|
||||
{"problem": f"Qdrant is not answering at {cfg.qdrant_url}"}
|
||||
)
|
||||
|
||||
report = {
|
||||
"generated_at": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
"today": today.isoformat(),
|
||||
"horizon": {
|
||||
"max_age_days": cfg.max_age_days,
|
||||
"warn_days": cfg.warn_days,
|
||||
},
|
||||
"corpus": {
|
||||
"documents": len(documents),
|
||||
"chunks": store.chunk_count(),
|
||||
"sources": store.source_count(),
|
||||
"collection": cfg.collection,
|
||||
"collection_points": points,
|
||||
"provider": cfg.provider,
|
||||
"model": cfg.model,
|
||||
"dim": cfg.dim,
|
||||
"chunker_version": CHUNKER_VERSION,
|
||||
},
|
||||
"findings": findings,
|
||||
"summary": {
|
||||
"stale": len(findings["stale"]),
|
||||
"due_soon": len(findings["due_soon"]),
|
||||
"index_drift": len(findings["index_drift"]),
|
||||
"contract_drift": len(findings["contract_drift"]),
|
||||
},
|
||||
# `due_soon` is a warning, not a failure: the document is still
|
||||
# inside its verification horizon. Stale, drift and contract
|
||||
# problems are failures.
|
||||
"ok": not (
|
||||
findings["stale"] or findings["index_drift"] or findings["contract_drift"]
|
||||
),
|
||||
}
|
||||
log(f"audit: {json.dumps(report['summary'], sort_keys=True)}")
|
||||
return report
|
||||
|
||||
|
||||
def render_text(report: dict) -> str:
|
||||
"""Human-readable audit, for a terminal or an Apprise body."""
|
||||
lines = []
|
||||
corpus = report["corpus"]
|
||||
lines.append(
|
||||
f"corpus: {corpus['documents']} document(s), {corpus['chunks']} chunk(s), "
|
||||
f"{corpus['collection_points']} vector(s) in {corpus['collection']}"
|
||||
)
|
||||
lines.append(
|
||||
f"horizon: last_verified within {report['horizon']['max_age_days']} days "
|
||||
f"(warning at {report['horizon']['max_age_days'] - report['horizon']['warn_days']} days)"
|
||||
)
|
||||
findings = report["findings"]
|
||||
for kind, label in (
|
||||
("stale", "STALE — past the verification horizon"),
|
||||
("due_soon", "DUE SOON — verify before it goes stale"),
|
||||
("index_drift", "INDEX DRIFT — the index does not match content/"),
|
||||
("contract_drift", "CONTRACT DRIFT — rebuild required"),
|
||||
):
|
||||
rows = findings[kind]
|
||||
if not rows:
|
||||
continue
|
||||
lines.append("")
|
||||
lines.append(f"{label} ({len(rows)})")
|
||||
for row in rows:
|
||||
if "problem" in row:
|
||||
lines.append(f" - {row.get('source_path', '<collection>')}: {row['problem']}")
|
||||
else:
|
||||
lines.append(
|
||||
f" - {row['source_path']} ({row['area']}, {row['confidence']}): "
|
||||
f"last_verified {row['last_verified']} ({row['age_days']} days ago)"
|
||||
)
|
||||
if report["ok"]:
|
||||
lines.append("")
|
||||
lines.append("OK: nothing stale, no drift." + (" (see warnings above)" if findings["due_soon"] else ""))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def alert(report: dict, apprise_url: str, timeout: float = 10.0) -> bool:
|
||||
"""POST the audit to the existing Apprise service. No-op without a URL."""
|
||||
if not apprise_url:
|
||||
return False
|
||||
title = "ai-resume corpus audit: findings" if not report["ok"] else "ai-resume corpus audit: OK"
|
||||
body = json.dumps({"title": title, "body": render_text(report), "type": "warning" if not report["ok"] else "info"})
|
||||
request = urllib.request.Request(
|
||||
apprise_url,
|
||||
data=body.encode("utf-8"),
|
||||
method="POST",
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=timeout):
|
||||
return True
|
||||
162
corpus/loader.py
Normal file
@@ -0,0 +1,162 @@
|
||||
"""Load `content/` into documents, reusing the Phase 1 validator's parser.
|
||||
|
||||
There is exactly one front-matter dialect in this repository and exactly one
|
||||
parser for it: `tools/validate_content.py`. This module imports that parser
|
||||
rather than reimplementing it, so the corpus can never disagree with the gate
|
||||
about what a document says.
|
||||
|
||||
The gate itself is `validate_tree` from the same file. The pipeline runs it
|
||||
before it chunks anything: an invalid content tree stops the run, it never
|
||||
reaches the vector store.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import os
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
VALIDATOR_NAME = "validate_content"
|
||||
|
||||
|
||||
class ContentError(RuntimeError):
|
||||
"""The content tree is unreadable or invalid; the run must stop."""
|
||||
|
||||
|
||||
def load_validator(root: str):
|
||||
"""Import tools/validate_content.py as a module (cached per root)."""
|
||||
cached = sys.modules.get(VALIDATOR_NAME)
|
||||
if cached is not None and getattr(cached, "__validator_root__", None) == root:
|
||||
return cached
|
||||
path = os.path.join(root, "tools", "validate_content.py")
|
||||
if not os.path.exists(path):
|
||||
raise ContentError(f"validator not found at {path}")
|
||||
spec = importlib.util.spec_from_file_location(VALIDATOR_NAME, path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise ContentError(f"cannot load validator at {path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
module.__validator_root__ = root # type: ignore[attr-defined]
|
||||
sys.modules[VALIDATOR_NAME] = module
|
||||
return module
|
||||
|
||||
|
||||
def to_python(node):
|
||||
"""Convert a validator front-matter Node into plain Python values."""
|
||||
if node is None or node.kind == "null":
|
||||
return None
|
||||
if node.kind == "scalar":
|
||||
return node.value
|
||||
if node.kind == "seq":
|
||||
return [to_python(item) for item in node.value]
|
||||
if node.kind == "map":
|
||||
return {key: to_python(value) for key, value in node.value.items()}
|
||||
raise ContentError(f"unexpected front-matter node kind {node.kind!r}")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Document:
|
||||
"""One Markdown file plus the front-matter the schema guarantees."""
|
||||
|
||||
path: str # repository-relative, posix, e.g. "content/edge/l3/edge-zigbee.md"
|
||||
source_id: str
|
||||
area: str
|
||||
level: int
|
||||
title: str
|
||||
claim: str
|
||||
confidence: str
|
||||
period: str
|
||||
tech: list
|
||||
metrics: list
|
||||
artefacts: list
|
||||
public: bool
|
||||
last_verified: str
|
||||
tags: list
|
||||
body: str
|
||||
digest: str # sha256 of the file bytes: the incremental-mode key
|
||||
url: str # canonical page URL, matching internal/content.PageURL
|
||||
|
||||
# Filled in by the pipeline as it goes; not part of the schema.
|
||||
chunk_count: int = field(default=0)
|
||||
|
||||
@property
|
||||
def artefact_urls(self) -> list:
|
||||
return [str(a.get("url", "")) for a in self.artefacts if a.get("url")]
|
||||
|
||||
|
||||
def page_url(area: str, doc_id: str, level: int) -> str:
|
||||
"""Canonical URL for a document. Mirrors internal/content.PageURL exactly."""
|
||||
if level == 1:
|
||||
return f"/{area}/"
|
||||
if level == 2:
|
||||
return f"/{area}/{doc_id}/"
|
||||
return f"/{area}/a/{doc_id}/"
|
||||
|
||||
|
||||
def digest_of(path: str) -> str:
|
||||
"""sha256 of the raw file bytes — the same digest for any platform."""
|
||||
with open(path, "rb") as handle:
|
||||
return hashlib.sha256(handle.read()).hexdigest()
|
||||
|
||||
|
||||
def load_documents(cfg) -> list:
|
||||
"""Load every document under content/, sorted by path.
|
||||
|
||||
Raises ContentError when the gate fails, so a broken content tree never
|
||||
produces a half-correct index.
|
||||
"""
|
||||
validator = load_validator(cfg.root)
|
||||
violations, count = validator.validate_tree(cfg.root, cfg.content_dir, quiet=True)
|
||||
if violations:
|
||||
detail = "\n ".join(violations[:20])
|
||||
more = "" if len(violations) <= 20 else f"\n … and {len(violations) - 20} more"
|
||||
raise ContentError(
|
||||
f"content gate failed with {len(violations)} violation(s) "
|
||||
f"across {count} document(s). Run `make validate`:\n {detail}{more}"
|
||||
)
|
||||
|
||||
documents = []
|
||||
for path in validator.iter_documents(cfg.content_dir):
|
||||
documents.append(_load_one(cfg, validator, path))
|
||||
documents.sort(key=lambda d: d.path)
|
||||
if not documents:
|
||||
raise ContentError(f"no documents found under {cfg.content_dir}")
|
||||
return documents
|
||||
|
||||
|
||||
def _load_one(cfg, validator, path: str) -> Document:
|
||||
rel = os.path.relpath(path, cfg.root).replace(os.sep, "/")
|
||||
fm_lines, body, problem = validator.split_front_matter(path)
|
||||
if problem:
|
||||
raise ContentError(f"{rel}: {problem}")
|
||||
try:
|
||||
root_node = validator.parse_front_matter(fm_lines)
|
||||
except validator.YamlError as exc:
|
||||
raise ContentError(f"{rel}:{exc.line}: invalid YAML: {exc.msg}") from exc
|
||||
fields = to_python(root_node)
|
||||
if not isinstance(fields, dict):
|
||||
raise ContentError(f"{rel}: front-matter must be a mapping")
|
||||
|
||||
level = int(fields["level"])
|
||||
doc = Document(
|
||||
path=rel,
|
||||
source_id=str(fields["id"]),
|
||||
area=str(fields["area"]),
|
||||
level=level,
|
||||
title=str(fields["title"]),
|
||||
claim=str(fields["claim"]),
|
||||
confidence=str(fields["confidence"]),
|
||||
period=str(fields["period"]),
|
||||
tech=list(fields.get("tech") or []),
|
||||
metrics=list(fields.get("metrics") or []),
|
||||
artefacts=list(fields.get("artefacts") or []),
|
||||
public=bool(fields.get("public")),
|
||||
last_verified=str(fields["last_verified"]),
|
||||
tags=list(fields.get("tags") or []),
|
||||
body=body,
|
||||
digest=digest_of(path),
|
||||
url=page_url(str(fields["area"]), str(fields["id"]), level),
|
||||
)
|
||||
return doc
|
||||
242
corpus/measure.py
Normal file
@@ -0,0 +1,242 @@
|
||||
"""Qdrant memory at 500 / 5,000 / 20,000 chunks.
|
||||
|
||||
`docs/phase-4-corpus.md` requires the measurement and `reference/stack.md` is
|
||||
where the number is recorded. The corpus itself is small; the point of the
|
||||
measurement is to show the *shape* of the growth so the 400 m cap is a known
|
||||
quantity rather than a hope.
|
||||
|
||||
Method:
|
||||
|
||||
1. Create an empty collection and sample memory — the baseline.
|
||||
2. For each size: recreate the collection, upsert that many chunks with payloads
|
||||
shaped exactly like production payloads (same fields, same text lengths),
|
||||
wait for the write to settle, then sample.
|
||||
|
||||
Memory is read from Qdrant's own Prometheus `app_process_resident_memory_bytes`,
|
||||
which is the process that would be capped, not the container sandbox. When
|
||||
`--container` is also given the container's counter is recorded next to it.
|
||||
|
||||
The measurement collection is dropped afterwards unless `--keep` is passed, and
|
||||
it is never the production collection name.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import statistics
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from . import chunker, embeddings, loader, payloads
|
||||
from .qdrant import Qdrant
|
||||
|
||||
# Qdrant 1.20 exposes memory_resident_bytes; older builds used the Prometheus
|
||||
# default app_process_resident_memory_bytes. Accept either.
|
||||
RSS_METRIC = re.compile(
|
||||
r"^(?:memory_resident_bytes|app_process_resident_memory_bytes)\s+([0-9.eE+]+)",
|
||||
re.MULTILINE,
|
||||
)
|
||||
INGEST_BATCH = 250
|
||||
INGEST_SETTLE = 0.4
|
||||
MEMUSAGE = re.compile(r"([0-9.]+)\s*([KMG]i?B)", re.IGNORECASE)
|
||||
CAP_MIB = 400 # reference/stack.md memory budget for Qdrant
|
||||
|
||||
|
||||
def measure(cfg, sizes=(500, 5000, 20000), container: str = "", keep: bool = False, log=None) -> dict:
|
||||
log = log or (lambda _msg: None)
|
||||
collection = f"{cfg.collection}_measure"
|
||||
provider = embeddings.build_provider(cfg)
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=max(cfg.timeout, 60.0))
|
||||
if not qdrant.health():
|
||||
raise RuntimeError(f"Qdrant is not answering at {cfg.qdrant_url}")
|
||||
|
||||
templates = _templates(cfg, provider)
|
||||
log(f"measure: {len(templates)} payload template(s), dim={cfg.dim}, collection={collection}")
|
||||
|
||||
qdrant.drop_collection(collection)
|
||||
_create(qdrant, cfg, collection)
|
||||
time.sleep(1.0)
|
||||
baseline = _sample(qdrant, container)
|
||||
|
||||
samples = []
|
||||
for size in sorted(int(value) for value in sizes):
|
||||
qdrant.drop_collection(collection)
|
||||
_create(qdrant, cfg, collection)
|
||||
points = _points(templates, size, provider)
|
||||
started = time.time()
|
||||
# Throttled ingest: the peak during a bulk write is what trips the
|
||||
# cgroup limit, not the settled size. One batch at a time, sampling as we
|
||||
# go, so the measurement reports the peak and not just the end state.
|
||||
sent = 0
|
||||
peak = baseline["rss_mib"]
|
||||
for start in range(0, len(points), INGEST_BATCH):
|
||||
sent += qdrant.upsert(collection, points[start : start + INGEST_BATCH], batch=INGEST_BATCH)
|
||||
time.sleep(INGEST_SETTLE)
|
||||
reading = _rss_mib(qdrant)
|
||||
if reading is not None:
|
||||
peak = max(peak, reading)
|
||||
elapsed = round(time.time() - started, 2)
|
||||
points_now = qdrant.point_count(collection)
|
||||
time.sleep(2.0)
|
||||
reading = _sample(qdrant, container)
|
||||
growth = reading["rss_mib"] - baseline["rss_mib"]
|
||||
samples.append(
|
||||
{
|
||||
"chunks_requested": size,
|
||||
"points": points_now,
|
||||
"sent": sent,
|
||||
"upsert_seconds": elapsed,
|
||||
"rss_mib": reading["rss_mib"],
|
||||
"peak_rss_mib": round(peak, 1),
|
||||
"container_mib": reading["container_mib"],
|
||||
"growth_mib": round(growth, 1),
|
||||
"bytes_per_chunk": int(growth * 1024 * 1024 / size) if size else None,
|
||||
}
|
||||
)
|
||||
log(
|
||||
f"measure: {size} chunk(s) -> {points_now} point(s), rss {reading['rss_mib']:.1f} MiB "
|
||||
f"(+{growth:.1f} MiB), upsert {elapsed}s"
|
||||
)
|
||||
|
||||
if not keep:
|
||||
qdrant.drop_collection(collection)
|
||||
|
||||
report = {
|
||||
"collection": collection,
|
||||
"dim": cfg.dim,
|
||||
"distance": "Cosine",
|
||||
"provider": cfg.provider,
|
||||
"model": cfg.model,
|
||||
"qdrant_version": qdrant.version(),
|
||||
"cap_mib": CAP_MIB,
|
||||
"baseline": baseline,
|
||||
"samples": samples,
|
||||
"within_cap": all(
|
||||
max(sample["rss_mib"], sample["peak_rss_mib"]) <= CAP_MIB for sample in samples
|
||||
),
|
||||
"measured_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"kept": keep,
|
||||
}
|
||||
return report
|
||||
|
||||
|
||||
def _create(qdrant: Qdrant, cfg, collection: str) -> None:
|
||||
"""Same collection settings as production, so the memory reading means something."""
|
||||
qdrant.create_collection(
|
||||
collection, cfg.dim, vectors_on_disk=cfg.vectors_on_disk
|
||||
)
|
||||
|
||||
|
||||
def _templates(cfg, provider) -> list:
|
||||
"""Real payloads from content/, so field count and text length are realistic."""
|
||||
documents = loader.load_documents(cfg)
|
||||
out = []
|
||||
for doc in documents:
|
||||
for chunk in chunker.chunk_document(doc.body):
|
||||
payload = payloads.build_payload(doc, chunk, cfg)
|
||||
payload["id"] = "template"
|
||||
out.append(payload)
|
||||
if not out:
|
||||
raise RuntimeError("no content to build a measurement from")
|
||||
return out
|
||||
|
||||
|
||||
def _points(templates: list, size: int, provider) -> list:
|
||||
"""`size` points with unique ids, unique source paths and real payload shape."""
|
||||
points = []
|
||||
texts = []
|
||||
for index in range(size):
|
||||
template = templates[index % len(templates)]
|
||||
payload = dict(template)
|
||||
payload["id"] = str(uuid.UUID(hex=hashlib.sha256(f"measure:{index}".encode()).hexdigest()[:32]))
|
||||
payload["source_path"] = f"measure/{template['area']}/{index}.md"
|
||||
payload["chunk_index"] = index
|
||||
points.append({"id": payload["id"], "payload": payload})
|
||||
texts.append(f"{index} {payload['text']}")
|
||||
vectors = provider.embed(texts)
|
||||
for point, vector in zip(points, vectors):
|
||||
point["vector"] = vector
|
||||
return points
|
||||
|
||||
|
||||
def _sample(qdrant: Qdrant, container: str) -> dict:
|
||||
"""Three readings two seconds apart; report the median so a GC pause is not the number."""
|
||||
readings = []
|
||||
for attempt in range(3):
|
||||
readings.append(_rss_mib(qdrant))
|
||||
if attempt < 2:
|
||||
time.sleep(2.0)
|
||||
readings = [value for value in readings if value is not None]
|
||||
rss = round(statistics.median(readings), 1) if readings else 0.0
|
||||
return {"rss_mib": rss, "container_mib": _container_mib(container) if container else None}
|
||||
|
||||
|
||||
def _rss_mib(qdrant: Qdrant):
|
||||
text = qdrant.metrics()
|
||||
if not text:
|
||||
return None
|
||||
match = RSS_METRIC.search(text)
|
||||
if not match:
|
||||
return None
|
||||
return float(match.group(1)) / (1024 * 1024)
|
||||
|
||||
|
||||
def _container_mib(container: str):
|
||||
"""Container memory from `docker stats`. None when docker or the container is absent."""
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
["docker", "stats", "--no-stream", "--format", "{{.MemUsage}}", container],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=20,
|
||||
check=False,
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
if completed.returncode != 0:
|
||||
return None
|
||||
match = MEMUSAGE.search(completed.stdout)
|
||||
if not match:
|
||||
return None
|
||||
value = float(match.group(1))
|
||||
unit = match.group(2).upper()
|
||||
if unit.startswith("G"):
|
||||
value *= 1024
|
||||
elif unit.startswith("K"):
|
||||
value /= 1024
|
||||
return round(value, 1)
|
||||
|
||||
|
||||
def render_text(report: dict) -> str:
|
||||
lines = [
|
||||
f"Qdrant {report['qdrant_version']} · collection {report['collection']} · "
|
||||
f"dim {report['dim']} cosine · provider {report['provider']}/{report['model']}",
|
||||
f"baseline (empty collection): {report['baseline']['rss_mib']:.1f} MiB",
|
||||
"",
|
||||
f"{'chunks':>8} {'points':>8} {'rss MiB':>8} {'peak':>7} {'growth':>7} {'B/chunk':>8} {'upsert s':>8}",
|
||||
]
|
||||
for sample in report["samples"]:
|
||||
per_chunk = sample["bytes_per_chunk"] or 0
|
||||
lines.append(
|
||||
f"{sample['chunks_requested']:>8} {sample['points']:>8} {sample['rss_mib']:>8.1f} "
|
||||
f"{sample['peak_rss_mib']:>7.1f} {sample['growth_mib']:>+7.1f} {per_chunk:>8} "
|
||||
f"{sample['upsert_seconds']:>8}"
|
||||
)
|
||||
lines.append("")
|
||||
verdict = "within" if report["within_cap"] else "OVER"
|
||||
lines.append(f"cap {report['cap_mib']} MiB: {verdict} cap at every measured size")
|
||||
if report["kept"]:
|
||||
lines.append(f"collection {report['collection']} was kept for inspection")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
from .config import Config
|
||||
|
||||
cfg = Config.from_env(dim=int(sys.argv[1]) if len(sys.argv) > 1 else None)
|
||||
print(render_text(measure(cfg, container="", log=print)))
|
||||
155
corpus/payloads.py
Normal file
@@ -0,0 +1,155 @@
|
||||
"""Chunk payloads: the thing the retrieval filter runs against.
|
||||
|
||||
`docs/phase-4-corpus.md` §4.2 fixes the payload. `area` and `confidence` are the
|
||||
two that matter: they let the query path narrow *before* it searches, and they
|
||||
let an answer qualify unfinished work instead of presenting it as delivered.
|
||||
|
||||
**A missing or invalid field fails the run.** Nothing in this module supplies a
|
||||
default value: `validate_payload` raises on the first field that is absent,
|
||||
empty or out of enum, and the pipeline reports which document and chunk it was.
|
||||
That is the DoD item "a missing field fails the run rather than defaulting".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
|
||||
AREAS = ("ai", "platform", "edge", "products", "history", "about")
|
||||
LEVELS = (1, 2, 3)
|
||||
SOURCE_TYPES = ("doc", "code", "adr", "metric")
|
||||
CONFIDENCE = ("delivered", "running", "in-progress", "abandoned", "historic")
|
||||
|
||||
# Fields every payload must carry (docs/phase-4-corpus.md §4.2), plus the ones
|
||||
# the index needs to answer "where did this come from?" without a second lookup.
|
||||
REQUIRED_FIELDS = (
|
||||
"area",
|
||||
"level",
|
||||
"source_type",
|
||||
"confidence",
|
||||
"period",
|
||||
"artefacts",
|
||||
"last_verified",
|
||||
"text",
|
||||
)
|
||||
PROVENANCE_FIELDS = (
|
||||
"id",
|
||||
"source_path",
|
||||
"source_id",
|
||||
"chunk_index",
|
||||
"title",
|
||||
"claim",
|
||||
"url",
|
||||
"content_hash",
|
||||
"embedding_model",
|
||||
"embedding_dim",
|
||||
"chunker_version",
|
||||
)
|
||||
|
||||
|
||||
class PayloadError(RuntimeError):
|
||||
"""A payload is incomplete or out of enum. The run must stop."""
|
||||
|
||||
|
||||
def source_type_for(doc) -> str:
|
||||
"""Classify a document. Everything in content/ is a doc unless it says so.
|
||||
|
||||
The schema has no `source_type` field and the schema is frozen, so the
|
||||
classification is derived from tags the schema does allow.
|
||||
"""
|
||||
tags = {str(tag).lower() for tag in doc.tags}
|
||||
if "adr" in tags:
|
||||
return "adr"
|
||||
if "metric" in tags:
|
||||
return "metric"
|
||||
if "code" in tags:
|
||||
return "code"
|
||||
return "doc"
|
||||
|
||||
|
||||
def build_payload(doc, chunk, cfg) -> dict:
|
||||
"""Build one chunk payload from a document and a chunk.
|
||||
|
||||
Values are copied, never guessed: if the document does not carry a field,
|
||||
the payload does not gain one.
|
||||
"""
|
||||
payload = {
|
||||
# -- the retrieval filter (docs/phase-4-corpus.md §4.2) ------------
|
||||
"area": doc.area,
|
||||
"level": doc.level,
|
||||
"source_type": source_type_for(doc),
|
||||
"confidence": doc.confidence,
|
||||
"period": doc.period,
|
||||
"artefacts": [str(a.get("url", "")) for a in doc.artefacts],
|
||||
"last_verified": doc.last_verified,
|
||||
"text": chunk.text,
|
||||
# -- provenance ----------------------------------------------------
|
||||
"id": None, # filled by the pipeline once the id is computed
|
||||
"source_path": doc.path,
|
||||
"source_id": doc.source_id,
|
||||
"chunk_index": chunk.index,
|
||||
"title": doc.title,
|
||||
"claim": doc.claim,
|
||||
"url": doc.url,
|
||||
"content_hash": doc.digest,
|
||||
"embedding_model": cfg.model,
|
||||
"embedding_dim": cfg.dim,
|
||||
"chunker_version": cfg.chunker_version,
|
||||
# -- supporting evidence, used by the answer to cite ---------------
|
||||
"public": bool(doc.public),
|
||||
"tech": list(doc.tech),
|
||||
"tags": list(doc.tags),
|
||||
"chunk_tokens": chunk.tokens,
|
||||
}
|
||||
return payload
|
||||
|
||||
|
||||
def validate_payload(payload: dict, where: str = "chunk") -> None:
|
||||
"""Raise PayloadError if a required field is missing, empty or out of enum."""
|
||||
for field in REQUIRED_FIELDS + PROVENANCE_FIELDS:
|
||||
if field == "id" and payload.get("id") is None:
|
||||
raise PayloadError(f"{where}: payload field 'id' was not set")
|
||||
if field not in payload:
|
||||
raise PayloadError(f"{where}: payload is missing required field {field!r}")
|
||||
|
||||
area = payload["area"]
|
||||
if area not in AREAS:
|
||||
raise PayloadError(f"{where}: area {area!r} is not one of {list(AREAS)}")
|
||||
if payload["level"] not in LEVELS:
|
||||
raise PayloadError(f"{where}: level {payload['level']!r} is not one of {list(LEVELS)}")
|
||||
if payload["source_type"] not in SOURCE_TYPES:
|
||||
raise PayloadError(
|
||||
f"{where}: source_type {payload['source_type']!r} is not one of {list(SOURCE_TYPES)}"
|
||||
)
|
||||
if payload["confidence"] not in CONFIDENCE:
|
||||
raise PayloadError(
|
||||
f"{where}: confidence {payload['confidence']!r} is not one of {list(CONFIDENCE)}"
|
||||
)
|
||||
|
||||
for field in ("period", "title", "claim", "url", "source_path", "source_id", "embedding_model"):
|
||||
value = payload[field]
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise PayloadError(f"{where}: field {field!r} must be a non-empty string")
|
||||
|
||||
text = payload["text"]
|
||||
if not isinstance(text, str) or not text.strip():
|
||||
raise PayloadError(f"{where}: field 'text' must be non-empty")
|
||||
|
||||
if not isinstance(payload["artefacts"], list):
|
||||
raise PayloadError(f"{where}: field 'artefacts' must be a list (empty is allowed)")
|
||||
|
||||
_check_date(payload["last_verified"], where)
|
||||
if not isinstance(payload["embedding_dim"], int) or payload["embedding_dim"] <= 0:
|
||||
raise PayloadError(f"{where}: field 'embedding_dim' must be a positive integer")
|
||||
if not isinstance(payload["chunk_index"], int) or payload["chunk_index"] < 0:
|
||||
raise PayloadError(f"{where}: field 'chunk_index' must be a non-negative integer")
|
||||
if not isinstance(payload["content_hash"], str) or len(payload["content_hash"]) != 64:
|
||||
raise PayloadError(f"{where}: field 'content_hash' must be a sha256 hex digest")
|
||||
|
||||
|
||||
def _check_date(value: str, where: str) -> None:
|
||||
try:
|
||||
datetime.date.fromisoformat(str(value))
|
||||
except ValueError as exc:
|
||||
raise PayloadError(
|
||||
f"{where}: field 'last_verified' must be an ISO date (YYYY-MM-DD), got {value!r}"
|
||||
) from exc
|
||||
426
corpus/pipeline.py
Normal file
@@ -0,0 +1,426 @@
|
||||
"""The pipeline: content/ -> validate -> chunk -> enrich -> embed -> upsert.
|
||||
|
||||
One entry point, `run`, with two modes:
|
||||
|
||||
* **rebuild** — drop the collection, empty the store, embed everything. This is
|
||||
the "one command rebuilds the corpus from an empty volume" path.
|
||||
* **sync** — the default. Only documents whose content hash, embedding model,
|
||||
embedding dimension or chunker version changed are re-chunked and
|
||||
re-embedded. Everything else is left alone.
|
||||
|
||||
The order matters. Payloads are built and validated for *every* changed document
|
||||
before a single vector is written, so an invalid payload stops the run with an
|
||||
empty diff rather than leaving half an index behind.
|
||||
|
||||
Deterministic ids (`sha256(source_path + chunk_index + content_hash)`) mean
|
||||
re-running updates points instead of duplicating them, and the id is a valid
|
||||
UUID so Qdrant accepts it as a point id without a mapping table.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
from dataclasses import asdict, dataclass, field
|
||||
|
||||
from . import chunker, embeddings, loader, payloads, store as store_module
|
||||
from .config import CHUNKER_VERSION
|
||||
from .qdrant import Qdrant, QdrantError
|
||||
|
||||
|
||||
class PipelineError(RuntimeError):
|
||||
"""The run cannot continue without corrupting the index."""
|
||||
|
||||
|
||||
def chunk_id(source_path: str, chunk_index: int, content_hash: str) -> str:
|
||||
"""Deterministic point id: sha256(source_path + chunk_index + content_hash).
|
||||
|
||||
Rendered as a UUID because Qdrant only accepts unsigned integers or UUIDs as
|
||||
point ids; the first 128 bits of the digest keep collisions improbable.
|
||||
"""
|
||||
digest = hashlib.sha256(
|
||||
f"{source_path}\x00{chunk_index}\x00{content_hash}".encode("utf-8")
|
||||
).hexdigest()
|
||||
return str(uuid.UUID(hex=digest[:32]))
|
||||
|
||||
|
||||
@dataclass
|
||||
class RunStats:
|
||||
mode: str = "sync"
|
||||
documents: int = 0
|
||||
added: int = 0
|
||||
updated: int = 0
|
||||
unchanged: int = 0
|
||||
removed: int = 0
|
||||
chunks_written: int = 0
|
||||
chunks_deleted: int = 0
|
||||
embed_calls: int = 0
|
||||
texts_embedded: int = 0
|
||||
points_sent: int = 0
|
||||
collection_points: int = 0
|
||||
store_chunks: int = 0
|
||||
duration_s: float = 0.0
|
||||
collection: str = ""
|
||||
provider: str = ""
|
||||
model: str = ""
|
||||
dim: int = 0
|
||||
chunker_version: str = CHUNKER_VERSION
|
||||
notes: list = field(default_factory=list)
|
||||
|
||||
def summary(self) -> str:
|
||||
rows = [
|
||||
f"mode={self.mode} documents={self.documents} chunks={self.store_chunks}",
|
||||
f"documents added={self.added} updated={self.updated} unchanged={self.unchanged} removed={self.removed}",
|
||||
f"chunks written={self.chunks_written} deleted={self.chunks_deleted}",
|
||||
f"embedding calls={self.embed_calls} texts={self.texts_embedded}",
|
||||
f"points sent={self.points_sent} collection points={self.collection_points}",
|
||||
f"provider={self.provider} model={self.model} dim={self.dim} chunker={self.chunker_version}",
|
||||
f"collection={self.collection} duration={self.duration_s:.2f}s",
|
||||
]
|
||||
rows.extend(self.notes)
|
||||
return "\n".join(rows)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
def run(cfg, mode: str = "sync", log=None) -> RunStats:
|
||||
"""Build or refresh the corpus. Returns what happened; raises on any error."""
|
||||
if mode not in ("sync", "rebuild"):
|
||||
raise PipelineError(f"unknown mode {mode!r}; expected 'sync' or 'rebuild'")
|
||||
started = time.time()
|
||||
log = log or (lambda _msg: None)
|
||||
stats = RunStats(
|
||||
mode=mode,
|
||||
collection=cfg.collection,
|
||||
provider=cfg.provider,
|
||||
model=cfg.model,
|
||||
dim=cfg.dim,
|
||||
chunker_version=cfg.chunker_version,
|
||||
)
|
||||
|
||||
documents = loader.load_documents(cfg)
|
||||
stats.documents = len(documents)
|
||||
log(f"validated {len(documents)} document(s)")
|
||||
|
||||
provider = embeddings.build_provider(cfg)
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
if not qdrant.health():
|
||||
raise PipelineError(
|
||||
f"Qdrant is not answering at {cfg.qdrant_url}. Start it "
|
||||
"(deploy/qdrant.yml) or point RESUME_QDRANT_URL at a running instance."
|
||||
)
|
||||
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
contract = _contract(cfg)
|
||||
mode = _check_contract(store, qdrant, cfg, contract, mode)
|
||||
|
||||
_ensure_collection(qdrant, cfg)
|
||||
|
||||
manifest = store.manifest()
|
||||
changed, removed = _classify(documents, manifest, cfg, mode)
|
||||
stats.unchanged = len(documents) - len(changed)
|
||||
stats.removed = len(removed)
|
||||
log(
|
||||
f"incremental plan: {len(changed)} to embed, "
|
||||
f"{stats.unchanged} unchanged, {len(removed)} removed"
|
||||
)
|
||||
|
||||
for path in removed:
|
||||
stats.chunks_deleted += store.delete_document(path)
|
||||
qdrant.delete_by_source(cfg.collection, path)
|
||||
log(f"removed {path}")
|
||||
|
||||
for doc in changed:
|
||||
payloads_for_doc = _prepare(doc, cfg, manifest)
|
||||
stats.chunks_written += len(payloads_for_doc)
|
||||
|
||||
vectors = provider.embed([p["text"] for p in payloads_for_doc])
|
||||
if len(vectors) != len(payloads_for_doc):
|
||||
raise PipelineError(
|
||||
f"{doc.path}: provider returned {len(vectors)} vectors for "
|
||||
f"{len(payloads_for_doc)} chunks"
|
||||
)
|
||||
|
||||
points = []
|
||||
for payload, vector in zip(payloads_for_doc, vectors):
|
||||
if len(vector) != cfg.dim:
|
||||
raise PipelineError(
|
||||
f"{doc.path}: vector dimension {len(vector)} != configured {cfg.dim}"
|
||||
)
|
||||
points.append({"id": payload["id"], "vector": vector, "payload": payload})
|
||||
|
||||
# Order: remove what this document used to have (chunk counts change),
|
||||
# then write the new rows and points. A crash between the two leaves an
|
||||
# index that reports as incomplete, never one that claims to be whole.
|
||||
stats.chunks_deleted += store.delete_document(doc.path)
|
||||
qdrant.delete_by_source(cfg.collection, doc.path)
|
||||
stats.points_sent += qdrant.upsert(cfg.collection, points)
|
||||
store.write_document(doc, payloads_for_doc)
|
||||
|
||||
if doc.path in manifest:
|
||||
stats.updated += 1
|
||||
log(f"updated {doc.path} ({len(points)} chunk(s))")
|
||||
else:
|
||||
stats.added += 1
|
||||
log(f"added {doc.path} ({len(points)} chunk(s))")
|
||||
|
||||
provider_stats = provider.stats()
|
||||
stats.embed_calls = provider_stats["calls"]
|
||||
stats.texts_embedded = provider_stats["texts"]
|
||||
stats.store_chunks = store.chunk_count()
|
||||
stats.collection_points = qdrant.point_count(cfg.collection)
|
||||
|
||||
if stats.collection_points != stats.store_chunks:
|
||||
raise PipelineError(
|
||||
"index is inconsistent after the run: "
|
||||
f"{stats.collection_points} point(s) in Qdrant but "
|
||||
f"{stats.store_chunks} chunk(s) in SQLite. Run `python3 -m corpus.cli verify`."
|
||||
)
|
||||
|
||||
store.set_meta("collection", contract)
|
||||
store.set_meta("last_run", {**stats.to_dict(), "finished_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
|
||||
_write_collection_file(cfg, store, stats)
|
||||
|
||||
stats.duration_s = round(time.time() - started, 3)
|
||||
return stats
|
||||
|
||||
|
||||
def _contract(cfg) -> dict:
|
||||
"""What this collection *is*. Recorded so a change is a migration, not drift."""
|
||||
return {
|
||||
"provider": cfg.provider,
|
||||
"model": cfg.model,
|
||||
"dim": cfg.dim,
|
||||
"chunker_version": cfg.chunker_version,
|
||||
"distance": "Cosine",
|
||||
}
|
||||
|
||||
|
||||
def _check_contract(store, qdrant, cfg, contract: dict, mode: str) -> str:
|
||||
stored = store.get_meta("collection")
|
||||
if mode == "rebuild":
|
||||
qdrant.drop_collection(cfg.collection)
|
||||
store.reset()
|
||||
return mode
|
||||
if stored is None:
|
||||
# No contract yet: either a fresh store, or one whose database was lost.
|
||||
if qdrant.collection_exists(cfg.collection) and qdrant.point_count(cfg.collection) > 0:
|
||||
raise PipelineError(
|
||||
f"collection {cfg.collection!r} holds points but this store has no "
|
||||
"collection contract (was data/corpus/corpus.db deleted?). "
|
||||
"Confirm you want to overwrite it: python3 -m corpus.cli rebuild"
|
||||
)
|
||||
return mode
|
||||
if stored != contract:
|
||||
raise PipelineError(
|
||||
"the collection contract does not match this configuration — refusing to "
|
||||
"mix vectors in one collection (docs/phase-4-corpus.md §4.3).\n"
|
||||
f" stored: {json.dumps(stored, sort_keys=True)}\n"
|
||||
f" config: {json.dumps(contract, sort_keys=True)}\n"
|
||||
"A provider or chunker change is a governed migration: run "
|
||||
"`python3 -m corpus.cli rebuild` once Sam has approved the change."
|
||||
)
|
||||
return mode
|
||||
|
||||
|
||||
def _ensure_collection(qdrant: Qdrant, cfg) -> None:
|
||||
if not qdrant.collection_exists(cfg.collection):
|
||||
qdrant.create_collection(
|
||||
cfg.collection, cfg.dim, vectors_on_disk=cfg.vectors_on_disk
|
||||
)
|
||||
return
|
||||
info = qdrant.collection_info(cfg.collection)
|
||||
vectors = ((info.get("config") or {}).get("params") or {}).get("vectors")
|
||||
size = None
|
||||
if isinstance(vectors, dict):
|
||||
size = vectors.get("size")
|
||||
if size is not None and int(size) != int(cfg.dim):
|
||||
raise PipelineError(
|
||||
f"collection {cfg.collection!r} has vectors of size {size} but this run "
|
||||
f"is configured for {cfg.dim}. That is a governed migration: rebuild."
|
||||
)
|
||||
|
||||
|
||||
def _classify(documents: list, manifest: dict, cfg, mode: str):
|
||||
"""Which documents need work, and which are gone."""
|
||||
changed = []
|
||||
on_disk = set()
|
||||
for doc in documents:
|
||||
on_disk.add(doc.path)
|
||||
if mode == "rebuild":
|
||||
changed.append(doc)
|
||||
continue
|
||||
entry = manifest.get(doc.path)
|
||||
if entry is None:
|
||||
changed.append(doc)
|
||||
continue
|
||||
if (
|
||||
entry["content_hash"] != doc.digest
|
||||
or entry["embedding_model"] != cfg.model
|
||||
or int(entry["embedding_dim"]) != int(cfg.dim)
|
||||
or entry["chunker_version"] != cfg.chunker_version
|
||||
):
|
||||
changed.append(doc)
|
||||
removed = sorted(path for path in manifest if path not in on_disk)
|
||||
return changed, removed
|
||||
|
||||
|
||||
def _prepare(doc, cfg, manifest: dict) -> list:
|
||||
"""Chunk one document and build validated payloads. No network, no writes."""
|
||||
try:
|
||||
chunks = chunker.chunk_document(doc.body)
|
||||
except chunker.ChunkError as exc:
|
||||
raise PipelineError(f"{doc.path}: {exc}") from exc
|
||||
if not chunks:
|
||||
raise PipelineError(f"{doc.path}: produced no chunks (empty body?)")
|
||||
doc.chunk_count = len(chunks)
|
||||
|
||||
out = []
|
||||
for chunk in chunks:
|
||||
payload = payloads.build_payload(doc, chunk, cfg)
|
||||
payload["id"] = chunk_id(doc.path, chunk.index, doc.digest)
|
||||
try:
|
||||
payloads.validate_payload(payload, where=f"{doc.path}#chunk-{chunk.index}")
|
||||
except payloads.PayloadError as exc:
|
||||
raise PipelineError(str(exc)) from exc
|
||||
out.append(payload)
|
||||
return out
|
||||
|
||||
|
||||
def _write_collection_file(cfg, store, stats: RunStats) -> None:
|
||||
"""Mirror the contract to data/corpus/collection.json for the operator."""
|
||||
body = {
|
||||
"collection": cfg.collection,
|
||||
"contract": _contract(cfg),
|
||||
"qdrant_url": cfg.qdrant_url,
|
||||
"chunks": store.chunk_count(),
|
||||
"sources": store.source_count(),
|
||||
"last_run": store.get_meta("last_run"),
|
||||
}
|
||||
with open(cfg.meta_path, "w", encoding="utf-8") as handle:
|
||||
json.dump(body, handle, indent=2, sort_keys=True)
|
||||
handle.write("\n")
|
||||
|
||||
|
||||
# -- verification and demonstration ---------------------------------------
|
||||
|
||||
|
||||
def verify(cfg, log=None) -> list:
|
||||
"""Return a list of problems. Empty means the index agrees with content/."""
|
||||
log = log or (lambda _msg: None)
|
||||
problems = []
|
||||
documents = loader.load_documents(cfg)
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
if not qdrant.health():
|
||||
return [f"Qdrant is not answering at {cfg.qdrant_url}"]
|
||||
|
||||
contract = _contract(cfg)
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
stored = store.get_meta("collection")
|
||||
if stored != contract:
|
||||
problems.append(f"collection contract mismatch: stored={stored} config={contract}")
|
||||
|
||||
if not qdrant.collection_exists(cfg.collection):
|
||||
problems.append(f"collection {cfg.collection!r} does not exist")
|
||||
return problems
|
||||
|
||||
info = qdrant.collection_info(cfg.collection)
|
||||
size = (((info.get("config") or {}).get("params") or {}).get("vectors") or {}).get("size")
|
||||
if size is not None and int(size) != cfg.dim:
|
||||
problems.append(f"collection vector size {size} != configured {cfg.dim}")
|
||||
|
||||
points_by_source = qdrant.scroll_sources(cfg.collection)
|
||||
rows = store.chunk_payloads()
|
||||
by_source = {}
|
||||
for row in rows:
|
||||
by_source.setdefault(row["source_path"], []).append(row)
|
||||
try:
|
||||
payloads.validate_payload(row, where=f"{row['source_path']}#chunk-{row['chunk_index']}")
|
||||
except payloads.PayloadError as exc:
|
||||
problems.append(str(exc))
|
||||
|
||||
for doc in documents:
|
||||
payload_rows = by_source.get(doc.path, [])
|
||||
if not payload_rows:
|
||||
problems.append(f"{doc.path}: no chunks in the index (run sync)")
|
||||
continue
|
||||
if payload_rows[0]["content_hash"] != doc.digest:
|
||||
problems.append(f"{doc.path}: source changed since it was embedded")
|
||||
expected = len(payload_rows)
|
||||
actual = points_by_source.get(doc.path, 0)
|
||||
if actual != expected:
|
||||
problems.append(
|
||||
f"{doc.path}: {expected} chunk row(s) but {actual} vector(s) in Qdrant"
|
||||
)
|
||||
|
||||
for source in sorted(set(by_source) - {doc.path for doc in documents}):
|
||||
problems.append(f"{source}: indexed but no longer in content/ (run sync)")
|
||||
|
||||
total_points = sum(points_by_source.values())
|
||||
if total_points != store.chunk_count():
|
||||
problems.append(
|
||||
f"{total_points} vector(s) in Qdrant but {store.chunk_count()} chunk(s) in SQLite"
|
||||
)
|
||||
log(
|
||||
f"verify: {store.chunk_count()} chunk(s) across {store.source_count()} source(s), "
|
||||
f"{total_points} vector(s) in Qdrant"
|
||||
)
|
||||
return problems
|
||||
|
||||
|
||||
def demo_area_filter(cfg, query: str, area: str = None, limit: int = 5) -> dict:
|
||||
"""Same query with and without an `area` filter — the DoD's filter evidence."""
|
||||
provider = embeddings.build_provider(cfg)
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
vector = provider.embed([query])[0]
|
||||
unfiltered = qdrant.search(cfg.collection, vector, limit=limit)
|
||||
filtered = qdrant.search(cfg.collection, vector, limit=limit, area=area) if area else []
|
||||
return {
|
||||
"query": query,
|
||||
"area": area,
|
||||
"unfiltered": [_hit(row) for row in unfiltered],
|
||||
"filtered": [_hit(row) for row in filtered],
|
||||
}
|
||||
|
||||
|
||||
def _hit(row: dict) -> dict:
|
||||
payload = row.get("payload") or {}
|
||||
return {
|
||||
"score": round(row["score"], 4),
|
||||
"area": payload.get("area"),
|
||||
"confidence": payload.get("confidence"),
|
||||
"title": payload.get("title"),
|
||||
"url": payload.get("url"),
|
||||
"source_path": payload.get("source_path"),
|
||||
"chunk_index": payload.get("chunk_index"),
|
||||
"preview": (payload.get("text") or "").replace("\n", " ")[:120],
|
||||
}
|
||||
|
||||
|
||||
def hybrid_search(cfg, query: str, area: str = None, limit: int = 5) -> list:
|
||||
"""Vectors + FTS5 fused with reciprocal rank fusion (decision D10).
|
||||
|
||||
Phase 5 owns the query graph; this is here because Phase 4 is the phase that
|
||||
has to prove the two halves line up — same ids, same metadata, one ranking.
|
||||
"""
|
||||
provider = embeddings.build_provider(cfg)
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
vector = provider.embed([query])[0]
|
||||
ranked = {}
|
||||
for rank, hit in enumerate(qdrant.search(cfg.collection, vector, limit=limit * 4, area=area)):
|
||||
entry = ranked.setdefault(hit["id"], {"id": hit["id"], "payload": hit["payload"], "score": 0.0})
|
||||
entry["score"] += 1.0 / (60 + rank)
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
for rank, hit in enumerate(store.keyword_search(query, area=area, limit=limit * 4)):
|
||||
entry = ranked.setdefault(hit["id"], {"id": hit["id"], "payload": {}, "score": 0.0})
|
||||
entry.setdefault("keyword", hit["snippet"])
|
||||
entry["score"] += 1.0 / (60 + rank)
|
||||
ordered = sorted(ranked.values(), key=lambda row: row["score"], reverse=True)[:limit]
|
||||
for row in ordered:
|
||||
payload = row.get("payload") or {}
|
||||
row["title"] = payload.get("title")
|
||||
row["url"] = payload.get("url")
|
||||
return ordered
|
||||
269
corpus/qdrant.py
Normal file
@@ -0,0 +1,269 @@
|
||||
"""Qdrant REST client — standard library only.
|
||||
|
||||
No `qdrant-client` dependency: the four calls this pipeline needs are HTTP
|
||||
requests, and AGENTS.md §4.1 freezes the stack. `_request` is the single
|
||||
transport hook, which is what the tests replace to run the whole pipeline
|
||||
without a container.
|
||||
|
||||
Qdrant is bound to `127.0.0.1:6333` on the runtime host (`.13`) and is never
|
||||
publicly reachable; Caddy is the only public listener.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import http.client
|
||||
import json
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
DISTANCE_COSINE = "Cosine"
|
||||
|
||||
# The runtime host (.13) is memory-bound: 2.3 GiB free and 8 GiB of swap already
|
||||
# in use, so Qdrant gets a 400 m cap and must live inside it
|
||||
# (reference/stack.md). Measured on 2026-10-26: the *default* HNSW and payload
|
||||
# settings are OOM-killed at the cap between 5,000 and 20,000 chunks of 1024
|
||||
# dimensions. Keeping the HNSW graph and the payloads on disk, and using a
|
||||
# smaller graph (m=8, ef_construct=64), fits the cap because the kernel can
|
||||
# reclaim those page-cache pages instead of killing the process. Vectors stay in
|
||||
# RAM because they are the part search reads on every query.
|
||||
#
|
||||
# The trade is build time and a little search latency; the corpus is ~50 chunks
|
||||
# today and the public site has no query burst. See docs/phase-4-notes.md §7.
|
||||
MEMORY_FRUGAL_DEFAULTS = {
|
||||
"on_disk_payload": True,
|
||||
"hnsw_m": 8,
|
||||
"hnsw_ef_construct": 64,
|
||||
"vectors_on_disk": False,
|
||||
}
|
||||
|
||||
# Segment behaviour under a cap. `memmap_threshold` makes Qdrant mmap segments
|
||||
# this size and larger (page cache the kernel can reclaim instead of OOM), and
|
||||
# `indexing_threshold` is raised so a bulk ingest does not build the HNSW graph
|
||||
# while it is still writing segments. Measured: 20,000 chunks at 1024
|
||||
# dimensions peaked at 296 MiB during ingest and settled to 96 MiB.
|
||||
MEMORY_FRUGAL_OPTIMIZERS = {
|
||||
"default_segment_number": 2,
|
||||
"memmap_threshold": 5000,
|
||||
"indexing_threshold": 20000,
|
||||
}
|
||||
|
||||
|
||||
class QdrantError(RuntimeError):
|
||||
"""A Qdrant call failed, or Qdrant is not reachable."""
|
||||
|
||||
|
||||
class Qdrant:
|
||||
def __init__(self, url: str = "http://127.0.0.1:6333", timeout: float = 30.0, attempts: int = 3) -> None:
|
||||
self.url = url.rstrip("/")
|
||||
self.timeout = timeout
|
||||
self.attempts = max(1, attempts)
|
||||
|
||||
# -- transport --------------------------------------------------------
|
||||
|
||||
def _request(self, method: str, path: str, body=None) -> dict:
|
||||
"""One HTTP call with retry/backoff. The only place that touches the network."""
|
||||
data = None if body is None else json.dumps(body).encode("utf-8")
|
||||
last_error = None
|
||||
for attempt in range(self.attempts):
|
||||
request = urllib.request.Request(
|
||||
f"{self.url}{path}",
|
||||
data=data,
|
||||
method=method,
|
||||
headers={"Content-Type": "application/json"} if data else {},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=self.timeout) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
return json.loads(raw) if raw.strip() else {}
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", "replace")[:400]
|
||||
exc.close()
|
||||
if exc.code == 404:
|
||||
raise QdrantNotFound(f"{method} {path}: {detail}") from None
|
||||
if exc.code != 429 and exc.code < 500:
|
||||
raise QdrantError(f"{method} {path} failed: HTTP {exc.code} {detail}") from None
|
||||
last_error = f"HTTP {exc.code} {detail}"
|
||||
except (urllib.error.URLError, TimeoutError, http.client.HTTPException, ConnectionError) as exc:
|
||||
# RemoteDisconnected and friends arrive here when the server dies
|
||||
# mid-request (an OOM kill looks exactly like this).
|
||||
last_error = f"cannot reach {self.url}: {getattr(exc, 'reason', exc)}"
|
||||
if attempt + 1 < self.attempts:
|
||||
time.sleep(0.5 * (2 ** attempt))
|
||||
raise QdrantError(f"{method} {path} failed after {self.attempts} attempts: {last_error}")
|
||||
|
||||
# -- cluster ----------------------------------------------------------
|
||||
|
||||
def health(self) -> bool:
|
||||
"""True when Qdrant answers. Never raises."""
|
||||
try:
|
||||
self._request("GET", "/")
|
||||
return True
|
||||
except (QdrantError, QdrantNotFound):
|
||||
return False
|
||||
|
||||
def version(self) -> str:
|
||||
response = self._request("GET", "/")
|
||||
return str(response.get("version", "unknown"))
|
||||
|
||||
# -- collections ------------------------------------------------------
|
||||
|
||||
def collection_exists(self, name: str) -> bool:
|
||||
try:
|
||||
self._request("GET", f"/collections/{name}")
|
||||
return True
|
||||
except QdrantNotFound:
|
||||
return False
|
||||
|
||||
def create_collection(
|
||||
self,
|
||||
name: str,
|
||||
dim: int,
|
||||
distance: str = DISTANCE_COSINE,
|
||||
on_disk_payload: bool = None,
|
||||
hnsw_m: int = None,
|
||||
hnsw_ef_construct: int = None,
|
||||
vectors_on_disk: bool = None,
|
||||
) -> None:
|
||||
"""Create a collection sized for a memory-bound host (see MEMORY_FRUGAL_DEFAULTS)."""
|
||||
defaults = MEMORY_FRUGAL_DEFAULTS
|
||||
body = {
|
||||
"vectors": {
|
||||
"size": int(dim),
|
||||
"distance": distance,
|
||||
"on_disk": bool(
|
||||
defaults["vectors_on_disk"] if vectors_on_disk is None else vectors_on_disk
|
||||
),
|
||||
},
|
||||
"hnsw_config": {
|
||||
"m": int(defaults["hnsw_m"] if hnsw_m is None else hnsw_m),
|
||||
"ef_construct": int(
|
||||
defaults["hnsw_ef_construct"] if hnsw_ef_construct is None else hnsw_ef_construct
|
||||
),
|
||||
"on_disk": True,
|
||||
},
|
||||
"on_disk_payload": bool(
|
||||
defaults["on_disk_payload"] if on_disk_payload is None else on_disk_payload
|
||||
),
|
||||
"optimizers_config": dict(MEMORY_FRUGAL_OPTIMIZERS),
|
||||
}
|
||||
self._request("PUT", f"/collections/{name}", body)
|
||||
|
||||
def drop_collection(self, name: str) -> None:
|
||||
try:
|
||||
self._request("DELETE", f"/collections/{name}")
|
||||
except QdrantNotFound:
|
||||
pass
|
||||
|
||||
def collection_info(self, name: str) -> dict:
|
||||
response = self._request("GET", f"/collections/{name}")
|
||||
return dict(response.get("result") or {})
|
||||
|
||||
def point_count(self, name: str) -> int:
|
||||
response = self._request("POST", f"/collections/{name}/points/count", {"exact": True})
|
||||
return int((response.get("result") or {}).get("count", 0))
|
||||
|
||||
# -- points -----------------------------------------------------------
|
||||
|
||||
def upsert(self, name: str, points: list, batch: int = 256) -> int:
|
||||
"""Upsert points in batches. Returns the number of points sent."""
|
||||
sent = 0
|
||||
for start in range(0, len(points), max(1, batch)):
|
||||
chunk = points[start : start + max(1, batch)]
|
||||
self._request(
|
||||
"PUT",
|
||||
f"/collections/{name}/points?wait=true",
|
||||
{"points": chunk},
|
||||
)
|
||||
sent += len(chunk)
|
||||
return sent
|
||||
|
||||
def delete_by_source(self, name: str, source_path: str) -> None:
|
||||
"""Delete every point that came from one document.
|
||||
|
||||
Re-chunking a document can produce fewer chunks than before, so the old
|
||||
points are removed rather than overwritten. Deterministic ids make
|
||||
duplicates impossible; this makes orphans impossible.
|
||||
"""
|
||||
self._request(
|
||||
"POST",
|
||||
f"/collections/{name}/points/delete?wait=true",
|
||||
{"filter": {"must": [{"key": "source_path", "match": {"value": source_path}}]}},
|
||||
)
|
||||
|
||||
def search(
|
||||
self,
|
||||
name: str,
|
||||
vector: list,
|
||||
limit: int = 5,
|
||||
area: str = None,
|
||||
confidence: str = None,
|
||||
with_payload: bool = True,
|
||||
) -> list:
|
||||
"""Nearest neighbours, optionally filtered *before* the search."""
|
||||
body = {
|
||||
"query": [float(value) for value in vector],
|
||||
"limit": int(limit),
|
||||
"with_payload": bool(with_payload),
|
||||
}
|
||||
filters = _filter_conditions(area, confidence)
|
||||
if filters:
|
||||
body["filter"] = {"must": filters}
|
||||
try:
|
||||
response = self._request("POST", f"/collections/{name}/points/query", body)
|
||||
points = ((response.get("result") or {}).get("points")) or []
|
||||
except QdrantNotFound:
|
||||
# Older Qdrant exposes the same operation as /points/search.
|
||||
response = self._request("POST", f"/collections/{name}/points/search", body)
|
||||
points = response.get("result") or []
|
||||
return [
|
||||
{
|
||||
"id": point.get("id"),
|
||||
"score": float(point.get("score") or 0.0),
|
||||
"payload": point.get("payload") or {},
|
||||
}
|
||||
for point in points
|
||||
]
|
||||
|
||||
def scroll_sources(self, name: str, limit: int = 1000) -> dict:
|
||||
"""Map source_path -> number of vectors, for consistency checks."""
|
||||
counts = {}
|
||||
offset = None
|
||||
while True:
|
||||
body = {"limit": limit, "with_payload": ["source_path"], "with_vector": False}
|
||||
if offset is not None:
|
||||
body["offset"] = offset
|
||||
response = self._request("POST", f"/collections/{name}/points/scroll", body)
|
||||
result = response.get("result") or {}
|
||||
points = result.get("points") or []
|
||||
for point in points:
|
||||
source = (point.get("payload") or {}).get("source_path", "<missing>")
|
||||
counts[source] = counts.get(source, 0) + 1
|
||||
offset = result.get("next_page_offset")
|
||||
if not points or offset is None:
|
||||
return counts
|
||||
|
||||
def metrics(self) -> str:
|
||||
"""Prometheus text from Qdrant's /metrics route (used by the memory measurement).
|
||||
|
||||
Returns "" when the route is not exposed, so callers can fall back to
|
||||
reading the container's own memory counters.
|
||||
"""
|
||||
try:
|
||||
with urllib.request.urlopen(f"{self.url}/metrics", timeout=self.timeout) as response:
|
||||
return response.read().decode("utf-8", "replace")
|
||||
except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError):
|
||||
return ""
|
||||
|
||||
|
||||
class QdrantNotFound(QdrantError):
|
||||
"""A 404 from Qdrant: the collection or route does not exist."""
|
||||
|
||||
|
||||
def _filter_conditions(area: str = None, confidence: str = None) -> list:
|
||||
conditions = []
|
||||
if area:
|
||||
conditions.append({"key": "area", "match": {"value": area}})
|
||||
if confidence:
|
||||
conditions.append({"key": "confidence", "match": {"value": confidence}})
|
||||
return conditions
|
||||
295
corpus/store.py
Normal file
@@ -0,0 +1,295 @@
|
||||
"""SQLite: the keyword index and the incremental state.
|
||||
|
||||
Two jobs, one file (`RESUME_CORPUS_DB`, default `data/corpus/corpus.db`):
|
||||
|
||||
* **`chunk_fts`** — an FTS5 index over chunk text with the same filter columns
|
||||
the vectors carry. This is the keyword half of hybrid retrieval (decision
|
||||
D10). Phase 2 indexes *pages* for the site's own search box; this indexes
|
||||
*chunks*, because retrieval hands the model a passage, not a page.
|
||||
* **`manifest`** — one row per source document: its content hash, its chunk
|
||||
count and the embedding model/dimension it was embedded with. Incremental mode
|
||||
is a hash comparison against this table, and nothing else.
|
||||
|
||||
`meta` records the collection contract (provider, model, dimension, chunker
|
||||
version) so a provider change is detected instead of silently corrupting the
|
||||
index.
|
||||
|
||||
Standard library `sqlite3` only. The same file is opened read-only by the Phase
|
||||
5 retrieval service.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import time
|
||||
import unicodedata
|
||||
|
||||
SCHEMA_VERSION = "1"
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS chunks (
|
||||
id TEXT PRIMARY KEY,
|
||||
source_path TEXT NOT NULL,
|
||||
source_id TEXT NOT NULL,
|
||||
chunk_index INTEGER NOT NULL,
|
||||
area TEXT NOT NULL,
|
||||
level INTEGER NOT NULL,
|
||||
source_type TEXT NOT NULL,
|
||||
confidence TEXT NOT NULL,
|
||||
period TEXT NOT NULL,
|
||||
last_verified TEXT NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
claim TEXT NOT NULL,
|
||||
url TEXT NOT NULL,
|
||||
content_hash TEXT NOT NULL,
|
||||
embedding_model TEXT NOT NULL,
|
||||
embedding_dim INTEGER NOT NULL,
|
||||
chunker_version TEXT NOT NULL,
|
||||
chunk_tokens INTEGER NOT NULL,
|
||||
public INTEGER NOT NULL,
|
||||
artefacts TEXT NOT NULL,
|
||||
tech TEXT NOT NULL,
|
||||
tags TEXT NOT NULL,
|
||||
text TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS chunks_source ON chunks(source_path);
|
||||
CREATE INDEX IF NOT EXISTS chunks_area ON chunks(area);
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS chunk_fts USING fts5(
|
||||
id UNINDEXED,
|
||||
source_path UNINDEXED,
|
||||
title,
|
||||
terms,
|
||||
text,
|
||||
area UNINDEXED,
|
||||
confidence UNINDEXED,
|
||||
tokenize = 'porter unicode61 remove_diacritics 2'
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS manifest (
|
||||
source_path TEXT PRIMARY KEY,
|
||||
content_hash TEXT NOT NULL,
|
||||
chunk_count INTEGER NOT NULL,
|
||||
embedding_model TEXT NOT NULL,
|
||||
embedding_dim INTEGER NOT NULL,
|
||||
chunker_version TEXT NOT NULL,
|
||||
last_verified TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS meta (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
"""
|
||||
|
||||
# Column order used by every INSERT in this module.
|
||||
_CHUNK_COLUMNS = (
|
||||
"id", "source_path", "source_id", "chunk_index", "area", "level", "source_type",
|
||||
"confidence", "period", "last_verified", "title", "claim", "url", "content_hash",
|
||||
"embedding_model", "embedding_dim", "chunker_version", "chunk_tokens",
|
||||
"public", "artefacts", "tech", "tags", "text",
|
||||
)
|
||||
|
||||
_JSON_FIELDS = ("artefacts", "tech", "tags")
|
||||
|
||||
|
||||
class StoreError(RuntimeError):
|
||||
"""The corpus database is unreadable or inconsistent."""
|
||||
|
||||
|
||||
class Store:
|
||||
def __init__(self, path: str) -> None:
|
||||
self.path = path
|
||||
os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
|
||||
self.db = sqlite3.connect(path)
|
||||
self.db.row_factory = sqlite3.Row
|
||||
self.db.execute("PRAGMA journal_mode = WAL")
|
||||
self.db.execute("PRAGMA synchronous = NORMAL")
|
||||
self.db.executescript(SCHEMA)
|
||||
self.set_meta("schema_version", SCHEMA_VERSION)
|
||||
|
||||
# -- lifecycle --------------------------------------------------------
|
||||
|
||||
def close(self) -> None:
|
||||
self.db.close()
|
||||
|
||||
def __enter__(self) -> "Store":
|
||||
return self
|
||||
|
||||
def __exit__(self, *_exc) -> None:
|
||||
self.close()
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Empty every table. Used by a from-scratch rebuild."""
|
||||
for table in ("chunk_fts", "chunks", "manifest", "meta"):
|
||||
self.db.execute(f"DELETE FROM {table}")
|
||||
self.db.commit()
|
||||
|
||||
# -- writing ----------------------------------------------------------
|
||||
|
||||
def write_document(self, doc, payloads: list) -> None:
|
||||
"""Replace all rows for one document: chunks, FTS rows and manifest."""
|
||||
self.delete_document(doc.path, keep_manifest=True)
|
||||
for payload in payloads:
|
||||
values = []
|
||||
for column in _CHUNK_COLUMNS:
|
||||
value = payload[column]
|
||||
if column in _JSON_FIELDS:
|
||||
value = json.dumps(value, sort_keys=True)
|
||||
values.append(value)
|
||||
cursor = self.db.execute(
|
||||
f"INSERT INTO chunks ({', '.join(_CHUNK_COLUMNS)}) "
|
||||
f"VALUES ({', '.join('?' for _ in _CHUNK_COLUMNS)})",
|
||||
values,
|
||||
)
|
||||
# `terms` is the keyword surface: the words a visitor or a recruiter
|
||||
# would type that live in front-matter rather than the body — tech
|
||||
# names, tags, the one-line claim. Without it, a search for "zigbee"
|
||||
# misses a page whose body says "mesh". It is indexed, never retrieved.
|
||||
terms = " ".join(
|
||||
str(value)
|
||||
for value in (
|
||||
payload["claim"],
|
||||
payload["title"],
|
||||
" ".join(payload["tech"]),
|
||||
" ".join(payload["tags"]),
|
||||
)
|
||||
if value
|
||||
)
|
||||
self.db.execute(
|
||||
"INSERT INTO chunk_fts (rowid, id, source_path, title, terms, text, area, confidence) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
cursor.lastrowid,
|
||||
payload["id"],
|
||||
payload["source_path"],
|
||||
payload["title"],
|
||||
terms,
|
||||
payload["text"],
|
||||
payload["area"],
|
||||
payload["confidence"],
|
||||
),
|
||||
)
|
||||
self.db.execute(
|
||||
"INSERT OR REPLACE INTO manifest "
|
||||
"(source_path, content_hash, chunk_count, embedding_model, embedding_dim, "
|
||||
" chunker_version, last_verified, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
doc.path,
|
||||
doc.digest,
|
||||
len(payloads),
|
||||
payloads[0]["embedding_model"] if payloads else "",
|
||||
payloads[0]["embedding_dim"] if payloads else 0,
|
||||
payloads[0]["chunker_version"] if payloads else "",
|
||||
doc.last_verified,
|
||||
time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
),
|
||||
)
|
||||
self.db.commit()
|
||||
|
||||
def delete_document(self, source_path: str, keep_manifest: bool = False) -> int:
|
||||
rowids = [
|
||||
row["rowid"]
|
||||
for row in self.db.execute("SELECT rowid FROM chunks WHERE source_path = ?", (source_path,))
|
||||
]
|
||||
for rowid in rowids:
|
||||
self.db.execute("DELETE FROM chunk_fts WHERE rowid = ?", (rowid,))
|
||||
self.db.execute("DELETE FROM chunks WHERE source_path = ?", (source_path,))
|
||||
if not keep_manifest:
|
||||
self.db.execute("DELETE FROM manifest WHERE source_path = ?", (source_path,))
|
||||
self.db.commit()
|
||||
return len(rowids)
|
||||
|
||||
# -- reading ----------------------------------------------------------
|
||||
|
||||
def manifest(self) -> dict:
|
||||
return {
|
||||
row["source_path"]: dict(row)
|
||||
for row in self.db.execute("SELECT * FROM manifest")
|
||||
}
|
||||
|
||||
def chunk_count(self) -> int:
|
||||
return int(self.db.execute("SELECT count(*) FROM chunks").fetchone()[0])
|
||||
|
||||
def source_count(self) -> int:
|
||||
return int(self.db.execute("SELECT count(DISTINCT source_path) FROM chunks").fetchone()[0])
|
||||
|
||||
def chunk_ids(self) -> list:
|
||||
return [row["id"] for row in self.db.execute("SELECT id FROM chunks ORDER BY id")]
|
||||
|
||||
def chunk_payloads(self, source_path: str = None) -> list:
|
||||
"""Every stored payload, optionally for one document. Used by verify()."""
|
||||
sql = "SELECT * FROM chunks"
|
||||
args = ()
|
||||
if source_path:
|
||||
sql += " WHERE source_path = ?"
|
||||
args = (source_path,)
|
||||
sql += " ORDER BY source_path, chunk_index"
|
||||
out = []
|
||||
for row in self.db.execute(sql, args):
|
||||
payload = dict(row)
|
||||
for field in _JSON_FIELDS:
|
||||
payload[field] = json.loads(payload[field])
|
||||
out.append(payload)
|
||||
return out
|
||||
|
||||
def keyword_search(self, query: str, area: str = None, confidence: str = None, limit: int = 10) -> list:
|
||||
"""BM25 over chunk text, optionally filtered before ranking."""
|
||||
match = match_expression(query)
|
||||
if not match:
|
||||
return []
|
||||
sql = [
|
||||
"SELECT c.id, c.url, c.title, c.area, c.confidence, c.source_path,",
|
||||
" c.chunk_index, snippet(chunk_fts, 4, '<mark>', '</mark>', '…', 18) AS snippet,",
|
||||
" bm25(chunk_fts) AS score",
|
||||
"FROM chunk_fts JOIN chunks c ON c.rowid = chunk_fts.rowid",
|
||||
"WHERE chunk_fts MATCH ?",
|
||||
]
|
||||
args = [match]
|
||||
if area:
|
||||
sql.append("AND c.area = ?")
|
||||
args.append(area)
|
||||
if confidence:
|
||||
sql.append("AND c.confidence = ?")
|
||||
args.append(confidence)
|
||||
sql.append("ORDER BY bm25(chunk_fts) LIMIT ?")
|
||||
args.append(int(limit))
|
||||
rows = self.db.execute(" ".join(sql), args)
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
# -- meta -------------------------------------------------------------
|
||||
|
||||
def set_meta(self, key: str, value) -> None:
|
||||
self.db.execute(
|
||||
"INSERT OR REPLACE INTO meta (key, value) VALUES (?, ?)",
|
||||
(key, json.dumps(value, sort_keys=True)),
|
||||
)
|
||||
self.db.commit()
|
||||
|
||||
def get_meta(self, key: str, default=None):
|
||||
row = self.db.execute("SELECT value FROM meta WHERE key = ?", (key,)).fetchone()
|
||||
if row is None:
|
||||
return default
|
||||
return json.loads(row["value"])
|
||||
|
||||
def meta(self) -> dict:
|
||||
return {row["key"]: json.loads(row["value"]) for row in self.db.execute("SELECT key, value FROM meta")}
|
||||
|
||||
|
||||
def match_expression(query: str) -> str:
|
||||
"""Free text -> a safe FTS5 expression: bare words, each a prefix, all required.
|
||||
|
||||
Mirrors `MatchExpression` in internal/search/search.go so the site's search
|
||||
box and the retrieval path treat a visitor's words identically. FTS5
|
||||
operators in the input are never honoured.
|
||||
"""
|
||||
terms = []
|
||||
for raw in re.split(r"[^0-9a-zA-Z\-]+", unicodedata.normalize("NFKC", query or "").lower()):
|
||||
term = raw.strip("-")
|
||||
if term:
|
||||
terms.append('"' + term.replace('"', "") + '"*')
|
||||
return " AND ".join(terms)
|
||||
0
corpus/tests/__init__.py
Normal file
112
corpus/tests/fake_qdrant.py
Normal file
@@ -0,0 +1,112 @@
|
||||
"""An in-memory stand-in for Qdrant, used by the pipeline tests.
|
||||
|
||||
It implements the routes the pipeline actually calls and nothing else, and it is
|
||||
deliberately dumb: brute-force cosine over a dict. If the pipeline passes
|
||||
against this, the pipeline's own logic is right; the real server is exercised
|
||||
separately in `docs/phase-4-notes.md` (rebuild against a live container).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
from corpus.qdrant import Qdrant, QdrantError, QdrantNotFound
|
||||
|
||||
|
||||
class FakeQdrant(Qdrant):
|
||||
def __init__(self, dim: int):
|
||||
super().__init__(url="fake://qdrant", timeout=1.0, attempts=1)
|
||||
self.dim = dim
|
||||
self.collections = {} # name -> {id: {"vector": [...], "payload": {...}}}
|
||||
self.calls = []
|
||||
|
||||
# -- transport --------------------------------------------------------
|
||||
|
||||
def _request(self, method: str, path: str, body=None) -> dict:
|
||||
self.calls.append((method, path, body))
|
||||
query = ""
|
||||
if "?" in path:
|
||||
path, query = path.split("?", 1)
|
||||
parts = [part for part in path.split("/") if part]
|
||||
|
||||
if not parts:
|
||||
return {"version": "fake-1.0"}
|
||||
|
||||
if parts[0] != "collections":
|
||||
raise QdrantNotFound(f"no route {path}")
|
||||
if len(parts) == 2:
|
||||
name = parts[1]
|
||||
if method == "PUT":
|
||||
self.collections.setdefault(name, {})
|
||||
return {"result": True}
|
||||
if method == "DELETE":
|
||||
if name not in self.collections:
|
||||
raise QdrantNotFound("collection not found")
|
||||
del self.collections[name]
|
||||
return {"result": True}
|
||||
if name not in self.collections:
|
||||
raise QdrantNotFound("collection not found")
|
||||
return {
|
||||
"result": {
|
||||
"points_count": len(self.collections[name]),
|
||||
"config": {"params": {"vectors": {"size": self.dim, "distance": "Cosine"}}},
|
||||
}
|
||||
}
|
||||
name = parts[1]
|
||||
if name not in self.collections:
|
||||
raise QdrantNotFound("collection not found")
|
||||
points = self.collections[name]
|
||||
|
||||
if parts[2:4] == ["points", "count"]:
|
||||
return {"result": {"count": len(points)}}
|
||||
if len(parts) == 3 and parts[2] == "points" and method == "PUT":
|
||||
for point in (body or {}).get("points", []):
|
||||
if len(point["vector"]) != self.dim:
|
||||
raise QdrantError("wrong vector dimension")
|
||||
points[point["id"]] = {"vector": point["vector"], "payload": point["payload"]}
|
||||
return {"result": {"status": "completed"}}
|
||||
if parts[2:4] == ["points", "delete"]:
|
||||
must = (((body or {}).get("filter") or {}).get("must")) or []
|
||||
for point_id in list(points):
|
||||
if _matches(points[point_id]["payload"], must):
|
||||
del points[point_id]
|
||||
return {"result": {"status": "completed"}}
|
||||
if parts[2:4] == ["points", "query"]:
|
||||
vector = body["query"]
|
||||
must = (((body or {}).get("filter") or {}).get("must")) or []
|
||||
scored = []
|
||||
for point_id, point in points.items():
|
||||
if not _matches(point["payload"], must):
|
||||
continue
|
||||
scored.append(
|
||||
{
|
||||
"id": point_id,
|
||||
"score": _cosine(vector, point["vector"]),
|
||||
"payload": point["payload"],
|
||||
}
|
||||
)
|
||||
scored.sort(key=lambda row: row["score"], reverse=True)
|
||||
return {"result": {"points": scored[: int(body.get("limit", 10))]}}
|
||||
if parts[2:4] == ["points", "scroll"]:
|
||||
limit = int((body or {}).get("limit", 100))
|
||||
rows = [
|
||||
{"id": point_id, "payload": {"source_path": point["payload"].get("source_path")}}
|
||||
for point_id, point in list(points.items())[:limit]
|
||||
]
|
||||
return {"result": {"points": rows, "next_page_offset": None}}
|
||||
raise QdrantNotFound(f"no route {path}")
|
||||
|
||||
|
||||
def _matches(payload: dict, conditions: list) -> bool:
|
||||
for condition in conditions:
|
||||
key = condition.get("key")
|
||||
expected = (condition.get("match") or {}).get("value")
|
||||
if payload.get(key) != expected:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _cosine(left: list, right: list) -> float:
|
||||
dot = sum(a * b for a, b in zip(left, right))
|
||||
norm = math.sqrt(sum(a * a for a in left)) * math.sqrt(sum(b * b for b in right))
|
||||
return dot / norm if norm else 0.0
|
||||
186
corpus/tests/support.py
Normal file
@@ -0,0 +1,186 @@
|
||||
"""Shared fixtures for the corpus tests: a throwaway content tree and a fake Qdrant."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from unittest import mock
|
||||
|
||||
from corpus.config import Config
|
||||
from corpus.tests.fake_qdrant import FakeQdrant
|
||||
|
||||
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
DOC = """---
|
||||
id: {id}
|
||||
area: {area}
|
||||
level: {level}
|
||||
title: "{title}"
|
||||
claim: "{claim}"
|
||||
confidence: {confidence}
|
||||
period: "2023–"
|
||||
tech: [Go, SQLite]
|
||||
metrics: [{{label: endpoints, value: 8}}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: {last_verified}
|
||||
tags: [{tags}]
|
||||
---
|
||||
|
||||
# {title}
|
||||
|
||||
{body}
|
||||
"""
|
||||
|
||||
FENCE = """A paragraph before the code block explains the constraint.
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
func main() {{
|
||||
println("chunk boundary must never land inside this fence")
|
||||
}}
|
||||
```
|
||||
|
||||
A paragraph after the code block.
|
||||
"""
|
||||
|
||||
TABLE = """Measured results, one row per endpoint:
|
||||
|
||||
| endpoint | latency | loss |
|
||||
|---|---|---|
|
||||
| alpha | 12ms | 0.0% |
|
||||
| beta | 14ms | 0.1% |
|
||||
| gamma | 9ms | 0.0% |
|
||||
|
||||
And a closing paragraph.
|
||||
"""
|
||||
|
||||
PROSE = "\n\n".join(
|
||||
f"Paragraph {index} describes a delivery constraint in corporate language, "
|
||||
"with enough words that the chunker has to think about where to break, and "
|
||||
"never inside a fenced block, a table or a front-matter block."
|
||||
for index in range(1, 40)
|
||||
)
|
||||
|
||||
DEFAULT_DOCS = {
|
||||
"ai/ai-overview.md": dict(
|
||||
id="ai-overview", area="ai", level=1, title="Agent Systems",
|
||||
claim="Harnesses, memory and routing.", confidence="running",
|
||||
last_verified="2026-10-01", tags="agents, llm", body=PROSE,
|
||||
),
|
||||
"edge/l3/edge-zigbee.md": dict(
|
||||
id="edge-zigbee", area="edge", level=3, title="Low Power Mesh Sensing",
|
||||
claim="A constrained mesh that survives power loss.", confidence="delivered",
|
||||
last_verified="2026-09-15", tags="zigbee, mesh", body=FENCE,
|
||||
),
|
||||
"platform/l3/platform-containers.md": dict(
|
||||
id="platform-containers", area="platform", level=3, title="Container Estate",
|
||||
claim="Fifty containers with measured resource limits.", confidence="in-progress",
|
||||
last_verified="2026-08-01", tags="docker", body=TABLE,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def temp_repo(docs: dict = None) -> str:
|
||||
"""Create a throwaway repository root with tools/ and content/."""
|
||||
root = tempfile.mkdtemp(prefix="corpus-test-")
|
||||
os.makedirs(os.path.join(root, "tools"), exist_ok=True)
|
||||
shutil.copy(
|
||||
os.path.join(REPO_ROOT, "tools", "validate_content.py"),
|
||||
os.path.join(root, "tools", "validate_content.py"),
|
||||
)
|
||||
write_docs(root, docs or DEFAULT_DOCS)
|
||||
return root
|
||||
|
||||
|
||||
def write_docs(root: str, docs: dict) -> None:
|
||||
for rel, fields in docs.items():
|
||||
path = os.path.join(root, "content", rel)
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8") as handle:
|
||||
handle.write(DOC.format(**fields))
|
||||
|
||||
|
||||
def write_doc(root: str, rel: str, **fields) -> None:
|
||||
path = os.path.join(root, "content", rel)
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8") as handle:
|
||||
handle.write(DOC.format(**fields))
|
||||
# A fresh interpreter's import cache is fine; the loader re-reads the file.
|
||||
|
||||
|
||||
def make_config(root: str, **overrides) -> Config:
|
||||
values = dict(
|
||||
provider="local-hashing",
|
||||
model="local-hashing-v1",
|
||||
dim=32,
|
||||
qdrant_url="fake://qdrant",
|
||||
collection="test_corpus",
|
||||
)
|
||||
values.update(overrides)
|
||||
return Config.from_env(root=root, **values)
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def fake_qdrant(cfg, fake: FakeQdrant = None):
|
||||
"""Point every module that builds a Qdrant client at one in-memory fake."""
|
||||
fake = fake or FakeQdrant(cfg.dim)
|
||||
with mock.patch("corpus.pipeline.Qdrant", return_value=fake), mock.patch(
|
||||
"corpus.freshness.Qdrant", return_value=fake
|
||||
):
|
||||
yield fake
|
||||
|
||||
|
||||
class StubServer:
|
||||
"""A throwaway HTTP server on a loopback ephemeral port, for provider tests.
|
||||
|
||||
`responder(method, path, body)` returns `(status, payload)`; every request is
|
||||
recorded with its parsed body so a test can assert the wire shape without a
|
||||
real provider or a real Qdrant.
|
||||
"""
|
||||
|
||||
def __init__(self, responder):
|
||||
import json
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
self.requests = []
|
||||
self._responder = responder
|
||||
outer = self
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def _handle(self):
|
||||
length = int(self.headers.get("Content-Length") or 0)
|
||||
raw = self.rfile.read(length).decode("utf-8") if length else ""
|
||||
body = json.loads(raw) if raw.strip() else None
|
||||
outer.requests.append(
|
||||
{"method": self.command, "path": self.path, "body": body,
|
||||
"headers": dict(self.headers)}
|
||||
)
|
||||
status, payload = outer._responder(self.command, self.path, body)
|
||||
data = payload.encode("utf-8") if isinstance(payload, str) else json.dumps(payload).encode("utf-8")
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
do_GET = do_POST = do_PUT = do_DELETE = _handle
|
||||
|
||||
def log_message(self, *_args):
|
||||
pass
|
||||
|
||||
self.server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
self.url = f"http://127.0.0.1:{self.server.server_address[1]}"
|
||||
self._thread = threading.Thread(target=self.server.serve_forever, daemon=True)
|
||||
|
||||
def __enter__(self):
|
||||
self._thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *_exc):
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
145
corpus/tests/test_chunker.py
Normal file
@@ -0,0 +1,145 @@
|
||||
"""Chunking rules: size, overlap, determinism, and the hard one — no split inside
|
||||
a fence, a table or front-matter (docs/phase-4-corpus.md DoD item 3)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import unittest
|
||||
|
||||
from corpus import chunker, loader
|
||||
from corpus.config import Config
|
||||
from corpus.tests import support
|
||||
|
||||
REPO_ROOT = support.REPO_ROOT
|
||||
|
||||
|
||||
def content_documents():
|
||||
cfg = Config.from_env(root=REPO_ROOT, provider="local-hashing", dim=32)
|
||||
return loader.load_documents(cfg)
|
||||
|
||||
|
||||
class BlockParsingTest(unittest.TestCase):
|
||||
def test_fence_is_one_block(self):
|
||||
blocks = chunker.parse_blocks(support.FENCE)
|
||||
kinds = [block.kind for block in blocks]
|
||||
self.assertEqual(kinds, ["para", "fence", "para"])
|
||||
self.assertIn("chunk boundary must never land", blocks[1].text)
|
||||
|
||||
def test_table_is_one_block(self):
|
||||
blocks = chunker.parse_blocks(support.TABLE)
|
||||
table = [block for block in blocks if block.kind == "table"]
|
||||
self.assertEqual(len(table), 1)
|
||||
self.assertEqual(len(table[0].text.splitlines()), 5) # header, rule, 3 rows
|
||||
|
||||
def test_front_matter_in_body_is_refused(self):
|
||||
with self.assertRaises(chunker.ChunkError):
|
||||
chunker.parse_blocks("---\nid: x\n---\n\n# Heading\n")
|
||||
|
||||
def test_unterminated_fence_runs_to_end(self):
|
||||
blocks = chunker.parse_blocks("intro\n\n```go\nfunc main() {}\n")
|
||||
self.assertEqual(blocks[-1].kind, "fence")
|
||||
self.assertTrue(blocks[-1].text.rstrip().endswith("}"))
|
||||
|
||||
def test_headings_start_blocks(self):
|
||||
blocks = chunker.parse_blocks("# One\n\ntext\n\n## Two\n\nmore\n")
|
||||
self.assertEqual([block.kind for block in blocks], ["heading", "para", "heading", "para"])
|
||||
|
||||
def test_list_items_are_separate_blocks(self):
|
||||
blocks = chunker.parse_blocks("- one\n- two\n continued\n- three\n")
|
||||
self.assertEqual([block.kind for block in blocks], ["list", "list", "list"])
|
||||
self.assertIn("continued", blocks[1].text)
|
||||
|
||||
|
||||
class ChunkingTest(unittest.TestCase):
|
||||
def test_small_body_is_one_chunk(self):
|
||||
chunks = chunker.chunk_document("# Title\n\nA short body.\n")
|
||||
self.assertEqual(len(chunks), 1)
|
||||
self.assertEqual(chunks[0].index, 0)
|
||||
self.assertFalse(chunks[0].oversize)
|
||||
|
||||
def test_long_body_respects_maximum_and_overlaps(self):
|
||||
chunks = chunker.chunk_document(support.PROSE)
|
||||
self.assertGreater(len(chunks), 1)
|
||||
for chunk in chunks:
|
||||
self.assertLessEqual(chunk.tokens, chunker.TOKEN_MAX, f"chunk {chunk.index} too large")
|
||||
# Overlap: consecutive chunks share at least one block.
|
||||
for previous, current in zip(chunks, chunks[1:]):
|
||||
self.assertTrue(
|
||||
set(id(block) for block in previous.blocks) & set(id(block) for block in current.blocks),
|
||||
f"chunks {previous.index} and {current.index} do not overlap",
|
||||
)
|
||||
|
||||
def test_chunk_indices_are_sequential(self):
|
||||
chunks = chunker.chunk_document(support.PROSE)
|
||||
self.assertEqual([chunk.index for chunk in chunks], list(range(len(chunks))))
|
||||
|
||||
def test_deterministic(self):
|
||||
first = chunker.chunk_document(support.PROSE)
|
||||
second = chunker.chunk_document(support.PROSE)
|
||||
self.assertEqual([chunk.text for chunk in first], [chunk.text for chunk in second])
|
||||
|
||||
def test_code_fence_is_never_split(self):
|
||||
body = support.PROSE + "\n\n" + support.FENCE + "\n\n" + support.PROSE
|
||||
chunks = chunker.chunk_document(body)
|
||||
fences = [block for block in chunker.parse_blocks(body) if block.kind == "fence"]
|
||||
self.assertTrue(fences)
|
||||
for fence in fences:
|
||||
holders = [chunk for chunk in chunks if fence in chunk.blocks]
|
||||
self.assertTrue(holders, "the fenced block disappeared from every chunk")
|
||||
for holder in holders:
|
||||
self.assertIn(fence.text, holder.text, "a chunk holds part of a fenced block")
|
||||
|
||||
def test_oversize_single_block_is_flagged_not_split(self):
|
||||
huge = "```go\n" + "\n".join(f"line{i} := {i}" for i in range(1200)) + "\n```\n"
|
||||
chunks = chunker.chunk_document(huge)
|
||||
self.assertEqual(len(chunks), 1)
|
||||
self.assertTrue(chunks[0].oversize)
|
||||
self.assertGreater(chunks[0].tokens, chunker.TOKEN_MAX)
|
||||
|
||||
|
||||
class RealContentTest(unittest.TestCase):
|
||||
"""The same rules, over every document in content/."""
|
||||
|
||||
def test_no_boundary_splits_a_protected_region(self):
|
||||
for doc in content_documents():
|
||||
regions = chunker.protected_regions(doc.body)
|
||||
chunks = chunker.chunk_document(doc.body)
|
||||
chunker.verify_no_split(chunks, regions) # raises on failure
|
||||
|
||||
def test_every_chunk_is_whole_blocks(self):
|
||||
for doc in content_documents():
|
||||
expected = [block.text for block in chunker.parse_blocks(doc.body)]
|
||||
seen = []
|
||||
for chunk in chunker.chunk_document(doc.body):
|
||||
for block in chunk.blocks:
|
||||
self.assertIn(block.text, chunk.text)
|
||||
if not seen or seen[-1] != block.text:
|
||||
seen.append(block.text)
|
||||
# Every block appears, in order, exactly as parsed (overlap repeats a
|
||||
# block at the start of the next chunk, which is expected).
|
||||
position = 0
|
||||
for text in expected:
|
||||
while position < len(seen) and seen[position] != text:
|
||||
position += 1
|
||||
self.assertLess(position, len(seen), f"{doc.path}: block was dropped: {text[:40]!r}")
|
||||
position += 1
|
||||
|
||||
def test_chunks_are_within_the_size_band(self):
|
||||
for doc in content_documents():
|
||||
for chunk in chunker.chunk_document(doc.body):
|
||||
if chunk.oversize:
|
||||
self.assertEqual(len(chunk.blocks), 1)
|
||||
continue
|
||||
self.assertLessEqual(chunk.tokens, chunker.TOKEN_MAX, f"{doc.path} chunk {chunk.index}")
|
||||
|
||||
def test_content_tree_is_chunked_completely(self):
|
||||
total = 0
|
||||
for doc in content_documents():
|
||||
chunks = chunker.chunk_document(doc.body)
|
||||
self.assertTrue(chunks, f"{doc.path}: no chunks")
|
||||
total += len(chunks)
|
||||
self.assertGreater(total, 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
125
corpus/tests/test_embeddings.py
Normal file
@@ -0,0 +1,125 @@
|
||||
"""Providers: determinism offline, and an OpenAI-compatible wire shape through OmniRoute.
|
||||
|
||||
There is no direct vendor SDK call anywhere (AGENTS.md §4.5), so the provider
|
||||
test asserts the HTTP request shape, not a client library's behaviour.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import unittest
|
||||
|
||||
from corpus import embeddings
|
||||
from corpus.tests import support
|
||||
|
||||
|
||||
class LocalHashingTest(unittest.TestCase):
|
||||
def test_deterministic_and_unit_length(self):
|
||||
provider = embeddings.LocalHashingProvider(dim=64)
|
||||
first = provider.embed(["zigbee mesh latency"])[0]
|
||||
second = embeddings.LocalHashingProvider(dim=64).embed(["zigbee mesh latency"])[0]
|
||||
self.assertEqual(first, second)
|
||||
self.assertEqual(len(first), 64)
|
||||
self.assertAlmostEqual(math.sqrt(sum(value * value for value in first)), 1.0, places=9)
|
||||
|
||||
def test_different_text_differs(self):
|
||||
provider = embeddings.LocalHashingProvider(dim=64)
|
||||
a, b = provider.embed(["zigbee", "lora"])
|
||||
self.assertNotEqual(a, b)
|
||||
|
||||
def test_empty_text_still_returns_a_unit_vector(self):
|
||||
provider = embeddings.LocalHashingProvider(dim=16)
|
||||
vector = provider.embed([""])[0]
|
||||
self.assertEqual(len(vector), 16)
|
||||
self.assertAlmostEqual(math.sqrt(sum(value * value for value in vector)), 1.0, places=9)
|
||||
|
||||
def test_stats_count_calls_and_texts(self):
|
||||
provider = embeddings.LocalHashingProvider(dim=16)
|
||||
provider.embed(["a", "b", "c"])
|
||||
provider.embed(["d"])
|
||||
self.assertEqual(provider.stats(), {"calls": 2, "texts": 4})
|
||||
provider.reset_stats()
|
||||
self.assertEqual(provider.stats(), {"calls": 0, "texts": 0})
|
||||
|
||||
def test_rejects_a_zero_dimension(self):
|
||||
with self.assertRaises(embeddings.EmbeddingError):
|
||||
embeddings.LocalHashingProvider(dim=0)
|
||||
|
||||
|
||||
class OmniRouteTest(unittest.TestCase):
|
||||
def responder_factory(self, dim=8, status_plan=None):
|
||||
plan = list(status_plan or [])
|
||||
state = {"n": 0}
|
||||
|
||||
def responder(method, path, body):
|
||||
state["n"] += 1
|
||||
if plan:
|
||||
status = plan.pop(0)
|
||||
if status != 200:
|
||||
return status, {"error": "planned failure"}
|
||||
vectors = [
|
||||
{"index": index, "embedding": [0.1] * dim}
|
||||
for index, _text in enumerate(body.get("input") or [])
|
||||
]
|
||||
return 200, {"model": body.get("model"), "data": vectors}
|
||||
|
||||
return responder
|
||||
|
||||
def test_posts_to_v1_embeddings_with_the_configured_model(self):
|
||||
with support.StubServer(self.responder_factory()) as server:
|
||||
provider = embeddings.OmniRouteProvider(
|
||||
api_url=server.url, api_key="test-key", model="text-embedding-3-small", dim=8
|
||||
)
|
||||
vectors = provider.embed(["one", "two"])
|
||||
self.assertEqual(len(vectors), 2)
|
||||
request = server.requests[0]
|
||||
self.assertEqual(request["method"], "POST")
|
||||
self.assertEqual(request["path"], "/v1/embeddings")
|
||||
self.assertEqual(request["body"]["model"], "text-embedding-3-small")
|
||||
self.assertEqual(request["body"]["input"], ["one", "two"])
|
||||
self.assertEqual(request["headers"]["Authorization"], "Bearer test-key")
|
||||
self.assertEqual(provider.stats(), {"calls": 1, "texts": 2})
|
||||
|
||||
def test_batches_requests(self):
|
||||
with support.StubServer(self.responder_factory()) as server:
|
||||
provider = embeddings.OmniRouteProvider(
|
||||
api_url=server.url, api_key="k", model="m", dim=8, batch_size=2
|
||||
)
|
||||
provider.embed(["a", "b", "c", "d", "e"])
|
||||
self.assertEqual(len(server.requests), 3)
|
||||
self.assertEqual(provider.stats(), {"calls": 3, "texts": 5})
|
||||
|
||||
def test_dimension_mismatch_is_refused(self):
|
||||
with support.StubServer(self.responder_factory(dim=4)) as server:
|
||||
provider = embeddings.OmniRouteProvider(
|
||||
api_url=server.url, api_key="k", model="m", dim=8
|
||||
)
|
||||
with self.assertRaises(embeddings.EmbeddingError) as caught:
|
||||
provider.embed(["one"])
|
||||
self.assertIn("dimension", str(caught.exception))
|
||||
|
||||
def test_retries_then_succeeds(self):
|
||||
with support.StubServer(self.responder_factory(status_plan=[429, 200])) as server:
|
||||
provider = embeddings.OmniRouteProvider(
|
||||
api_url=server.url, api_key="k", model="m", dim=8, backoff=0.01
|
||||
)
|
||||
vectors = provider.embed(["one"])
|
||||
self.assertEqual(len(vectors), 1)
|
||||
self.assertEqual(len(server.requests), 2)
|
||||
|
||||
def test_client_error_is_not_retried(self):
|
||||
with support.StubServer(self.responder_factory(status_plan=[400])) as server:
|
||||
provider = embeddings.OmniRouteProvider(
|
||||
api_url=server.url, api_key="k", model="m", dim=8, backoff=0.01
|
||||
)
|
||||
with self.assertRaises(embeddings.EmbeddingError):
|
||||
provider.embed(["one"])
|
||||
self.assertEqual(len(server.requests), 1)
|
||||
|
||||
def test_missing_key_is_refused_before_any_call(self):
|
||||
with self.assertRaises(embeddings.EmbeddingError):
|
||||
embeddings.OmniRouteProvider(api_url="http://127.0.0.1:1", api_key="", model="m", dim=8)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
110
corpus/tests/test_freshness.py
Normal file
@@ -0,0 +1,110 @@
|
||||
"""Freshness audit: stale documents, index drift and contract drift."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import unittest
|
||||
|
||||
from corpus import freshness, pipeline, store as store_module
|
||||
from corpus.tests import support
|
||||
|
||||
|
||||
class AuditTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = support.temp_repo()
|
||||
self.cfg = support.make_config(self.root)
|
||||
self.addCleanup(_cleanup, self.root)
|
||||
with support.fake_qdrant(self.cfg):
|
||||
pipeline.run(self.cfg, mode="rebuild")
|
||||
|
||||
def audit(self, today=None, cfg=None):
|
||||
with support.fake_qdrant(cfg or self.cfg):
|
||||
return freshness.audit(cfg or self.cfg, today=today)
|
||||
|
||||
def test_clean_soon_after_rebuild(self):
|
||||
report = self.audit(today=datetime.date(2026, 10, 5))
|
||||
self.assertTrue(report["ok"], report["findings"])
|
||||
self.assertEqual(report["corpus"]["documents"], 3)
|
||||
|
||||
def test_reports_stale_documents(self):
|
||||
report = self.audit(today=datetime.date(2027, 10, 5))
|
||||
stale = {row["source_path"] for row in report["findings"]["stale"]}
|
||||
self.assertEqual(
|
||||
stale,
|
||||
{
|
||||
"content/ai/ai-overview.md",
|
||||
"content/edge/l3/edge-zigbee.md",
|
||||
"content/platform/l3/platform-containers.md",
|
||||
},
|
||||
)
|
||||
self.assertFalse(report["ok"])
|
||||
text = freshness.render_text(report)
|
||||
self.assertIn("STALE", text)
|
||||
self.assertIn("platform-containers", text)
|
||||
|
||||
def test_warns_before_a_document_goes_stale(self):
|
||||
# platform-containers was verified 2026-08-01. Horizon 180 days, warning
|
||||
# 30, so on 2027-01-20 it is 172 days old: due soon, not yet stale.
|
||||
report = self.audit(today=datetime.date(2027, 1, 20))
|
||||
due = {row["source_path"] for row in report["findings"]["due_soon"]}
|
||||
self.assertEqual(due, {"content/platform/l3/platform-containers.md"})
|
||||
self.assertEqual(report["findings"]["stale"], [])
|
||||
self.assertTrue(report["ok"])
|
||||
|
||||
def test_reports_index_drift_when_a_source_changes(self):
|
||||
support.write_doc(
|
||||
self.root,
|
||||
"ai/ai-overview.md",
|
||||
id="ai-overview", area="ai", level=1, title="Agent Systems",
|
||||
claim="Edited but not re-indexed.", confidence="running",
|
||||
last_verified="2026-10-05", tags="agents", body="New words for the corpus.",
|
||||
)
|
||||
report = self.audit(today=datetime.date(2026, 10, 6))
|
||||
problems = [row["problem"] for row in report["findings"]["index_drift"]]
|
||||
self.assertIn("source changed since it was embedded", problems)
|
||||
self.assertFalse(report["ok"])
|
||||
|
||||
def test_reports_contract_drift_when_the_model_changes(self):
|
||||
other = support.make_config(self.root, model="different-model")
|
||||
report = self.audit(today=datetime.date(2026, 10, 5), cfg=other)
|
||||
self.assertTrue(report["findings"]["contract_drift"])
|
||||
|
||||
def test_report_is_json_serialisable(self):
|
||||
import json
|
||||
|
||||
report = self.audit(today=datetime.date(2026, 10, 5))
|
||||
json.dumps(report)
|
||||
|
||||
|
||||
class AlertTest(unittest.TestCase):
|
||||
def test_no_url_means_no_alert(self):
|
||||
report = {"ok": True, "findings": {"stale": []}, "corpus": {}, "horizon": {}, "summary": {}}
|
||||
self.assertFalse(freshness.alert(report, ""))
|
||||
|
||||
def test_posts_to_apprise(self):
|
||||
report = {
|
||||
"generated_at": "2026-10-26T00:00:00Z",
|
||||
"today": "2026-10-26",
|
||||
"horizon": {"max_age_days": 180, "warn_days": 30},
|
||||
"corpus": {
|
||||
"documents": 1, "chunks": 1, "sources": 1, "collection": "c",
|
||||
"collection_points": 1, "provider": "p", "model": "m", "dim": 8,
|
||||
"chunker_version": "v",
|
||||
},
|
||||
"findings": {"stale": [], "due_soon": [], "index_drift": [], "contract_drift": []},
|
||||
"summary": {"stale": 0},
|
||||
"ok": True,
|
||||
}
|
||||
with support.StubServer(lambda method, path, body: (200, "ok")) as server:
|
||||
self.assertTrue(freshness.alert(report, server.url))
|
||||
self.assertIn("title", server.requests[0]["body"])
|
||||
|
||||
|
||||
def _cleanup(root):
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(root, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
100
corpus/tests/test_payloads.py
Normal file
@@ -0,0 +1,100 @@
|
||||
"""Payload rules: every field present, enums enforced, missing field fails the run."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
from corpus import chunker, loader, payloads
|
||||
from corpus.config import Config
|
||||
from corpus.pipeline import chunk_id
|
||||
from corpus.tests import support
|
||||
|
||||
|
||||
def first_payload():
|
||||
cfg = Config.from_env(root=support.REPO_ROOT, provider="local-hashing", dim=32)
|
||||
doc = loader.load_documents(cfg)[0]
|
||||
chunk = chunker.chunk_document(doc.body)[0]
|
||||
payload = payloads.build_payload(doc, chunk, cfg)
|
||||
payload["id"] = chunk_id(doc.path, chunk.index, doc.digest)
|
||||
return cfg, doc, payload
|
||||
|
||||
|
||||
class BuildTest(unittest.TestCase):
|
||||
def test_payload_has_every_required_field(self):
|
||||
cfg, doc, payload = first_payload()
|
||||
payloads.validate_payload(payload)
|
||||
for field in payloads.REQUIRED_FIELDS:
|
||||
self.assertIn(field, payload)
|
||||
self.assertEqual(payload["area"], doc.area)
|
||||
self.assertEqual(payload["confidence"], doc.confidence)
|
||||
self.assertEqual(payload["last_verified"], doc.last_verified)
|
||||
self.assertEqual(payload["embedding_model"], cfg.model)
|
||||
self.assertEqual(payload["embedding_dim"], cfg.dim)
|
||||
|
||||
def test_every_real_document_produces_valid_payloads(self):
|
||||
cfg = Config.from_env(root=support.REPO_ROOT, provider="local-hashing", dim=32)
|
||||
for doc in loader.load_documents(cfg):
|
||||
for chunk in chunker.chunk_document(doc.body):
|
||||
payload = payloads.build_payload(doc, chunk, cfg)
|
||||
payload["id"] = chunk_id(doc.path, chunk.index, doc.digest)
|
||||
payloads.validate_payload(payload, where=doc.path)
|
||||
|
||||
def test_source_type_is_doc_by_default(self):
|
||||
_cfg, doc, payload = first_payload()
|
||||
self.assertEqual(payload["source_type"], "doc")
|
||||
|
||||
def test_source_type_from_tags(self):
|
||||
doc = loader.Document(
|
||||
path="content/ai/adr-x.md", source_id="adr-x", area="ai", level=3, title="ADR",
|
||||
claim="c", confidence="delivered", period="2024", tech=[], metrics=[], artefacts=[],
|
||||
public=True, last_verified="2026-01-01", tags=["adr"], body="", digest="0" * 64,
|
||||
url="/ai/a/adr-x/",
|
||||
)
|
||||
self.assertEqual(payloads.source_type_for(doc), "adr")
|
||||
|
||||
|
||||
class ValidationTest(unittest.TestCase):
|
||||
def test_missing_field_raises(self):
|
||||
_cfg, _doc, payload = first_payload()
|
||||
for field in ("area", "level", "source_type", "confidence", "period", "artefacts", "last_verified", "text"):
|
||||
broken = dict(payload)
|
||||
del broken[field]
|
||||
with self.assertRaises(payloads.PayloadError, msg=f"missing {field} passed validation"):
|
||||
payloads.validate_payload(broken)
|
||||
|
||||
def test_missing_provenance_field_raises(self):
|
||||
_cfg, _doc, payload = first_payload()
|
||||
for field in payloads.PROVENANCE_FIELDS:
|
||||
broken = dict(payload)
|
||||
del broken[field]
|
||||
with self.assertRaises(payloads.PayloadError, msg=f"missing {field} passed validation"):
|
||||
payloads.validate_payload(broken)
|
||||
|
||||
def test_bad_enums_raise(self):
|
||||
_cfg, _doc, payload = first_payload()
|
||||
for field, value in (
|
||||
("area", "sales"),
|
||||
("level", 4),
|
||||
("source_type", "blog"),
|
||||
("confidence", "vibes"),
|
||||
):
|
||||
broken = dict(payload)
|
||||
broken[field] = value
|
||||
with self.assertRaises(payloads.PayloadError, msg=f"{field}={value!r} passed validation"):
|
||||
payloads.validate_payload(broken)
|
||||
|
||||
def test_empty_text_raises(self):
|
||||
_cfg, _doc, payload = first_payload()
|
||||
payload["text"] = " "
|
||||
with self.assertRaises(payloads.PayloadError):
|
||||
payloads.validate_payload(payload)
|
||||
|
||||
def test_bad_date_raises(self):
|
||||
_cfg, _doc, payload = first_payload()
|
||||
payload["last_verified"] = "last october"
|
||||
with self.assertRaises(payloads.PayloadError):
|
||||
payloads.validate_payload(payload)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
179
corpus/tests/test_pipeline.py
Normal file
@@ -0,0 +1,179 @@
|
||||
"""The pipeline: deterministic ids, no duplicate vectors on re-run, incremental mode,
|
||||
incomplete payloads never reach the store (docs/phase-4-corpus.md DoD 1-6)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
from corpus import chunker, loader, pipeline, store as store_module
|
||||
from corpus.tests import support
|
||||
|
||||
|
||||
class RebuildTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = support.temp_repo()
|
||||
self.cfg = support.make_config(self.root)
|
||||
self.addCleanup(_cleanup, self.root)
|
||||
# One fake for the whole test: a second run must see the first run's points.
|
||||
self.fake = support.FakeQdrant(self.cfg.dim)
|
||||
|
||||
def run_pipeline(self, mode="sync", cfg=None):
|
||||
cfg = cfg or self.cfg
|
||||
with support.fake_qdrant(cfg, self.fake):
|
||||
return pipeline.run(cfg, mode=mode)
|
||||
|
||||
def test_rebuild_from_empty_volume(self):
|
||||
stats = self.run_pipeline("rebuild")
|
||||
self.assertEqual(stats.documents, 3)
|
||||
self.assertEqual(stats.added, 3)
|
||||
self.assertGreater(stats.store_chunks, 0)
|
||||
self.assertEqual(stats.store_chunks, stats.collection_points)
|
||||
self.assertEqual(stats.collection_points, len(self.fake.collections[self.cfg.collection]))
|
||||
self.assertGreater(stats.embed_calls, 0)
|
||||
|
||||
def test_running_twice_does_not_duplicate_vectors(self):
|
||||
first = self.run_pipeline("rebuild")
|
||||
points_after_first = len(self.fake.collections[self.cfg.collection])
|
||||
second = self.run_pipeline("sync")
|
||||
points_after_second = len(self.fake.collections[self.cfg.collection])
|
||||
self.assertEqual(points_after_first, points_after_second)
|
||||
self.assertEqual(first.store_chunks, second.store_chunks)
|
||||
self.assertEqual(second.added, 0)
|
||||
self.assertEqual(second.updated, 0)
|
||||
self.assertEqual(second.chunks_written, 0)
|
||||
|
||||
def test_incremental_reembeds_only_changed_documents(self):
|
||||
self.run_pipeline("rebuild")
|
||||
|
||||
# Nothing changed: no embedding calls at all.
|
||||
unchanged = self.run_pipeline("sync")
|
||||
self.assertEqual(unchanged.embed_calls, 0)
|
||||
self.assertEqual(unchanged.texts_embedded, 0)
|
||||
|
||||
# One document changes: exactly one embedding call, for that document.
|
||||
support.write_doc(
|
||||
self.root,
|
||||
"edge/l3/edge-zigbee.md",
|
||||
id="edge-zigbee", area="edge", level=3, title="Low Power Mesh Sensing",
|
||||
claim="A constrained mesh that survives power loss.", confidence="delivered",
|
||||
last_verified="2026-09-15", tags="zigbee, mesh",
|
||||
body="The mesh now also reports battery voltage on every node.",
|
||||
)
|
||||
changed = self.run_pipeline("sync")
|
||||
self.assertEqual(changed.added + changed.updated, 1)
|
||||
self.assertEqual(changed.embed_calls, 1)
|
||||
expected_chunks = len(
|
||||
chunker.chunk_document(
|
||||
loader.load_documents(support.make_config(self.root))[1].body
|
||||
)
|
||||
)
|
||||
self.assertEqual(changed.texts_embedded, expected_chunks)
|
||||
self.assertEqual(changed.unchanged, 2)
|
||||
|
||||
def test_removed_document_is_removed_from_both_indexes(self):
|
||||
self.run_pipeline("rebuild")
|
||||
before = self.run_pipeline("sync").store_chunks
|
||||
os.remove(os.path.join(self.root, "content", "platform", "l3", "platform-containers.md"))
|
||||
after = self.run_pipeline("sync")
|
||||
self.assertEqual(after.removed, 1)
|
||||
self.assertLess(after.store_chunks, before)
|
||||
self.assertEqual(after.store_chunks, after.collection_points)
|
||||
with store_module.Store(self.cfg.db_path) as store:
|
||||
self.assertEqual(store.keyword_search("endpoint latency loss"), [])
|
||||
|
||||
def test_ids_are_deterministic(self):
|
||||
first = pipeline.chunk_id("content/ai/ai-overview.md", 0, "a" * 64)
|
||||
second = pipeline.chunk_id("content/ai/ai-overview.md", 0, "a" * 64)
|
||||
other = pipeline.chunk_id("content/ai/ai-overview.md", 1, "a" * 64)
|
||||
self.assertEqual(first, second)
|
||||
self.assertNotEqual(first, other)
|
||||
self.assertEqual(len(first), 36)
|
||||
|
||||
def test_invalid_payload_aborts_before_any_write(self):
|
||||
real_build = pipeline.payloads.build_payload
|
||||
|
||||
def broken(doc, chunk, cfg):
|
||||
payload = real_build(doc, chunk, cfg)
|
||||
payload.pop("confidence")
|
||||
return payload
|
||||
|
||||
with mock.patch("corpus.payloads.build_payload", side_effect=broken):
|
||||
with self.assertRaises(pipeline.PipelineError):
|
||||
self.run_pipeline("rebuild")
|
||||
self.assertEqual(self.fake.collections.get(self.cfg.collection, {}), {})
|
||||
|
||||
def test_provider_change_is_refused_without_rebuild(self):
|
||||
self.run_pipeline("rebuild")
|
||||
other = support.make_config(self.root, model="local-hashing-v2")
|
||||
with self.assertRaises(pipeline.PipelineError) as caught:
|
||||
self.run_pipeline("sync", cfg=other)
|
||||
self.assertIn("governed migration", str(caught.exception))
|
||||
|
||||
def test_contract_is_recorded(self):
|
||||
self.run_pipeline("rebuild")
|
||||
with store_module.Store(self.cfg.db_path) as store:
|
||||
contract = store.get_meta("collection")
|
||||
self.assertEqual(contract["model"], self.cfg.model)
|
||||
self.assertEqual(contract["dim"], self.cfg.dim)
|
||||
self.assertEqual(contract["chunker_version"], self.cfg.chunker_version)
|
||||
self.assertTrue(os.path.exists(self.cfg.meta_path))
|
||||
|
||||
|
||||
class VerifyTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = support.temp_repo()
|
||||
self.cfg = support.make_config(self.root)
|
||||
self.addCleanup(_cleanup, self.root)
|
||||
|
||||
def test_clean_after_rebuild(self):
|
||||
with support.fake_qdrant(self.cfg):
|
||||
pipeline.run(self.cfg, mode="rebuild")
|
||||
self.assertEqual(pipeline.verify(self.cfg), [])
|
||||
|
||||
def test_detects_source_changed_without_sync(self):
|
||||
with support.fake_qdrant(self.cfg):
|
||||
pipeline.run(self.cfg, mode="rebuild")
|
||||
support.write_doc(
|
||||
self.root,
|
||||
"ai/ai-overview.md",
|
||||
id="ai-overview", area="ai", level=1, title="Agent Systems",
|
||||
claim="Changed.", confidence="running", last_verified="2026-10-02",
|
||||
tags="agents", body="A different body entirely.",
|
||||
)
|
||||
problems = pipeline.verify(self.cfg)
|
||||
self.assertTrue(any("changed since it was embedded" in problem for problem in problems), problems)
|
||||
|
||||
def test_detects_missing_document(self):
|
||||
with support.fake_qdrant(self.cfg):
|
||||
pipeline.run(self.cfg, mode="rebuild")
|
||||
os.remove(os.path.join(self.root, "content", "ai", "ai-overview.md"))
|
||||
problems = pipeline.verify(self.cfg)
|
||||
self.assertTrue(any("no longer in content/" in problem for problem in problems), problems)
|
||||
|
||||
|
||||
class AreaFilterTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = support.temp_repo()
|
||||
self.cfg = support.make_config(self.root)
|
||||
self.addCleanup(_cleanup, self.root)
|
||||
|
||||
def test_filter_narrows_the_result_set(self):
|
||||
with support.fake_qdrant(self.cfg):
|
||||
pipeline.run(self.cfg, mode="rebuild")
|
||||
result = pipeline.demo_area_filter(self.cfg, "container estate limits", area="edge")
|
||||
self.assertTrue(result["unfiltered"])
|
||||
self.assertTrue(result["filtered"])
|
||||
self.assertEqual({row["area"] for row in result["filtered"]}, {"edge"})
|
||||
self.assertGreater(len({row["area"] for row in result["unfiltered"]}), 1)
|
||||
|
||||
|
||||
def _cleanup(root):
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(root, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
104
corpus/tests/test_qdrant_client.py
Normal file
@@ -0,0 +1,104 @@
|
||||
"""Qdrant client: the wire shape the pipeline depends on, and the fallback route."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
from corpus import qdrant as qdrant_module
|
||||
from corpus.tests import support
|
||||
|
||||
|
||||
class ClientTest(unittest.TestCase):
|
||||
def test_create_collection_sends_size_and_distance(self):
|
||||
with support.StubServer(lambda method, path, body: (200, {"result": True})) as server:
|
||||
client = qdrant_module.Qdrant(server.url, attempts=1)
|
||||
client.create_collection("resume_corpus", 1536)
|
||||
request = server.requests[0]
|
||||
self.assertEqual(request["method"], "PUT")
|
||||
self.assertEqual(request["path"], "/collections/resume_corpus")
|
||||
body = request["body"]
|
||||
self.assertEqual(body["vectors"]["size"], 1536)
|
||||
self.assertEqual(body["vectors"]["distance"], "Cosine")
|
||||
# A memory-bound host: payloads and the HNSW graph live on disk.
|
||||
self.assertTrue(body["on_disk_payload"])
|
||||
self.assertTrue(body["hnsw_config"]["on_disk"])
|
||||
self.assertFalse(body["vectors"]["on_disk"])
|
||||
self.assertGreater(body["optimizers_config"]["memmap_threshold"], 0)
|
||||
|
||||
def test_upsert_waits_and_batches(self):
|
||||
with support.StubServer(lambda method, path, body: (200, {"result": {}})) as server:
|
||||
client = qdrant_module.Qdrant(server.url, attempts=1)
|
||||
points = [{"id": f"id-{index}", "vector": [0.0], "payload": {}} for index in range(5)]
|
||||
sent = client.upsert("c", points, batch=2)
|
||||
self.assertEqual(sent, 5)
|
||||
self.assertEqual(len(server.requests), 3)
|
||||
self.assertTrue(all("wait=true" in request["path"] for request in server.requests))
|
||||
self.assertEqual(sum(len(r["body"]["points"]) for r in server.requests), 5)
|
||||
|
||||
def test_delete_by_source_sends_a_source_filter(self):
|
||||
with support.StubServer(lambda method, path, body: (200, {"result": {}})) as server:
|
||||
client = qdrant_module.Qdrant(server.url, attempts=1)
|
||||
client.delete_by_source("c", "content/edge/l3/edge-zigbee.md")
|
||||
request = server.requests[0]
|
||||
self.assertEqual(request["method"], "POST")
|
||||
self.assertIn("/collections/c/points/delete", request["path"])
|
||||
self.assertEqual(
|
||||
request["body"],
|
||||
{"filter": {"must": [{"key": "source_path", "match": {"value": "content/edge/l3/edge-zigbee.md"}}]}},
|
||||
)
|
||||
|
||||
def test_search_filters_before_searching(self):
|
||||
responder = lambda method, path, body: (200, {"result": {"points": []}}) # noqa: E731
|
||||
with support.StubServer(responder) as server:
|
||||
client = qdrant_module.Qdrant(server.url, attempts=1)
|
||||
client.search("c", [0.1, 0.2], limit=3, area="edge", confidence="delivered")
|
||||
body = server.requests[0]["body"]
|
||||
self.assertEqual(server.requests[0]["path"], "/collections/c/points/query")
|
||||
self.assertEqual(body["limit"], 3)
|
||||
self.assertEqual(
|
||||
body["filter"],
|
||||
{
|
||||
"must": [
|
||||
{"key": "area", "match": {"value": "edge"}},
|
||||
{"key": "confidence", "match": {"value": "delivered"}},
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
def test_search_falls_back_to_the_legacy_route(self):
|
||||
def responder(method, path, body):
|
||||
if path.endswith("/points/query"):
|
||||
return 404, {"status": "not found"}
|
||||
return 200, {"result": [{"id": "p1", "score": 0.9, "payload": {"area": "ai"}}]}
|
||||
|
||||
with support.StubServer(responder) as server:
|
||||
client = qdrant_module.Qdrant(server.url, attempts=1)
|
||||
hits = client.search("c", [0.1])
|
||||
self.assertEqual([hit["id"] for hit in hits], ["p1"])
|
||||
self.assertEqual([r["path"] for r in server.requests][-1], "/collections/c/points/search")
|
||||
|
||||
def test_point_count_is_exact(self):
|
||||
responder = lambda method, path, body: (200, {"result": {"count": 42}}) # noqa: E731
|
||||
with support.StubServer(responder) as server:
|
||||
client = qdrant_module.Qdrant(server.url, attempts=1)
|
||||
self.assertEqual(client.point_count("c"), 42)
|
||||
self.assertEqual(server.requests[0]["body"], {"exact": True})
|
||||
|
||||
def test_health_is_false_when_unreachable(self):
|
||||
client = qdrant_module.Qdrant("http://127.0.0.1:1", timeout=0.2, attempts=1)
|
||||
self.assertFalse(client.health())
|
||||
|
||||
def test_upsert_retries_a_server_error(self):
|
||||
plan = [500, 200]
|
||||
|
||||
def responder(method, path, body):
|
||||
return plan.pop(0), {"result": {}}
|
||||
|
||||
with support.StubServer(responder) as server:
|
||||
client = qdrant_module.Qdrant(server.url, attempts=2)
|
||||
client.upsert("c", [{"id": "x", "vector": [0.0], "payload": {}}])
|
||||
self.assertEqual(len(server.requests), 2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
106
corpus/tests/test_store.py
Normal file
@@ -0,0 +1,106 @@
|
||||
"""FTS5 chunk index, manifest and meta: the keyword half of hybrid retrieval."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
from corpus import chunker, loader, payloads, store as store_module
|
||||
from corpus.config import Config
|
||||
from corpus.pipeline import chunk_id
|
||||
from corpus.tests import support
|
||||
|
||||
|
||||
def build_store(path, cfg, documents):
|
||||
store = store_module.Store(path)
|
||||
for doc in documents:
|
||||
payload_list = []
|
||||
for chunk in chunker.chunk_document(doc.body):
|
||||
payload = payloads.build_payload(doc, chunk, cfg)
|
||||
payload["id"] = chunk_id(doc.path, chunk.index, doc.digest)
|
||||
payload_list.append(payload)
|
||||
store.write_document(doc, payload_list)
|
||||
return store
|
||||
|
||||
|
||||
class StoreTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = support.temp_repo()
|
||||
self.cfg = support.make_config(self.root)
|
||||
self.addCleanup(_cleanup, self.root)
|
||||
self.documents = loader.load_documents(self.cfg)
|
||||
self.store = build_store(self.cfg.db_path, self.cfg, self.documents)
|
||||
self.addCleanup(self.store.close)
|
||||
|
||||
def test_counts_match_the_documents(self):
|
||||
self.assertEqual(self.store.source_count(), len(self.documents))
|
||||
expected = sum(len(chunker.chunk_document(doc.body)) for doc in self.documents)
|
||||
self.assertEqual(self.store.chunk_count(), expected)
|
||||
|
||||
def test_keyword_search_finds_exact_tokens(self):
|
||||
hits = self.store.keyword_search("zigbee")
|
||||
self.assertTrue(hits, "no hit for a token that is literally in the document")
|
||||
self.assertEqual(hits[0]["area"], "edge")
|
||||
|
||||
def test_keyword_search_respects_area_filter(self):
|
||||
all_hits = self.store.keyword_search("latency")
|
||||
edge_hits = self.store.keyword_search("latency", area="edge")
|
||||
self.assertTrue(all_hits)
|
||||
for hit in edge_hits:
|
||||
self.assertEqual(hit["area"], "edge")
|
||||
|
||||
def test_keyword_search_returns_snippets_and_scores(self):
|
||||
hits = self.store.keyword_search("fence")
|
||||
self.assertTrue(hits)
|
||||
self.assertIn("score", hits[0])
|
||||
self.assertIn("snippet", hits[0])
|
||||
|
||||
def test_operator_injection_is_not_honoured(self):
|
||||
# FTS5 operators must be treated as words, never as query syntax.
|
||||
hits = self.store.keyword_search('edge OR "NEAR"(a b)')
|
||||
self.assertIsInstance(hits, list)
|
||||
|
||||
def test_manifest_records_the_content_hash(self):
|
||||
manifest = self.store.manifest()
|
||||
self.assertEqual(set(manifest), {doc.path for doc in self.documents})
|
||||
for doc in self.documents:
|
||||
self.assertEqual(manifest[doc.path]["content_hash"], doc.digest)
|
||||
self.assertGreater(manifest[doc.path]["chunk_count"], 0)
|
||||
|
||||
def test_delete_removes_rows_from_both_tables(self):
|
||||
target = self.documents[1]
|
||||
self.store.delete_document(target.path)
|
||||
self.assertNotIn(target.path, self.store.manifest())
|
||||
for row in self.store.chunk_payloads():
|
||||
self.assertNotEqual(row["source_path"], target.path)
|
||||
|
||||
def test_meta_roundtrip(self):
|
||||
self.store.set_meta("collection", {"model": "x", "dim": 8})
|
||||
self.assertEqual(self.store.get_meta("collection"), {"model": "x", "dim": 8})
|
||||
self.assertIn("collection", self.store.meta())
|
||||
|
||||
def test_payloads_survive_the_roundtrip(self):
|
||||
rows = self.store.chunk_payloads()
|
||||
self.assertTrue(rows)
|
||||
for row in rows:
|
||||
payloads.validate_payload(row)
|
||||
|
||||
|
||||
class MatchExpressionTest(unittest.TestCase):
|
||||
def test_terms_become_prefix_queries(self):
|
||||
self.assertEqual(store_module.match_expression("Zigbee mesh"), '"zigbee"* AND "mesh"*')
|
||||
|
||||
def test_empty_query_is_empty(self):
|
||||
self.assertEqual(store_module.match_expression(" "), "")
|
||||
|
||||
def test_punctuation_is_dropped(self):
|
||||
self.assertEqual(store_module.match_expression("-edge- LoRaWAN, 2023"), '"edge"* AND "lorawan"* AND "2023"*')
|
||||
|
||||
|
||||
def _cleanup(root):
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(root, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
29
deploy/Dockerfile
Normal file
@@ -0,0 +1,29 @@
|
||||
# Runtime image for the Mode A site (docs/phase-2-static-site.md §2.1).
|
||||
#
|
||||
# The image contains no toolchain: `make build dist` on the build host produces
|
||||
# bin/resume-server and dist/, and the image just carries them. That keeps the
|
||||
# runtime offline and small, and it means the deployed bytes are the bytes that
|
||||
# were built and tested, not a second compile.
|
||||
#
|
||||
# Build context is the repository root:
|
||||
#
|
||||
# make build dist
|
||||
# docker build -f deploy/Dockerfile -t ai-resume:local .
|
||||
#
|
||||
# Phase 3 (§3.1) owns the release directory, the compose invocation and the
|
||||
# rollback; this file only describes the process that runs.
|
||||
|
||||
FROM gcr.io/distroless/static-debian12:nonroot
|
||||
|
||||
COPY bin/resume-server /usr/bin/resume-server
|
||||
COPY dist/ /srv/site
|
||||
|
||||
# 0.0.0.0 is inside the container's own network namespace only. The host
|
||||
# publishes the port on 127.0.0.1 (see deploy/compose.yml), which is what keeps
|
||||
# Caddy the only public listener.
|
||||
ENV RESUME_ADDR=0.0.0.0:8096 \
|
||||
RESUME_DIST=/srv/site
|
||||
|
||||
EXPOSE 8096
|
||||
USER nonroot:nonroot
|
||||
ENTRYPOINT ["/usr/bin/resume-server"]
|
||||
71
deploy/compose.yml
Normal file
@@ -0,0 +1,71 @@
|
||||
# Container shape for the Mode A site (docs/phase-2-static-site.md §2.1).
|
||||
#
|
||||
# Values are fixed by the specification and must not drift:
|
||||
# mem_limit == memswap_limit == 120m so the app cannot eat the host's swap
|
||||
# ports published on 127.0.0.1 only Caddy on .35 is the only public listener
|
||||
# logging capped one bad day cannot fill the disk
|
||||
#
|
||||
# Run it (build the artefacts first — Phase 3's deploy.sh does both):
|
||||
#
|
||||
# make build dist
|
||||
# docker compose -f deploy/compose.yml up -d
|
||||
#
|
||||
# `curl http://127.0.0.1:8096/healthz` must answer with static ready.
|
||||
|
||||
# The compose project name is deliberately not `ai-resume`: /home/sam/deployment/
|
||||
# ai-resume on .13 already runs a compose project by that name (backend + db on
|
||||
# the ai-mesh network). Sharing the name would make this site's containers show
|
||||
# up as orphans in that project and vice versa, so anything running `compose
|
||||
# down --remove-orphans` under either project could stop the other. The image
|
||||
# tag stays ai-resume:local (Phase 3 fixes it); only the project namespace is
|
||||
# distinct.
|
||||
name: ai-resume-site
|
||||
|
||||
services:
|
||||
resume:
|
||||
image: ai-resume:local
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/Dockerfile
|
||||
restart: unless-stopped
|
||||
mem_limit: 120m
|
||||
memswap_limit: 120m
|
||||
# Published on this host's LAN address, not loopback. Caddy lives on .35 and
|
||||
# proxies to 192.168.20.13:8096, exactly as it does for offsite (:8095),
|
||||
# photo-filter (:8092), garage-ui (:3909) and prefect (:4200). The port is
|
||||
# reachable inside the house only; Caddy on .35 remains the sole public
|
||||
# listener, which is the intent of AGENTS.md section 4.
|
||||
ports:
|
||||
- "192.168.20.13:8096:8096"
|
||||
read_only: true
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: 10m
|
||||
max-file: "3"
|
||||
healthcheck:
|
||||
test: ["CMD", "/usr/bin/resume-server", "--health-check"]
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 5s
|
||||
|
||||
# The project network gets an explicit, small subnet. The host's docker daemon
|
||||
# has every default address pool allocated by other projects ("all predefined
|
||||
# address pools have been fully subnetted"), so an implicit compose network
|
||||
# cannot be created. Pinning /28 of RFC1918 space that no other bridge or route
|
||||
# on .13 uses keeps this deploy independent of the host's pool exhaustion and
|
||||
# does not touch any other container. Single-service stack: two addresses is
|
||||
# plenty.
|
||||
networks:
|
||||
default:
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 10.89.0.0/28
|
||||
gateway: 10.89.0.1
|
||||
|
||||
# Phase 4/5 add at most three more containers (Qdrant, Prefect worker, query
|
||||
# service) inside the 4-container and 1.0 GiB budget in AGENTS.md §4.3. Mode A
|
||||
# needs none of them and must keep working when they are stopped (test A1).
|
||||
400
deploy/deploy.sh
Executable file
@@ -0,0 +1,400 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy.sh — release, deploy and roll back the Mode A site (docs/phase-3-deploy.md §3.1).
|
||||
#
|
||||
# ./deploy/deploy.sh build, test, stage a release, flip `current`,
|
||||
# start the container, gate on /healthz
|
||||
# ./deploy/deploy.sh --rollback put the previous release back, in one command
|
||||
# ./deploy/deploy.sh --status what is live, what is previous, is it healthy
|
||||
# ./deploy/deploy.sh --release-dir D deploy an already-built release directory
|
||||
# (copied into releases/ first; used by the
|
||||
# rollback rehearsal below)
|
||||
#
|
||||
# Layout on the runtime host — the release directory, not the image tag, is the
|
||||
# unit that runs, so a rollback cannot publish bytes that were never tested:
|
||||
#
|
||||
# /home/sam/deployment/resume/
|
||||
# releases/<utc-timestamp>-<git-sha>/{bin/resume-server,dist/,release.json}
|
||||
# current -> releases/<…> what the container is running
|
||||
# previous -> releases/<…> what --rollback restores
|
||||
#
|
||||
# A deploy is: build → test → stage → flip `current` → build the image from that
|
||||
# release directory → `compose up -d` → `curl /healthz` until ready. Any failure
|
||||
# after the flip puts `current` and the container back on the previous release
|
||||
# and exits non-zero. No step is a model decision (AGENTS.md §4.6); this script
|
||||
# is the only thing that publishes.
|
||||
#
|
||||
# Rollback rehearsal (Definition of Done: "a deliberate bad build proves rollback"):
|
||||
#
|
||||
# good=$(readlink ~/deployment/resume/current)
|
||||
# bad=$(mktemp -d) && mkdir -p "$bad/bin" "$bad/dist"
|
||||
# printf '#!/bin/sh\nexit 1\n' >"$bad/bin/resume-server" # binary that cannot start
|
||||
# chmod +x "$bad/bin/resume-server"
|
||||
# cp -a ~/deployment/resume/"$good"/dist/. "$bad/dist/" # valid pages
|
||||
# ./deploy/deploy.sh --release-dir "$bad" # expect exit 1
|
||||
# readlink ~/deployment/resume/current # still $good
|
||||
#
|
||||
# Run it on the runtime host (.13) from a checkout of this repository. It never
|
||||
# reads content/ and never calls a model.
|
||||
|
||||
set -Eeuo pipefail
|
||||
|
||||
APP=ai-resume # image tag (fixed by Phase 3 §3.2)
|
||||
PROJECT=ai-resume-site # compose project; distinct from the older
|
||||
# /home/sam/deployment/ai-resume project on .13
|
||||
PORT=8096
|
||||
HEALTH_PATH=/healthz
|
||||
|
||||
# The host address the port is published on. Caddy runs on .35 and proxies to
|
||||
# this address, so a loopback publish is unreachable from the edge (Sam's
|
||||
# decision, 2026-10-26, interaction 54aeb1a2). Never 0.0.0.0: the port must not
|
||||
# be reachable off the LAN. The health gate below follows this address — once the
|
||||
# port is published on the LAN address the host no longer listens on
|
||||
# 127.0.0.1:8096, so a loopback probe would refuse and every deploy would roll
|
||||
# back.
|
||||
PUBLISH_IP=${RESUME_PUBLISH_IP:-192.168.20.13}
|
||||
|
||||
ROOT=${RESUME_DEPLOY_ROOT:-/home/sam/deployment/resume}
|
||||
RELEASES=$ROOT/releases
|
||||
CURRENT=$ROOT/current
|
||||
PREVIOUS=$ROOT/previous
|
||||
|
||||
KEEP=${RESUME_KEEP_RELEASES:-5} # release directories kept on disk
|
||||
HEALTH_TRIES=${RESUME_HEALTH_TRIES:-40} # seconds to wait for /healthz
|
||||
|
||||
REPO=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)
|
||||
COMPOSE=$REPO/deploy/compose.yml
|
||||
DOCKERFILE=$REPO/deploy/Dockerfile
|
||||
|
||||
MODE=deploy
|
||||
RELEASE_DIR_OVERRIDE=
|
||||
|
||||
log() { printf '%s %s
|
||||
' "$(date -u +%H:%M:%SZ)" "$*" >&2; }
|
||||
warn() { printf '%s WARN %s
|
||||
' "$(date -u +%H:%M:%SZ)" "$*" >&2; }
|
||||
die() { printf '%s FAIL %s
|
||||
' "$(date -u +%H:%M:%SZ)" "$*" >&2; exit 1; }
|
||||
|
||||
on_err() {
|
||||
local rc=$? line=$1
|
||||
printf '%s FAIL line %s exited %s
|
||||
' "$(date -u +%H:%M:%SZ)" "$line" "$rc" >&2
|
||||
exit "$rc"
|
||||
}
|
||||
trap 'on_err $LINENO' ERR
|
||||
|
||||
usage() {
|
||||
sed -n '2,13p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
|
||||
exit "${1:-0}"
|
||||
}
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case $1 in
|
||||
--rollback) MODE=rollback ;;
|
||||
--status) MODE=status ;;
|
||||
--release-dir) shift; RELEASE_DIR_OVERRIDE=${1:-} ;;
|
||||
--keep) shift; KEEP=${1:-5} ;;
|
||||
-h|--help) usage 0 ;;
|
||||
*) printf 'unknown argument: %s
|
||||
' "$1" >&2; usage 1 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
# ---------------------------------------------------------------- helpers ---
|
||||
|
||||
compose() { docker compose -f "$COMPOSE" -p "$PROJECT" "$@"; }
|
||||
|
||||
current_target() { readlink -- "$CURRENT" 2>/dev/null || true; }
|
||||
previous_target() { readlink -- "$PREVIOUS" 2>/dev/null || true; }
|
||||
|
||||
# flip <relative-path-under-ROOT> — replace the symlink atomically. A reader sees
|
||||
# either the old release or the new one, never a missing link.
|
||||
flip() {
|
||||
ln -sfn "$1" "$CURRENT.tmp"
|
||||
mv -Tf "$CURRENT.tmp" "$CURRENT"
|
||||
}
|
||||
|
||||
release_name_of() { basename "$1"; }
|
||||
|
||||
# The tag deploy/compose.yml runs. Releases are distinguished by content, not by
|
||||
# tag, so compose.yml keeps its fixed image name.
|
||||
image_name() { printf '%s:local' "$APP"; }
|
||||
|
||||
# The port must be ours, or free. 8095 belongs to offsite.lab.
|
||||
check_port() {
|
||||
local listening
|
||||
listening=$(ss -ltnH "sport = :$PORT" 2>/dev/null || true)
|
||||
if [ -n "$listening" ]; then
|
||||
if [ -z "$(compose ps -q resume 2>/dev/null)" ]; then
|
||||
printf '%s
|
||||
' "$listening" >&2
|
||||
die "port $PORT is held by something that is not $APP; pick the next free port and record it in reference/stack.md"
|
||||
fi
|
||||
log "port $PORT is held by our own container (expected on a redeploy)"
|
||||
fi
|
||||
}
|
||||
|
||||
check_tools() {
|
||||
command -v docker >/dev/null || die "docker is not on PATH"
|
||||
docker compose version >/dev/null 2>&1 || die "docker compose v2 is not available"
|
||||
command -v curl >/dev/null || die "curl is not on PATH"
|
||||
command -v ss >/dev/null || die "ss (iproute2) is not on PATH"
|
||||
mkdir -p "$RELEASES"
|
||||
}
|
||||
|
||||
memory_report() {
|
||||
local avail
|
||||
avail=$(awk '/^MemAvailable:/ {print int($2/1024)}' /proc/meminfo)
|
||||
log "host memory available: ${avail} MiB — the container cap is 120 MiB; .13 is memory-tight by design"
|
||||
if [ "${avail:-0}" -lt 512 ]; then
|
||||
warn "under 512 MiB available; the Go build may push this host into swap"
|
||||
fi
|
||||
}
|
||||
|
||||
# --------------------------------------------------------------- build ------
|
||||
|
||||
build_and_test() {
|
||||
log "build + test in $REPO"
|
||||
(
|
||||
cd "$REPO"
|
||||
rm -rf dist
|
||||
if command -v nix >/dev/null 2>&1; then
|
||||
nix develop . -c bash -c 'make validate test dist'
|
||||
elif command -v templ >/dev/null 2>&1 && command -v go >/dev/null 2>&1; then
|
||||
make validate test dist
|
||||
else
|
||||
echo "no toolchain: neither 'nix develop' nor templ+go is available" >&2
|
||||
exit 1
|
||||
fi
|
||||
)
|
||||
[ -f "$REPO/bin/resume-server" ] || die "build produced no bin/resume-server"
|
||||
[ -f "$REPO/dist/index.html" ] || die "build produced no dist/index.html"
|
||||
log "build ok: $(jq -r '"\(.pages) pages, \(.documents) documents, \(.fragments) fragments"' "$REPO/dist/build.json" 2>/dev/null || echo 'dist/build.json unreadable')"
|
||||
}
|
||||
|
||||
# -------------------------------------------------------------- release -----
|
||||
|
||||
write_manifest() {
|
||||
local dir=$1 sha dirty
|
||||
sha=$(git -C "$REPO" rev-parse HEAD 2>/dev/null || echo unknown)
|
||||
dirty=false
|
||||
git -C "$REPO" diff --quiet 2>/dev/null || dirty=true
|
||||
cat >"$dir/release.json" <<JSON
|
||||
{
|
||||
"release": "$(basename "$dir")",
|
||||
"image": "$(image_name)",
|
||||
"git_sha": "$sha",
|
||||
"git_dirty": $dirty,
|
||||
"built_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
|
||||
"host": "$(hostname)",
|
||||
"pages": $(jq -r '.pages // 0' "$REPO/dist/build.json" 2>/dev/null || echo 0),
|
||||
"documents": $(jq -r '.documents // 0' "$REPO/dist/build.json" 2>/dev/null || echo 0)
|
||||
}
|
||||
JSON
|
||||
}
|
||||
|
||||
stage_release() {
|
||||
local name dest sha
|
||||
sha=$(git -C "$REPO" rev-parse HEAD 2>/dev/null || echo unknown)
|
||||
name=$(date -u +%Y%m%dT%H%M%SZ)-${sha:0:7}
|
||||
dest=$RELEASES/$name
|
||||
|
||||
mkdir -p "$dest/bin" "$dest/dist"
|
||||
install -m 0755 "$REPO/bin/resume-server" "$dest/bin/resume-server"
|
||||
cp -a "$REPO/dist/." "$dest/dist/"
|
||||
write_manifest "$dest"
|
||||
printf '%s
|
||||
' "$dest"
|
||||
}
|
||||
|
||||
# Copy an externally built release into releases/ so that every deployable
|
||||
# directory lives under exactly one root and `current` never dangles.
|
||||
adopt_release() {
|
||||
local src dest
|
||||
src=$(cd "$1" && pwd)
|
||||
[ -f "$src/bin/resume-server" ] || die "$src has no bin/resume-server"
|
||||
[ -f "$src/dist/index.html" ] || die "$src has no dist/index.html"
|
||||
dest=$RELEASES/$(date -u +%Y%m%dT%H%M%SZ)-manual
|
||||
mkdir -p "$dest"
|
||||
cp -a "$src/." "$dest/"
|
||||
write_manifest "$dest"
|
||||
printf '%s
|
||||
' "$dest"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- image -----
|
||||
|
||||
build_image() {
|
||||
log "docker build from $1"
|
||||
docker build --file "$DOCKERFILE" --tag "$(image_name)" "$1" 1>&2
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ run -----
|
||||
|
||||
start() {
|
||||
log "docker compose up -d ($1)"
|
||||
compose up -d --no-build --force-recreate 1>&2
|
||||
}
|
||||
|
||||
wait_healthy() {
|
||||
local i body
|
||||
for i in $(seq 1 "$HEALTH_TRIES"); do
|
||||
if body=$(curl -fsS --max-time 3 "http://$PUBLISH_IP:$PORT$HEALTH_PATH" 2>/dev/null); then
|
||||
if printf '%s' "$body" | grep -Eq '"status"[[:space:]]*:[[:space:]]*"ok"'; then
|
||||
log "health gate passed after ${i}s"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
warn "health gate failed after ${HEALTH_TRIES}s; last response:"
|
||||
curl -sS --max-time 3 "http://$PUBLISH_IP:$PORT$HEALTH_PATH" >&2 || true
|
||||
compose logs --tail 40 resume >&2 || true
|
||||
return 1
|
||||
}
|
||||
|
||||
# Definition-of-Done evidence, checked on every deploy instead of by hand:
|
||||
# mem_limit == memswap_limit == the value declared in compose.yml, and the port
|
||||
# published on $PUBLISH_IP only (the LAN address Caddy proxies to — never
|
||||
# 0.0.0.0 and no longer loopback).
|
||||
assert_shape() {
|
||||
local cid mem swap ip declared
|
||||
declared=$(awk '/^[[:space:]]*mem_limit:/ {gsub(/[^0-9]/, "", $2); print $2}' "$COMPOSE")
|
||||
cid=$(compose ps -q resume 2>/dev/null || true)
|
||||
[ -n "$cid" ] || die "no container for service 'resume'"
|
||||
mem=$(docker inspect -f '{{.HostConfig.Memory}}' "$cid")
|
||||
swap=$(docker inspect -f '{{.HostConfig.MemorySwap}}' "$cid")
|
||||
[ "$mem" = "$swap" ] || die "mem_limit ($mem) != memswap_limit ($swap)"
|
||||
[ "$mem" = "$((declared * 1024 * 1024))" ] || die "mem_limit is $mem, expected ${declared} MiB"
|
||||
ip=$(docker inspect -f '{{range $p, $b := .HostConfig.PortBindings}}{{range $b}}{{.HostIp}} {{end}}{{end}}' "$cid" | xargs)
|
||||
[ "$ip" = "$PUBLISH_IP" ] || die "published on '${ip}', expected $PUBLISH_IP only"
|
||||
log "shape ok: mem_limit == memswap_limit == ${declared} MiB, published on $PUBLISH_IP:$PORT"
|
||||
}
|
||||
|
||||
deploy_release() {
|
||||
local dir old name
|
||||
dir=$1
|
||||
name=$(release_name_of "$dir")
|
||||
[ -d "$dir" ] || die "release directory does not exist: $dir"
|
||||
[ -f "$dir/bin/resume-server" ] || die "$dir has no bin/resume-server"
|
||||
[ -f "$dir/dist/index.html" ] || die "$dir has no dist/index.html — a build that cannot serve a landing page is not a release"
|
||||
|
||||
build_image "$dir"
|
||||
old=$(current_target)
|
||||
|
||||
# The flip happens before the container change, so `current` always describes
|
||||
# what is being started. The release it replaced becomes `previous`.
|
||||
flip "releases/$name"
|
||||
if [ -n "$old" ] && [ "$old" != "releases/$name" ]; then
|
||||
ln -sfn "$old" "$PREVIOUS"
|
||||
log "previous -> $old"
|
||||
fi
|
||||
|
||||
if start "$name" && wait_healthy && assert_shape; then
|
||||
printf '%s
|
||||
' "$(date -u +%Y-%m-%dT%H:%M:%SZ) deploy releases/$name ok" >>"$ROOT/deploy.log"
|
||||
log "deployed $name"
|
||||
prune
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [ -z "$old" ] || [ "$old" = "releases/$name" ]; then
|
||||
warn "first deploy failed and there is no previous release; leaving the container down"
|
||||
compose down >&2 || true
|
||||
rm -f "$CURRENT"
|
||||
return 1
|
||||
fi
|
||||
|
||||
warn "deploy of $name failed; restoring $old"
|
||||
flip "$old"
|
||||
build_image "$ROOT/$old" || die "could not rebuild the previous image; container state unknown"
|
||||
if start "$old" && wait_healthy && assert_shape; then
|
||||
printf '%s
|
||||
' "$(date -u +%Y-%m-%dT%H:%M:%SZ) rolled back to $old after failed deploy of releases/$name" >>"$ROOT/deploy.log"
|
||||
die "deploy of releases/$name failed; rolled back to $old (the failed release is kept for inspection)"
|
||||
fi
|
||||
die "deploy of releases/$name failed AND rollback to $old failed — container state is unknown; run '$0 --status' and read $ROOT/deploy.log"
|
||||
}
|
||||
|
||||
do_rollback() {
|
||||
local cur prev
|
||||
cur=$(current_target)
|
||||
prev=$(previous_target)
|
||||
[ -n "$prev" ] || die "no previous release recorded; nothing to roll back to"
|
||||
[ "$prev" != "$cur" ] || die "previous and current are the same release ($cur); nothing to roll back"
|
||||
[ -d "$ROOT/$prev" ] || die "previous release directory is missing: $ROOT/$prev"
|
||||
|
||||
log "rolling back: $cur -> $prev"
|
||||
build_image "$ROOT/$prev"
|
||||
flip "$prev"
|
||||
if start "$prev" && wait_healthy && assert_shape; then
|
||||
[ -n "$cur" ] && ln -sfn "$cur" "$PREVIOUS"
|
||||
printf '%s
|
||||
' "$(date -u +%Y-%m-%dT%H:%M:%SZ) rollback $cur -> $prev" >>"$ROOT/deploy.log"
|
||||
log "rolled back to $prev; $cur is now previous"
|
||||
return 0
|
||||
fi
|
||||
|
||||
warn "rollback to $prev failed; returning to $cur"
|
||||
flip "$cur"
|
||||
build_image "$ROOT/$cur" || die "could not rebuild $cur; container state unknown"
|
||||
start "$cur" && wait_healthy || die "return to $cur failed; container state unknown"
|
||||
die "rollback to $prev failed; $cur was restored"
|
||||
}
|
||||
|
||||
# Keep the newest KEEP releases plus current and previous, nothing else.
|
||||
prune() {
|
||||
local cur prev name dir
|
||||
cur=$(current_target); prev=$(previous_target)
|
||||
for dir in $(ls -1dt "$RELEASES"/*/ 2>/dev/null | tail -n +"$((KEEP + 1))"); do
|
||||
name="releases/$(basename "$dir")"
|
||||
if [ "$name" = "$cur" ] || [ "$name" = "$prev" ]; then continue; fi
|
||||
rm -rf -- "$dir"
|
||||
log "pruned $name"
|
||||
done
|
||||
}
|
||||
|
||||
do_status() {
|
||||
local cur prev body
|
||||
cur=$(current_target); prev=$(previous_target)
|
||||
body=$(curl -fsS --max-time 3 "http://$PUBLISH_IP:$PORT$HEALTH_PATH" 2>/dev/null || true)
|
||||
printf 'root %s
|
||||
' "$ROOT"
|
||||
printf 'current %s
|
||||
' "${cur:-<none>}"
|
||||
printf 'previous %s
|
||||
' "${prev:-<none>}"
|
||||
if [ -n "$body" ]; then
|
||||
printf 'health %s
|
||||
' "$(printf '%s' "$body" | jq -c '{status, pages: .static.pages, search: .search.ready}' 2>/dev/null || printf '%s' "$body")"
|
||||
else
|
||||
printf 'health unreachable
|
||||
'
|
||||
fi
|
||||
compose ps || true
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ main ----
|
||||
|
||||
case $MODE in
|
||||
status)
|
||||
do_status
|
||||
;;
|
||||
rollback)
|
||||
check_tools
|
||||
do_rollback
|
||||
;;
|
||||
deploy)
|
||||
check_tools
|
||||
check_port
|
||||
memory_report
|
||||
if [ -n "$RELEASE_DIR_OVERRIDE" ]; then
|
||||
deploy_release "$(adopt_release "$RELEASE_DIR_OVERRIDE")"
|
||||
else
|
||||
build_and_test
|
||||
deploy_release "$(stage_release)"
|
||||
fi
|
||||
log "live: $(current_target)"
|
||||
;;
|
||||
esac
|
||||
55
deploy/qdrant.yml
Normal file
@@ -0,0 +1,55 @@
|
||||
# Qdrant for the corpus (docs/phase-4-corpus.md, reference/stack.md).
|
||||
#
|
||||
# mem_limit == memswap_limit == 400m the cap in the memory budget, and the
|
||||
# container cannot reach host swap
|
||||
# ports on 127.0.0.1 only Caddy on .35 is the only public listener
|
||||
# its own compose project stopping or recreating it cannot disturb
|
||||
# the Mode A site (test A1 still passes:
|
||||
# the site never calls Qdrant)
|
||||
#
|
||||
# Run on the runtime host (.13):
|
||||
#
|
||||
# docker compose -f deploy/qdrant.yml up -d
|
||||
# curl -s http://127.0.0.1:6333/ # must answer with a version
|
||||
#
|
||||
# Then, in the corpus environment (RESUME_QDRANT_URL is already the default):
|
||||
#
|
||||
# python3 -m corpus.cli rebuild
|
||||
#
|
||||
# Deleting the volume is always recoverable: `rebuild` re-derives every vector
|
||||
# from content/.
|
||||
|
||||
name: ai-resume-index
|
||||
|
||||
services:
|
||||
qdrant:
|
||||
image: qdrant/qdrant:v1.20.0
|
||||
container_name: ai-resume-qdrant
|
||||
restart: unless-stopped
|
||||
mem_limit: 400m
|
||||
memswap_limit: 400m
|
||||
ports:
|
||||
- "127.0.0.1:6333:6333" # REST — the pipeline and the query service use this
|
||||
- "127.0.0.1:6334:6334" # gRPC — reserved for the Phase 5 query service
|
||||
volumes:
|
||||
- qdrant_storage:/qdrant/storage
|
||||
environment:
|
||||
# The corpus is ~1k chunks today and the measurement in
|
||||
# reference/stack.md records the ceiling. Keep the HNSW defaults; do not
|
||||
# tune for speed on a memory-bound host.
|
||||
QDRANT__TELEMETRY_DISABLED: "true"
|
||||
QDRANT__LOG_LEVEL: "WARN"
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: 10m
|
||||
max-file: "3"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "bash -c '</dev/tcp/127.0.0.1/6333' || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
volumes:
|
||||
qdrant_storage:
|
||||
64
docs/content-redaction.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# Content redaction pass — what was excluded and why
|
||||
|
||||
**Scope:** every document under `content/`, authored in Phase 1 (issue AUD-131).
|
||||
**Date of pass:** 2026-10-26.
|
||||
**Sources read for this pass:** `../resume_ai_system/plan/EVIDENCE-NOTES.md`, `../resume_ai_system/plan/draft-content.md`, `../resume_ai_system/plan/20-content-spine.md`, `../resume_ai_system/old_projects.md`, `resume_2025.docx`, `reference/content-map.md`, `reference/decisions.md`, `reference/stack.md`, and the Obsidian backup notes.
|
||||
|
||||
Acceptance test C6 covers this: *no family, address, credential or third-party personal data in pages or corpus.* The list below is what was removed to satisfy it, and the reason for each removal.
|
||||
|
||||
## Excluded — personal identifiers
|
||||
|
||||
| Item | Where it appeared | Why excluded |
|
||||
|---|---|---|
|
||||
| Date of birth | `resume_2025.docx`, Personal Details | Not needed to assess capability. Age is a discriminator and an identity-theft input. |
|
||||
| Phone number | `resume_2025.docx`, Personal Details | Publishing a mobile exposes it to scrapers. Sam has not confirmed a phone number belongs on the site (`reference/stack.md` open items). |
|
||||
| Street address / suburb | `resume_2025.docx` ("live in Ivanhoe"), `draft-content.md` | Address-level detail is a safety issue and is irrelevant to the work. "Melbourne based" is the published granularity. |
|
||||
| Email address | `resume_2025.docx` | Kept off the pages in Phase 1. Contact details are an open decision for Sam; the narrative states the policy rather than the address. |
|
||||
| Passport and visa status | `resume_2025.docx` | Immigration status is not published. Working rights are a conversation, not a web page. |
|
||||
| Family names and details | `EVIDENCE-NOTES.md` (school portals, children, household) | Never published. The 2020–2026 period is described as "full-time parent to two children" with no names, ages, schools or locations. |
|
||||
|
||||
## Excluded — third-party personal data
|
||||
|
||||
| Item | Where it appeared | Why excluded |
|
||||
|---|---|---|
|
||||
| Names of individual client contacts, agency colleagues and referees | résumé prose | No third party has consented to being named. Clients are represented by brand only, which was public campaign work. |
|
||||
| Details identifying minors in a personal anecdote | `EVIDENCE-NOTES.md` interests ("a 2 day hike … with 7 13 year old boys") | Rewritten as "a group of teenagers in tow". A count of identifiable children is not needed to make the point. |
|
||||
| Names of household members appearing in automation examples | `EVIDENCE-NOTES.md` | Household automation is described by function (lighting, audio, doors) and never by person. |
|
||||
|
||||
## Excluded — credentials and infrastructure detail
|
||||
|
||||
| Item | Where it appeared | Why excluded |
|
||||
|---|---|---|
|
||||
| API keys, tokens, passphrases, secret file names | n/a — never copied | Nothing from `~/.config/environment.d/` was read or quoted. No key material enters the repo, per `AGENTS.md` §4.4. |
|
||||
| Backup passphrase file path and repository passphrase | Obsidian backup note | Publishing the location of a passphrase file is a hint worth withholding. Backups are described by tier, tool and retention only. |
|
||||
| Hostnames, internal IP addresses, ports-by-host, public DNS names | `reference/stack.md`, Obsidian notes, live inspection | Internal addressing is not needed to judge the platform work. Hosts are described by role ("the runtime host", "the edge host"). |
|
||||
| Internal service URLs and dashboards | `EVIDENCE-NOTES.md` (the pipeline dashboard) | Described as "a dashboard built to keep track of it". A private URL on a public page is an invitation. |
|
||||
| Third-party account identifiers, billing detail | n/a | Not in the sources and not wanted. |
|
||||
|
||||
## Excluded — unverifiable or invented content
|
||||
|
||||
| Item | Where it appeared | Why excluded |
|
||||
|---|---|---|
|
||||
| "42 devices" | recovered site content | Explicitly withdrawn in `draft-content.md`. Not used. |
|
||||
| "~800ms end-to-end" latency | recovered site content | Explicitly withdrawn. Not used. |
|
||||
| Invented venues, competition results, swell sizes, distances and restaurants | recovered pursuits content | Marked ❌ SUSPECT and confirmed as generated. Replaced with one honest line per pursuit. |
|
||||
| "Diajio" | recovered client list | Spelling error for **Diageo**. Corrected, not carried. |
|
||||
| Screenshot of the BBC National Archive engagement | recovered site content | The work was an internal tool. Described only; no image. |
|
||||
|
||||
## Metric discipline
|
||||
|
||||
Every numeric `metric` in `content/` is traced to a source line, and the trace is recorded per metric in the completion comment on AUD-131. Two figures were deliberately **not** promoted:
|
||||
|
||||
- **Businesses: 3 active, 12 cumulative.** The site says three are running now and twelve adopted it over time. Conflating them would be the easiest inflation available on this site.
|
||||
- **Where-Woof: deployed, pre-market.** The word "commercial" is not used about it anywhere.
|
||||
|
||||
## Two numbers reported, not smoothed
|
||||
|
||||
1. **Agencies: 22 published, 23 in the summary material.** The recovered, verified list holds twenty-two agency names. The published count is the count of the list. If a name is missing, the number rises to twenty-three and the note in `content/history/l3/history-client-wall.md` is deleted.
|
||||
2. **Period start dates.** Where the sources give no start date, the period is anchored on the homelab year (2020) and the page says so. Exact start dates are Sam's to correct.
|
||||
|
||||
## Open questions for Sam
|
||||
|
||||
- Contact details: email only, or also a phone number? (`about-narrative.md` states the current policy: email, supplied directly.)
|
||||
- Is a Melbourne suburb to be published? Currently excluded.
|
||||
- The missing twenty-third agency, if there is one.
|
||||