Compare commits
8 Commits
main
...
feat/phase
| Author | SHA1 | Date | |
|---|---|---|---|
|
447a0523fb | ||
|
|
0b910d216e | ||
|
|
720feddfcc | ||
|
|
4896e18a41 | ||
|
|
da674b5555 | ||
|
|
4e52015fd2 | ||
|
|
e3f54f58c1 | ||
|
|
1e46ec009a |
22
Makefile
22
Makefile
@@ -23,7 +23,8 @@ BIN := bin
|
||||
DIST := dist
|
||||
|
||||
.PHONY: build dist generate dev test validate validate-self-test fmt vet clean help \
|
||||
corpus corpus-sync corpus-rebuild corpus-audit corpus-verify corpus-demo corpus-test
|
||||
corpus corpus-sync corpus-rebuild corpus-offline corpus-audit corpus-verify \
|
||||
corpus-demo corpus-test corpus-graph corpus-scan corpus-sources
|
||||
|
||||
help:
|
||||
@echo "make build templ generate, compile bin/, no dist"
|
||||
@@ -36,11 +37,15 @@ help:
|
||||
@echo "make clean remove bin/ and dist/"
|
||||
@echo ""
|
||||
@echo "corpus (Phase 4, python3 -m corpus.cli):"
|
||||
@echo "make corpus rebuild the corpus from content/ (drops the collection)"
|
||||
@echo "make corpus rebuild the pool from the estate (drops both collections)"
|
||||
@echo "make corpus-offline rebuild content/ only, no Gitea (recovery and CI)"
|
||||
@echo "make corpus-sync incremental update; only changed documents are re-embedded"
|
||||
@echo "make corpus-audit freshness + drift audit (exit 1 on findings)"
|
||||
@echo "make corpus-verify check the index against content/ and Qdrant"
|
||||
@echo "make corpus-demo same query with and without an area filter"
|
||||
@echo "make corpus-graph Q= query graph nodes and edges (with their source note)"
|
||||
@echo "make corpus-scan search the pool for credential-shaped content"
|
||||
@echo "make corpus-sources show the platform guides and the last harvest"
|
||||
@echo "make corpus-test corpus unit tests (no container, no network)"
|
||||
|
||||
build: generate
|
||||
@@ -87,6 +92,9 @@ corpus: corpus-rebuild
|
||||
corpus-rebuild:
|
||||
@$(CORPUS) rebuild
|
||||
|
||||
corpus-offline:
|
||||
@$(CORPUS) --offline rebuild
|
||||
|
||||
corpus-sync:
|
||||
@$(CORPUS) sync
|
||||
|
||||
@@ -99,6 +107,16 @@ corpus-verify:
|
||||
corpus-demo:
|
||||
@$(CORPUS) demo-filter --query "distributed audio endpoints" --area edge
|
||||
|
||||
corpus-graph:
|
||||
@test -n "$(Q)" || { echo "usage: make corpus-graph Q='photo pipeline'"; exit 2; }
|
||||
@$(CORPUS) graph --query "$(Q)"
|
||||
|
||||
corpus-scan:
|
||||
@$(CORPUS) scan
|
||||
|
||||
corpus-sources:
|
||||
@$(CORPUS) sources
|
||||
|
||||
corpus-test:
|
||||
@$(PYTHON) -m unittest discover -s corpus/tests -t .
|
||||
|
||||
|
||||
@@ -49,7 +49,7 @@ The honest framing is in the evidence notes and is repeated here: *full-time par
|
||||
|
||||
## What next
|
||||
|
||||
The same three areas at a larger scale: [AI and agentic systems](../ai/ai-systems.md), the [platform engineering underneath them](../platform/platform-estate.md), and the [constrained devices at the edge](../edge/edge-fleet.md). Melbourne based and flexible about location and format — remote, hybrid, office, or travel.
|
||||
The same three areas at a larger scale: AI and agentic systems, the platform engineering underneath them, and the constrained devices at the edge. Melbourne based and flexible about location and format — remote, hybrid, office, or travel.
|
||||
|
||||
## Contact policy
|
||||
|
||||
@@ -60,3 +60,16 @@ This site publishes no address, no phone number and no family details. Contact i
|
||||
- **No computer science degree.** The trade is a longer, broader practical education; the pages on this site are the argument, not the certificate.
|
||||
- **A period of self-employment is described as such.** It is not dressed up as a sabbatical or as consulting.
|
||||
- **Interests are one honest sentence each.** The previous version of this site invented venues and results. That has been deleted rather than corrected in place.
|
||||
|
||||
## Drill-downs
|
||||
|
||||
- [Non-IT working history](l3/about-non-it-work.md) — level 3
|
||||
- [Education](l3/about-education.md) — level 3
|
||||
- [Interests — one honest line each](l3/about-interests.md) — level 3
|
||||
- [The AI estate](../ai/ai-systems.md) — level 2
|
||||
- [The platform estate](../platform/platform-estate.md) — level 2
|
||||
- [The edge fleet](../edge/edge-fleet.md) — level 2
|
||||
|
||||
## Period and confidence
|
||||
|
||||
`period: 1999–`. Confidence is `running`: this is the current self-description, verified against the evidence notes on 2026-10-26.
|
||||
|
||||
@@ -3,7 +3,7 @@ id: about-overview
|
||||
area: about
|
||||
level: 1
|
||||
title: "About"
|
||||
claim: "Sam has shipped software for twenty-five years — self-taught, through a non-IT working life and five years overseas — and now builds AI systems, the platforms under them and the devices at the edge."
|
||||
claim: "Twenty-five years of shipping software — built through self-teaching · a non-IT working life · and five years overseas — now aimed at AI systems · the platforms under them and the devices at the edge."
|
||||
confidence: running
|
||||
period: "1999–"
|
||||
tech: [NixOS, Neovim, Python, JavaScript, React, Go, HTMX]
|
||||
@@ -13,4 +13,13 @@ public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [about, narrative, background]
|
||||
---
|
||||
The other five areas prove what can be built; this one explains where the judgement came from. The narrative, including 2020 to 2026, is on the [narrative page](about-narrative.md).
|
||||
|
||||
# About
|
||||
|
||||
**Claim.** A career built by self-teaching — from a childhood in the 1980s with a borrowed language reference, through fifteen years of agency delivery, a working life outside technology, and five years overseas — now concentrated on AI systems, the platforms they run on, and the hardware at the edge.
|
||||
|
||||
**Status.** Current. Melbourne based, and flexible about how the work is done — remote, hybrid, office, or travel.
|
||||
|
||||
**What this page is for.** The other five areas prove what can be built. This one explains where the judgement came from: a non-technical working life, a fine-arts and philosophy education, and a period spent building a business while raising two children.
|
||||
|
||||
**Start here.** [The narrative](about-narrative.md) — including an honest account of 2020 to 2026.
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [about, education, artefact]
|
||||
---
|
||||
|
||||
Formal presentation lands in Phase 7. No certificates, transcripts or student identifiers are published.
|
||||
# Artefact — Education
|
||||
|
||||
**Status: stub.** Formal presentation lands in Phase 7. No certificates, transcripts or student identifiers are published.
|
||||
|
||||
## The record
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [about, interests, artefact]
|
||||
---
|
||||
|
||||
# Artefact — Interests · one honest line each
|
||||
|
||||
**Status: stub for the visual treatment.**
|
||||
|
||||
## Music
|
||||
Blues and jazz piano and blues harp; played in two bands. Studied music composition at university and kept the piano as the thing that resets the week.
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [about, history, work, artefact]
|
||||
---
|
||||
|
||||
The roles are final; the presentation lands in Phase 7.
|
||||
# Artefact — Non-IT working history
|
||||
|
||||
**Status: stub for the visual treatment.** The roles are final; the presentation lands in Phase 7.
|
||||
|
||||
## The roles
|
||||
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
---
|
||||
id: about-sources
|
||||
area: about
|
||||
level: 3
|
||||
title: "What the assistant reads"
|
||||
claim: "Four sources, one pool, and a published rule for every one of them."
|
||||
confidence: running
|
||||
period: "2026–"
|
||||
tech: [Gitea, Obsidian, Graphify, Qdrant, SQLite]
|
||||
metrics: []
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, sources, corpus, retrieval]
|
||||
---
|
||||
|
||||
# What the assistant reads
|
||||
|
||||
The assistant answers from a pool built out of four sources, not from a single
|
||||
folder of copy.
|
||||
|
||||
| Source | Tier | What it contributes |
|
||||
|---|---|---|
|
||||
| This site's content | 1 | The curated claims: what Sam says is true about the work |
|
||||
| The Gitea estate | 2 | README files, file trees, selected source files, commit messages and language statistics |
|
||||
| The notes vault | 2 | Project and area notes, in Sam's own words |
|
||||
| The knowledge graph | 2 | Nodes and edges over the vault, each joined back to the note it came from |
|
||||
|
||||
Tier 1 wins when the two disagree. A harvested README can be out of date; the
|
||||
curated claim is the one that is maintained.
|
||||
|
||||
Every chunk records the platform it came from, the URL or repository path it can
|
||||
be opened at, the area it feeds, its tier, its honest confidence and when it last
|
||||
changed. A chunk without an origin is refused, not defaulted.
|
||||
|
||||
The rules are not hidden in code. Each platform has a guide that says what it is,
|
||||
where it lives, what is included, what is excluded and whether it is enabled.
|
||||
Those guides are shown at [the sources page](/platforms/) and are the same files
|
||||
the ingestion pipeline reads.
|
||||
|
||||
## What is excluded
|
||||
|
||||
`node_modules`, virtualenvs, git internals, trash, editor state, the SilverBullet
|
||||
auth file, credential-shaped files, binaries and media. Media is recorded as an
|
||||
image reference and never turned into text. Personal and family notes are not
|
||||
read.
|
||||
|
||||
The finished pool is scanned for credential-shaped content before a run can
|
||||
succeed, so the exclusion list is checked rather than trusted.
|
||||
@@ -3,7 +3,7 @@ id: ai-overview
|
||||
area: ai
|
||||
level: 1
|
||||
title: "AI & Agentic Systems"
|
||||
claim: "Sam builds and operates sixteen or more AI pipelines: agent harnesses, model routing and cost control, retrieval over a tagged corpus, and a voice front-end that runs a house."
|
||||
claim: "Builds and operates sixteen or more AI pipelines — agent harnesses · model routing and cost control · retrieval over a tagged corpus · and a voice front-end that runs a house."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [LangGraph, Prefect, n8n, Qdrant, OmniRoute, Whisper, Piper, MiniMax]
|
||||
@@ -13,4 +13,13 @@ public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [ai, agents, routing, retrieval, voice]
|
||||
---
|
||||
The pipelines run against real obligations — school portals, calendars, email, household audio — and they are operated rather than demonstrated. What actually runs, and what it cost to get there, is on the [capability page](ai-systems.md).
|
||||
|
||||
# AI & Agentic Systems
|
||||
|
||||
**Claim.** Sixteen or more AI pipelines are maintained in production — ten or more on n8n, Prefect and LangGraph, six on Paperclip — with one gateway in front of six model providers, retrieval over a metadata-tagged corpus, and a voice front-end that controls the house.
|
||||
|
||||
**Status.** Running. Every figure on this page traces to the evidence notes; the verification date is 2026-10-26.
|
||||
|
||||
**Why it reads as engineering rather than enthusiasm.** The pipelines run against real obligations — school portals, calendars, email, household audio — and they are operated: routed for cost, guarded before they act, and observed while they run.
|
||||
|
||||
**Start here.** [The capability page](ai-systems.md) explains what is actually run and what the trade-offs were. The three chips above resolve to artefacts below it.
|
||||
|
||||
@@ -14,6 +14,8 @@ last_verified: 2026-10-26
|
||||
tags: [ai, agents, routing, retrieval, voice, guardrails]
|
||||
---
|
||||
|
||||
# Agent systems that are operated rather than demonstrated
|
||||
|
||||
**Capability statement.** This is a working estate, not a prototype: more than sixteen pipelines in production, six providers behind a single gateway, retrieval that filters on metadata before it searches, and a voice front-end used daily by people who did not build it.
|
||||
|
||||
## What is actually run
|
||||
@@ -53,4 +55,16 @@ tags: [ai, agents, routing, retrieval, voice, guardrails]
|
||||
- **Not everything is automated.** Parts of the house are automated end to end and parts still need a person to press the button. The estate is described as it is run, not as it was imagined.
|
||||
- **Cost is bounded by routing, not by hope.** Caps and a refusal path exist so a runaway session cannot become a bill; the routing layer keeps the median request on a cheap model.
|
||||
|
||||
The period starts at the homelab year in the evidence notes: the AI estate and the homelab have the same origin.
|
||||
## Drill-downs
|
||||
|
||||
- [House AI hub — service topology](l3/ai-hub-architecture.md) — level 3
|
||||
- [Model routing and cost control](l3/ai-omniroute-routing.md) — level 3
|
||||
- [Paperclip and n8n pipelines in production](l3/ai-paperclip-pipelines.md) — level 3
|
||||
- [Retrieval corpus and query graph](l3/ai-rag-corpus.md) — level 3
|
||||
- [Voice pipeline — speech in and speech out](l3/ai-voice-pipeline.md) — level 3
|
||||
- [Guardrails — hooks in the model path](l3/ai-guardrails.md) — level 3
|
||||
- [Memory in two layers](l3/ai-memory-layers.md) — level 3
|
||||
|
||||
## Period and confidence
|
||||
|
||||
`period: 2020–`. The start is anchored on the homelab year in the evidence notes, because the AI estate and the homelab have the same origin. Confidence is `running`: this is the current arrangement, verified 2026-10-26 against the evidence notes.
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [ai, guardrails, security, artefact]
|
||||
---
|
||||
|
||||
A redacted injection-test result lands in Phase 6, when the guardrail library is wired into this site's own assistant.
|
||||
# Artefact — Guardrails · hooks in the model path
|
||||
|
||||
**Status: stub.** A redacted injection-test result lands in Phase 6, when the guardrail library is wired into this site's own assistant.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [ai, architecture, artefact]
|
||||
---
|
||||
|
||||
The diagram and the generated topology dump land in Phase 7 (polish). This stub exists so the level 2 page has no dangling reference.
|
||||
# Artefact — House AI hub service topology
|
||||
|
||||
**Status: stub.** The diagram and the generated topology dump land in Phase 7 (polish). This stub exists so the level 2 page has no dangling reference.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [ai, memory, context, artefact]
|
||||
---
|
||||
|
||||
A redacted session handoff and a before-and-after context measurement land in Phase 7.
|
||||
# Artefact — Memory in two layers · Engram and Headroom
|
||||
|
||||
**Status: stub.** A redacted session handoff and a before-and-after context measurement land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [ai, routing, cost, artefact]
|
||||
---
|
||||
|
||||
The routing decision path and a redacted cost table land in Phase 7. No keys or account identifiers are ever published.
|
||||
# Artefact — Model routing and cost control
|
||||
|
||||
**Status: stub.** The routing decision path and a redacted cost table land in Phase 7. No keys or account identifiers are ever published.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [ai, pipelines, agents, artefact]
|
||||
---
|
||||
|
||||
A redacted run log of one pipeline lands in Phase 7, alongside the pipeline inventory.
|
||||
# Artefact — Paperclip and n8n pipelines in production
|
||||
|
||||
**Status: stub.** A redacted run log of one pipeline lands in Phase 7, alongside the pipeline inventory.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [ai, retrieval, rag, artefact]
|
||||
---
|
||||
|
||||
Ingest and index are live; the diagram below shows both sides, and the query path is labelled as the design it is until Phase 5 ships.
|
||||
# Artefact — Retrieval corpus and query graph
|
||||
|
||||
**Status: stub.** The query graph diagram and an ingest run log land later; this site's own corpus build is Phase 4.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
@@ -24,12 +26,6 @@ The ingest and query path: documents are chunked and tagged with metadata at ing
|
||||
|
||||
Retrieval designed around payload filtering and a hard stop, plus the reasoning for moving off ChromaDB — heavier footprint and weaker filtering — recorded as a decision rather than a migration.
|
||||
|
||||
## The pipeline
|
||||
|
||||
|
||||
|
||||
The separating line is the design decision: metadata is written at ingest, so the query can filter by area, level and source before it pays for a similarity search. The fourth node is the hard stop — an answer that cannot be supported by what was retrieved is refused rather than improvised.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Ingest is delivered and measured (this site's own corpus, Phase 4). The query path is delivered in Phase 5; the streaming console is the visitor-facing half of it.
|
||||
The query graph diagram in Phase 7. The live demonstration of the same pattern is this site's own retrieval, delivered in Phase 4 and Phase 5.
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [ai, voice, audio, artefact]
|
||||
---
|
||||
|
||||
A short recording of music ducking under a spoken announcement lands in the optional voice phase. Text describes the chain until then.
|
||||
# Artefact — Voice pipeline · speech in and speech out
|
||||
|
||||
**Status: stub.** A short recording of music ducking under a spoken announcement lands in the optional voice phase. Text describes the chain until then.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -1,36 +0,0 @@
|
||||
package thebeer {
|
||||
import utils.Bg;
|
||||
|
||||
import com.greensock.TweenMax;
|
||||
|
||||
import flash.display.MovieClip;
|
||||
import flash.events.Event;
|
||||
|
||||
/**
|
||||
* @author samrolfe
|
||||
*/
|
||||
public class Siren extends MovieClip {
|
||||
private var targ : Bg;
|
||||
private var delays : Number;
|
||||
private var wide: Number;
|
||||
private var high: Number;
|
||||
|
||||
public function Siren(col:Number,t:Bg,d:Number) {
|
||||
delays=d;
|
||||
targ=t;
|
||||
x=targ.x;
|
||||
y=targ.y;
|
||||
wide=this.width;
|
||||
high=this.high;
|
||||
TweenMax.to(this,0,{autoAlpha:0,colorMatrixFilter:{colorize:col}});
|
||||
}
|
||||
|
||||
private function setPos(e:Event) : void {
|
||||
x=parent[targ].x;
|
||||
y=parent[targ].y;
|
||||
}
|
||||
|
||||
public function flash() : void {
|
||||
TweenMax.to(this,0,{autoAlpha:1,width:wide,height:high});
|
||||
TweenMax.to(this,.1,{width:0,height:0,repeat:5,repeatDelay:delays,yoyo:true,onComplete:finish});
|
||||
}
|
||||
@@ -1,53 +0,0 @@
|
||||
// Excerpt from js/utils/text_effects.js (2,156 lines) in the WebsiteBuilder
|
||||
// source. Two regions are reproduced verbatim: the timeline loop that derives
|
||||
// each letter's delay from an easing function (lines 1111-1147), and the
|
||||
// hand-written easing functions themselves (lines 1340-1349).
|
||||
$(mySplitTextTarg).each(function(i){
|
||||
|
||||
tweenData.letter=$(this);
|
||||
tweenData.text_static.letter=$(this);
|
||||
tweenData.text_text_to.letter=$(this);
|
||||
tweenData.text_text_from.letter=$(this);
|
||||
|
||||
if(text_effect_type_animation=="standard"){
|
||||
if(tweenData.ignoreItems=="on" && Math.random()<tweenData.ignore_amount)return;
|
||||
|
||||
if(!keepHide(tweenData,i,$(this).text()))return;
|
||||
|
||||
var transforms=getTransforms(tweenData);
|
||||
|
||||
|
||||
//currenttime,start,change,duration
|
||||
if(tweenData.staggerTimeRnd=="random")stagger=min_max_random(0,tweenData.staggerTime);
|
||||
// stagger=positionsCubic(tweenData.alternatecount, 0, tweenData.duration, tweenData.total)
|
||||
var ease=tweenData.easeFunction;
|
||||
// var ease="easeInCubic";
|
||||
stime=tweenData.duration;
|
||||
if(tweenData.direc=="in"){
|
||||
stagger=jQuery.easing[ease](null,tweenData.alternatecount, 0, tweenData.duration, tweenData.total)+tweenData.startWait
|
||||
transforms.delay=stagger;
|
||||
TweenMax.from($(this), tweenData.duration, transforms);
|
||||
tt=tweenData.duration+stagger;
|
||||
}
|
||||
if(tweenData.direc=="out"){
|
||||
stagger=jQuery.easing[ease](null,tweenData.alternatecount, 0, tweenData.duration, tweenData.total)
|
||||
transforms.delay=stagger;
|
||||
transforms.onComplete=tweenRemove($(this),stagger+stime+tweenData.removeWait)
|
||||
TweenMax.to($(this), tweenData.duration, transforms);
|
||||
tt=tweenData.duration+stagger+tweenData.removeWait+1;
|
||||
}
|
||||
|
||||
setCssTweens(stime,text_effect_type_animation,tweenData,tweenData.text_static,tweenData.text_text_to,tweenData.text_text_from,$(this),stagger);
|
||||
|
||||
|
||||
// --- easing functions, lines 1340-1349 ---
|
||||
//t,b,c,d
|
||||
//currenttime,start,change,duration
|
||||
//t and do can be time or frames
|
||||
function positionsCubic(t, b, c, d){
|
||||
t /= d;
|
||||
return c*t*t*t + b;
|
||||
}
|
||||
function positionSine(t, b, c, d) {
|
||||
return -c * Math.cos(t/d * (Math.PI/2)) + c + b;
|
||||
}
|
||||
@@ -1,54 +0,0 @@
|
||||
// Excerpt from php/setMediaEncoder.php (3,402 lines) in the WebsiteBuilder
|
||||
// source: the create_gif encode operation, lines 614-665, reproduced verbatim.
|
||||
if($output_options->encode_media=="create_gif") {
|
||||
$output_gif= ".." . $settings->directory . $output_file .$output_options->format .".gif";
|
||||
|
||||
$docmime= checkFinfo($input_file);
|
||||
$doctype=explode("/",$docmime)[0];
|
||||
if ($doctype!="video"){
|
||||
$output_options->error_message= "We only create gifs from a video - it must be the first video in the media list.";
|
||||
$returnFiles = returnFiles ( $output_options, $output_file .$output_options->format . '.mp4', "video/mp4", '', null );
|
||||
return $returnFiles;
|
||||
}
|
||||
|
||||
$cmd = "ffmpeg -i ".$input_file." -y -vf fps=".$settings->settings->movie_frame_rate.",scale=".$settings->settings->video_width.":".$settings->settings->video_height.",palettegen palette.png 2>&1";
|
||||
// $cmd = "ffmpeg -i ".$input_file." -y -vf fps=10,scale=".$settings->settings->video_width.":".$settings->settings->video_height.",palettegen palette.png 2>&1";
|
||||
$result=shell_exec($cmd);
|
||||
$first_media=json_decode($settings->add_media)[0];
|
||||
|
||||
// $cmd = "ffmpeg -ss ".$first_media->add_media_start." -t ".$first_media->add_media_duration." -i ".$input_file." -i palette.png -y -filter_complex \"fps=10[x];[x][1:v]paletteuse\" ".$output_gif." ";
|
||||
$cmd = "ffmpeg -ss ".$first_media->add_media_start." -t ".$first_media->add_media_duration." -i ".$input_file." -i palette.png -y -filter_complex \"fps=".$settings->settings->movie_frame_rate.",scale=".$settings->settings->video_width.":".$settings->settings->video_height.":flags=lanczos[x];[x][1:v]paletteuse\" ".$output_gif." ";
|
||||
ffmpegMsg("ffmpeg","Creating Gif ", 0,$cmd);
|
||||
|
||||
// $cmd_2='mogrify -delay '.$_POST['compress_gif_delay'].' -alpha remove '.$matte.' -depth '.$_POST['compress_gif_depth'].' '.$deconstruct.' '.$dither.' -colors '.$_POST['compress_gif_colors'].' -fuzz '.$_POST['compress_gif_fuzz'].'% '.$loop.' '.$optimize.' '.$path.' 2>&1';
|
||||
|
||||
$_POST['loop_gif']="true";
|
||||
$_POST['compress_gif_matte']="false";
|
||||
$_POST['compress_gif_deconstruct']="false";
|
||||
$_POST['compress_gif_dither']="true";
|
||||
$_POST['compress_gif_optimize']="true";
|
||||
|
||||
$loop=($_POST['loop_gif']=="false")?"-loop 1":"";
|
||||
$matte=($_POST['compress_gif_matte']=="true")?"-matte":"";
|
||||
$deconstruct=($_POST['compress_gif_deconstruct']=="true")?"-deconstruct":"";
|
||||
$dither=($_POST['compress_gif_dither']=="true")?"-dither none ":"";
|
||||
$optimize=($_POST['compress_gif_optimize']=="true")?"-layers optimize ":"";
|
||||
// $optimize=($_POST['compress_gif_optimize']=="true")?" -coalesce -layers RemoveDups ":"";
|
||||
|
||||
$_POST['compress_gif_depth']=16;
|
||||
$_POST['compress_gif_delay']=0;
|
||||
$_POST['compress_gif_fuzz']="4";
|
||||
$_POST['compress_gif_colors']="16";
|
||||
|
||||
$cmd_2='mogrify -delay '.$_POST['compress_gif_delay'].' -alpha remove '.$matte.' -depth '.$_POST['compress_gif_depth'].' '.$deconstruct.' '.$dither.' -colors '.$_POST['compress_gif_colors'].' -fuzz '.$_POST['compress_gif_fuzz'].'% '.$loop.' '.$optimize.' '.$output_gif.' 2>&1';
|
||||
$result=shell_exec($cmd_2);
|
||||
|
||||
|
||||
$output_options->edited_url= $output_options->options->directory.$output_file .$output_options->format . '.gif';
|
||||
$output_options->edited_name= $output_file .$output_options->format . '.gif';
|
||||
$output_options->edited_type = "image/gif";
|
||||
$output_options->edited_filetype = explode ( "/", "image/gif");
|
||||
insert_document($output_options->edited_url,"image/gif",$output_options->edited_name,"image","edited");
|
||||
$returnFiles = returnFiles ( $output_options, $output_file .$output_options->format . '.mp4', "video/mp4", '', null );
|
||||
return $returnFiles;
|
||||
}
|
||||
@@ -1,38 +0,0 @@
|
||||
// In file: src/server/api/routers/commandTemplate.ts
|
||||
import { z } from "zod";
|
||||
import { createTRPCRouter, publicProcedure } from "~/server/api/trpc";
|
||||
|
||||
export const commandTemplateRouter = createTRPCRouter({
|
||||
// Procedure to get all templates
|
||||
getAll: publicProcedure.query(({ ctx }) => {
|
||||
return ctx.db.commandTemplate.findMany({
|
||||
orderBy: { createdAt: "desc" },
|
||||
});
|
||||
}),
|
||||
|
||||
// Procedure to create a new template
|
||||
create: publicProcedure
|
||||
.input(z.object({
|
||||
name: z.string().min(1),
|
||||
type: z.string().min(1),
|
||||
payload: z.record(z.unknown()), // Accepts any JSON object
|
||||
}))
|
||||
.mutation(async ({ ctx, input }) => {
|
||||
return ctx.db.commandTemplate.create({
|
||||
data: {
|
||||
name: input.name,
|
||||
type: input.type,
|
||||
payload: input.payload,
|
||||
},
|
||||
});
|
||||
}),
|
||||
|
||||
// Procedure to delete a template
|
||||
delete: publicProcedure
|
||||
.input(z.object({ id: z.number() }))
|
||||
.mutation(async ({ ctx, input }) => {
|
||||
return ctx.db.commandTemplate.delete({
|
||||
where: { id: input.id },
|
||||
});
|
||||
}),
|
||||
});
|
||||
@@ -1,31 +0,0 @@
|
||||
// Two files from firmware/src/hw, reproduced verbatim: WiFiMgr.hpp (the
|
||||
// interface) and WiFiMgr.cpp (the implementation). The boundary is marked.
|
||||
// WiFiMgr.hpp — the interface the rest of the firmware depends on
|
||||
// WiFiMgr.hpp
|
||||
#pragma once
|
||||
#include <Arduino.h>
|
||||
|
||||
namespace WiFiMgr {
|
||||
bool connectHome(const char* ssid, const char* psk, uint32_t timeoutMs = 15000);
|
||||
void disconnect();
|
||||
}
|
||||
// WiFiMgr.cpp — the only translation unit that includes <WiFi.h>
|
||||
// WiFiMgr.cpp
|
||||
#include "WiFiMgr.hpp"
|
||||
#include <WiFi.h> // ESP32 WiFi
|
||||
|
||||
namespace WiFiMgr {
|
||||
bool connectHome(const char* ssid, const char* psk, uint32_t timeoutMs) {
|
||||
WiFi.mode(WIFI_STA);
|
||||
WiFi.begin(ssid, psk);
|
||||
uint32_t t0 = millis();
|
||||
while (WiFi.status() != WL_CONNECTED && millis() - t0 < timeoutMs) {
|
||||
delay(200);
|
||||
}
|
||||
return WiFi.status() == WL_CONNECTED;
|
||||
}
|
||||
void disconnect() {
|
||||
WiFi.disconnect(true, true);
|
||||
WiFi.mode(WIFI_OFF);
|
||||
}
|
||||
}
|
||||
@@ -1,33 +0,0 @@
|
||||
{ lib, stdenv, fetchurl }:
|
||||
|
||||
# tuios — terminal window manager (https://github.com/Gaurav-Gosain/tuios)
|
||||
# OVERRIDE: nixpkgs ships tuios 0.7.0; the multi-machine `hosts` UX needs 0.9+.
|
||||
# Official pure-Go prebuilt release binary (no runtime deps, runs bare on NixOS).
|
||||
# To return to nixpkgs packaging: use plain `tuios` in home.packages again and
|
||||
# delete this file (do this once nixpkgs ships tuios >= 0.10).
|
||||
stdenv.mkDerivation {
|
||||
pname = "tuios";
|
||||
version = "0.10.1";
|
||||
|
||||
src = fetchurl {
|
||||
url = "https://github.com/Gaurav-Gosain/tuios/releases/download/v0.10.1/tuios_0.10.1_Linux_x86_64.tar.gz";
|
||||
sha256 = "85a4bf339a9a6793e0b3d15abdccd9a96ca8d2ff76072007ec99be31f1a16a5f";
|
||||
};
|
||||
|
||||
# Tarball has no wrapping directory (tuios / LICENSE / README.md at root)
|
||||
sourceRoot = ".";
|
||||
|
||||
installPhase = ''
|
||||
runHook preInstall
|
||||
install -Dm755 tuios $out/bin/tuios
|
||||
runHook postInstall
|
||||
'';
|
||||
|
||||
meta = with lib; {
|
||||
description = "Terminal window manager — tiling panes, sessions that survive restarts (multi-machine daemon)";
|
||||
homepage = "https://github.com/Gaurav-Gosain/tuios";
|
||||
license = licenses.mit;
|
||||
mainProgram = "tuios";
|
||||
platforms = [ "x86_64-linux" ];
|
||||
};
|
||||
}
|
||||
@@ -1,109 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1110" height="600" viewBox="0 0 1110 600" role="img" aria-labelledby="topoTitle topoDesc" font-family="ui-sans-serif, system-ui, -apple-system, 'Segoe UI', sans-serif">
|
||||
<title id="topoTitle">Homelab topology: build host, runtime host and public edge</title>
|
||||
<desc id="topoDesc">Three hosts. The build host produces a release; the runtime host runs the site, the vector store, a workflow worker and the model gateway, all bound to loopback; the public edge terminates TLS and is the only host reachable from the internet. All three join one private mesh, and backups run in three tiers.</desc>
|
||||
|
||||
<style>
|
||||
/* On paper the diagram is ink on white, not a dark panel. Presentation
|
||||
attributes are the weakest source, so these rules win without editing
|
||||
the drawing. */
|
||||
@media print {
|
||||
rect { fill: #fff; }
|
||||
text { fill: #111; }
|
||||
text[fill="#6ee7c6"] { fill: #0b5c48; }
|
||||
text[fill="#8b98a8"] { fill: #444; }
|
||||
line, path { stroke: #555; }
|
||||
marker path { fill: #555; }
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||||
<path d="M0 0 10 5 0 10z" fill="#6ee7c6"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<g fill="#12161b" stroke="#33404d" stroke-width="1.5">
|
||||
<rect x="30" y="104" width="310" height="268" rx="10"/>
|
||||
<rect x="400" y="104" width="310" height="268" rx="10"/>
|
||||
<rect x="770" y="104" width="310" height="268" rx="10"/>
|
||||
</g>
|
||||
|
||||
<!-- internet -->
|
||||
<g>
|
||||
<rect x="810" y="24" width="230" height="46" rx="23" fill="#171d24" stroke="#33404d" stroke-width="1.5"/>
|
||||
<text x="925" y="52" text-anchor="middle" font-size="13" fill="#b3bcc8">public internet · https :443</text>
|
||||
<line x1="925" y1="70" x2="925" y2="100" stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)"/>
|
||||
</g>
|
||||
|
||||
<!-- host headers -->
|
||||
<g fill="#e6edf3" font-size="15" font-weight="600">
|
||||
<text x="54" y="136">Build and development</text>
|
||||
<text x="424" y="136">Runtime host</text>
|
||||
<text x="794" y="136">Public edge</text>
|
||||
</g>
|
||||
<g fill="#8b98a8" font-size="12" font-family="ui-monospace, SFMono-Regular, Menlo, monospace">
|
||||
<text x="54" y="156">.27 · 4-screen desktop</text>
|
||||
<text x="424" y="156">.13 · 15 GiB · memory-bound</text>
|
||||
<text x="794" y="156">.35 · Ubuntu · 2 vCPU</text>
|
||||
</g>
|
||||
|
||||
<!-- build host items -->
|
||||
<g fill="#b3bcc8" font-size="12.5">
|
||||
<text x="54" y="192">· repository working copy</text>
|
||||
<text x="54" y="219">· agent fleet, panes and approvals</text>
|
||||
<text x="54" y="246">· content, diagrams and exhibits</text>
|
||||
<text x="54" y="273">· release bundles and hashes</text>
|
||||
</g>
|
||||
|
||||
<!-- runtime: loopback trust boundary -->
|
||||
<rect x="416" y="172" width="278" height="150" rx="8" fill="none" stroke="#6ee7c6" stroke-width="1.2" stroke-dasharray="5 4"/>
|
||||
<text x="430" y="192" font-size="11" fill="#6ee7c6" font-family="ui-monospace, SFMono-Regular, Menlo, monospace">bound to 127.0.0.1</text>
|
||||
<g fill="#b3bcc8" font-size="12.5">
|
||||
<text x="430" y="216">· site server :8096</text>
|
||||
<text x="430" y="241">· Qdrant :6333 / :6334</text>
|
||||
<text x="430" y="266">· Prefect worker</text>
|
||||
<text x="430" y="291">· model gateway (shared)</text>
|
||||
</g>
|
||||
<text x="424" y="345" font-size="11.5" fill="#8b98a8">Caps equal swap; nothing listens publicly.</text>
|
||||
|
||||
<!-- edge host items -->
|
||||
<g fill="#b3bcc8" font-size="12.5">
|
||||
<text x="794" y="192">· Caddy terminates TLS</text>
|
||||
<text x="794" y="219">· proxies to the runtime host</text>
|
||||
<text x="794" y="246">· Gitea · Immich · Nextcloud</text>
|
||||
<text x="794" y="273">· one public listener, many sites</text>
|
||||
</g>
|
||||
|
||||
<!-- hand-off arrows -->
|
||||
<line x1="340" y1="230" x2="396" y2="230" stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)"/>
|
||||
<text x="368" y="218" text-anchor="middle" font-size="10.5" fill="#8b98a8">deploy</text>
|
||||
<line x1="710" y1="230" x2="766" y2="230" stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)"/>
|
||||
<text x="738" y="218" text-anchor="middle" font-size="10.5" fill="#8b98a8">proxy</text>
|
||||
|
||||
<!-- mesh band -->
|
||||
<rect x="30" y="410" width="1050" height="56" rx="10" fill="#12161b" stroke="#33404d" stroke-width="1.2" stroke-dasharray="6 5"/>
|
||||
<text x="555" y="436" text-anchor="middle" font-size="13" fill="#b3bcc8">private mesh · SSH on every host · remote access without an exposed port</text>
|
||||
<text x="555" y="454" text-anchor="middle" font-size="11" fill="#8b98a8">no service is reachable because it happened to bind to every interface</text>
|
||||
<g stroke="#33404d" stroke-width="1" stroke-dasharray="3 4">
|
||||
<line x1="185" y1="372" x2="185" y2="410"/>
|
||||
<line x1="555" y1="372" x2="555" y2="410"/>
|
||||
<line x1="925" y1="372" x2="925" y2="410"/>
|
||||
</g>
|
||||
|
||||
<!-- backup tiers -->
|
||||
<text x="30" y="506" font-size="13" font-weight="600" fill="#e6edf3">Three backup tiers — each covers a different failure</text>
|
||||
<g fill="#12161b" stroke="#33404d" stroke-width="1.2">
|
||||
<rect x="30" y="522" width="330" height="52" rx="8"/>
|
||||
<rect x="390" y="522" width="330" height="52" rx="8"/>
|
||||
<rect x="750" y="522" width="330" height="52" rx="8"/>
|
||||
</g>
|
||||
<g fill="#b3bcc8" font-size="12.5">
|
||||
<text x="50" y="545">local snapshots, on the machine</text>
|
||||
<text x="410" y="545">a copy on another machine</text>
|
||||
<text x="770" y="545">encrypted object storage, offsite</text>
|
||||
</g>
|
||||
<g fill="#8b98a8" font-size="11">
|
||||
<text x="50" y="562">restore tested, runbook beside it</text>
|
||||
<text x="410" y="562">over the private mesh</text>
|
||||
<text x="770" y="562">retention stated in the job</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 5.6 KiB |
@@ -1,119 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1120" height="580" viewBox="0 0 1120 580" role="img" aria-labelledby="ragTitle ragDesc" font-family="ui-sans-serif, system-ui, -apple-system, 'Segoe UI', sans-serif">
|
||||
<title id="ragTitle">Retrieval pipeline: ingest on top, four-node query path below</title>
|
||||
<desc id="ragDesc">Ingest reads content and estate notes, excludes secrets, chunks and tags each block, embeds through the model gateway and upserts into two Qdrant collections on loopback. The query path classifies the question, routes it to an area, retrieves with a payload filter, composes an answer from the retrieved context only, and verifies that every citation resolves to a page.</desc>
|
||||
|
||||
<style>
|
||||
/* On paper the diagram is ink on white, not a dark panel. Presentation
|
||||
attributes are the weakest source, so these rules win without editing
|
||||
the drawing. */
|
||||
@media print {
|
||||
rect { fill: #fff; }
|
||||
text { fill: #111; }
|
||||
text[fill="#6ee7c6"] { fill: #0b5c48; }
|
||||
text[fill="#8b98a8"] { fill: #444; }
|
||||
line, path { stroke: #555; }
|
||||
marker path { fill: #555; }
|
||||
}
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||||
<path d="M0 0 10 5 0 10z" fill="#6ee7c6"/>
|
||||
</marker>
|
||||
<marker id="arrowDim" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||||
<path d="M0 0 10 5 0 10z" fill="#8b98a8"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- ingest lane -->
|
||||
<text x="30" y="42" font-size="14" font-weight="600" fill="#e6edf3">Ingest — one source of truth, one index</text>
|
||||
|
||||
<g fill="#12161b" stroke="#33404d" stroke-width="1.4">
|
||||
<rect x="30" y="60" width="244" height="92" rx="9"/>
|
||||
<rect x="302" y="60" width="244" height="92" rx="9"/>
|
||||
<rect x="574" y="60" width="244" height="92" rx="9"/>
|
||||
<rect x="846" y="60" width="244" height="92" rx="9"/>
|
||||
</g>
|
||||
<g fill="#e6edf3" font-size="13.5" font-weight="600">
|
||||
<text x="48" y="86">Sources</text>
|
||||
<text x="320" y="86">Normalise</text>
|
||||
<text x="592" y="86">Chunk and tag</text>
|
||||
<text x="864" y="86">Embed and upsert</text>
|
||||
</g>
|
||||
<g fill="#b3bcc8" font-size="12">
|
||||
<text x="48" y="110">content/ Markdown</text>
|
||||
<text x="48" y="129">estate notes</text>
|
||||
<text x="48" y="148">platform guides</text>
|
||||
<text x="320" y="110">front-matter parsed</text>
|
||||
<text x="320" y="129">secrets and drafts</text>
|
||||
<text x="320" y="148">excluded by rule</text>
|
||||
<text x="592" y="110">blocks, not lines</text>
|
||||
<text x="592" y="129">area · level · source</text>
|
||||
<text x="592" y="148">title · tags</text>
|
||||
<text x="864" y="110">model gateway only</text>
|
||||
<text x="864" y="129">deterministic ids</text>
|
||||
<text x="864" y="148">batched, resumable</text>
|
||||
</g>
|
||||
<g stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)">
|
||||
<line x1="277" y1="106" x2="298" y2="106"/>
|
||||
<line x1="549" y1="106" x2="570" y2="106"/>
|
||||
<line x1="821" y1="106" x2="842" y2="106"/>
|
||||
</g>
|
||||
|
||||
<line x1="968" y1="152" x2="968" y2="184" stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)"/>
|
||||
|
||||
<rect x="30" y="186" width="1060" height="62" rx="9" fill="#12161b" stroke="#6ee7c6" stroke-width="1.2" stroke-dasharray="5 4"/>
|
||||
<text x="50" y="210" font-size="13" font-weight="600" fill="#e6edf3">Qdrant on 127.0.0.1</text>
|
||||
<text x="50" y="231" font-size="12" fill="#b3bcc8">resume_corpus · one point per chunk resume_graph · one point per node and edge HNSW and payloads on disk · within a 400 MiB cap</text>
|
||||
<text x="50" y="272" font-size="11.5" fill="#8b98a8">A sync run re-embeds only the documents whose content, model or chunker version changed.</text>
|
||||
|
||||
<!-- query lane -->
|
||||
<text x="30" y="316" font-size="14" font-weight="600" fill="#e6edf3">Query</text>
|
||||
<text x="108" y="316" font-size="12" fill="#8b98a8">— four graph nodes. It grows past four only with a written reason.</text>
|
||||
|
||||
<line x1="776" y1="248" x2="776" y2="352" stroke="#8b98a8" stroke-width="1.3" stroke-dasharray="4 4" marker-end="url(#arrowDim)"/>
|
||||
<text x="788" y="330" font-size="11" fill="#8b98a8">payload-filtered search</text>
|
||||
|
||||
<g fill="#12161b" stroke="#33404d" stroke-width="1.4">
|
||||
<rect x="30" y="360" width="196" height="112" rx="9"/>
|
||||
<rect x="246" y="360" width="196" height="112" rx="9"/>
|
||||
<rect x="462" y="360" width="196" height="112" rx="9"/>
|
||||
<rect x="678" y="360" width="196" height="112" rx="9"/>
|
||||
<rect x="894" y="360" width="196" height="112" rx="9"/>
|
||||
</g>
|
||||
<g font-size="10.5" fill="#6ee7c6" font-family="ui-monospace, SFMono-Regular, Menlo, monospace">
|
||||
<text x="46" y="382">CLASSIFY</text>
|
||||
<text x="262" y="382">ROUTE</text>
|
||||
<text x="478" y="382">RETRIEVE</text>
|
||||
<text x="694" y="382">COMPOSE</text>
|
||||
<text x="910" y="382">VERIFY</text>
|
||||
</g>
|
||||
<g fill="#e6edf3" font-size="13" font-weight="600">
|
||||
<text x="46" y="406">Read the question</text>
|
||||
<text x="262" y="406">Pick the areas</text>
|
||||
<text x="478" y="406">Find the evidence</text>
|
||||
<text x="694" y="406">Answer from it</text>
|
||||
<text x="910" y="406">Check citations</text>
|
||||
</g>
|
||||
<g fill="#b3bcc8" font-size="11.5">
|
||||
<text x="46" y="430">intent · injection</text>
|
||||
<text x="46" y="448">off-topic · PII</text>
|
||||
<text x="262" y="430">which pages can</text>
|
||||
<text x="262" y="448">answer this</text>
|
||||
<text x="478" y="430">vector + keyword</text>
|
||||
<text x="478" y="448">ranked, top five</text>
|
||||
<text x="694" y="430">context only</text>
|
||||
<text x="694" y="448">no outside claims</text>
|
||||
<text x="910" y="430">every claim maps</text>
|
||||
<text x="910" y="448">to a real page</text>
|
||||
</g>
|
||||
<g stroke="#6ee7c6" stroke-width="1.5" marker-end="url(#arrow)">
|
||||
<line x1="227" y1="416" x2="242" y2="416"/>
|
||||
<line x1="443" y1="416" x2="458" y2="416"/>
|
||||
<line x1="659" y1="416" x2="674" y2="416"/>
|
||||
<line x1="875" y1="416" x2="890" y2="416"/>
|
||||
</g>
|
||||
|
||||
<text x="30" y="500" font-size="12" fill="#b3bcc8">A refusal is a real outcome: an off-topic question, an injection attempt or a request over the cost cap stops at the first node with zero model spend.</text>
|
||||
<text x="30" y="526" font-size="11.5" fill="#8b98a8">Ingest runs today. The query path is the Phase 5 design; until it ships, every page is pre-rendered and needs no model.</text>
|
||||
<text x="30" y="550" font-size="11.5" fill="#8b98a8">The public AI has no tools, no filesystem, no network fetch and no memory between visitors — architectural, not a prompt instruction.</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 6.5 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 29 KiB |
@@ -8,12 +8,14 @@ confidence: running
|
||||
period: "2020–"
|
||||
tech: [ESP32, LoRa, Zigbee, MQTT, Home Assistant, Snapcast, Pi, PIR, mmWave]
|
||||
metrics: [{label: "edge nodes", value: 20}, {label: "lighting zones", value: 9}, {label: "automations", value: 20}, {label: "hand-built Zigbee routers", value: 2}]
|
||||
artefacts: [{type: doc, label: "Edge architecture — power · radio · deep sleep", url: "content/edge/l3/edge-architecture.md"}, {type: doc, label: "LoRa irrigation under solar power", url: "content/edge/l3/edge-lora-sprinkler.md"}, {type: doc, label: "Hand-built Zigbee routers and lighting", url: "content/edge/l3/edge-zigbee.md"}, {type: doc, label: "Distributed audio — eight endpoints with ducking", url: "content/edge/l3/edge-snapcast.md"}, {type: doc, label: "Front door — vision · voice · and a rebuild", url: "content/edge/l3/edge-front-door.md"}, {type: doc, label: "Sensors and actuation", url: "content/edge/l3/edge-sensors.md"}, {type: doc, label: "The failures, kept in the record", url: "content/edge/l3/edge-failures.md"}, {type: repo, label: "Embedded C++ — the Wi-Fi manager interface and implementation", url: "content/assets/code/05-embedded-cpp-interface.hpp"}]
|
||||
artefacts: [{type: doc, label: "Edge architecture — power · radio · deep sleep", url: "content/edge/l3/edge-architecture.md"}, {type: doc, label: "LoRa irrigation under solar power", url: "content/edge/l3/edge-lora-sprinkler.md"}, {type: doc, label: "Hand-built Zigbee routers and lighting", url: "content/edge/l3/edge-zigbee.md"}, {type: doc, label: "Distributed audio — eight endpoints with ducking", url: "content/edge/l3/edge-snapcast.md"}, {type: doc, label: "Front door — vision · voice · and a rebuild", url: "content/edge/l3/edge-front-door.md"}, {type: doc, label: "Sensors and actuation", url: "content/edge/l3/edge-sensors.md"}, {type: doc, label: "The failures, kept in the record", url: "content/edge/l3/edge-failures.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, iot, esp32, lora, zigbee, audio]
|
||||
---
|
||||
|
||||
# Constrained devices · custom radio · and a record of what broke
|
||||
|
||||
**Capability statement.** Devices that must run for months on a small battery, talk over radio that was chosen for range rather than convenience, and fail in ways that are visible. Twenty or more nodes, nine zones, twenty automations, and a voice layer on top of all of it.
|
||||
|
||||
## What is actually run
|
||||
@@ -30,10 +32,6 @@ tags: [edge, iot, esp32, lora, zigbee, audio]
|
||||
|
||||
**Cameras and vision.** The front door takes an image, logs it, has the image described, and speaks the result into the house before deciding what to do next. RFID tags on boxes and on trees resolve to records — inventory with prices for the boxes, care notes for the trees — written by an automation rather than typed by hand.
|
||||
|
||||
## Why this exhibit is good code
|
||||
|
||||
The header declares the interface and includes only `Arduino.h`, so a caller compiles without pulling the ESP32 Wi-Fi stack into its translation unit. Only the implementation file knows about station mode, polling and millisecond timeouts, and it returns a boolean so the caller decides what a failed join means. The two files are shown together because the point is the split, not either file alone.
|
||||
|
||||
## Evidence
|
||||
|
||||
| Claim | Evidence | Artefact |
|
||||
@@ -52,4 +50,16 @@ The header declares the interface and includes only `Arduino.h`, so a caller com
|
||||
- **Some things are still manual.** The garden automation is partly manual — some watering and lighting is scheduled, some is a person turning something on. The page says which is which instead of implying full coverage.
|
||||
- **The failures cost money and are kept.** A lizard destroyed a feeder built for it. The garden cameras were postponed with the parts already bought. The doorbell is being rebuilt rather than declared finished. Those are the most honest evidence of production experience on this site.
|
||||
|
||||
Individual nodes carry their own label, and the ones being rebuilt or postponed are labelled that way on their pages.
|
||||
## Drill-downs
|
||||
|
||||
- [Edge architecture — power · radio · deep sleep](l3/edge-architecture.md) — level 3
|
||||
- [LoRa irrigation under solar power](l3/edge-lora-sprinkler.md) — level 3
|
||||
- [Hand-built Zigbee routers and lighting](l3/edge-zigbee.md) — level 3
|
||||
- [Distributed audio — eight endpoints with ducking](l3/edge-snapcast.md) — level 3
|
||||
- [Front door — vision · voice · and a rebuild](l3/edge-front-door.md) — level 3
|
||||
- [Sensors and actuation](l3/edge-sensors.md) — level 3
|
||||
- [The failures, kept in the record](l3/edge-failures.md) — level 3
|
||||
|
||||
## Period and confidence
|
||||
|
||||
`period: 2020–`, anchored on the homelab year in the evidence notes. Confidence is `running` for the fleet as a whole; individual nodes carry their own label, and the ones being rebuilt or postponed are labelled that way on their pages.
|
||||
|
||||
@@ -3,14 +3,23 @@ id: edge-overview
|
||||
area: edge
|
||||
level: 1
|
||||
title: "Edge & Devices"
|
||||
claim: "Sam builds and maintains a twenty-node edge fleet — constrained devices on solar power and deep sleep, with custom radio and sensors — that reports into a single automation layer."
|
||||
claim: "Builds and maintains a twenty-node edge fleet — constrained devices · custom radio · solar power · deep sleep · and sensors — reporting into a single automation layer."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [ESP32, LoRa, Zigbee, MQTT, Home Assistant, Snapcast, Raspberry Pi]
|
||||
metrics: [{label: "edge nodes", value: 20}, {label: "lighting zones", value: 9}, {label: "automations", value: 20}]
|
||||
artefacts: [{type: doc, label: "Edge architecture — power · radio · deep sleep", url: "content/edge/l3/edge-architecture.md"}, {type: doc, label: "The failures, kept in the record", url: "content/edge/l3/edge-failures.md"}, {type: doc, label: "Distributed audio — eight endpoints with ducking", url: "content/edge/l3/edge-snapcast.md"}, {type: doc, label: "Vehicle tracker on LTE Cat-M", url: "content/edge/l3/edge-car-tracker.md"}]
|
||||
artefacts: [{type: doc, label: "Edge architecture — power · radio · deep sleep", url: "content/edge/l3/edge-architecture.md"}, {type: doc, label: "The failures, kept in the record", url: "content/edge/l3/edge-failures.md"}, {type: doc, label: "Distributed audio — eight endpoints with ducking", url: "content/edge/l3/edge-snapcast.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [edge, iot, esp32, lora, zigbee]
|
||||
---
|
||||
The devices are outdoors on solar and asleep most of their lives, and they have to keep working when nobody is looking; what failed is written down. The architecture, the radio and the failures are on the [fleet page](edge-fleet.md).
|
||||
|
||||
# Edge & Devices
|
||||
|
||||
**Claim.** Twenty or more nodes — ten ESP32 devices, two hand-built Zigbee routers and eight Pi audio endpoints — across nine lighting zones and twenty automations, all reporting into one automation layer.
|
||||
|
||||
**Status.** Running. Some of it is finished, some is being rebuilt, and at least one thing was destroyed by the animal it was built for. All three states are shown.
|
||||
|
||||
**Why it is credible.** These devices are outdoors, on solar, in deep sleep, and they must keep working when nobody is looking. The failures on this page cost real time and are documented as failures.
|
||||
|
||||
**Start here.** [The capability page](edge-fleet.md) covers the architecture, the radio, the audio and what broke.
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [edge, architecture, power, artefact]
|
||||
---
|
||||
|
||||
One annotated diagram of a representative node and the radio path lands in Phase 7.
|
||||
# Artefact — Edge architecture · power · radio · deep sleep
|
||||
|
||||
**Status: stub.** One annotated diagram of a representative node and the radio path lands in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
---
|
||||
id: edge-car-tracker
|
||||
area: edge
|
||||
level: 3
|
||||
title: "Artefact — Vehicle tracker on LTE Cat-M"
|
||||
claim: "A tracker that sleeps between reports · wakes on a schedule or when the car moves · and reports when it leaves a boundary set from the web interface."
|
||||
confidence: in-progress
|
||||
period: "2024–"
|
||||
tech: [ESP32, LTE Cat-M, NB-IoT, SIM7080G, GPS, LiFePO4]
|
||||
metrics: [{label: "battery packs", value: 2}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
tags: [edge, tracker, lte, low-power, artefact]
|
||||
---
|
||||
|
||||
The build is on the bench: two LiFePO batteries, a buck converter, an ESP32 with GPS, and a
|
||||
cellular module for the uplink. The car keeps the packs charged while it is running.
|
||||
|
||||
|
||||
|
||||
## What it does
|
||||
|
||||
Two behaviours, chosen by the settings rather than hard-coded. When the car is parked the
|
||||
ESP32 sits in deep sleep, wakes on a timer, checks its own configuration, and reports only
|
||||
what the settings ask for. When the car is moving it stays awake and reports on a shorter
|
||||
interval. Either way the interval and the boundary can be changed from the web interface,
|
||||
and a change can force an immediate update rather than waiting for the next wake.
|
||||
|
||||
The geofence is the part that matters. A boundary is drawn from the web interface, and the
|
||||
device is responsible for deciding whether it has left it — which is why the settings have
|
||||
to travel to the device rather than live only in the server.
|
||||
|
||||
## Why LTE Cat-M rather than WiFi
|
||||
|
||||
The tracker lives in a car. WiFi coverage is a driveway, not a road. Cat-M and NB-IoT were
|
||||
chosen because they are designed for exactly this shape of traffic — small payloads, long
|
||||
sleeps, coverage beyond what 4G handsets need — and because the modules are cheap enough to
|
||||
leave installed.
|
||||
|
||||
## What it proves
|
||||
|
||||
A complete path rather than a demo: constrained hardware, a power budget that dictates the
|
||||
schedule, a device that holds enough state to act on its own, and a web application that
|
||||
configures it and shows where it has been.
|
||||
|
||||
## Still open
|
||||
|
||||
Field testing across the full battery cycle. The sleep interval that is right for a parked
|
||||
car and the interval that is right for a moving one are guesses until the discharge curve
|
||||
has been measured over a week rather than a day.
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [edge, failures, artefact]
|
||||
---
|
||||
|
||||
Two items are being rebuilt, one is postponed. The page is not finished because the work is not finished.
|
||||
# Artefact — The failures · kept in the record
|
||||
|
||||
**Status: in progress.** Two items are being rebuilt, one is postponed. The page is not finished because the work is not finished.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [edge, doorbell, vision, voice, artefact]
|
||||
---
|
||||
|
||||
The device was built and is being rebuilt. The page states this rather than describing it as finished.
|
||||
# Artefact — Front door · vision · voice · and a rebuild
|
||||
|
||||
**Status: in progress.** The device was built and is being rebuilt. The page states this rather than describing it as finished.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [edge, lora, solar, irrigation, artefact]
|
||||
---
|
||||
|
||||
The bill of materials and the wake-on-radio change land in Phase 7.
|
||||
# Artefact — LoRa irrigation under solar power
|
||||
|
||||
**Status: stub.** The bill of materials and the wake-on-radio change land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [edge, sensors, actuation, artefact]
|
||||
---
|
||||
|
||||
A sensor and actuator table lands in Phase 7.
|
||||
# Artefact — Sensors and actuation
|
||||
|
||||
**Status: stub.** A sensor and actuator table lands in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [edge, audio, snapcast, artefact]
|
||||
---
|
||||
|
||||
The topology diagram and a short recording of ducking land in Phase 7.
|
||||
# Artefact — Distributed audio · eight endpoints with ducking
|
||||
|
||||
**Status: stub.** The topology diagram and a short recording of ducking land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [edge, zigbee, lighting, artefact]
|
||||
---
|
||||
|
||||
The router build and a zone map land in Phase 7.
|
||||
# Artefact — Hand-built Zigbee routers and lighting
|
||||
|
||||
**Status: stub.** The router build and a zone map land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,6 +14,8 @@ last_verified: 2026-10-26
|
||||
tags: [history, agencies, brands, exhibits]
|
||||
---
|
||||
|
||||
# Fifteen years of delivery for agencies and their clients
|
||||
|
||||
**Capability statement.** Range under deadline. The work in this period ran from car classifieds and online betting to a bank's pitch piece, a broadcaster's on-demand banners and a publisher's education platform — usually with a date attached that could not move.
|
||||
|
||||
## The eras, briefly
|
||||
@@ -50,4 +52,12 @@ The point of the exhibit set is that the tooling changed repeatedly and the deli
|
||||
- **No client has been named beyond a brand.** Brands are named because they were public campaigns. Individuals, internal contacts and unpublished commercial detail are excluded.
|
||||
- **The exhibits are excerpts.** Around forty lines each, chosen to show judgement — structure, error handling, an exit plan — not to demonstrate volume.
|
||||
|
||||
This work is finished, and it is presented in the past tense throughout.
|
||||
## Drill-downs
|
||||
|
||||
- [The client wall](l3/history-client-wall.md) — level 3
|
||||
- [Six code exhibits across five eras](l3/history-code-exhibits.md) — level 3
|
||||
- [Products — the commercial era that followed](../products/products-business.md) — level 2
|
||||
|
||||
## Period and confidence
|
||||
|
||||
`period: 1999–2014`. Confidence is `historic`: this is work that is finished, and it is presented in the past tense throughout.
|
||||
|
||||
@@ -3,7 +3,7 @@ id: history-overview
|
||||
area: history
|
||||
level: 1
|
||||
title: "History"
|
||||
claim: "Sam delivered interactive work for twenty-two agencies and twenty-three brands between 1999 and 2014 — the commercial apprenticeship that produced the range."
|
||||
claim: "Delivered interactive work for twenty-two agencies and twenty-three brands between 1999 and 2014 — the commercial apprenticeship that produced the range."
|
||||
confidence: historic
|
||||
period: "1999–2014"
|
||||
tech: [ActionScript, PHP, JavaScript, MySQL, XML, React, Python, Go, HTMX]
|
||||
@@ -13,4 +13,13 @@ public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [history, agencies, brands, career]
|
||||
---
|
||||
The languages changed six times across the period — ActionScript, PHP, JavaScript, React, Python, and now Go with HTMX — and the exhibits are chosen to show that rather than to show a decade. The era-by-era record is on the [engagement record](history-engagements.md).
|
||||
|
||||
# History
|
||||
|
||||
**Claim.** Fifteen years of agency and product work — national campaigns, bank and broadcaster platforms, betting front-ends, and a lot of rescue missions — for twenty-two agencies and twenty-three brands.
|
||||
|
||||
**Status.** Historic. This is the 1999–2014 era, presented as what it was: the commercial apprenticeship that produced the range.
|
||||
|
||||
**What it is not.** It is not a Flash page. The languages changed six times across this period — ActionScript, PHP, JavaScript, React, Python, and now Go with HTMX — and the exhibits are chosen to show that rather than to show a decade.
|
||||
|
||||
**Start here.** [The engagement record](history-engagements.md) explains the eras, the work and what each exhibit demonstrates.
|
||||
|
||||
@@ -14,66 +14,17 @@ last_verified: 2026-10-26
|
||||
tags: [history, clients, artefact]
|
||||
---
|
||||
|
||||
The names are final. The treatment is deliberate: one typeface, one size, one grid, and no logo collage — see *Why there are no logos here* below.
|
||||
# Artefact — The client wall
|
||||
|
||||
**Status: stub for the visual treatment.** The names are final; the layout, logo set and typography land in Phase 7 (the logo set is explicitly a Phase 7 decision).
|
||||
|
||||
## The agencies
|
||||
|
||||
<div class="client-wall" role="list" aria-label="Agencies">
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Clemenger</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Agency Republic</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Brand Union</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Jackpot Joy</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Bryce Ford</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">RMG Connect</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Sporting Index</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Adventis Digital</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Trinity Street</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Inferno</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">SAS Design</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Ogilvy</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Wunderman</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Illumina</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Barkers</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Mook</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Pearsons</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Penguin</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Draft London</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Behan Consulting</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">MCM Entertainment</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Carsales</span></div>
|
||||
</div>
|
||||
Clemenger · Agency Republic · Brand Union · Jackpot Joy · Bryce Ford · RMG Connect · Sporting Index · Adventis Digital · Trinity Street · Inferno · SAS Design · Ogilvy · Wunderman · Illumina · Barkers · Mook · Pearsons · Penguin · Draft London · Behan Consulting · MCM Entertainment · Carsales
|
||||
|
||||
## The brands
|
||||
|
||||
<div class="client-wall" role="list" aria-label="Brands">
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Carlton Draught</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Diageo</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">KPMG</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">McDonald's</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Accenture</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">BBC National Archive</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Bank of America</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">War of the Worlds</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Madonna (Hard Candy)</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">PlayStation</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">HSBC</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">DHL</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Reuters</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Volvic</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">AutoCAD</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Channel 4</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Pearsons</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Shell</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Stella Artois</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Baileys Irish Cream</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Parkgate Homes</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Carlton Dry</span></div>
|
||||
<div class="client-tile" role="listitem"><span class="client-name">Cottees Cordial</span></div>
|
||||
</div>
|
||||
|
||||
## Why there are no logos here
|
||||
|
||||
A wall of mixed marks — some full colour, some monochrome, some missing — reads as broken, and the eight marks that exist for these names are not the eight the work is best remembered by. One typeface, one size and one grid says the same thing more clearly, and it sits beside the technical pages without competing with them. A mark is added only when it is the client's official mark and Sam has approved its use; none has been invented.
|
||||
Carlton Draught · Diageo · KPMG · McDonald's · Accenture · BBC National Archive · Bank of America · War of the Worlds · Madonna (Hard Candy) · PlayStation · HSBC · DHL · Reuters · Volvic · AutoCAD · Channel 4 · Pearsons · Shell · Stella Artois · Baileys Irish Cream · Parkgate Homes · Carlton Dry · Cottees Cordial
|
||||
|
||||
## Corrections carried into this list
|
||||
|
||||
@@ -87,4 +38,4 @@ The summary material said twenty-three agencies. The recovered, verified list ho
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Delivered as a body of names and descriptions, and presented as a typographic wall from Phase 7. Names and one-line descriptions are approved as recovered and corrected.
|
||||
Phase 7 for the visual treatment. Names and one-line descriptions are approved as recovered and corrected.
|
||||
|
||||
@@ -8,34 +8,28 @@ confidence: historic
|
||||
period: "2006–2026"
|
||||
tech: [ActionScript, JavaScript, PHP, TypeScript, C++, Nix]
|
||||
metrics: [{label: "code exhibits", value: 6}, {label: "technology eras spanned", value: 5}]
|
||||
artefacts: [{type: repo, label: "ActionScript 3 — the Siren class", url: "content/assets/code/01-actionscript-class.as"}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
last_verified: 2026-10-26
|
||||
tags: [history, code, exhibits, artefact]
|
||||
---
|
||||
|
||||
**Status: published.** This page shows the ActionScript class and indexes the other five, each of which is exhibited on the page where it belongs.
|
||||
# Artefact — Six code exhibits across five eras
|
||||
|
||||
**Status: stub.** The excerpts are selected from source that exists on disk; each one lands in Phase 7 with two sentences on why it is good code.
|
||||
|
||||
## The six
|
||||
|
||||
| # | Era | Exhibit | Where it is shown |
|
||||
| # | Era | Source | What the excerpt shows |
|
||||
|---|---|---|---|
|
||||
| 1 | Object-oriented ActionScript | `01-actionscript-class.as` — class, imports, encapsulation | this page |
|
||||
| 2 | Hand-written JavaScript engine | `02-js-easing-core.js` — easing functions and the timeline loop | [the site builder](../../products/l3/products-websitebuilder.md) |
|
||||
| 3 | PHP media pipeline | `03-php-encode-op.php` — one GIF encode operation and its FFmpeg commands | [the site builder](../../products/l3/products-websitebuilder.md) |
|
||||
| 4 | TypeScript | `04-typescript-service-layer.ts` — a typed tRPC boundary | [the estate](../../platform/platform-estate.md) |
|
||||
| 5 | Embedded C++ | `05-embedded-cpp-interface.hpp` — interface split from implementation | [the edge fleet](../../edge/edge-fleet.md) |
|
||||
| 6 | Declarative infrastructure | `06-nix-derivation.nix` — pinned version, verified hash, exit condition | [the Nix package](../../platform/l3/platform-tuios-package.md) |
|
||||
| 1 | Object-oriented ActionScript | the brief-case builder and the national beer campaign | class design, imports, encapsulation |
|
||||
| 2 | Hand-written JavaScript engine | the text-effects engine from the site builder (about 2 200 lines) | easing and a timeline core |
|
||||
| 3 | PHP media pipeline | the media encoder from the site builder (about 3 400 lines) | one encode operation and the FFmpeg piping around it |
|
||||
| 4 | TypeScript | the vehicle-tracking application's schema and typed service layer | types, interfaces, a router |
|
||||
| 5 | Embedded C++ | the Wi-Fi manager interface and implementation | interface versus implementation in firmware |
|
||||
| 6 | Declarative infrastructure | the 25-line Nix package definition | pinned version, verified hash, written exit condition |
|
||||
|
||||
Total is 243 lines across twenty years, which is the point: each excerpt is chosen to show judgement rather than volume.
|
||||
|
||||
## Why this exhibit is good code
|
||||
|
||||
Every field is private and explicitly typed, and the public surface is two methods, so a caller cannot push the object into a state it does not understand. The constructor takes what it needs as parameters instead of reaching for it, which is why the same class can drive every siren on the screen without knowing where any of them are.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
The excerpt is verbatim, including a latent typo (`this.high` rather than `this.height`). It is left as found: the excerpt is evidence, not a portfolio piece.
|
||||
Total is roughly 225 lines across twenty years, which is the point: each excerpt is chosen to show judgement rather than volume.
|
||||
|
||||
## What it proves
|
||||
|
||||
@@ -43,4 +37,4 @@ That the stack changed six times and the discipline did not. Exhibit 6 is the le
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Published. The six files are in this repository under `content/assets/code/`; the archive path is withheld. The other five are linked above from the pages that carry their claims.
|
||||
Phase 7. Source paths are recorded in the planning notes; nothing is published from a path that has not been read at capture time.
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [platform, backups, restore, artefact]
|
||||
---
|
||||
|
||||
Redacted job definitions and the restore runbook land in Phase 7. Passphrase locations and remote paths are removed before publication.
|
||||
# Artefact — Backups and restore runbook
|
||||
|
||||
**Status: stub.** Redacted job definitions and the restore runbook land in Phase 7. Passphrase locations and remote paths are removed before publication.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [platform, docker, artefact]
|
||||
---
|
||||
|
||||
A digest of the inventory — grouped by purpose, with addresses removed — lands in Phase 7.
|
||||
# Artefact — Container inventory
|
||||
|
||||
**Status: stub.** A digest of the inventory — grouped by purpose, with addresses removed — lands in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [platform, networking, security, artefact]
|
||||
---
|
||||
|
||||
The diagram below shows three hosts and one trust boundary; internal addresses, hostnames and public DNS names are generalised, not published.
|
||||
# Artefact — Network layout and remote access
|
||||
|
||||
**Status: stub.** The topology diagram lands in Phase 7 with addresses and hostnames redacted.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
@@ -24,12 +26,6 @@ The layout: hosts on a private mesh for remote access, SSH on every machine, a p
|
||||
|
||||
Security posture chosen rather than inherited: no service is reachable because it happened to bind to all interfaces, and there is exactly one place where the outside world meets the estate.
|
||||
|
||||
## The layout
|
||||
|
||||
|
||||
|
||||
Read it left to right: work is built on one machine, runs on a memory-bound machine with every service bound to loopback, and is published by a single edge that terminates TLS. The dashed boundary is the point of the whole arrangement — crossing it is a deliberate change to the edge configuration, not a side effect of starting a container. Below both, the mesh carries remote access and the second backup tier, so neither depends on a port being open.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Delivered. Internal addressing, hostnames and public DNS names are excluded from publication by the redaction pass.
|
||||
Phase 7. Internal addressing, hostnames and public DNS names are excluded from publication by the redaction pass.
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [platform, nixos, artefact]
|
||||
---
|
||||
|
||||
A redacted excerpt of the configuration tree lands in Phase 7. No hostnames, addresses or keys are published.
|
||||
# Artefact — Declarative multi-host configuration
|
||||
|
||||
**Status: stub.** A redacted excerpt of the configuration tree lands in Phase 7. No hostnames, addresses or keys are published.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [platform, signage, hdmi-cec, artefact]
|
||||
---
|
||||
|
||||
The CEC command set and a layout diagram land in Phase 7.
|
||||
# Artefact — Signage infrastructure · screens as a managed fleet
|
||||
|
||||
**Status: stub.** The CEC command set and a layout diagram land in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -8,13 +8,15 @@ confidence: running
|
||||
period: "2026–"
|
||||
tech: [Nix, NixOS]
|
||||
metrics: [{label: "pinned version", value: "0.10.1"}, {label: "nixpkgs version it overrides", value: "0.7.0"}]
|
||||
artefacts: [{type: repo, label: "Nix derivation — tuios 0.10.1 override with a removal condition", url: "content/assets/code/06-nix-derivation.nix"}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
last_verified: 2026-10-26
|
||||
tags: [platform, nix, packaging, artefact]
|
||||
---
|
||||
|
||||
**Status: published.** The whole file is shown verbatim.
|
||||
# Artefact — A hand-written Nix package with an exit plan
|
||||
|
||||
**Status: stub.** The whole file is short enough to publish verbatim; it lands in Phase 7.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
@@ -24,10 +26,6 @@ A package definition for the agent operations console: `fetchurl` against a rele
|
||||
|
||||
More than knowing Nix exists. The signals are that someone noticed a version gap, fixed it locally instead of waiting, pinned it so the fleet is reproducible, and wrote down when to remove their own patch. Knowing when to delete your own override is the senior part.
|
||||
|
||||
## Why this exhibit is good code
|
||||
|
||||
The override records why it exists and the exact condition under which it should be deleted, which is the part of a local patch that usually goes missing. Pinning a released binary by `fetchurl` with a verified hash keeps the result reproducible without waiting for nixpkgs.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Published. The file is in this repository at `content/assets/code/06-nix-derivation.nix`; the infrastructure repository remains the source. Source fact: content spine section 4.2.
|
||||
Phase 7. The file is in the infrastructure repository; source fact: content spine section 4.2.
|
||||
|
||||
@@ -8,12 +8,14 @@ confidence: running
|
||||
period: "2020–"
|
||||
tech: [NixOS, Proxmox, Docker, Caddy, Tailscale, Borg, Kopia, AWS S3, HDMI-CEC]
|
||||
metrics: [{label: "containers in the estate", value: 50}, {label: "hosts", value: 4}, {label: "backup tiers", value: "Borg · Kopia · AWS"}, {label: "public listeners", value: 1}]
|
||||
artefacts: [{type: doc, label: "Declarative multi-host configuration", url: "content/platform/l3/platform-nixos.md"}, {type: doc, label: "A hand-written Nix package with an exit plan", url: "content/platform/l3/platform-tuios-package.md"}, {type: doc, label: "Container inventory", url: "content/platform/l3/platform-containers.md"}, {type: doc, label: "Backups and restore runbook", url: "content/platform/l3/platform-backups.md"}, {type: doc, label: "Network layout and remote access", url: "content/platform/l3/platform-network.md"}, {type: doc, label: "Signage infrastructure — screens as a managed fleet", url: "content/platform/l3/platform-signage.md"}, {type: repo, label: "TypeScript — a typed tRPC service boundary", url: "content/assets/code/04-typescript-service-layer.ts"}]
|
||||
artefacts: [{type: doc, label: "Declarative multi-host configuration", url: "content/platform/l3/platform-nixos.md"}, {type: doc, label: "A hand-written Nix package with an exit plan", url: "content/platform/l3/platform-tuios-package.md"}, {type: doc, label: "Container inventory", url: "content/platform/l3/platform-containers.md"}, {type: doc, label: "Backups and restore runbook", url: "content/platform/l3/platform-backups.md"}, {type: doc, label: "Network layout and remote access", url: "content/platform/l3/platform-network.md"}, {type: doc, label: "Signage infrastructure — screens as a managed fleet", url: "content/platform/l3/platform-signage.md"}]
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
last_verified: 2026-10-26
|
||||
tags: [platform, devops, nixos, docker, backups, networking]
|
||||
---
|
||||
|
||||
# A small estate run to a documented standard
|
||||
|
||||
**Capability statement.** Four hosts, fifty or more containers, configuration in version control instead of in a shell history, and one place where traffic enters. The estate is deliberately small because it is run rather than demonstrated.
|
||||
|
||||
## What is actually run
|
||||
@@ -32,10 +34,6 @@ tags: [platform, devops, nixos, docker, backups, networking]
|
||||
|
||||
**Signage infrastructure.** The display work is treated as a fleet to be managed: HDMI-CEC control for power, rotation, mirroring and multi-screen layout, so screens are driven by the platform rather than by a person with a remote control.
|
||||
|
||||
## Why this exhibit is good code
|
||||
|
||||
The Prisma schema defines the stored types, and the tRPC router validates input with Zod at the boundary, so the data layer receives a shape it can trust rather than one it hopes for. Each procedure is a thin call on the database context, which keeps the router a description of the API rather than a second copy of the queries.
|
||||
|
||||
## Evidence
|
||||
|
||||
| Claim | Evidence | Artefact |
|
||||
@@ -53,3 +51,16 @@ The Prisma schema defines the stored types, and the tRPC router validates input
|
||||
- **Self-hosting is a choice with a cost.** Monitoring, backups, ingress and the AI gateway are all operated here, which means the failure modes are also operated here. The mitigation is documentation and measurement, not optimism.
|
||||
- **Rejected on purpose.** A long list of tools was evaluated and refused — Airflow for RAM, self-hosted Langfuse for six containers it could not afford, LiteLLM because it would duplicate the existing gateway, and a pile of documentation platforms. "Alternatives considered" is kept because the reasoning is the senior artifact.
|
||||
- **Not everything is automated.** Some operations are still a documented command run by a person. The runbook exists precisely because the automations do not cover everything.
|
||||
|
||||
## Drill-downs
|
||||
|
||||
- [Declarative multi-host configuration](l3/platform-nixos.md) — level 3
|
||||
- [A hand-written Nix package with an exit plan](l3/platform-tuios-package.md) — level 3
|
||||
- [Container inventory](l3/platform-containers.md) — level 3
|
||||
- [Backups and restore runbook](l3/platform-backups.md) — level 3
|
||||
- [Network layout and remote access](l3/platform-network.md) — level 3
|
||||
- [Signage infrastructure — screens as a managed fleet](l3/platform-signage.md) — level 3
|
||||
|
||||
## Period and confidence
|
||||
|
||||
`period: 2020–`, anchored on the homelab year in the evidence notes. Confidence is `running`: this is the current estate, verified 2026-10-26.
|
||||
|
||||
@@ -3,7 +3,7 @@ id: platform-overview
|
||||
area: platform
|
||||
level: 1
|
||||
title: "Platform & Infrastructure"
|
||||
claim: "Sam runs a multi-machine NixOS and Proxmox estate sized to the hardware rather than the budget: fifty-plus containers, declarative configuration, encrypted backups with a written restore runbook, and public ingress."
|
||||
claim: "Runs a multi-machine NixOS and Proxmox estate — fifty-plus containers · declarative configuration · encrypted backups with a written restore runbook · and public ingress — sized to the hardware rather than the budget."
|
||||
confidence: running
|
||||
period: "2020–"
|
||||
tech: [NixOS, Proxmox, Docker, Caddy, Tailscale, Borg, Kopia, AWS S3]
|
||||
@@ -13,4 +13,13 @@ public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [platform, devops, nixos, docker, backups]
|
||||
---
|
||||
The runtime host is memory-bound rather than CPU-bound, so every service carries a memory cap and is bound to loopback. What runs, where, and what was rejected is on the [estate page](platform-estate.md).
|
||||
|
||||
# Platform & Infrastructure
|
||||
|
||||
**Claim.** Fifty or more containers on a four-host estate, configuration declared rather than hand-applied, encrypted backups on three machines with a restore runbook beside them, and one public listener at the edge.
|
||||
|
||||
**Status.** Running. Figures trace to the approved metric set and the infrastructure reference; verified 2026-10-26.
|
||||
|
||||
**The constraint that shaped it.** The runtime host is memory-bound rather than CPU-bound. Every service in this estate carries a memory cap and is bound to loopback, which is why the platform reads as engineering under constraint instead of hardware shopping.
|
||||
|
||||
**Start here.** [The capability page](platform-estate.md) covers what runs, where, and what was rejected.
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [products, signage, artefact]
|
||||
---
|
||||
|
||||
The product shipped and customers still run it. The rebuild on the current stack is not finished and is labelled as unfinished everywhere it is mentioned.
|
||||
# Artefact — Commercial signage on commodity hardware
|
||||
|
||||
**Status: delivered, with a rebuild in progress.** The product shipped and customers still run it. The rebuild on the current stack is not finished and is labelled as unfinished everywhere it is mentioned.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -8,12 +8,14 @@ confidence: delivered
|
||||
period: "2014–"
|
||||
tech: [PHP, JavaScript, three.js, FFmpeg, ImageMagick, MySQL]
|
||||
metrics: [{label: "media editors in the product", value: "drawing · image · video · text animation · transitions · 3D import"}]
|
||||
artefacts: [{type: repo, label: "Text-effects engine — easing functions and the timeline loop", url: "content/assets/code/02-js-easing-core.js"}, {type: repo, label: "Media encoder — one GIF encode operation and its FFmpeg commands", url: "content/assets/code/03-php-encode-op.php"}]
|
||||
artefacts: []
|
||||
public: true
|
||||
last_verified: 2026-10-28
|
||||
last_verified: 2026-10-26
|
||||
tags: [products, cms, media, artefact]
|
||||
---
|
||||
|
||||
# Artefact — The site builder behind the signage
|
||||
|
||||
**Status: delivered, retained.** Retired as a general-purpose product; kept as the content engine for the signage.
|
||||
|
||||
## What the artefact is
|
||||
@@ -24,16 +26,10 @@ The templated builder: pick a template, build the site, publish it — with edit
|
||||
|
||||
Two things at once — depth in media pipelines (FFmpeg, ImageMagick, PDF and office formats, three.js) and the discipline of shipping a general product that strangers could operate without help.
|
||||
|
||||
## Why these exhibits are good code
|
||||
|
||||
The animation engine evaluates a named easing function to give each letter its delay, so the spacing of an animation follows the same curve as its motion instead of a guessed linear stagger. Both directions build from one transform object, which is what stops the two halves of the same effect drifting apart over years of edits.
|
||||
|
||||
The media pipeline checks the real MIME type before it builds a command, so a wrong input returns a message instead of a half-written file. The GIF encode takes two passes — a palette from the clip, then frames mapped onto it — which is the difference between a GIF that holds its colour and one that does not.
|
||||
|
||||
## Limits stated on the page
|
||||
|
||||
The PHP is 2014-era procedural code: global state, interpolated shell commands, and long functions. The exhibit is here for the operational judgement, not as a model of structure. The surface was too large for one person to support indefinitely; the plan recorded here is to strip it back to signage plus the content engine and stop selling the rest, which is a decision about focus rather than a claim of failure.
|
||||
The surface was too large for one person to support indefinitely. The plan recorded here is to strip it back to signage plus the content engine and stop selling the rest, which is a decision about focus rather than a claim of failure.
|
||||
|
||||
## When it is delivered
|
||||
|
||||
Phase 7. Both code exhibits are published at `content/assets/code/`; a screen recording of the editors is still to come. Source: résumé (2014–2020 freelance) and evidence notes section 3.
|
||||
Phase 7 for a screen recording of the editors. Source: résumé (2014–2020 freelance) and evidence notes section 3.
|
||||
|
||||
@@ -14,7 +14,9 @@ last_verified: 2026-10-26
|
||||
tags: [products, where-woof, artefact]
|
||||
---
|
||||
|
||||
Deployed is true, commercial is not — that is the exact wording the site uses, deliberately.
|
||||
# Artefact — Where-Woof · deployed and pre-market
|
||||
|
||||
**Status: running, pre-market.** This is the exact wording the site uses, deliberately: deployed is true, commercial is not.
|
||||
|
||||
## What the artefact is
|
||||
|
||||
|
||||
@@ -14,6 +14,8 @@ last_verified: 2026-10-26
|
||||
tags: [products, business, signage, where-woof, positioning]
|
||||
---
|
||||
|
||||
# Software sold and operated — with the status stated
|
||||
|
||||
**Capability statement.** Building software for a client and building software as a product are different jobs. This page is the second one: a market that was overpriced, a substitute built on hardware that costs a fraction of the incumbent systems, customers who adopted it, and the operational cost of keeping it running.
|
||||
|
||||
## The commercial story
|
||||
@@ -53,6 +55,13 @@ The third product is different in kind: Where-Woof is a QR and RFID tag system f
|
||||
- **One product is pre-revenue by design.** Where-Woof is deployed before it is sold. Describing it as commercial would be a lie that a careful reader would find in a minute.
|
||||
- **Support is the hidden cost.** Customers running the product for years is the strongest evidence of value and the strongest argument for keeping the surface small.
|
||||
|
||||
Signage infrastructure is also platform work: [screens as a managed fleet](../platform/l3/platform-signage.md).
|
||||
## Drill-downs
|
||||
|
||||
Each product carries its own status, and the two caveats — a rebuild in progress and a launch not yet made — are in the table above rather than in a footnote.
|
||||
- [Commercial signage on commodity hardware](l3/products-signage.md) — level 3
|
||||
- [The site builder behind it](l3/products-websitebuilder.md) — level 3
|
||||
- [Where-Woof — deployed and pre-market](l3/products-where-woof.md) — level 3
|
||||
- [Signage infrastructure — screens as a managed fleet](../platform/l3/platform-signage.md) — level 3
|
||||
|
||||
## Period and confidence
|
||||
|
||||
`period: 2014–`, anchored on the first freelance product work in the résumé. Confidence is `running`: the products exist and customers still use one of them. Individual products carry their own status, and the two caveats — a rebuild in progress and a launch not yet made — are in the table above rather than in a footnote.
|
||||
|
||||
@@ -3,7 +3,7 @@ id: products-overview
|
||||
area: products
|
||||
level: 1
|
||||
title: "Products"
|
||||
claim: "Sam identified an overpriced market and shipped a cheaper substitute on commodity hardware: twelve adoptions over time and three businesses still running it."
|
||||
claim: "Identified an overpriced market and shipped a cheaper substitute on commodity hardware — twelve adoptions over time and three businesses still running it."
|
||||
confidence: running
|
||||
period: "2014–"
|
||||
tech: [PHP, JavaScript, three.js, FFmpeg, ImageMagick, HDMI-CEC, Raspberry Pi, ESP32]
|
||||
@@ -13,4 +13,13 @@ public: true
|
||||
last_verified: 2026-10-26
|
||||
tags: [products, signage, business, where-woof]
|
||||
---
|
||||
Building, selling, deploying and still supporting software for other businesses is a different discipline from writing it, and the caveats on these three products are stated rather than hidden. The commercial story is on the [products page](products-business.md).
|
||||
|
||||
# Products
|
||||
|
||||
**Claim.** A digital signage product built as deliberate disruption to thousand-dollar wall systems — three businesses still run it — plus the site builder that produced the marketing sites and a lost-property tag system that is deployed but not yet sold.
|
||||
|
||||
**Status.** Running. The counts are the corrected ones: three businesses active now, twelve adoptions over time, and Where-Woof deployed without a market launch.
|
||||
|
||||
**Why it is on an engineering CV.** Selling, deploying and still supporting software for other businesses is a different discipline from writing it. Two of the three products carry honest caveats that are stated rather than hidden: the signage platform is being rebuilt, and Where-Woof has not gone to market.
|
||||
|
||||
**Start here.** [The commercial story](products-business.md) covers what each product is, what state it is in, and what it cost to run.
|
||||
|
||||
94
corpus/areas.py
Normal file
94
corpus/areas.py
Normal file
@@ -0,0 +1,94 @@
|
||||
"""Which resume area a harvested document feeds.
|
||||
|
||||
The payload `area` is what lets the query path filter *before* it searches, so a
|
||||
harvested file has to be classified once, deterministically, and never promoted
|
||||
to make a page read better.
|
||||
|
||||
The classification is keyword evidence, in this order: the path, the repository
|
||||
name, the description and topics, then the languages. The result is constrained
|
||||
to the platform guide's `fits` list: a platform cannot feed an area it does not
|
||||
declare. When nothing matches, the first `fits` entry wins, which is why every
|
||||
guide lists its strongest area first.
|
||||
|
||||
This is a heuristic and it is labelled as one. It is not a claim about Sam's
|
||||
career; the curated `content/` tree carries the claims.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from .payloads import AREAS
|
||||
|
||||
# Order matters: the first area with a hit wins, so the more specific evidence
|
||||
# (an ESP32 repository is edge, not platform) is checked before the general.
|
||||
_RULES = (
|
||||
("edge", (
|
||||
"esp32", "esp8266", "arduino", "lora", "lorawan", "mqtt", "sensor",
|
||||
"zigbee", "z-wave", "relay", "gpio", "pir", "modbus", "iot",
|
||||
"firmware", "breadboard", "raspberry pi", "rpi", "camera", "cam",
|
||||
"temperature", "watering", "fish feeder", "driveway", "light",
|
||||
"snapcast", "mopidy", "audio", "voice bridge",
|
||||
)),
|
||||
("ai", (
|
||||
"ai", "llm", "agent", "agentic", "rag", "embedding", "vector",
|
||||
"model", "ml", "prompt", "graphify", "omniroute", "paperclip",
|
||||
"langgraph", "prefect", "whisper", "piper", "chat", "memory",
|
||||
"knowledge graph", "vision", "triage",
|
||||
)),
|
||||
("platform", (
|
||||
"nixos", "nix", "docker", "compose", "kubernetes", "k8s", "deploy",
|
||||
"deployment", "backup", "borg", "kopia", "restic", "network", "dns",
|
||||
"email", "caddy", "proxmox", "postgres", "terraform", "ansible",
|
||||
"linux", "systemd", "homelab", "server", "infrastructure", "git",
|
||||
"gitea", "monitor", "logging", "security", "firewall", "vpn",
|
||||
"tailscale", "storage", "garage", "s3",
|
||||
)),
|
||||
("products", (
|
||||
"product", "app", "web", "dashboard", "react", "laravel", "flutter",
|
||||
"kotlin", "calendar", "photo", "kontra", "store", "resume", "site",
|
||||
"gps", "tracker", "api", "frontend", "ui", "ux",
|
||||
)),
|
||||
("history", (
|
||||
"history", "agency", "client", "archive", "old", "legacy", "2019",
|
||||
"2018", "2020", "2021", "career", "resume copy",
|
||||
)),
|
||||
("about", ("about", "profile", "bio", "personal", "cv")),
|
||||
)
|
||||
|
||||
_WORD_SPLIT = re.compile(r"[^0-9a-z]+")
|
||||
|
||||
|
||||
def area_for(evidence: str, fits, fallback: str = None) -> str:
|
||||
"""Classify one document from its path, name, description and topics."""
|
||||
haystack = " " + _WORD_SPLIT.sub(" ", (evidence or "").lower()) + " "
|
||||
allowed = [area for area in (fits or AREAS) if area in AREAS]
|
||||
if not allowed:
|
||||
allowed = list(AREAS)
|
||||
for area, needles in _RULES:
|
||||
if area not in allowed:
|
||||
continue
|
||||
for needle in needles:
|
||||
if f" {needle} " in haystack or f" {needle}s " in haystack:
|
||||
return area
|
||||
if fallback in allowed:
|
||||
return fallback
|
||||
return allowed[0]
|
||||
|
||||
|
||||
def area_for_repo(repo: dict, fits) -> str:
|
||||
topics = " ".join(repo.get("topics") or [])
|
||||
evidence = " ".join(
|
||||
str(part)
|
||||
for part in (
|
||||
repo.get("full_name", ""),
|
||||
repo.get("name", ""),
|
||||
repo.get("description", ""),
|
||||
topics,
|
||||
)
|
||||
)
|
||||
return area_for(evidence, fits, fallback=(fits or [None])[0])
|
||||
|
||||
|
||||
def area_for_note(path: str, fits) -> str:
|
||||
return area_for(path, fits, fallback=(fits or [None])[0])
|
||||
@@ -285,3 +285,124 @@ def verify_no_split(chunks: list, regions: list) -> None:
|
||||
f"chunk {chunk.index} splits protected lines {first}-{last} "
|
||||
f"(chunk covers {chunk.line_start}-{chunk.line_end})"
|
||||
)
|
||||
|
||||
|
||||
# -- code -----------------------------------------------------------------
|
||||
|
||||
|
||||
def _code_depth(line: str, depth: int) -> int:
|
||||
"""Approximate bracket depth for one line, ignoring strings and comments.
|
||||
|
||||
This is a heuristic lexer, not a parser: it is here to answer one question —
|
||||
"is this a safe line to break after?" — and it errs towards *not* breaking.
|
||||
A line with an ambiguous quote or an unterminated bracket simply keeps the
|
||||
chunk open, which can only produce a larger chunk, never a broken one.
|
||||
"""
|
||||
index = 0
|
||||
quote = None
|
||||
length = len(line)
|
||||
while index < length:
|
||||
char = line[index]
|
||||
if quote:
|
||||
if char == "\\":
|
||||
index += 2
|
||||
continue
|
||||
if line.startswith(quote, index):
|
||||
index += len(quote)
|
||||
quote = None
|
||||
continue
|
||||
index += 1
|
||||
continue
|
||||
if line.startswith("//", index) or line.startswith("#", index) and line.lstrip().startswith("#"):
|
||||
break
|
||||
if char in "\"'`":
|
||||
triple = char * 3
|
||||
if line.startswith(triple, index):
|
||||
quote = triple
|
||||
index += 3
|
||||
continue
|
||||
quote = char
|
||||
index += 1
|
||||
continue
|
||||
if char in "([{":
|
||||
depth += 1
|
||||
elif char in ")]}":
|
||||
depth = max(0, depth - 1)
|
||||
index += 1
|
||||
return depth
|
||||
|
||||
|
||||
def code_segments(text: str) -> list:
|
||||
"""Split code into whole top-level segments. A function is never cut."""
|
||||
lines = text.split("\n")
|
||||
segments = []
|
||||
current = []
|
||||
depth = 0
|
||||
start = 1
|
||||
for number, line in enumerate(lines, start=1):
|
||||
if not current:
|
||||
start = number
|
||||
depth = _code_depth(line, depth)
|
||||
current.append(line)
|
||||
# A blank line at bracket depth zero is the only automatic boundary.
|
||||
if depth == 0 and not line.strip() and len(current) > 1:
|
||||
segments.append((start, number, "\n".join(current)))
|
||||
current = []
|
||||
if current:
|
||||
segments.append((start, len(lines), "\n".join(current)))
|
||||
return [(start, end, body) for start, end, body in segments if body.strip()]
|
||||
|
||||
|
||||
def chunk_code(text: str) -> list:
|
||||
"""Chunk a source file without cutting a function, class or block.
|
||||
|
||||
Segments are whole top-level units. A unit larger than the ceiling is kept
|
||||
as one oversize chunk and flagged, exactly like an oversized Markdown fence:
|
||||
cutting it would break the rule this function exists to keep.
|
||||
|
||||
Overlap repeats the previous unit, never part of one, so an overlapping
|
||||
chunk is still syntactically whole.
|
||||
"""
|
||||
segments = code_segments(text)
|
||||
if not segments:
|
||||
raise ChunkError("source file produced no segments (empty?)")
|
||||
|
||||
chunks = []
|
||||
current = [] # list of (start, end, body)
|
||||
tokens = 0
|
||||
|
||||
def flush():
|
||||
nonlocal current, tokens
|
||||
if not current:
|
||||
return
|
||||
body = "\n".join(segment[2] for segment in current).strip("\n")
|
||||
chunks.append(
|
||||
Chunk(
|
||||
index=len(chunks),
|
||||
text=body,
|
||||
tokens=estimate_tokens(body),
|
||||
line_start=current[0][0],
|
||||
line_end=current[-1][1],
|
||||
blocks=tuple(
|
||||
Block("code", segment[2], estimate_tokens(segment[2]), segment[0], segment[1])
|
||||
for segment in current
|
||||
),
|
||||
oversize=len(current) == 1 and estimate_tokens(current[0][2]) > TOKEN_MAX,
|
||||
)
|
||||
)
|
||||
current = []
|
||||
tokens = 0
|
||||
|
||||
for segment in segments:
|
||||
segment_tokens = estimate_tokens(segment[2])
|
||||
if current and tokens + segment_tokens > TOKEN_MAX:
|
||||
previous = current[-1]
|
||||
flush()
|
||||
budget = int(round(TOKEN_TARGET * OVERLAP_RATIO))
|
||||
if estimate_tokens(previous[2]) <= budget:
|
||||
current = [previous]
|
||||
tokens = estimate_tokens(previous[2])
|
||||
current.append(segment)
|
||||
tokens += segment_tokens
|
||||
flush()
|
||||
return chunks
|
||||
|
||||
122
corpus/cli.py
122
corpus/cli.py
@@ -28,9 +28,10 @@ import os
|
||||
import sys
|
||||
|
||||
from . import config as config_module
|
||||
from . import freshness, pipeline, store as store_module
|
||||
from . import freshness, guides, pipeline, store as store_module
|
||||
from .config import Config, ConfigError
|
||||
from .qdrant import Qdrant, QdrantError
|
||||
from .sources import graph as graph_module
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
@@ -49,6 +50,11 @@ def main(argv=None) -> int:
|
||||
"dim": args.dim,
|
||||
"db_path": args.db,
|
||||
"data_dir": args.data,
|
||||
"offline": True if args.offline else None,
|
||||
"platforms": tuple(part.strip() for part in args.platforms.split(",") if part.strip())
|
||||
if args.platforms
|
||||
else None,
|
||||
"graph_path": args.graph_path or None,
|
||||
}
|
||||
try:
|
||||
cfg = Config.from_env(**overrides)
|
||||
@@ -68,6 +74,10 @@ def main(argv=None) -> int:
|
||||
"hybrid": _cmd_hybrid,
|
||||
"measure": _cmd_measure,
|
||||
"info": _cmd_info,
|
||||
"graph": _cmd_graph,
|
||||
"graph-neighbours": _cmd_graph_neighbours,
|
||||
"scan": _cmd_scan,
|
||||
"sources": _cmd_sources,
|
||||
}[args.command]
|
||||
return handler(cfg, args, log)
|
||||
except (pipeline.PipelineError, ConfigError) as exc:
|
||||
@@ -92,6 +102,9 @@ def _build_parser() -> argparse.ArgumentParser:
|
||||
parser.add_argument("--db", help="SQLite corpus database path")
|
||||
parser.add_argument("--data", help="corpus data directory")
|
||||
parser.add_argument("--quiet", action="store_true", help="progress on stderr only becomes silent")
|
||||
parser.add_argument("--offline", action="store_true", help="read content/ only; do not harvest the estate")
|
||||
parser.add_argument("--platforms", help="comma-separated platforms to harvest (default: all enabled)")
|
||||
parser.add_argument("--graph-path", help="read graph.json from this path instead of Gitea")
|
||||
|
||||
sub = parser.add_subparsers(dest="command")
|
||||
|
||||
@@ -128,6 +141,23 @@ def _build_parser() -> argparse.ArgumentParser:
|
||||
measure.add_argument("--keep", action="store_true", help="keep the measurement collection")
|
||||
measure.add_argument("--json", action="store_true")
|
||||
|
||||
graph = sub.add_parser("graph", help="query graph nodes and edges by keyword or vector")
|
||||
graph.add_argument("--query", required=True)
|
||||
graph.add_argument("--area")
|
||||
graph.add_argument("--limit", type=int, default=10)
|
||||
graph.add_argument("--json", action="store_true")
|
||||
|
||||
neighbours = sub.add_parser("graph-neighbours", help="edges attached to a graph node id")
|
||||
neighbours.add_argument("--node", required=True)
|
||||
neighbours.add_argument("--limit", type=int, default=50)
|
||||
neighbours.add_argument("--json", action="store_true")
|
||||
|
||||
scan = sub.add_parser("scan", help="search the finished pool for credential-shaped content")
|
||||
scan.add_argument("--json", action="store_true")
|
||||
|
||||
sources = sub.add_parser("sources", help="show the platform guides and the last harvest")
|
||||
sources.add_argument("--json", action="store_true")
|
||||
|
||||
sub.add_parser("info", help="print the resolved configuration")
|
||||
return parser
|
||||
|
||||
@@ -235,6 +265,88 @@ def _cmd_measure(cfg, args, log) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_graph(cfg, args, log) -> int:
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
report = graph_module.query(cfg, store, args.query, area=args.area, limit=args.limit)
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, sort_keys=True, default=str))
|
||||
return 0
|
||||
print(f"graph query {args.query!r} area={args.area!r}")
|
||||
print("")
|
||||
print(f"keyword ({len(report['keyword'])} row(s)):")
|
||||
for row in report["keyword"]:
|
||||
kind = row.get("kind")
|
||||
print(f" [{kind}] {row.get('label')} -> {row.get('source_note')} {row.get('origin_url')}")
|
||||
print("")
|
||||
print(f"vector ({len(report['vector'])} row(s)):")
|
||||
for row in report["vector"]:
|
||||
print(f" {row['score']:.3f} [{row['kind']}] {row['label']} -> {row['source_note']}")
|
||||
if report.get("vector_note"):
|
||||
print(f" note: {report['vector_note']}")
|
||||
return 0 if report["ok"] else 1
|
||||
|
||||
|
||||
def _cmd_graph_neighbours(cfg, args, log) -> int:
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
node = store.graph_node(args.node)
|
||||
edges = store.graph_neighbours(args.node, limit=args.limit)
|
||||
if args.json:
|
||||
print(json.dumps({"node": node, "edges": edges}, indent=2, sort_keys=True, default=str))
|
||||
return 0
|
||||
if not node:
|
||||
print(f"no graph node with id {args.node!r}")
|
||||
return 1
|
||||
print(f"{node.get('label')} ({node.get('node_type')})")
|
||||
print(f" note: {node.get('source_note')}")
|
||||
print(f" origin: {node.get('origin_url')}")
|
||||
for edge in edges:
|
||||
print(
|
||||
f" {edge.get('source_label')} --{edge.get('relation')}--> {edge.get('target_label')}"
|
||||
f" [{edge.get('source_note')}]"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_scan(cfg, args, log) -> int:
|
||||
report = pipeline.scan_pool(cfg)
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, sort_keys=True))
|
||||
else:
|
||||
print(
|
||||
f"scanned {report['chunks']} chunk(s): "
|
||||
+ ("clean" if report["ok"] else f"{len(report['hits'])} hit(s)")
|
||||
)
|
||||
for hit in report["hits"]:
|
||||
print(f" {hit}")
|
||||
return 0 if report["ok"] else 1
|
||||
|
||||
|
||||
def _cmd_sources(cfg, args, log) -> int:
|
||||
loaded = guides.load_guides(cfg.root)
|
||||
public = guides.build_public_manifest(loaded)
|
||||
last = {}
|
||||
if os.path.exists(cfg.db_path):
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
last = store.get_meta("sources") or {}
|
||||
if args.json:
|
||||
print(json.dumps({"guides": public, "last_harvest": last}, indent=2, sort_keys=True))
|
||||
return 0
|
||||
for row in public["platforms"]:
|
||||
state = "enabled" if row["enabled"] else "disabled"
|
||||
print(f"{row['name']} ({state}) — {row['what']}")
|
||||
print(f" base: {row['base']} fits: {', '.join(row['fits'])} read: {', '.join(row['read'])}")
|
||||
print(f" include: {json.dumps(row['include'])}")
|
||||
print(f" exclude: {json.dumps(row['exclude'])}")
|
||||
seen = last.get(row["name"]) or {}
|
||||
if seen:
|
||||
print(
|
||||
f" last harvest: {seen.get('documents', 0)} document(s), "
|
||||
f"newest {seen.get('newest') or '—'}"
|
||||
)
|
||||
print("")
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_info(cfg, args, log) -> int:
|
||||
print(cfg.describe())
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
@@ -248,6 +360,14 @@ def _cmd_info(cfg, args, log) -> int:
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
print(f"store: {store.chunk_count()} chunk(s) across {store.source_count()} source(s)")
|
||||
print(f"contract: {json.dumps(store.get_meta('collection'), sort_keys=True)}")
|
||||
print(f"graph: {json.dumps(store.graph_counts(), sort_keys=True)}")
|
||||
sources = store.get_meta("sources") or {}
|
||||
for name in sorted(sources):
|
||||
row = sources[name]
|
||||
print(
|
||||
f"source {name}: {row.get('documents', 0)} document(s), "
|
||||
f"newest {row.get('newest') or '—'}"
|
||||
)
|
||||
else:
|
||||
print(f"store: {cfg.db_path} does not exist")
|
||||
return 0
|
||||
|
||||
@@ -13,7 +13,7 @@ from dataclasses import dataclass
|
||||
|
||||
# Bumped whenever chunk boundaries or payload shape change. A stored corpus
|
||||
# carrying a different version is stale by definition and gets rebuilt.
|
||||
CHUNKER_VERSION = "blocks-v1"
|
||||
CHUNKER_VERSION = "blocks-v2-code-v1"
|
||||
|
||||
# Provider defaults. `local-hashing` is a deterministic, offline provider used
|
||||
# by the tests, by `make corpus-test`, and by any rebuild on a host with no
|
||||
@@ -29,6 +29,14 @@ DEFAULT_QDRANT_URL = "http://127.0.0.1:6333"
|
||||
DEFAULT_COLLECTION = "resume_corpus"
|
||||
DEFAULT_PROVIDER = "local-hashing"
|
||||
DEFAULT_OMNIROUTE_URL = "https://omniroute.lab.audasmedia.com.au"
|
||||
DEFAULT_GITEA_URL = "https://gitea.lab.audasmedia.com.au"
|
||||
DEFAULT_PLATFORMS = ("gitea", "obsidian", "graphify")
|
||||
DEFAULT_GRAPH_COLLECTION = "resume_graph"
|
||||
|
||||
# A source that has not changed in this window is reported by the audit. The
|
||||
# estate is active; silence is evidence that ingestion stopped, not that the
|
||||
# work stopped.
|
||||
DEFAULT_SOURCE_SILENCE_DAYS = 30
|
||||
|
||||
# Freshness horizon: a document whose `last_verified` is older than this is
|
||||
# reported stale by the nightly audit. docs/phase-4-notes.md section 5.
|
||||
@@ -60,6 +68,13 @@ class Config:
|
||||
max_age_days: int
|
||||
warn_days: int
|
||||
apprise_url: str
|
||||
gitea_url: str = DEFAULT_GITEA_URL
|
||||
gitea_token: str = ""
|
||||
platforms: tuple = DEFAULT_PLATFORMS
|
||||
offline: bool = False
|
||||
graph_path: str = ""
|
||||
graph_collection: str = DEFAULT_GRAPH_COLLECTION
|
||||
source_silence_days: int = DEFAULT_SOURCE_SILENCE_DAYS
|
||||
vectors_on_disk: bool = False
|
||||
|
||||
# -- construction -----------------------------------------------------
|
||||
@@ -102,6 +117,19 @@ class Config:
|
||||
),
|
||||
warn_days=int(os.environ.get("RESUME_WARN_DAYS") or DEFAULT_WARN_DAYS),
|
||||
apprise_url=os.environ.get("RESUME_APPRISE_URL") or "",
|
||||
gitea_url=(os.environ.get("GITEA_URL") or os.environ.get("RESUME_GITEA_URL") or DEFAULT_GITEA_URL).rstrip("/"),
|
||||
gitea_token=os.environ.get("GITEA_TOKEN") or "",
|
||||
platforms=tuple(
|
||||
part.strip()
|
||||
for part in (os.environ.get("RESUME_PLATFORMS") or ",".join(DEFAULT_PLATFORMS)).split(",")
|
||||
if part.strip()
|
||||
),
|
||||
offline=(os.environ.get("RESUME_OFFLINE") or "").lower() in ("1", "true", "yes"),
|
||||
graph_path=os.environ.get("RESUME_GRAPH_PATH") or "",
|
||||
graph_collection=os.environ.get("RESUME_GRAPH_COLLECTION") or DEFAULT_GRAPH_COLLECTION,
|
||||
source_silence_days=int(
|
||||
os.environ.get("RESUME_SOURCE_SILENCE_DAYS") or DEFAULT_SOURCE_SILENCE_DAYS
|
||||
),
|
||||
vectors_on_disk=(os.environ.get("RESUME_QDRANT_VECTORS_ON_DISK") or "").lower()
|
||||
in ("1", "true", "yes"),
|
||||
)
|
||||
@@ -124,6 +152,21 @@ class Config:
|
||||
)
|
||||
if not os.path.isdir(self.content_dir):
|
||||
raise ConfigError(f"content directory not found: {self.content_dir}")
|
||||
unknown = [name for name in self.platforms if name not in ("gitea", "obsidian", "graphify")]
|
||||
if unknown:
|
||||
raise ConfigError(
|
||||
f"RESUME_PLATFORMS contains unknown platform(s) {unknown}; "
|
||||
"expected gitea, obsidian, graphify"
|
||||
)
|
||||
if not self.offline and self.gitea_token == "":
|
||||
# Only a hard error when a platform that needs Gitea is selected; the
|
||||
# token itself is never printed or stored.
|
||||
if set(self.platforms) & {"gitea", "obsidian", "graphify"}:
|
||||
raise ConfigError(
|
||||
"GITEA_TOKEN is not set. Harvesting needs read access to Gitea "
|
||||
"(~/.config/environment.d/10-secrets.conf), or set RESUME_OFFLINE=1 "
|
||||
"to rebuild only the curated tier."
|
||||
)
|
||||
|
||||
# -- presentation -----------------------------------------------------
|
||||
|
||||
@@ -140,7 +183,10 @@ class Config:
|
||||
f"batch={self.batch_size}\n"
|
||||
f"api_url={self.api_url} api_key={key}\n"
|
||||
f"freshness=max_age_days:{self.max_age_days} warn_days:{self.warn_days}\n"
|
||||
f"qdrant vectors_on_disk={self.vectors_on_disk}\n"
|
||||
f"estate=platforms:{','.join(self.platforms)} offline:{self.offline} "
|
||||
f"gitea:{self.gitea_url} token:{'set' if self.gitea_token else 'unset'} "
|
||||
f"graph_path:{self.graph_path or 'from gitea'} silence_days:{self.source_silence_days}\n"
|
||||
f"qdrant vectors_on_disk={self.vectors_on_disk} graph_collection={self.graph_collection}\n"
|
||||
f"chunker={CHUNKER_VERSION}"
|
||||
)
|
||||
|
||||
|
||||
256
corpus/exclusions.py
Normal file
256
corpus/exclusions.py
Normal file
@@ -0,0 +1,256 @@
|
||||
"""What must never reach the pool.
|
||||
|
||||
AGENTS.md §4.4 and `docs/phase-4-corpus.md` §6 make this a hard gate, not a
|
||||
suggestion: node_modules, virtualenvs, git internals, `.trash`, `.obsidian`, the
|
||||
SilverBullet auth file, credential-shaped files and binaries are refused. The
|
||||
platform guide excludes are *additional*; they cannot widen this list.
|
||||
|
||||
Two different outcomes, because they are different things:
|
||||
|
||||
* **reject** — the file is not read at all. The reason is recorded and reported
|
||||
so a run says *why* it skipped something.
|
||||
* **image** — a media file is recorded as an image node reference (path only,
|
||||
never bytes, never extracted text). Phase 4 §1.
|
||||
|
||||
Credential-shaped *content* is a third thing: a file with a valid path can still
|
||||
contain a private key or an API token. `scan_text` rejects the whole document
|
||||
rather than redacting part of it, and the same scan runs over the finished pool
|
||||
so "no secret in the pool" is verified by search, not asserted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from . import globs
|
||||
|
||||
# Directory names that are never descended into. Checked per path component, so
|
||||
# a file deep inside one is refused by the same rule as its parent.
|
||||
EXCLUDED_DIRS = frozenset(
|
||||
{
|
||||
"node_modules",
|
||||
".venv",
|
||||
"venv",
|
||||
".git",
|
||||
".trash",
|
||||
".obsidian",
|
||||
".pi",
|
||||
"__pycache__",
|
||||
".mypy_cache",
|
||||
".pytest_cache",
|
||||
".ruff_cache",
|
||||
"graphify-out",
|
||||
"graphify-out-temp",
|
||||
".next",
|
||||
".cache",
|
||||
"site-packages",
|
||||
}
|
||||
)
|
||||
|
||||
# Basenames (and simple suffix patterns) that are credential-shaped by name.
|
||||
CREDENTIAL_FILES = (
|
||||
".silverbullet.auth.json",
|
||||
".env",
|
||||
".env.*",
|
||||
"*.env",
|
||||
"*.env.*",
|
||||
".netrc",
|
||||
".git-credentials",
|
||||
".npmrc",
|
||||
".pypirc",
|
||||
"credentials",
|
||||
"credentials.*",
|
||||
"secrets.*",
|
||||
"secret.*",
|
||||
"auth.json",
|
||||
"*.pem",
|
||||
"*.key",
|
||||
"*.p12",
|
||||
"*.pfx",
|
||||
"*.jks",
|
||||
"*.keystore",
|
||||
"*.kdbx",
|
||||
"*.age",
|
||||
"id_rsa",
|
||||
"id_ed25519",
|
||||
"id_ecdsa",
|
||||
"*.lock",
|
||||
"*.lockb",
|
||||
)
|
||||
|
||||
# Media is never text. It becomes an image node reference and nothing else.
|
||||
IMAGE_EXTENSIONS = frozenset(
|
||||
{"png", "jpg", "jpeg", "gif", "webp", "bmp", "tif", "tiff", "heic", "avif", "ico"}
|
||||
)
|
||||
VIDEO_EXTENSIONS = frozenset({"mp4", "mov", "mkv", "webm", "avi", "m4v", "mpg", "mpeg"})
|
||||
AUDIO_EXTENSIONS = frozenset({"mp3", "wav", "flac", "ogg", "m4a", "aac", "opus", "wma"})
|
||||
MEDIA_EXTENSIONS = IMAGE_EXTENSIONS | VIDEO_EXTENSIONS | AUDIO_EXTENSIONS
|
||||
|
||||
# Everything else that is bytes, not prose. Rejected with a reason; not indexed
|
||||
# and not recorded as media.
|
||||
BINARY_EXTENSIONS = frozenset(
|
||||
{
|
||||
"pdf", "zip", "tar", "gz", "bz2", "xz", "7z", "rar", "iso", "dmg",
|
||||
"exe", "dll", "so", "dylib", "o", "a", "class", "jar", "war",
|
||||
"pyc", "pyo", "wasm", "ttf", "otf", "woff", "woff2", "eot",
|
||||
"sqlite", "sqlite3", "db", "mdb", "bin", "dat", "pkl", "npy", "npz",
|
||||
"stl", "step", "obj", "blend", "psd", "ai", "sketch", "fig",
|
||||
}
|
||||
)
|
||||
|
||||
# Text-ish extensions the estate actually contains. A file with an unknown
|
||||
# extension is admitted only when it decodes as UTF-8 and does not look binary.
|
||||
TEXT_EXTENSIONS = frozenset(
|
||||
{
|
||||
"md", "markdown", "mdx", "txt", "rst", "adoc", "org",
|
||||
"py", "go", "ts", "tsx", "js", "jsx", "mjs", "cjs", "rs", "rb",
|
||||
"php", "kt", "kts", "java", "cs", "c", "h", "cc", "cpp", "hpp",
|
||||
"ino", "lua", "pl", "r", "scala", "swift", "dart", "ex", "exs",
|
||||
"sh", "bash", "zsh", "fish", "ps1", "bat",
|
||||
"nix", "toml", "yml", "yaml", "ini", "conf", "cfg", "properties",
|
||||
"json", "jsonc", "xml", "html", "htm", "css", "scss", "sass", "less",
|
||||
"sql", "graphql", "proto", "tf", "hcl", "dockerfile", "makefile",
|
||||
"gitignore", "dockerignore", "editorconfig", "service", "timer",
|
||||
"csv", "tsv", "svg", "tex", "patch", "diff", "env.example",
|
||||
"example", "sample", "template", "j2", "tmpl",
|
||||
}
|
||||
)
|
||||
|
||||
_CREDENTIAL_PATTERNS = (
|
||||
("private-key", re.compile(r"-----BEGIN (?:[A-Z]+ )?PRIVATE KEY-----")),
|
||||
("aws-access-key", re.compile(r"\bAKIA[0-9A-Z]{16}\b")),
|
||||
("github-token", re.compile(r"\bgh[pousr]_[A-Za-z0-9]{20,}\b")),
|
||||
("slack-token", re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{10,}\b")),
|
||||
("openai-key", re.compile(r"\bsk-[A-Za-z0-9]{20,}\b")),
|
||||
("anthropic-key", re.compile(r"\bsk-ant-[A-Za-z0-9_\-]{20,}\b")),
|
||||
("jwt", re.compile(r"\beyJ[A-Za-z0-9_\-]{10,}\.[A-Za-z0-9_\-]{10,}\.[A-Za-z0-9_\-]{10,}\b")),
|
||||
("age-key", re.compile(r"\bAGE-SECRET-KEY-1[0-9A-Z]{20,}\b")),
|
||||
(
|
||||
"assigned-secret",
|
||||
re.compile(
|
||||
r"(?im)^\s*(?:export\s+)?[A-Z0-9_]*(?:SECRET|TOKEN|PASSWORD|PASSWD|API_KEY|"
|
||||
r"ACCESS_KEY|PRIVATE_KEY)[A-Z0-9_]*\s*[:=]\s*[\"']?[^\s\"'#]{8,}"
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
# A `.env.example`-style file may document the *name* of a variable. It may not
|
||||
# carry a value. This filter keeps `GITEA_TOKEN=` and `TOKEN=changeme` out.
|
||||
_PLACEHOLDERS = {"", "changeme", "change-me", "xxx", "todo", "placeholder", "your-token-here", "redacted"}
|
||||
|
||||
|
||||
class Rejection:
|
||||
"""One file that was not admitted, and the rule that refused it."""
|
||||
|
||||
__slots__ = ("path", "platform", "reason", "detail")
|
||||
|
||||
def __init__(self, path: str, platform: str, reason: str, detail: str = "") -> None:
|
||||
self.path = path
|
||||
self.platform = platform
|
||||
self.reason = reason
|
||||
self.detail = detail
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"path": self.path,
|
||||
"platform": self.platform,
|
||||
"reason": self.reason,
|
||||
"detail": self.detail,
|
||||
}
|
||||
|
||||
|
||||
def extension(path: str) -> str:
|
||||
base = path.rsplit("/", 1)[-1].lower()
|
||||
if "." not in base:
|
||||
return base
|
||||
return base.rsplit(".", 1)[-1]
|
||||
|
||||
|
||||
def basename(path: str) -> str:
|
||||
return path.rsplit("/", 1)[-1]
|
||||
|
||||
|
||||
def classify(path: str, guide=None, size: int | None = None) -> tuple:
|
||||
"""Return `(kind, reason)` for a path.
|
||||
|
||||
`kind` is one of `text`, `image`, `reject`. `reason` is empty for text and
|
||||
names the rule for the other two. Guide excludes are applied here so a
|
||||
caller cannot forget them.
|
||||
"""
|
||||
path = globs.normalise(path)
|
||||
if not path:
|
||||
return "reject", "empty path"
|
||||
|
||||
parts = path.split("/")
|
||||
for part in parts[:-1]:
|
||||
if part.lower() in EXCLUDED_DIRS:
|
||||
return "reject", f"excluded directory {part!r}"
|
||||
|
||||
name = basename(path)
|
||||
lowered = name.lower()
|
||||
if lowered in EXCLUDED_DIRS:
|
||||
return "reject", f"excluded directory {lowered!r}"
|
||||
for pattern in CREDENTIAL_FILES:
|
||||
if globs.matches(lowered, pattern) or globs.matches(name, pattern):
|
||||
return "reject", "credential-shaped file name"
|
||||
|
||||
if guide is not None:
|
||||
exclude = getattr(guide, "exclude", {}) or {}
|
||||
if globs.under_any(path, exclude.get("paths")):
|
||||
return "reject", "excluded by platform guide"
|
||||
if globs.matches_any(path, exclude.get("files") or ()):
|
||||
return "reject", "excluded by platform guide"
|
||||
|
||||
ext = extension(path)
|
||||
if ext in MEDIA_EXTENSIONS:
|
||||
return "image", f"media ({ext}) recorded as an image reference"
|
||||
if ext in BINARY_EXTENSIONS:
|
||||
return "reject", f"binary ({ext})"
|
||||
|
||||
if size is not None:
|
||||
limit = int(((getattr(guide, "limits", {}) or {}).get("max_file_kb") or 200) * 1024)
|
||||
if size > limit:
|
||||
return "reject", f"oversize ({size} bytes > {limit})"
|
||||
|
||||
if guide is not None:
|
||||
include = getattr(guide, "include", {}) or {}
|
||||
include_paths = include.get("paths") or []
|
||||
if include_paths and not globs.under_any(path, include_paths):
|
||||
return "reject", "not under an included path"
|
||||
|
||||
if ext not in TEXT_EXTENSIONS and name.lower() not in TEXT_EXTENSIONS:
|
||||
# Unknown extension: admitted only if the caller proves it is UTF-8 text.
|
||||
return "unknown", ""
|
||||
return "text", ""
|
||||
|
||||
|
||||
def looks_binary(data: bytes) -> bool:
|
||||
if b"\x00" in data[:4096]:
|
||||
return True
|
||||
try:
|
||||
data.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def scan_text(text: str) -> list:
|
||||
"""Return `[(kind, line_number, excerpt)]` for credential-shaped content.
|
||||
|
||||
The excerpt is short and never includes the matched value: a rejection
|
||||
report must not become the leak it is reporting.
|
||||
"""
|
||||
hits = []
|
||||
for number, line in enumerate(text.splitlines(), start=1):
|
||||
for kind, pattern in _CREDENTIAL_PATTERNS:
|
||||
match = pattern.search(line)
|
||||
if not match:
|
||||
continue
|
||||
value = match.group(0)
|
||||
if kind == "assigned-secret":
|
||||
assigned = value.split("=", 1)[-1].split(":", 1)[-1].strip().strip("\"'")
|
||||
if assigned.lower() in _PLACEHOLDERS:
|
||||
continue
|
||||
hits.append((kind, number, line.strip()[:60].split("=", 1)[0]))
|
||||
break
|
||||
return hits
|
||||
@@ -30,15 +30,74 @@ from . import loader, store as store_module
|
||||
from .config import CHUNKER_VERSION
|
||||
from .pipeline import _contract
|
||||
from .qdrant import Qdrant
|
||||
from .sources import estate as estate_module
|
||||
|
||||
|
||||
def audit(cfg, today: datetime.date = None, log=None) -> dict:
|
||||
"""Return the audit report as a plain dict (JSON-serialisable)."""
|
||||
log = log or (lambda _msg: None)
|
||||
today = today or datetime.date.today()
|
||||
documents = loader.load_documents(cfg)
|
||||
|
||||
findings = {"stale": [], "due_soon": [], "index_drift": [], "contract_drift": []}
|
||||
# The audit reads the same estate the pipeline reads, so "the source has
|
||||
# stopped changing" is measured against the source, not against the index.
|
||||
harvest = None
|
||||
try:
|
||||
harvest = estate_module.harvest(cfg, log=log)
|
||||
documents = harvest.documents
|
||||
except estate_module.EstateError as exc:
|
||||
documents = loader.load_documents(cfg)
|
||||
estate_error = str(exc)
|
||||
else:
|
||||
estate_error = ""
|
||||
|
||||
findings = {
|
||||
"stale": [],
|
||||
"due_soon": [],
|
||||
"index_drift": [],
|
||||
"contract_drift": [],
|
||||
"stale_source": [],
|
||||
"source_missing": [],
|
||||
"source_unavailable": [],
|
||||
}
|
||||
if estate_error:
|
||||
findings["source_unavailable"].append({"platform": "estate", "problem": estate_error})
|
||||
|
||||
if harvest is not None:
|
||||
for name, row in sorted((harvest.sources or {}).items()):
|
||||
if not row.get("enabled"):
|
||||
continue
|
||||
# Curated content is measured by last_verified above; source silence
|
||||
# is about the harvested estate, which should change on its own.
|
||||
if name == "curated":
|
||||
continue
|
||||
if row.get("missing") or not row.get("documents"):
|
||||
findings["source_missing"].append(
|
||||
{
|
||||
"platform": name,
|
||||
"problem": "no documents were harvested",
|
||||
"repo": row.get("repo") or row.get("base") or "",
|
||||
}
|
||||
)
|
||||
continue
|
||||
newest = row.get("newest") or ""
|
||||
try:
|
||||
changed = datetime.date.fromisoformat(newest[:10])
|
||||
except ValueError:
|
||||
findings["source_missing"].append(
|
||||
{"platform": name, "problem": f"no usable last_modified ({newest!r})"}
|
||||
)
|
||||
continue
|
||||
age = (today - changed).days
|
||||
if age > cfg.source_silence_days:
|
||||
findings["stale_source"].append(
|
||||
{
|
||||
"platform": name,
|
||||
"repo": row.get("repo") or row.get("base") or "",
|
||||
"newest": newest,
|
||||
"age_days": age,
|
||||
"problem": f"the source has not changed in {age} days",
|
||||
}
|
||||
)
|
||||
|
||||
for doc in documents:
|
||||
try:
|
||||
@@ -90,6 +149,7 @@ def audit(cfg, today: datetime.date = None, log=None) -> dict:
|
||||
{"source_path": source, "problem": "indexed but no longer in content/"}
|
||||
)
|
||||
|
||||
graph_counts = store.graph_counts()
|
||||
stored_contract = store.get_meta("collection")
|
||||
if stored_contract != _contract(cfg):
|
||||
findings["contract_drift"].append(
|
||||
@@ -120,23 +180,36 @@ def audit(cfg, today: datetime.date = None, log=None) -> dict:
|
||||
"sources": store.source_count(),
|
||||
"collection": cfg.collection,
|
||||
"collection_points": points,
|
||||
"graph_collection": cfg.graph_collection,
|
||||
"graph_nodes": graph_counts["nodes"],
|
||||
"graph_edges": graph_counts["edges"],
|
||||
"provider": cfg.provider,
|
||||
"model": cfg.model,
|
||||
"dim": cfg.dim,
|
||||
"chunker_version": CHUNKER_VERSION,
|
||||
},
|
||||
"findings": findings,
|
||||
"sources": (harvest.sources if harvest is not None else {}),
|
||||
"warnings": (harvest.warnings if harvest is not None else []),
|
||||
"summary": {
|
||||
"stale": len(findings["stale"]),
|
||||
"due_soon": len(findings["due_soon"]),
|
||||
"index_drift": len(findings["index_drift"]),
|
||||
"contract_drift": len(findings["contract_drift"]),
|
||||
"stale_source": len(findings["stale_source"]),
|
||||
"source_missing": len(findings["source_missing"]),
|
||||
"source_unavailable": len(findings["source_unavailable"]),
|
||||
},
|
||||
# `due_soon` is a warning, not a failure: the document is still
|
||||
# inside its verification horizon. Stale, drift and contract
|
||||
# problems are failures.
|
||||
"ok": not (
|
||||
findings["stale"] or findings["index_drift"] or findings["contract_drift"]
|
||||
findings["stale"]
|
||||
or findings["index_drift"]
|
||||
or findings["contract_drift"]
|
||||
or findings["stale_source"]
|
||||
or findings["source_missing"]
|
||||
or findings["source_unavailable"]
|
||||
),
|
||||
}
|
||||
log(f"audit: {json.dumps(report['summary'], sort_keys=True)}")
|
||||
@@ -147,9 +220,12 @@ def render_text(report: dict) -> str:
|
||||
"""Human-readable audit, for a terminal or an Apprise body."""
|
||||
lines = []
|
||||
corpus = report["corpus"]
|
||||
graph = ""
|
||||
if corpus.get("graph_nodes") is not None:
|
||||
graph = f", graph {corpus['graph_nodes']} node(s) / {corpus['graph_edges']} edge(s)"
|
||||
lines.append(
|
||||
f"corpus: {corpus['documents']} document(s), {corpus['chunks']} chunk(s), "
|
||||
f"{corpus['collection_points']} vector(s) in {corpus['collection']}"
|
||||
f"{corpus['collection_points']} vector(s) in {corpus['collection']}{graph}"
|
||||
)
|
||||
lines.append(
|
||||
f"horizon: last_verified within {report['horizon']['max_age_days']} days "
|
||||
@@ -159,10 +235,13 @@ def render_text(report: dict) -> str:
|
||||
for kind, label in (
|
||||
("stale", "STALE — past the verification horizon"),
|
||||
("due_soon", "DUE SOON — verify before it goes stale"),
|
||||
("index_drift", "INDEX DRIFT — the index does not match content/"),
|
||||
("index_drift", "INDEX DRIFT — the index does not match the sources"),
|
||||
("contract_drift", "CONTRACT DRIFT — rebuild required"),
|
||||
("stale_source", "STALE SOURCE — the estate has stopped changing"),
|
||||
("source_missing", "SOURCE MISSING — an enabled platform produced nothing"),
|
||||
("source_unavailable", "SOURCE UNAVAILABLE — the estate could not be read"),
|
||||
):
|
||||
rows = findings[kind]
|
||||
rows = findings.get(kind) or []
|
||||
if not rows:
|
||||
continue
|
||||
lines.append("")
|
||||
@@ -177,7 +256,10 @@ def render_text(report: dict) -> str:
|
||||
)
|
||||
if report["ok"]:
|
||||
lines.append("")
|
||||
lines.append("OK: nothing stale, no drift." + (" (see warnings above)" if findings["due_soon"] else ""))
|
||||
lines.append(
|
||||
"OK: nothing stale, no drift, every source current."
|
||||
+ (" (see warnings above)" if findings["due_soon"] else "")
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
|
||||
89
corpus/globs.py
Normal file
89
corpus/globs.py
Normal file
@@ -0,0 +1,89 @@
|
||||
"""Path globs, with the one extension the guides need: `**/` matches nothing.
|
||||
|
||||
`fnmatch` is close but wrong for this job in two ways: it lets `*` cross a `/`,
|
||||
and it has no `**`. Platform guides are read by a human and written as ordinary
|
||||
gitignore-style globs (`**/node_modules/**`, `200 projects/**`), so this module
|
||||
implements exactly that little language and nothing else.
|
||||
|
||||
Matching is always against a normalised, repository-relative POSIX path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
_CACHE: dict = {}
|
||||
|
||||
|
||||
def normalise(path: str) -> str:
|
||||
"""Repository-relative, POSIX separators, no leading `./` or `/`."""
|
||||
path = str(path).replace("\\", "/").strip()
|
||||
while path.startswith("./"):
|
||||
path = path[2:]
|
||||
return path.lstrip("/")
|
||||
|
||||
|
||||
def compile_glob(pattern: str):
|
||||
"""Compile one glob to a regular expression. Cached."""
|
||||
cached = _CACHE.get(pattern)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
pattern = normalise(pattern)
|
||||
out = ["^"]
|
||||
index = 0
|
||||
while index < len(pattern):
|
||||
char = pattern[index]
|
||||
if char == "*":
|
||||
if pattern[index : index + 2] == "**":
|
||||
index += 2
|
||||
if pattern[index : index + 1] == "/":
|
||||
# `**/` — any number of whole directories, including none.
|
||||
out.append("(?:[^/]+/)*")
|
||||
index += 1
|
||||
else:
|
||||
out.append(".*")
|
||||
continue
|
||||
out.append("[^/]*")
|
||||
elif char == "?":
|
||||
out.append("[^/]")
|
||||
elif char == "[":
|
||||
close = pattern.find("]", index + 1)
|
||||
if close == -1:
|
||||
out.append(re.escape(char))
|
||||
else:
|
||||
body = pattern[index + 1 : close]
|
||||
if body.startswith("!"):
|
||||
body = "^" + body[1:]
|
||||
out.append("[" + body + "]")
|
||||
index = close
|
||||
else:
|
||||
out.append(re.escape(char))
|
||||
index += 1
|
||||
out.append("$")
|
||||
compiled = re.compile("".join(out))
|
||||
_CACHE[pattern] = compiled
|
||||
return compiled
|
||||
|
||||
|
||||
def matches(path: str, pattern: str) -> bool:
|
||||
return bool(compile_glob(pattern).match(normalise(path)))
|
||||
|
||||
|
||||
def matches_any(path: str, patterns) -> bool:
|
||||
return any(matches(path, pattern) for pattern in (patterns or ()))
|
||||
|
||||
|
||||
def under_any(path: str, patterns) -> bool:
|
||||
"""True when `path` is a pattern's directory itself or inside it.
|
||||
|
||||
`200 projects/**` should also exclude the bare directory entry `200 projects`
|
||||
that a tree listing returns, not only the files underneath it.
|
||||
"""
|
||||
path = normalise(path)
|
||||
for pattern in patterns or ():
|
||||
if matches(path, pattern):
|
||||
return True
|
||||
if pattern.endswith("/**") and path == normalise(pattern[:-3]):
|
||||
return True
|
||||
return False
|
||||
227
corpus/guides.py
Normal file
227
corpus/guides.py
Normal file
@@ -0,0 +1,227 @@
|
||||
"""Platform guides: one small file per platform, configuration and public copy.
|
||||
|
||||
`platforms/<name>.yml` says what a platform is, where it lives, what to include,
|
||||
what to exclude, which resume area it feeds, and whether it is enabled. The same
|
||||
wording is shown on the public site, so every value here is written to be read
|
||||
by a visitor, not just by the pipeline.
|
||||
|
||||
The schema is fixed and validated at load time. An unknown area, a missing
|
||||
`what`, or a guide whose filename and `name` disagree is a configuration error,
|
||||
not a default. A disabled guide is loaded and shown, but not harvested.
|
||||
|
||||
There is exactly one YAML dialect in this repository and one parser for it
|
||||
(`tools/validate_content.py`); this module imports that parser rather than
|
||||
adding a second one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from . import loader
|
||||
from .payloads import AREAS # one area enum for guides and payloads
|
||||
|
||||
# The platforms the pipeline knows how to harvest. `curated` is content/ and is
|
||||
# always on: it is tier 1 and has no guide because it is not an external estate.
|
||||
KINDS = ("gitea", "obsidian", "graphify")
|
||||
REQUIRED = ("name", "enabled", "what", "fits", "read")
|
||||
DEFAULT_LIMITS = {
|
||||
"max_documents": 4000,
|
||||
"max_file_kb": 200,
|
||||
"max_files_per_repo": 40,
|
||||
"max_commits": 20,
|
||||
"max_tree_entries": 3000,
|
||||
"max_nodes": 20000,
|
||||
"max_edges": 40000,
|
||||
}
|
||||
|
||||
|
||||
class GuideError(RuntimeError):
|
||||
"""A platform guide is missing, malformed, or inconsistent."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Guide:
|
||||
name: str
|
||||
enabled: bool
|
||||
what: str
|
||||
fits: list
|
||||
read: list
|
||||
base: str = ""
|
||||
auth: str = ""
|
||||
include: dict = field(default_factory=dict)
|
||||
exclude: dict = field(default_factory=dict)
|
||||
limits: dict = field(default_factory=dict)
|
||||
public: bool = True
|
||||
path: str = "" # repository-relative, for the public page and error messages
|
||||
|
||||
# -- derived ----------------------------------------------------------
|
||||
|
||||
@property
|
||||
def kind(self) -> str:
|
||||
return self.name
|
||||
|
||||
@property
|
||||
def primary_area(self) -> str:
|
||||
return self.fits[0]
|
||||
|
||||
def limit(self, name: str) -> int:
|
||||
return int(self.limits.get(name) or DEFAULT_LIMITS.get(name) or 0)
|
||||
|
||||
def to_public_dict(self) -> dict:
|
||||
"""What the site shows. No auth variable values, only its name."""
|
||||
return {
|
||||
"name": self.name,
|
||||
"enabled": self.enabled,
|
||||
"what": self.what,
|
||||
"fits": list(self.fits),
|
||||
"read": list(self.read),
|
||||
"base": self.base,
|
||||
"include": {
|
||||
"repos": _as_list((self.include or {}).get("repos")),
|
||||
"paths": _as_list((self.include or {}).get("paths")),
|
||||
},
|
||||
"exclude": {
|
||||
"repos": _as_list((self.exclude or {}).get("repos")),
|
||||
"paths": _as_list((self.exclude or {}).get("paths")),
|
||||
},
|
||||
"public": self.public,
|
||||
"path": self.path,
|
||||
}
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
data = self.to_public_dict()
|
||||
data["limits"] = dict(self.limits)
|
||||
data["auth"] = self.auth # the variable name, never its value
|
||||
return data
|
||||
|
||||
|
||||
def load_guides(root: str, directory: str = None) -> dict:
|
||||
"""Load every guide under platforms/, keyed by name. Sorted, deterministic."""
|
||||
directory = directory or os.path.join(root, "platforms")
|
||||
if not os.path.isdir(directory):
|
||||
raise GuideError(f"platform guide directory not found: {directory}")
|
||||
|
||||
validator = loader.load_validator(root)
|
||||
guides = {}
|
||||
for filename in sorted(os.listdir(directory)):
|
||||
if not filename.endswith((".yml", ".yaml")):
|
||||
continue
|
||||
path = os.path.join(directory, filename)
|
||||
rel = os.path.relpath(path, root).replace(os.sep, "/")
|
||||
guide = _load_one(validator, path, rel, os.path.splitext(filename)[0])
|
||||
if guide.name in guides:
|
||||
raise GuideError(f"{rel}: duplicate platform name {guide.name!r}")
|
||||
guides[guide.name] = guide
|
||||
|
||||
missing = sorted(set(KINDS) - set(guides))
|
||||
if missing:
|
||||
raise GuideError(
|
||||
f"no platform guide for {', '.join(missing)} under {directory}. "
|
||||
"One small file per platform is required (docs/phase-4-corpus.md §2)."
|
||||
)
|
||||
return guides
|
||||
|
||||
|
||||
def _load_one(validator, path: str, rel: str, stem: str) -> Guide:
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
text = handle.read()
|
||||
lines = [(number, line) for number, line in enumerate(text.splitlines(), start=1)]
|
||||
try:
|
||||
node = validator.parse_front_matter(lines)
|
||||
except validator.YamlError as exc:
|
||||
raise GuideError(f"{rel}:{exc.line}: invalid YAML: {exc.msg}") from exc
|
||||
values = loader.to_python(node)
|
||||
if not isinstance(values, dict):
|
||||
raise GuideError(f"{rel}: a platform guide must be a mapping")
|
||||
|
||||
for key in REQUIRED:
|
||||
if values.get(key) in (None, "", []):
|
||||
raise GuideError(f"{rel}: missing required key {key!r}")
|
||||
|
||||
name = str(values["name"]).strip()
|
||||
if name != stem:
|
||||
raise GuideError(f"{rel}: name {name!r} does not match the filename {stem!r}")
|
||||
if name not in KINDS:
|
||||
raise GuideError(f"{rel}: unknown platform {name!r}; expected one of {list(KINDS)}")
|
||||
|
||||
fits = [str(area) for area in values["fits"]]
|
||||
unknown = [area for area in fits if area not in AREAS]
|
||||
if unknown:
|
||||
raise GuideError(f"{rel}: fits contains unknown area(s) {unknown}; allowed {list(AREAS)}")
|
||||
|
||||
read = [str(item) for item in values["read"]]
|
||||
if not read:
|
||||
raise GuideError(f"{rel}: read must name at least one thing to harvest")
|
||||
|
||||
include = _mapping(values.get("include"), f"{rel}: include")
|
||||
exclude = _mapping(values.get("exclude"), f"{rel}: exclude")
|
||||
limits = _mapping(values.get("limits"), f"{rel}: limits")
|
||||
for key, value in limits.items():
|
||||
if isinstance(value, list):
|
||||
# `graph_paths` is the one list-valued limit: the places a graph may
|
||||
# be committed, tried in order.
|
||||
continue
|
||||
try:
|
||||
limits[key] = int(value)
|
||||
except (TypeError, ValueError):
|
||||
raise GuideError(f"{rel}: limit {key!r} must be an integer, got {value!r}") from None
|
||||
|
||||
return Guide(
|
||||
name=name,
|
||||
enabled=bool(values.get("enabled")),
|
||||
what=str(values["what"]).strip(),
|
||||
fits=fits,
|
||||
read=read,
|
||||
base=str(values.get("base") or "").strip(),
|
||||
auth=str(values.get("auth") or "").strip(),
|
||||
include=include,
|
||||
exclude=exclude,
|
||||
limits=limits,
|
||||
public=bool(values.get("public", True)),
|
||||
path=rel,
|
||||
)
|
||||
|
||||
|
||||
def _mapping(value, where: str) -> dict:
|
||||
if value is None:
|
||||
return {}
|
||||
if not isinstance(value, dict):
|
||||
raise GuideError(f"{where} must be a mapping")
|
||||
out = {}
|
||||
for key, item in value.items():
|
||||
if isinstance(item, list):
|
||||
out[key] = [str(entry) for entry in item]
|
||||
else:
|
||||
out[key] = item
|
||||
return out
|
||||
|
||||
|
||||
def _as_list(value) -> list:
|
||||
"""`repos: all` is a scalar; the public manifest still emits a list."""
|
||||
if value in (None, ""):
|
||||
return []
|
||||
if isinstance(value, list):
|
||||
return [str(item) for item in value]
|
||||
return [str(value)]
|
||||
|
||||
|
||||
def enabled_guides(guides: dict) -> list:
|
||||
return [guide for _, guide in sorted(guides.items()) if guide.enabled]
|
||||
|
||||
|
||||
def auth_value(guide: Guide, environ=None) -> str:
|
||||
"""Read a guide's token from the environment. Never from a file in the repo."""
|
||||
if not guide.auth:
|
||||
return ""
|
||||
environ = os.environ if environ is None else environ
|
||||
return environ.get(guide.auth, "")
|
||||
|
||||
|
||||
def build_public_manifest(guides: dict) -> dict:
|
||||
"""The guides as the public site shows them. Deterministic JSON shape."""
|
||||
return {
|
||||
"version": 1,
|
||||
"platforms": [guide.to_public_dict() for _, guide in sorted(guides.items())],
|
||||
}
|
||||
@@ -78,6 +78,16 @@ class Document:
|
||||
digest: str # sha256 of the file bytes: the incremental-mode key
|
||||
url: str # canonical page URL, matching internal/content.PageURL
|
||||
|
||||
# Provenance (docs/phase-4-corpus.md §3). Curated content is tier 1 and has
|
||||
# no external origin; harvested documents set these in corpus/sources/.
|
||||
platform: str = "curated"
|
||||
tier: int = 1
|
||||
origin_url: str = ""
|
||||
last_modified: str = ""
|
||||
source_type: str = "" # empty means: derive from tags (payloads.source_type_for)
|
||||
kind: str = "markdown" # "markdown" or "code", selects the chunker
|
||||
repo_path: str = "" # origin-relative path; `path` above is the unique source key
|
||||
|
||||
# Filled in by the pipeline as it goes; not part of the schema.
|
||||
chunk_count: int = field(default=0)
|
||||
|
||||
@@ -158,5 +168,13 @@ def _load_one(cfg, validator, path: str) -> Document:
|
||||
body=body,
|
||||
digest=digest_of(path),
|
||||
url=page_url(str(fields["area"]), str(fields["id"]), level),
|
||||
platform="curated",
|
||||
tier=1,
|
||||
# A curated claim is traceable to the page it appears on. The URL is
|
||||
# site-relative because the public origin is a deployment concern.
|
||||
origin_url=page_url(str(fields["area"]), str(fields["id"]), level),
|
||||
last_modified=f"{fields['last_verified']}T00:00:00Z",
|
||||
kind="markdown",
|
||||
repo_path=rel,
|
||||
)
|
||||
return doc
|
||||
|
||||
@@ -16,8 +16,14 @@ import datetime
|
||||
|
||||
AREAS = ("ai", "platform", "edge", "products", "history", "about")
|
||||
LEVELS = (1, 2, 3)
|
||||
SOURCE_TYPES = ("doc", "code", "adr", "metric")
|
||||
# docs/phase-4-corpus.md §3 fixes the harvested types; adr and metric were
|
||||
# already used by curated content and stay valid.
|
||||
SOURCE_TYPES = (
|
||||
"doc", "code", "commit", "graph_node", "graph_edge", "claim", "adr", "metric",
|
||||
)
|
||||
CONFIDENCE = ("delivered", "running", "in-progress", "abandoned", "historic")
|
||||
PLATFORMS = ("curated", "gitea", "obsidian", "graphify")
|
||||
TIERS = (1, 2)
|
||||
|
||||
# Fields every payload must carry (docs/phase-4-corpus.md §4.2), plus the ones
|
||||
# the index needs to answer "where did this come from?" without a second lookup.
|
||||
@@ -30,6 +36,13 @@ REQUIRED_FIELDS = (
|
||||
"artefacts",
|
||||
"last_verified",
|
||||
"text",
|
||||
# docs/phase-4-corpus.md §3: provenance is part of the payload, not an
|
||||
# optional extra. A chunk that cannot say where it came from is not admitted.
|
||||
"platform",
|
||||
"tier",
|
||||
"origin_url",
|
||||
"path",
|
||||
"last_modified",
|
||||
)
|
||||
PROVENANCE_FIELDS = (
|
||||
"id",
|
||||
@@ -51,11 +64,15 @@ class PayloadError(RuntimeError):
|
||||
|
||||
|
||||
def source_type_for(doc) -> str:
|
||||
"""Classify a document. Everything in content/ is a doc unless it says so.
|
||||
"""Classify a document.
|
||||
|
||||
The schema has no `source_type` field and the schema is frozen, so the
|
||||
classification is derived from tags the schema does allow.
|
||||
Harvested documents carry an explicit `source_type`. Curated content does
|
||||
not: the schema is frozen and has no such field, so it is derived from the
|
||||
tags the schema does allow.
|
||||
"""
|
||||
explicit = str(getattr(doc, "source_type", "") or "").strip()
|
||||
if explicit:
|
||||
return explicit
|
||||
tags = {str(tag).lower() for tag in doc.tags}
|
||||
if "adr" in tags:
|
||||
return "adr"
|
||||
@@ -82,6 +99,12 @@ def build_payload(doc, chunk, cfg) -> dict:
|
||||
"artefacts": [str(a.get("url", "")) for a in doc.artefacts],
|
||||
"last_verified": doc.last_verified,
|
||||
"text": chunk.text,
|
||||
# -- provenance (docs/phase-4-corpus.md §3) ------------------------
|
||||
"platform": doc.platform,
|
||||
"tier": int(doc.tier),
|
||||
"origin_url": doc.origin_url,
|
||||
"path": getattr(doc, "repo_path", "") or doc.path,
|
||||
"last_modified": doc.last_modified,
|
||||
# -- provenance ----------------------------------------------------
|
||||
"id": None, # filled by the pipeline once the id is computed
|
||||
"source_path": doc.path,
|
||||
@@ -125,7 +148,17 @@ def validate_payload(payload: dict, where: str = "chunk") -> None:
|
||||
f"{where}: confidence {payload['confidence']!r} is not one of {list(CONFIDENCE)}"
|
||||
)
|
||||
|
||||
for field in ("period", "title", "claim", "url", "source_path", "source_id", "embedding_model"):
|
||||
if payload["platform"] not in PLATFORMS:
|
||||
raise PayloadError(
|
||||
f"{where}: platform {payload['platform']!r} is not one of {list(PLATFORMS)}"
|
||||
)
|
||||
if payload["tier"] not in TIERS:
|
||||
raise PayloadError(f"{where}: tier {payload['tier']!r} is not one of {list(TIERS)}")
|
||||
|
||||
for field in (
|
||||
"period", "title", "claim", "url", "source_path", "source_id",
|
||||
"embedding_model", "origin_url", "path",
|
||||
):
|
||||
value = payload[field]
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise PayloadError(f"{where}: field {field!r} must be a non-empty string")
|
||||
@@ -138,6 +171,7 @@ def validate_payload(payload: dict, where: str = "chunk") -> None:
|
||||
raise PayloadError(f"{where}: field 'artefacts' must be a list (empty is allowed)")
|
||||
|
||||
_check_date(payload["last_verified"], where)
|
||||
_check_timestamp(payload["last_modified"], where)
|
||||
if not isinstance(payload["embedding_dim"], int) or payload["embedding_dim"] <= 0:
|
||||
raise PayloadError(f"{where}: field 'embedding_dim' must be a positive integer")
|
||||
if not isinstance(payload["chunk_index"], int) or payload["chunk_index"] < 0:
|
||||
@@ -146,6 +180,19 @@ def validate_payload(payload: dict, where: str = "chunk") -> None:
|
||||
raise PayloadError(f"{where}: field 'content_hash' must be a sha256 hex digest")
|
||||
|
||||
|
||||
def _check_timestamp(value: str, where: str) -> None:
|
||||
"""`last_modified` is an ISO-8601 instant, e.g. 2026-10-14T05:00:00Z."""
|
||||
text = str(value)
|
||||
if text.endswith("Z"):
|
||||
text = text[:-1] + "+00:00"
|
||||
try:
|
||||
datetime.datetime.fromisoformat(text)
|
||||
except ValueError as exc:
|
||||
raise PayloadError(
|
||||
f"{where}: field 'last_modified' must be an ISO-8601 timestamp, got {value!r}"
|
||||
) from exc
|
||||
|
||||
|
||||
def _check_date(value: str, where: str) -> None:
|
||||
try:
|
||||
datetime.date.fromisoformat(str(value))
|
||||
|
||||
@@ -1,20 +1,32 @@
|
||||
"""The pipeline: content/ -> validate -> chunk -> enrich -> embed -> upsert.
|
||||
"""The ingestion pipeline: estate -> normalise -> chunk -> enrich -> embed -> upsert.
|
||||
|
||||
One entry point, `run`, with two modes:
|
||||
|
||||
* **rebuild** — drop the collection, empty the store, embed everything. This is
|
||||
the "one command rebuilds the corpus from an empty volume" path.
|
||||
* **rebuild** — drop the collections, empty the store, embed everything. This is
|
||||
the "one command rebuilds the pool from an empty volume" path.
|
||||
* **sync** — the default. Only documents whose content hash, embedding model,
|
||||
embedding dimension or chunker version changed are re-chunked and
|
||||
re-embedded. Everything else is left alone.
|
||||
embedding dimension or chunker version changed are re-chunked and re-embedded.
|
||||
Everything else is left alone, including the graph when graph.json is byte-for-byte
|
||||
unchanged.
|
||||
|
||||
The order matters. Payloads are built and validated for *every* changed document
|
||||
before a single vector is written, so an invalid payload stops the run with an
|
||||
empty diff rather than leaving half an index behind.
|
||||
The order matters. Every payload for every changed document is built and
|
||||
validated *before* a single vector is written, so an invalid payload stops the
|
||||
run with an empty diff rather than leaving half an index behind. Embedding is one
|
||||
flat pass over the changed chunks, batched, so the provider call count is a
|
||||
direct measure of how much work incremental mode actually did.
|
||||
|
||||
Deterministic ids (`sha256(source_path + chunk_index + content_hash)`) mean
|
||||
re-running updates points instead of duplicating them, and the id is a valid
|
||||
UUID so Qdrant accepts it as a point id without a mapping table.
|
||||
|
||||
Two Qdrant collections, one store:
|
||||
|
||||
* `resume_corpus` — one point per chunk of a document, from every platform;
|
||||
* `resume_graph` — one point per graph node and one per graph edge, joined to the
|
||||
note it came from by `source_note`.
|
||||
|
||||
Neither is ever read by the public site (acceptance test A1): the site serves
|
||||
pre-rendered files and does not open a socket to Qdrant.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -25,9 +37,15 @@ import time
|
||||
import uuid
|
||||
from dataclasses import asdict, dataclass, field
|
||||
|
||||
from . import chunker, embeddings, loader, payloads, store as store_module
|
||||
from . import chunker, embeddings, exclusions, payloads, store as store_module
|
||||
from .config import CHUNKER_VERSION
|
||||
from .qdrant import Qdrant, QdrantError
|
||||
from .sources import estate as estate_module
|
||||
from .sources import graph as graph_module
|
||||
|
||||
# How many texts one embed call carries. The provider may batch smaller
|
||||
# internally (OmniRoute does); this bounds memory on the ingest host.
|
||||
EMBED_BATCH = 128
|
||||
|
||||
|
||||
class PipelineError(RuntimeError):
|
||||
@@ -61,12 +79,22 @@ class RunStats:
|
||||
points_sent: int = 0
|
||||
collection_points: int = 0
|
||||
store_chunks: int = 0
|
||||
graph_nodes: int = 0
|
||||
graph_edges: int = 0
|
||||
graph_points: int = 0
|
||||
graph_rebuilt: bool = False
|
||||
rejections: int = 0
|
||||
images: int = 0
|
||||
duration_s: float = 0.0
|
||||
collection: str = ""
|
||||
graph_collection: str = ""
|
||||
provider: str = ""
|
||||
model: str = ""
|
||||
dim: int = 0
|
||||
chunker_version: str = CHUNKER_VERSION
|
||||
platforms: dict = field(default_factory=dict)
|
||||
rejection_summary: dict = field(default_factory=dict)
|
||||
warnings: list = field(default_factory=list)
|
||||
notes: list = field(default_factory=list)
|
||||
|
||||
def summary(self) -> str:
|
||||
@@ -76,10 +104,18 @@ class RunStats:
|
||||
f"chunks written={self.chunks_written} deleted={self.chunks_deleted}",
|
||||
f"embedding calls={self.embed_calls} texts={self.texts_embedded}",
|
||||
f"points sent={self.points_sent} collection points={self.collection_points}",
|
||||
f"graph nodes={self.graph_nodes} edges={self.graph_edges} points={self.graph_points} "
|
||||
f"rebuilt={self.graph_rebuilt}",
|
||||
f"rejections={self.rejections} image references={self.images}",
|
||||
f"platforms={json.dumps(self.platforms, sort_keys=True)}",
|
||||
f"provider={self.provider} model={self.model} dim={self.dim} chunker={self.chunker_version}",
|
||||
f"collection={self.collection} duration={self.duration_s:.2f}s",
|
||||
f"collection={self.collection} graph_collection={self.graph_collection} "
|
||||
f"duration={self.duration_s:.2f}s",
|
||||
]
|
||||
if self.rejection_summary:
|
||||
rows.append(f"rejected: {json.dumps(self.rejection_summary, sort_keys=True)}")
|
||||
rows.extend(self.notes)
|
||||
rows.extend(f"warning: {warning}" for warning in self.warnings)
|
||||
return "\n".join(rows)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
@@ -87,7 +123,7 @@ class RunStats:
|
||||
|
||||
|
||||
def run(cfg, mode: str = "sync", log=None) -> RunStats:
|
||||
"""Build or refresh the corpus. Returns what happened; raises on any error."""
|
||||
"""Build or refresh the pool. Returns what happened; raises on any error."""
|
||||
if mode not in ("sync", "rebuild"):
|
||||
raise PipelineError(f"unknown mode {mode!r}; expected 'sync' or 'rebuild'")
|
||||
started = time.time()
|
||||
@@ -95,15 +131,26 @@ def run(cfg, mode: str = "sync", log=None) -> RunStats:
|
||||
stats = RunStats(
|
||||
mode=mode,
|
||||
collection=cfg.collection,
|
||||
graph_collection=cfg.graph_collection,
|
||||
provider=cfg.provider,
|
||||
model=cfg.model,
|
||||
dim=cfg.dim,
|
||||
chunker_version=cfg.chunker_version,
|
||||
)
|
||||
|
||||
documents = loader.load_documents(cfg)
|
||||
# 1. Read the estate. Curated content is validated first (loader does it),
|
||||
# so a broken content tree stops the run before Gitea is touched.
|
||||
harvest = estate_module.harvest(cfg, log=log)
|
||||
documents = harvest.documents
|
||||
stats.documents = len(documents)
|
||||
log(f"validated {len(documents)} document(s)")
|
||||
stats.rejections = len(harvest.rejections)
|
||||
stats.images = len(harvest.images)
|
||||
stats.rejection_summary = harvest.rejection_summary()
|
||||
stats.warnings = list(harvest.warnings)
|
||||
stats.platforms = {
|
||||
name: {"documents": row.get("documents", 0)} for name, row in harvest.sources.items()
|
||||
}
|
||||
log(f"estate: {len(documents)} document(s), {stats.rejections} rejection(s), {stats.images} image reference(s)")
|
||||
|
||||
provider = embeddings.build_provider(cfg)
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
@@ -116,8 +163,7 @@ def run(cfg, mode: str = "sync", log=None) -> RunStats:
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
contract = _contract(cfg)
|
||||
mode = _check_contract(store, qdrant, cfg, contract, mode)
|
||||
|
||||
_ensure_collection(qdrant, cfg)
|
||||
_ensure_collection(qdrant, cfg.collection, cfg)
|
||||
|
||||
manifest = store.manifest()
|
||||
changed, removed = _classify(documents, manifest, cfg, mode)
|
||||
@@ -133,33 +179,45 @@ def run(cfg, mode: str = "sync", log=None) -> RunStats:
|
||||
qdrant.delete_by_source(cfg.collection, path)
|
||||
log(f"removed {path}")
|
||||
|
||||
# 2. Build and validate every payload before any write.
|
||||
prepared = [] # [(doc, payloads)]
|
||||
for doc in changed:
|
||||
payloads_for_doc = _prepare(doc, cfg, manifest)
|
||||
stats.chunks_written += len(payloads_for_doc)
|
||||
prepared.append((doc, _prepare(doc, cfg)))
|
||||
all_payloads = [payload for _, rows in prepared for payload in rows]
|
||||
|
||||
vectors = provider.embed([p["text"] for p in payloads_for_doc])
|
||||
if len(vectors) != len(payloads_for_doc):
|
||||
raise PipelineError(
|
||||
f"{doc.path}: provider returned {len(vectors)} vectors for "
|
||||
f"{len(payloads_for_doc)} chunks"
|
||||
)
|
||||
# 3. Embed in one flat, batched pass. This is the number the DoD uses to
|
||||
# prove incremental mode: it is proportional to changed chunks only.
|
||||
vectors = _embed_all(provider, [payload["text"] for payload in all_payloads])
|
||||
if len(vectors) != len(all_payloads):
|
||||
raise PipelineError(
|
||||
f"provider returned {len(vectors)} vectors for {len(all_payloads)} chunks"
|
||||
)
|
||||
|
||||
cursor = 0
|
||||
platform_changes = stats.platforms
|
||||
for doc, rows in prepared:
|
||||
points = []
|
||||
for payload, vector in zip(payloads_for_doc, vectors):
|
||||
for payload in rows:
|
||||
vector = vectors[cursor]
|
||||
cursor += 1
|
||||
if len(vector) != cfg.dim:
|
||||
raise PipelineError(
|
||||
f"{doc.path}: vector dimension {len(vector)} != configured {cfg.dim}"
|
||||
)
|
||||
points.append({"id": payload["id"], "vector": vector, "payload": payload})
|
||||
|
||||
# Order: remove what this document used to have (chunk counts change),
|
||||
# then write the new rows and points. A crash between the two leaves an
|
||||
# index that reports as incomplete, never one that claims to be whole.
|
||||
# Remove what this document used to have, then write the new rows and
|
||||
# points. A crash between the two leaves an index that reports as
|
||||
# incomplete, never one that claims to be whole.
|
||||
stats.chunks_deleted += store.delete_document(doc.path)
|
||||
qdrant.delete_by_source(cfg.collection, doc.path)
|
||||
stats.points_sent += qdrant.upsert(cfg.collection, points)
|
||||
store.write_document(doc, payloads_for_doc)
|
||||
store.write_document(doc, rows)
|
||||
stats.chunks_written += len(rows)
|
||||
|
||||
bucket = platform_changes.setdefault(doc.platform, {"documents": 0})
|
||||
key = "updated" if doc.path in manifest else "added"
|
||||
bucket[key] = bucket.get(key, 0) + 1
|
||||
if doc.path in manifest:
|
||||
stats.updated += 1
|
||||
log(f"updated {doc.path} ({len(points)} chunk(s))")
|
||||
@@ -170,24 +228,47 @@ def run(cfg, mode: str = "sync", log=None) -> RunStats:
|
||||
provider_stats = provider.stats()
|
||||
stats.embed_calls = provider_stats["calls"]
|
||||
stats.texts_embedded = provider_stats["texts"]
|
||||
|
||||
# 4. Graph: its own collection, rebuilt only when graph.json changed.
|
||||
graph_stats = _sync_graph(qdrant, store, cfg, harvest, contract, mode, provider, log)
|
||||
stats.graph_nodes = graph_stats["nodes"]
|
||||
stats.graph_edges = graph_stats["edges"]
|
||||
stats.graph_points = graph_stats["points"]
|
||||
stats.graph_rebuilt = graph_stats["rebuilt"]
|
||||
stats.notes.extend(graph_stats["notes"])
|
||||
|
||||
# 5. Consistency and the credential scan. Both are checks, not hopes.
|
||||
stats.store_chunks = store.chunk_count()
|
||||
stats.collection_points = qdrant.point_count(cfg.collection)
|
||||
|
||||
if stats.collection_points != stats.store_chunks:
|
||||
raise PipelineError(
|
||||
"index is inconsistent after the run: "
|
||||
f"{stats.collection_points} point(s) in Qdrant but "
|
||||
f"{stats.store_chunks} chunk(s) in SQLite. Run `python3 -m corpus.cli verify`."
|
||||
)
|
||||
_scan_pool(store)
|
||||
_scan_graph(store)
|
||||
|
||||
store.set_meta("collection", contract)
|
||||
store.set_meta("last_run", {**stats.to_dict(), "finished_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
|
||||
_write_collection_file(cfg, store, stats)
|
||||
store.set_meta("sources", harvest.sources)
|
||||
store.set_meta("rejections", harvest.rejections[:200])
|
||||
store.set_meta("images", harvest.images[:500])
|
||||
store.set_meta(
|
||||
"last_run",
|
||||
{
|
||||
**stats.to_dict(),
|
||||
"finished_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
},
|
||||
)
|
||||
_write_collection_file(cfg, store, stats, harvest)
|
||||
|
||||
stats.duration_s = round(time.time() - started, 3)
|
||||
return stats
|
||||
|
||||
|
||||
# -- planning -------------------------------------------------------------
|
||||
|
||||
|
||||
def _contract(cfg) -> dict:
|
||||
"""What this collection *is*. Recorded so a change is a migration, not drift."""
|
||||
return {
|
||||
@@ -200,13 +281,24 @@ def _contract(cfg) -> dict:
|
||||
|
||||
|
||||
def _check_contract(store, qdrant, cfg, contract: dict, mode: str) -> str:
|
||||
stored = store.get_meta("collection")
|
||||
if mode == "rebuild":
|
||||
qdrant.drop_collection(cfg.collection)
|
||||
store.reset()
|
||||
qdrant.drop_collection(cfg.graph_collection)
|
||||
if store.schema_stale:
|
||||
store.rebuild_schema()
|
||||
else:
|
||||
store.reset()
|
||||
store.set_meta("schema_version", store_module.SCHEMA_VERSION)
|
||||
return mode
|
||||
|
||||
if store.schema_stale:
|
||||
raise PipelineError(
|
||||
"the corpus database was written by an older payload shape and has no "
|
||||
"provenance columns. It is derived data: run `python3 -m corpus.cli rebuild`."
|
||||
)
|
||||
|
||||
stored = store.get_meta("collection")
|
||||
if stored is None:
|
||||
# No contract yet: either a fresh store, or one whose database was lost.
|
||||
if qdrant.collection_exists(cfg.collection) and qdrant.point_count(cfg.collection) > 0:
|
||||
raise PipelineError(
|
||||
f"collection {cfg.collection!r} holds points but this store has no "
|
||||
@@ -217,7 +309,7 @@ def _check_contract(store, qdrant, cfg, contract: dict, mode: str) -> str:
|
||||
if stored != contract:
|
||||
raise PipelineError(
|
||||
"the collection contract does not match this configuration — refusing to "
|
||||
"mix vectors in one collection (docs/phase-4-corpus.md §4.3).\n"
|
||||
"mix vectors in one collection (docs/phase-4-corpus.md §4).\n"
|
||||
f" stored: {json.dumps(stored, sort_keys=True)}\n"
|
||||
f" config: {json.dumps(contract, sort_keys=True)}\n"
|
||||
"A provider or chunker change is a governed migration: run "
|
||||
@@ -226,20 +318,18 @@ def _check_contract(store, qdrant, cfg, contract: dict, mode: str) -> str:
|
||||
return mode
|
||||
|
||||
|
||||
def _ensure_collection(qdrant: Qdrant, cfg) -> None:
|
||||
if not qdrant.collection_exists(cfg.collection):
|
||||
qdrant.create_collection(
|
||||
cfg.collection, cfg.dim, vectors_on_disk=cfg.vectors_on_disk
|
||||
)
|
||||
def _ensure_collection(qdrant: Qdrant, name: str, cfg) -> None:
|
||||
if not qdrant.collection_exists(name):
|
||||
qdrant.create_collection(name, cfg.dim, vectors_on_disk=cfg.vectors_on_disk)
|
||||
return
|
||||
info = qdrant.collection_info(cfg.collection)
|
||||
info = qdrant.collection_info(name)
|
||||
vectors = ((info.get("config") or {}).get("params") or {}).get("vectors")
|
||||
size = None
|
||||
if isinstance(vectors, dict):
|
||||
size = vectors.get("size")
|
||||
if size is not None and int(size) != int(cfg.dim):
|
||||
raise PipelineError(
|
||||
f"collection {cfg.collection!r} has vectors of size {size} but this run "
|
||||
f"collection {name!r} has vectors of size {size} but this run "
|
||||
f"is configured for {cfg.dim}. That is a governed migration: rebuild."
|
||||
)
|
||||
|
||||
@@ -268,10 +358,13 @@ def _classify(documents: list, manifest: dict, cfg, mode: str):
|
||||
return changed, removed
|
||||
|
||||
|
||||
def _prepare(doc, cfg, manifest: dict) -> list:
|
||||
def _prepare(doc, cfg) -> list:
|
||||
"""Chunk one document and build validated payloads. No network, no writes."""
|
||||
try:
|
||||
chunks = chunker.chunk_document(doc.body)
|
||||
if getattr(doc, "kind", "markdown") == "code":
|
||||
chunks = chunker.chunk_code(doc.body)
|
||||
else:
|
||||
chunks = chunker.chunk_document(doc.body)
|
||||
except chunker.ChunkError as exc:
|
||||
raise PipelineError(f"{doc.path}: {exc}") from exc
|
||||
if not chunks:
|
||||
@@ -290,14 +383,118 @@ def _prepare(doc, cfg, manifest: dict) -> list:
|
||||
return out
|
||||
|
||||
|
||||
def _write_collection_file(cfg, store, stats: RunStats) -> None:
|
||||
"""Mirror the contract to data/corpus/collection.json for the operator."""
|
||||
def _embed_all(provider, texts: list) -> list:
|
||||
out = []
|
||||
for start in range(0, len(texts), EMBED_BATCH):
|
||||
out.extend(provider.embed(texts[start : start + EMBED_BATCH]))
|
||||
return out
|
||||
|
||||
|
||||
# -- graph ----------------------------------------------------------------
|
||||
|
||||
|
||||
def _sync_graph(qdrant, store, cfg, harvest, contract, mode, provider, log) -> dict:
|
||||
graph = harvest.graph
|
||||
stored = store.get_meta("graph") or {}
|
||||
notes = []
|
||||
if graph is None:
|
||||
if mode == "rebuild":
|
||||
qdrant.drop_collection(cfg.graph_collection)
|
||||
store.write_graph([], [])
|
||||
store.set_meta("graph", {})
|
||||
notes.append("graph: no graph.json was available; the graph collection is empty")
|
||||
return {"nodes": 0, "edges": 0, "points": 0, "rebuilt": True, "notes": notes}
|
||||
counts = store.graph_counts()
|
||||
notes.append(
|
||||
f"graph: graph.json was not available this run; kept the previous graph "
|
||||
f"({counts['nodes']} nodes, {counts['edges']} edges)"
|
||||
)
|
||||
return {**counts, "points": counts["nodes"] + counts["edges"], "rebuilt": False, "notes": notes}
|
||||
|
||||
graph_contract = {
|
||||
"digest": graph.digest,
|
||||
"source_path": graph.source_path,
|
||||
"provider": contract["provider"],
|
||||
"model": contract["model"],
|
||||
"dim": contract["dim"],
|
||||
}
|
||||
rebuild = mode == "rebuild" or stored != graph_contract
|
||||
if not rebuild:
|
||||
counts = store.graph_counts()
|
||||
return {
|
||||
**counts,
|
||||
"points": counts["nodes"] + counts["edges"],
|
||||
"rebuilt": False,
|
||||
"notes": [f"graph: unchanged ({graph.digest[:12]}), not re-embedded"],
|
||||
}
|
||||
|
||||
if not qdrant.collection_exists(cfg.graph_collection):
|
||||
_ensure_collection(qdrant, cfg.graph_collection, cfg)
|
||||
else:
|
||||
qdrant.drop_collection(cfg.graph_collection)
|
||||
qdrant.create_collection(cfg.graph_collection, cfg.dim, vectors_on_disk=cfg.vectors_on_disk)
|
||||
|
||||
rows = [("node", row) for row in graph.nodes] + [("edge", row) for row in graph.edges]
|
||||
payloads_for_graph = [graph_module.point_payload(row, kind) for kind, row in rows]
|
||||
vectors = _embed_all(provider, [payload["text"] for payload in payloads_for_graph])
|
||||
points = [
|
||||
{"id": row["point_id"], "vector": vector, "payload": payload}
|
||||
for (_, row), vector, payload in zip(rows, vectors, payloads_for_graph)
|
||||
]
|
||||
sent = qdrant.upsert(cfg.graph_collection, points)
|
||||
store.write_graph(graph.nodes, graph.edges)
|
||||
store.set_meta("graph", graph_contract)
|
||||
log(f"graph: rebuilt {len(graph.nodes)} node(s) and {len(graph.edges)} edge(s), {sent} point(s)")
|
||||
return {
|
||||
"nodes": len(graph.nodes),
|
||||
"edges": len(graph.edges),
|
||||
"points": sent,
|
||||
"rebuilt": True,
|
||||
"notes": notes,
|
||||
}
|
||||
|
||||
|
||||
# -- checks ---------------------------------------------------------------
|
||||
|
||||
|
||||
def _scan_pool(store) -> None:
|
||||
"""Search the finished pool for credential-shaped content. Fail, do not warn."""
|
||||
hits = []
|
||||
for source_path, chunk_index, text in store.pool_texts():
|
||||
for kind, line, _excerpt in exclusions.scan_text(text):
|
||||
hits.append((source_path, chunk_index, kind, line))
|
||||
if hits:
|
||||
detail = ", ".join(f"{path}#chunk-{index} ({kind} line {line})" for path, index, kind, line in hits[:10])
|
||||
raise PipelineError(
|
||||
f"credential-shaped content reached the pool in {len(hits)} place(s): {detail}. "
|
||||
"The run is failed rather than published; fix the source or the exclusion rule."
|
||||
)
|
||||
|
||||
|
||||
def _scan_graph(store) -> None:
|
||||
rows = store.db.execute("SELECT id, text FROM graph_nodes UNION ALL SELECT id, text FROM graph_edges")
|
||||
hits = []
|
||||
for row in rows:
|
||||
for kind, line, _excerpt in exclusions.scan_text(row["text"]):
|
||||
hits.append((row["id"], kind, line))
|
||||
if hits:
|
||||
detail = ", ".join(f"{node} ({kind} line {line})" for node, kind, line in hits[:10])
|
||||
raise PipelineError(f"credential-shaped content reached the graph in {len(hits)} place(s): {detail}")
|
||||
|
||||
|
||||
def _write_collection_file(cfg, store, stats: RunStats, harvest) -> None:
|
||||
"""Mirror the contract and the last run to data/corpus/collection.json."""
|
||||
body = {
|
||||
"collection": cfg.collection,
|
||||
"contract": _contract(cfg),
|
||||
"qdrant_url": cfg.qdrant_url,
|
||||
"chunks": store.chunk_count(),
|
||||
"sources": store.source_count(),
|
||||
"graph_collection": cfg.graph_collection,
|
||||
"graph": store.graph_counts(),
|
||||
"platforms": harvest.sources,
|
||||
"rejections": harvest.rejection_summary(),
|
||||
"images": len(harvest.images),
|
||||
"last_run": store.get_meta("last_run"),
|
||||
}
|
||||
with open(cfg.meta_path, "w", encoding="utf-8") as handle:
|
||||
@@ -309,10 +506,11 @@ def _write_collection_file(cfg, store, stats: RunStats) -> None:
|
||||
|
||||
|
||||
def verify(cfg, log=None) -> list:
|
||||
"""Return a list of problems. Empty means the index agrees with content/."""
|
||||
"""Return a list of problems. Empty means the index agrees with the sources."""
|
||||
log = log or (lambda _msg: None)
|
||||
problems = []
|
||||
documents = loader.load_documents(cfg)
|
||||
harvest = estate_module.harvest(cfg)
|
||||
documents = harvest.documents
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
if not qdrant.health():
|
||||
return [f"Qdrant is not answering at {cfg.qdrant_url}"]
|
||||
@@ -342,6 +540,7 @@ def verify(cfg, log=None) -> list:
|
||||
except payloads.PayloadError as exc:
|
||||
problems.append(str(exc))
|
||||
|
||||
on_disk = {doc.path: doc for doc in documents}
|
||||
for doc in documents:
|
||||
payload_rows = by_source.get(doc.path, [])
|
||||
if not payload_rows:
|
||||
@@ -356,18 +555,48 @@ def verify(cfg, log=None) -> list:
|
||||
f"{doc.path}: {expected} chunk row(s) but {actual} vector(s) in Qdrant"
|
||||
)
|
||||
|
||||
for source in sorted(set(by_source) - {doc.path for doc in documents}):
|
||||
problems.append(f"{source}: indexed but no longer in content/ (run sync)")
|
||||
for source in sorted(set(by_source) - set(on_disk)):
|
||||
problems.append(f"{source}: indexed but no longer in content/ or the estate (run sync)")
|
||||
|
||||
total_points = sum(points_by_source.values())
|
||||
if total_points != store.chunk_count():
|
||||
problems.append(
|
||||
f"{total_points} vector(s) in Qdrant but {store.chunk_count()} chunk(s) in SQLite"
|
||||
)
|
||||
|
||||
# Graph: every row must resolve to a note, and Qdrant must agree.
|
||||
counts = store.graph_counts()
|
||||
graph_meta = store.get_meta("graph") or {}
|
||||
if harvest.graph is not None:
|
||||
if not graph_meta:
|
||||
problems.append("graph: graph.json is available but the graph was never indexed")
|
||||
if qdrant.collection_exists(cfg.graph_collection):
|
||||
graph_points = qdrant.point_count(cfg.graph_collection)
|
||||
if graph_points != counts["nodes"] + counts["edges"]:
|
||||
problems.append(
|
||||
f"graph: {graph_points} point(s) in Qdrant but "
|
||||
f"{counts['nodes'] + counts['edges']} row(s) in SQLite"
|
||||
)
|
||||
elif counts["nodes"] or counts["edges"]:
|
||||
problems.append(f"graph: collection {cfg.graph_collection!r} does not exist")
|
||||
for row in store.db.execute("SELECT id, source_note, origin_url FROM graph_nodes LIMIT 5"):
|
||||
if not row["source_note"] or not row["origin_url"]:
|
||||
problems.append(f"graph node {row['id']}: no source note or origin")
|
||||
missing_notes = [
|
||||
row["id"]
|
||||
for row in store.db.execute(
|
||||
"SELECT id FROM graph_nodes WHERE source_note = '' OR origin_url = '' LIMIT 10"
|
||||
)
|
||||
]
|
||||
if missing_notes:
|
||||
problems.append(f"graph: {len(missing_notes)} node(s) carry no source note")
|
||||
|
||||
log(
|
||||
f"verify: {store.chunk_count()} chunk(s) across {store.source_count()} source(s), "
|
||||
f"{total_points} vector(s) in Qdrant"
|
||||
f"{total_points} vector(s) in Qdrant, {counts['nodes']} node(s), {counts['edges']} edge(s)"
|
||||
)
|
||||
for warning in harvest.warnings:
|
||||
problems.append(f"estate warning: {warning}")
|
||||
return problems
|
||||
|
||||
|
||||
@@ -391,6 +620,8 @@ def _hit(row: dict) -> dict:
|
||||
return {
|
||||
"score": round(row["score"], 4),
|
||||
"area": payload.get("area"),
|
||||
"platform": payload.get("platform"),
|
||||
"tier": payload.get("tier"),
|
||||
"confidence": payload.get("confidence"),
|
||||
"title": payload.get("title"),
|
||||
"url": payload.get("url"),
|
||||
@@ -400,7 +631,7 @@ def _hit(row: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def hybrid_search(cfg, query: str, area: str = None, limit: int = 5) -> list:
|
||||
def hybrid_search(cfg, query: str, area: str = None, limit: int = 5, platform: str = None, tier: int = None) -> list:
|
||||
"""Vectors + FTS5 fused with reciprocal rank fusion (decision D10).
|
||||
|
||||
Phase 5 owns the query graph; this is here because Phase 4 is the phase that
|
||||
@@ -410,11 +641,15 @@ def hybrid_search(cfg, query: str, area: str = None, limit: int = 5) -> list:
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
vector = provider.embed([query])[0]
|
||||
ranked = {}
|
||||
for rank, hit in enumerate(qdrant.search(cfg.collection, vector, limit=limit * 4, area=area)):
|
||||
for rank, hit in enumerate(
|
||||
qdrant.search(cfg.collection, vector, limit=limit * 4, area=area, platform=platform, tier=tier)
|
||||
):
|
||||
entry = ranked.setdefault(hit["id"], {"id": hit["id"], "payload": hit["payload"], "score": 0.0})
|
||||
entry["score"] += 1.0 / (60 + rank)
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
for rank, hit in enumerate(store.keyword_search(query, area=area, limit=limit * 4)):
|
||||
for rank, hit in enumerate(
|
||||
store.keyword_search(query, area=area, platform=platform, tier=tier, limit=limit * 4)
|
||||
):
|
||||
entry = ranked.setdefault(hit["id"], {"id": hit["id"], "payload": {}, "score": 0.0})
|
||||
entry.setdefault("keyword", hit["snippet"])
|
||||
entry["score"] += 1.0 / (60 + rank)
|
||||
@@ -424,3 +659,21 @@ def hybrid_search(cfg, query: str, area: str = None, limit: int = 5) -> list:
|
||||
row["title"] = payload.get("title")
|
||||
row["url"] = payload.get("url")
|
||||
return ordered
|
||||
|
||||
|
||||
def scan_pool(cfg) -> dict:
|
||||
"""Run the credential scan without writing anything. Used by `make corpus-scan`."""
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
hits = []
|
||||
texts = store.pool_texts()
|
||||
for source_path, chunk_index, text in texts:
|
||||
for kind, line, _excerpt in exclusions.scan_text(text):
|
||||
hits.append(
|
||||
{"source_path": source_path, "chunk": chunk_index, "kind": kind, "line": line}
|
||||
)
|
||||
try:
|
||||
_scan_graph(store)
|
||||
except PipelineError as exc:
|
||||
hits.append({"source_path": "graph", "kind": "credential-shaped content", "detail": str(exc)})
|
||||
chunks = len(texts)
|
||||
return {"ok": not hits, "hits": hits, "chunks": chunks}
|
||||
|
||||
@@ -198,6 +198,8 @@ class Qdrant:
|
||||
limit: int = 5,
|
||||
area: str = None,
|
||||
confidence: str = None,
|
||||
platform: str = None,
|
||||
tier: int = None,
|
||||
with_payload: bool = True,
|
||||
) -> list:
|
||||
"""Nearest neighbours, optionally filtered *before* the search."""
|
||||
@@ -206,7 +208,7 @@ class Qdrant:
|
||||
"limit": int(limit),
|
||||
"with_payload": bool(with_payload),
|
||||
}
|
||||
filters = _filter_conditions(area, confidence)
|
||||
filters = _filter_conditions(area, confidence, platform, tier)
|
||||
if filters:
|
||||
body["filter"] = {"must": filters}
|
||||
try:
|
||||
@@ -260,10 +262,16 @@ class QdrantNotFound(QdrantError):
|
||||
"""A 404 from Qdrant: the collection or route does not exist."""
|
||||
|
||||
|
||||
def _filter_conditions(area: str = None, confidence: str = None) -> list:
|
||||
def _filter_conditions(
|
||||
area: str = None, confidence: str = None, platform: str = None, tier: int = None
|
||||
) -> list:
|
||||
conditions = []
|
||||
if area:
|
||||
conditions.append({"key": "area", "match": {"value": area}})
|
||||
if confidence:
|
||||
conditions.append({"key": "confidence", "match": {"value": confidence}})
|
||||
if platform:
|
||||
conditions.append({"key": "platform", "match": {"value": platform}})
|
||||
if tier is not None:
|
||||
conditions.append({"key": "tier", "match": {"value": int(tier)}})
|
||||
return conditions
|
||||
|
||||
848
corpus/sources/estate.py
Normal file
848
corpus/sources/estate.py
Normal file
@@ -0,0 +1,848 @@
|
||||
"""Harvest the estate into one set of documents.
|
||||
|
||||
Four sources, one shape (`loader.Document`), one tier rule:
|
||||
|
||||
| Platform | Tier | Reached by |
|
||||
|---|---|---|
|
||||
| `curated` (`content/`) | 1 | the local repository, already validated |
|
||||
| `gitea` | 2 | the Gitea API |
|
||||
| `obsidian` | 2 | the same Gitea API, one repository |
|
||||
| `graphify` | 2 | `graph.json` committed to the vault repository |
|
||||
|
||||
`content/` is loaded first and is never displaced: the query layer uses `tier`
|
||||
to prefer a curated claim over harvested text that disagrees with it.
|
||||
|
||||
Every refused file is recorded with the rule that refused it. The DoD asks for
|
||||
this explicitly ("the run reports the rejection"), and it is the difference
|
||||
between an exclusion list that works and one nobody can audit.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
|
||||
from .. import areas, exclusions, globs, guides, loader
|
||||
from . import graph as graph_module
|
||||
from .gitea import GiteaClient, GiteaError, GiteaNotFound, file_url, repo_url
|
||||
|
||||
PLATFORM_ORDER = ("curated", "gitea", "obsidian", "graphify")
|
||||
|
||||
|
||||
class EstateError(RuntimeError):
|
||||
"""The estate could not be read. The run stops rather than indexing half of it."""
|
||||
|
||||
|
||||
class Harvest:
|
||||
def __init__(self) -> None:
|
||||
self.documents: list = []
|
||||
self.rejections: list = []
|
||||
self.images: list = []
|
||||
self.warnings: list = []
|
||||
self.sources: dict = {}
|
||||
self.graph = None # graph_module.GraphResult | None
|
||||
self.offline = False
|
||||
|
||||
# -- reporting --------------------------------------------------------
|
||||
|
||||
def add_rejection(self, rejection) -> None:
|
||||
self.rejections.append(rejection.to_dict() if hasattr(rejection, "to_dict") else rejection)
|
||||
|
||||
def rejection_summary(self) -> dict:
|
||||
counts = {}
|
||||
for row in self.rejections:
|
||||
reason = str(row.get("reason") or "unknown")
|
||||
# Keep the summary readable: 200 distinct oversize byte counts are
|
||||
# one finding, not two hundred.
|
||||
if reason.startswith("oversize"):
|
||||
reason = "oversize"
|
||||
elif reason.startswith("binary"):
|
||||
reason = "binary"
|
||||
key = f"{row.get('platform')}/{reason}"
|
||||
counts[key] = counts.get(key, 0) + 1
|
||||
return dict(sorted(counts.items()))
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"documents": len(self.documents),
|
||||
"images": len(self.images),
|
||||
"rejections": len(self.rejections),
|
||||
"rejection_summary": self.rejection_summary(),
|
||||
"warnings": list(self.warnings),
|
||||
"sources": self.sources,
|
||||
"graph": self.graph.counts() if self.graph else None,
|
||||
"offline": self.offline,
|
||||
}
|
||||
|
||||
|
||||
def harvest(cfg, log=None) -> Harvest:
|
||||
"""Read every enabled platform. Curated content is always read."""
|
||||
log = log or (lambda _msg: None)
|
||||
result = Harvest()
|
||||
result.offline = bool(cfg.offline)
|
||||
|
||||
curated = loader.load_documents(cfg)
|
||||
result.documents.extend(curated)
|
||||
result.sources["curated"] = _source_row(
|
||||
"curated", enabled=True, documents=curated, extra={"path": "content/"}
|
||||
)
|
||||
log(f"curated: {len(curated)} document(s) from content/")
|
||||
|
||||
if cfg.offline:
|
||||
# Not a warning: offline is a deliberate mode, not a failure.
|
||||
log("offline mode: only content/ was read (RESUME_OFFLINE=1)")
|
||||
return result
|
||||
|
||||
all_guides = guides.load_guides(cfg.root)
|
||||
selected = {
|
||||
name: guide
|
||||
for name, guide in all_guides.items()
|
||||
if guide.enabled and name in cfg.platforms
|
||||
}
|
||||
missing = [name for name in cfg.platforms if name not in all_guides]
|
||||
if missing:
|
||||
result.warnings.append(f"platform(s) {missing} have no guide and were skipped")
|
||||
|
||||
needs_gitea = {"gitea", "obsidian", "graphify"} & set(selected)
|
||||
if not needs_gitea:
|
||||
return result
|
||||
|
||||
base = cfg.gitea_url or selected.get("gitea", all_guides.get("gitea")).base
|
||||
client = GiteaClient(base, cfg.gitea_token, timeout=cfg.timeout)
|
||||
try:
|
||||
repos = client.repos()
|
||||
except GiteaError as exc:
|
||||
raise EstateError(f"cannot list Gitea repositories: {exc}") from exc
|
||||
log(f"gitea: {len(repos)} repositories visible")
|
||||
|
||||
vault_repo = _vault_repo(selected)
|
||||
vault_tree = None
|
||||
if vault_repo:
|
||||
try:
|
||||
vault_tree = client.tree(vault_repo, "main")
|
||||
except GiteaNotFound:
|
||||
try:
|
||||
vault_tree = client.tree(vault_repo, "master")
|
||||
except GiteaError as exc:
|
||||
raise EstateError(f"cannot read {vault_repo}: {exc}") from exc
|
||||
except GiteaError as exc:
|
||||
raise EstateError(f"cannot read {vault_repo}: {exc}") from exc
|
||||
|
||||
if "gitea" in selected:
|
||||
_harvest_repos(result, client, selected["gitea"], repos, base, log)
|
||||
if "obsidian" in selected:
|
||||
_harvest_obsidian(result, client, selected["obsidian"], repos, vault_repo, vault_tree, cfg, base, log)
|
||||
if "graphify" in selected:
|
||||
_harvest_graph(result, client, selected["graphify"], repos, vault_repo, vault_tree, cfg, base, log)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# -- gitea repositories ---------------------------------------------------
|
||||
|
||||
|
||||
def _harvest_repos(result: Harvest, client, guide, repos, base_url, log) -> None:
|
||||
excluded = set(_as_list((guide.exclude or {}).get("repos")))
|
||||
include = (guide.include or {}).get("repos") or "all"
|
||||
wanted = None if include == "all" or _as_list(include) == [] else set(_as_list(include))
|
||||
|
||||
documents = []
|
||||
seen_repos = set()
|
||||
for repo in repos:
|
||||
full_name = repo.get("full_name") or ""
|
||||
if not full_name or full_name in excluded:
|
||||
continue
|
||||
if wanted is not None and full_name not in wanted:
|
||||
continue
|
||||
seen_repos.add(full_name)
|
||||
try:
|
||||
documents.extend(_harvest_one_repo(result, client, guide, repo, base_url))
|
||||
except GiteaError as exc:
|
||||
result.warnings.append(f"gitea/{full_name}: {exc}")
|
||||
continue
|
||||
if len(documents) >= guide.limit("max_documents"):
|
||||
result.warnings.append(
|
||||
f"gitea: document limit {guide.limit('max_documents')} reached; "
|
||||
"remaining repositories were skipped"
|
||||
)
|
||||
break
|
||||
|
||||
result.documents.extend(documents)
|
||||
result.sources["gitea"] = _source_row(
|
||||
"gitea", enabled=True, documents=documents,
|
||||
extra={"repos": len(seen_repos), "base": base_url},
|
||||
)
|
||||
log(f"gitea: {len(documents)} document(s), {len(result.rejections)} rejection(s) so far")
|
||||
|
||||
|
||||
def _harvest_one_repo(result: Harvest, client, guide, repo, base_url) -> list:
|
||||
full_name = repo["full_name"]
|
||||
branch = repo.get("default_branch") or "main"
|
||||
updated = _timestamp(repo.get("updated_at"))
|
||||
created = _timestamp(repo.get("created_at"))
|
||||
area = areas.area_for_repo(repo, guide.fits)
|
||||
topics = [str(topic) for topic in (repo.get("topics") or [])]
|
||||
description = str(repo.get("description") or "").strip()
|
||||
confidence = "abandoned" if repo.get("archived") else "historic"
|
||||
|
||||
documents = []
|
||||
try:
|
||||
entries = client.tree(full_name, branch)
|
||||
except GiteaNotFound:
|
||||
entries = []
|
||||
|
||||
blobs = [entry for entry in entries if entry.get("type") == "blob"]
|
||||
admitted = []
|
||||
for entry in blobs:
|
||||
path = globs.normalise(entry.get("path") or "")
|
||||
kind, reason = exclusions.classify(path, guide, size=entry.get("size"))
|
||||
if kind == "reject":
|
||||
result.add_rejection(exclusions.Rejection(path, "gitea", reason, full_name))
|
||||
continue
|
||||
if kind == "image":
|
||||
result.images.append(
|
||||
{"path": path, "platform": "gitea", "repo": full_name,
|
||||
"origin_url": file_url(base_url, full_name, branch, path)}
|
||||
)
|
||||
continue
|
||||
admitted.append(entry)
|
||||
|
||||
# The file tree is paths only. It is what lets a question like "does he have
|
||||
# a Prometheus config anywhere?" find a repository whose README never says so.
|
||||
tree_doc = _tree_document(guide, repo, area, updated, admitted, base_url)
|
||||
if tree_doc:
|
||||
documents.append(tree_doc)
|
||||
|
||||
# README first: it is the document a reader would open.
|
||||
readmes = [
|
||||
entry for entry in admitted
|
||||
if _is_readme(entry.get("path") or "") and globs.normalise(entry.get("path") or "").count("/") <= 1
|
||||
][:3]
|
||||
used = set()
|
||||
for entry in readmes:
|
||||
doc = _file_document(
|
||||
result, client, guide, repo, branch, area, updated, created,
|
||||
confidence, description, topics, entry, base_url,
|
||||
)
|
||||
if doc:
|
||||
documents.append(doc)
|
||||
used.add(globs.normalise(entry["path"]))
|
||||
|
||||
# Selected source files: the evidence behind the README's claims.
|
||||
candidates = [
|
||||
entry for entry in admitted
|
||||
if globs.normalise(entry.get("path") or "") not in used
|
||||
and not _is_readme(entry.get("path") or "")
|
||||
and _source_rank(entry.get("path") or "")
|
||||
]
|
||||
candidates.sort(key=lambda entry: (_depth(entry.get("path") or ""), globs.normalise(entry["path"])))
|
||||
for entry in candidates[: guide.limit("max_files_per_repo")]:
|
||||
doc = _file_document(
|
||||
result, client, guide, repo, branch, area, updated, created,
|
||||
confidence, description, topics, entry, base_url,
|
||||
)
|
||||
if doc:
|
||||
documents.append(doc)
|
||||
|
||||
# One document per repository for commits and one for languages. These are
|
||||
# short, stable, and answer "what is being worked on?" without a filesystem.
|
||||
commits = _commit_document(result, client, guide, repo, area, updated, created, confidence, base_url)
|
||||
if commits:
|
||||
documents.append(commits)
|
||||
languages = _language_document(result, client, guide, repo, area, updated, created, confidence, base_url)
|
||||
if languages:
|
||||
documents.append(languages)
|
||||
|
||||
if not any(not doc.source_id.endswith(("@tree", "@commits", "@languages")) for doc in documents):
|
||||
# No README and no admitted source file: still admit a summary so the
|
||||
# repository is discoverable and its silence is visible, not invisible.
|
||||
summary = _summary_document(guide, repo, area, updated, created, confidence, description, topics, base_url)
|
||||
if summary:
|
||||
documents.insert(0, summary)
|
||||
|
||||
return documents
|
||||
|
||||
|
||||
def _file_document(result, client, guide, repo, branch, area, updated, created, confidence, description, topics, entry, base_url) -> object:
|
||||
path = globs.normalise(entry.get("path") or "")
|
||||
full_name = repo["full_name"]
|
||||
try:
|
||||
raw = client.raw(full_name, branch, path)
|
||||
except GiteaError as exc:
|
||||
result.warnings.append(f"gitea/{full_name}/{path}: {exc}")
|
||||
return None
|
||||
if exclusions.looks_binary(raw):
|
||||
result.add_rejection(exclusions.Rejection(path, "gitea", "binary content", full_name))
|
||||
return None
|
||||
text = raw.decode("utf-8", "replace")
|
||||
if not text.strip():
|
||||
result.add_rejection(exclusions.Rejection(path, "gitea", "empty file", full_name))
|
||||
return None
|
||||
hits = exclusions.scan_text(text)
|
||||
if hits:
|
||||
kinds = sorted({hit[0] for hit in hits})
|
||||
result.add_rejection(
|
||||
exclusions.Rejection(path, "gitea", "credential-shaped content", full_name + ": " + ", ".join(kinds))
|
||||
)
|
||||
return None
|
||||
|
||||
kind = "code" if exclusions.extension(path) not in ("md", "markdown", "mdx", "txt", "rst") else "markdown"
|
||||
title = f"{full_name} — {path}"
|
||||
body = _strip_front_matter(text) if kind == "markdown" else text
|
||||
if not body.strip():
|
||||
result.add_rejection(exclusions.Rejection(path, "gitea", "front-matter only", full_name))
|
||||
return None
|
||||
if description or topics:
|
||||
header = [f"Repository: {full_name}"]
|
||||
if description:
|
||||
header.append(description)
|
||||
if topics:
|
||||
header.append("Topics: " + ", ".join(topics))
|
||||
body = "\n\n".join(["\n".join(header), text])
|
||||
return _make_document(
|
||||
key=f"gitea:{full_name}/{path}",
|
||||
platform="gitea",
|
||||
area=area,
|
||||
title=title,
|
||||
claim=_claim(text, title),
|
||||
body=body,
|
||||
digest=hashlib.sha256(raw).hexdigest(),
|
||||
origin_url=file_url(base_url, full_name, branch, path),
|
||||
path=path,
|
||||
repo_path=path,
|
||||
source_id=f"{full_name}/{path}",
|
||||
source_type="code" if kind == "code" else "doc",
|
||||
last_modified=updated,
|
||||
confidence=confidence,
|
||||
period=_period(created, updated),
|
||||
tags=topics + ["gitea", area],
|
||||
tech=[],
|
||||
public=guide.public,
|
||||
kind=kind,
|
||||
)
|
||||
|
||||
|
||||
def _tree_document(guide, repo, area, updated, blobs, base_url) -> object:
|
||||
if not blobs:
|
||||
return None
|
||||
full_name = repo["full_name"]
|
||||
branch = repo.get("default_branch") or "main"
|
||||
limit = guide.limit("max_tree_entries")
|
||||
paths = [globs.normalise(entry.get("path") or "") for entry in blobs][:limit]
|
||||
body = "\n".join(
|
||||
[f"Repository: {full_name}", f"Branch: {branch}", f"Files: {len(blobs)}", "", "File tree:"]
|
||||
+ [f"- {path}" for path in paths]
|
||||
)
|
||||
if len(blobs) > limit:
|
||||
body += f"\n- … {len(blobs) - limit} more"
|
||||
digest = hashlib.sha256(body.encode("utf-8")).hexdigest()
|
||||
return _make_document(
|
||||
key=f"gitea:{full_name}@tree",
|
||||
platform="gitea",
|
||||
area=area,
|
||||
title=f"{full_name} — file tree",
|
||||
claim=f"File tree of {full_name}, {len(blobs)} files.",
|
||||
body=body,
|
||||
digest=digest,
|
||||
origin_url=repo_url(base_url, full_name),
|
||||
path="",
|
||||
repo_path="",
|
||||
source_id=f"{full_name}@tree",
|
||||
source_type="doc",
|
||||
last_modified=updated,
|
||||
confidence="abandoned" if repo.get("archived") else "historic",
|
||||
period="",
|
||||
tags=["gitea", "tree", area],
|
||||
tech=[],
|
||||
public=guide.public,
|
||||
kind="markdown",
|
||||
)
|
||||
|
||||
|
||||
def _commit_document(result, client, guide, repo, area, updated, created, confidence, base_url) -> object:
|
||||
full_name = repo["full_name"]
|
||||
branch = repo.get("default_branch") or "main"
|
||||
try:
|
||||
commits = client.commits(full_name, branch, limit=guide.limit("max_commits"))
|
||||
except GiteaError:
|
||||
return None
|
||||
if not commits:
|
||||
return None
|
||||
lines = [f"Repository: {full_name}", f"Recent commits on {branch}:", ""]
|
||||
newest = updated
|
||||
for item in commits:
|
||||
commit = item.get("commit") or {}
|
||||
message = (commit.get("message") or "").strip().splitlines()
|
||||
headline = message[0] if message else "(no message)"
|
||||
date = _timestamp(((commit.get("author") or {}).get("date")) or ((commit.get("committer") or {}).get("date")))
|
||||
if date:
|
||||
newest = max(newest, date)
|
||||
lines.append(f"- {date[:10]} {headline}")
|
||||
body = "\n".join(lines)
|
||||
digest = hashlib.sha256(body.encode("utf-8")).hexdigest()
|
||||
return _make_document(
|
||||
key=f"gitea:{full_name}@commits",
|
||||
platform="gitea",
|
||||
area=area,
|
||||
title=f"{full_name} — recent commits",
|
||||
claim=f"Recent commit history for {full_name}.",
|
||||
body=body,
|
||||
digest=digest,
|
||||
origin_url=f"{repo_url(base_url, full_name)}/commits/branch/{branch}",
|
||||
path="",
|
||||
repo_path="",
|
||||
source_id=f"{full_name}@commits",
|
||||
source_type="commit",
|
||||
last_modified=newest,
|
||||
confidence=confidence,
|
||||
period=_period(created, newest),
|
||||
tags=["gitea", "commits", area],
|
||||
tech=[],
|
||||
public=guide.public,
|
||||
kind="markdown",
|
||||
)
|
||||
|
||||
|
||||
def _language_document(result, client, guide, repo, area, updated, created, confidence, base_url) -> object:
|
||||
full_name = repo["full_name"]
|
||||
try:
|
||||
languages = client.languages(full_name)
|
||||
except GiteaError:
|
||||
return None
|
||||
if not languages:
|
||||
return None
|
||||
total = sum(int(value) for value in languages.values()) or 1
|
||||
rows = sorted(languages.items(), key=lambda item: int(item[1]), reverse=True)
|
||||
lines = [f"Repository: {full_name}", "", "Language statistics:"]
|
||||
for name, size in rows:
|
||||
lines.append(f"- {name}: {int(size)} bytes ({100.0 * int(size) / total:.1f}%)")
|
||||
body = "\n".join(lines)
|
||||
digest = hashlib.sha256(body.encode("utf-8")).hexdigest()
|
||||
return _make_document(
|
||||
key=f"gitea:{full_name}@languages",
|
||||
platform="gitea",
|
||||
area=area,
|
||||
title=f"{full_name} — languages",
|
||||
claim=f"Language breakdown for {full_name}: " + ", ".join(name for name, _ in rows[:4]) + ".",
|
||||
body=body,
|
||||
digest=digest,
|
||||
origin_url=repo_url(base_url, full_name),
|
||||
path="",
|
||||
repo_path="",
|
||||
source_id=f"{full_name}@languages",
|
||||
source_type="metric",
|
||||
last_modified=updated,
|
||||
confidence=confidence,
|
||||
period=_period(created, updated),
|
||||
tags=["gitea", "languages", area],
|
||||
tech=[name for name, _ in rows[:6]],
|
||||
public=guide.public,
|
||||
kind="markdown",
|
||||
)
|
||||
|
||||
|
||||
def _summary_document(guide, repo, area, updated, created, confidence, description, topics, base_url) -> object:
|
||||
full_name = repo["full_name"]
|
||||
body = "\n".join(
|
||||
[f"Repository: {full_name}", f"Description: {description or '(none)'}",
|
||||
f"Topics: {', '.join(topics) if topics else '(none)'}"]
|
||||
)
|
||||
digest = hashlib.sha256(body.encode("utf-8")).hexdigest()
|
||||
return _make_document(
|
||||
key=f"gitea:{full_name}@summary",
|
||||
platform="gitea",
|
||||
area=area,
|
||||
title=full_name,
|
||||
claim=description or f"Repository {full_name}.",
|
||||
body=body,
|
||||
digest=digest,
|
||||
origin_url=repo_url(base_url, full_name),
|
||||
path="",
|
||||
repo_path="",
|
||||
source_id=f"{full_name}@summary",
|
||||
source_type="doc",
|
||||
last_modified=updated,
|
||||
confidence=confidence,
|
||||
period=_period(created, updated),
|
||||
tags=topics + ["gitea", area],
|
||||
tech=[],
|
||||
public=guide.public,
|
||||
kind="markdown",
|
||||
)
|
||||
|
||||
|
||||
# -- obsidian vault -------------------------------------------------------
|
||||
|
||||
|
||||
def _harvest_obsidian(result, client, guide, repos, vault_repo, tree, cfg, base_url, log) -> None:
|
||||
if not vault_repo or tree is None:
|
||||
result.warnings.append("obsidian: the vault repository could not be listed")
|
||||
result.sources["obsidian"] = _source_row("obsidian", enabled=True, documents=[], extra={"missing": True})
|
||||
return
|
||||
repo = next((row for row in repos if row.get("full_name") == vault_repo), {})
|
||||
branch = repo.get("default_branch") or "main"
|
||||
updated = _timestamp(repo.get("updated_at"))
|
||||
created = _timestamp(repo.get("created_at"))
|
||||
validator = loader.load_validator(cfg.root)
|
||||
documents = []
|
||||
|
||||
blobs = [entry for entry in tree if entry.get("type") == "blob"]
|
||||
for entry in blobs:
|
||||
if len(documents) >= guide.limit("max_documents"):
|
||||
result.warnings.append(
|
||||
f"obsidian: document limit {guide.limit('max_documents')} reached"
|
||||
)
|
||||
break
|
||||
path = globs.normalise(entry.get("path") or "")
|
||||
kind, reason = exclusions.classify(path, guide, size=entry.get("size"))
|
||||
if kind == "reject":
|
||||
# A path outside the include list is expected and is not a finding;
|
||||
# only hard exclusions are reported, so the report stays readable.
|
||||
if reason != "not under an included path":
|
||||
result.add_rejection(exclusions.Rejection(path, "obsidian", reason))
|
||||
continue
|
||||
if kind == "image":
|
||||
result.images.append(
|
||||
{"path": path, "platform": "obsidian",
|
||||
"origin_url": file_url(base_url, vault_repo, branch, path)}
|
||||
)
|
||||
continue
|
||||
try:
|
||||
raw = client.raw(vault_repo, branch, path)
|
||||
except GiteaError as exc:
|
||||
result.warnings.append(f"obsidian/{path}: {exc}")
|
||||
continue
|
||||
if exclusions.looks_binary(raw):
|
||||
result.add_rejection(exclusions.Rejection(path, "obsidian", "binary content"))
|
||||
continue
|
||||
text = raw.decode("utf-8", "replace")
|
||||
if not text.strip():
|
||||
result.add_rejection(exclusions.Rejection(path, "obsidian", "empty file"))
|
||||
continue
|
||||
hits = exclusions.scan_text(text)
|
||||
if hits:
|
||||
result.add_rejection(
|
||||
exclusions.Rejection(
|
||||
path, "obsidian", "credential-shaped content",
|
||||
", ".join(sorted({hit[0] for hit in hits})),
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
fields, body = _split_note(text, validator)
|
||||
if not body.strip():
|
||||
result.add_rejection(exclusions.Rejection(path, "obsidian", "empty note"))
|
||||
continue
|
||||
title = str(fields.get("title") or _heading(body) or os.path.splitext(os.path.basename(path))[0])
|
||||
tag_values = fields.get("tags") or []
|
||||
if isinstance(tag_values, str):
|
||||
tag_values = [tag_values]
|
||||
tags = [str(tag) for tag in tag_values] + _folder_tags(path)
|
||||
area = areas.area_for_note(path, guide.fits)
|
||||
documents.append(
|
||||
_make_document(
|
||||
key=f"obsidian:{path}",
|
||||
platform="obsidian",
|
||||
area=area,
|
||||
title=title,
|
||||
claim=_claim(body, title),
|
||||
body=body,
|
||||
digest=hashlib.sha256(raw).hexdigest(),
|
||||
origin_url=file_url(base_url, vault_repo, branch, path),
|
||||
path=path,
|
||||
repo_path=path,
|
||||
source_id=path,
|
||||
source_type="doc",
|
||||
last_modified=updated,
|
||||
confidence="historic",
|
||||
period=_period(created, updated),
|
||||
tags=tags + ["obsidian", area],
|
||||
tech=[],
|
||||
public=guide.public,
|
||||
kind="markdown",
|
||||
)
|
||||
)
|
||||
|
||||
result.documents.extend(documents)
|
||||
result.sources["obsidian"] = _source_row(
|
||||
"obsidian", enabled=True, documents=documents,
|
||||
extra={"repo": vault_repo, "base": base_url},
|
||||
)
|
||||
log(f"obsidian: {len(documents)} note(s)")
|
||||
|
||||
|
||||
# -- graphify -------------------------------------------------------------
|
||||
|
||||
|
||||
def _harvest_graph(result, client, guide, repos, vault_repo, tree, cfg, base_url, log) -> None:
|
||||
if not vault_repo:
|
||||
result.warnings.append("graphify: no vault repository configured in the guide")
|
||||
result.sources["graphify"] = _source_row("graphify", enabled=True, documents=[], extra={"missing": True})
|
||||
return
|
||||
repo = next((row for row in repos if row.get("full_name") == vault_repo), {})
|
||||
branch = repo.get("default_branch") or "main"
|
||||
updated = _timestamp(repo.get("updated_at"))
|
||||
vault_paths = {
|
||||
globs.normalise(entry.get("path") or "")
|
||||
for entry in (tree or [])
|
||||
if entry.get("type") == "blob"
|
||||
}
|
||||
|
||||
raw = None
|
||||
source_path = ""
|
||||
if cfg.graph_path:
|
||||
if not os.path.exists(cfg.graph_path):
|
||||
raise EstateError(f"RESUME_GRAPH_PATH is set but {cfg.graph_path} does not exist")
|
||||
with open(cfg.graph_path, "rb") as handle:
|
||||
raw = handle.read()
|
||||
source_path = cfg.graph_path
|
||||
else:
|
||||
for candidate in guide.limits.get("graph_paths") or ["graph.json"]:
|
||||
try:
|
||||
raw = client.raw(vault_repo, branch, candidate)
|
||||
source_path = f"gitea:{vault_repo}/{candidate}"
|
||||
break
|
||||
except GiteaNotFound:
|
||||
continue
|
||||
except GiteaError as exc:
|
||||
raise EstateError(f"cannot read graph.json from {vault_repo}: {exc}") from exc
|
||||
|
||||
if raw is None:
|
||||
result.warnings.append(
|
||||
f"graphify: graph.json is not committed to {vault_repo} "
|
||||
f"(looked in {guide.limits.get('graph_paths')}). The graph was not ingested; "
|
||||
"the nightly audit will keep reporting this until it is committed."
|
||||
)
|
||||
result.sources["graphify"] = _source_row(
|
||||
"graphify", enabled=True, documents=[], extra={"missing": True, "repo": vault_repo}
|
||||
)
|
||||
return
|
||||
|
||||
graph = graph_module.parse(
|
||||
raw,
|
||||
platform="graphify",
|
||||
base=base_url,
|
||||
repo=vault_repo,
|
||||
ref=branch,
|
||||
last_modified=updated,
|
||||
area=guide.primary_area,
|
||||
vault_paths=vault_paths,
|
||||
max_nodes=guide.limit("max_nodes"),
|
||||
max_edges=guide.limit("max_edges"),
|
||||
source_path=source_path,
|
||||
)
|
||||
result.graph = graph
|
||||
for rejection in graph.rejections:
|
||||
result.add_rejection(rejection)
|
||||
result.images.extend(graph.images)
|
||||
result.sources["graphify"] = _source_row(
|
||||
"graphify", enabled=True, documents=[],
|
||||
extra={
|
||||
"repo": vault_repo,
|
||||
"path": source_path,
|
||||
"nodes": len(graph.nodes),
|
||||
"edges": len(graph.edges),
|
||||
"digest": graph.digest,
|
||||
"built_at_commit": graph.built_at_commit,
|
||||
},
|
||||
)
|
||||
if graph.warnings:
|
||||
result.warnings.extend(f"graphify: {warning}" for warning in graph.warnings)
|
||||
log(
|
||||
f"graphify: {len(graph.nodes)} node(s), {len(graph.edges)} edge(s), "
|
||||
f"{len(graph.images)} image node(s)"
|
||||
)
|
||||
|
||||
|
||||
# -- shared helpers -------------------------------------------------------
|
||||
|
||||
|
||||
def _make_document(**values) -> loader.Document:
|
||||
"""One Document constructor, so every source produces the same shape."""
|
||||
defaults = dict(
|
||||
metrics=[], artefacts=[], tech=[], tags=[], public=True, kind="markdown",
|
||||
platform="gitea", tier=2, level=3, period="", source_type="doc",
|
||||
)
|
||||
defaults.update(values)
|
||||
title = defaults["title"]
|
||||
origin = defaults["origin_url"]
|
||||
path = defaults.get("path") or ""
|
||||
return loader.Document(
|
||||
path=defaults["key"],
|
||||
source_id=defaults.get("source_id") or (path or defaults["key"]),
|
||||
repo_path=defaults.get("repo_path") or path,
|
||||
area=defaults["area"],
|
||||
level=defaults.get("level", 3),
|
||||
title=title,
|
||||
claim=defaults.get("claim") or title,
|
||||
confidence=defaults.get("confidence") or "historic",
|
||||
period=defaults.get("period") or "—",
|
||||
tech=defaults.get("tech") or [],
|
||||
metrics=defaults.get("metrics") or [],
|
||||
artefacts=defaults.get("artefacts") or [],
|
||||
public=bool(defaults.get("public", True)),
|
||||
last_verified=str(defaults["last_modified"])[:10],
|
||||
tags=defaults.get("tags") or [],
|
||||
body=defaults["body"],
|
||||
digest=defaults["digest"],
|
||||
url=origin,
|
||||
platform=defaults["platform"],
|
||||
tier=int(defaults.get("tier", 2)),
|
||||
origin_url=origin,
|
||||
last_modified=defaults["last_modified"],
|
||||
source_type=defaults.get("source_type") or "doc",
|
||||
kind=defaults.get("kind") or "markdown",
|
||||
)
|
||||
|
||||
|
||||
def _strip_front_matter(text: str) -> str:
|
||||
"""Drop leading YAML blocks. The Note path parses them; files only need them gone.
|
||||
|
||||
Some notes carry two blocks in a row (a stale export, or a block followed by
|
||||
a horizontal rule that closes), so this repeats. It never strips a body that
|
||||
does not have a closing delimiter.
|
||||
"""
|
||||
body = text
|
||||
for _ in range(3):
|
||||
lines = body.splitlines()
|
||||
if not lines or lines[0].strip() != "---":
|
||||
return body
|
||||
closing = None
|
||||
for index in range(1, len(lines)):
|
||||
if lines[index].strip() in ("---", "..."):
|
||||
closing = index
|
||||
break
|
||||
if closing is None:
|
||||
return body
|
||||
body = "\n".join(lines[closing + 1 :]).strip()
|
||||
return body
|
||||
|
||||
|
||||
def _split_note(text: str, validator) -> tuple:
|
||||
"""Tolerant front-matter split. A note without front-matter is still a note."""
|
||||
lines = text.splitlines()
|
||||
if not lines or lines[0].strip() != "---":
|
||||
return {}, text
|
||||
end = None
|
||||
for index in range(1, len(lines)):
|
||||
if lines[index].strip() in ("---", "..."):
|
||||
end = index
|
||||
break
|
||||
if end is None:
|
||||
return {}, text
|
||||
fields = {}
|
||||
try:
|
||||
node = validator.parse_front_matter(
|
||||
[(number, line) for number, line in enumerate(lines[1:end], start=2)]
|
||||
)
|
||||
parsed = loader.to_python(node)
|
||||
if isinstance(parsed, dict):
|
||||
fields = parsed
|
||||
except validator.YamlError:
|
||||
fields = {}
|
||||
# Strip any further leading block as well, so a duplicate front-matter block
|
||||
# never reaches the chunker.
|
||||
return fields, _strip_front_matter("\n".join(lines[end + 1 :]).strip())
|
||||
|
||||
|
||||
def _source_row(name: str, enabled: bool, documents: list, extra: dict = None) -> dict:
|
||||
stamps = sorted(doc.last_modified for doc in documents if doc.last_modified)
|
||||
row = {
|
||||
"platform": name,
|
||||
"enabled": enabled,
|
||||
"documents": len(documents),
|
||||
"newest": stamps[-1] if stamps else "",
|
||||
"oldest": stamps[0] if stamps else "",
|
||||
"fetched_at": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
}
|
||||
row.update(extra or {})
|
||||
return row
|
||||
|
||||
|
||||
def _vault_repo(selected: dict) -> str:
|
||||
for name in ("obsidian", "graphify"):
|
||||
guide = selected.get(name)
|
||||
if guide and guide.base and "/" in guide.base and not guide.base.startswith("http"):
|
||||
return guide.base
|
||||
return ""
|
||||
|
||||
|
||||
def _as_list(value) -> list:
|
||||
if value in (None, ""):
|
||||
return []
|
||||
if isinstance(value, list):
|
||||
return [str(item) for item in value]
|
||||
return [str(value)]
|
||||
|
||||
|
||||
def _timestamp(value) -> str:
|
||||
"""Normalise any RFC3339 timestamp to UTC `…Z`. Empty stays empty."""
|
||||
if not value:
|
||||
return "1970-01-01T00:00:00Z"
|
||||
text = str(value).strip()
|
||||
if text.endswith("Z"):
|
||||
text = text[:-1] + "+00:00"
|
||||
try:
|
||||
moment = datetime.datetime.fromisoformat(text)
|
||||
except ValueError:
|
||||
return "1970-01-01T00:00:00Z"
|
||||
if moment.tzinfo is None:
|
||||
moment = moment.replace(tzinfo=datetime.timezone.utc)
|
||||
return moment.astimezone(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def _period(created: str, updated: str) -> str:
|
||||
start = (created or "")[:4]
|
||||
end = (updated or "")[:4]
|
||||
if not start or start == "1970":
|
||||
return end or "—"
|
||||
if start == end:
|
||||
return start
|
||||
return f"{start}–{end}"
|
||||
|
||||
|
||||
def _claim(text: str, title: str) -> str:
|
||||
for line in (text or "").splitlines():
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith(("#", "|", "```", "---", ">")):
|
||||
continue
|
||||
cleaned = re.sub(r"[*_`\[\]]", "", stripped)
|
||||
if len(cleaned) > 20:
|
||||
return cleaned[:180]
|
||||
return title[:180]
|
||||
|
||||
|
||||
def _heading(text: str) -> str:
|
||||
for line in (text or "").splitlines():
|
||||
if line.startswith("# "):
|
||||
return line[2:].strip()
|
||||
return ""
|
||||
|
||||
|
||||
def _folder_tags(path: str) -> list:
|
||||
parts = [part for part in path.split("/")[:-1] if part and not part[0].isdigit()]
|
||||
return [part.lower() for part in parts[-2:]]
|
||||
|
||||
|
||||
def _is_readme(path: str) -> bool:
|
||||
return os.path.basename(path).lower().startswith("readme")
|
||||
|
||||
|
||||
def _depth(path: str) -> int:
|
||||
return globs.normalise(path).count("/")
|
||||
|
||||
|
||||
def _source_rank(path: str) -> bool:
|
||||
ext = exclusions.extension(path)
|
||||
if ext not in exclusions.TEXT_EXTENSIONS:
|
||||
return False
|
||||
if ext in ("json", "csv", "tsv", "svg"):
|
||||
return False
|
||||
# Prefer files that carry decisions over generated or vendored files.
|
||||
return not any(
|
||||
part.lower() in ("vendor", "third_party", "examples", "testdata", "fixtures", "migrations")
|
||||
for part in globs.normalise(path).split("/")[:-1]
|
||||
)
|
||||
165
corpus/sources/gitea.py
Normal file
165
corpus/sources/gitea.py
Normal file
@@ -0,0 +1,165 @@
|
||||
"""Gitea API client — the only network dependency in the pipeline.
|
||||
|
||||
`docs/phase-4-corpus.md` §1: Gitea is reached over HTTPS, and the vault and the
|
||||
graph arrive through the same API, so there is no cross-machine copy and no SSH
|
||||
from the runtime host. This client is standard library only (`urllib`), it never
|
||||
prints or stores the token, and every call names only the path it requested.
|
||||
|
||||
Read-only by construction: there is no method here that writes, creates, deletes
|
||||
or changes anything in Gitea. The token only needs read scope.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import http.client
|
||||
import json
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
|
||||
class GiteaError(RuntimeError):
|
||||
"""A Gitea call failed. The message never contains the token."""
|
||||
|
||||
|
||||
class GiteaNotFound(GiteaError):
|
||||
"""The requested repository, branch or file does not exist."""
|
||||
|
||||
|
||||
class GiteaClient:
|
||||
def __init__(self, base: str, token: str, timeout: float = 30.0, attempts: int = 3) -> None:
|
||||
if not base:
|
||||
raise GiteaError("a Gitea base URL is required (GITEA_URL or the guide's base)")
|
||||
if not token:
|
||||
raise GiteaError(
|
||||
"a Gitea token is required. It is read from the environment variable "
|
||||
"named by the platform guide (GITEA_TOKEN) and never from this repository."
|
||||
)
|
||||
self.base = base.rstrip("/")
|
||||
self._token = token
|
||||
self.timeout = timeout
|
||||
self.attempts = max(1, attempts)
|
||||
self.calls = 0
|
||||
|
||||
# -- transport --------------------------------------------------------
|
||||
|
||||
def _request(self, path: str, expect_bytes: bool = False):
|
||||
"""One GET with retry/backoff. The only place that touches the network."""
|
||||
url = f"{self.base}/api/v1{path}"
|
||||
last_error = None
|
||||
for attempt in range(self.attempts):
|
||||
request = urllib.request.Request(
|
||||
url,
|
||||
method="GET",
|
||||
headers={
|
||||
"Authorization": f"token {self._token}",
|
||||
"Accept": "application/vnd.gitea+json, application/json",
|
||||
"User-Agent": "ai-resume-corpus/1",
|
||||
},
|
||||
)
|
||||
try:
|
||||
self.calls += 1
|
||||
with urllib.request.urlopen(request, timeout=self.timeout) as response:
|
||||
raw = response.read()
|
||||
if expect_bytes:
|
||||
return raw
|
||||
return json.loads(raw.decode("utf-8")) if raw.strip() else {}
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", "replace")[:200]
|
||||
exc.close()
|
||||
if exc.code == 404:
|
||||
raise GiteaNotFound(f"GET {path}: not found") from None
|
||||
if exc.code == 401:
|
||||
raise GiteaError(
|
||||
f"GET {path}: authentication failed. Check the token named by the "
|
||||
"platform guide; it is not printed here."
|
||||
) from None
|
||||
if exc.code == 403:
|
||||
raise GiteaError(f"GET {path}: forbidden (the token may lack read scope)") from None
|
||||
if exc.code != 429 and exc.code < 500:
|
||||
raise GiteaError(f"GET {path}: HTTP {exc.code} {detail}") from None
|
||||
last_error = f"HTTP {exc.code}"
|
||||
except (urllib.error.URLError, TimeoutError, http.client.HTTPException, ConnectionError) as exc:
|
||||
last_error = f"cannot reach {self.base}: {getattr(exc, 'reason', exc)}"
|
||||
if attempt + 1 < self.attempts:
|
||||
time.sleep(0.5 * (2 ** attempt))
|
||||
raise GiteaError(f"GET {path} failed after {self.attempts} attempts: {last_error}")
|
||||
|
||||
# -- repositories -----------------------------------------------------
|
||||
|
||||
def repos(self, limit: int = 50) -> list:
|
||||
"""Every repository the token can read, sorted by full name."""
|
||||
out = []
|
||||
page = 1
|
||||
while True:
|
||||
response = self._request(f"/repos/search?limit={limit}&page={page}")
|
||||
rows = response.get("data") or []
|
||||
out.extend(rows)
|
||||
if len(rows) < limit:
|
||||
break
|
||||
page += 1
|
||||
if page > 200: # a guard, not a limit anyone should reach
|
||||
break
|
||||
return sorted(out, key=lambda repo: repo.get("full_name", ""))
|
||||
|
||||
def repo(self, full_name: str) -> dict:
|
||||
return self._request(f"/repos/{_repo_path(full_name)}")
|
||||
|
||||
# -- contents ---------------------------------------------------------
|
||||
|
||||
def tree(self, full_name: str, ref: str) -> list:
|
||||
"""Recursive blob/tree listing. Returns every entry, following pages."""
|
||||
entries = []
|
||||
page = 1
|
||||
while True:
|
||||
response = self._request(
|
||||
f"/repos/{_repo_path(full_name)}/git/trees/{urllib.parse.quote(ref, safe='')}"
|
||||
f"?recursive=true&per_page=1000&page={page}"
|
||||
)
|
||||
rows = response.get("tree") or []
|
||||
entries.extend(rows)
|
||||
if not rows or not response.get("truncated") or len(rows) < 1000:
|
||||
break
|
||||
page += 1
|
||||
if page > 50:
|
||||
break
|
||||
return entries
|
||||
|
||||
def raw(self, full_name: str, ref: str, path: str) -> bytes:
|
||||
quoted = urllib.parse.quote(path, safe="/")
|
||||
return self._request(
|
||||
f"/repos/{_repo_path(full_name)}/raw/{quoted}?ref={urllib.parse.quote(ref, safe='')}",
|
||||
expect_bytes=True,
|
||||
)
|
||||
|
||||
def commits(self, full_name: str, ref: str, limit: int = 20) -> list:
|
||||
return self._request(
|
||||
f"/repos/{_repo_path(full_name)}/commits?sha={urllib.parse.quote(ref, safe='')}"
|
||||
f"&limit={int(limit)}&page=1"
|
||||
)
|
||||
|
||||
def languages(self, full_name: str) -> dict:
|
||||
return self._request(f"/repos/{_repo_path(full_name)}/languages")
|
||||
|
||||
# -- presentation -----------------------------------------------------
|
||||
|
||||
def describe(self) -> dict:
|
||||
return {"base": self.base, "token": "set", "calls": self.calls}
|
||||
|
||||
|
||||
def file_url(base: str, full_name: str, ref: str, path: str) -> str:
|
||||
"""The URL a reader can open: the file at its branch, in Gitea."""
|
||||
return (
|
||||
f"{base.rstrip('/')}/{full_name}/src/branch/{urllib.parse.quote(ref, safe='')}/"
|
||||
f"{urllib.parse.quote(path, safe='/')}"
|
||||
)
|
||||
|
||||
|
||||
def repo_url(base: str, full_name: str) -> str:
|
||||
return f"{base.rstrip('/')}/{full_name}"
|
||||
|
||||
|
||||
def _repo_path(full_name: str) -> str:
|
||||
owner, _, name = full_name.partition("/")
|
||||
return f"{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(name, safe='')}"
|
||||
335
corpus/sources/graph.py
Normal file
335
corpus/sources/graph.py
Normal file
@@ -0,0 +1,335 @@
|
||||
"""Graphify output: nodes and edges, each joined back to the note it came from.
|
||||
|
||||
`docs/phase-4-corpus.md` §4: *"Graph is not chunked. Nodes and edges go into
|
||||
their own collection, joined to note chunks by `path`, so a graph answer can name
|
||||
the note it came from."*
|
||||
|
||||
The graph is not rebuilt here. It already exists and is rebuilt on the build host
|
||||
by its own skill; this module consumes `graph.json`.
|
||||
|
||||
Two rules from §3 drive the whole file:
|
||||
|
||||
* **A node with no resolvable source note is refused, not defaulted.** Graphify
|
||||
emits nodes for files outside the vault (`settings.json`, `.pi/…`). Those are
|
||||
not traceable to a note, so they are rejected and counted.
|
||||
* **Media never becomes text.** A media node becomes an image reference: path,
|
||||
origin URL and note, and nothing else.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from .. import areas, exclusions, globs
|
||||
from ..sources.gitea import file_url
|
||||
|
||||
|
||||
class GraphError(RuntimeError):
|
||||
"""graph.json is not a graph this pipeline can admit."""
|
||||
|
||||
|
||||
class GraphResult:
|
||||
def __init__(self, platform: str = "graphify"):
|
||||
self.platform = platform
|
||||
self.nodes: list = []
|
||||
self.edges: list = []
|
||||
self.images: list = []
|
||||
self.rejections: list = []
|
||||
self.warnings: list = []
|
||||
self.digest = ""
|
||||
self.source_path = ""
|
||||
self.built_at_commit = ""
|
||||
|
||||
def counts(self) -> dict:
|
||||
return {
|
||||
"nodes": len(self.nodes),
|
||||
"edges": len(self.edges),
|
||||
"images": len(self.images),
|
||||
"rejected": len(self.rejections),
|
||||
}
|
||||
|
||||
|
||||
def parse(
|
||||
raw: bytes,
|
||||
*,
|
||||
platform: str,
|
||||
base: str,
|
||||
repo: str,
|
||||
ref: str,
|
||||
last_modified: str,
|
||||
area: str,
|
||||
vault_paths: set,
|
||||
max_nodes: int = 20000,
|
||||
max_edges: int = 40000,
|
||||
source_path: str = "",
|
||||
) -> GraphResult:
|
||||
"""Turn graph.json bytes into rows that carry their note. Never guesses."""
|
||||
result = GraphResult(platform)
|
||||
result.digest = hashlib.sha256(raw).hexdigest()
|
||||
result.source_path = source_path
|
||||
try:
|
||||
data = json.loads(raw.decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise GraphError(f"graph.json is not valid UTF-8 JSON: {exc}") from exc
|
||||
if not isinstance(data, dict):
|
||||
raise GraphError("graph.json must be a JSON object")
|
||||
result.built_at_commit = str(data.get("built_at_commit") or "")
|
||||
|
||||
raw_nodes = data.get("nodes") or []
|
||||
raw_links = data.get("links") or data.get("edges") or []
|
||||
if not isinstance(raw_nodes, list) or not isinstance(raw_links, list):
|
||||
raise GraphError("graph.json must contain `nodes` and `links` arrays")
|
||||
|
||||
labels = {}
|
||||
notes = {}
|
||||
for item in raw_nodes:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
node_id = str(item.get("id") or "").strip()
|
||||
if not node_id:
|
||||
continue
|
||||
labels[node_id] = str(item.get("label") or node_id)
|
||||
notes[node_id] = _note_for(item, vault_paths)
|
||||
|
||||
for item in raw_nodes:
|
||||
if len(result.nodes) >= max_nodes:
|
||||
result.warnings.append(f"graph node limit {max_nodes} reached; remaining nodes skipped")
|
||||
break
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
node_id = str(item.get("id") or "").strip()
|
||||
if not node_id:
|
||||
result.rejections.append(
|
||||
_rejection("graph.json", platform, "graph node has no id")
|
||||
)
|
||||
continue
|
||||
source_file = globs.normalise(item.get("source_file") or "")
|
||||
note = notes.get(node_id)
|
||||
if not note:
|
||||
result.rejections.append(
|
||||
_rejection(
|
||||
source_file or f"node:{node_id}", platform,
|
||||
"no resolvable source note", "graph node does not point at a file in the vault",
|
||||
)
|
||||
)
|
||||
continue
|
||||
if globs.normalise(source_file) != note:
|
||||
# A node can be derived from one note while naming another file; the
|
||||
# note is the traceable origin either way.
|
||||
pass
|
||||
if exclusions.classify(note, None)[0] == "image" or exclusions.extension(note) in exclusions.MEDIA_EXTENSIONS:
|
||||
result.images.append(
|
||||
{
|
||||
"path": note,
|
||||
"label": str(item.get("label") or node_id),
|
||||
"origin_url": file_url(base, repo, ref, note),
|
||||
"platform": platform,
|
||||
"note": note,
|
||||
}
|
||||
)
|
||||
continue
|
||||
|
||||
node_area = areas.area_for(note, (area,), fallback=area)
|
||||
label = str(item.get("label") or node_id)
|
||||
content_hash = _hash(
|
||||
{"kind": "node", "id": node_id, "label": label, "note": note,
|
||||
"community": item.get("community"), "type": item.get("node_type")}
|
||||
)
|
||||
text = (
|
||||
f"{label}. Knowledge graph node"
|
||||
+ (f" in community {item.get('community')}" if item.get("community") is not None else "")
|
||||
+ f", from the note {note}."
|
||||
)
|
||||
result.nodes.append(
|
||||
{
|
||||
"id": node_id,
|
||||
"point_id": _point_id("node", node_id, content_hash),
|
||||
"graph_id": node_id,
|
||||
"label": label,
|
||||
"norm_label": str(item.get("norm_label") or _norm(label)),
|
||||
"node_type": str(item.get("node_type") or item.get("file_type") or "entity"),
|
||||
"community": item.get("community"),
|
||||
"source_note": note,
|
||||
"origin_url": file_url(base, repo, ref, note),
|
||||
"area": node_area,
|
||||
"tier": 2,
|
||||
"platform": platform,
|
||||
"last_modified": last_modified,
|
||||
"content_hash": content_hash,
|
||||
"text": text,
|
||||
}
|
||||
)
|
||||
|
||||
admitted_nodes = {row["graph_id"]: row for row in result.nodes}
|
||||
for item in raw_links:
|
||||
if len(result.edges) >= max_edges:
|
||||
result.warnings.append(f"graph edge limit {max_edges} reached; remaining edges skipped")
|
||||
break
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
source = str(item.get("source") or "").strip()
|
||||
target = str(item.get("target") or "").strip()
|
||||
if not source or not target:
|
||||
result.rejections.append(_rejection("graph.json", platform, "graph edge has no endpoints"))
|
||||
continue
|
||||
note = notes.get(source) or _note_for(item, vault_paths)
|
||||
if not note:
|
||||
result.rejections.append(
|
||||
_rejection(
|
||||
f"edge:{source}->{target}", platform,
|
||||
"no resolvable source note", "graph edge does not point at a file in the vault",
|
||||
)
|
||||
)
|
||||
continue
|
||||
source_label = labels.get(source, source)
|
||||
target_label = labels.get(target, target)
|
||||
relation = str(item.get("relation") or item.get("rel") or "related_to")
|
||||
area_name = areas.area_for(note, (area,), fallback=area)
|
||||
content_hash = _hash(
|
||||
{"kind": "edge", "source": source, "target": target,
|
||||
"relation": relation, "note": note}
|
||||
)
|
||||
text = f"{source_label} {relation} {target_label}, from the note {note}."
|
||||
result.edges.append(
|
||||
{
|
||||
"id": _edge_id(source, relation, target, content_hash),
|
||||
"point_id": _point_id("edge", f"{source}>{relation}>{target}", content_hash),
|
||||
"source_id": source,
|
||||
"target_id": target,
|
||||
"relation": relation,
|
||||
"confidence": str(item.get("confidence") or "INFERRED"),
|
||||
"confidence_score": item.get("confidence_score"),
|
||||
"source_note": note,
|
||||
"origin_url": file_url(base, repo, ref, note),
|
||||
"area": area_name,
|
||||
"tier": 2,
|
||||
"platform": platform,
|
||||
"last_modified": last_modified,
|
||||
"content_hash": content_hash,
|
||||
"text": text,
|
||||
}
|
||||
)
|
||||
|
||||
# Edges whose endpoints were not admitted are still traceable and are kept;
|
||||
# an answer can name the relation and its note even when the node itself had
|
||||
# no origin. `admitted_node_ids` is exposed for verify, which reports the gap.
|
||||
result.admitted_node_ids = set(admitted_nodes)
|
||||
result.orphan_edges = [
|
||||
edge for edge in result.edges
|
||||
if edge["source_id"] not in admitted_nodes or edge["target_id"] not in admitted_nodes
|
||||
]
|
||||
return result
|
||||
|
||||
|
||||
def point_payload(row: dict, kind: str) -> dict:
|
||||
"""One Qdrant point payload for a node or edge. Same provenance as the row."""
|
||||
payload = {
|
||||
"platform": row["platform"],
|
||||
"tier": row["tier"],
|
||||
"source_type": "graph_node" if kind == "node" else "graph_edge",
|
||||
"area": row["area"],
|
||||
"origin_url": row["origin_url"],
|
||||
"path": row["source_note"],
|
||||
"source_note": row["source_note"],
|
||||
"title": row.get("label") or row.get("relation") or row["source_note"],
|
||||
"confidence": row.get("confidence") or "INFERRED",
|
||||
"last_modified": row["last_modified"],
|
||||
"content_hash": row["content_hash"],
|
||||
"text": row["text"],
|
||||
"kind": kind,
|
||||
"graph_id": row.get("graph_id") or row.get("id"),
|
||||
}
|
||||
if kind == "node":
|
||||
payload["community"] = row.get("community")
|
||||
payload["node_type"] = row.get("node_type")
|
||||
else:
|
||||
payload["relation"] = row.get("relation")
|
||||
payload["source_id"] = row.get("source_id")
|
||||
payload["target_id"] = row.get("target_id")
|
||||
payload["confidence_score"] = row.get("confidence_score")
|
||||
return payload
|
||||
|
||||
|
||||
def query(cfg, store, text: str, area: str = None, limit: int = 10) -> dict:
|
||||
"""Graph query: keyword hits from SQLite, vector hits from Qdrant when up.
|
||||
|
||||
Returning both is deliberate: the keyword side always works, and the vector
|
||||
side proves the graph collection is live. Each row carries `source_note`.
|
||||
"""
|
||||
from ..qdrant import Qdrant, QdrantError
|
||||
|
||||
report = {"query": text, "area": area, "keyword": [], "vector": [], "ok": True}
|
||||
report["keyword"] = store.graph_search(text, area=area, limit=limit)
|
||||
if not report["keyword"]:
|
||||
report["ok"] = False
|
||||
report["reason"] = "no graph rows matched the query"
|
||||
qdrant = Qdrant(cfg.qdrant_url, timeout=cfg.timeout)
|
||||
if not qdrant.health() or not qdrant.collection_exists(cfg.graph_collection):
|
||||
report["vector"] = []
|
||||
report["vector_note"] = f"graph collection {cfg.graph_collection} is not available"
|
||||
return report
|
||||
from .. import embeddings
|
||||
|
||||
provider = embeddings.build_provider(cfg)
|
||||
try:
|
||||
vector = provider.embed([text])[0]
|
||||
hits = qdrant.search(cfg.graph_collection, vector, limit=limit, area=area)
|
||||
except (QdrantError, Exception) as exc: # noqa: BLE001 - a graph query must not crash a shell
|
||||
report["vector_note"] = f"graph vector search unavailable: {exc}"
|
||||
return report
|
||||
report["vector"] = [
|
||||
{
|
||||
"score": round(hit.get("score", 0.0), 4),
|
||||
"kind": (hit.get("payload") or {}).get("kind"),
|
||||
"label": (hit.get("payload") or {}).get("title"),
|
||||
"source_note": (hit.get("payload") or {}).get("source_note"),
|
||||
"origin_url": (hit.get("payload") or {}).get("origin_url"),
|
||||
}
|
||||
for hit in hits
|
||||
]
|
||||
return report
|
||||
|
||||
|
||||
# -- helpers -------------------------------------------------------------
|
||||
|
||||
|
||||
def _note_for(item: dict, vault_paths: set) -> str:
|
||||
source = globs.normalise(item.get("source_file") or "")
|
||||
if not source:
|
||||
return ""
|
||||
if source in vault_paths:
|
||||
return source
|
||||
# Graphify sometimes prefixes `./` or uses an absolute scan root. A suffix
|
||||
# match is only accepted when it is unique, so a note is never guessed.
|
||||
candidates = [path for path in vault_paths if path.endswith("/" + source)]
|
||||
if len(candidates) == 1:
|
||||
return candidates[0]
|
||||
return ""
|
||||
|
||||
|
||||
def _rejection(path: str, platform: str, reason: str, detail: str = ""):
|
||||
rejection = exclusions.Rejection(path, platform, reason, detail)
|
||||
return rejection.to_dict()
|
||||
|
||||
|
||||
def _hash(value) -> str:
|
||||
body = json.dumps(value, sort_keys=True, separators=(",", ":")).encode("utf-8")
|
||||
return hashlib.sha256(body).hexdigest()
|
||||
|
||||
|
||||
def _point_id(kind: str, key: str, content_hash: str) -> str:
|
||||
digest = hashlib.sha256(f"graph\x00{kind}\x00{key}\x00{content_hash}".encode("utf-8")).hexdigest()
|
||||
return str(uuid.UUID(hex=digest[:32]))
|
||||
|
||||
|
||||
def _edge_id(source: str, relation: str, target: str, content_hash: str) -> str:
|
||||
return hashlib.sha256(
|
||||
f"edge\x00{source}\x00{relation}\x00{target}\x00{content_hash}".encode("utf-8")
|
||||
).hexdigest()[:32]
|
||||
|
||||
|
||||
def _norm(label: str) -> str:
|
||||
return re.sub(r"[^0-9a-z]+", "_", (label or "").lower()).strip("_")
|
||||
196
corpus/store.py
196
corpus/store.py
@@ -27,7 +27,7 @@ import sqlite3
|
||||
import time
|
||||
import unicodedata
|
||||
|
||||
SCHEMA_VERSION = "1"
|
||||
SCHEMA_VERSION = "2"
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS chunks (
|
||||
@@ -49,6 +49,11 @@ CREATE TABLE IF NOT EXISTS chunks (
|
||||
embedding_dim INTEGER NOT NULL,
|
||||
chunker_version TEXT NOT NULL,
|
||||
chunk_tokens INTEGER NOT NULL,
|
||||
platform TEXT NOT NULL,
|
||||
tier INTEGER NOT NULL,
|
||||
origin_url TEXT NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
last_modified TEXT NOT NULL,
|
||||
public INTEGER NOT NULL,
|
||||
artefacts TEXT NOT NULL,
|
||||
tech TEXT NOT NULL,
|
||||
@@ -66,6 +71,8 @@ CREATE VIRTUAL TABLE IF NOT EXISTS chunk_fts USING fts5(
|
||||
text,
|
||||
area UNINDEXED,
|
||||
confidence UNINDEXED,
|
||||
platform UNINDEXED,
|
||||
tier UNINDEXED,
|
||||
tokenize = 'porter unicode61 remove_diacritics 2'
|
||||
);
|
||||
|
||||
@@ -84,6 +91,55 @@ CREATE TABLE IF NOT EXISTS meta (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
|
||||
-- The knowledge graph is not chunked (docs/phase-4-corpus.md §4). Nodes and
|
||||
-- edges live in their own tables and their own Qdrant collection, joined to note
|
||||
-- chunks by source_note. Every row carries the note it came from.
|
||||
CREATE TABLE IF NOT EXISTS graph_nodes (
|
||||
id TEXT PRIMARY KEY,
|
||||
label TEXT NOT NULL,
|
||||
norm_label TEXT NOT NULL,
|
||||
node_type TEXT NOT NULL,
|
||||
community INTEGER,
|
||||
source_note TEXT NOT NULL,
|
||||
origin_url TEXT NOT NULL,
|
||||
area TEXT NOT NULL,
|
||||
tier INTEGER NOT NULL,
|
||||
platform TEXT NOT NULL,
|
||||
last_modified TEXT NOT NULL,
|
||||
content_hash TEXT NOT NULL,
|
||||
text TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS graph_nodes_note ON graph_nodes(source_note);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS graph_edges (
|
||||
id TEXT PRIMARY KEY,
|
||||
source_id TEXT NOT NULL,
|
||||
target_id TEXT NOT NULL,
|
||||
relation TEXT NOT NULL,
|
||||
confidence TEXT NOT NULL,
|
||||
confidence_score REAL,
|
||||
source_note TEXT NOT NULL,
|
||||
origin_url TEXT NOT NULL,
|
||||
area TEXT NOT NULL,
|
||||
tier INTEGER NOT NULL,
|
||||
platform TEXT NOT NULL,
|
||||
last_modified TEXT NOT NULL,
|
||||
content_hash TEXT NOT NULL,
|
||||
text TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS graph_edges_source ON graph_edges(source_id);
|
||||
CREATE INDEX IF NOT EXISTS graph_edges_target ON graph_edges(target_id);
|
||||
CREATE INDEX IF NOT EXISTS graph_edges_note ON graph_edges(source_note);
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS graph_fts USING fts5(
|
||||
id UNINDEXED,
|
||||
kind UNINDEXED,
|
||||
label,
|
||||
text,
|
||||
area UNINDEXED,
|
||||
tokenize = 'porter unicode61 remove_diacritics 2'
|
||||
);
|
||||
"""
|
||||
|
||||
# Column order used by every INSERT in this module.
|
||||
@@ -91,6 +147,7 @@ _CHUNK_COLUMNS = (
|
||||
"id", "source_path", "source_id", "chunk_index", "area", "level", "source_type",
|
||||
"confidence", "period", "last_verified", "title", "claim", "url", "content_hash",
|
||||
"embedding_model", "embedding_dim", "chunker_version", "chunk_tokens",
|
||||
"platform", "tier", "origin_url", "path", "last_modified",
|
||||
"public", "artefacts", "tech", "tags", "text",
|
||||
)
|
||||
|
||||
@@ -110,7 +167,13 @@ class Store:
|
||||
self.db.execute("PRAGMA journal_mode = WAL")
|
||||
self.db.execute("PRAGMA synchronous = NORMAL")
|
||||
self.db.executescript(SCHEMA)
|
||||
self.set_meta("schema_version", SCHEMA_VERSION)
|
||||
stored = self.get_meta("schema_version")
|
||||
# A database from an older payload shape has no provenance columns. It is
|
||||
# derived data, so the fix is a rebuild, but it must never be silently
|
||||
# mixed with the new shape: `rebuild_schema` is called by a rebuild only.
|
||||
self.schema_stale = stored not in (None, SCHEMA_VERSION)
|
||||
if not self.schema_stale:
|
||||
self.set_meta("schema_version", SCHEMA_VERSION)
|
||||
|
||||
# -- lifecycle --------------------------------------------------------
|
||||
|
||||
@@ -125,10 +188,22 @@ class Store:
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Empty every table. Used by a from-scratch rebuild."""
|
||||
for table in ("chunk_fts", "chunks", "manifest", "meta"):
|
||||
for table in ("chunk_fts", "chunks", "manifest", "meta", "graph_fts", "graph_nodes", "graph_edges"):
|
||||
self.db.execute(f"DELETE FROM {table}")
|
||||
self.db.commit()
|
||||
|
||||
def rebuild_schema(self) -> None:
|
||||
"""Drop and recreate every table. Only a rebuild may call this."""
|
||||
for table in (
|
||||
"chunk_fts", "chunks", "manifest", "meta",
|
||||
"graph_fts", "graph_nodes", "graph_edges",
|
||||
):
|
||||
self.db.execute(f"DROP TABLE IF EXISTS {table}")
|
||||
self.db.commit()
|
||||
self.db.executescript(SCHEMA)
|
||||
self.schema_stale = False
|
||||
self.set_meta("schema_version", SCHEMA_VERSION)
|
||||
|
||||
# -- writing ----------------------------------------------------------
|
||||
|
||||
def write_document(self, doc, payloads: list) -> None:
|
||||
@@ -161,8 +236,9 @@ class Store:
|
||||
if value
|
||||
)
|
||||
self.db.execute(
|
||||
"INSERT INTO chunk_fts (rowid, id, source_path, title, terms, text, area, confidence) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
"INSERT INTO chunk_fts "
|
||||
"(rowid, id, source_path, title, terms, text, area, confidence, platform, tier) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
cursor.lastrowid,
|
||||
payload["id"],
|
||||
@@ -172,6 +248,8 @@ class Store:
|
||||
payload["text"],
|
||||
payload["area"],
|
||||
payload["confidence"],
|
||||
payload["platform"],
|
||||
payload["tier"],
|
||||
),
|
||||
)
|
||||
self.db.execute(
|
||||
@@ -237,7 +315,10 @@ class Store:
|
||||
out.append(payload)
|
||||
return out
|
||||
|
||||
def keyword_search(self, query: str, area: str = None, confidence: str = None, limit: int = 10) -> list:
|
||||
def keyword_search(
|
||||
self, query: str, area: str = None, confidence: str = None,
|
||||
platform: str = None, tier: int = None, limit: int = 10,
|
||||
) -> list:
|
||||
"""BM25 over chunk text, optionally filtered before ranking."""
|
||||
match = match_expression(query)
|
||||
if not match:
|
||||
@@ -256,11 +337,110 @@ class Store:
|
||||
if confidence:
|
||||
sql.append("AND c.confidence = ?")
|
||||
args.append(confidence)
|
||||
if platform:
|
||||
sql.append("AND c.platform = ?")
|
||||
args.append(platform)
|
||||
if tier is not None:
|
||||
sql.append("AND c.tier = ?")
|
||||
args.append(int(tier))
|
||||
sql.append("ORDER BY bm25(chunk_fts) LIMIT ?")
|
||||
args.append(int(limit))
|
||||
rows = self.db.execute(" ".join(sql), args)
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
def pool_texts(self) -> list:
|
||||
"""Every chunk text in the pool. The credential scan reads this."""
|
||||
return [
|
||||
(row["source_path"], row["chunk_index"], row["text"])
|
||||
for row in self.db.execute("SELECT source_path, chunk_index, text FROM chunks")
|
||||
]
|
||||
|
||||
# -- graph ------------------------------------------------------------
|
||||
|
||||
def write_graph(self, nodes: list, edges: list) -> None:
|
||||
"""Replace the whole graph. Nodes and edges are rebuilt together."""
|
||||
self.db.execute("DELETE FROM graph_fts")
|
||||
self.db.execute("DELETE FROM graph_nodes")
|
||||
self.db.execute("DELETE FROM graph_edges")
|
||||
for node in nodes:
|
||||
self.db.execute(
|
||||
"INSERT INTO graph_nodes (id, label, norm_label, node_type, community, "
|
||||
"source_note, origin_url, area, tier, platform, last_modified, content_hash, text) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
node["id"], node["label"], node.get("norm_label") or _norm(node["label"]),
|
||||
node.get("node_type") or "entity", node.get("community"),
|
||||
node["source_note"], node["origin_url"], node["area"], node["tier"],
|
||||
node["platform"], node["last_modified"], node["content_hash"], node["text"],
|
||||
),
|
||||
)
|
||||
self.db.execute(
|
||||
"INSERT INTO graph_fts (id, kind, label, text, area) VALUES (?, 'node', ?, ?, ?)",
|
||||
(node["id"], node["label"], node["text"], node["area"]),
|
||||
)
|
||||
for edge in edges:
|
||||
self.db.execute(
|
||||
"INSERT INTO graph_edges (id, source_id, target_id, relation, confidence, "
|
||||
"confidence_score, source_note, origin_url, area, tier, platform, "
|
||||
"last_modified, content_hash, text) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
edge["id"], edge["source_id"], edge["target_id"], edge["relation"],
|
||||
edge["confidence"], edge.get("confidence_score"), edge["source_note"],
|
||||
edge["origin_url"], edge["area"], edge["tier"], edge["platform"],
|
||||
edge["last_modified"], edge["content_hash"], edge["text"],
|
||||
),
|
||||
)
|
||||
self.db.execute(
|
||||
"INSERT INTO graph_fts (id, kind, label, text, area) VALUES (?, 'edge', ?, ?, ?)",
|
||||
(edge["id"], edge["relation"], edge["text"], edge["area"]),
|
||||
)
|
||||
self.db.commit()
|
||||
|
||||
def graph_counts(self) -> dict:
|
||||
return {
|
||||
"nodes": int(self.db.execute("SELECT count(*) FROM graph_nodes").fetchone()[0]),
|
||||
"edges": int(self.db.execute("SELECT count(*) FROM graph_edges").fetchone()[0]),
|
||||
}
|
||||
|
||||
def graph_node(self, node_id: str) -> dict:
|
||||
row = self.db.execute("SELECT * FROM graph_nodes WHERE id = ?", (node_id,)).fetchone()
|
||||
return dict(row) if row else {}
|
||||
|
||||
def graph_neighbours(self, node_id: str, limit: int = 50) -> list:
|
||||
rows = self.db.execute(
|
||||
"SELECT e.*, sn.label AS source_label, tn.label AS target_label "
|
||||
"FROM graph_edges e "
|
||||
"LEFT JOIN graph_nodes sn ON sn.id = e.source_id "
|
||||
"LEFT JOIN graph_nodes tn ON tn.id = e.target_id "
|
||||
"WHERE e.source_id = ? OR e.target_id = ? LIMIT ?",
|
||||
(node_id, node_id, int(limit)),
|
||||
)
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
def graph_search(self, query: str, area: str = None, limit: int = 10) -> list:
|
||||
"""Keyword search over node labels and edge relations, with provenance."""
|
||||
match = match_expression(query)
|
||||
if not match:
|
||||
return []
|
||||
sql = [
|
||||
"SELECT f.id, f.kind, f.label, f.text, f.area,",
|
||||
" COALESCE(n.source_note, e.source_note) AS source_note,",
|
||||
" COALESCE(n.origin_url, e.origin_url) AS origin_url,",
|
||||
" n.community, e.relation, e.source_id, e.target_id",
|
||||
"FROM graph_fts f",
|
||||
"LEFT JOIN graph_nodes n ON f.kind = 'node' AND n.id = f.id",
|
||||
"LEFT JOIN graph_edges e ON f.kind = 'edge' AND e.id = f.id",
|
||||
"WHERE graph_fts MATCH ?",
|
||||
]
|
||||
args = [match]
|
||||
if area:
|
||||
sql.append("AND f.area = ?")
|
||||
args.append(area)
|
||||
sql.append("ORDER BY bm25(graph_fts) LIMIT ?")
|
||||
args.append(int(limit))
|
||||
return [dict(row) for row in self.db.execute(" ".join(sql), args)]
|
||||
|
||||
# -- meta -------------------------------------------------------------
|
||||
|
||||
def set_meta(self, key: str, value) -> None:
|
||||
@@ -280,6 +460,10 @@ class Store:
|
||||
return {row["key"]: json.loads(row["value"]) for row in self.db.execute("SELECT key, value FROM meta")}
|
||||
|
||||
|
||||
def _norm(label: str) -> str:
|
||||
return re.sub(r"[^0-9a-z]+", "_", (label or "").lower()).strip("_")
|
||||
|
||||
|
||||
def match_expression(query: str) -> str:
|
||||
"""Free text -> a safe FTS5 expression: bare words, each a prefix, all required.
|
||||
|
||||
|
||||
@@ -119,6 +119,8 @@ def make_config(root: str, **overrides) -> Config:
|
||||
dim=32,
|
||||
qdrant_url="fake://qdrant",
|
||||
collection="test_corpus",
|
||||
graph_collection="test_graph",
|
||||
offline=True,
|
||||
)
|
||||
values.update(overrides)
|
||||
return Config.from_env(root=root, **values)
|
||||
|
||||
346
corpus/tests/test_ingestion.py
Normal file
346
corpus/tests/test_ingestion.py
Normal file
@@ -0,0 +1,346 @@
|
||||
"""Ingestion from the estate: exclusions, provenance, incremental work, graph.
|
||||
|
||||
These tests never touch the network. `FakeGitea` implements the four read calls
|
||||
the harvester uses; `FakeQdrant` implements the collection routes the pipeline
|
||||
uses. The real estate is exercised separately (docs/phase-4-notes.md), so a
|
||||
failure here is a logic failure, not a connectivity one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
from corpus import exclusions, guides, loader, pipeline, store as store_module
|
||||
from corpus.sources import estate, graph as graph_module
|
||||
from corpus.tests import support
|
||||
|
||||
|
||||
def gitea_guide(**overrides) -> guides.Guide:
|
||||
values = dict(
|
||||
name="gitea", enabled=True, what="Self-hosted Git.", fits=["ai", "platform", "edge", "products"],
|
||||
read=["readme", "tree", "selected_source", "commits", "languages"],
|
||||
base="https://gitea.example", auth="GITEA_TOKEN",
|
||||
include={"repos": "all", "paths": []},
|
||||
exclude={"repos": [], "paths": []},
|
||||
limits={"max_file_kb": 200, "max_files_per_repo": 10, "max_commits": 5, "max_tree_entries": 100},
|
||||
)
|
||||
values.update(overrides)
|
||||
return guides.Guide(**values)
|
||||
|
||||
|
||||
class FakeGitea:
|
||||
def __init__(self, repos=None, trees=None, files=None, commits=None, languages=None):
|
||||
self.repos_list = repos or []
|
||||
self.trees = trees or {}
|
||||
self.files = files or {}
|
||||
self.commit_list = commits or []
|
||||
self.language_map = languages or {}
|
||||
self.calls = []
|
||||
|
||||
def repos(self):
|
||||
return list(self.repos_list)
|
||||
|
||||
def tree(self, full_name, ref):
|
||||
self.calls.append(("tree", full_name, ref))
|
||||
return list(self.trees.get(full_name, []))
|
||||
|
||||
def raw(self, full_name, ref, path):
|
||||
self.calls.append(("raw", full_name, ref, path))
|
||||
try:
|
||||
return self.files[(full_name, path)]
|
||||
except KeyError:
|
||||
from corpus.sources.gitea import GiteaNotFound
|
||||
|
||||
raise GiteaNotFound(f"{full_name}/{path}") from None
|
||||
|
||||
def commits(self, full_name, ref, limit=20):
|
||||
return list(self.commit_list)
|
||||
|
||||
def languages(self, full_name):
|
||||
return dict(self.language_map)
|
||||
|
||||
|
||||
def repo(full_name="sam/photo-pipeline", **overrides):
|
||||
values = dict(
|
||||
full_name=full_name, name=full_name.split("/")[-1], description="Photo ingestion pipeline",
|
||||
topics=["photo", "python"], default_branch="main", archived=False,
|
||||
created_at="2024-01-01T00:00:00Z", updated_at="2026-10-14T05:00:00Z",
|
||||
html_url=f"https://gitea.example/{full_name}",
|
||||
)
|
||||
values.update(overrides)
|
||||
return values
|
||||
|
||||
|
||||
def blob(path, size=None):
|
||||
return {"type": "blob", "path": path, "size": len(path) if size is None else size}
|
||||
|
||||
|
||||
class ExclusionTest(unittest.TestCase):
|
||||
"""The DoD probe: a file in an excluded path never appears in the pool."""
|
||||
|
||||
def setUp(self):
|
||||
self.guide = gitea_guide()
|
||||
self.result = estate.Harvest()
|
||||
tree = [
|
||||
blob("README.md"),
|
||||
blob("node_modules/left-pad/probe.md"),
|
||||
blob(".venv/lib/probe.py"),
|
||||
blob(".git/config"),
|
||||
blob(".trash/old.md"),
|
||||
blob("docs/.silverbullet.auth.json"),
|
||||
blob("src/main.py"),
|
||||
blob("assets/photo.png"),
|
||||
]
|
||||
files = {
|
||||
("sam/photo-pipeline", "README.md"): b"# Photo pipeline\n\nDeduplicates by hash.\n",
|
||||
("sam/photo-pipeline", "node_modules/left-pad/probe.md"): b"# PROBE\n",
|
||||
("sam/photo-pipeline", ".venv/lib/probe.py"): b"PROBE = 1\n",
|
||||
("sam/photo-pipeline", ".git/config"): b"[core]\n",
|
||||
("sam/photo-pipeline", ".trash/old.md"): b"# old\n",
|
||||
("sam/photo-pipeline", "docs/.silverbullet.auth.json"): b'{"secret_key": "x"}\n',
|
||||
("sam/photo-pipeline", "src/main.py"): b"def dedup():\n return True\n",
|
||||
}
|
||||
self.client = FakeGitea(
|
||||
repos=[repo()],
|
||||
trees={"sam/photo-pipeline": tree},
|
||||
files=files,
|
||||
commits=[{"commit": {"message": "dedup by hash", "author": {"date": "2026-10-14T05:00:00Z"}}}],
|
||||
languages={"Python": 900, "Shell": 100},
|
||||
)
|
||||
self.documents = estate._harvest_one_repo(
|
||||
self.result, self.client, self.guide, repo(), "https://gitea.example"
|
||||
)
|
||||
|
||||
def test_probe_in_excluded_paths_never_appears(self):
|
||||
paths = {doc.repo_path for doc in self.documents}
|
||||
for probe in (
|
||||
"node_modules/left-pad/probe.md",
|
||||
".venv/lib/probe.py",
|
||||
".git/config",
|
||||
".trash/old.md",
|
||||
"docs/.silverbullet.auth.json",
|
||||
):
|
||||
self.assertNotIn(probe, paths)
|
||||
self.assertIn("README.md", paths)
|
||||
self.assertIn("src/main.py", paths)
|
||||
|
||||
def test_rejections_are_reported_with_a_reason(self):
|
||||
reasons = {(row["path"], row["reason"]) for row in self.result.rejections}
|
||||
self.assertIn(("node_modules/left-pad/probe.md", "excluded directory 'node_modules'"), reasons)
|
||||
self.assertIn(("docs/.silverbullet.auth.json", "credential-shaped file name"), reasons)
|
||||
self.assertEqual([image["path"] for image in self.result.images], ["assets/photo.png"])
|
||||
|
||||
def test_media_is_an_image_reference_not_text(self):
|
||||
self.assertEqual([image["path"] for image in self.result.images], ["assets/photo.png"])
|
||||
for doc in self.documents:
|
||||
self.assertNotIn("assets/photo.png", doc.body)
|
||||
|
||||
def test_every_document_carries_provenance(self):
|
||||
for doc in self.documents:
|
||||
self.assertEqual(doc.platform, "gitea")
|
||||
self.assertEqual(doc.tier, 2)
|
||||
self.assertTrue(doc.origin_url.startswith("https://gitea.example/sam/photo-pipeline"))
|
||||
self.assertTrue(doc.last_modified)
|
||||
self.assertIn(doc.area, self.guide.fits)
|
||||
|
||||
|
||||
class CredentialContentTest(unittest.TestCase):
|
||||
def test_a_valid_path_with_a_private_key_is_rejected(self):
|
||||
result = estate.Harvest()
|
||||
guide = gitea_guide()
|
||||
client = FakeGitea(
|
||||
repos=[repo()],
|
||||
trees={"sam/photo-pipeline": [blob("README.md"), blob("src/leak.py")]},
|
||||
files={
|
||||
("sam/photo-pipeline", "README.md"): b"# Photo pipeline\n",
|
||||
("sam/photo-pipeline", "src/leak.py"): (
|
||||
("KEY = '" + "-----BEGIN " + "PRIVATE KEY-----'\n").encode("utf-8")
|
||||
),
|
||||
},
|
||||
commits=[],
|
||||
languages={"Python": 10},
|
||||
)
|
||||
estate._harvest_one_repo(result, client, guide, repo(), "https://gitea.example")
|
||||
reasons = {row["reason"]: row for row in result.rejections}
|
||||
self.assertIn("credential-shaped content", reasons)
|
||||
self.assertEqual(reasons["credential-shaped content"]["path"], "src/leak.py")
|
||||
|
||||
|
||||
class PayloadProvenanceTest(unittest.TestCase):
|
||||
def test_missing_provenance_field_fails_the_payload(self):
|
||||
root = support.temp_repo()
|
||||
self.addCleanup(lambda: __import__("shutil").rmtree(root, ignore_errors=True))
|
||||
docs = loader.load_documents(support.make_config(root))
|
||||
chunk = __import__("corpus.chunker", fromlist=["chunk_document"]).chunk_document(docs[0].body)[0]
|
||||
payload = __import__("corpus.payloads", fromlist=["build_payload"]).build_payload(
|
||||
docs[0], chunk, support.make_config(root)
|
||||
)
|
||||
payload["id"] = pipeline.chunk_id(docs[0].path, 0, docs[0].digest)
|
||||
for field in ("platform", "tier", "origin_url", "path", "last_modified"):
|
||||
broken = dict(payload)
|
||||
broken.pop(field)
|
||||
with self.assertRaises(Exception):
|
||||
__import__("corpus.payloads", fromlist=["validate_payload"]).validate_payload(broken)
|
||||
|
||||
|
||||
GRAPH = {
|
||||
"built_at_commit": "abc123",
|
||||
"nodes": [
|
||||
{"id": "photo_ingest", "label": "Photo ingest", "source_file": "300 areas/Photo.md",
|
||||
"community": 7, "node_type": "entity"},
|
||||
{"id": "settings", "label": "settings.json", "source_file": "settings.json",
|
||||
"community": 118, "node_type": "code"},
|
||||
{"id": "image_node", "label": "diagram", "source_file": "300 areas/diagram.png",
|
||||
"community": 7, "node_type": "image"},
|
||||
],
|
||||
"links": [
|
||||
{"source": "photo_ingest", "target": "settings", "relation": "configured_by",
|
||||
"source_file": "300 areas/Photo.md", "confidence": "EXTRACTED", "confidence_score": 0.9},
|
||||
{"source": "settings", "target": "photo_ingest", "relation": "configures",
|
||||
"source_file": "settings.json", "confidence": "INFERRED"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class GraphTest(unittest.TestCase):
|
||||
def parse(self):
|
||||
return graph_module.parse(
|
||||
json.dumps(GRAPH).encode("utf-8"),
|
||||
platform="graphify", base="https://gitea.example", repo="sam/obsidian-vault",
|
||||
ref="main", last_modified="2026-10-14T05:00:00Z", area="ai",
|
||||
vault_paths={"300 areas/Photo.md", "300 areas/diagram.png"},
|
||||
)
|
||||
|
||||
def test_nodes_and_edges_carry_their_source_note(self):
|
||||
result = self.parse()
|
||||
self.assertEqual(len(result.nodes), 1)
|
||||
self.assertEqual(result.nodes[0]["id"], "photo_ingest")
|
||||
self.assertEqual(result.nodes[0]["source_note"], "300 areas/Photo.md")
|
||||
self.assertIn("/sam/obsidian-vault/src/branch/main/300%20areas/Photo.md", result.nodes[0]["origin_url"])
|
||||
self.assertTrue(result.edges)
|
||||
for edge in result.edges:
|
||||
self.assertTrue(edge["source_note"])
|
||||
|
||||
def test_a_node_without_a_resolvable_note_is_refused(self):
|
||||
result = self.parse()
|
||||
rejected = {row["path"] for row in result.rejections}
|
||||
self.assertIn("settings.json", rejected)
|
||||
self.assertNotIn("settings", {row["id"] for row in result.nodes})
|
||||
|
||||
def test_media_becomes_an_image_reference(self):
|
||||
result = self.parse()
|
||||
self.assertEqual([image["path"] for image in result.images], ["300 areas/diagram.png"])
|
||||
self.assertNotIn("image_node", {row["id"] for row in result.nodes})
|
||||
|
||||
def test_graph_query_returns_the_note(self):
|
||||
root = support.temp_repo()
|
||||
self.addCleanup(lambda: __import__("shutil").rmtree(root, ignore_errors=True))
|
||||
cfg = support.make_config(root)
|
||||
harvest = estate.Harvest()
|
||||
harvest.documents = loader.load_documents(cfg)
|
||||
harvest.graph = self.parse()
|
||||
harvest.sources["graphify"] = {"platform": "graphify", "enabled": True, "documents": 0}
|
||||
fake = support.FakeQdrant(cfg.dim)
|
||||
with mock.patch("corpus.sources.estate.harvest", return_value=harvest):
|
||||
with support.fake_qdrant(cfg, fake):
|
||||
pipeline.run(cfg, mode="rebuild")
|
||||
with store_module.Store(cfg.db_path) as store:
|
||||
rows = store.graph_search("photo")
|
||||
self.assertTrue(rows)
|
||||
self.assertTrue(all(row["source_note"] for row in rows))
|
||||
self.assertEqual(store.graph_node("photo_ingest")["source_note"], "300 areas/Photo.md")
|
||||
self.assertTrue(store.graph_neighbours("photo_ingest"))
|
||||
|
||||
|
||||
class IncrementalEstateTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = support.temp_repo()
|
||||
self.addCleanup(lambda: __import__("shutil").rmtree(self.root, ignore_errors=True))
|
||||
self.cfg = support.make_config(self.root)
|
||||
self.fake = support.FakeQdrant(self.cfg.dim)
|
||||
self.harvest = estate.Harvest()
|
||||
self.harvest.documents = loader.load_documents(self.cfg)
|
||||
self.harvest.documents.append(self.harvested("a" * 64, "First version of the repository."))
|
||||
self.harvest.sources["gitea"] = {"platform": "gitea", "enabled": True, "documents": 1}
|
||||
|
||||
def harvested(self, digest, body):
|
||||
return loader.Document(
|
||||
path="gitea:sam/photo-pipeline/README.md", source_id="sam/photo-pipeline/README.md",
|
||||
area="platform", level=3, title="sam/photo-pipeline — README.md",
|
||||
claim=body, confidence="historic", period="2024–2026", tech=[], metrics=[], artefacts=[],
|
||||
public=True, last_verified="2026-10-14", tags=["gitea"], body=body, digest=digest,
|
||||
url="https://gitea.example/sam/photo-pipeline/src/branch/main/README.md",
|
||||
platform="gitea", tier=2, origin_url="https://gitea.example/sam/photo-pipeline/src/branch/main/README.md",
|
||||
last_modified="2026-10-14T05:00:00Z", source_type="doc", kind="markdown",
|
||||
repo_path="README.md",
|
||||
)
|
||||
|
||||
def run_pipeline(self, mode="sync"):
|
||||
with mock.patch("corpus.sources.estate.harvest", return_value=self.harvest):
|
||||
with support.fake_qdrant(self.cfg, self.fake):
|
||||
return pipeline.run(self.cfg, mode=mode)
|
||||
|
||||
def test_rebuild_and_second_run_do_not_duplicate(self):
|
||||
first = self.run_pipeline("rebuild")
|
||||
points = len(self.fake.collections[self.cfg.collection])
|
||||
self.assertEqual(first.platforms["gitea"]["documents"], 1)
|
||||
second = self.run_pipeline("sync")
|
||||
self.assertEqual(len(self.fake.collections[self.cfg.collection]), points)
|
||||
self.assertEqual(second.added, 0)
|
||||
self.assertEqual(second.updated, 0)
|
||||
self.assertEqual(second.texts_embedded, 0)
|
||||
|
||||
def test_only_the_changed_harvested_document_is_reembedded(self):
|
||||
self.run_pipeline("rebuild")
|
||||
self.harvest.documents[-1] = self.harvested("b" * 64, "Second version, one line changed.")
|
||||
stats = self.run_pipeline("sync")
|
||||
self.assertEqual(stats.updated, 1)
|
||||
expected = len(__import__("corpus.chunker", fromlist=["chunk_document"]).chunk_document(
|
||||
"Second version, one line changed."
|
||||
))
|
||||
self.assertEqual(stats.texts_embedded, expected)
|
||||
self.assertEqual(stats.embed_calls, 1)
|
||||
self.assertEqual(stats.unchanged, 3)
|
||||
|
||||
def test_pool_scan_finds_a_planted_secret(self):
|
||||
self.harvest.documents[-1] = self.harvested(
|
||||
"c" * 64,
|
||||
"AWS_ACCESS_KEY_ID=" + "AKIA" + "ABCDEFGHIJKLMNOP" + "\nThis line is a planted credential.",
|
||||
)
|
||||
with self.assertRaises(pipeline.PipelineError) as caught:
|
||||
self.run_pipeline("rebuild")
|
||||
self.assertIn("credential-shaped content", str(caught.exception))
|
||||
|
||||
|
||||
class CodeChunkerTest(unittest.TestCase):
|
||||
def test_a_function_is_never_cut(self):
|
||||
from corpus import chunker
|
||||
|
||||
body = "\n\n".join(
|
||||
f"def function_{index}():\n value = {{'index': {index}}}\n return value"
|
||||
for index in range(120)
|
||||
)
|
||||
chunks = chunker.chunk_code(body)
|
||||
self.assertGreater(len(chunks), 1)
|
||||
for chunk in chunks:
|
||||
depth = 0
|
||||
for line in chunk.text.splitlines():
|
||||
depth = chunker._code_depth(line, depth)
|
||||
self.assertEqual(depth, 0, f"chunk {chunk.index} ends inside a block")
|
||||
self.assertIn("def function_", chunk.text)
|
||||
|
||||
def test_oversize_single_function_is_flagged_not_cut(self):
|
||||
from corpus import chunker
|
||||
|
||||
body = "def huge():\n" + "\n".join(f" x{index} = {index}" for index in range(2000))
|
||||
chunks = chunker.chunk_code(body)
|
||||
self.assertEqual(len(chunks), 1)
|
||||
self.assertTrue(chunks[0].oversize)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -44,15 +44,6 @@ PROJECT=ai-resume-site # compose project; distinct from the older
|
||||
PORT=8096
|
||||
HEALTH_PATH=/healthz
|
||||
|
||||
# The host address the port is published on. Caddy runs on .35 and proxies to
|
||||
# this address, so a loopback publish is unreachable from the edge (Sam's
|
||||
# decision, 2026-10-26, interaction 54aeb1a2). Never 0.0.0.0: the port must not
|
||||
# be reachable off the LAN. The health gate below follows this address — once the
|
||||
# port is published on the LAN address the host no longer listens on
|
||||
# 127.0.0.1:8096, so a loopback probe would refuse and every deploy would roll
|
||||
# back.
|
||||
PUBLISH_IP=${RESUME_PUBLISH_IP:-192.168.20.13}
|
||||
|
||||
ROOT=${RESUME_DEPLOY_ROOT:-/home/sam/deployment/resume}
|
||||
RELEASES=$ROOT/releases
|
||||
CURRENT=$ROOT/current
|
||||
@@ -240,7 +231,7 @@ start() {
|
||||
wait_healthy() {
|
||||
local i body
|
||||
for i in $(seq 1 "$HEALTH_TRIES"); do
|
||||
if body=$(curl -fsS --max-time 3 "http://$PUBLISH_IP:$PORT$HEALTH_PATH" 2>/dev/null); then
|
||||
if body=$(curl -fsS --max-time 3 "http://127.0.0.1:$PORT$HEALTH_PATH" 2>/dev/null); then
|
||||
if printf '%s' "$body" | grep -Eq '"status"[[:space:]]*:[[:space:]]*"ok"'; then
|
||||
log "health gate passed after ${i}s"
|
||||
return 0
|
||||
@@ -249,15 +240,14 @@ wait_healthy() {
|
||||
sleep 1
|
||||
done
|
||||
warn "health gate failed after ${HEALTH_TRIES}s; last response:"
|
||||
curl -sS --max-time 3 "http://$PUBLISH_IP:$PORT$HEALTH_PATH" >&2 || true
|
||||
curl -sS --max-time 3 "http://127.0.0.1:$PORT$HEALTH_PATH" >&2 || true
|
||||
compose logs --tail 40 resume >&2 || true
|
||||
return 1
|
||||
}
|
||||
|
||||
# Definition-of-Done evidence, checked on every deploy instead of by hand:
|
||||
# mem_limit == memswap_limit == the value declared in compose.yml, and the port
|
||||
# published on $PUBLISH_IP only (the LAN address Caddy proxies to — never
|
||||
# 0.0.0.0 and no longer loopback).
|
||||
# published on 127.0.0.1 only.
|
||||
assert_shape() {
|
||||
local cid mem swap ip declared
|
||||
declared=$(awk '/^[[:space:]]*mem_limit:/ {gsub(/[^0-9]/, "", $2); print $2}' "$COMPOSE")
|
||||
@@ -268,8 +258,8 @@ assert_shape() {
|
||||
[ "$mem" = "$swap" ] || die "mem_limit ($mem) != memswap_limit ($swap)"
|
||||
[ "$mem" = "$((declared * 1024 * 1024))" ] || die "mem_limit is $mem, expected ${declared} MiB"
|
||||
ip=$(docker inspect -f '{{range $p, $b := .HostConfig.PortBindings}}{{range $b}}{{.HostIp}} {{end}}{{end}}' "$cid" | xargs)
|
||||
[ "$ip" = "$PUBLISH_IP" ] || die "published on '${ip}', expected $PUBLISH_IP only"
|
||||
log "shape ok: mem_limit == memswap_limit == ${declared} MiB, published on $PUBLISH_IP:$PORT"
|
||||
[ "$ip" = "127.0.0.1" ] || die "published on '${ip}', expected 127.0.0.1 only"
|
||||
log "shape ok: mem_limit == memswap_limit == ${declared} MiB, published on 127.0.0.1:$PORT"
|
||||
}
|
||||
|
||||
deploy_release() {
|
||||
@@ -358,7 +348,7 @@ prune() {
|
||||
do_status() {
|
||||
local cur prev body
|
||||
cur=$(current_target); prev=$(previous_target)
|
||||
body=$(curl -fsS --max-time 3 "http://$PUBLISH_IP:$PORT$HEALTH_PATH" 2>/dev/null || true)
|
||||
body=$(curl -fsS --max-time 3 "http://127.0.0.1:$PORT$HEALTH_PATH" 2>/dev/null || true)
|
||||
printf 'root %s
|
||||
' "$ROOT"
|
||||
printf 'current %s
|
||||
|
||||
@@ -51,5 +51,16 @@ services:
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
# The host's docker daemon has every default address pool allocated by other
|
||||
# projects, so an implicit compose network cannot be created here. Pin a /28 of
|
||||
# RFC1918 space that no other bridge uses; it is separate from the site's
|
||||
# 10.89.0.0/28 so stopping one project cannot disturb the other.
|
||||
networks:
|
||||
default:
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 10.89.0.16/28
|
||||
gateway: 10.89.0.17
|
||||
|
||||
volumes:
|
||||
qdrant_storage:
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
# Image brief
|
||||
|
||||
Every image on this site gets the same treatment. Consistency is what makes a set look
|
||||
designed; six images treated six ways look worse than six treated identically.
|
||||
|
||||
The palette is fixed by `internal/web/static/site.css`:
|
||||
|
||||
| Token | Value | Use |
|
||||
|---|---|---|
|
||||
| `--bg` | `#0b0d10` | canvas and fade colour. Never pure black |
|
||||
| `--surface` | `#12161b` | screenshot backgrounds |
|
||||
| `--line` | `#242c35` | screenshot borders |
|
||||
| `--line-strong` | `#33404d` | diagram strokes |
|
||||
| `--text-dim` | `#b3bcc8` | diagram labels |
|
||||
| `--link` | `#8ecbff` | rim glow |
|
||||
| `--accent` | `#6ee7c6` | sparing emphasis only |
|
||||
|
||||
## Why white-background product shots fail here
|
||||
|
||||
`#FFFFFF` on a `#0b0d10` page is roughly twenty times the brightness of everything around
|
||||
it. The brightest thing in the frame becomes the backdrop rather than the object, and the
|
||||
eye lands on the backdrop first. One reads as an accident; six read as a scrapbook.
|
||||
|
||||
## Photographs
|
||||
|
||||
- **Canvas** 1600×1000 for hero images, 1200×1200 for detail. Landscape unless the object demands otherwise.
|
||||
- **Background** `#0b0d10`, the page colour. Not black — pure black always looks like a void.
|
||||
- **Subject** cut out, placed right of centre, occupying 55–70% of the canvas height. Off-centre because the fade comes from the left.
|
||||
- **Fade** a `#0b0d10` gradient, 100% opacity at the left edge to 0% at about 55% across. The object emerges from the page and dissolves to the right. This is the move that stops an image looking pasted on.
|
||||
- **Rim glow** `#8ecbff` at 25–40% opacity, 8–14px blur, derived from the cut-out's own alpha channel so it follows real edges. Enough to lift the object off the background. Not enough to look like a gaming mouse.
|
||||
- **Shadow lift** 5–8% on the subject itself. Cut-outs on dark backgrounds lose their dark detail and read as flat stickers otherwise.
|
||||
- **Format** WebP, quality 82, plus a JPEG fallback. Target under 120 KB for a hero.
|
||||
|
||||
## Screenshots
|
||||
|
||||
Never cut out. Frame them like the site's own panels:
|
||||
|
||||
- corner radius `10px` (`--radius`)
|
||||
- 1px border in `--line`
|
||||
- edges darkened 8%
|
||||
- soft drop shadow
|
||||
- crop out browser chrome where possible
|
||||
|
||||
## Diagrams
|
||||
|
||||
Stay SVG and stay in the palette: `--line-strong` strokes, `--text-dim` labels, `--accent`
|
||||
and `--link` for emphasis. No white fills, no default Mermaid blues.
|
||||
|
||||
## Logos
|
||||
|
||||
One colour, inheriting `currentColor` from the surrounding text. Forty-six client logos in
|
||||
full colour would shriek. If one deserves emphasis, give it `--accent`; the rest stay muted.
|
||||
|
||||
## Prohibited
|
||||
|
||||
White boxes · stock photography · lens flares · 3D renders of objects not owned ·
|
||||
glow strong enough to be the point · any image containing a family member, an address,
|
||||
a credential, or a screen showing personal data.
|
||||
|
||||
## Capture guidance
|
||||
|
||||
Treatment rescues a product shot; it cannot rescue a photograph of a white wall in daylight.
|
||||
|
||||
- dark surface, one light source from one side
|
||||
- shoot slightly above the object
|
||||
- keep the object off-centre in frame
|
||||
- no overhead lighting, no visible background clutter
|
||||
- a phone is fine
|
||||
|
||||
Rack and cabinet shots are the exception: keep them whole and wide, framed like
|
||||
screenshots rather than cut out.
|
||||
|
||||
## Output contract
|
||||
|
||||
Every image lands in `content/assets/images/`, named lowercase and hyphenated with no
|
||||
spaces or dates. Every image used on a page carries a caption naming the object; a product
|
||||
photograph additionally states that it is a product image and where it came from. The
|
||||
original, if it is Sam's own photograph or a sourced product image, is also filed in the
|
||||
repository of the project it belongs to, under `docs/images/`, with provenance recorded in
|
||||
that folder's `README.md`.
|
||||
@@ -30,7 +30,7 @@ deploy.sh
|
||||
├─ copy to releases/<timestamp>/
|
||||
├─ flip `current` symlink (atomic)
|
||||
├─ docker compose up -d
|
||||
├─ curl http://192.168.20.13:8096/healthz → must be ready
|
||||
├─ curl http://127.0.0.1:8096/healthz → must be ready
|
||||
└─ on failure: flip back to `previous`, exit non-zero
|
||||
```
|
||||
|
||||
@@ -40,12 +40,7 @@ deploy.sh
|
||||
|
||||
### 3.2 Container
|
||||
- `mem_limit` = `memswap_limit` = `120m`.
|
||||
- `ports: ["192.168.20.13:8096:8096"]` — **never** `0.0.0.0`.
|
||||
Caddy runs on `.35`, so the port has to be reachable from the LAN; a
|
||||
`127.0.0.1` publish answers only on `.13` and the public URL returns 502.
|
||||
LAN-only publish, Caddy on `.35` still the sole public listener. Decision:
|
||||
Sam, 2026-10-26, interaction `54aeb1a2`. The health gate and `assert_shape()`
|
||||
in `deploy.sh` follow the same address (`RESUME_PUBLISH_IP`).
|
||||
- `ports: ["127.0.0.1:8096:8096"]` — never `0.0.0.0`.
|
||||
- `logging` capped (`max-size: 10m`, `max-file: 3`).
|
||||
- `restart: unless-stopped`.
|
||||
|
||||
@@ -80,7 +75,7 @@ Repo stays private through this phase. Sam decides on public visibility after th
|
||||
|
||||
- [ ] `deploy.sh` and `deploy.sh --rollback` both work; a deliberate bad build proves rollback.
|
||||
- [ ] `https://resume.lab.audasmedia.com.au` serves the site.
|
||||
- [ ] Container published on `192.168.20.13:8096` only, reachable from `.35`; confirmed with `ss -ltn`.
|
||||
- [ ] Container bound to `127.0.0.1` only; confirmed with `ss -ltn`.
|
||||
- [ ] `memswap_limit == mem_limit`; confirmed with `docker inspect`.
|
||||
- [ ] Security headers present; CSP does not break the page.
|
||||
- [ ] A1 still passes over the public URL with the AI layer stopped.
|
||||
|
||||
@@ -1,71 +0,0 @@
|
||||
# Phase 7 — notes
|
||||
|
||||
Running record for the visual layer (AUD-149). Each batch states what changed,
|
||||
what it measures, and what it still needs.
|
||||
|
||||
---
|
||||
|
||||
## Batch 1 — marks, area icons, client wall, two diagrams
|
||||
|
||||
### What is in the tree
|
||||
|
||||
| Piece | Where | Note |
|
||||
|---|---|---|
|
||||
| Technology marks | `internal/brand/` | 50 Simple Icons paths (CC0), inlined as 16 px `currentColor` SVG. No request, no font, no bundler. |
|
||||
| Regenerator | `tools/gen_brand_icons.sh` | Development-time only. The generated Go file is committed, so `make build` never needs the network. |
|
||||
| Area icons | `internal/web/templates/icons.templ` | Six marks drawn for this site, plus one for the evidence index. 24×24, 1.6 px stroke, `aria-hidden`. |
|
||||
| Client wall | `content/history/l3/history-client-wall.md` + `.client-wall` in `site.css` | 45 names, one typeface, one size, one grid of equal tiles. No client mark is used, because none has been approved. |
|
||||
| Diagrams | `content/assets/diagrams/` | Homelab topology and RAG pipeline. Both referenced from the page they explain. |
|
||||
| Image sizing | `internal/content/markdown.go` | Every content image gets intrinsic `width`/`height`, `loading="lazy"`, `decoding="async"`. |
|
||||
| Guard | `internal/site/site_test.go` | `TestImagesReserveTheirSpace` fails if an image ships without size or alt text. |
|
||||
| Topbar | `site.css` | Two rows at desktop. One row overlapped the search form once the marks were added; Lighthouse `target-size` caught it. |
|
||||
| Favicon | `internal/web/static/favicon.svg` | Stops the browser asking for a missing `/favicon.ico` (it was the only console error). |
|
||||
|
||||
### Adding a technology mark
|
||||
|
||||
1. Add the label → slug pair to `slugs` in `internal/brand/brand.go`.
|
||||
2. Add the slug to `slugs()` in `tools/gen_brand_icons.sh` and run it.
|
||||
3. `make dist`, then check the chip on the page.
|
||||
|
||||
A technology with no published mark renders as text alone, on purpose: a chip
|
||||
with a missing image would read as a broken page.
|
||||
|
||||
### Adding an image
|
||||
|
||||
- File goes under `content/assets/` (published at `/assets/...`).
|
||||
- SVG needs explicit `width` and `height` on the root element; PNG and JPEG are
|
||||
measured with `image.DecodeConfig`. Both give the page its reserved space.
|
||||
- Alt text describes the thing, not the file name.
|
||||
- The diagrams carry their own print style, so they print as ink on white.
|
||||
|
||||
### Measurements (2026-10-28)
|
||||
|
||||
Lighthouse 12.8.2, headless Chrome, against the local server on `127.0.0.1`.
|
||||
|
||||
| Page | Mode | Performance | Accessibility | Best practices | SEO | CLS |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Landing | mobile | 100 | 100 | 100 | 100 | 0 |
|
||||
| Landing | desktop | 100 | 100 | 100 | 100 | 0 |
|
||||
| Platform network (diagram) | mobile | 100 | 100 | 100 | 100 | 0 |
|
||||
| AI retrieval (diagram) | desktop | 100 | 100 | 100 | 100 | 0 |
|
||||
| Client wall | mobile | 100 | 100 | 100 | 100 | — |
|
||||
| Edge audio artefact | mobile | 100 | 100 | 100 | 100 | — |
|
||||
|
||||
Total page weight: landing 153 KiB, diagram page 105 KiB. Diagram files are 5.7
|
||||
KiB and 6.6 KiB; the stylesheet is 24 KiB; htmx is 51 KiB and is the largest
|
||||
single asset.
|
||||
|
||||
Two defects were found by measurement rather than by eye and fixed in this
|
||||
batch: the desktop topbar overlap, and the missing favicon.
|
||||
|
||||
### Still open — needs Sam
|
||||
|
||||
| DoD line | State | What is needed |
|
||||
|---|---|---|
|
||||
| At least one real photograph in use | Not started | Which photographs may be used (Immich on `.35` is the likely source). Anything used is checked for people, address and screens in shot. |
|
||||
| Two screenshots in use | Not started | Approval to capture Home Assistant, Gitea, TUIOS with agent panes, or Graphify's `graph.html`. Screens must contain nothing personal. |
|
||||
| One recording, 30–60 s | Not started | Either Sam records it, or Sam approves a recording of the agent console. |
|
||||
| Client wall marks | Deliberately absent | Official marks only, with Sam's approval. Nothing invented. |
|
||||
|
||||
Nothing above is blocking the site: it ships, measures 100 across the board, and
|
||||
every page works with the AI services stopped.
|
||||
@@ -48,13 +48,49 @@ from corpus.config import Config # noqa: E402
|
||||
|
||||
@_flow(name="corpus-sync", log_prints=True)
|
||||
def sync_corpus() -> dict:
|
||||
"""Incremental refresh. Runs on every push that touches content/."""
|
||||
"""Incremental refresh of the whole estate. Runs on a push or on demand."""
|
||||
logger = _get_run_logger()
|
||||
cfg = Config.from_env()
|
||||
stats = pipeline.run(cfg, mode="sync", log=logger.info)
|
||||
return stats.to_dict()
|
||||
|
||||
|
||||
@_flow(name="corpus-scheduled-ingest", log_prints=True)
|
||||
def scheduled_ingest(alert: bool = True) -> dict:
|
||||
"""Nightly: harvest, report what changed, then audit and alert on staleness.
|
||||
|
||||
This is the scheduled run the DoD asks for: it reports documents added,
|
||||
updated and removed per platform, and it raises an Apprise alert when a
|
||||
source is missing, unavailable or has stopped changing.
|
||||
"""
|
||||
logger = _get_run_logger()
|
||||
cfg = Config.from_env()
|
||||
stats = pipeline.run(cfg, mode="sync", log=logger.info)
|
||||
logger.info(stats.summary())
|
||||
report = freshness.audit(cfg, log=logger.info)
|
||||
logger.info(freshness.render_text(report))
|
||||
if alert and not report["ok"]:
|
||||
try:
|
||||
freshness.alert(report, cfg.apprise_url)
|
||||
except Exception as exc: # noqa: BLE001 - the alert must not fail the run
|
||||
logger.warning(f"Apprise alert failed: {exc}")
|
||||
return {
|
||||
"changed": {
|
||||
"documents": stats.documents,
|
||||
"added": stats.added,
|
||||
"updated": stats.updated,
|
||||
"removed": stats.removed,
|
||||
"unchanged": stats.unchanged,
|
||||
"embedding_calls": stats.embed_calls,
|
||||
"texts_embedded": stats.texts_embedded,
|
||||
"platforms": stats.platforms,
|
||||
"rejections": stats.rejection_summary,
|
||||
"graph_rebuilt": stats.graph_rebuilt,
|
||||
},
|
||||
"audit": report,
|
||||
}
|
||||
|
||||
|
||||
@_flow(name="corpus-rebuild", log_prints=True)
|
||||
def rebuild_corpus() -> dict:
|
||||
"""From-scratch rebuild. The recovery path: deleting the volume is always safe."""
|
||||
@@ -85,5 +121,7 @@ if __name__ == "__main__": # pragma: no cover - convenience for a manual run
|
||||
print(freshness.render_text(freshness_audit(alert=False)))
|
||||
elif command == "rebuild":
|
||||
print(rebuild_corpus())
|
||||
elif command == "scheduled":
|
||||
print(scheduled_ingest(alert=False))
|
||||
else:
|
||||
print(sync_corpus())
|
||||
|
||||
@@ -1,90 +0,0 @@
|
||||
// Package brand holds the small monochrome technology marks the site inlines
|
||||
// beside a technology name.
|
||||
//
|
||||
// A mark is path data for a 24x24 viewBox, drawn with fill="currentColor" so it
|
||||
// inherits the chip's colour and adds no request, no font and no layout shift.
|
||||
// Marks are opt-in: a technology with no published mark renders as text alone,
|
||||
// which is the honest default and keeps the chips from looking broken.
|
||||
//
|
||||
// The path data is vendored (see icons_gen.go and tools/gen_brand_icons.sh),
|
||||
// not fetched at build or request time. The marks are trademarks of their
|
||||
// owners and are used only to identify the technology named beside them.
|
||||
package brand
|
||||
|
||||
import "strings"
|
||||
|
||||
// slugs maps a technology label, lowercased, to its Simple Icons slug. Labels
|
||||
// are the ones the content uses; aliases point at the same mark.
|
||||
var slugs = map[string]string{
|
||||
"javascript": "javascript",
|
||||
"php": "php",
|
||||
"python": "python",
|
||||
"home assistant": "homeassistant",
|
||||
"raspberry pi": "raspberrypi",
|
||||
"pi": "raspberrypi",
|
||||
"nixos": "nixos",
|
||||
"nix": "nixos",
|
||||
"qdrant": "qdrant",
|
||||
"prefect": "prefect",
|
||||
"n8n": "n8n",
|
||||
"mqtt": "mqtt",
|
||||
"langgraph": "langgraph",
|
||||
"langchain": "langchain",
|
||||
"ffmpeg": "ffmpeg",
|
||||
"zigbee": "zigbee",
|
||||
"three.js": "threedotjs",
|
||||
"react": "react",
|
||||
"proxmox": "proxmox",
|
||||
"mysql": "mysql",
|
||||
"postgres": "postgresql",
|
||||
"postgresql": "postgresql",
|
||||
"caddy": "caddy",
|
||||
"tailscale": "tailscale",
|
||||
"neovim": "neovim",
|
||||
"htmx": "htmx",
|
||||
"go": "go",
|
||||
"docker": "docker",
|
||||
"compose": "docker",
|
||||
"borg": "borgbackup",
|
||||
"sqlite": "sqlite",
|
||||
"spotify": "spotify",
|
||||
"rabbitmq": "rabbitmq",
|
||||
"openrouter": "openrouter",
|
||||
"obsidian": "obsidian",
|
||||
"niri": "niri",
|
||||
"gitea": "gitea",
|
||||
"ghostty": "ghostty",
|
||||
"gemini": "googlegemini",
|
||||
"deepseek": "deepseek",
|
||||
"c++": "cplusplus",
|
||||
"typescript": "typescript",
|
||||
"zsh": "zsh",
|
||||
"minimax": "minimax",
|
||||
"esp32": "espressif",
|
||||
"esp32-cam": "espressif",
|
||||
"grafana": "grafana",
|
||||
"wireguard": "wireguard",
|
||||
"nginx": "nginx",
|
||||
"linux": "linux",
|
||||
"zig": "zig",
|
||||
"huggingface": "huggingface",
|
||||
"ollama": "ollama",
|
||||
}
|
||||
|
||||
// Mark returns the path data for a technology label and whether one exists.
|
||||
// The lookup is case-insensitive and tolerates the surrounding whitespace that
|
||||
// hand-written front-matter tends to carry.
|
||||
func Mark(label string) ([]string, bool) {
|
||||
slug, ok := slugs[strings.ToLower(strings.TrimSpace(label))]
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
paths, ok := marks[slug]
|
||||
return paths, ok
|
||||
}
|
||||
|
||||
// Has reports whether a technology label has a published mark.
|
||||
func Has(label string) bool {
|
||||
_, ok := Mark(label)
|
||||
return ok
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -37,21 +37,22 @@ type Area struct {
|
||||
Title string
|
||||
Role string
|
||||
Blurb string
|
||||
Gutter string // two-letter mark used by the nav and the stage panel
|
||||
}
|
||||
|
||||
// Areas is the fixed navigation order from reference/content-map.md.
|
||||
var Areas = []Area{
|
||||
{Key: "ai", Title: "AI & Agentic Systems", Role: "AI / LLM Engineer",
|
||||
{Key: "ai", Title: "AI & Agentic Systems", Role: "AI / LLM Engineer", Gutter: "AI",
|
||||
Blurb: "Harnesses, subagents and skills, memory, model routing and cost, retrieval, voice, guardrails, agent observability."},
|
||||
{Key: "platform", Title: "Platform & Infrastructure", Role: "Platform / DevOps Engineer",
|
||||
{Key: "platform", Title: "Platform & Infrastructure", Role: "Platform / DevOps Engineer", Gutter: "PL",
|
||||
Blurb: "Declarative hosts, packaging, a fifty-container Docker estate, networking, storage and restore-tested backups, observability."},
|
||||
{Key: "edge", Title: "Edge & Devices", Role: "Edge / IoT Engineer",
|
||||
{Key: "edge", Title: "Edge & Devices", Role: "Edge / IoT Engineer", Gutter: "ED",
|
||||
Blurb: "Twenty-node fleet on constrained devices, custom radio, solar power, deep sleep, sensors and actuation — and the failures."},
|
||||
{Key: "products", Title: "Products", Role: "",
|
||||
{Key: "products", Title: "Products", Role: "", Gutter: "PR",
|
||||
Blurb: "Three active businesses, built and run rather than advised on."},
|
||||
{Key: "history", Title: "History", Role: "",
|
||||
{Key: "history", Title: "History", Role: "", Gutter: "HI",
|
||||
Blurb: "2000–2014: agencies, clients, engagements, and the code that survives from them."},
|
||||
{Key: "about", Title: "About", Role: "",
|
||||
{Key: "about", Title: "About", Role: "", Gutter: "AB",
|
||||
Blurb: "The narrative, the non-IT work that came first, education, and how to get in touch."},
|
||||
}
|
||||
|
||||
|
||||
@@ -4,15 +4,9 @@ import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
stdhtml "html"
|
||||
"image"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
"io"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/alecthomas/chroma/v2"
|
||||
@@ -82,7 +76,6 @@ func (r *Renderer) Transform(doc *ast.Document, reader text.Reader, pc parser.Co
|
||||
node.Destination = r.rewrite(current, string(node.Destination), false)
|
||||
case *ast.Image:
|
||||
node.Destination = r.rewrite(current, string(node.Destination), true)
|
||||
r.sizeImage(current, node)
|
||||
}
|
||||
return ast.WalkContinue, nil
|
||||
})
|
||||
@@ -115,70 +108,6 @@ func (r *Renderer) rewrite(from *Document, dest string, image bool) []byte {
|
||||
return []byte(dest)
|
||||
}
|
||||
|
||||
// sizeImage records an image's intrinsic size and loading hints on the node, so
|
||||
// the rendered page reserves the space before the file arrives. Without this the
|
||||
// text jumps as each image lands, which is the most visible performance defect a
|
||||
// page of diagrams can have.
|
||||
//
|
||||
// Only the four attributes in goldmark's image filter are used, and only when a
|
||||
// real size is known: an unknown asset keeps the browser's own behaviour.
|
||||
func (r *Renderer) sizeImage(from *Document, node *ast.Image) {
|
||||
if r.set == nil || from == nil {
|
||||
return
|
||||
}
|
||||
dest := string(node.Destination)
|
||||
if strings.Contains(dest, "://") || strings.HasPrefix(dest, "data:") {
|
||||
return
|
||||
}
|
||||
rel := dest
|
||||
if strings.HasPrefix(rel, "/assets/") {
|
||||
rel = "assets/" + strings.TrimPrefix(rel, "/assets/")
|
||||
} else if i := strings.IndexAny(rel, "#?"); i >= 0 {
|
||||
rel = rel[:i]
|
||||
}
|
||||
w, h, ok := imageSize(filepath.Join(r.set.Root, filepath.FromSlash(path.Clean(rel))))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
node.SetAttributeString("width", []byte(strconv.Itoa(w)))
|
||||
node.SetAttributeString("height", []byte(strconv.Itoa(h)))
|
||||
node.SetAttributeString("loading", []byte("lazy"))
|
||||
node.SetAttributeString("decoding", []byte("async"))
|
||||
}
|
||||
|
||||
// svgSize reads width and height from the root element of an SVG. The diagrams
|
||||
// in content/assets are authored in this repository with explicit dimensions,
|
||||
// so a viewBox alone is not expected to occur.
|
||||
var svgSizeRe = regexp.MustCompile(`<svg[^>]*\swidth="(\d+(?:\.\d+)?)(?:px)?"[^>]*\sheight="(\d+(?:\.\d+)?)(?:px)?"`)
|
||||
|
||||
// imageSize returns the intrinsic pixel size of a content asset.
|
||||
func imageSize(abs string) (int, int, bool) {
|
||||
f, err := os.Open(abs)
|
||||
if err != nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
defer f.Close()
|
||||
if strings.HasSuffix(strings.ToLower(abs), ".svg") {
|
||||
head := make([]byte, 4096)
|
||||
n, _ := f.Read(head)
|
||||
m := svgSizeRe.FindSubmatch(head[:n])
|
||||
if m == nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
w, err1 := strconv.ParseFloat(string(m[1]), 64)
|
||||
h, err2 := strconv.ParseFloat(string(m[2]), 64)
|
||||
if err1 != nil || err2 != nil || w <= 0 || h <= 0 {
|
||||
return 0, 0, false
|
||||
}
|
||||
return int(w + 0.5), int(h + 0.5), true
|
||||
}
|
||||
cfg, _, err := image.DecodeConfig(f)
|
||||
if err != nil || cfg.Width <= 0 || cfg.Height <= 0 {
|
||||
return 0, 0, false
|
||||
}
|
||||
return cfg.Width, cfg.Height, true
|
||||
}
|
||||
|
||||
// docKey carries the document being rendered through the parser context so
|
||||
// the link rewriter knows what "relative" means.
|
||||
var docKey = parser.NewContextKey()
|
||||
|
||||
@@ -413,10 +413,9 @@ func (s *Server) fragmentNotFound(w http.ResponseWriter, msg string) {
|
||||
_, _ = io.WriteString(w, `<div class="stage-empty"><p class="stage-empty-title">`+msg+`</p><p class="stage-hint">The <a href="/evidence/">evidence index</a> lists every fragment in this build.</p></div>`)
|
||||
}
|
||||
|
||||
// handleSearch runs a keyword search. With HTMX it returns just the contents of
|
||||
// the results block, which the search page's own form swaps into #search-results;
|
||||
// without JavaScript the same handler returns the whole page, so the form works
|
||||
// in both cases.
|
||||
// handleSearch runs a keyword search. With HTMX it returns just the results
|
||||
// block; without JavaScript the same handler returns the whole page, so the form
|
||||
// works in both cases.
|
||||
func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) {
|
||||
q := strings.TrimSpace(r.URL.Query().Get("q"))
|
||||
failed := false
|
||||
@@ -452,9 +451,7 @@ func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) {
|
||||
Render(context.Context, io.Writer) error
|
||||
} = templates.SearchPage(view)
|
||||
if r.Header.Get("HX-Request") == "true" {
|
||||
// The element that holds #search-results stays where it is, so the
|
||||
// fragment carries the contents and not the wrapper.
|
||||
c = templates.SearchResultsBody(view)
|
||||
c = templates.SearchResults(view)
|
||||
}
|
||||
if err := c.Render(r.Context(), &buf); err != nil {
|
||||
s.log.Error("render search", "error", err)
|
||||
|
||||
@@ -7,7 +7,6 @@ import (
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -275,14 +274,8 @@ func TestSearchWorksWithAndWithoutHTMX(t *testing.T) {
|
||||
if strings.Contains(frag, "<html") {
|
||||
t.Errorf("htmx search %q returned a whole page, want the results block", tc.q)
|
||||
}
|
||||
// The fragment is swapped into #search-results with innerHTML, so it
|
||||
// must not carry the wrapper again: a second element with the same id
|
||||
// inside the first is invalid and makes the live region announce twice.
|
||||
if strings.Contains(frag, `id="search-results"`) {
|
||||
t.Errorf("htmx search %q re-declares #search-results inside itself", tc.q)
|
||||
}
|
||||
if !strings.Contains(frag, "results for") {
|
||||
t.Errorf("htmx search %q returned no results", tc.q)
|
||||
if !strings.Contains(frag, `id="search-results"`) {
|
||||
t.Errorf("htmx search %q did not return #search-results", tc.q)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -376,190 +369,3 @@ func TestFirstByteBudget(t *testing.T) {
|
||||
t.Errorf("first byte in %s, budget is 100ms", elapsed)
|
||||
}
|
||||
}
|
||||
|
||||
// navSearchForm isolates the navigation search form's opening tag and markup.
|
||||
// The layout has one such form on every page; there is nothing to search for if
|
||||
// it is missing.
|
||||
func navSearchForm(t *testing.T, path, body string) string {
|
||||
t.Helper()
|
||||
marker := strings.Index(body, `class="search nav-search"`)
|
||||
if marker < 0 {
|
||||
t.Fatalf("%s: no navigation search form", path)
|
||||
}
|
||||
start := strings.LastIndex(body[:marker], "<form")
|
||||
end := strings.Index(body[marker:], "</form>")
|
||||
if start < 0 || end < 0 {
|
||||
t.Fatalf("%s: could not isolate the navigation search form", path)
|
||||
}
|
||||
return body[start : marker+end]
|
||||
}
|
||||
|
||||
// TestNavigationSearchNavigatesFromEveryPage is the AUD-151 regression: the nav
|
||||
// form used to carry hx-target="#search-results", which only exists on /search,
|
||||
// so htmx silently discarded the response everywhere else. The nav form must be
|
||||
// a plain GET, and no page may target a fragment it does not carry.
|
||||
func TestNavigationSearchNavigatesFromEveryPage(t *testing.T) {
|
||||
ts := newServer(t)
|
||||
pages := []string{
|
||||
"/",
|
||||
"/ai/", "/platform/", "/edge/", "/products/", "/history/", "/about/",
|
||||
"/evidence/", "/edge/edge-fleet/", "/404.html",
|
||||
}
|
||||
for _, path := range pages {
|
||||
code, _, body := get(t, ts, path, nil)
|
||||
if code != http.StatusOK {
|
||||
t.Errorf("%s: status %d", path, code)
|
||||
continue
|
||||
}
|
||||
form := navSearchForm(t, path, body)
|
||||
if strings.Contains(form, "hx-") {
|
||||
t.Errorf("%s: navigation search form carries an htmx attribute:\n%s", path, form)
|
||||
}
|
||||
if !strings.Contains(form, `action="/search"`) {
|
||||
t.Errorf("%s: navigation search form does not post to /search:\n%s", path, form)
|
||||
}
|
||||
if !strings.Contains(form, `method="get"`) {
|
||||
t.Errorf("%s: navigation search form is not a GET:\n%s", path, form)
|
||||
}
|
||||
if strings.Contains(body, `hx-target="#search-results"`) {
|
||||
t.Errorf("%s: page targets #search-results, which only the search page carries", path)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSearchFromAnyPageLandsOnResults follows the nav form the way a browser with
|
||||
// JavaScript disabled would: GET the form action with q set, expect a page.
|
||||
func TestSearchFromAnyPageLandsOnResults(t *testing.T) {
|
||||
ts := newServer(t)
|
||||
for _, path := range []string{"/", "/edge/", "/platform/"} {
|
||||
code, _, body := get(t, ts, path, nil)
|
||||
if code != http.StatusOK {
|
||||
t.Errorf("%s: status %d", path, code)
|
||||
continue
|
||||
}
|
||||
form := navSearchForm(t, path, body)
|
||||
if !strings.Contains(form, `action="/search"`) {
|
||||
t.Errorf("%s: nav form has no /search action to follow", path)
|
||||
continue
|
||||
}
|
||||
// No HX-Request header: this is the no-JavaScript path.
|
||||
code, _, landed := get(t, ts, "/search?q=zigbee", nil)
|
||||
if code != http.StatusOK {
|
||||
t.Errorf("search from %s: status %d", path, code)
|
||||
continue
|
||||
}
|
||||
if !strings.Contains(strings.ToLower(landed), "<!doctype html>") {
|
||||
t.Errorf("search from %s did not land on a page", path)
|
||||
}
|
||||
if !strings.Contains(landed, "results for") || !strings.Contains(landed, "/edge/") {
|
||||
t.Errorf("search from %s landed on a page with no results", path)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestNoHTMXTargetIsMissingFromItsPage walks the whole built site: every
|
||||
// hx-target selector must name an element in the same document. This is the
|
||||
// general form of the AUD-151 bug, so the next one fails a test instead of
|
||||
// failing silently in a browser.
|
||||
func TestNoHTMXTargetIsMissingFromItsPage(t *testing.T) {
|
||||
target := regexp.MustCompile(`hx-target="([^"]+)"`)
|
||||
files := 0
|
||||
err := filepath.WalkDir(distDir, func(p string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if d.IsDir() || !strings.HasSuffix(p, ".html") {
|
||||
return nil
|
||||
}
|
||||
raw, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
files++
|
||||
body := string(raw)
|
||||
for _, m := range target.FindAllStringSubmatch(body, -1) {
|
||||
selector := m[1]
|
||||
if !strings.HasPrefix(selector, "#") {
|
||||
continue // closest/find targets are resolved from the trigger
|
||||
}
|
||||
id := strings.TrimPrefix(selector, "#")
|
||||
if !strings.Contains(body, `id="`+id+`"`) {
|
||||
t.Errorf("%s: hx-target=%q has no matching element", p, selector)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("walk %s: %v", distDir, err)
|
||||
}
|
||||
if files == 0 {
|
||||
t.Fatal("no HTML files scanned; the build fixture is empty")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSearchEmptyQueryShowsAPrompt covers "an empty query shows a prompt, not an
|
||||
// empty list".
|
||||
func TestSearchEmptyQueryShowsAPrompt(t *testing.T) {
|
||||
ts := newServer(t)
|
||||
code, _, body := get(t, ts, "/search", nil)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("status = %d", code)
|
||||
}
|
||||
if !strings.Contains(body, "Type one or two words above") {
|
||||
t.Error("empty search shows no prompt")
|
||||
}
|
||||
if strings.Contains(body, `class="results"`) {
|
||||
t.Error("empty search renders a results list")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSearchWithNoMatchesSuggestsTheAreas covers "a query with no matches shows a
|
||||
// clear nothing-found message and suggests browsing the areas".
|
||||
func TestSearchWithNoMatchesSuggestsTheAreas(t *testing.T) {
|
||||
ts := newServer(t)
|
||||
code, _, body := get(t, ts, "/search?q=appaloosafoundation", nil)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("status = %d", code)
|
||||
}
|
||||
if !strings.Contains(body, "Nothing found for") {
|
||||
t.Error("no nothing-found message")
|
||||
}
|
||||
for _, href := range []string{`href="/ai/"`, `href="/platform/"`, `href="/edge/"`, `href="/evidence/"`} {
|
||||
if !strings.Contains(body, href) {
|
||||
t.Errorf("nothing-found message does not offer %s", href)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSearchCountIsAnnounced checks the count is inside the polite live region
|
||||
// and marked as a status, so a screen reader reads "4 results for …" rather than
|
||||
// an undifferentiated block of titles.
|
||||
func TestSearchCountIsAnnounced(t *testing.T) {
|
||||
ts := newServer(t)
|
||||
_, _, body := get(t, ts, "/search?q=zigbee", nil)
|
||||
if !strings.Contains(body, `id="search-results" aria-live="polite"`) {
|
||||
t.Error("results container is not a live region")
|
||||
}
|
||||
if !regexp.MustCompile(`<p class="search-count" role="status">\d+ results for`).MatchString(body) {
|
||||
t.Error("result count is not announced as a status inside the live region")
|
||||
}
|
||||
}
|
||||
|
||||
// TestHTMXSearchStillReturnsTheFragment keeps the search page's own upgrade: with
|
||||
// the header set, the handler returns the block and not a whole page.
|
||||
func TestHTMXSearchStillReturnsTheFragment(t *testing.T) {
|
||||
ts := newServer(t)
|
||||
code, _, frag := get(t, ts, "/search?q=zigbee", map[string]string{"HX-Request": "true"})
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("status = %d", code)
|
||||
}
|
||||
if strings.Contains(frag, "<html") {
|
||||
t.Error("htmx search returned a whole page")
|
||||
}
|
||||
if strings.Contains(frag, `id="search-results"`) {
|
||||
t.Error("htmx search re-declares #search-results inside itself")
|
||||
}
|
||||
if !strings.Contains(frag, `role="status"`) || !strings.Contains(frag, "results for") {
|
||||
t.Error("htmx search did not return the results block")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -100,13 +100,6 @@ func (s *Site) Build() error {
|
||||
if err := writePage("/404.html", templates.NotFoundPage(s.NotFoundPage())); err != nil {
|
||||
return err
|
||||
}
|
||||
// The platform guides are configuration and public copy at the same time.
|
||||
// Rendering them here keeps the page and the pipeline's include/exclude rules
|
||||
// from drifting: both read platforms/*.yml.
|
||||
if _, err := RenderPlatforms(s.Config.Root, s.Config.OutDir); err != nil {
|
||||
return fmt.Errorf("platform guides: %w", err)
|
||||
}
|
||||
pages++
|
||||
|
||||
for _, f := range s.Fragments {
|
||||
if err := writeFile(filepath.Join(s.Config.OutDir, filepath.FromSlash(f.File)), []byte(f.HTML)); err != nil {
|
||||
|
||||
@@ -120,7 +120,7 @@ func (s *Site) docFragment(d *content.Document) *Fragment {
|
||||
DocID: d.ID,
|
||||
Kind: stage.KindText,
|
||||
Title: d.Title,
|
||||
subtitle: area.Title,
|
||||
subtitle: area.Title + " · " + templates.LevelLabel(d.Level),
|
||||
body: templ.Raw(d.HTML),
|
||||
confidence: d.Confidence,
|
||||
docURL: d.URL,
|
||||
|
||||
@@ -1,128 +0,0 @@
|
||||
package site
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"html/template"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// PlatformGuide is one `platforms/<name>.yml` file: configuration and public
|
||||
// copy at the same time. The pipeline reads include/exclude from here; the site
|
||||
// shows what, where it lives, and what is read.
|
||||
type PlatformGuide struct {
|
||||
Name string `yaml:"name" json:"name"`
|
||||
Enabled bool `yaml:"enabled" json:"enabled"`
|
||||
What string `yaml:"what" json:"what"`
|
||||
Fits []string `yaml:"fits" json:"fits"`
|
||||
Base string `yaml:"base" json:"base"`
|
||||
Read []string `yaml:"read" json:"read"`
|
||||
Public bool `yaml:"public" json:"public"`
|
||||
Include struct {
|
||||
Repos any `yaml:"repos" json:"repos"`
|
||||
Paths []string `yaml:"paths" json:"paths"`
|
||||
} `yaml:"include" json:"include"`
|
||||
Exclude struct {
|
||||
Repos []string `yaml:"repos" json:"repos"`
|
||||
Paths []string `yaml:"paths" json:"paths"`
|
||||
} `yaml:"exclude" json:"exclude"`
|
||||
}
|
||||
|
||||
// LoadPlatforms reads every guide. A file that does not parse is an error: the
|
||||
// site must not silently show a different list from the one the pipeline reads.
|
||||
func LoadPlatforms(root string) ([]PlatformGuide, error) {
|
||||
dir := filepath.Join(root, "platforms")
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []PlatformGuide
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() || !(strings.HasSuffix(entry.Name(), ".yml") || strings.HasSuffix(entry.Name(), ".yaml")) {
|
||||
continue
|
||||
}
|
||||
raw, err := os.ReadFile(filepath.Join(dir, entry.Name()))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var guide PlatformGuide
|
||||
if err := yaml.Unmarshal(raw, &guide); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", entry.Name(), err)
|
||||
}
|
||||
if guide.Name == "" || guide.What == "" {
|
||||
return nil, fmt.Errorf("%s: name and what are required", entry.Name())
|
||||
}
|
||||
out = append(out, guide)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// RenderPlatforms writes /platforms/index.html and /platforms/index.json.
|
||||
//
|
||||
// The page is generated outside the templ layout because templ needs a Node-free
|
||||
// generator on PATH and the build host has no network requirement. It uses the
|
||||
// same stylesheet, the same navigation and the same wording as the rest of the
|
||||
// site; the only thing it does not have is the presentation panel, which has no
|
||||
// meaning on a reference page.
|
||||
func RenderPlatforms(root, outDir string) (int, error) {
|
||||
guides, err := LoadPlatforms(root)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
target := filepath.Join(outDir, "platforms")
|
||||
if err := os.MkdirAll(target, 0o755); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
var body strings.Builder
|
||||
body.WriteString(`<p class="lede">The assistant reads four sources. Three are harvested from the estate; the fourth is the curated text this site is built from. Everything below is the same configuration the pipeline runs.</p>`)
|
||||
for _, guide := range guides {
|
||||
state := "enabled"
|
||||
if !guide.Enabled {
|
||||
state = "disabled"
|
||||
}
|
||||
body.WriteString(`<section class="platform">`)
|
||||
fmt.Fprintf(&body, `<h2>%s <span class="conf">%s</span></h2>`, template.HTMLEscapeString(guide.Name), state)
|
||||
fmt.Fprintf(&body, `<p>%s</p>`, template.HTMLEscapeString(strings.TrimSpace(guide.What)))
|
||||
fmt.Fprintf(&body, `<dl><dt>Where it lives</dt><dd>%s</dd>`, template.HTMLEscapeString(guide.Base))
|
||||
fmt.Fprintf(&body, `<dt>Feeds</dt><dd>%s</dd>`, template.HTMLEscapeString(strings.Join(guide.Fits, ", ")))
|
||||
fmt.Fprintf(&body, `<dt>Reads</dt><dd>%s</dd>`, template.HTMLEscapeString(strings.Join(guide.Read, ", ")))
|
||||
if len(guide.Include.Paths) > 0 {
|
||||
fmt.Fprintf(&body, `<dt>Includes</dt><dd>%s</dd>`, template.HTMLEscapeString(strings.Join(guide.Include.Paths, ", ")))
|
||||
}
|
||||
if len(guide.Exclude.Paths) > 0 {
|
||||
fmt.Fprintf(&body, `<dt>Excludes</dt><dd>%s</dd>`, template.HTMLEscapeString(strings.Join(guide.Exclude.Paths, ", ")))
|
||||
}
|
||||
body.WriteString(`</dl></section>`)
|
||||
}
|
||||
|
||||
page, err := template.New("platforms").Parse(platformsPage)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
file, err := os.Create(filepath.Join(target, "index.html"))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer file.Close()
|
||||
if err := page.Execute(file, map[string]any{"Body": template.HTML(body.String())}); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
raw, err := json.MarshalIndent(map[string]any{"version": 1, "platforms": guides}, "", " ")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(target, "index.json"), append(raw, '\n'), 0o644); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return len(guides), nil
|
||||
}
|
||||
|
||||
const platformsPage = `<!doctype html><html lang="en-AU"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>Sources and platform guides — Sam Rolfe</title><meta name="description" content="Every source the assistant reads, with the include and exclude rules, shown as configuration and as public copy."><meta name="color-scheme" content="dark"><link rel="stylesheet" href="/static/site.css"></head><body><a class="skip-link" href="#main">Skip to content</a><header class="topbar"><a class="brand" href="/"><span class="brand-mark">SR</span> <span class="brand-text">Sam Rolfe <span class="brand-sub">AI · Platform · Edge</span></span></a><nav class="nav nav-rail" aria-label="Sections"><a href="/ai/" class="nav-link"><span class="nav-gutter" aria-hidden="true">AI</span> <span class="nav-title">AI & Agentic Systems</span></a><a href="/platform/" class="nav-link"><span class="nav-gutter" aria-hidden="true">PL</span> <span class="nav-title">Platform & Infrastructure</span></a><a href="/edge/" class="nav-link"><span class="nav-gutter" aria-hidden="true">ED</span> <span class="nav-title">Edge & Devices</span></a><a href="/products/" class="nav-link"><span class="nav-gutter" aria-hidden="true">PR</span> <span class="nav-title">Products</span></a><a href="/history/" class="nav-link"><span class="nav-gutter" aria-hidden="true">HI</span> <span class="nav-title">History</span></a><a href="/about/" class="nav-link"><span class="nav-gutter" aria-hidden="true">AB</span> <span class="nav-title">About</span></a><a href="/evidence/" class="nav-link"><span class="nav-gutter" aria-hidden="true">EV</span> <span class="nav-title">Evidence index</span></a></nav></header><div class="shell"><main id="main" class="main"><nav class="crumbs" aria-label="Breadcrumb"><ol><li><a href="/">Home</a></li><li><span aria-current="page">Sources</span></li></ol></nav><article class="doc"><header class="doc-head"><p class="doc-kicker">AI overlay</p><h1>What the assistant reads</h1><p class="doc-claim">Four sources, one pool. Content is tier 1: a curated claim outranks harvested text when the two disagree.</p></header>{{.Body}}<p class="doc-foot"><a href="/evidence/">Every artefact</a> · <a href="/search">Search the site</a></p></article></main></div><footer class="footer"><p class="footer-line">Twenty-five years shipping software · Melbourne · remote · hybrid · office · travel</p><p class="footer-links"><a href="/evidence/">Evidence index</a> <a href="/search">Search</a></p></footer></body></html>`
|
||||
@@ -1,42 +0,0 @@
|
||||
package site
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRenderPlatformsShowsTheGuides(t *testing.T) {
|
||||
root, err := filepath.Abs(filepath.Join("..", ".."))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out := t.TempDir()
|
||||
count, err := RenderPlatforms(root, out)
|
||||
if err != nil {
|
||||
t.Fatalf("RenderPlatforms: %v", err)
|
||||
}
|
||||
if count != 3 {
|
||||
t.Fatalf("got %d guides, want 3", count)
|
||||
}
|
||||
raw, err := os.ReadFile(filepath.Join(out, "platforms", "index.html"))
|
||||
if err != nil {
|
||||
t.Fatalf("read page: %v", err)
|
||||
}
|
||||
page := strings.ToLower(string(raw))
|
||||
for _, want := range []string{
|
||||
"what the assistant reads",
|
||||
"self-hosted git",
|
||||
"personal knowledge base",
|
||||
"knowledge graph over the notes",
|
||||
"node_modules",
|
||||
} {
|
||||
if !strings.Contains(page, want) {
|
||||
t.Errorf("page does not contain %q", want)
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(out, "platforms", "index.json")); err != nil {
|
||||
t.Errorf("index.json missing: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -59,23 +59,13 @@ func (Shell) Nav(area string) []templates.NavItem {
|
||||
items = append(items, templates.NavItem{
|
||||
Title: a.Title,
|
||||
Href: "/" + a.Key + "/",
|
||||
Area: a.Key,
|
||||
Gutter: a.Gutter,
|
||||
Current: a.Key == area,
|
||||
})
|
||||
}
|
||||
return items
|
||||
}
|
||||
|
||||
// AreaLinks is the six areas as plain links. The search page offers them when a
|
||||
// query matches nothing, so an empty result is still a way into the site.
|
||||
func (Shell) AreaLinks() []templates.LinkView {
|
||||
out := make([]templates.LinkView, 0, len(content.Areas))
|
||||
for _, a := range content.Areas {
|
||||
out = append(out, templates.LinkView{Label: a.Title, Href: "/" + a.Key + "/"})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// RouteBinding is the Mode A driver: one pre-rendered fragment per click.
|
||||
func (Shell) RouteBinding() stage.Binding {
|
||||
return stage.RouteDriver{BaseURL: "/stage"}.Bind()
|
||||
@@ -124,7 +114,6 @@ func (sh Shell) SearchView(indexed int, q string, results []search.Result, faile
|
||||
Results: results,
|
||||
Indexed: indexed,
|
||||
Failed: failed,
|
||||
Areas: sh.AreaLinks(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -268,6 +257,7 @@ func (s *Site) Home() templates.HomeView {
|
||||
// Doc builds the view for one document at any depth.
|
||||
func (s *Site) Doc(d *content.Document) templates.DocView {
|
||||
area, _ := content.AreaByKey(d.Area)
|
||||
levelLabel := templates.LevelLabel(d.Level)
|
||||
|
||||
view := templates.DocView{
|
||||
Layout: s.Layout(LayoutOpts{
|
||||
@@ -277,14 +267,16 @@ func (s *Site) Doc(d *content.Document) templates.DocView {
|
||||
Crumbs: s.crumbs(d),
|
||||
Binding: s.RouteBinding(),
|
||||
}),
|
||||
Doc: d,
|
||||
Role: area.Role,
|
||||
Body: templates.MarkdownBody(d.HTML),
|
||||
Metrics: make([]templates.MetricView, 0, len(d.Metrics)),
|
||||
Tech: d.Tech,
|
||||
Evidence: s.evidence(d),
|
||||
Backlinks: s.backlinks(d),
|
||||
SubPages: s.subPages(d),
|
||||
Doc: d,
|
||||
AreaTitle: area.Title,
|
||||
Role: area.Role,
|
||||
Body: templates.MarkdownBody(d.HTML),
|
||||
Metrics: make([]templates.MetricView, 0, len(d.Metrics)),
|
||||
Tech: d.Tech,
|
||||
Evidence: s.evidence(d),
|
||||
Backlinks: s.backlinks(d),
|
||||
SubPages: s.subPages(d),
|
||||
LevelLabel: levelLabel,
|
||||
}
|
||||
for _, m := range d.Metrics {
|
||||
view.Metrics = append(view.Metrics, templates.MetricView{Label: m.Label, Value: content.MetricValue(m.Value)})
|
||||
@@ -342,7 +334,7 @@ func (s *Site) evidence(d *content.Document) []templates.LinkView {
|
||||
func (s *Site) backlinks(d *content.Document) []templates.LinkView {
|
||||
out := make([]templates.LinkView, 0, len(d.Backlinks))
|
||||
for _, b := range d.Backlinks {
|
||||
out = append(out, templates.LinkView{Label: b.Title, Href: b.URL})
|
||||
out = append(out, templates.LinkView{Label: b.Title, Href: b.URL, Kind: templates.LevelLabel(b.Level)})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -7,7 +7,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"gitea.lab.audasmedia.com.au/sam/ai-resume/internal/content"
|
||||
"gitea.lab.audasmedia.com.au/sam/ai-resume/internal/search"
|
||||
)
|
||||
|
||||
// newSite loads the repository's own content. If this fails, every other test
|
||||
@@ -173,125 +172,3 @@ func TestBuildWritesEveryPublishedPage(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// copyTree copies a directory tree. The freshness test needs its own content
|
||||
// tree so it can change a page without touching the repository.
|
||||
func copyTree(src, dst string) error {
|
||||
return filepath.WalkDir(src, func(p string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rel, err := filepath.Rel(src, p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
target := filepath.Join(dst, rel)
|
||||
if d.IsDir() {
|
||||
return os.MkdirAll(target, 0o755)
|
||||
}
|
||||
raw, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(target, raw, 0o644)
|
||||
})
|
||||
}
|
||||
|
||||
// TestSearchIndexFollowsContent is the AUD-151 freshness line: a content change
|
||||
// followed by a build is reflected in search. The deploy path is exactly this —
|
||||
// deploy/deploy.sh runs `make dist` before it stages a release, and the build
|
||||
// rewrites dist/search.db from content/ every time (search.Build removes the old
|
||||
// file), so search can never serve an index from a previous content tree.
|
||||
func TestSearchIndexFollowsContent(t *testing.T) {
|
||||
root, err := filepath.Abs("../..")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dir := t.TempDir()
|
||||
contentDir := filepath.Join(dir, "content")
|
||||
if err := copyTree(filepath.Join(root, "content"), contentDir); err != nil {
|
||||
t.Fatalf("copy content: %v", err)
|
||||
}
|
||||
out := filepath.Join(dir, "dist")
|
||||
|
||||
build := func() {
|
||||
t.Helper()
|
||||
s, err := New(Config{Root: root, ContentDir: contentDir, OutDir: out})
|
||||
if err != nil {
|
||||
t.Fatalf("load site: %v", err)
|
||||
}
|
||||
if err := s.Build(); err != nil {
|
||||
t.Fatalf("build: %v", err)
|
||||
}
|
||||
}
|
||||
query := func(q string) []search.Result {
|
||||
t.Helper()
|
||||
idx, err := search.Open(filepath.Join(out, IndexFile))
|
||||
if err != nil {
|
||||
t.Fatalf("open index: %v", err)
|
||||
}
|
||||
defer idx.Close()
|
||||
results, err := idx.Query(q, 20)
|
||||
if err != nil {
|
||||
t.Fatalf("query %q: %v", q, err)
|
||||
}
|
||||
return results
|
||||
}
|
||||
|
||||
const token = "kumquatprotocol"
|
||||
build()
|
||||
if got := query(token); len(got) != 0 {
|
||||
t.Fatalf("token is already indexed in %d results; the test proves nothing", len(got))
|
||||
}
|
||||
|
||||
page := filepath.Join(contentDir, "edge", "edge-fleet.md")
|
||||
raw, err := os.ReadFile(page)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
added := string(raw) + "\n\nA " + token + " harness keeps the nodes in step.\n"
|
||||
if err := os.WriteFile(page, []byte(added), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
build()
|
||||
results := query(token)
|
||||
if len(results) == 0 {
|
||||
t.Fatal("content change is not in the rebuilt index")
|
||||
}
|
||||
if results[0].URL != "/edge/edge-fleet/" {
|
||||
t.Errorf("token found at %s, want /edge/edge-fleet/", results[0].URL)
|
||||
}
|
||||
}
|
||||
|
||||
// TestImagesReserveTheirSpace is the layout-shift guard: every image a content
|
||||
// page carries must declare its intrinsic size and load lazily, so a page of
|
||||
// diagrams and screenshots never reflows as the files arrive.
|
||||
func TestImagesReserveTheirSpace(t *testing.T) {
|
||||
s := newSite(t)
|
||||
seen := 0
|
||||
for _, d := range s.Content.Docs {
|
||||
html := d.HTML
|
||||
for i := 0; ; i++ {
|
||||
start := strings.Index(html, "<img ")
|
||||
if start < 0 {
|
||||
break
|
||||
}
|
||||
end := strings.Index(html[start:], ">")
|
||||
if end < 0 {
|
||||
break
|
||||
}
|
||||
tag := html[start : start+end+1]
|
||||
html = html[start+end+1:]
|
||||
seen++
|
||||
for _, want := range []string{`width="`, `height="`, `loading="lazy"`, `alt="`} {
|
||||
if !strings.Contains(tag, want) {
|
||||
t.Errorf("%s: image tag missing %s: %s", d.Path, want, tag)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if seen == 0 {
|
||||
t.Fatal("no images found in content; the guard is not exercising anything")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32">
|
||||
<rect width="32" height="32" rx="7" fill="#0b0d10"/>
|
||||
<rect x="0.75" y="0.75" width="30.5" height="30.5" rx="6.25" fill="none" stroke="#33404d" stroke-width="1.5"/>
|
||||
<path d="M11.2 22.2c-1.9-.5-3-1.7-3-3.5h2.6c0 .9.5 1.4 1.5 1.6.9.2 1.6 0 1.8-.6.2-.6-.2-1-.9-1.2l-2-.4c-2-.4-3-1.4-3-3 0-2 1.6-3.3 4.1-3.4v-1.9h1.9v1.9c2 .2 3.1 1.4 3.2 3.2h-2.6c-.1-.8-.6-1.2-1.4-1.3-.8-.1-1.4.2-1.4.7-.1.5.3.8 1 .9l2 .4c2 .4 3 1.5 3 3.2 0 2.1-1.7 3.4-4.2 3.5v1.9h-1.9z" fill="#6ee7c6"/>
|
||||
<path d="M19.6 10.3h4.9c1.9 0 3.1 1.1 3.1 2.8 0 1.1-.5 1.9-1.4 2.3 1.1.3 1.7 1.2 1.7 2.5v1.9h-2.6v-1.6c0-.9-.4-1.3-1.3-1.3h-1.8v-2.3h1.5c.7 0 1.1-.3 1.1-.9s-.4-.9-1.1-.9h-1.5z" fill="#b3bcc8"/>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 761 B |
@@ -151,22 +151,13 @@ samp {
|
||||
|
||||
/* ---- chrome ------------------------------------------------------------ */
|
||||
|
||||
/*
|
||||
Two rows: the name and the search on the first, the six areas plus the index
|
||||
on the second. The area names are long and the marks made them longer, so a
|
||||
single row either overlapped the search form or scrolled the nav out of
|
||||
reach. Two rows give every link a real target at every width.
|
||||
*/
|
||||
.topbar {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 20;
|
||||
display: grid;
|
||||
grid-template-columns: var(--nav-w) minmax(0, 1fr);
|
||||
grid-template-areas:
|
||||
"brand search"
|
||||
"nav nav";
|
||||
gap: 0.5rem var(--pad);
|
||||
grid-template-columns: var(--nav-w) minmax(0, 1fr) auto;
|
||||
gap: var(--pad);
|
||||
align-items: center;
|
||||
padding: 0.7rem var(--pad);
|
||||
background: color-mix(in srgb, var(--bg) 92%, transparent);
|
||||
@@ -174,21 +165,6 @@ samp {
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.brand {
|
||||
grid-area: brand;
|
||||
}
|
||||
|
||||
.nav-search {
|
||||
grid-area: search;
|
||||
justify-self: end;
|
||||
width: min(28rem, 100%);
|
||||
}
|
||||
|
||||
.nav-rail {
|
||||
grid-area: nav;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.brand {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
@@ -265,25 +241,17 @@ samp {
|
||||
box-shadow: inset 0 -2px 0 var(--accent);
|
||||
}
|
||||
|
||||
/* The area mark replaces the old two-letter badge. It is decoration; the nav
|
||||
label beside it is the accessible name. */
|
||||
.nav-mark {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
.nav-gutter {
|
||||
font-family: var(--font-mono);
|
||||
font-size: 0.68rem;
|
||||
letter-spacing: 0.08em;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.nav-link:hover .nav-mark,
|
||||
.nav-current .nav-mark {
|
||||
.nav-current .nav-gutter {
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
.area-icon {
|
||||
display: block;
|
||||
width: 20px;
|
||||
height: 20px;
|
||||
}
|
||||
|
||||
/* ---- shell ------------------------------------------------------------- */
|
||||
|
||||
.shell {
|
||||
@@ -596,9 +564,8 @@ samp {
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.pillar-mark {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
.pillar-gutter {
|
||||
font-family: var(--font-mono);
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
@@ -682,11 +649,8 @@ samp {
|
||||
list-style: none;
|
||||
}
|
||||
|
||||
.tech-chip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.35rem;
|
||||
padding: 0.14rem 0.45rem;
|
||||
.tech li {
|
||||
padding: 0.12rem 0.45rem;
|
||||
border: 1px solid var(--line);
|
||||
border-radius: var(--radius-sm);
|
||||
font-family: var(--font-mono);
|
||||
@@ -694,65 +658,26 @@ samp {
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
/* Monochrome and currentColor, so the chip and the mark share one colour and
|
||||
no icon ever adds a request. */
|
||||
.tech-mark {
|
||||
display: block;
|
||||
flex: none;
|
||||
width: 16px;
|
||||
height: 16px;
|
||||
fill: currentColor;
|
||||
opacity: 0.85;
|
||||
}
|
||||
|
||||
.tech-chip:hover {
|
||||
border-color: var(--line-strong);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.tech-chip:hover .tech-mark {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
/* ---- client wall ------------------------------------------------------- */
|
||||
/*
|
||||
One typeface, one size, one tile. Mixed brand marks would read as broken;
|
||||
a consistent grid reads as deliberate. Tiles are the same height whatever
|
||||
the length of the name, so the wall never looks like a failed image load.
|
||||
*/
|
||||
.client-wall {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fill, minmax(10.5rem, 1fr));
|
||||
gap: 1px;
|
||||
margin: 1rem 0 1.6rem;
|
||||
padding: 1px;
|
||||
background: var(--line);
|
||||
border: 1px solid var(--line);
|
||||
border-radius: var(--radius-sm);
|
||||
overflow: hidden;
|
||||
list-style: none;
|
||||
}
|
||||
|
||||
.client-tile {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
min-height: 4.2rem;
|
||||
padding: 0.6rem 0.75rem;
|
||||
.honesty {
|
||||
padding: 1.2rem 1.4rem;
|
||||
background: var(--surface);
|
||||
text-align: center;
|
||||
border: 1px solid var(--line);
|
||||
border-left: 3px solid var(--accent);
|
||||
border-radius: var(--radius);
|
||||
}
|
||||
|
||||
.client-name {
|
||||
font-size: 0.88rem;
|
||||
line-height: 1.25;
|
||||
letter-spacing: 0.01em;
|
||||
.honesty h2 {
|
||||
margin-top: 0;
|
||||
font-size: 1.05rem;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.08em;
|
||||
color: var(--text-dim);
|
||||
text-wrap: balance;
|
||||
}
|
||||
|
||||
.client-tile:hover .client-name {
|
||||
color: var(--text);
|
||||
.honesty ul {
|
||||
margin: 0;
|
||||
padding-left: 1.1rem;
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.doc-head {
|
||||
@@ -984,15 +909,6 @@ samp {
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.search-areas {
|
||||
margin: 0.6rem 0 0;
|
||||
padding: 0;
|
||||
list-style: none;
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 0.4rem 1.1rem;
|
||||
}
|
||||
|
||||
.results {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
@@ -1373,6 +1289,7 @@ figure.media figcaption {
|
||||
.hero,
|
||||
.pillar,
|
||||
.result,
|
||||
.honesty,
|
||||
.drill-list li,
|
||||
.stage,
|
||||
.prose blockquote {
|
||||
@@ -1397,29 +1314,6 @@ figure.media figcaption {
|
||||
color: #222;
|
||||
}
|
||||
|
||||
/* The wall and the chips are surfaces, not decoration: on paper they go
|
||||
white with a hairline, and the marks keep their shape in one ink. */
|
||||
.client-wall {
|
||||
background: #bbb;
|
||||
border-color: #bbb;
|
||||
}
|
||||
|
||||
.client-tile,
|
||||
.tech-chip {
|
||||
background: #fff;
|
||||
border-color: #bbb;
|
||||
}
|
||||
|
||||
.client-name,
|
||||
.tech-chip {
|
||||
color: #111;
|
||||
}
|
||||
|
||||
.nav-mark,
|
||||
.pillar-mark {
|
||||
color: #000;
|
||||
}
|
||||
|
||||
a {
|
||||
color: #000;
|
||||
border-bottom: 0;
|
||||
|
||||
@@ -17,7 +17,6 @@ templ Layout(v LayoutView) {
|
||||
<title>{ v.Title }</title>
|
||||
<meta name="description" content={ v.Description }/>
|
||||
<meta name="color-scheme" content="dark"/>
|
||||
<link rel="icon" href="/static/favicon.svg" type="image/svg+xml"/>
|
||||
<link rel="stylesheet" href="/static/site.css"/>
|
||||
<link rel="stylesheet" href="/static/chroma.css"/>
|
||||
<script src="/static/htmx.min.js" defer></script>
|
||||
@@ -69,8 +68,7 @@ templ Nav(items []NavItem, current string) {
|
||||
templ navLinks(items []NavItem, current string) {
|
||||
for _, it := range items {
|
||||
<a href={ templ.SafeURL(it.Href) } class={ navClass(it) } aria-current={ currentAttr(it) }>
|
||||
<span class="nav-mark">@AreaIcon(it.Area)
|
||||
</span>
|
||||
<span class="nav-gutter" aria-hidden="true">{ it.Gutter }</span>
|
||||
<span class="nav-title">{ it.Title }</span>
|
||||
</a>
|
||||
}
|
||||
@@ -78,8 +76,7 @@ templ navLinks(items []NavItem, current string) {
|
||||
|
||||
templ evidenceLink(current string) {
|
||||
<a href="/evidence/" class={ navClassFor(current == "evidence") }>
|
||||
<span class="nav-mark">@AreaIcon("evidence")
|
||||
</span>
|
||||
<span class="nav-gutter" aria-hidden="true">EV</span>
|
||||
<span class="nav-title">Evidence index</span>
|
||||
</a>
|
||||
}
|
||||
@@ -104,43 +101,17 @@ templ Breadcrumbs(crumbs []Crumb) {
|
||||
}
|
||||
}
|
||||
|
||||
// SearchForm is the plain GET form used in the navigation and the landing page.
|
||||
// It navigates to /search and carries no htmx attributes, because #search-results
|
||||
// only exists on the search page: an hx-target that is not in the document makes
|
||||
// htmx discard the response, which is a form that looks like it works and does
|
||||
// nothing. A plain GET works from every page and with JavaScript disabled.
|
||||
// SearchForm is a GET form: it works with JavaScript disabled and is upgraded by
|
||||
// HTMX when scripts are available.
|
||||
templ SearchForm(q string, class string) {
|
||||
<form class={ "search", class } action="/search" method="get" role="search">
|
||||
@searchField(q, class)
|
||||
<form class={ "search", class } action="/search" method="get" role="search"
|
||||
hx-get="/search" hx-target="#search-results" hx-swap="innerHTML" hx-push-url="true">
|
||||
<label class="visually-hidden" for={ "q-" + class }>Search the site</label>
|
||||
<input id={ "q-" + class } type="search" name="q" value={ q } placeholder="Search — try “zigbee”, “signage”, “routing”" autocomplete="off"/>
|
||||
<button type="submit">Search</button>
|
||||
</form>
|
||||
}
|
||||
|
||||
// SearchFormLive is the search page's own form. HTMX swaps the results block in
|
||||
// place, which is safe here and only here: #search-results is part of the page.
|
||||
templ SearchFormLive(q string, class string) {
|
||||
<form
|
||||
class={ "search", class }
|
||||
action="/search"
|
||||
method="get"
|
||||
role="search"
|
||||
hx-get="/search"
|
||||
hx-target="#search-results"
|
||||
hx-swap="innerHTML"
|
||||
hx-push-url="true"
|
||||
>
|
||||
@searchField(q, class)
|
||||
<button type="submit">Search</button>
|
||||
</form>
|
||||
}
|
||||
|
||||
// searchField is the label and input both forms share. The id is derived from the
|
||||
// form class, so two forms on one page never collide.
|
||||
templ searchField(q string, class string) {
|
||||
<label class="visually-hidden" for={ "q-" + class }>Search the site</label>
|
||||
<input id={ "q-" + class } type="search" name="q" value={ q } placeholder="Search — try “zigbee”, “signage”, “routing”" autocomplete="off"/>
|
||||
}
|
||||
|
||||
// Stage is the presentation panel. Everything about how it is fed comes from
|
||||
// the binding: route mode swaps in pre-rendered fragments, stream mode listens
|
||||
// on an event stream. The markup is identical either way.
|
||||
@@ -181,8 +152,7 @@ templ FragmentView(f stage.Fragment) {
|
||||
</header>
|
||||
switch f.Kind {
|
||||
case stage.KindText:
|
||||
<div class="frag-text">@f.Body
|
||||
</div>
|
||||
<div class="frag-text">@f.Body</div>
|
||||
case stage.KindCode:
|
||||
if f.Code != nil {
|
||||
<figure class="code">
|
||||
@@ -267,8 +237,7 @@ templ EvidenceTable(rows []EvidenceRow) {
|
||||
templ PillarCard(c PillarCardView) {
|
||||
<article class="pillar">
|
||||
<p class="pillar-kicker">
|
||||
<span class="pillar-mark">@AreaIcon(c.Area.Key)
|
||||
</span>
|
||||
<span class="pillar-gutter" aria-hidden="true">{ c.Area.Gutter }</span>
|
||||
if c.Area.Role != "" {
|
||||
<span>{ c.Area.Role }</span>
|
||||
}
|
||||
@@ -288,9 +257,20 @@ templ PillarCard(c PillarCardView) {
|
||||
</ul>
|
||||
}
|
||||
if len(c.Tech) > 0 {
|
||||
@TechChips(c.Tech)
|
||||
<ul class="tech" aria-label="Technology">
|
||||
for _, t := range c.Tech {
|
||||
<li>{ t }</li>
|
||||
}
|
||||
</ul>
|
||||
}
|
||||
<p class="pillar-foot">
|
||||
<a class="pillar-go" href={ templ.SafeURL(c.Href) }>
|
||||
if c.Level == 1 {
|
||||
<span>Start here</span>
|
||||
} else {
|
||||
<span>Read the capability page</span>
|
||||
}
|
||||
</a>
|
||||
<span class="pillar-count">{ strconv.Itoa(c.Pages) } pages</span>
|
||||
<span class={ ConfidenceClass(c.Confidence) }>{ c.Confidence }</span>
|
||||
</p>
|
||||
@@ -321,10 +301,15 @@ templ CodeBlock(c stage.Code) {
|
||||
</figure>
|
||||
}
|
||||
|
||||
// Footer closes the page.
|
||||
// Footer closes the page with the honesty notes a recruiter needs.
|
||||
templ Footer(v LayoutView) {
|
||||
<footer class="footer">
|
||||
<p class="footer-line">Twenty-five years shipping software · Melbourne · remote · hybrid · office · travel</p>
|
||||
<p class="footer-note">
|
||||
Status labels are literal: <span class="conf conf-running">running</span> and
|
||||
<span class="conf conf-delivered">delivered</span> are claims; <span class="conf conf-in-progress">in-progress</span> and
|
||||
<span class="conf conf-abandoned">abandoned</span> are stated as such.
|
||||
</p>
|
||||
<p class="footer-links">
|
||||
<a href="/evidence/">Evidence index</a>
|
||||
<a href="/search">Search</a>
|
||||
|
||||
@@ -1,67 +0,0 @@
|
||||
package templates
|
||||
|
||||
import "gitea.lab.audasmedia.com.au/sam/ai-resume/internal/brand"
|
||||
|
||||
// AreaIcon is the mark for one area of the site. The six marks are drawn here,
|
||||
// geometrically, in one visual language: 24x24, 1.6 px stroke, round caps, no
|
||||
// fill, colour inherited from the text. They are decoration only — the label
|
||||
// beside them carries the meaning — so every one is aria-hidden.
|
||||
//
|
||||
// "evidence" is the seventh mark: the index, not an area.
|
||||
templ AreaIcon(area string) {
|
||||
<svg class="area-icon" viewBox="0 0 24 24" width="20" height="20" aria-hidden="true" focusable="false" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round">
|
||||
switch area {
|
||||
case "ai":
|
||||
<circle cx="5.5" cy="6.5" r="2.5"/>
|
||||
<circle cx="18.5" cy="6.5" r="2.5"/>
|
||||
<circle cx="12" cy="17.5" r="2.5"/>
|
||||
<path d="M8 6.5h8"/>
|
||||
<path d="M6.7 8.8 10.8 15.6"/>
|
||||
<path d="M17.3 8.8 13.2 15.6"/>
|
||||
case "platform":
|
||||
<rect x="3.5" y="3.5" width="17" height="4.6" rx="1.5"/>
|
||||
<rect x="3.5" y="9.7" width="17" height="4.6" rx="1.5"/>
|
||||
<rect x="3.5" y="15.9" width="17" height="4.6" rx="1.5"/>
|
||||
<path d="M7 5.8h2M7 12h2M7 18.2h2"/>
|
||||
case "edge":
|
||||
<rect x="2.5" y="13" width="9" height="8" rx="1.5"/>
|
||||
<circle cx="7" cy="17" r="1.6"/>
|
||||
<path d="M15 11.5a3.5 3.5 0 0 1-3.5 3.5"/>
|
||||
<path d="M18 11.5a6.5 6.5 0 0 1-6.5 6.5"/>
|
||||
case "products":
|
||||
<path d="M12 3.2 19 7.3v9.4L12 20.8 5 16.7V7.3z"/>
|
||||
<path d="M5 7.3l7 4.1 7-4.1"/>
|
||||
<path d="M12 11.4v9.4"/>
|
||||
case "history":
|
||||
<circle cx="12" cy="12" r="8.5"/>
|
||||
<path d="M12 7v5.2l3.4 2.1"/>
|
||||
case "about":
|
||||
<circle cx="12" cy="8.4" r="3.6"/>
|
||||
<path d="M5.4 20a6.6 6.6 0 0 1 13.2 0"/>
|
||||
case "evidence":
|
||||
<path d="M6.8 3.4h7.2l4 4v13.2H6.8z"/>
|
||||
<path d="M14 3.4v4h4"/>
|
||||
<path d="M9.6 14.2l2 2 3.6-4.1"/>
|
||||
}
|
||||
</svg>
|
||||
}
|
||||
|
||||
// TechChips is the technology list on a page. A chip carries a published mark
|
||||
// when one exists and plain text when it does not, so a missing icon reads as a
|
||||
// design decision rather than a broken image.
|
||||
templ TechChips(items []string) {
|
||||
<ul class="tech" aria-label="Technology">
|
||||
for _, t := range items {
|
||||
<li class="tech-chip">
|
||||
if paths, ok := brand.Mark(t); ok {
|
||||
<svg class="tech-mark" viewBox="0 0 24 24" width="16" height="16" aria-hidden="true" focusable="false">
|
||||
for _, d := range paths {
|
||||
<path d={ d } fill="currentColor"/>
|
||||
}
|
||||
</svg>
|
||||
}
|
||||
<span>{ t }</span>
|
||||
</li>
|
||||
}
|
||||
</ul>
|
||||
}
|
||||
@@ -17,11 +17,23 @@ templ HomePage(v HomeView) {
|
||||
</p>
|
||||
@SearchForm("", "hero-search")
|
||||
</section>
|
||||
|
||||
<section class="pillars" aria-label="Six areas">
|
||||
for _, c := range v.Cards {
|
||||
@PillarCard(c)
|
||||
}
|
||||
</section>
|
||||
|
||||
<section class="honesty" aria-labelledby="how-to-read">
|
||||
<h2 id="how-to-read">How to read this site</h2>
|
||||
<ul>
|
||||
<li><strong>Level 1</strong> is one screen per area: the claim, the numbers, the way in.</li>
|
||||
<li><strong>Level 2</strong> is what is actually run, with an evidence table and the trade-offs.</li>
|
||||
<li><strong>Level 3</strong> is the artefact: code, diagram, screenshot, log.</li>
|
||||
<li>Every claim carries a <strong>status</strong>, and finished, running, in-progress and abandoned are all shown as they are.</li>
|
||||
<li>Every artefact is at most three clicks from here, and the <a href="/evidence/">evidence index</a> lists all of them on one page.</li>
|
||||
</ul>
|
||||
</section>
|
||||
}
|
||||
}
|
||||
|
||||
@@ -30,6 +42,7 @@ templ DocPage(v DocView) {
|
||||
@Layout(v.Layout) {
|
||||
<article class="doc">
|
||||
<header class="doc-head">
|
||||
<p class="doc-kicker">{ v.AreaTitle } · { v.LevelLabel }</p>
|
||||
<h1 class="doc-title">{ v.Doc.Title }</h1>
|
||||
<p class="doc-claim">{ v.Doc.Claim }</p>
|
||||
<div class="doc-chips">
|
||||
@@ -45,42 +58,47 @@ templ DocPage(v DocView) {
|
||||
}
|
||||
</div>
|
||||
if len(v.Tech) > 0 {
|
||||
@TechChips(v.Tech)
|
||||
<ul class="tech" aria-label="Technology">
|
||||
for _, t := range v.Tech {
|
||||
<li>{ t }</li>
|
||||
}
|
||||
</ul>
|
||||
}
|
||||
</header>
|
||||
|
||||
<div class="prose">
|
||||
@v.Body
|
||||
</div>
|
||||
|
||||
// Level 1 is one screen: the claim, the figures, the technology and the
|
||||
// link in the prose. The evidence table starts at level 2, which is the
|
||||
// page that argues for the claim.
|
||||
if v.Doc.Level >= 2 && (len(v.Evidence) > 0 || len(v.SubPages) > 0) {
|
||||
if len(v.Evidence) > 0 {
|
||||
<section class="doc-evidence" aria-labelledby="evidence-heading">
|
||||
<h2 id="evidence-heading">Artefacts</h2>
|
||||
if len(v.Evidence) > 0 {
|
||||
@EvidenceTable(evidenceRows(v.Evidence))
|
||||
}
|
||||
if len(v.SubPages) > 0 {
|
||||
<ul class="drill-list">
|
||||
for _, s := range v.SubPages {
|
||||
<li>
|
||||
<a href={ templ.SafeURL(s.Href) }>{ s.Label }</a>
|
||||
<span class="drill-kind">{ s.Kind }</span>
|
||||
</li>
|
||||
}
|
||||
</ul>
|
||||
}
|
||||
<h2 id="evidence-heading">Evidence for this page</h2>
|
||||
<p class="section-note">Select a row to load the artefact into the presentation panel. Each row is also a link, so this works without JavaScript.</p>
|
||||
@EvidenceTable(evidenceRows(v.Evidence))
|
||||
</section>
|
||||
}
|
||||
|
||||
if v.Doc.Level == 2 && len(v.Timeline) > 0 {
|
||||
if len(v.Timeline) > 0 {
|
||||
<section class="doc-timeline" aria-labelledby="timeline-heading">
|
||||
<h2 id="timeline-heading">In time order</h2>
|
||||
@Timeline(v.Timeline)
|
||||
</section>
|
||||
}
|
||||
|
||||
if len(v.SubPages) > 0 {
|
||||
<section class="doc-sub" aria-labelledby="sub-heading">
|
||||
<h2 id="sub-heading">{ subHeading(v.Doc.Level) }</h2>
|
||||
<ul class="drill-list">
|
||||
for _, s := range v.SubPages {
|
||||
<li>
|
||||
<a href={ templ.SafeURL(s.Href) }>{ s.Label }</a>
|
||||
<span class="drill-kind">{ s.Kind }</span>
|
||||
</li>
|
||||
}
|
||||
</ul>
|
||||
</section>
|
||||
}
|
||||
|
||||
if len(v.Backlinks) > 0 {
|
||||
<section class="doc-back" aria-labelledby="back-heading">
|
||||
<h2 id="back-heading">Referenced from</h2>
|
||||
@@ -91,8 +109,10 @@ templ DocPage(v DocView) {
|
||||
</ul>
|
||||
</section>
|
||||
}
|
||||
|
||||
<footer class="doc-foot">
|
||||
<p>Last verified { v.Doc.LastVerified } · <code>content/{ v.Doc.Path }</code></p>
|
||||
<p>Status <span class={ ConfidenceClass(v.Doc.Confidence) }>{ v.Doc.Confidence }</span> — { ConfidenceNote(v.Doc.Confidence) }</p>
|
||||
<p>Last verified { v.Doc.LastVerified } · id <code>{ v.Doc.ID }</code> · { v.Doc.Path }</p>
|
||||
</footer>
|
||||
</article>
|
||||
}
|
||||
@@ -128,65 +148,38 @@ templ SearchPage(v SearchView) {
|
||||
<h1>Search { strconv.Itoa(v.Indexed) } pages</h1>
|
||||
<p class="doc-claim">Full-text search over page titles and body text. Served from a local index — no model, no network call.</p>
|
||||
</header>
|
||||
@SearchFormLive(v.Query, "page-search")
|
||||
@SearchForm(v.Query, "page-search")
|
||||
@SearchResults(v)
|
||||
</article>
|
||||
}
|
||||
}
|
||||
|
||||
// SearchResults is the results block as a page element: the live region wrapper
|
||||
// plus its contents, which is what the whole page renders. Without JavaScript the
|
||||
// form reloads the page and this markup arrives in the document.
|
||||
// SearchResults is swapped into #search-results by HTMX.
|
||||
templ SearchResults(v SearchView) {
|
||||
<div id="search-results" aria-live="polite">
|
||||
@SearchResultsBody(v)
|
||||
</div>
|
||||
}
|
||||
|
||||
// SearchResultsBody is the results block without the live region. This is what an
|
||||
// htmx request returns: the search page's form swaps it into #search-results with
|
||||
// innerHTML, so returning the wrapper too would nest a second element with the
|
||||
// same id inside the first and make the live region announce twice.
|
||||
//
|
||||
// Everything here arrives inside the polite live region, and the count carries
|
||||
// role="status" on top of that, so the number is read as one utterance instead of
|
||||
// a run of result titles.
|
||||
templ SearchResultsBody(v SearchView) {
|
||||
if v.Query == "" {
|
||||
<p class="search-empty">Type one or two words above and press Enter. Good starting points: “zigbee”, “signage”, “routing”, “backups”, “guardrails”.</p>
|
||||
} else if v.Failed {
|
||||
<p class="search-empty">That query could not be run. Try fewer or simpler words.</p>
|
||||
} else if len(v.Results) == 0 {
|
||||
<p class="search-empty">Nothing found for <q>{ v.Query }</q>. Try a shorter or more general word, or browse an area:</p>
|
||||
@searchAreaLinks(v.Areas)
|
||||
} else {
|
||||
<p class="search-count" role="status">{ strconv.Itoa(len(v.Results)) } results for <q>{ v.Query }</q></p>
|
||||
<ol class="results">
|
||||
for _, r := range v.Results {
|
||||
<li class="result">
|
||||
<h2><a href={ templ.SafeURL(r.URL) }>{ r.Title }</a></h2>
|
||||
<p class="result-meta">
|
||||
<span class="kind">{ r.Area }</span>
|
||||
<span class="kind">L{ strconv.Itoa(r.Level) }</span>
|
||||
<span class={ ConfidenceClass(r.Confidence) }>{ r.Confidence }</span>
|
||||
</p>
|
||||
<p class="result-snippet">@SafeSnippet(r.Snippet)
|
||||
</p>
|
||||
</li>
|
||||
}
|
||||
</ol>
|
||||
}
|
||||
}
|
||||
|
||||
// searchAreaLinks is the way out of an empty result: the six areas plus the
|
||||
// evidence index, so a failed search never leaves the reader at a dead end.
|
||||
templ searchAreaLinks(areas []LinkView) {
|
||||
<ul class="search-areas">
|
||||
for _, a := range areas {
|
||||
<li><a href={ templ.SafeURL(a.Href) }>{ a.Label }</a></li>
|
||||
if v.Query == "" {
|
||||
<p class="section-note">Try “zigbee”, “signage”, “routing”, “backups” or “guardrails”.</p>
|
||||
} else if v.Failed {
|
||||
<p class="search-empty">That query could not be run. Try fewer or simpler words.</p>
|
||||
} else if len(v.Results) == 0 {
|
||||
<p class="search-empty">No page matches <q>{ v.Query }</q>. The <a href="/evidence/">evidence index</a> lists everything by area.</p>
|
||||
} else {
|
||||
<p class="search-count">{ strconv.Itoa(len(v.Results)) } results for <q>{ v.Query }</q></p>
|
||||
<ol class="results">
|
||||
for _, r := range v.Results {
|
||||
<li class="result">
|
||||
<h2><a href={ templ.SafeURL(r.URL) }>{ r.Title }</a></h2>
|
||||
<p class="result-meta">
|
||||
<span class="kind">{ r.Area }</span>
|
||||
<span class="kind">L{ strconv.Itoa(r.Level) }</span>
|
||||
<span class={ ConfidenceClass(r.Confidence) }>{ r.Confidence }</span>
|
||||
</p>
|
||||
<p class="result-snippet">@SafeSnippet(r.Snippet)</p>
|
||||
</li>
|
||||
}
|
||||
</ol>
|
||||
}
|
||||
<li><a href="/evidence/">Evidence index</a></li>
|
||||
</ul>
|
||||
</div>
|
||||
}
|
||||
|
||||
// StreamPage proves the panel can be re-pointed at a stream. The layout and the
|
||||
@@ -200,21 +193,15 @@ templ StreamPage(v StreamView) {
|
||||
<p class="doc-claim">This page renders the same layout and the same panel as every other page. Its panel is bound to an event stream instead of pre-rendered files, and no layout code changed.</p>
|
||||
</header>
|
||||
<div class="prose">
|
||||
<p>
|
||||
The presentation panel is a content sink. In Mode A it takes pre-rendered fragments over
|
||||
<p>The presentation panel is a content sink. In Mode A it takes pre-rendered fragments over
|
||||
HTMX: you click a row in an evidence table and one file is swapped in. On this page the same
|
||||
panel is bound to <code>/api/stage/stream</code> and receives { strconv.Itoa(v.Frames) } fragments
|
||||
over server-sent events, one after another.
|
||||
</p>
|
||||
<p>
|
||||
Fragments are pushed, not requested. Nothing about the panel — its markup, its fragment
|
||||
over server-sent events, one after another.</p>
|
||||
<p>Fragments are pushed, not requested. Nothing about the panel — its markup, its fragment
|
||||
view, its empty state — knows the difference. That is the point: the Phase 5 console
|
||||
re-points this panel at retrieval output and the layout does not change.
|
||||
</p>
|
||||
<p class="section-note">
|
||||
This demonstration streams the artefacts already built for the site, with
|
||||
no model in the loop. It exists to prove the transport, not to pretend an AI is answering.
|
||||
</p>
|
||||
re-points this panel at retrieval output and the layout does not change.</p>
|
||||
<p class="section-note">This demonstration streams the artefacts already built for the site, with
|
||||
no model in the loop. It exists to prove the transport, not to pretend an AI is answering.</p>
|
||||
</div>
|
||||
if len(v.Fragment) > 0 {
|
||||
<section class="doc-sub">
|
||||
@@ -259,3 +246,9 @@ func evidenceRows(links []LinkView) []EvidenceRow {
|
||||
return rows
|
||||
}
|
||||
|
||||
func subHeading(level int) string {
|
||||
if level == 1 {
|
||||
return "Drill-downs in this area"
|
||||
}
|
||||
return "Artefacts behind this page"
|
||||
}
|
||||
|
||||
@@ -23,7 +23,7 @@ import (
|
||||
type NavItem struct {
|
||||
Title string
|
||||
Href string
|
||||
Area string
|
||||
Gutter string
|
||||
Current bool
|
||||
}
|
||||
|
||||
@@ -124,16 +124,18 @@ type (
|
||||
|
||||
// DocView is an L1, L2 or L3 content page.
|
||||
DocView struct {
|
||||
Layout LayoutView
|
||||
Doc *content.Document
|
||||
Role string
|
||||
Body templ.Component
|
||||
Metrics []MetricView
|
||||
Tech []string
|
||||
Evidence []LinkView
|
||||
Backlinks []LinkView
|
||||
SubPages []LinkView
|
||||
Timeline []TimelineItem
|
||||
Layout LayoutView
|
||||
Doc *content.Document
|
||||
AreaTitle string
|
||||
Role string
|
||||
Body templ.Component
|
||||
Metrics []MetricView
|
||||
Tech []string
|
||||
Evidence []LinkView
|
||||
Backlinks []LinkView
|
||||
SubPages []LinkView
|
||||
Timeline []TimelineItem
|
||||
LevelLabel string
|
||||
}
|
||||
|
||||
// EvidenceView is the index of every artefact.
|
||||
@@ -151,9 +153,6 @@ type (
|
||||
Total int
|
||||
Indexed int
|
||||
Failed bool
|
||||
// Areas are the six area landings, offered as the way out of an
|
||||
// empty result. Nothing here was ever a search hit.
|
||||
Areas []LinkView
|
||||
}
|
||||
|
||||
// StreamView is the SSE-driven stage demonstration page.
|
||||
@@ -198,3 +197,33 @@ func ConfidenceClass(c string) string {
|
||||
}
|
||||
return "conf"
|
||||
}
|
||||
|
||||
// ConfidenceNote is the sentence shown under a status badge.
|
||||
func ConfidenceNote(c string) string {
|
||||
switch c {
|
||||
case content.ConfidenceDelivered:
|
||||
return "Delivered."
|
||||
case content.ConfidenceRunning:
|
||||
return "Running today."
|
||||
case content.ConfidenceInProgress:
|
||||
return "In progress — not finished."
|
||||
case content.ConfidenceAbandoned:
|
||||
return "Abandoned. Kept in the record deliberately."
|
||||
case content.ConfidenceHistoric:
|
||||
return "Historic — worked on, not maintained today."
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// LevelLabel names a depth level for the page kicker.
|
||||
func LevelLabel(level int) string {
|
||||
switch level {
|
||||
case 1:
|
||||
return "Level 1 · capability in one screen"
|
||||
case 2:
|
||||
return "Level 2 · what is actually run"
|
||||
case 3:
|
||||
return "Level 3 · artefact"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
| Service | Host | Port | State |
|
||||
|---|---|---|---|
|
||||
| Caddy | `.35` | 443 (+ LAN 8000) | existing — **Sam applies any change** |
|
||||
| **Go app** | `.13` | **8096** | new — verify free (`8095` = offsite.lab); published on `192.168.20.13:8096`, **never** `0.0.0.0` |
|
||||
| **Go app** | `.13` | **8096** | new — verify free (`8095` = offsite.lab) |
|
||||
| Qdrant REST | `.13` | 6333 | new, `127.0.0.1` only |
|
||||
| Qdrant gRPC | `.13` | 6334 | new, `127.0.0.1` only |
|
||||
| Prefect server | `.13` | existing | reuse — do not deploy a second |
|
||||
@@ -55,9 +55,7 @@ Ceiling: 20,000 chunks at 1024 dimensions. Beyond that, set
|
||||
`RESUME_QDRANT_VECTORS_ON_DISK=1`, reduce the dimension, or raise the cap — in
|
||||
that order. Detail: `docs/phase-4-notes.md` §7.
|
||||
|
||||
Every service sets `mem_limit` **and** `memswap_limit` to the same value so it cannot consume host swap. Bind services to `127.0.0.1`; Caddy on `.35` is the only public listener.
|
||||
|
||||
**One documented amendment (Go app, port 8096):** the app publishes on the LAN address `192.168.20.13:8096`, not `127.0.0.1`. Caddy runs on `.35` and proxies to `192.168.20.13:8096`, so a loopback-only publish is unreachable from the edge and the public URL returns 502. This is the same way `8095` (offsite), `8092` (photo-filter), `3909` (garage-ui) and `4200` (prefect) are published on `.13`. The port is LAN-only — never `0.0.0.0` — and Caddy on `.35` is still the only public listener, so `AGENTS.md` §4.2 holds in substance. Decision: Sam, 2026-10-26, interaction `54aeb1a2`. `deploy/deploy.sh` follows the same address (`RESUME_PUBLISH_IP`, default `192.168.20.13`) in its health gate and shape assertion.
|
||||
Every service sets `mem_limit` **and** `memswap_limit` to the same value so it cannot consume host swap. Bind to `127.0.0.1`; Caddy is the only public listener.
|
||||
|
||||
## Paths
|
||||
|
||||
|
||||
@@ -1,71 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerate internal/brand/icons_gen.go from Simple Icons.
|
||||
#
|
||||
# Simple Icons (https://simpleicons.org) is CC0 1.0: the icon paths are public
|
||||
# domain. The marks themselves remain the trademarks of their owners and are
|
||||
# used here only to identify the technology being described.
|
||||
#
|
||||
# This script runs at development time on a machine with network access. The
|
||||
# generated Go file IS committed, so `make build` never needs the network, never
|
||||
# needs Node and never needs node_modules. Re-run it only when the icon set
|
||||
# changes:
|
||||
#
|
||||
# tools/gen_brand_icons.sh
|
||||
#
|
||||
# Every icon is normalised to a 24x24 viewBox and inlined as path data, so the
|
||||
# page carries no image request and the mark takes the chip's text colour.
|
||||
set -euo pipefail
|
||||
|
||||
root="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
out="$root/internal/brand/icons_gen.go"
|
||||
cdn="${SIMPLE_ICONS_CDN:-https://cdn.simpleicons.org}"
|
||||
|
||||
# slug:name as published by Simple Icons. Keep this list to marks the content
|
||||
# actually uses; an unused icon is dead weight in the binary.
|
||||
slugs=(
|
||||
javascript php python homeassistant raspberrypi nixos
|
||||
qdrant prefect n8n mqtt langgraph langchain
|
||||
ffmpeg zigbee threedotjs react proxmox mysql postgresql
|
||||
caddy tailscale neovim htmx go docker borgbackup
|
||||
sqlite spotify rabbitmq openrouter obsidian niri gitea ghostty
|
||||
googlegemini deepseek cplusplus typescript zsh minimax espressif
|
||||
grafana wireguard nginx hashicorp vault linux zig huggingface ollama
|
||||
)
|
||||
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
|
||||
{
|
||||
echo "// Code generated by tools/gen_brand_icons.sh; DO NOT EDIT."
|
||||
echo "//"
|
||||
echo "// Path data from Simple Icons (CC0 1.0), retrieved from $cdn."
|
||||
echo "// Every mark is a 24x24 viewBox drawn in currentColor."
|
||||
echo
|
||||
echo "package brand"
|
||||
echo
|
||||
echo "// marks maps a Simple Icons slug to its path data."
|
||||
echo "var marks = map[string][]string{"
|
||||
} > "$out"
|
||||
|
||||
for slug in "${slugs[@]}"; do
|
||||
if ! curl -fsS -m 20 -o "$tmp/$slug.svg" "$cdn/$slug"; then
|
||||
echo "gen_brand_icons: could not fetch $slug" >&2
|
||||
exit 1
|
||||
fi
|
||||
# One or more <path d="..."> elements; monochrome, so the path data is all we keep.
|
||||
mapfile -t paths < <(grep -o 'd="[^"]*"' "$tmp/$slug.svg" | sed 's/^d="//; s/"$//')
|
||||
if [ "${#paths[@]}" -eq 0 ]; then
|
||||
echo "gen_brand_icons: $slug has no path data" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '\t"%s": {\n' "$slug" >> "$out"
|
||||
for p in "${paths[@]}"; do
|
||||
printf '\t\t`%s`,\n' "$p" >> "$out"
|
||||
done
|
||||
printf '\t},\n' >> "$out"
|
||||
done
|
||||
|
||||
echo "}" >> "$out"
|
||||
|
||||
gofmt -w "$out" 2>/dev/null || true
|
||||
echo "gen_brand_icons: wrote $out (${#slugs[@]} marks)"
|
||||
@@ -1,35 +0,0 @@
|
||||
# Image treatment
|
||||
|
||||
Implements `docs/image-brief.md`. Two steps, in this order.
|
||||
|
||||
## 1. Cut the subject out
|
||||
|
||||
Needs `rembg`. It is not installed on the host, and installing Python packages on NixOS
|
||||
into the system interpreter is a bad idea. Use a container:
|
||||
|
||||
```
|
||||
docker run --rm -v "$PWD/raw:/in" -v "$PWD/build/cut:/out" \
|
||||
-v "$PWD/tools/images/treat.py:/treat.py" \
|
||||
python:3.12-slim sh -c "pip install -q rembg[cpu] pillow onnxruntime && \
|
||||
python /treat.py cutout /in/*.jpg --out /out"
|
||||
```
|
||||
|
||||
Or cut out by hand in GIMP and skip to step 2. For fewer than ten images, hand-cutting is
|
||||
usually better — automatic segmentation struggles with reflective or thin edges.
|
||||
|
||||
## 2. Apply the treatment
|
||||
|
||||
```
|
||||
pip install pillow # or the same container, without rembg
|
||||
python3 tools/images/treat.py treat build/cut/*.png --size hero
|
||||
```
|
||||
|
||||
Output lands in `content/assets/images/` as WebP at quality 82, canvas `#0b0d10`, subject
|
||||
right of centre, left-to-right fade, rim glow from the alpha channel.
|
||||
|
||||
## Then
|
||||
|
||||
1. Add the page reference: ``
|
||||
2. File the original in the project's own repo under `docs/images/`, with a note in that
|
||||
folder's `README.md` saying where it came from.
|
||||
3. Deploy and check it on the live site at 1440×900 and on a phone.
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Apply the site image treatment described in docs/image-brief.md.
|
||||
|
||||
Two jobs, both idempotent and re-runnable:
|
||||
|
||||
cutout remove the background from a product photograph (needs rembg)
|
||||
treat compose onto the dark canvas with the fade and glow (needs Pillow)
|
||||
|
||||
Typical use:
|
||||
|
||||
python3 tools/images/treat.py cutout raw/*.jpg -> build/cut/*.png
|
||||
python3 tools/images/treat.py treat build/cut/*.png -> content/assets/images/*.webp
|
||||
|
||||
If rembg is not available, cut the subject out by hand in GIMP, save as PNG with
|
||||
transparency, and run only the `treat` step.
|
||||
"""
|
||||
import argparse, pathlib, sys
|
||||
|
||||
BG = (0x0b, 0x0d, 0x10) # --bg
|
||||
GLOW = (0x8e, 0xcb, 0xff) # --link
|
||||
SIZES = {"hero": (1600, 1000), "detail": (1200, 1200)}
|
||||
FADE_END = 0.55 # gradient reaches 0 opacity here
|
||||
GLOW_A = 0.32
|
||||
GLOW_BLUR = 11
|
||||
LIFT = 1.06 # shadow lift on the subject
|
||||
|
||||
|
||||
def cutout(paths, outdir):
|
||||
try:
|
||||
from rembg import remove
|
||||
except ImportError:
|
||||
sys.exit("rembg not installed. See tools/images/README.md, or cut out by hand and skip this step.")
|
||||
from PIL import Image
|
||||
outdir.mkdir(parents=True, exist_ok=True)
|
||||
for p in paths:
|
||||
src = Image.open(p).convert("RGBA")
|
||||
dst = remove(src)
|
||||
target = outdir / (pathlib.Path(p).stem + ".png")
|
||||
dst.save(target)
|
||||
print(f"cut {p} -> {target}")
|
||||
|
||||
|
||||
def treat(paths, outdir, size="hero"):
|
||||
from PIL import Image, ImageFilter, ImageEnhance
|
||||
W, H = SIZES[size]
|
||||
outdir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for p in paths:
|
||||
subject = Image.open(p).convert("RGBA")
|
||||
|
||||
# fit to 68% of canvas height, keep aspect
|
||||
scale = (H * 0.68) / subject.height
|
||||
subject = subject.resize((max(1, int(subject.width * scale)), int(H * 0.68)), Image.LANCZOS)
|
||||
|
||||
# shadow lift
|
||||
rgb = ImageEnhance.Brightness(subject.convert("RGB")).enhance(LIFT)
|
||||
subject = Image.merge("RGBA", (*rgb.split(), subject.split()[3]))
|
||||
|
||||
canvas = Image.new("RGB", (W, H), BG)
|
||||
|
||||
# place right of centre
|
||||
x = int(W * 0.52)
|
||||
y = (H - subject.height) // 2
|
||||
canvas.paste(subject, (x, y), subject)
|
||||
|
||||
# rim glow from the subject's own alpha
|
||||
alpha = subject.split()[3].filter(ImageFilter.GaussianBlur(GLOW_BLUR))
|
||||
glow = Image.new("RGB", subject.size, GLOW)
|
||||
canvas.paste(glow, (x, y), alpha.point(lambda a: int(a * GLOW_A / 255)))
|
||||
|
||||
# left-to-right fade back to the page colour
|
||||
fade = Image.new("L", (W, 1))
|
||||
end = int(W * FADE_END)
|
||||
for px in range(W):
|
||||
fade.putpixel((px, 0), 255 if px < end else max(0, int(255 * (1 - (px - end) / max(1, W - end)))))
|
||||
fade = fade.resize((W, H))
|
||||
canvas = Image.composite(Image.new("RGB", (W, H), BG), canvas, fade)
|
||||
|
||||
target = outdir / (pathlib.Path(p).stem + ".webp")
|
||||
canvas.save(target, "WEBP", quality=82, method=6)
|
||||
kb = target.stat().st_size / 1024
|
||||
print(f"treat {p} -> {target} ({kb:.0f} KB)")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("step", choices=["cutout", "treat"])
|
||||
ap.add_argument("paths", nargs="+", type=pathlib.Path)
|
||||
ap.add_argument("--out", type=pathlib.Path, default=None)
|
||||
ap.add_argument("--size", choices=list(SIZES), default="hero")
|
||||
a = ap.parse_args()
|
||||
out = a.out or pathlib.Path("build" if a.step == "cutout" else "content/assets/images")
|
||||
(cutout if a.step == "cutout" else treat)(a.paths, out) if a.step == "cutout" else treat(a.paths, out, a.size)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user