Phase 3: parse stored Compass pages into structured items

Turn the raw pages stored by Phase 2 into rows in the SQLite items table
(AGENTS.md §11 Phase 3, FR-3/FR-4/FR-5/FR-12).

- internal/parse: dependency-free HTML reader plus extractors for the ExtJS
  learning-task grid (subject, title, category, kind, due date, status), the
  home news feed (author, posted time, title, body) and the calendar week grid
  (day column + grid offset give the time; label gives title/room/teacher).
- internal/parse: fail loudly. A challenge page, login page, changed layout or
  unknown due-date format is an explicit error, never an empty day.
- FR-4: structure is parsed deterministically; importance is classified by an
  OmniRoute model (default openrouter/google/gemini-3.5-flash-lite) with a
  deterministic fallback. The classifier used is printed and recorded.
- internal/llm: small OpenAI-compatible OmniRoute client (handles SSE replies).
- internal/store: migration 002 adds source_page, importance and external_ref
  with a unique (child_id, source, type, external_ref) index; UpsertItems,
  CountItemsByType, SampleItems and ChildByName.
- cmd/school parse: reads var/raw/<child>/<run>, upserts items, records the scan
  row (ok=0 plus the error on failure) and prints counts and samples.
- scripts/parse-compass.sh and make parse run it in a one-off container against
  the service's own volume without restarting the service.
- docs/phase-3-parse-notes.md records the approach, the FR-4 decision and the
  known gaps.

Evidence on the live database: 56 rows for Harry (20 task, 1 message, 35 event).
Re-running the same raw pages inserts 0 and updates 56.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
2026-10-15 17:27:25 +11:00
parent faed46b494
commit d7d529eb0f
19 changed files with 2419 additions and 18 deletions

View File

@@ -1,5 +1,10 @@
IMAGE ?= compass-teams-nhs/school:0.1.0-phase1
BUILD_VERSION ?= 0.1.0-phase1
IMAGE ?= compass-teams-nhs/school:0.2.0-phase3
BUILD_VERSION ?= 0.2.0-phase3
# Parse options (see scripts/parse-compass.sh).
CHILD ?= Harry
RUN ?= latest
CLASSIFY ?= auto
# Bounded docker helper for the Go toolchain. The container runs as the current
# user, so the repository and the module cache stay owned by you.
@@ -8,7 +13,7 @@ GO_RUN = timeout 600 docker run --rm --network host --user "$$(id -u):$$(id -g)"
-e HOME=/src/.gocache -e GOMODCACHE=/src/.gocache/mod -e GOCACHE=/src/.gocache/build \
-v "$$PWD":/src -w /src $(GO_IMAGE)
.PHONY: help test vet build up down restart logs ps deps fmt hash-password gen-secret verify-phase1
.PHONY: help test vet build up down restart logs ps deps fmt hash-password gen-secret verify-phase1 parse
help:
@echo "make test run go test ./... in golang:1.25-alpine"
@@ -21,6 +26,7 @@ help:
@echo "make deps go mod tidy + download (writes go.mod/go.sum)"
@echo "make hash-password generate a bcrypt hash (PASSWORD=...)"
@echo "make gen-secret print a fresh session secret"
@echo "make parse parse stored Compass pages into SQLite (Phase 3)"
@echo "make verify-phase1 prove /health, the login and the Caddy route"
test:
@@ -60,6 +66,10 @@ hash-password:
gen-secret:
docker run --rm --entrypoint /app/school $(IMAGE) gen-secret
# Phase 3: parse the stored raw pages for one child into the items table.
parse:
CHILD=$(CHILD) RUN=$(RUN) CLASSIFY=$(CLASSIFY) IMAGE=$(IMAGE) scripts/parse-compass.sh
# Phase 1 definition of done: /health 200, login page + session login, and the
# Caddy route on .35.
verify-phase1: