Phase 3: parse stored Compass pages into structured items
Turn the raw pages stored by Phase 2 into rows in the SQLite items table (AGENTS.md §11 Phase 3, FR-3/FR-4/FR-5/FR-12). - internal/parse: dependency-free HTML reader plus extractors for the ExtJS learning-task grid (subject, title, category, kind, due date, status), the home news feed (author, posted time, title, body) and the calendar week grid (day column + grid offset give the time; label gives title/room/teacher). - internal/parse: fail loudly. A challenge page, login page, changed layout or unknown due-date format is an explicit error, never an empty day. - FR-4: structure is parsed deterministically; importance is classified by an OmniRoute model (default openrouter/google/gemini-3.5-flash-lite) with a deterministic fallback. The classifier used is printed and recorded. - internal/llm: small OpenAI-compatible OmniRoute client (handles SSE replies). - internal/store: migration 002 adds source_page, importance and external_ref with a unique (child_id, source, type, external_ref) index; UpsertItems, CountItemsByType, SampleItems and ChildByName. - cmd/school parse: reads var/raw/<child>/<run>, upserts items, records the scan row (ok=0 plus the error on failure) and prints counts and samples. - scripts/parse-compass.sh and make parse run it in a one-off container against the service's own volume without restarting the service. - docs/phase-3-parse-notes.md records the approach, the FR-4 decision and the known gaps. Evidence on the live database: 56 rows for Harry (20 task, 1 message, 35 event). Re-running the same raw pages inserts 0 and updates 56. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
16
Makefile
16
Makefile
@@ -1,5 +1,10 @@
|
||||
IMAGE ?= compass-teams-nhs/school:0.1.0-phase1
|
||||
BUILD_VERSION ?= 0.1.0-phase1
|
||||
IMAGE ?= compass-teams-nhs/school:0.2.0-phase3
|
||||
BUILD_VERSION ?= 0.2.0-phase3
|
||||
|
||||
# Parse options (see scripts/parse-compass.sh).
|
||||
CHILD ?= Harry
|
||||
RUN ?= latest
|
||||
CLASSIFY ?= auto
|
||||
|
||||
# Bounded docker helper for the Go toolchain. The container runs as the current
|
||||
# user, so the repository and the module cache stay owned by you.
|
||||
@@ -8,7 +13,7 @@ GO_RUN = timeout 600 docker run --rm --network host --user "$$(id -u):$$(id -g)"
|
||||
-e HOME=/src/.gocache -e GOMODCACHE=/src/.gocache/mod -e GOCACHE=/src/.gocache/build \
|
||||
-v "$$PWD":/src -w /src $(GO_IMAGE)
|
||||
|
||||
.PHONY: help test vet build up down restart logs ps deps fmt hash-password gen-secret verify-phase1
|
||||
.PHONY: help test vet build up down restart logs ps deps fmt hash-password gen-secret verify-phase1 parse
|
||||
|
||||
help:
|
||||
@echo "make test run go test ./... in golang:1.25-alpine"
|
||||
@@ -21,6 +26,7 @@ help:
|
||||
@echo "make deps go mod tidy + download (writes go.mod/go.sum)"
|
||||
@echo "make hash-password generate a bcrypt hash (PASSWORD=...)"
|
||||
@echo "make gen-secret print a fresh session secret"
|
||||
@echo "make parse parse stored Compass pages into SQLite (Phase 3)"
|
||||
@echo "make verify-phase1 prove /health, the login and the Caddy route"
|
||||
|
||||
test:
|
||||
@@ -60,6 +66,10 @@ hash-password:
|
||||
gen-secret:
|
||||
docker run --rm --entrypoint /app/school $(IMAGE) gen-secret
|
||||
|
||||
# Phase 3: parse the stored raw pages for one child into the items table.
|
||||
parse:
|
||||
CHILD=$(CHILD) RUN=$(RUN) CLASSIFY=$(CLASSIFY) IMAGE=$(IMAGE) scripts/parse-compass.sh
|
||||
|
||||
# Phase 1 definition of done: /health 200, login page + session login, and the
|
||||
# Caddy route on .35.
|
||||
verify-phase1:
|
||||
|
||||
Reference in New Issue
Block a user