Files

Sam Rolfe d7d529eb0f Phase 3: parse stored Compass pages into structured items

Turn the raw pages stored by Phase 2 into rows in the SQLite items table
(AGENTS.md §11 Phase 3, FR-3/FR-4/FR-5/FR-12).

- internal/parse: dependency-free HTML reader plus extractors for the ExtJS
  learning-task grid (subject, title, category, kind, due date, status), the
  home news feed (author, posted time, title, body) and the calendar week grid
  (day column + grid offset give the time; label gives title/room/teacher).
- internal/parse: fail loudly. A challenge page, login page, changed layout or
  unknown due-date format is an explicit error, never an empty day.
- FR-4: structure is parsed deterministically; importance is classified by an
  OmniRoute model (default openrouter/google/gemini-3.5-flash-lite) with a
  deterministic fallback. The classifier used is printed and recorded.
- internal/llm: small OpenAI-compatible OmniRoute client (handles SSE replies).
- internal/store: migration 002 adds source_page, importance and external_ref
  with a unique (child_id, source, type, external_ref) index; UpsertItems,
  CountItemsByType, SampleItems and ChildByName.
- cmd/school parse: reads var/raw/<child>/<run>, upserts items, records the scan
  row (ok=0 plus the error on failure) and prints counts and samples.
- scripts/parse-compass.sh and make parse run it in a one-off container against
  the service's own volume without restarting the service.
- docs/phase-3-parse-notes.md records the approach, the FR-4 decision and the
  known gaps.

Evidence on the live database: 56 rows for Harry (20 task, 1 message, 35 event).
Re-running the same raw pages inserts 0 and updates 56.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

2026-10-15 17:27:25 +11:00

3.6 KiB

Raw Blame History

Phase 3 — parse Compass pages into structured items

Status: done. school parse turns the raw pages stored by Phase 2 into rows in the SQLite items table. See AUD-107.

What runs

# one child, from the newest parseable raw run
scripts/parse-compass.sh --child Harry --run latest --classify auto

# offline (no model call)
scripts/parse-compass.sh --child Harry --classify none

scripts/parse-compass.sh runs school parse in a one-off container that mounts the compass-teams-nhs_compass-teams-nhs-data volume read/write and var/raw read-only. It never restarts the running service.

The subcommand itself:

school parse --child Harry --run latest --raw-root /raw --classify auto

Where the input comes from

Raw pages under var/raw/<child>/<runId>/ (git-ignored, written by scripts/compass-fetch-patchright.mjs). The store reads every *.html page and the url from the matching *.meta.json. --run latest picks the newest run directory that holds at least one recognised page, so a failed login run (only login.html) is never chosen.

What becomes an item

Source page	Item type	Fields taken from
`learning-tasks` (ExtJS grid)	`task`	subject group header, task title, category, task kind, due date, submission status
`home` (news feed)	`message`	author, posted time, title, body
`calendar` (week grid)	`event`	day column + grid offset for the time, label for title/room/teacher
`child-profile` (React/MUI)	—	no server-rendered grid; used only when it carries the legacy grid

Each row stores child, source, type, subject, teacher/author, title, body, due-at, link, source page, importance, and an external reference (FR-5).

FR-4 decision: parse deterministically, classify with a model

The Compass markup is machine-generated, so the structure is read deterministically. Asking a model to transcribe a grid invites hallucinated titles and due dates, and cannot be verified. The model is used only for importance, where judgement is needed (FR-4):

--classify auto (default) uses an OmniRoute model when OMNIROUTE_API_KEY is set, otherwise the offline rule.
--classify omniroute forces the model.
--classify none uses the offline rule.

Model default: openrouter/google/gemini-3.5-flash-lite via OMNIROUTE_BASE_URL (http://127.0.0.1:20129/v1). Override with SCHOOL_LLM_MODEL or --model. The classifier name is printed and recorded.

Offline rule: a task whose due date is in the past or within seven days is high; everything else is normal. Calendar and message items never become high offline.

Fail loudly (FR-12)

A parse or fetch failure is an explicit error, and a failing run is written to the scans table with ok = 0 and the message. A challenge page ("Just a moment…"), a login page, or a changed layout makes the extractor return an error — the run never reports "nothing due". An unrecognised due-date string is also an error, never a silent null.

De-duplication

Every item carries an external_ref and there is a unique index on (child_id, source, type, external_ref). Re-parsing the same page updates the same row. Second run of the same raw pages: inserted 0, updated 56.

Known gaps

The learning-tasks grid is capped at the most recent 20 tasks (Compass.attributes.LearningTask_Default_Records_Count = 20). The parent landing page reports more overdue work than the grid contains, so a later fetch must open the overdue-filtered view. This blocks the dashboard's "overdue" column, not Phase 3.
Team homework is not parsed here; that is FR-15, a separate source.
No model is used to classify the Teams source yet.

3.6 KiB Raw Blame History