Files

Sam Rolfe 5d263987dd Phase 4a: fetch and parse the authoritative Compass learning tasks

The Learning Tasks grid is capped at 20 rows and, on the parent page, mixes
both children, so it cannot answer "what is overdue". The parent home page
reported 6 overdue tasks for Harry while the stored grid held 1.

Fetch the per-child task list (read-only) from
POST /Services/LearningTasks.svc/GetAllLearningTasksByUserId, parse it into
task rows, and refuse to store it when the overdue count disagrees with the
home page summary (FR-12).

- fetcher: store learning-tasks-all.json + meta (userId, status, task count)
- parse: new learning-tasks-all page, JSON parser, home-page reconciliation
- store: drop superseded grid-derived task rows before the upsert
- docs: docs/phase-4a-overdue-tasks.md; close the Phase 3 known gap

Verified with live runs: Harry 70 tasks / 6 overdue, Finn 42 tasks / 2 overdue,
both equal to the home page; a tampered home page stops the parse.

Note: the fetcher and store.go files also carry the per-child session work
that was already uncommitted in this shared workspace (AUD-108).

Co-Authored-By: Paperclip <noreply@paperclip.ing>

2026-10-15 17:39:11 +11:00

3.7 KiB

Raw Blame History

Phase 3 — parse Compass pages into structured items

Status: done. school parse turns the raw pages stored by Phase 2 into rows in the SQLite items table. See AUD-107.

What runs

# one child, from the newest parseable raw run
scripts/parse-compass.sh --child Harry --run latest --classify auto

# offline (no model call)
scripts/parse-compass.sh --child Harry --classify none

scripts/parse-compass.sh runs school parse in a one-off container that mounts the compass-teams-nhs_compass-teams-nhs-data volume read/write and var/raw read-only. It never restarts the running service.

The subcommand itself:

school parse --child Harry --run latest --raw-root /raw --classify auto

Where the input comes from

Raw pages under var/raw/<child>/<runId>/ (git-ignored, written by scripts/compass-fetch-patchright.mjs). The store reads every *.html page and the url from the matching *.meta.json. --run latest picks the newest run directory that holds at least one recognised page, so a failed login run (only login.html) is never chosen.

What becomes an item

Source page	Item type	Fields taken from
`learning-tasks` (ExtJS grid)	`task`	subject group header, task title, category, task kind, due date, submission status
`home` (news feed)	`message`	author, posted time, title, body
`calendar` (week grid)	`event`	day column + grid offset for the time, label for title/room/teacher
`child-profile` (React/MUI)	—	no server-rendered grid; used only when it carries the legacy grid

Each row stores child, source, type, subject, teacher/author, title, body, due-at, link, source page, importance, and an external reference (FR-5).

FR-4 decision: parse deterministically, classify with a model

The Compass markup is machine-generated, so the structure is read deterministically. Asking a model to transcribe a grid invites hallucinated titles and due dates, and cannot be verified. The model is used only for importance, where judgement is needed (FR-4):

--classify auto (default) uses an OmniRoute model when OMNIROUTE_API_KEY is set, otherwise the offline rule.
--classify omniroute forces the model.
--classify none uses the offline rule.

Model default: openrouter/google/gemini-3.5-flash-lite via OMNIROUTE_BASE_URL (http://127.0.0.1:20129/v1). Override with SCHOOL_LLM_MODEL or --model. The classifier name is printed and recorded.

Offline rule: a task whose due date is in the past or within seven days is high; everything else is normal. Calendar and message items never become high offline.

Fail loudly (FR-12)

A parse or fetch failure is an explicit error, and a failing run is written to the scans table with ok = 0 and the message. A challenge page ("Just a moment…"), a login page, or a changed layout makes the extractor return an error — the run never reports "nothing due". An unrecognised due-date string is also an error, never a silent null.

De-duplication

Every item carries an external_ref and there is a unique index on (child_id, source, type, external_ref). Re-parsing the same page updates the same row. Second run of the same raw pages: inserted 0, updated 56.

Known gaps

The learning-tasks grid is capped at the most recent 20 tasks (Compass.attributes.LearningTask_Default_Records_Count = 20). The parent landing page reports more overdue work than the grid contains. Closed by Phase 4a: learning-tasks-all is now the authoritative per-child list, and the parser fails loudly when it disagrees with the home page. See docs/phase-4a-overdue-tasks.md.
Team homework is not parsed here; that is FR-15, a separate source.
No model is used to classify the Teams source yet.

3.7 KiB Raw Blame History