Sam Rolfe
5d263987dd
The Learning Tasks grid is capped at 20 rows and, on the parent page, mixes both children, so it cannot answer "what is overdue". The parent home page reported 6 overdue tasks for Harry while the stored grid held 1. Fetch the per-child task list (read-only) from POST /Services/LearningTasks.svc/GetAllLearningTasksByUserId, parse it into task rows, and refuse to store it when the overdue count disagrees with the home page summary (FR-12). - fetcher: store learning-tasks-all.json + meta (userId, status, task count) - parse: new learning-tasks-all page, JSON parser, home-page reconciliation - store: drop superseded grid-derived task rows before the upsert - docs: docs/phase-4a-overdue-tasks.md; close the Phase 3 known gap Verified with live runs: Harry 70 tasks / 6 overdue, Finn 42 tasks / 2 overdue, both equal to the home page; a tampered home page stops the parse. Note: the fetcher and store.go files also carry the per-child session work that was already uncommitted in this shared workspace (AUD-108). Co-Authored-By: Paperclip <noreply@paperclip.ing>
90 lines
3.7 KiB
Markdown
90 lines
3.7 KiB
Markdown
# Phase 3 — parse Compass pages into structured items
|
|
|
|
Status: **done.** `school parse` turns the raw pages stored by Phase 2 into
|
|
rows in the SQLite `items` table. See AUD-107.
|
|
|
|
## What runs
|
|
|
|
```bash
|
|
# one child, from the newest parseable raw run
|
|
scripts/parse-compass.sh --child Harry --run latest --classify auto
|
|
|
|
# offline (no model call)
|
|
scripts/parse-compass.sh --child Harry --classify none
|
|
```
|
|
|
|
`scripts/parse-compass.sh` runs `school parse` in a one-off container that
|
|
mounts the `compass-teams-nhs_compass-teams-nhs-data` volume read/write and
|
|
`var/raw` read-only. It never restarts the running service.
|
|
|
|
The subcommand itself:
|
|
|
|
```bash
|
|
school parse --child Harry --run latest --raw-root /raw --classify auto
|
|
```
|
|
|
|
## Where the input comes from
|
|
|
|
Raw pages under `var/raw/<child>/<runId>/` (git-ignored, written by
|
|
`scripts/compass-fetch-patchright.mjs`). The store reads every `*.html` page and
|
|
the `url` from the matching `*.meta.json`. `--run latest` picks the newest run
|
|
directory that holds at least one recognised page, so a failed login run (only
|
|
`login.html`) is never chosen.
|
|
|
|
## What becomes an item
|
|
|
|
| Source page | Item type | Fields taken from |
|
|
|---|---|---|
|
|
| `learning-tasks` (ExtJS grid) | `task` | subject group header, task title, category, task kind, due date, submission status |
|
|
| `home` (news feed) | `message` | author, posted time, title, body |
|
|
| `calendar` (week grid) | `event` | day column + grid offset for the time, label for title/room/teacher |
|
|
| `child-profile` (React/MUI) | — | no server-rendered grid; used only when it carries the legacy grid |
|
|
|
|
Each row stores child, source, type, subject, teacher/author, title, body,
|
|
due-at, link, source page, importance, and an external reference (FR-5).
|
|
|
|
## FR-4 decision: parse deterministically, classify with a model
|
|
|
|
The Compass markup is machine-generated, so the **structure** is read
|
|
deterministically. Asking a model to transcribe a grid invites hallucinated
|
|
titles and due dates, and cannot be verified. The model is used only for
|
|
**importance**, where judgement is needed (FR-4):
|
|
|
|
- `--classify auto` (default) uses an OmniRoute model when `OMNIROUTE_API_KEY`
|
|
is set, otherwise the offline rule.
|
|
- `--classify omniroute` forces the model.
|
|
- `--classify none` uses the offline rule.
|
|
|
|
Model default: `openrouter/google/gemini-3.5-flash-lite` via
|
|
`OMNIROUTE_BASE_URL` (`http://127.0.0.1:20129/v1`). Override with
|
|
`SCHOOL_LLM_MODEL` or `--model`. The classifier name is printed and recorded.
|
|
|
|
Offline rule: a `task` whose due date is in the past or within seven days is
|
|
`high`; everything else is `normal`. Calendar and message items never become
|
|
`high` offline.
|
|
|
|
## Fail loudly (FR-12)
|
|
|
|
A parse or fetch failure is an explicit error, and a failing run is written to
|
|
the `scans` table with `ok = 0` and the message. A challenge page ("Just a
|
|
moment…"), a login page, or a changed layout makes the extractor return an
|
|
error — the run never reports "nothing due". An unrecognised due-date string is
|
|
also an error, never a silent null.
|
|
|
|
## De-duplication
|
|
|
|
Every item carries an `external_ref` and there is a unique index on
|
|
`(child_id, source, type, external_ref)`. Re-parsing the same page updates the
|
|
same row. Second run of the same raw pages: `inserted 0, updated 56`.
|
|
|
|
## Known gaps
|
|
|
|
- The learning-tasks grid is capped at the most recent 20 tasks
|
|
(`Compass.attributes.LearningTask_Default_Records_Count = 20`). The parent
|
|
landing page reports more overdue work than the grid contains. **Closed by
|
|
Phase 4a:** `learning-tasks-all` is now the authoritative per-child list, and
|
|
the parser fails loudly when it disagrees with the home page. See
|
|
`docs/phase-4a-overdue-tasks.md`.
|
|
- Team homework is not parsed here; that is FR-15, a separate source.
|
|
- No model is used to classify the Teams source yet.
|