Sam Rolfe
d7d529eb0f
Turn the raw pages stored by Phase 2 into rows in the SQLite items table (AGENTS.md §11 Phase 3, FR-3/FR-4/FR-5/FR-12). - internal/parse: dependency-free HTML reader plus extractors for the ExtJS learning-task grid (subject, title, category, kind, due date, status), the home news feed (author, posted time, title, body) and the calendar week grid (day column + grid offset give the time; label gives title/room/teacher). - internal/parse: fail loudly. A challenge page, login page, changed layout or unknown due-date format is an explicit error, never an empty day. - FR-4: structure is parsed deterministically; importance is classified by an OmniRoute model (default openrouter/google/gemini-3.5-flash-lite) with a deterministic fallback. The classifier used is printed and recorded. - internal/llm: small OpenAI-compatible OmniRoute client (handles SSE replies). - internal/store: migration 002 adds source_page, importance and external_ref with a unique (child_id, source, type, external_ref) index; UpsertItems, CountItemsByType, SampleItems and ChildByName. - cmd/school parse: reads var/raw/<child>/<run>, upserts items, records the scan row (ok=0 plus the error on failure) and prints counts and samples. - scripts/parse-compass.sh and make parse run it in a one-off container against the service's own volume without restarting the service. - docs/phase-3-parse-notes.md records the approach, the FR-4 decision and the known gaps. Evidence on the live database: 56 rows for Harry (20 task, 1 message, 35 event). Re-running the same raw pages inserts 0 and updates 56. Co-Authored-By: Paperclip <noreply@paperclip.ing>
89 lines
3.6 KiB
Markdown
89 lines
3.6 KiB
Markdown
# Phase 3 — parse Compass pages into structured items
|
|
|
|
Status: **done.** `school parse` turns the raw pages stored by Phase 2 into
|
|
rows in the SQLite `items` table. See AUD-107.
|
|
|
|
## What runs
|
|
|
|
```bash
|
|
# one child, from the newest parseable raw run
|
|
scripts/parse-compass.sh --child Harry --run latest --classify auto
|
|
|
|
# offline (no model call)
|
|
scripts/parse-compass.sh --child Harry --classify none
|
|
```
|
|
|
|
`scripts/parse-compass.sh` runs `school parse` in a one-off container that
|
|
mounts the `compass-teams-nhs_compass-teams-nhs-data` volume read/write and
|
|
`var/raw` read-only. It never restarts the running service.
|
|
|
|
The subcommand itself:
|
|
|
|
```bash
|
|
school parse --child Harry --run latest --raw-root /raw --classify auto
|
|
```
|
|
|
|
## Where the input comes from
|
|
|
|
Raw pages under `var/raw/<child>/<runId>/` (git-ignored, written by
|
|
`scripts/compass-fetch-patchright.mjs`). The store reads every `*.html` page and
|
|
the `url` from the matching `*.meta.json`. `--run latest` picks the newest run
|
|
directory that holds at least one recognised page, so a failed login run (only
|
|
`login.html`) is never chosen.
|
|
|
|
## What becomes an item
|
|
|
|
| Source page | Item type | Fields taken from |
|
|
|---|---|---|
|
|
| `learning-tasks` (ExtJS grid) | `task` | subject group header, task title, category, task kind, due date, submission status |
|
|
| `home` (news feed) | `message` | author, posted time, title, body |
|
|
| `calendar` (week grid) | `event` | day column + grid offset for the time, label for title/room/teacher |
|
|
| `child-profile` (React/MUI) | — | no server-rendered grid; used only when it carries the legacy grid |
|
|
|
|
Each row stores child, source, type, subject, teacher/author, title, body,
|
|
due-at, link, source page, importance, and an external reference (FR-5).
|
|
|
|
## FR-4 decision: parse deterministically, classify with a model
|
|
|
|
The Compass markup is machine-generated, so the **structure** is read
|
|
deterministically. Asking a model to transcribe a grid invites hallucinated
|
|
titles and due dates, and cannot be verified. The model is used only for
|
|
**importance**, where judgement is needed (FR-4):
|
|
|
|
- `--classify auto` (default) uses an OmniRoute model when `OMNIROUTE_API_KEY`
|
|
is set, otherwise the offline rule.
|
|
- `--classify omniroute` forces the model.
|
|
- `--classify none` uses the offline rule.
|
|
|
|
Model default: `openrouter/google/gemini-3.5-flash-lite` via
|
|
`OMNIROUTE_BASE_URL` (`http://127.0.0.1:20129/v1`). Override with
|
|
`SCHOOL_LLM_MODEL` or `--model`. The classifier name is printed and recorded.
|
|
|
|
Offline rule: a `task` whose due date is in the past or within seven days is
|
|
`high`; everything else is `normal`. Calendar and message items never become
|
|
`high` offline.
|
|
|
|
## Fail loudly (FR-12)
|
|
|
|
A parse or fetch failure is an explicit error, and a failing run is written to
|
|
the `scans` table with `ok = 0` and the message. A challenge page ("Just a
|
|
moment…"), a login page, or a changed layout makes the extractor return an
|
|
error — the run never reports "nothing due". An unrecognised due-date string is
|
|
also an error, never a silent null.
|
|
|
|
## De-duplication
|
|
|
|
Every item carries an `external_ref` and there is a unique index on
|
|
`(child_id, source, type, external_ref)`. Re-parsing the same page updates the
|
|
same row. Second run of the same raw pages: `inserted 0, updated 56`.
|
|
|
|
## Known gaps
|
|
|
|
- The learning-tasks grid is capped at the most recent 20 tasks
|
|
(`Compass.attributes.LearningTask_Default_Records_Count = 20`). The parent
|
|
landing page reports more overdue work than the grid contains, so a later
|
|
fetch must open the overdue-filtered view. This blocks the dashboard's
|
|
"overdue" column, not Phase 3.
|
|
- Team homework is not parsed here; that is FR-15, a separate source.
|
|
- No model is used to classify the Teams source yet.
|