Phase 3: parse stored Compass pages into structured items
Turn the raw pages stored by Phase 2 into rows in the SQLite items table (AGENTS.md §11 Phase 3, FR-3/FR-4/FR-5/FR-12). - internal/parse: dependency-free HTML reader plus extractors for the ExtJS learning-task grid (subject, title, category, kind, due date, status), the home news feed (author, posted time, title, body) and the calendar week grid (day column + grid offset give the time; label gives title/room/teacher). - internal/parse: fail loudly. A challenge page, login page, changed layout or unknown due-date format is an explicit error, never an empty day. - FR-4: structure is parsed deterministically; importance is classified by an OmniRoute model (default openrouter/google/gemini-3.5-flash-lite) with a deterministic fallback. The classifier used is printed and recorded. - internal/llm: small OpenAI-compatible OmniRoute client (handles SSE replies). - internal/store: migration 002 adds source_page, importance and external_ref with a unique (child_id, source, type, external_ref) index; UpsertItems, CountItemsByType, SampleItems and ChildByName. - cmd/school parse: reads var/raw/<child>/<run>, upserts items, records the scan row (ok=0 plus the error on failure) and prints counts and samples. - scripts/parse-compass.sh and make parse run it in a one-off container against the service's own volume without restarting the service. - docs/phase-3-parse-notes.md records the approach, the FR-4 decision and the known gaps. Evidence on the live database: 56 rows for Harry (20 task, 1 message, 35 event). Re-running the same raw pages inserts 0 and updates 56. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
28
README.md
28
README.md
@@ -4,8 +4,17 @@ A school service for Harry and Finn: **Compass** for the daily schedule and
|
||||
notices, **Teams** for homework and assignments. It stores what is due per child,
|
||||
puts overdue work first, and tells the family before work is late.
|
||||
|
||||
> Build spec: [`AGENTS.md`](AGENTS.md). This repository is at **Phase 1** (the
|
||||
> skeleton). See §11 of the spec for the phase order.
|
||||
> Build spec: [`AGENTS.md`](AGENTS.md). This repository is at **Phase 3** (parsing
|
||||
> the stored Compass pages). See §11 of the spec for the phase order.
|
||||
|
||||
## Status — Phase 3
|
||||
|
||||
Phase 3 parses the raw pages stored by Phase 2 into the SQLite `items` table
|
||||
(tasks, messages, events). See [`docs/phase-3-parse-notes.md`](docs/phase-3-parse-notes.md).
|
||||
|
||||
```bash
|
||||
scripts/parse-compass.sh --child Harry --run latest --classify auto
|
||||
```
|
||||
|
||||
## Status — Phase 1
|
||||
|
||||
@@ -20,9 +29,9 @@ Built and deployed on `.13`:
|
||||
- Its own container and port **8792**, behind Caddy at
|
||||
`school.lab.audasmedia.com.au`.
|
||||
|
||||
Not built yet (later phases): the Compass login and fetch, parsing, both
|
||||
children's data, the dashboard data, the calendar write, notifications, and the
|
||||
Hub API. It does **not** scan the school in this phase.
|
||||
Not built yet (later phases): both children's data, the dashboard data, the
|
||||
calendar write, notifications, and the Hub API. The service itself does **not**
|
||||
scan the school; the daily scan and the Teams source come later.
|
||||
|
||||
### Fail loudly (FR-12)
|
||||
|
||||
@@ -36,12 +45,15 @@ empty day. The dashboard reads the `scans` table and shows one of:
|
||||
## Layout
|
||||
|
||||
```
|
||||
cmd/school/ service entry point
|
||||
cmd/school/ service entry point (run, parse, hash-password, gen-secret)
|
||||
internal/config/ environment configuration
|
||||
internal/store/ SQLite: schema, migrations, queries
|
||||
internal/parse/ HTML -> structured items (tasks, messages, events)
|
||||
internal/llm/ OmniRoute client (importance classification only)
|
||||
internal/auth/ session login (signed cookies, CSRF)
|
||||
internal/web/ HTTP handlers, templates, static assets
|
||||
deploy/ Caddy snippet, deployment notes, verify script
|
||||
scripts/ Compass fetchers and the parse wrapper
|
||||
environment.d/ env example (the real file is gitignored)
|
||||
```
|
||||
|
||||
@@ -56,8 +68,8 @@ docker compose up -d --build
|
||||
Generate secrets with the built image:
|
||||
|
||||
```bash
|
||||
docker run --rm --entrypoint /app/school compass-teams-nhs/school:0.1.0-phase1 gen-secret
|
||||
docker run --rm --entrypoint /app/school compass-teams-nhs/school:0.1.0-phase1 hash-password 'the-password'
|
||||
docker run --rm --entrypoint /app/school compass-teams-nhs/school:0.2.0-phase3 gen-secret
|
||||
docker run --rm --entrypoint /app/school compass-teams-nhs/school:0.2.0-phase3 hash-password 'the-password'
|
||||
```
|
||||
|
||||
Note: Docker Compose interpolates `$VAR` inside `env_file`. A bcrypt hash
|
||||
|
||||
Reference in New Issue
Block a user