Phase 3: parse stored Compass pages into structured items
Turn the raw pages stored by Phase 2 into rows in the SQLite items table (AGENTS.md §11 Phase 3, FR-3/FR-4/FR-5/FR-12). - internal/parse: dependency-free HTML reader plus extractors for the ExtJS learning-task grid (subject, title, category, kind, due date, status), the home news feed (author, posted time, title, body) and the calendar week grid (day column + grid offset give the time; label gives title/room/teacher). - internal/parse: fail loudly. A challenge page, login page, changed layout or unknown due-date format is an explicit error, never an empty day. - FR-4: structure is parsed deterministically; importance is classified by an OmniRoute model (default openrouter/google/gemini-3.5-flash-lite) with a deterministic fallback. The classifier used is printed and recorded. - internal/llm: small OpenAI-compatible OmniRoute client (handles SSE replies). - internal/store: migration 002 adds source_page, importance and external_ref with a unique (child_id, source, type, external_ref) index; UpsertItems, CountItemsByType, SampleItems and ChildByName. - cmd/school parse: reads var/raw/<child>/<run>, upserts items, records the scan row (ok=0 plus the error on failure) and prints counts and samples. - scripts/parse-compass.sh and make parse run it in a one-off container against the service's own volume without restarting the service. - docs/phase-3-parse-notes.md records the approach, the FR-4 decision and the known gaps. Evidence on the live database: 56 rows for Harry (20 task, 1 message, 35 event). Re-running the same raw pages inserts 0 and updates 56. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
13
internal/store/migrations/002_items_parse.sql
Normal file
13
internal/store/migrations/002_items_parse.sql
Normal file
@@ -0,0 +1,13 @@
|
||||
-- Migration 002 — Phase 3 parse columns (AGENTS.md §4 FR-4/FR-5, §11 Phase 3).
|
||||
--
|
||||
-- Every item records the page it came from (FR-5), a classification of
|
||||
-- importance (FR-4) and a stable external reference so re-parsing the same page
|
||||
-- updates a row instead of inserting a second one.
|
||||
|
||||
ALTER TABLE items ADD COLUMN source_page TEXT NOT NULL DEFAULT '';
|
||||
ALTER TABLE items ADD COLUMN importance TEXT NOT NULL DEFAULT 'normal';
|
||||
ALTER TABLE items ADD COLUMN external_ref TEXT NOT NULL DEFAULT '';
|
||||
|
||||
-- One row per real item: the same source item parsed twice maps to one row.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_items_dedupe
|
||||
ON items(child_id, source, type, external_ref);
|
||||
Reference in New Issue
Block a user