From d7d529eb0f94eabfce0b82df253446e131851f04 Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Thu, 15 Oct 2026 17:27:25 +1100 Subject: [PATCH] Phase 3: parse stored Compass pages into structured items MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turn the raw pages stored by Phase 2 into rows in the SQLite items table (AGENTS.md §11 Phase 3, FR-3/FR-4/FR-5/FR-12). - internal/parse: dependency-free HTML reader plus extractors for the ExtJS learning-task grid (subject, title, category, kind, due date, status), the home news feed (author, posted time, title, body) and the calendar week grid (day column + grid offset give the time; label gives title/room/teacher). - internal/parse: fail loudly. A challenge page, login page, changed layout or unknown due-date format is an explicit error, never an empty day. - FR-4: structure is parsed deterministically; importance is classified by an OmniRoute model (default openrouter/google/gemini-3.5-flash-lite) with a deterministic fallback. The classifier used is printed and recorded. - internal/llm: small OpenAI-compatible OmniRoute client (handles SSE replies). - internal/store: migration 002 adds source_page, importance and external_ref with a unique (child_id, source, type, external_ref) index; UpsertItems, CountItemsByType, SampleItems and ChildByName. - cmd/school parse: reads var/raw//, upserts items, records the scan row (ok=0 plus the error on failure) and prints counts and samples. - scripts/parse-compass.sh and make parse run it in a one-off container against the service's own volume without restarting the service. - docs/phase-3-parse-notes.md records the approach, the FR-4 decision and the known gaps. Evidence on the live database: 56 rows for Harry (20 task, 1 message, 35 event). Re-running the same raw pages inserts 0 and updates 56. Co-Authored-By: Paperclip --- Makefile | 16 +- README.md | 28 +- cmd/school/main.go | 3 + cmd/school/parse.go | 358 ++++++++++++++++++ docker-compose.yml | 4 +- docs/phase-3-parse-notes.md | 88 +++++ internal/config/config.go | 20 + internal/llm/omniroute.go | 152 ++++++++ internal/parse/calendar.go | 169 +++++++++ internal/parse/html.go | 323 ++++++++++++++++ internal/parse/importance.go | 204 ++++++++++ internal/parse/news.go | 102 +++++ internal/parse/parse.go | 163 ++++++++ internal/parse/parse_test.go | 305 +++++++++++++++ internal/parse/tasks.go | 263 +++++++++++++ internal/store/migrations/002_items_parse.sql | 13 + internal/store/store.go | 160 ++++++++ internal/web/server.go | 10 +- scripts/parse-compass.sh | 56 +++ 19 files changed, 2419 insertions(+), 18 deletions(-) create mode 100644 cmd/school/parse.go create mode 100644 docs/phase-3-parse-notes.md create mode 100644 internal/llm/omniroute.go create mode 100644 internal/parse/calendar.go create mode 100644 internal/parse/html.go create mode 100644 internal/parse/importance.go create mode 100644 internal/parse/news.go create mode 100644 internal/parse/parse.go create mode 100644 internal/parse/parse_test.go create mode 100644 internal/parse/tasks.go create mode 100644 internal/store/migrations/002_items_parse.sql create mode 100755 scripts/parse-compass.sh diff --git a/Makefile b/Makefile index f242650..2e0962b 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,10 @@ -IMAGE ?= compass-teams-nhs/school:0.1.0-phase1 -BUILD_VERSION ?= 0.1.0-phase1 +IMAGE ?= compass-teams-nhs/school:0.2.0-phase3 +BUILD_VERSION ?= 0.2.0-phase3 + +# Parse options (see scripts/parse-compass.sh). +CHILD ?= Harry +RUN ?= latest +CLASSIFY ?= auto # Bounded docker helper for the Go toolchain. The container runs as the current # user, so the repository and the module cache stay owned by you. @@ -8,7 +13,7 @@ GO_RUN = timeout 600 docker run --rm --network host --user "$$(id -u):$$(id -g)" -e HOME=/src/.gocache -e GOMODCACHE=/src/.gocache/mod -e GOCACHE=/src/.gocache/build \ -v "$$PWD":/src -w /src $(GO_IMAGE) -.PHONY: help test vet build up down restart logs ps deps fmt hash-password gen-secret verify-phase1 +.PHONY: help test vet build up down restart logs ps deps fmt hash-password gen-secret verify-phase1 parse help: @echo "make test run go test ./... in golang:1.25-alpine" @@ -21,6 +26,7 @@ help: @echo "make deps go mod tidy + download (writes go.mod/go.sum)" @echo "make hash-password generate a bcrypt hash (PASSWORD=...)" @echo "make gen-secret print a fresh session secret" + @echo "make parse parse stored Compass pages into SQLite (Phase 3)" @echo "make verify-phase1 prove /health, the login and the Caddy route" test: @@ -60,6 +66,10 @@ hash-password: gen-secret: docker run --rm --entrypoint /app/school $(IMAGE) gen-secret +# Phase 3: parse the stored raw pages for one child into the items table. +parse: + CHILD=$(CHILD) RUN=$(RUN) CLASSIFY=$(CLASSIFY) IMAGE=$(IMAGE) scripts/parse-compass.sh + # Phase 1 definition of done: /health 200, login page + session login, and the # Caddy route on .35. verify-phase1: diff --git a/README.md b/README.md index 0e91f02..3ff8f44 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,17 @@ A school service for Harry and Finn: **Compass** for the daily schedule and notices, **Teams** for homework and assignments. It stores what is due per child, puts overdue work first, and tells the family before work is late. -> Build spec: [`AGENTS.md`](AGENTS.md). This repository is at **Phase 1** (the -> skeleton). See §11 of the spec for the phase order. +> Build spec: [`AGENTS.md`](AGENTS.md). This repository is at **Phase 3** (parsing +> the stored Compass pages). See §11 of the spec for the phase order. + +## Status — Phase 3 + +Phase 3 parses the raw pages stored by Phase 2 into the SQLite `items` table +(tasks, messages, events). See [`docs/phase-3-parse-notes.md`](docs/phase-3-parse-notes.md). + +```bash +scripts/parse-compass.sh --child Harry --run latest --classify auto +``` ## Status — Phase 1 @@ -20,9 +29,9 @@ Built and deployed on `.13`: - Its own container and port **8792**, behind Caddy at `school.lab.audasmedia.com.au`. -Not built yet (later phases): the Compass login and fetch, parsing, both -children's data, the dashboard data, the calendar write, notifications, and the -Hub API. It does **not** scan the school in this phase. +Not built yet (later phases): both children's data, the dashboard data, the +calendar write, notifications, and the Hub API. The service itself does **not** +scan the school; the daily scan and the Teams source come later. ### Fail loudly (FR-12) @@ -36,12 +45,15 @@ empty day. The dashboard reads the `scans` table and shows one of: ## Layout ``` -cmd/school/ service entry point +cmd/school/ service entry point (run, parse, hash-password, gen-secret) internal/config/ environment configuration internal/store/ SQLite: schema, migrations, queries +internal/parse/ HTML -> structured items (tasks, messages, events) +internal/llm/ OmniRoute client (importance classification only) internal/auth/ session login (signed cookies, CSRF) internal/web/ HTTP handlers, templates, static assets deploy/ Caddy snippet, deployment notes, verify script +scripts/ Compass fetchers and the parse wrapper environment.d/ env example (the real file is gitignored) ``` @@ -56,8 +68,8 @@ docker compose up -d --build Generate secrets with the built image: ```bash -docker run --rm --entrypoint /app/school compass-teams-nhs/school:0.1.0-phase1 gen-secret -docker run --rm --entrypoint /app/school compass-teams-nhs/school:0.1.0-phase1 hash-password 'the-password' +docker run --rm --entrypoint /app/school compass-teams-nhs/school:0.2.0-phase3 gen-secret +docker run --rm --entrypoint /app/school compass-teams-nhs/school:0.2.0-phase3 hash-password 'the-password' ``` Note: Docker Compose interpolates `$VAR` inside `env_file`. A bcrypt hash diff --git a/cmd/school/main.go b/cmd/school/main.go index 348f16b..92ae2b9 100644 --- a/cmd/school/main.go +++ b/cmd/school/main.go @@ -41,6 +41,9 @@ func main() { case "gen-secret": genSecret() return + case "parse": + parseCommand(os.Args[2:]) + return case "version": fmt.Println(version) return diff --git a/cmd/school/parse.go b/cmd/school/parse.go new file mode 100644 index 0000000..4cec0a4 --- /dev/null +++ b/cmd/school/parse.go @@ -0,0 +1,358 @@ +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "time" + + "gitea.lab.audasmedia.com.au/sam/compass_teams_nhs/internal/config" + "gitea.lab.audasmedia.com.au/sam/compass_teams_nhs/internal/llm" + "gitea.lab.audasmedia.com.au/sam/compass_teams_nhs/internal/parse" + "gitea.lab.audasmedia.com.au/sam/compass_teams_nhs/internal/store" +) + +// parseFlags holds the `school parse` options. +type parseFlags struct { + child string + run string + rawRoot string + classify string + model string + asJSON bool +} + +// parseCommand is the `school parse` entry point. It reads the stored raw +// Compass pages for one child, turns them into items and writes them to SQLite +// (AGENTS.md §11 Phase 3). +func parseCommand(args []string) { + fs := flag.NewFlagSet("parse", flag.ContinueOnError) + f := parseFlags{} + fs.StringVar(&f.child, "child", "Harry", "child name (Harry or Finn)") + fs.StringVar(&f.run, "run", "latest", "raw run id under var/raw//, or 'latest'") + fs.StringVar(&f.rawRoot, "raw-root", "var/raw", "root directory of the stored raw pages") + fs.StringVar(&f.classify, "classify", "auto", "importance classifier: auto, none or omniroute") + fs.StringVar(&f.model, "model", "", "model id for importance (default SCHOOL_LLM_MODEL or OmniRoute default)") + fs.BoolVar(&f.asJSON, "json", false, "print the summary as JSON") + _ = fs.Parse(args) + + if err := runParse(f); err != nil { + fmt.Fprintln(os.Stderr, "school parse:", err) + os.Exit(1) + } +} + +// parseSummary is the machine-readable result of one parse run. +type parseSummary struct { + Child string `json:"child"` + RunDir string `json:"run_dir"` + Classifier string `json:"classifier"` + Model string `json:"model,omitempty"` + Pages []string `json:"pages"` + Total int `json:"total"` + Inserted int `json:"inserted"` + Updated int `json:"updated"` + Counts []store.TypeCount `json:"counts"` + Samples []itemSample `json:"samples"` + ScannedAt string `json:"scanned_at"` +} + +type itemSample struct { + Type string `json:"type"` + Subject string `json:"subject,omitempty"` + Title string `json:"title"` + DueAt string `json:"due_at,omitempty"` + SourcePage string `json:"source_page"` + Importance string `json:"importance"` + URL string `json:"url,omitempty"` +} + +func runParse(f parseFlags) error { + ctx := context.Background() + + cfg, err := config.LoadDB() + if err != nil { + return err + } + + st, err := store.Open(cfg.DBPath) + if err != nil { + return err + } + defer st.Close() + + if err := st.Migrate(ctx); err != nil { + return err + } + if err := st.SeedChildren(ctx, seedChildren...); err != nil { + return err + } + + child, err := st.ChildByName(ctx, f.child) + if err != nil { + return err + } + if child == nil { + return fmt.Errorf("unknown child %q", f.child) + } + + runDir, pages, err := loadRunPages(f.rawRoot, f.child, f.run) + if err != nil { + // FR-12: a failed load is an explicit failed scan, never an empty day. + _, _ = st.RecordScan(ctx, &child.ID, 0, 0, false, err.Error()) + return err + } + + items, err := parse.Parse(f.child, pages, cfg.Location) + if err != nil { + _, _ = st.RecordScan(ctx, &child.ID, len(pages), 0, false, err.Error()) + return err + } + + classifyCtx, cancelClassify := context.WithTimeout(ctx, 3*time.Minute) + classifier, model, err := classifyItems(classifyCtx, f, items, cfg.Location) + cancelClassify() + if err != nil { + _, _ = st.RecordScan(ctx, &child.ID, len(pages), len(items), false, err.Error()) + return err + } + + stored := make([]store.Item, 0, len(items)) + for _, it := range items { + stored = append(stored, store.Item{ + Source: it.Source, + Type: it.Type, + Subject: it.Subject, + Teacher: it.Teacher, + Title: it.Title, + Body: it.Body, + DueAt: it.DueAt, + URL: it.URL, + SourcePage: it.SourcePage, + Importance: it.Importance, + ExternalRef: it.ExternalRef, + }) + } + + inserted, updated, err := st.UpsertItems(ctx, child.ID, stored) + if err != nil { + _, _ = st.RecordScan(ctx, &child.ID, len(pages), len(items), false, err.Error()) + return err + } + + if _, err := st.RecordScan(ctx, &child.ID, len(pages), len(items), true, ""); err != nil { + return err + } + + counts, err := st.CountItemsByType(ctx) + if err != nil { + return err + } + samples, err := st.SampleItems(ctx, child.ID, 12) + if err != nil { + return err + } + + pageSlugs := make([]string, 0, len(pages)) + for _, p := range pages { + pageSlugs = append(pageSlugs, p.Slug) + } + sort.Strings(pageSlugs) + + summary := parseSummary{ + Child: f.child, + RunDir: runDir, + Classifier: classifier, + Model: model, + Pages: pageSlugs, + Total: len(items), + Inserted: inserted, + Updated: updated, + Counts: counts, + ScannedAt: time.Now().In(cfg.Location).Format(time.RFC3339), + } + for _, s := range samples { + summary.Samples = append(summary.Samples, itemSample{ + Type: s.Type, + Subject: s.Subject, + Title: s.Title, + DueAt: dueString(s.DueAt, cfg.Location), + SourcePage: s.SourcePage, + Importance: s.Importance, + URL: s.URL, + }) + } + + if f.asJSON { + enc := json.NewEncoder(os.Stdout) + enc.SetIndent("", " ") + return enc.Encode(summary) + } + printParseSummary(summary) + return nil +} + +func printParseSummary(s parseSummary) { + fmt.Printf("child %s\n", s.Child) + fmt.Printf("run %s\n", s.RunDir) + fmt.Printf("pages %s\n", strings.Join(s.Pages, ", ")) + fmt.Printf("classifier %s", s.Classifier) + if s.Model != "" { + fmt.Printf(" (%s)", s.Model) + } + fmt.Printf("\nitems %d (inserted %d, updated %d)\n", s.Total, s.Inserted, s.Updated) + for _, c := range s.Counts { + fmt.Printf(" %-6s %-8s %d\n", c.Child, c.Type, c.Count) + } + fmt.Println("samples:") + for _, x := range s.Samples { + due := x.DueAt + if due == "" { + due = "-" + } + fmt.Printf(" [%s] %-8s %-28s %-45s due=%s\n", x.Importance, x.Type, x.Subject, truncate(x.Title, 45), due) + } +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n-1] + "…" +} + +func dueString(t *time.Time, loc *time.Location) string { + if t == nil { + return "" + } + return t.In(loc).Format("2006-01-02 15:04") +} + +// classifyItems applies the configured importance classifier and returns its +// name and model. It records which classifier was used (AGENTS.md §4 FR-4). +func classifyItems(ctx context.Context, f parseFlags, items []parse.Item, loc *time.Location) (string, string, error) { + mode := strings.ToLower(strings.TrimSpace(f.classify)) + switch mode { + case "none", "": + parse.ClassifyDeterministic(items, time.Now()) + return parse.ClassifierNone, "", nil + case "auto", "omniroute", "llm": + key := strings.TrimSpace(os.Getenv("OMNIROUTE_API_KEY")) + if key == "" { + if mode == "auto" { + parse.ClassifyDeterministic(items, time.Now()) + return parse.ClassifierNone, "", nil + } + return "", "", fmt.Errorf("classify: OMNIROUTE_API_KEY is not set") + } + model := strings.TrimSpace(f.model) + if model == "" { + model = strings.TrimSpace(os.Getenv("SCHOOL_LLM_MODEL")) + } + if model == "" { + model = llm.DefaultModel + } + base := strings.TrimSpace(os.Getenv("OMNIROUTE_BASE_URL")) + if base == "" { + base = llm.DefaultBaseURL + } + client := llm.New(base, key, model) + if err := parse.ClassifyModel(ctx, client, items, model, time.Now().In(loc)); err != nil { + return "", "", err + } + return parse.ClassifierOmniRoute, model, nil + default: + return "", "", fmt.Errorf("unknown --classify %q (want auto, none or omniroute)", f.classify) + } +} + +// loadRunPages finds the raw run directory for a child and reads every +// .html page in it, with the URL from the matching .meta.json. +func loadRunPages(rawRoot, child, run string) (string, []parse.Page, error) { + childDir := filepath.Join(rawRoot, strings.ToLower(strings.TrimSpace(child))) + entries, err := os.ReadDir(childDir) + if err != nil { + return "", nil, fmt.Errorf("read raw pages for %s at %s: %w", child, childDir, err) + } + + var candidates []string + for _, e := range entries { + if e.IsDir() { + candidates = append(candidates, e.Name()) + } + } + sort.Sort(sort.Reverse(sort.StringSlice(candidates))) + + runDir := "" + if strings.TrimSpace(run) == "latest" { + for _, name := range candidates { + if runHasKnownPage(filepath.Join(childDir, name)) { + runDir = filepath.Join(childDir, name) + break + } + } + if runDir == "" { + return "", nil, fmt.Errorf("no parseable Compass run for %s under %s", child, childDir) + } + } else { + runDir = filepath.Join(childDir, run) + } + + files, err := os.ReadDir(runDir) + if err != nil { + return "", nil, fmt.Errorf("read run %s: %w", runDir, err) + } + + pages := make([]parse.Page, 0, len(files)) + for _, e := range files { + name := e.Name() + if e.IsDir() || !strings.HasSuffix(strings.ToLower(name), ".html") { + continue + } + slug := strings.TrimSuffix(name, filepath.Ext(name)) + raw, err := os.ReadFile(filepath.Join(runDir, name)) + if err != nil { + return "", nil, fmt.Errorf("read %s: %w", name, err) + } + pages = append(pages, parse.Page{ + Slug: slug, + URL: pageURL(runDir, slug), + HTML: raw, + }) + } + if len(pages) == 0 { + return "", nil, fmt.Errorf("run %s has no .html pages", runDir) + } + return runDir, pages, nil +} + +// runHasKnownPage reports whether a run directory holds at least one page the +// parser understands, so a failed login run is never chosen as "latest". +func runHasKnownPage(dir string) bool { + for _, slug := range []string{parse.PageHome, parse.PageLearningTasks, parse.PageCalendar, parse.PageChildProfile} { + if _, err := os.Stat(filepath.Join(dir, slug+".html")); err == nil { + return true + } + } + return false +} + +// pageURL reads the source URL recorded by the Phase 2 fetcher. +func pageURL(runDir, slug string) string { + raw, err := os.ReadFile(filepath.Join(runDir, slug+".meta.json")) + if err != nil { + return "" + } + var meta struct { + URL string `json:"url"` + } + if err := json.Unmarshal(raw, &meta); err != nil { + return "" + } + return strings.TrimSpace(meta.URL) +} diff --git a/docker-compose.yml b/docker-compose.yml index 878ce42..9a847cd 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -10,8 +10,8 @@ services: build: context: . args: - VERSION: "0.1.0-phase1" - image: compass-teams-nhs/school:0.1.0-phase1 + VERSION: "0.2.0-phase3" + image: compass-teams-nhs/school:0.2.0-phase3 container_name: compass-teams-nhs restart: unless-stopped env_file: diff --git a/docs/phase-3-parse-notes.md b/docs/phase-3-parse-notes.md new file mode 100644 index 0000000..f02b695 --- /dev/null +++ b/docs/phase-3-parse-notes.md @@ -0,0 +1,88 @@ +# Phase 3 — parse Compass pages into structured items + +Status: **done.** `school parse` turns the raw pages stored by Phase 2 into +rows in the SQLite `items` table. See AUD-107. + +## What runs + +```bash +# one child, from the newest parseable raw run +scripts/parse-compass.sh --child Harry --run latest --classify auto + +# offline (no model call) +scripts/parse-compass.sh --child Harry --classify none +``` + +`scripts/parse-compass.sh` runs `school parse` in a one-off container that +mounts the `compass-teams-nhs_compass-teams-nhs-data` volume read/write and +`var/raw` read-only. It never restarts the running service. + +The subcommand itself: + +```bash +school parse --child Harry --run latest --raw-root /raw --classify auto +``` + +## Where the input comes from + +Raw pages under `var/raw///` (git-ignored, written by +`scripts/compass-fetch-patchright.mjs`). The store reads every `*.html` page and +the `url` from the matching `*.meta.json`. `--run latest` picks the newest run +directory that holds at least one recognised page, so a failed login run (only +`login.html`) is never chosen. + +## What becomes an item + +| Source page | Item type | Fields taken from | +|---|---|---| +| `learning-tasks` (ExtJS grid) | `task` | subject group header, task title, category, task kind, due date, submission status | +| `home` (news feed) | `message` | author, posted time, title, body | +| `calendar` (week grid) | `event` | day column + grid offset for the time, label for title/room/teacher | +| `child-profile` (React/MUI) | — | no server-rendered grid; used only when it carries the legacy grid | + +Each row stores child, source, type, subject, teacher/author, title, body, +due-at, link, source page, importance, and an external reference (FR-5). + +## FR-4 decision: parse deterministically, classify with a model + +The Compass markup is machine-generated, so the **structure** is read +deterministically. Asking a model to transcribe a grid invites hallucinated +titles and due dates, and cannot be verified. The model is used only for +**importance**, where judgement is needed (FR-4): + +- `--classify auto` (default) uses an OmniRoute model when `OMNIROUTE_API_KEY` + is set, otherwise the offline rule. +- `--classify omniroute` forces the model. +- `--classify none` uses the offline rule. + +Model default: `openrouter/google/gemini-3.5-flash-lite` via +`OMNIROUTE_BASE_URL` (`http://127.0.0.1:20129/v1`). Override with +`SCHOOL_LLM_MODEL` or `--model`. The classifier name is printed and recorded. + +Offline rule: a `task` whose due date is in the past or within seven days is +`high`; everything else is `normal`. Calendar and message items never become +`high` offline. + +## Fail loudly (FR-12) + +A parse or fetch failure is an explicit error, and a failing run is written to +the `scans` table with `ok = 0` and the message. A challenge page ("Just a +moment…"), a login page, or a changed layout makes the extractor return an +error — the run never reports "nothing due". An unrecognised due-date string is +also an error, never a silent null. + +## De-duplication + +Every item carries an `external_ref` and there is a unique index on +`(child_id, source, type, external_ref)`. Re-parsing the same page updates the +same row. Second run of the same raw pages: `inserted 0, updated 56`. + +## Known gaps + +- The learning-tasks grid is capped at the most recent 20 tasks + (`Compass.attributes.LearningTask_Default_Records_Count = 20`). The parent + landing page reports more overdue work than the grid contains, so a later + fetch must open the overdue-filtered view. This blocks the dashboard's + "overdue" column, not Phase 3. +- Team homework is not parsed here; that is FR-15, a separate source. +- No model is used to classify the Teams source yet. diff --git a/internal/config/config.go b/internal/config/config.go index f39b6d3..b1016cc 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -94,6 +94,26 @@ func Load() (*Config, error) { return cfg, nil } +// DBConfig is the minimum a database-only command needs. The parse subcommand +// opens the same SQLite file as the service without requiring the service's +// session secret or login. +type DBConfig struct { + DBPath string + Location *time.Location +} + +// LoadDB reads the database path and the family timezone only. +func LoadDB() (*DBConfig, error) { + loc, err := time.LoadLocation(env("SCHOOL_TIMEZONE", DefaultTimezone)) + if err != nil { + return nil, fmt.Errorf("SCHOOL_TIMEZONE: %w", err) + } + return &DBConfig{ + DBPath: env("SCHOOL_DB_PATH", "/data/school.db"), + Location: loc, + }, nil +} + func env(key, fallback string) string { if v := strings.TrimSpace(os.Getenv(key)); v != "" { return v diff --git a/internal/llm/omniroute.go b/internal/llm/omniroute.go new file mode 100644 index 0000000..2c621c7 --- /dev/null +++ b/internal/llm/omniroute.go @@ -0,0 +1,152 @@ +// Package llm is a small OpenAI-compatible client for OmniRoute. All model +// traffic goes through OmniRoute (AGENTS.md §10 rule 4); this package holds no +// model list and no credentials of its own. +package llm + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "strings" + "time" +) + +// DefaultBaseURL is OmniRoute on the host that runs this service (.13). +const DefaultBaseURL = "http://127.0.0.1:20129/v1" + +// DefaultModel is a cheap, reliable text model used for importance labels. It is +// overridable with SCHOOL_LLM_MODEL. +const DefaultModel = "openrouter/google/gemini-3.5-flash-lite" + +// Client calls one OmniRoute chat-completions endpoint. +type Client struct { + BaseURL string + APIKey string + Model string + HTTP *http.Client +} + +// New builds a client. Empty baseURL and model fall back to the defaults. +func New(baseURL, apiKey, model string) *Client { + if strings.TrimSpace(baseURL) == "" { + baseURL = DefaultBaseURL + } + if strings.TrimSpace(model) == "" { + model = DefaultModel + } + return &Client{ + BaseURL: strings.TrimRight(baseURL, "/"), + APIKey: apiKey, + Model: model, + HTTP: &http.Client{Timeout: 90 * time.Second}, + } +} + +// Complete sends one system+user prompt and returns the assistant text. It +// accepts both a normal JSON reply and a streaming SSE reply, because some +// OmniRoute providers stream regardless of the stream flag. +func (c *Client) Complete(ctx context.Context, system, prompt string) (string, error) { + if c.APIKey == "" { + return "", fmt.Errorf("omniroute: no API key") + } + body, err := json.Marshal(map[string]any{ + "model": c.Model, + "messages": []map[string]string{ + {"role": "system", "content": system}, + {"role": "user", "content": prompt}, + }, + "temperature": 0, + "max_tokens": 4096, + "stream": false, + }) + if err != nil { + return "", fmt.Errorf("omniroute: encode request: %w", err) + } + + req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.BaseURL+"/chat/completions", bytes.NewReader(body)) + if err != nil { + return "", fmt.Errorf("omniroute: request: %w", err) + } + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Authorization", "Bearer "+c.APIKey) + + resp, err := c.HTTP.Do(req) + if err != nil { + return "", fmt.Errorf("omniroute: %w", err) + } + defer resp.Body.Close() + raw, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20)) + if err != nil { + return "", fmt.Errorf("omniroute: read response: %w", err) + } + if resp.StatusCode != http.StatusOK { + return "", fmt.Errorf("omniroute: HTTP %d: %s", resp.StatusCode, truncate(string(raw), 400)) + } + + if strings.Contains(resp.Header.Get("Content-Type"), "text/event-stream") || bytes.HasPrefix(bytes.TrimSpace(raw), []byte("data:")) { + return parseSSE(raw) + } + + var parsed struct { + Choices []struct { + Message struct { + Content string `json:"content"` + } `json:"message"` + } `json:"choices"` + } + if err := json.Unmarshal(raw, &parsed); err != nil { + return "", fmt.Errorf("omniroute: decode response: %w", err) + } + if len(parsed.Choices) == 0 { + return "", fmt.Errorf("omniroute: no choices in response: %s", truncate(string(raw), 400)) + } + return parsed.Choices[0].Message.Content, nil +} + +// parseSSE concatenates the delta content of an SSE chat-completions stream. +func parseSSE(raw []byte) (string, error) { + var b strings.Builder + scanner := bufio.NewScanner(bytes.NewReader(raw)) + scanner.Buffer(make([]byte, 0, 64*1024), 1<<20) + for scanner.Scan() { + line := strings.TrimSpace(scanner.Text()) + if !strings.HasPrefix(line, "data:") { + continue + } + payload := strings.TrimSpace(strings.TrimPrefix(line, "data:")) + if payload == "" || payload == "[DONE]" { + continue + } + var chunk struct { + Choices []struct { + Delta struct { + Content string `json:"content"` + } `json:"delta"` + } `json:"choices"` + } + if err := json.Unmarshal([]byte(payload), &chunk); err != nil { + continue + } + for _, ch := range chunk.Choices { + b.WriteString(ch.Delta.Content) + } + } + if err := scanner.Err(); err != nil { + return "", fmt.Errorf("omniroute: read stream: %w", err) + } + if b.Len() == 0 { + return "", fmt.Errorf("omniroute: empty streamed response") + } + return b.String(), nil +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] + "..." +} diff --git a/internal/parse/calendar.go b/internal/parse/calendar.go new file mode 100644 index 0000000..0b49d93 --- /dev/null +++ b/internal/parse/calendar.go @@ -0,0 +1,169 @@ +package parse + +import ( + "fmt" + "regexp" + "strconv" + "strings" + "time" +) + +// calendarPxPerHour is the Compass week-grid scale: the day gutter is 1200px +// tall for 24 hours, so 50px is one hour. The event's top offset is therefore +// the authoritative hour; the printed "6:00" prefix is 12-hour and ambiguous. +const calendarPxPerHour = 50.0 + +var ( + // calendarDayCol matches the day column id, for example + // calendar-manager-week-bd-day-col-20261016. + calendarDayCol = regexp.MustCompile(`calendar-manager-week-bd-day-col-(\d{8})`) + // calendarTop matches the top offset in an event's inline style. + calendarTop = regexp.MustCompile(`top:\s*([0-9.]+)px`) + // clockPrefix matches the leading "8:55: " of an event label. + clockPrefix = regexp.MustCompile(`^(\d{1,2}):(\d{2}):\s*(.*)$`) + // periodLabel matches the period part of a lesson label: "5", "C" or + // "RLE GROUP D". + periodLabel = regexp.MustCompile(`^(\d{1,2}|[A-Z]|RLE GROUP [A-Z])$`) +) + +// parseCalendar parses the week calendar into event items. It fails loudly when +// the page has no calendar grid (FR-12). +func parseCalendar(child string, page Page, loc *time.Location) ([]Item, error) { + items, found, err := extractCalendar(child, page, loc) + if err != nil { + return nil, err + } + if !found { + return nil, fmt.Errorf("parse: %s: no calendar grid found in %s (layout changed or fetch failed)", child, page.Slug) + } + return items, nil +} + +// extractCalendar returns the calendar events in one page. found reports whether +// the page carried a week grid at all. +func extractCalendar(child string, page Page, loc *time.Location) (items []Item, found bool, err error) { + root := parseHTML(page.HTML) + + dayCols := findAll(root, func(n *node) bool { + return n.Tag == "div" && calendarDayCol.MatchString(n.Attrs["id"]) + }) + events := findAll(root, func(n *node) bool { + return n.Tag == "div" && n.hasClassPart("calendar-manager-week-bd-evt-") + }) + found = len(dayCols) > 0 || len(events) > 0 + if !found { + return nil, false, nil + } + + for _, ev := range events { + it, ok, err := eventItem(child, ev, page, loc) + if err != nil { + return nil, false, err + } + if ok { + items = append(items, it) + } + } + return items, true, nil +} + +func eventItem(child string, ev *node, page Page, loc *time.Location) (Item, bool, error) { + labelSpan := find(ev, func(n *node) bool { return n.hasClass("ext-evt-bd") }) + if labelSpan == nil { + return Item{}, false, nil // not an event body + } + label := collapse(labelSpan.Text()) + if label == "" { + return Item{}, false, nil + } + + day := nearest(ev, func(n *node) bool { + return n.Tag == "div" && calendarDayCol.MatchString(n.Attrs["id"]) + }) + if day == nil { + return Item{}, false, fmt.Errorf("parse: %s: event %q has no day column", page.Slug, label) + } + m := calendarDayCol.FindStringSubmatch(day.Attrs["id"]) + date, err := time.ParseInLocation("20060102", m[1], loc) + if err != nil { + return Item{}, false, fmt.Errorf("parse: %s: event %q: bad day column %q", page.Slug, label, day.Attrs["id"]) + } + + topMatch := calendarTop.FindStringSubmatch(ev.Attrs["style"]) + if topMatch == nil { + return Item{}, false, fmt.Errorf("parse: %s: event %q has no top offset", page.Slug, label) + } + topPx, err := strconv.ParseFloat(topMatch[1], 64) + if err != nil { + return Item{}, false, fmt.Errorf("parse: %s: event %q: bad top offset %q", page.Slug, label, topMatch[1]) + } + + // The hour comes from the grid offset; the minutes come from the label. + hour := int(topPx / calendarPxPerHour) + minute := 0 + title := label + if cm := clockPrefix.FindStringSubmatch(label); cm != nil { + if v, err := strconv.Atoi(cm[2]); err == nil { + minute = v + } + title = strings.TrimSpace(cm[3]) + } + if hour < 0 || hour > 23 || minute < 0 || minute > 59 { + return Item{}, false, fmt.Errorf("parse: %s: event %q: bad time %02d:%02d", page.Slug, label, hour, minute) + } + at := time.Date(date.Year(), date.Month(), date.Day(), hour, minute, 0, 0, loc) + + subject, teacher, body := splitEvent(label, title) + + url := page.URL + if url == "" { + url = compassHost + "/Organise/Calendar/" + } + + return Item{ + Child: child, + Source: SourceCompass, + Type: TypeEvent, + Subject: subject, + Teacher: teacher, + Title: title, + Body: body, + DueAt: &at, + URL: url, + SourcePage: page.Slug, + ExternalRef: dedupeKey(at.Format(time.RFC3339), title), + Importance: importanceNormal, + }, true, nil +} + +// splitEvent breaks a calendar label into subject, teacher and body. A lesson +// label is "5 - 11HLEGB5 - D8 - LHT"; a school event is +// "End of Year Senior Music Concert - HALL (...) - KSH, CFR, ...". +func splitEvent(label, title string) (subject, teacher, body string) { + parts := splitDash(title) + if len(parts) >= 2 && periodLabel.MatchString(parts[0]) { + for _, p := range parts[1:] { + if subject == "" && classCode.MatchString(p) { + subject = p + } + } + if !classCode.MatchString(parts[0]) { + teacher = parts[len(parts)-1] + } + } + body = label + if teacher != "" { + body = label + "\nTeacher(s): " + teacher + } + return subject, teacher, body +} + +// splitDash splits on " - " and trims each part. +func splitDash(s string) []string { + raw := strings.Split(s, " - ") + out := make([]string, 0, len(raw)) + for _, p := range raw { + out = append(out, strings.TrimSpace(p)) + } + return out +} diff --git a/internal/parse/html.go b/internal/parse/html.go new file mode 100644 index 0000000..88aa19a --- /dev/null +++ b/internal/parse/html.go @@ -0,0 +1,323 @@ +// Package parse turns stored Compass pages into structured items (AGENTS.md +// §4 FR-3/FR-4/FR-5, §11 Phase 3). It is a deterministic, dependency-free HTML +// reader: the Compass pages are machine-generated, so parsing the markup is more +// reliable than asking a model to transcribe a table. +package parse + +import ( + "bytes" + "html" + "strings" +) + +// voidElements never have children and are therefore never pushed on the stack. +var voidElements = map[string]bool{ + "area": true, "base": true, "br": true, "col": true, "embed": true, + "hr": true, "img": true, "input": true, "link": true, "meta": true, + "param": true, "source": true, "track": true, "wbr": true, +} + +// rawTextElements have text content that must not be tokenised as markup. +var rawTextElements = map[string]bool{ + "script": true, "style": true, "textarea": true, +} + +// node is one element in the parsed document. Text content is stored as +// "#text" children so that text and elements keep their document order; use +// Text for the concatenated descendant text. +type node struct { + Tag string + Attrs map[string]string + Kids []*node + Parent *node + text string +} + +// Class returns the element's class attribute. +func (n *node) Class() string { return n.Attrs["class"] } + +// hasClass reports whether the element carries the exact class. +func (n *node) hasClass(cls string) bool { + for _, f := range strings.Fields(n.Class()) { + if f == cls { + return true + } + } + return false +} + +// hasClassPart reports whether any class token contains part. +func (n *node) hasClassPart(part string) bool { + for _, f := range strings.Fields(n.Class()) { + if strings.Contains(f, part) { + return true + } + } + return false +} + +// hasAttr reports whether the attribute is present with a non-empty value. +func (n *node) hasAttr(key string) bool { return strings.TrimSpace(n.Attrs[key]) != "" } + +// Text returns the concatenated text of the element and all descendants, +// whitespace-collapsed. +func (n *node) Text() string { + var b strings.Builder + n.writeText(&b) + return collapse(b.String()) +} + +func (n *node) writeText(b *strings.Builder) { + b.WriteString(n.text) + for _, k := range n.Kids { + k.writeText(b) + } +} +func collapse(s string) string { return strings.Join(strings.Fields(s), " ") } + +// textUntil returns the concatenated text of the node's descendants up to and +// including the first matching direct-or-descendant element, exclusive of that +// element's subtree. +func (n *node) textUntil(pred func(*node) bool) string { + var b strings.Builder + var w func(x *node) bool + w = func(x *node) bool { + if pred(x) { + return false + } + b.WriteString(x.text) + for _, k := range x.Kids { + if !w(k) { + return false + } + } + return true + } + w(n) + return collapse(b.String()) +} + +// parseHTML tokenises and nests the byte slice. It never fails: malformed +// markup, stray tags and unclosed elements are tolerated, because ExtJS emits +// group headers whose is not closed before the next table starts. +func parseHTML(data []byte) *node { + root := &node{Tag: "#root", Attrs: map[string]string{}} + stack := []*node{root} + n := len(data) + i := 0 + + top := func() *node { return stack[len(stack)-1] } + + for i < n { + if data[i] != '<' { + j := bytes.IndexByte(data[i:], '<') + if j < 0 { + j = n - i + } + appendText(top(), string(data[i:i+j])) + i += j + continue + } + + switch { + case bytes.HasPrefix(data[i:], []byte("")) + if end < 0 { + i = n + continue + } + i += end + 3 + case bytes.HasPrefix(data[i:], []byte("') + if end < 0 { + i = n + continue + } + i += end + 1 + case bytes.HasPrefix(data[i:], []byte("') + if gt < 0 { + i = n + continue + } + i = k + gt + 1 + stack = closeTag(stack, name) + default: + k := i + 1 + start := k + for k < n && isNameChar(data[k]) { + k++ + } + name := strings.ToLower(string(data[start:k])) + if name == "" { // a bare '<' + appendText(top(), "<") + i++ + continue + } + attrs, selfClose, next := parseAttrs(data, k) + i = next + + child := &node{Tag: name, Attrs: attrs, Parent: top()} + top().Kids = append(top().Kids, child) + + if selfClose || voidElements[name] { + continue + } + stack = append(stack, child) + + if rawTextElements[name] { + // Skip the content; scripts and styles are not parsed as markup. + if idx := indexFold(data[i:], "= 0 { + i += idx + } else { + i = n + } + } + } + } + return root +} + +// parseAttrs reads tag attributes starting at p (just past the tag name). It +// returns the attributes, whether the tag closed with "/>", and the index just +// past the closing '>'. +func parseAttrs(data []byte, p int) (map[string]string, bool, int) { + attrs := map[string]string{} + selfClose := false + n := len(data) + for p < n { + for p < n && isSpace(data[p]) { + p++ + } + if p >= n { + break + } + if data[p] == '>' { + p++ + break + } + if data[p] == '/' { + selfClose = true + p++ + continue + } + start := p + for p < n && !isSpace(data[p]) && data[p] != '=' && data[p] != '>' && data[p] != '/' { + p++ + } + name := strings.ToLower(string(data[start:p])) + q := p + for q < n && isSpace(data[q]) { + q++ + } + var value string + if q < n && data[q] == '=' { + q++ + for q < n && isSpace(data[q]) { + q++ + } + if q < n && (data[q] == '"' || data[q] == '\'') { + quote := data[q] + q++ + v := q + for q < n && data[q] != quote { + q++ + } + value = string(data[v:q]) + if q < n { + q++ + } + } else { + v := q + for q < n && !isSpace(data[q]) && data[q] != '>' { + q++ + } + value = string(data[v:q]) + } + p = q + } + if name != "" { + attrs[name] = html.UnescapeString(value) + } + } + return attrs, selfClose, p +} + +// appendText appends an entity-decoded text node to the parent, keeping its +// position in document order. +func appendText(n *node, s string) { + if s == "" { + return + } + n.Kids = append(n.Kids, &node{Tag: "#text", text: html.UnescapeString(s), Parent: n}) +} + +// closeTag pops elements up to and including the nearest matching tag and +// returns the remaining stack. +func closeTag(stack []*node, name string) []*node { + for i := len(stack) - 1; i > 0; i-- { + if stack[i].Tag == name { + return stack[:i] + } + } + return stack +} + +func isNameChar(c byte) bool { + return c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || + c >= '0' && c <= '9' || c == '-' || c == '_' || c == ':' || c == '.' +} + +func isSpace(c byte) bool { return c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\f' } + +// indexFold finds needle in haystack, ignoring ASCII case. +func indexFold(haystack []byte, needle string) int { + if len(needle) == 0 { + return 0 + } + lower := bytes.ToLower(haystack) + return bytes.Index(lower, []byte(strings.ToLower(needle))) +} + +// walk visits n and its descendants in document order. When fn returns false the +// node's children are skipped; siblings are still visited. To abort a whole +// traversal, the predicate must track its own stop flag. +func walk(n *node, fn func(*node) bool) { + if !fn(n) { + return + } + for _, k := range n.Kids { + walk(k, fn) + } +} + +// find returns the first descendant (including n) matching pred. +func find(n *node, pred func(*node) bool) *node { + if pred(n) { + return n + } + for _, k := range n.Kids { + if got := find(k, pred); got != nil { + return got + } + } + return nil +} + +// findAll returns every descendant (including n) matching pred, in document order. +func findAll(n *node, pred func(*node) bool) []*node { + var out []*node + walk(n, func(x *node) bool { + if pred(x) { + out = append(out, x) + } + return true + }) + return out +} diff --git a/internal/parse/importance.go b/internal/parse/importance.go new file mode 100644 index 0000000..256329b --- /dev/null +++ b/internal/parse/importance.go @@ -0,0 +1,204 @@ +package parse + +import ( + "context" + "encoding/json" + "fmt" + "sort" + "strings" + "time" +) + +// Importance labels stored in items.importance (FR-4). +const ( + importanceHigh = "high" + importanceNormal = "normal" + importanceLow = "low" +) + +// ImportanceClassifiers names the classifier used for a run, so the choice is +// recorded rather than implied. +const ( + ClassifierNone = "deterministic" + ClassifierOmniRoute = "omniroute" + ClassifierJEV = "jev" + classifyBatchSize = 40 + classifyMaxResponseChar = 16 * 1024 +) + +// Completer is one text-completion call. internal/llm supplies the OmniRoute +// client; tests supply a stub. +type Completer interface { + Complete(ctx context.Context, system, prompt string) (string, error) +} + +// ClassifyDeterministic is the offline importance rule. Overdue work and work +// due within the next week is high; everything else is normal. It is the +// fallback when no model is configured, and it is deliberately conservative: it +// never marks work "done". +func ClassifyDeterministic(items []Item, now time.Time) { + week := now.AddDate(0, 0, 7) + for i := range items { + items[i].Importance = deterministicImportance(items[i], week) + } +} + +func deterministicImportance(it Item, week time.Time) string { + // Only work carries a deadline that makes it important. A calendar entry's + // due_at is its start time, and a lesson that starts this week is not + // overdue work, so events and messages stay normal under the offline rule. + if it.Type == TypeTask { + if it.DueAt != nil && it.DueAt.Before(week) { + return importanceHigh + } + if strings.Contains(strings.ToLower(it.Body), "overdue") { + return importanceHigh + } + } + return importanceNormal +} + +// ClassifyModel asks a model to classify importance for every item (FR-4). It +// fails, loudly, on a model error or an unparseable answer; it never silently +// leaves a wrong label. Items are updated in place. now lets the model judge +// whether a due date is overdue. +func ClassifyModel(ctx context.Context, c Completer, items []Item, model string, now time.Time) error { + if c == nil { + return fmt.Errorf("classify: no model client configured") + } + const system = "You classify school items for a parent's morning brief. " + + "Reply with JSON only. No prose, no code fences." + + for start := 0; start < len(items); start += classifyBatchSize { + end := start + classifyBatchSize + if end > len(items) { + end = len(items) + } + batch := items[start:end] + + prompt := classifyPrompt(batch, model, now) + raw, err := c.Complete(ctx, system, prompt) + if err != nil { + return fmt.Errorf("classify: items %d-%d: %w", start, end-1, err) + } + labels, err := parseLabels(raw) + if err != nil { + return fmt.Errorf("classify: items %d-%d: %w", start, end-1, err) + } + for i, label := range labels { + if i >= len(batch) { + return fmt.Errorf("classify: items %d-%d: model returned %d labels for %d items", start, end-1, len(labels), len(batch)) + } + switch label { + case importanceHigh, importanceNormal, importanceLow: + batch[i].Importance = label + default: + return fmt.Errorf("classify: item %d: invalid importance %q", start+i, label) + } + } + if len(labels) != len(batch) { + return fmt.Errorf("classify: items %d-%d: model returned %d labels for %d items", start, end-1, len(labels), len(batch)) + } + } + return nil +} + +// classifyPrompt builds the user message for one batch. +func classifyPrompt(batch []Item, model string, now time.Time) string { + type row struct { + I int `json:"i"` + Child string `json:"child"` + Type string `json:"type"` + Subject string `json:"subject,omitempty"` + Title string `json:"title"` + Due string `json:"due,omitempty"` + Detail string `json:"detail,omitempty"` + } + rows := make([]row, 0, len(batch)) + for i, it := range batch { + due := "" + if it.DueAt != nil { + due = it.DueAt.Format(time.RFC3339) + } + rows = append(rows, row{ + I: i, + Child: it.Child, + Type: it.Type, + Subject: it.Subject, + Title: it.Title, + Due: due, + Detail: detail(it.Body), + }) + } + payload, _ := json.Marshal(rows) + var b strings.Builder + fmt.Fprintf(&b, "Model: %s\n", model) + fmt.Fprintf(&b, "Now: %s\n", now.Format(time.RFC3339)) + b.WriteString("Label every item with importance high, normal or low.\n") + b.WriteString("high: overdue work, work due today or in the next 48 hours, or a school event that affects the family.\n") + b.WriteString("low: routine recurring lessons, past submitted work, or notices with no action.\n") + b.WriteString("normal: anything else.\n") + b.WriteString("A task with a due date before Now and a submission status of Pending or Overdue is high.\n") + b.WriteString("Return a JSON array with one object per item, in order, exactly: " + + `{"i":,"importance":"high|normal|low"}` + "\n\nItems:\n") + b.Write(payload) + return b.String() +} + +// detail flattens an item body into one line for the model. +func detail(s string) string { + return collapse(strings.ReplaceAll(s, "\n", "; ")) +} + +// parseLabels extracts the importance labels from a model reply. It tolerates +// code fences and surrounding prose, and returns the labels ordered by index. +func parseLabels(raw string) ([]string, error) { + raw = strings.TrimSpace(raw) + if len(raw) > classifyMaxResponseChar { + return nil, fmt.Errorf("model reply too large (%d bytes)", len(raw)) + } + raw = strings.TrimPrefix(raw, "```json") + raw = strings.TrimPrefix(raw, "```") + raw = strings.TrimSuffix(raw, "```") + raw = strings.TrimSpace(raw) + + if i := strings.IndexByte(raw, '['); i >= 0 { + if j := strings.LastIndexByte(raw, ']'); j > i { + raw = raw[i : j+1] + } + } + + var entries []struct { + I *int `json:"i"` + Importance string `json:"importance"` + } + if err := json.Unmarshal([]byte(raw), &entries); err != nil { + return nil, fmt.Errorf("model reply is not JSON: %w", err) + } + if len(entries) == 0 { + return nil, fmt.Errorf("model reply has no items") + } + + type indexed struct { + i int + label string + } + got := make([]indexed, 0, len(entries)) + for pos, e := range entries { + idx := pos + if e.I != nil { + idx = *e.I + } + got = append(got, indexed{i: idx, label: strings.ToLower(strings.TrimSpace(e.Importance))}) + } + sort.Slice(got, func(a, b int) bool { return got[a].i < got[b].i }) + + out := make([]string, len(got)) + for k, g := range got { + if g.i < 0 || g.i >= len(got) { + return nil, fmt.Errorf("model returned out-of-range index %d", g.i) + } + out[k] = g.label + } + return out, nil +} diff --git a/internal/parse/news.go b/internal/parse/news.go new file mode 100644 index 0000000..1e1c148 --- /dev/null +++ b/internal/parse/news.go @@ -0,0 +1,102 @@ +package parse + +import ( + "fmt" + "strings" + "time" +) + +// newsTimeLayout is the aria-label format on the feed timestamp, for example +// "14/10/2026 09:00 AM". +const newsTimeLayout = "02/01/2006 03:04 PM" + +// parseNews parses the home-page news feed into message items. It fails loudly +// when the page carries neither the feed toolbar nor a post, so a login or +// challenge page is never mistaken for "no news" (FR-12). +func parseNews(child string, page Page, loc *time.Location) ([]Item, error) { + items, found, err := extractNews(child, page, loc) + if err != nil { + return nil, err + } + if !found { + return nil, fmt.Errorf("parse: %s: no news feed or posts found in %s (layout changed or fetch failed)", child, page.Slug) + } + return items, nil +} + +// extractNews returns news posts from the home page. found reports whether the +// feed was present at all. +func extractNews(child string, page Page, loc *time.Location) (items []Item, found bool, err error) { + root := parseHTML(page.HTML) + + toolbar := find(root, func(n *node) bool { + return strings.EqualFold(n.Attrs["data-testid"], "feed-toolbar") + }) + cards := findAll(root, func(n *node) bool { return n.hasClass("MuiCardContent-root") }) + found = toolbar != nil || len(cards) > 0 + if !found { + return nil, false, nil + } + + for _, card := range cards { + titleNode := find(card, func(n *node) bool { return n.hasClassPart("postTitle") }) + if titleNode == nil { + continue // not a news post + } + title := collapse(titleNode.Text()) + if title == "" { + continue + } + + var body string + if b := find(card, func(n *node) bool { return n.hasClassPart("froala-viewer") }); b != nil { + body = b.Text() + } + + var author string + if a := find(card, func(n *node) bool { return n.hasClassPart("MuiTypography-dialogMdStrong") }); a != nil { + author = collapse(a.Text()) + } + + var posted *time.Time + if t := find(card, func(n *node) bool { + return n.hasClassPart("MuiTypography-dialogSmStrong") && n.hasAttr("aria-label") + }); t != nil { + raw := strings.TrimSpace(t.Attrs["aria-label"]) + parsed, perr := time.ParseInLocation(newsTimeLayout, raw, loc) + if perr != nil { + return nil, false, fmt.Errorf("parse: %s: news %q: unrecognised posted time %q", page.Slug, title, raw) + } + posted = &parsed + } + + pageURL := page.URL + if pageURL == "" { + pageURL = compassHost + "/" + } + ref := dedupeKey(author, title, relativeOrISO(posted)) + + items = append(items, Item{ + Child: child, + Source: SourceCompass, + Type: TypeMessage, + Teacher: author, // messages carry the author, not a teacher + Title: title, + Body: body, + DueAt: posted, + URL: pageURL, + SourcePage: page.Slug, + ExternalRef: ref, + Importance: importanceNormal, + }) + } + return items, true, nil +} + +// relativeOrISO renders a posted time for the dedupe key. +func relativeOrISO(t *time.Time) string { + if t == nil { + return "" + } + return t.Format(time.RFC3339) +} diff --git a/internal/parse/parse.go b/internal/parse/parse.go new file mode 100644 index 0000000..91c8a62 --- /dev/null +++ b/internal/parse/parse.go @@ -0,0 +1,163 @@ +package parse + +import ( + "fmt" + "sort" + "strings" + "time" +) + +// Source and item types match the items table CHECK constraints (AGENTS.md §7). +const ( + SourceCompass = "compass" + + TypeMessage = "message" + TypeTask = "task" + TypeEvent = "event" +) + +// Recognised page slugs, matching the .html files stored by the Phase 2 +// fetchers under var/raw///. +const ( + PageHome = "home" + PageChildProfile = "child-profile" + PageLearningTasks = "learning-tasks" + PageCalendar = "calendar" +) + +// Item is one message, learning task or calendar event read from Compass. +// Every field maps to a column of the items table (FR-5). +type Item struct { + Child string `json:"child"` + Source string `json:"source"` + Type string `json:"type"` + Subject string `json:"subject,omitempty"` + Teacher string `json:"teacher,omitempty"` + Title string `json:"title"` + Body string `json:"body,omitempty"` + DueAt *time.Time `json:"due_at,omitempty"` + URL string `json:"url,omitempty"` + SourcePage string `json:"source_page"` + ExternalRef string `json:"external_ref"` + Importance string `json:"importance"` +} + +// Page is one stored raw page: its slug, the URL it was fetched from, and the +// HTML bytes. +type Page struct { + Slug string + URL string + HTML []byte +} + +// Parse turns the stored pages for one child into structured items. It fails, +// loudly and explicitly, when a page it recognises cannot be parsed, so a broken +// scan can never be shown as "nothing due" (FR-12). +// +// Pages with an unknown slug are ignored. When no recognised page yields any +// structure at all, Parse returns an error. +func Parse(child string, pages []Page, loc *time.Location) ([]Item, error) { + if strings.TrimSpace(child) == "" { + return nil, fmt.Errorf("parse: child is empty") + } + if loc == nil { + loc = time.UTC + } + + var items []Item + recognised := 0 + for _, p := range pagesOf(pages) { + switch p.Slug { + case PageLearningTasks: + got, err := parseLearningTasks(child, p, loc) + if err != nil { + return nil, err + } + items = append(items, got...) + recognised++ + case PageChildProfile: + // The React profile page has no server-rendered grid; when it does + // carry the legacy grid, use it. + got, found, err := extractLearningTasks(child, p, loc) + if err != nil { + return nil, err + } + if found { + items = append(items, got...) + } + recognised++ + case PageHome: + got, err := parseNews(child, p, loc) + if err != nil { + return nil, err + } + items = append(items, got...) + recognised++ + case PageCalendar: + got, err := parseCalendar(child, p, loc) + if err != nil { + return nil, err + } + items = append(items, got...) + recognised++ + } + } + + if recognised == 0 { + return nil, fmt.Errorf("parse: no recognised Compass page in %d stored page(s)", len(pages)) + } + + // A task row is the same task wherever it appears (learning tasks and child + // profile pages both carry the grid). Keep the first, drop the repeats. + seen := map[string]bool{} + out := items[:0] + for _, it := range items { + key := it.Type + "\x1f" + it.ExternalRef + if seen[key] { + continue + } + seen[key] = true + if it.Importance == "" { + it.Importance = importanceNormal + } + out = append(out, it) + } + + sort.SliceStable(out, func(i, j int) bool { + if out[i].Type != out[j].Type { + return out[i].Type < out[j].Type + } + return out[i].ExternalRef < out[j].ExternalRef + }) + return out, nil +} + +// pagesOf returns the pages with duplicates removed, keeping the last copy of a +// slug (the fetcher overwrites a slug file on a re-fetch, so the last is +// authoritative). +func pagesOf(pages []Page) []Page { + bySlug := map[string]Page{} + var order []string + for _, p := range pages { + if _, ok := bySlug[p.Slug]; !ok { + order = append(order, p.Slug) + } + bySlug[p.Slug] = p + } + out := make([]Page, 0, len(order)) + for _, slug := range order { + out = append(out, bySlug[slug]) + } + return out +} + +// dedupeKey joins the identity fields of an item into a stable external +// reference. Re-parsing the same page must update the same row, not insert a +// second one. +func dedupeKey(parts ...string) string { + clean := make([]string, 0, len(parts)) + for _, p := range parts { + clean = append(clean, collapse(p)) + } + return strings.Join(clean, "|") +} diff --git a/internal/parse/parse_test.go b/internal/parse/parse_test.go new file mode 100644 index 0000000..5507bc2 --- /dev/null +++ b/internal/parse/parse_test.go @@ -0,0 +1,305 @@ +package parse + +import ( + "context" + "encoding/json" + "strings" + "testing" + "time" + + // Embed the zone database so the tests can use Australia/Melbourne. + _ "time/tzdata" +) + +// melbourne is the family timezone used in the tests. +func melbourne(t *testing.T) *time.Location { + t.Helper() + loc, err := time.LoadLocation("Australia/Melbourne") + if err != nil { + t.Fatalf("load location: %v", err) + } + return loc +} + +// tasksFixture mirrors the malformed ExtJS grid: the group header is not +// closed before the group's inner table, exactly as Compass emits it. +const tasksFixture = ` + + +
+
+ Test Child - 11EENG 11 ENGLISH +
+ + + + + + + + + + + + + +
+ Assessment + Essay draft
+ Class Task | Report: Yes +
+ Due date: 19/10/26 04:00 PM
+ Submitted: Pending +
-
+ Homework + Chapter 4 questions
+ Subject-wide Task +
+ Due date: 09/10/26 04:00 PM
+ Submitted: Overdue +
-
+
+` + +func TestParseLearningTasks(t *testing.T) { + loc := melbourne(t) + items, err := parseLearningTasks("Harry", Page{ + Slug: PageLearningTasks, + URL: "https://nhs-vic.compass.education/Records/UserNew.aspx#learningTasks", + HTML: []byte(tasksFixture), + }, loc) + if err != nil { + t.Fatalf("parseLearningTasks: %v", err) + } + if len(items) != 2 { + t.Fatalf("got %d items, want 2", len(items)) + } + + first := items[0] + if first.Type != TypeTask { + t.Errorf("type = %q, want %q", first.Type, TypeTask) + } + if first.Subject != "11EENG 11 ENGLISH" { + t.Errorf("subject = %q", first.Subject) + } + if first.Title != "Essay draft" { + t.Errorf("title = %q", first.Title) + } + if !strings.Contains(first.Body, "Category: Assessment") || !strings.Contains(first.Body, "Task: Class Task") { + t.Errorf("body = %q", first.Body) + } + if first.DueAt == nil || first.DueAt.Format("2006-01-02 15:04") != "2026-10-19 16:00" { + t.Errorf("due = %v", first.DueAt) + } + if !strings.Contains(first.URL, "userId=999") || !strings.HasSuffix(first.URL, "#learningTasks") { + t.Errorf("url = %q", first.URL) + } + if first.SourcePage != PageLearningTasks { + t.Errorf("source page = %q", first.SourcePage) + } + if first.ExternalRef == "" { + t.Error("empty external ref") + } + + second := items[1] + if second.DueAt == nil || second.DueAt.Format("2006-01-02") != "2026-10-09" { + t.Errorf("second due = %v", second.DueAt) + } + if second.ExternalRef == first.ExternalRef { + t.Error("two tasks share an external ref") + } + + // Re-parsing must produce the same references, so the store de-duplicates. + again, err := parseLearningTasks("Harry", Page{Slug: PageLearningTasks, HTML: []byte(tasksFixture)}, loc) + if err != nil { + t.Fatalf("re-parse: %v", err) + } + if again[0].ExternalRef != first.ExternalRef { + t.Errorf("external ref not stable: %q vs %q", again[0].ExternalRef, first.ExternalRef) + } +} + +const newsFixture = ` +

News

+
+ T Smith + 1 day ago + Health Information +

Dear families, there is a notice.

+
+` + +func TestParseNews(t *testing.T) { + loc := melbourne(t) + items, err := parseNews("Harry", Page{Slug: PageHome, URL: "https://nhs-vic.compass.education/", HTML: []byte(newsFixture)}, loc) + if err != nil { + t.Fatalf("parseNews: %v", err) + } + if len(items) != 1 { + t.Fatalf("got %d items, want 1", len(items)) + } + it := items[0] + if it.Type != TypeMessage { + t.Errorf("type = %q", it.Type) + } + if it.Title != "Health Information" { + t.Errorf("title = %q", it.Title) + } + if it.Teacher != "T Smith" { + t.Errorf("author = %q", it.Teacher) + } + if !strings.Contains(it.Body, "Dear families") { + t.Errorf("body = %q", it.Body) + } + if it.DueAt == nil || it.DueAt.Format("2006-01-02 15:04") != "2026-10-14 09:00" { + t.Errorf("posted = %v", it.DueAt) + } +} + +const calendarFixture = ` +
+
+
+ 6:00: End of Year Concert - HALL - KSH +
+
+ 2:06: 5 - 11HLEGB5 - D8 - LHT +
+
+
+` + +func TestParseCalendar(t *testing.T) { + loc := melbourne(t) + items, err := parseCalendar("Harry", Page{Slug: PageCalendar, URL: "https://nhs-vic.compass.education/Organise/Calendar/", HTML: []byte(calendarFixture)}, loc) + if err != nil { + t.Fatalf("parseCalendar: %v", err) + } + if len(items) != 2 { + t.Fatalf("got %d items, want 2", len(items)) + } + if items[0].Type != TypeEvent { + t.Errorf("type = %q", items[0].Type) + } + if got := items[0].DueAt.Format("2006-01-02 15:04"); got != "2026-10-15 18:00" { + t.Errorf("event 1 time = %s", got) + } + if items[0].Title != "End of Year Concert - HALL - KSH" { + t.Errorf("event 1 title = %q", items[0].Title) + } + if items[1].Subject != "11HLEGB5" || items[1].Teacher != "LHT" { + t.Errorf("event 2 subject/teacher = %q/%q", items[1].Subject, items[1].Teacher) + } + if got := items[1].DueAt.Format("15:04"); got != "14:06" { + t.Errorf("event 2 time = %q", got) + } +} + +// TestParseFailsLoudly proves FR-12: a challenge or login page must not be read +// as "nothing due". +func TestParseFailsLoudly(t *testing.T) { + challenge := []byte(`Just a moment...

Just a moment...

`) + + if _, err := parseNews("Harry", Page{Slug: PageHome, HTML: challenge}, time.UTC); err == nil { + t.Error("parseNews accepted a challenge page") + } + if _, err := parseLearningTasks("Harry", Page{Slug: PageLearningTasks, HTML: challenge}, time.UTC); err == nil { + t.Error("parseLearningTasks accepted a page with no grid") + } + if _, err := parseCalendar("Harry", Page{Slug: PageCalendar, HTML: challenge}, time.UTC); err == nil { + t.Error("parseCalendar accepted a page with no grid") + } + if _, err := Parse("Harry", []Page{{Slug: "unknown", HTML: challenge}}, time.UTC); err == nil { + t.Error("Parse accepted a page set with no recognised page") + } +} + +func TestParseCompassDue(t *testing.T) { + loc := melbourne(t) + cases := []struct { + in string + want string + }{ + {"19/10/26 04:00 PM", "2026-10-19 16:00"}, + {"09/10/26 04:00 PM", "2026-10-09 16:00"}, + {"No due date", ""}, + {"", ""}, + } + for _, tc := range cases { + got, err := parseCompassDue(tc.in, loc) + if err != nil { + t.Errorf("parseCompassDue(%q): %v", tc.in, err) + continue + } + if tc.want == "" { + if got != nil { + t.Errorf("parseCompassDue(%q) = %v, want nil", tc.in, got) + } + continue + } + if got == nil || got.Format("2006-01-02 15:04") != tc.want { + t.Errorf("parseCompassDue(%q) = %v, want %s", tc.in, got, tc.want) + } + } + if _, err := parseCompassDue("tomorrow", loc); err == nil { + t.Error("parseCompassDue accepted an unknown format") + } +} + +// stubCompleter returns a fixed reply. +type stubCompleter struct { + reply string + err error + calls int +} + +func (s *stubCompleter) Complete(context.Context, string, string) (string, error) { + s.calls++ + if s.err != nil { + return "", s.err + } + return s.reply, nil +} + +func TestClassifyModel(t *testing.T) { + items := []Item{ + {Type: TypeTask, Title: "Essay", DueAt: ptr(time.Date(2026, 10, 9, 16, 0, 0, 0, time.UTC))}, + {Type: TypeMessage, Title: "Notice"}, + } + reply, _ := json.Marshal([]map[string]any{ + {"i": 0, "importance": "high"}, + {"i": 1, "importance": "low"}, + }) + c := &stubCompleter{reply: "```json\n" + string(reply) + "\n```"} + if err := ClassifyModel(context.Background(), c, items, "test-model", time.Date(2026, 10, 15, 9, 0, 0, 0, time.UTC)); err != nil { + t.Fatalf("ClassifyModel: %v", err) + } + if items[0].Importance != importanceHigh || items[1].Importance != importanceLow { + t.Errorf("labels = %q/%q", items[0].Importance, items[1].Importance) + } + + bad := &stubCompleter{reply: `[{"i":0,"importance":"urgent"}]`} + if err := ClassifyModel(context.Background(), bad, items[:1], "test-model", time.Now()); err == nil { + t.Error("ClassifyModel accepted an invalid label") + } +} + +func TestClassifyDeterministic(t *testing.T) { + now := time.Date(2026, 10, 15, 9, 0, 0, 0, time.UTC) + overdue := now.Add(-24 * time.Hour) + nextWeek := now.Add(48 * time.Hour) + items := []Item{ + {Type: TypeTask, Title: "overdue", DueAt: &overdue}, + {Type: TypeTask, Title: "soon", DueAt: &nextWeek}, + {Type: TypeMessage, Title: "plain"}, + } + ClassifyDeterministic(items, now) + if items[0].Importance != importanceHigh || items[1].Importance != importanceHigh { + t.Errorf("overdue/soon = %q/%q", items[0].Importance, items[1].Importance) + } + if items[2].Importance != importanceNormal { + t.Errorf("plain = %q", items[2].Importance) + } +} + +func ptr(t time.Time) *time.Time { return &t } diff --git a/internal/parse/tasks.go b/internal/parse/tasks.go new file mode 100644 index 0000000..ba9cfdd --- /dev/null +++ b/internal/parse/tasks.go @@ -0,0 +1,263 @@ +package parse + +import ( + "fmt" + "regexp" + "strings" + "time" +) + +// compassDueLayouts are the due-date formats Compass writes. The grid prints +// "19/10/26 04:00 PM"; the fallback accepts single-digit fields. +var compassDueLayouts = []string{"02/01/06 03:04 PM", "2/1/06 3:04 PM"} + +// noDueDate is the literal the grid prints when a task has no due date. +const noDueDate = "No due date" + +// taskFieldLabels are the learning-task datafield titles that carry a value. +var taskFieldLabels = map[string]bool{ + "Due date:": true, + "Submitted:": true, + "Report:": true, +} + +// parseLearningTasks parses the legacy ExtJS learning-task grid into task items. +// It fails loudly when the grid is missing entirely, because that means the page +// changed or the fetch returned something else (FR-12). +func parseLearningTasks(child string, page Page, loc *time.Location) ([]Item, error) { + items, found, err := extractLearningTasks(child, page, loc) + if err != nil { + return nil, err + } + if !found { + return nil, fmt.Errorf("parse: %s: no learning-task grid found in %s (layout changed or fetch failed)", child, page.Slug) + } + return items, nil +} + +// extractLearningTasks returns the task rows in one grid page. found reports +// whether the page contained a learning-task grid at all, so a caller can accept +// an empty option page (for example the React child-profile page) without +// treating it as a failure. +func extractLearningTasks(child string, page Page, loc *time.Location) (items []Item, found bool, err error) { + root := parseHTML(page.HTML) + if !hasGrid(root) { + return nil, false, nil + } + + subject := "" + profileURL := "" + var parseErr error + stop := false + + walk(root, func(n *node) bool { + if stop || parseErr != nil { + return false + } + if n.Tag == "div" && n.hasClass("x-grid-group-title") { + if s, u := groupSubject(n); s != "" { + subject = s + if u != "" { + profileURL = u + } + } + return true + } + if n.Tag == "a" && n.hasClass("task-submission-launcher") { + it, err := taskFromAnchor(child, n, subject, profileURL, page, loc) + if err != nil { + parseErr = err + stop = true + return false + } + items = append(items, it) + return false + } + return true + }) + if parseErr != nil { + return nil, false, parseErr + } + return items, true, nil +} + +// hasGrid reports whether the document carries an ExtJS grid table. +func hasGrid(root *node) bool { + return find(root, func(n *node) bool { + return n.Tag == "table" && n.hasClass("x-grid-table") + }) != nil +} + +// groupSubject reads the subject and the child profile URL from an ExtJS group +// header such as: +// +// Harrison ROLFE - +// 11HLEG ... +func groupSubject(title *node) (subject, profileURL string) { + for _, a := range findAll(title, func(n *node) bool { return n.Tag == "a" }) { + href := a.Attrs["href"] + text := collapse(a.Text()) + switch { + case strings.Contains(href, "Subject.aspx"): + if subject == "" { + subject = text + } + case strings.Contains(href, "Records/User.aspx"): + if profileURL == "" && text != "" { + profileURL = href + } + } + } + if subject == "" { + // Fall back to the text after "NAME - ": "Harrison ROLFE - 11HLEG ...". + text := title.Text() + if i := strings.Index(text, " - "); i >= 0 { + subject = strings.TrimSpace(text[i+3:]) + } + } + return subject, profileURL +} + +// taskFromAnchor builds one task item from a task-submission-launcher anchor. +func taskFromAnchor(child string, a *node, subject, profileURL string, page Page, loc *time.Location) (Item, error) { + title := collapse(a.Attrs["data-qtip"]) + if title == "" { + title = collapse(a.Text()) + } + if title == "" { + return Item{}, fmt.Errorf("parse: %s: learning-task row with no title", page.Slug) + } + + tr := nearest(a, func(n *node) bool { return n.Tag == "tr" }) + if tr == nil { + return Item{}, fmt.Errorf("parse: %s: task %q has no row", page.Slug, title) + } + + var category string + if c := find(tr, func(n *node) bool { return n.hasClass("task-details-category-label") }); c != nil { + category = collapse(c.Text()) + } + + fields, kind := taskFields(tr) + rawDue := fields["Due date:"] + + due, err := parseCompassDue(rawDue, loc) + if err != nil { + return Item{}, fmt.Errorf("parse: %s: task %q: %w", page.Slug, title, err) + } + + submitted := fields["Submitted:"] + var body []string + if category != "" { + body = append(body, "Category: "+category) + } + if kind != "" { + body = append(body, "Task: "+kind) + } + if submitted != "" { + body = append(body, "Submitted: "+submitted) + } + if rawDue != "" { + body = append(body, "Due: "+rawDue) + } + + url := page.URL + if profileURL != "" { + url = absoluteCompassURL(profileURL) + "#learningTasks" + } else if page.URL != "" { + url = strings.Split(page.URL, "#")[0] + "#learningTasks" + } + + return Item{ + Child: child, + Source: SourceCompass, + Type: TypeTask, + Subject: subject, + Title: title, + Body: strings.Join(body, "\n"), + DueAt: due, + URL: url, + SourcePage: page.Slug, + ExternalRef: dedupeKey(subject, title, kind), + Importance: importanceNormal, + }, nil +} + +// taskFields walks a learning-task row and returns the datafield values by +// label, plus the task kind ("Class Task", "Subject-wide Task"). It relies on +// document order: each value span follows its title span. +func taskFields(tr *node) (fields map[string]string, kind string) { + fields = map[string]string{} + last := "" + walk(tr, func(n *node) bool { + if n.Tag != "span" { + return true + } + switch { + case n.hasClass("learning-task-datafield-title"): + text := collapse(n.Text()) + if !taskFieldLabels[text] && kind == "" { + kind = text + } + last = text + return false + case n.hasClass("learning-task-datafield-value"): + if last != "" { + fields[last] = collapse(n.Text()) + } + return false + } + return true + }) + return fields, kind +} + +// parseCompassDue parses a Compass due-date string in loc. "No due date" and an +// empty value mean no due date; anything else that does not parse is an error, +// never a silent zero (FR-12). +func parseCompassDue(s string, loc *time.Location) (*time.Time, error) { + s = collapse(s) + if s == "" || strings.EqualFold(s, noDueDate) { + return nil, nil + } + if loc == nil { + loc = time.UTC + } + for _, layout := range compassDueLayouts { + if t, err := time.ParseInLocation(layout, s, loc); err == nil { + return &t, nil + } + } + return nil, fmt.Errorf("unrecognised due date %q", s) +} + +// absoluteCompassURL turns a site-relative Compass href into a full URL. +func absoluteCompassURL(href string) string { + href = strings.TrimSpace(href) + if href == "" { + return "" + } + if strings.HasPrefix(href, "http://") || strings.HasPrefix(href, "https://") { + return href + } + if !strings.HasPrefix(href, "/") { + href = "/" + href + } + return compassHost + href +} + +// compassHost is the Northcote High School Compass origin (AGENTS.md §2a). +const compassHost = "https://nhs-vic.compass.education" + +// nearest returns the closest ancestor (or the node itself) matching pred. +func nearest(n *node, pred func(*node) bool) *node { + for cur := n; cur != nil; cur = cur.Parent { + if pred(cur) { + return cur + } + } + return nil +} + +// classCode matches Compass class codes such as 11HLEGB5 or 7CHI2. +var classCode = regexp.MustCompile(`^[0-9]{1,2}[A-Z]{2,}[A-Z0-9.]*$`) diff --git a/internal/store/migrations/002_items_parse.sql b/internal/store/migrations/002_items_parse.sql new file mode 100644 index 0000000..8e0e7d2 --- /dev/null +++ b/internal/store/migrations/002_items_parse.sql @@ -0,0 +1,13 @@ +-- Migration 002 — Phase 3 parse columns (AGENTS.md §4 FR-4/FR-5, §11 Phase 3). +-- +-- Every item records the page it came from (FR-5), a classification of +-- importance (FR-4) and a stable external reference so re-parsing the same page +-- updates a row instead of inserting a second one. + +ALTER TABLE items ADD COLUMN source_page TEXT NOT NULL DEFAULT ''; +ALTER TABLE items ADD COLUMN importance TEXT NOT NULL DEFAULT 'normal'; +ALTER TABLE items ADD COLUMN external_ref TEXT NOT NULL DEFAULT ''; + +-- One row per real item: the same source item parsed twice maps to one row. +CREATE UNIQUE INDEX IF NOT EXISTS idx_items_dedupe + ON items(child_id, source, type, external_ref); diff --git a/internal/store/store.go b/internal/store/store.go index a52b630..25f504d 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -269,3 +269,163 @@ func (s *Store) RecordScan(ctx context.Context, childID *int64, pages, items int } return res.LastInsertId() } + +// Item is one row of the items table (AGENTS.md §7). It mirrors parse.Item +// without importing the parse package, so the store stays independent of the +// scraper. +type Item struct { + ID int64 + ChildID int64 + Child string + Source string + Type string + Subject string + Teacher string + Title string + Body string + DueAt *time.Time + URL string + SourcePage string + Importance string + ExternalRef string +} + +// UpsertItems writes every item for one child, de-duplicating on +// (child_id, source, type, external_ref). It returns how many rows were +// inserted and how many were refreshed, so a re-parse never duplicates an item. +func (s *Store) UpsertItems(ctx context.Context, childID int64, items []Item) (inserted, updated int, err error) { + tx, err := s.db.BeginTx(ctx, nil) + if err != nil { + return 0, 0, fmt.Errorf("upsert items: begin: %w", err) + } + defer func() { _ = tx.Rollback() }() + + for _, it := range items { + if it.ExternalRef == "" { + return 0, 0, fmt.Errorf("upsert items: %q has no external reference", it.Title) + } + var existing int64 + err := tx.QueryRowContext(ctx, + `SELECT id FROM items WHERE child_id = ? AND source = ? AND type = ? AND external_ref = ?`, + childID, it.Source, it.Type, it.ExternalRef).Scan(&existing) + switch { + case err == sql.ErrNoRows: + if _, err := tx.ExecContext(ctx, + `INSERT INTO items (child_id, source, type, subject, teacher, title, body, + due_at, url, source_page, importance, external_ref) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + childID, it.Source, it.Type, it.Subject, it.Teacher, it.Title, it.Body, + formatDue(it.DueAt), it.URL, it.SourcePage, it.Importance, it.ExternalRef); err != nil { + return 0, 0, fmt.Errorf("insert item %q: %w", it.Title, err) + } + inserted++ + case err != nil: + return 0, 0, fmt.Errorf("look up item %q: %w", it.Title, err) + default: + if _, err := tx.ExecContext(ctx, + `UPDATE items SET subject = ?, teacher = ?, title = ?, body = ?, due_at = ?, + url = ?, source_page = ?, importance = ?, updated_at = datetime('now') + WHERE id = ?`, + it.Subject, it.Teacher, it.Title, it.Body, formatDue(it.DueAt), + it.URL, it.SourcePage, it.Importance, existing); err != nil { + return 0, 0, fmt.Errorf("update item %q: %w", it.Title, err) + } + updated++ + } + } + if err := tx.Commit(); err != nil { + return 0, 0, fmt.Errorf("upsert items: commit: %w", err) + } + return inserted, updated, nil +} + +// formatDue stores a due time in UTC RFC3339 so ordering is DST-safe, or NULL +// when there is no due date. +func formatDue(t *time.Time) any { + if t == nil { + return nil + } + return t.UTC().Format(time.RFC3339) +} + +// TypeCount is the number of items of one type for one child. +type TypeCount struct { + Child string + Type string + Count int +} + +// CountItemsByType returns per-child, per-type row counts, ordered by child then +// type. It is the evidence query for Phase 3. +func (s *Store) CountItemsByType(ctx context.Context) ([]TypeCount, error) { + rows, err := s.db.QueryContext(ctx, + `SELECT COALESCE(ch.name, '(unknown)'), i.type, COUNT(*) + FROM items i LEFT JOIN children ch ON ch.id = i.child_id + GROUP BY ch.name, i.type + ORDER BY ch.name, i.type`) + if err != nil { + return nil, fmt.Errorf("count items: %w", err) + } + defer rows.Close() + + var out []TypeCount + for rows.Next() { + var tc TypeCount + if err := rows.Scan(&tc.Child, &tc.Type, &tc.Count); err != nil { + return nil, fmt.Errorf("scan item count: %w", err) + } + out = append(out, tc) + } + return out, rows.Err() +} + +// SampleItems returns up to limit items for a child, newest due date first. +func (s *Store) SampleItems(ctx context.Context, childID int64, limit int) ([]Item, error) { + rows, err := s.db.QueryContext(ctx, + `SELECT i.id, i.child_id, COALESCE(ch.name, ''), i.source, i.type, + COALESCE(i.subject, ''), COALESCE(i.teacher, ''), i.title, + COALESCE(i.body, ''), i.due_at, COALESCE(i.url, ''), + COALESCE(i.source_page, ''), COALESCE(i.importance, ''), COALESCE(i.external_ref, '') + FROM items i LEFT JOIN children ch ON ch.id = i.child_id + WHERE i.child_id = ? + ORDER BY i.type, i.due_at DESC, i.id + LIMIT ?`, childID, limit) + if err != nil { + return nil, fmt.Errorf("sample items: %w", err) + } + defer rows.Close() + + var out []Item + for rows.Next() { + var it Item + var due sql.NullString + if err := rows.Scan(&it.ID, &it.ChildID, &it.Child, &it.Source, &it.Type, + &it.Subject, &it.Teacher, &it.Title, &it.Body, &due, &it.URL, + &it.SourcePage, &it.Importance, &it.ExternalRef); err != nil { + return nil, fmt.Errorf("scan item: %w", err) + } + if due.Valid { + t := parseTime(due.String) + it.DueAt = &t + } + out = append(out, it) + } + return out, rows.Err() +} + +// ChildByName returns one child by name. +func (s *Store) ChildByName(ctx context.Context, name string) (*Child, error) { + row := s.db.QueryRowContext(ctx, + `SELECT id, name, COALESCE(year, ''), COALESCE(compass_id, ''), created_at + FROM children WHERE name = ?`, name) + var c Child + var created string + if err := row.Scan(&c.ID, &c.Name, &c.Year, &c.CompassID, &created); err != nil { + if err == sql.ErrNoRows { + return nil, nil + } + return nil, fmt.Errorf("child %q: %w", name, err) + } + c.CreatedAt = parseTime(created) + return &c, nil +} diff --git a/internal/web/server.go b/internal/web/server.go index 8d9e831..8dcc0a0 100644 --- a/internal/web/server.go +++ b/internal/web/server.go @@ -133,12 +133,12 @@ func (s *Server) requireAuth(next http.HandlerFunc) http.HandlerFunc { // ---------- handlers ---------- type healthResponse struct { - Status string `json:"status"` - Service string `json:"service"` - Version string `json:"version"` - Database string `json:"database"` + Status string `json:"status"` + Service string `json:"service"` + Version string `json:"version"` + Database string `json:"database"` Scans store.ScanStatus `json:"scans"` - Time string `json:"time"` + Time string `json:"time"` } func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) { diff --git a/scripts/parse-compass.sh b/scripts/parse-compass.sh new file mode 100755 index 0000000..ba3e585 --- /dev/null +++ b/scripts/parse-compass.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Phase 3: parse the stored raw Compass pages for one child into SQLite. +# +# This runs the `school parse` subcommand in a one-off container that mounts the +# service's own data volume (read/write) and the stored raw pages (read-only). +# It does not restart or touch the running service. +# +# Usage: +# scripts/parse-compass.sh [--child Harry] [--run latest] [--classify auto] +# +# Environment overrides: +# IMAGE image tag (default 0.2.0-phase3) +# VOLUME SQLite volume (default compass-teams-nhs_compass-teams-nhs-data) +# RAW_ROOT host path to var/raw (default /var/raw) +# SCHOOL_TIMEZONE family timezone (default Australia/Melbourne) +# OMNIROUTE_BASE_URL OmniRoute endpoint (default http://127.0.0.1:20129/v1) +# TIMEOUT docker command timeout (default 300) +# +# Importance classification (FR-4) uses OmniRoute when OMNIROUTE_API_KEY is set; +# pass --classify none for the offline deterministic rule. +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +IMAGE="${IMAGE:-compass-teams-nhs/school:0.2.0-phase3}" +VOLUME="${VOLUME:-compass-teams-nhs_compass-teams-nhs-data}" +RAW_ROOT="${RAW_ROOT:-$REPO_ROOT/var/raw}" +SCHOOL_TIMEZONE="${SCHOOL_TIMEZONE:-Australia/Melbourne}" +OMNIROUTE_BASE_URL="${OMNIROUTE_BASE_URL:-http://127.0.0.1:20129/v1}" +TIMEOUT="${TIMEOUT:-300}" + +CHILD="Harry" +RUN="latest" +CLASSIFY="auto" +while [[ $# -gt 0 ]]; do + case "$1" in + --child) CHILD="$2"; shift 2 ;; + --run) RUN="$2"; shift 2 ;; + --classify) CLASSIFY="$2"; shift 2 ;; + *) echo "unknown argument: $1" >&2; exit 2 ;; + esac +done + +if [[ ! -d "$RAW_ROOT" ]]; then + echo "raw pages not found at $RAW_ROOT" >&2 + exit 1 +fi + +exec timeout "$TIMEOUT" docker run --rm --network host \ + -v "$VOLUME:/data" \ + -v "$RAW_ROOT:/raw:ro" \ + -e SCHOOL_DB_PATH=/data/school.db \ + -e SCHOOL_TIMEZONE="$SCHOOL_TIMEZONE" \ + -e OMNIROUTE_API_KEY \ + -e OMNIROUTE_BASE_URL="$OMNIROUTE_BASE_URL" \ + "$IMAGE" parse --child "$CHILD" --run "$RUN" --raw-root /raw --classify "$CLASSIFY"