Sam Rolfe
d7d529eb0f
Turn the raw pages stored by Phase 2 into rows in the SQLite items table (AGENTS.md §11 Phase 3, FR-3/FR-4/FR-5/FR-12). - internal/parse: dependency-free HTML reader plus extractors for the ExtJS learning-task grid (subject, title, category, kind, due date, status), the home news feed (author, posted time, title, body) and the calendar week grid (day column + grid offset give the time; label gives title/room/teacher). - internal/parse: fail loudly. A challenge page, login page, changed layout or unknown due-date format is an explicit error, never an empty day. - FR-4: structure is parsed deterministically; importance is classified by an OmniRoute model (default openrouter/google/gemini-3.5-flash-lite) with a deterministic fallback. The classifier used is printed and recorded. - internal/llm: small OpenAI-compatible OmniRoute client (handles SSE replies). - internal/store: migration 002 adds source_page, importance and external_ref with a unique (child_id, source, type, external_ref) index; UpsertItems, CountItemsByType, SampleItems and ChildByName. - cmd/school parse: reads var/raw/<child>/<run>, upserts items, records the scan row (ok=0 plus the error on failure) and prints counts and samples. - scripts/parse-compass.sh and make parse run it in a one-off container against the service's own volume without restarting the service. - docs/phase-3-parse-notes.md records the approach, the FR-4 decision and the known gaps. Evidence on the live database: 56 rows for Harry (20 task, 1 message, 35 event). Re-running the same raw pages inserts 0 and updates 56. Co-Authored-By: Paperclip <noreply@paperclip.ing>
57 lines
2.1 KiB
Bash
Executable File
57 lines
2.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Phase 3: parse the stored raw Compass pages for one child into SQLite.
|
|
#
|
|
# This runs the `school parse` subcommand in a one-off container that mounts the
|
|
# service's own data volume (read/write) and the stored raw pages (read-only).
|
|
# It does not restart or touch the running service.
|
|
#
|
|
# Usage:
|
|
# scripts/parse-compass.sh [--child Harry] [--run latest] [--classify auto]
|
|
#
|
|
# Environment overrides:
|
|
# IMAGE image tag (default 0.2.0-phase3)
|
|
# VOLUME SQLite volume (default compass-teams-nhs_compass-teams-nhs-data)
|
|
# RAW_ROOT host path to var/raw (default <repo>/var/raw)
|
|
# SCHOOL_TIMEZONE family timezone (default Australia/Melbourne)
|
|
# OMNIROUTE_BASE_URL OmniRoute endpoint (default http://127.0.0.1:20129/v1)
|
|
# TIMEOUT docker command timeout (default 300)
|
|
#
|
|
# Importance classification (FR-4) uses OmniRoute when OMNIROUTE_API_KEY is set;
|
|
# pass --classify none for the offline deterministic rule.
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
|
|
IMAGE="${IMAGE:-compass-teams-nhs/school:0.2.0-phase3}"
|
|
VOLUME="${VOLUME:-compass-teams-nhs_compass-teams-nhs-data}"
|
|
RAW_ROOT="${RAW_ROOT:-$REPO_ROOT/var/raw}"
|
|
SCHOOL_TIMEZONE="${SCHOOL_TIMEZONE:-Australia/Melbourne}"
|
|
OMNIROUTE_BASE_URL="${OMNIROUTE_BASE_URL:-http://127.0.0.1:20129/v1}"
|
|
TIMEOUT="${TIMEOUT:-300}"
|
|
|
|
CHILD="Harry"
|
|
RUN="latest"
|
|
CLASSIFY="auto"
|
|
while [[ $# -gt 0 ]]; do
|
|
case "$1" in
|
|
--child) CHILD="$2"; shift 2 ;;
|
|
--run) RUN="$2"; shift 2 ;;
|
|
--classify) CLASSIFY="$2"; shift 2 ;;
|
|
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
|
esac
|
|
done
|
|
|
|
if [[ ! -d "$RAW_ROOT" ]]; then
|
|
echo "raw pages not found at $RAW_ROOT" >&2
|
|
exit 1
|
|
fi
|
|
|
|
exec timeout "$TIMEOUT" docker run --rm --network host \
|
|
-v "$VOLUME:/data" \
|
|
-v "$RAW_ROOT:/raw:ro" \
|
|
-e SCHOOL_DB_PATH=/data/school.db \
|
|
-e SCHOOL_TIMEZONE="$SCHOOL_TIMEZONE" \
|
|
-e OMNIROUTE_API_KEY \
|
|
-e OMNIROUTE_BASE_URL="$OMNIROUTE_BASE_URL" \
|
|
"$IMAGE" parse --child "$CHILD" --run "$RUN" --raw-root /raw --classify "$CLASSIFY"
|