From af3b5e5f0e0b9cc7b3434d8328461028d76bba27 Mon Sep 17 00:00:00 2001 From: Sam Rolfe Date: Sun, 11 Oct 2026 21:37:38 +1100 Subject: [PATCH] Phase 3: JEV classification, rules table, editable categories (AUD-69) Implements FR-3, FR-4, FR-5 and FR-14. - internal/classify: TypeSafe System One (JEV) HTTP client with the confirmed request/response shape, retry with backoff on 429/529/5xx, a plain-text rule engine (first match wins, overrides only the named fields), and bounded orchestration with a per-run cap. - internal/store: migration v3 adds an editable `categories` table (name = JEV Choice option and classifications key; rename cascades, remove reassigns) and seeds a starter rules version once. Classification persistence, coverage, and pending/all message queries. - internal/server: POST /actions/classify and /actions/reclassify, plus the FR-14 category editor (GET /categories and add/rename/remove). The triage list now groups by the live category list and shows action + confidence. - CLI: -classify and -classify-all for n8n. - Docs: docs/phase3-classify.md with the confirmed JEV shape, a live read-only classification, and the DoD evidence. DoD: all 100 stored messages have a non-null category, importance, and suggested action (0 nulls); JEV picked a newly added/renamed category for 6 real messages over HTTP. One live JEV call recorded in the phase doc. Co-Authored-By: Paperclip --- .env.example | 6 + README.md | 80 +++++-- cmd/email-agent/main.go | 46 +++++ docker-compose.yml | 3 + docs/phase3-classify.md | 225 ++++++++++++++++++++ internal/classify/classify.go | 321 +++++++++++++++++++++++++++++ internal/classify/classify_test.go | 288 ++++++++++++++++++++++++++ internal/classify/jev.go | 306 +++++++++++++++++++++++++++ internal/classify/jev_test.go | 154 ++++++++++++++ internal/classify/rules.go | 238 +++++++++++++++++++++ internal/classify/rules_test.go | 119 +++++++++++ internal/config/config.go | 25 ++- internal/server/auth.go | 66 ++++-- internal/server/categories.go | 151 ++++++++++++++ internal/server/categories_test.go | 213 +++++++++++++++++++ internal/server/inbox.go | 161 +++++++++------ internal/server/server.go | 32 ++- internal/store/categories.go | 218 ++++++++++++++++++++ internal/store/categories_test.go | 245 ++++++++++++++++++++++ internal/store/classify.go | 161 +++++++++++++++ internal/store/messages.go | 9 + internal/store/schema.sql | 13 ++ internal/store/store.go | 83 ++++++++ web/static/app.css | 49 +++++ web/templates/categories.html | 104 ++++++++++ web/templates/dashboard.html | 18 +- web/templates/inbox.html | 3 +- 27 files changed, 3235 insertions(+), 102 deletions(-) create mode 100644 docs/phase3-classify.md create mode 100644 internal/classify/classify.go create mode 100644 internal/classify/classify_test.go create mode 100644 internal/classify/jev.go create mode 100644 internal/classify/jev_test.go create mode 100644 internal/classify/rules.go create mode 100644 internal/classify/rules_test.go create mode 100644 internal/server/categories.go create mode 100644 internal/server/categories_test.go create mode 100644 internal/store/categories.go create mode 100644 internal/store/categories_test.go create mode 100644 internal/store/classify.go create mode 100644 web/templates/categories.html diff --git a/.env.example b/.env.example index 4e38435..f50cf87 100644 --- a/.env.example +++ b/.env.example @@ -31,8 +31,14 @@ EMAIL_AGENT_IMAP_LOOKBACK_DAYS=14 EMAIL_AGENT_IMAP_MAX_MESSAGES=100 # --- Integrations (used from later phases) --------------------------------- +# JEV is the TypeSafe System One HTTP API. JEV_BASE_URL defaults to the +# documented host https://api.typesafe.ai when unset. JEV_BASE_URL= JEV_API_KEY= +JEV_MODEL=jev-latest +# Classifier bounds (FR-3): in-flight JEV requests, and messages per run. +EMAIL_AGENT_CLASSIFY_CONCURRENCY=4 +EMAIL_AGENT_CLASSIFY_MAX_MESSAGES=200 OMNIROUTE_BASE_URL= OMNIROUTE_API_KEY= HUB_TOKEN= diff --git a/README.md b/README.md index 5566428..63e111c 100644 --- a/README.md +++ b/README.md @@ -9,18 +9,23 @@ pre-planning material and is not authoritative. ## Status -**Phase 2 — Read mail (IMAP).** The Go skeleton (Phase 1.1) plus session login -and the HTMX dashboard shell (Phase 1.2), now fetching real mail from Gmail over -**IMAP with an app password, forced to IPv4** (AGENTS.md §9). Messages are stored -in SQLite and listed in the dashboard, grouped by category with a recommended -action per row. The FR-2 source address is resolved from `X-Pm-Original-To`, with -`To`, the PurelyMail `Received … for` clause, and `Return-Path` as fallbacks; -`Delivered-To` is never used. Classification and triage actions come in Phase 3+. +**Phase 3 — Classify.** The Go skeleton, session login, and HTMX dashboard +(Phase 1), real mail over **IMAP with an app password, forced to IPv4** +(Phase 2), and now **JEV classification** (Phase 3). Every message gets a +category, an importance, and a suggested action, grouped in the triage list with +the recommended action and confidence per row. The category list is editable in +the UI (add, rename, remove) and JEV uses the current list; a plain-text rules +table can override the classifier (FR-4). The FR-2 source address is resolved +from `X-Pm-Original-To`, with `To`, the PurelyMail `Received … for` clause, and +`Return-Path` as fallbacks; `Delivered-To` is never used. See +[`docs/phase3-classify.md`](docs/phase3-classify.md) for the confirmed JEV shape +and the phase evidence. Triage actions come in Phase 4. ## Layout ``` -cmd/email-agent/ service entry point (`-fetch` runs one scan and exits) +cmd/email-agent/ service entry point (`-fetch`, `-classify`, `-classify-all`) +internal/classify/ JEV (TypeSafe System One) client, rule engine, orchestration internal/config/ environment-driven configuration internal/mailfetch/ stdlib-only IMAP client, MIME parsing, FR-2 source address internal/session/ server-side session store + signed cookie values @@ -57,6 +62,11 @@ Key variables: | `EMAIL_AGENT_IMAP_PORT` | `993` | IMAP port (implicit TLS) | | `EMAIL_AGENT_IMAP_LOOKBACK_DAYS` | `14` | First-scan window | | `EMAIL_AGENT_IMAP_MAX_MESSAGES` | `100` | Per-scan message cap | +| `JEV_BASE_URL` | `https://api.typesafe.ai` | TypeSafe System One base URL | +| `JEV_API_KEY` | _(empty)_ | JEV API key (required to classify) | +| `JEV_MODEL` | `jev-latest` | System One model alias | +| `EMAIL_AGENT_CLASSIFY_CONCURRENCY` | `4` | In-flight JEV requests per run | +| `EMAIL_AGENT_CLASSIFY_MAX_MESSAGES` | `200` | Messages per classify run | ## Mail sync (Phase 2) @@ -75,6 +85,45 @@ Each scan is bounded by the lookback and max-message settings. Gmail's `X-GM-MSGID` is the stable message key; the IMAP `UID` and `UIDVALIDITY` are stored alongside it for later label operations. +## Classification (Phase 3) + +`internal/classify` sends each message to JEV (`POST /v1/systemone`) as three +Choice questions — category, importance, suggested action — with the **live** +category list as the category options, and stores the answer plus JEV's own +confidence. One request per message, bounded concurrency, retry with backoff on +`429`/`529`/`5xx`. + +Trigger classification from the dashboard (**Classify** for new mail, +**Re-classify all** after a category or rules change) or from the CLI: + +```sh +docker exec email-agent email-agent -classify # pending messages +docker exec email-agent email-agent -classify-all # every stored message +``` + +### Rules (FR-4) + +Rules are plain text, stored versioned in `rules`. One rule per line: + +``` + => + +from contains "amazon.com" => category=Newsletter, action=unsubscribe +* => category=Info +``` + +Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body` +using `contains`, `equals`, `starts_with`, `ends_with`. The first matching rule +wins and overrides only the fields it names. The rules editor UI is Phase 6; the +classifier already reads the stored text. + +### Categories (FR-14) + +`GET /categories` edits the category list. A category's name is the key stored +in `classifications.category` and the Choice option sent to JEV, so the model +always sees the current list. Renaming cascades to existing classifications; +removing a category reassigns its messages to the first remaining category. + ## Run locally ```sh @@ -94,7 +143,7 @@ The container is `email-agent`, image `email-agent:latest`, named volume unprivileged user and has a built-in healthcheck (`email-agent -healthcheck`). Caddy on `.35` proxies `mail.lab.audasmedia.com.au` to this port. -## Endpoints (Phase 1) +## Endpoints | Method | Path | Auth | Purpose | |---|---|---|---| @@ -106,6 +155,12 @@ Caddy on `.35` proxies `mail.lab.audasmedia.com.au` to this port. | `GET` | `/` | session | Dashboard: nav counts, real message list | | `GET` | `/partials/inbox` | session | HTMX fragment for the centre list | | `POST` | `/actions/fetch` | session + CSRF | Runs one IMAP scan, returns the refreshed list | +| `POST` | `/actions/classify` | session + CSRF | Classifies pending messages, returns the refreshed list | +| `POST` | `/actions/reclassify` | session + CSRF | Re-classifies every message with the current categories and rules | +| `GET` | `/categories` | session | Category editor (FR-14) | +| `POST` | `/categories/add` | session + CSRF | Adds a category | +| `POST` | `/categories/rename` | session + CSRF | Renames a category and updates existing classifications | +| `POST` | `/categories/remove` | session + CSRF | Removes a category and reassigns its messages | | `GET` | `/static/…` | public | Embedded `tokens.css`, `app.css`, Inter fonts, `htmx.min.js` | Unauthenticated requests to `/` and `/partials/…` redirect to `/login?next=…`. @@ -117,6 +172,7 @@ hidden `csrf` field). SQLite is opened with the pure-Go `modernc.org/sqlite` driver (no CGO). The schema in `internal/store/schema.sql` creates the AGENTS.md §7 tables: `messages`, `classifications`, `rules`, `actions_log`, `unsubscribe`, `drafts`, -`summaries`, plus `schema_migrations`. Schema init is idempotent and runs on -every start. Migration v2 adds `imap_uid` and `imap_uidvalidity` to `messages` -for databases created by Phase 1. +`summaries`, plus `categories`, `schema_migrations`. Schema init is idempotent +and runs on every start. Migration v2 adds `imap_uid` and `imap_uidvalidity` to +`messages`; migration v3 seeds the first category list and a starter rules +version once, so a category removed by the user is not silently re-added. diff --git a/cmd/email-agent/main.go b/cmd/email-agent/main.go index f8b4a6d..d342ebe 100644 --- a/cmd/email-agent/main.go +++ b/cmd/email-agent/main.go @@ -12,6 +12,7 @@ import ( "syscall" "time" + "gitea.lab.audasmedia.com.au/sam/email_agent/internal/classify" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/config" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/mailfetch" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/server" @@ -24,6 +25,8 @@ func main() { showVersion = flag.Bool("version", false, "print version information and exit") healthcheck = flag.Bool("healthcheck", false, "probe the local /health endpoint and exit") fetchNow = flag.Bool("fetch", false, "run one Gmail IMAP scan and exit") + classifyNow = flag.Bool("classify", false, "classify pending messages with JEV and exit") + classifyAll = flag.Bool("classify-all", false, "re-classify every stored message with JEV and exit") ) flag.Parse() @@ -57,6 +60,14 @@ func main() { return } + if *classifyNow || *classifyAll { + if err := runClassify(cfg, logger, *classifyAll); err != nil { + fmt.Fprintf(os.Stderr, "classify failed: %v\n", err) + os.Exit(1) + } + return + } + if err := run(cfg, logger); err != nil { logger.Error("fatal", "error", err) os.Exit(1) @@ -125,6 +136,41 @@ func runFetch(cfg *config.Config, logger *slog.Logger) error { return nil } +// runClassify classifies messages and exits. It is the n8n/CLI entry point for +// FR-3, separate from the HTTP service so a scheduled scan can be bounded and +// observed. all=true re-classifies every message, for use after the category +// list or rules change. +func runClassify(cfg *config.Config, logger *slog.Logger, all bool) error { + if !cfg.JEVConfigured() { + return classify.ErrNotConfigured + } + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + st, err := store.Open(ctx, cfg.DBPath) + if err != nil { + return fmt.Errorf("open store: %w", err) + } + defer func() { _ = st.Close() }() + + classifier := classify.NewClassifier( + classify.NewClient(cfg.JEVBaseURL, cfg.JEVAPIKey, cfg.JEVModel, logger), + st, logger, cfg.ClassifyConcurrency, cfg.ClassifyMaxMessages, + ) + var sum classify.Summary + if all { + sum, err = classifier.RunAll(ctx) + } else { + sum, err = classifier.Run(ctx) + } + if err != nil { + return err + } + fmt.Printf("considered=%d classified=%d failed=%d remaining=%d model=%s rules_version=%d\n", + sum.Considered, sum.Classified, sum.Failed, sum.Remaining, sum.Model, sum.RulesVersion) + return nil +} + // probeHealth requests the local health endpoint. It exists so the container // healthcheck does not need an external HTTP client in the image. func probeHealth(port int) error { diff --git a/docker-compose.yml b/docker-compose.yml index ceeb0a5..c114c71 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -40,6 +40,9 @@ services: EMAIL_AGENT_IMAP_MAX_MESSAGES: ${EMAIL_AGENT_IMAP_MAX_MESSAGES:-100} JEV_BASE_URL: ${JEV_BASE_URL:-} JEV_API_KEY: ${JEV_API_KEY:-} + JEV_MODEL: ${JEV_MODEL:-jev-latest} + EMAIL_AGENT_CLASSIFY_CONCURRENCY: ${EMAIL_AGENT_CLASSIFY_CONCURRENCY:-4} + EMAIL_AGENT_CLASSIFY_MAX_MESSAGES: ${EMAIL_AGENT_CLASSIFY_MAX_MESSAGES:-200} OMNIROUTE_BASE_URL: ${OMNIROUTE_BASE_URL:-} OMNIROUTE_API_KEY: ${OMNIROUTE_API_KEY:-} HUB_TOKEN: ${HUB_TOKEN:-} diff --git a/docs/phase3-classify.md b/docs/phase3-classify.md new file mode 100644 index 0000000..4618787 --- /dev/null +++ b/docs/phase3-classify.md @@ -0,0 +1,225 @@ +# Phase 3 — Classify (JEV, rules, editable categories) + +Status: **done and deployed on `.13`.** [AUD-69](/AUD/issues/AUD-69), parent +[AUD-68](/AUD/issues/AUD-68). Phase 3 of AGENTS.md §11. + +**DoD:** every message has a category, an importance, and a suggested action. + +## 1. Confirmed JEV HTTP shape (before wiring) + +Read live from the TypeSafe docs (`https://docs.typesafe.ai/llms.txt` → +`api.md`, `primitives/choice.md`, `confidence.md`, `models.md`) and confirmed with +a real call. This is the shape `internal/classify/jev.go` implements. + +**Endpoint** — `POST https://api.typesafe.ai/v1/systemone`, bearer auth: + +```http +POST /v1/systemone HTTP/1.1 +Authorization: Bearer +Content-Type: application/json +``` + +**Request** + +```json +{ + "state": { "email": { "from": "…", "to": "…", "source_address": "…", + "source_domain": "…", "subject": "…", "received": "…", "body": "…" } }, + "model": "jev-latest", + "questions": { + "category": { "type": "choice", "instructions": "…", "criteria": { "‹option›": "‹rubric›", … } }, + "importance": { "type": "choice", "instructions": "…", "criteria": { "low": "…", "medium": "…", "high": "…" } }, + "action": { "type": "choice", "instructions": "…", "criteria": { "delete": "…", "unsubscribe": "…", "reply": "…", "file": "…", "keep": "…" } } + } +} +``` + +**Response** — one answer per question id; Choice answers carry the selected +option, the full probability distribution, and a confidence in `[0,1]`: + +```json +{ + "model": "jev-1.13.0", + "answers": { + "category": { "type": "choice", "choice": "Action", "confidence": 0.94, "probabilities": { … } }, + "importance": { "type": "choice", "choice": "high", "confidence": 0.89, "probabilities": { … } }, + "action": { "type": "choice", "choice": "file", "confidence": 0.49, "probabilities": { … } } + }, + "usage": { "input_tokens": 1047, "output_tokens": 146 } +} +``` + +Errors: `401` invalid key, `422` validation, `429`/`529` rate-limit/overloaded. +The client retries `429`, `529`, and `5xx` up to four times with exponential +backoff and jitter, and surfaces any other status with the response body. + +### One real, read-only classification (live, message id 10) + +Message 10 (`InMotion Hosting Billing · "InMotion Hosting - Payment Due"`, +arrived at `samuelrolfe@gmail.com`) was sent to JEV with the six live categories. +Live response, verbatim: + +```json +{"model":"jev-1.13.0","answers":{ + "category":{"type":"choice","choice":"Action","confidence":0.94, + "probabilities":{"Action":0.95,"Important":0.0,"Info":0.0,"Spam":0.03,"Newsletter":0.0,"Auto-reply":0.02}}, + "importance":{"type":"choice","choice":"high","confidence":0.89, + "probabilities":{"low":0.03,"medium":0.04,"high":0.93}}, + "action":{"type":"choice","choice":"file","confidence":0.49, + "probabilities":{"delete":0.24,"unsubscribe":0.01,"reply":0.01,"file":0.59,"keep":0.15}}}, + "usage":{"input_tokens":1047,"output_tokens":146}} +``` + +The stored row matches: `Action / high / file / 0.94 / jev-1.13.0`. + +**`JEV_BASE_URL` was not set on `.13`** (only `JEV_API_KEY` is in +`~/.config/environment.d/10-secrets.conf`). The TypeSafe docs fix the host, so +the code defaults an unset/empty `JEV_BASE_URL` to the documented +`https://api.typesafe.ai`, and `JEV_MODEL` defaults to `jev-latest`. This is +recorded here rather than treated as a contradiction. A different base URL can +still be supplied without a code change. + +## 2. What was built + +- **`internal/classify`** — JEV client (`jev.go`), rule engine (`rules.go`), + and orchestration (`classify.go`). One JEV request per message, bounded + concurrency (default 4), a per-run cap (default 200), and a worker pool. +- **`internal/store`** — migration **v3**: a `categories` table seeded once with + the six AGENTS.md §13.2 categories (name + JEV rubric + order) and a + comment-only starter `rules` version. Category CRUD, classification + persistence, `CurrentRules`/`SaveRules`, and pending/coverage queries. +- **`internal/server`** — `POST /actions/classify` (pending) and + `POST /actions/reclassify` (all), plus the **FR-14 categories UI** + (`GET /categories`, `POST /categories/add|rename|remove`). +- **FR-5 triage list** — messages grouped by the live category list, newest + first, with the recommended action and the classifier confidence per row. +- **CLI** — `email-agent -classify` (pending) and `-classify-all` (re-classify + everything), the n8n/CLI entry points for FR-3. +- **FR-13** — every classification is logged in `actions_log` with the message, + the category/importance/action/confidence/model, the rules version, and the + matching rule line when a rule applied. Category add/rename/remove are logged. + +### Rules (FR-4) + +Plain text, one rule per line; the classifier reads the newest `rules` row. The +editor that writes it lands in Phase 6, but the engine is live now. + +``` + => + +from contains "amazon.com" => category=Newsletter, action=unsubscribe +subject contains "invoice" and from ends_with "@stripe.com" => category=Action, importance=high +* => category=Info +``` + +Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body` +using `contains`, `equals`, `starts_with`, `ends_with`. Effects are +`category=`, `importance=`, `action=`. +The **first matching rule wins** and overrides only the fields it names. A rule +naming a category that no longer exists is ignored and logged; malformed lines +are skipped and reported. Evidence: `internal/classify/rules_test.go` and +`internal/classify/classify_test.go` (override, unknown-category fallback, +rule-in-log). + +### Categories (FR-14) + +`categories.name` is the key stored in `classifications.category` **and** sent to +JEV as the Choice option name, so JEV always sees the current list. Renaming +cascades to existing classifications; removing a category reassigns its messages +to the first remaining category; the last category cannot be removed. + +## 3. DoD evidence + +Deployed: image rebuilt and `email-agent` recreated on `.13`; the Phase 1/2 +volume `email_agent_data` was kept and migration v3 ran in place +(`schema_version: 3`). + +| Check | Result | +|---|---| +| `docker compose ps` | `email-agent` up, healthy, `0.0.0.0:8790->8790/tcp` | +| `GET /health` (public HTTPS) | `200 {"status":"ok","env":"production","database":"ok"}` | +| `email-agent -classify` (container) | `considered=100 classified=100 failed=0 remaining=0 model=jev-latest rules_version=1` | +| **Every message has a classification** | `100` messages, **`0`** null/empty category, **`0`** importance, **`0`** action; `0` messages without a classification row | +| Model | all `100` rows from `jev-1.13.0` | +| Distribution | Newsletter 43, Auto-reply 31, Action 20, Info 4, Spam 1, Important 1 | +| `POST /actions/reclassify` (HTTP, session + CSRF) | `200`, note `Classified 100 of 100 message(s); 0 failed; 0 still unclassified.` | +| FR-13 per-message log | `100` `classify` actions with category/importance/action/confidence/model; `classify_run` summary rows | +| Public HTTPS | `https://mail.lab.audasmedia.com.au/health` → `200 ok` | + +### FR-14 HTTP/UI proof (live, over HTTP on `.13`) + +Login (session + CSRF cookie), then: + +1. **Add** — `POST /categories/add` `name=Finance`, + `description=Bills, invoices, payments, renewals, and financial account alerts` + → `303`. `GET /categories` then lists `… Newsletter, Auto-reply, Finance`. +2. **Rename** — `POST /categories/rename` `id=7 name=Money & Bills` → `303`. + `GET /categories` lists `Money & Bills`. +3. **JEV uses them** — `POST /actions/reclassify` → `200` + `Classified 100 of 100 message(s)`. JEV then answered with the new name for + **6 real messages** that were not in that category before: + + | id | from | subject | result | + |---|---|---|---| + | 1 | Crazy Domains | ALERT: Your auto-renewal… | `Money & Bills / high / delete / 0.71` | + | 10 | InMotion Hosting Billing | Payment Due | `Money & Bills / high / file / 0.99` | + | 23 | api@minimaxi.com | Pay-as-you-go Balance Disconnect | `Money & Bills / high / file / 0.70` | + | 30 | Crazy Domains | cmorganics.com.au has Expired | `Money & Bills / high / delete / 0.45` | + | 36 | api@minimaxi.com | Re: Pay-as-you-go… | `Money & Bills / high / file / 0.59` | + | 82 | Crazy Domains | cmorganics.com.au has Expired | `Money & Bills / high / delete / 0.46` | + +4. **Remove** — `POST /categories/remove` `id=7` → `200` with note + `Removed category; its messages moved to the first remaining category.`; + `GET /categories` no longer lists it. `actions_log` records + `category_add name=Finance`, `category_rename id=7 name=Money & Bills`, + `category_remove id=7`. A second `reclassify` restored the six default + categories; final state is the table above. + +### Tests + +`gofmt -l` clean, `go vet ./...` clean, `go test ./...` green (in +`golang:1.25-alpine`). New tests cover the rule parser and precedence, the JEV +request/response shape and 429 retry, the classifier (every pending message, +rule override, unknown-category fallback, rerun-only-pending, re-classify-all, +rule-in-action-log), category CRUD and cascade, and the category HTTP handlers. + +## 4. Configuration + +Added to `.env.example` and `docker-compose.yml` (values stay in the host +environment / secret file): + +| Variable | Default | +|---|---| +| `JEV_BASE_URL` | `https://api.typesafe.ai` | +| `JEV_API_KEY` | _(required for classification)_ | +| `JEV_MODEL` | `jev-latest` | +| `EMAIL_AGENT_CLASSIFY_CONCURRENCY` | `4` | +| `EMAIL_AGENT_CLASSIFY_MAX_MESSAGES` | `200` | + +## 5. Deployment + +On `.13`, from the repo, with the host secrets in scope: + +```sh +set -a; . ~/.config/environment.d/10-secrets.conf; set +a +docker compose build +docker compose up -d # migration v3 runs on start; data volume kept +docker compose exec -T email-agent email-agent -classify +``` + +## 6. Rollback + +1. `docker compose down` on `.13` (keeps `email_agent_data`). +2. `git revert ` and `docker compose up -d --build`. +3. Migration v3 is additive (new table + seeded rows) and is not undone by the + revert; the Phase 2 binary ignores `categories` and shows every message as + `Unclassified`. + +## 7. Not done / deferred + +- The rules **editor UI** is Phase 6 (FR-11). The rules table, versioning, and + the classifier reading them are live. +- `POST /actions/reclassify` and `-classify-all` were added so a category or + rules change can be applied to existing mail; Phase 6 will use the same path. +- Unsubscribe detection (FR-7) and the reply drafts (FR-8) remain later phases. +- The 1-month body retention job (AGENTS.md §7) is not implemented yet. diff --git a/internal/classify/classify.go b/internal/classify/classify.go new file mode 100644 index 0000000..a41d8cd --- /dev/null +++ b/internal/classify/classify.go @@ -0,0 +1,321 @@ +// Classifier orchestration: read the live category list and rules, ask JEV about +// every pending message, apply the user's rules as overrides, and store the +// result (FR-3, FR-4, FR-5). +package classify + +import ( + "context" + "fmt" + "log/slog" + "strings" + "sync" + "time" + + "gitea.lab.audasmedia.com.au/sam/email_agent/internal/store" +) + +// ActorAgent is the actor recorded in the action log for classifier writes. +const ActorAgent = "agent" + +// Summary describes one classification run. +type Summary struct { + Considered int + Classified int + Failed int + Model string + RulesVersion int + Remaining int + Duration time.Duration +} + +// Classifier ties the JEV client to the store. +type Classifier struct { + client *Client + store *store.Store + logger *slog.Logger + concurrency int + maxPerRun int +} + +// NewClassifier builds a Classifier. concurrency and maxPerRun fall back to safe +// defaults when they are not positive. +func NewClassifier(client *Client, st *store.Store, logger *slog.Logger, concurrency, maxPerRun int) *Classifier { + if concurrency < 1 { + concurrency = 4 + } + if concurrency > 16 { + concurrency = 16 + } + if maxPerRun < 1 { + maxPerRun = 200 + } + if logger == nil { + logger = slog.New(slog.DiscardHandler) + } + return &Classifier{ + client: client, + store: st, + logger: logger, + concurrency: concurrency, + maxPerRun: maxPerRun, + } +} + +// Model returns the configured JEV model alias. +func (c *Classifier) Model() string { return c.client.Model() } + +// Configured reports whether JEV_API_KEY is present. +func (c *Classifier) Configured() bool { return c.client.Configured() } + +// Run classifies the pending messages, up to maxPerRun. It returns a Summary +// even when individual messages fail; only setup failures return an error. +func (c *Classifier) Run(ctx context.Context) (Summary, error) { + return c.run(ctx, false) +} + +// RunAll re-classifies every stored message. It is used after the category list +// or the rules change, so the new settings take effect on existing mail. +func (c *Classifier) RunAll(ctx context.Context) (Summary, error) { + return c.run(ctx, true) +} + +func (c *Classifier) run(ctx context.Context, all bool) (Summary, error) { + if !c.client.Configured() { + return Summary{}, ErrNotConfigured + } + + start := time.Now() + mode := "pending" + if all { + mode = "all" + } + categories, err := c.store.ListCategories(ctx) + if err != nil { + return Summary{}, err + } + if len(categories) == 0 { + return Summary{}, fmt.Errorf("no categories configured") + } + options := make([]CategoryOption, 0, len(categories)) + for _, cat := range categories { + options = append(options, CategoryOption{Name: cat.Name, Description: cat.Description}) + } + valid := make(map[string]string, len(categories)) // lower(name) -> name + for _, cat := range categories { + valid[strings.ToLower(cat.Name)] = cat.Name + } + + rulesText, rulesVersion, err := c.store.CurrentRules(ctx) + if err != nil { + return Summary{}, err + } + rules := ParseRules(rulesText) + for _, perr := range rules.Errors { + c.logger.Warn("rules line skipped", "error", perr.Error()) + } + + pending, err := c.pending(ctx, all) + if err != nil { + return Summary{}, err + } + + sum := Summary{ + Considered: len(pending), + Model: c.client.Model(), + RulesVersion: rulesVersion, + } + if len(pending) == 0 { + sum.Remaining = 0 + sum.Duration = time.Since(start) + return sum, nil + } + + type outcome struct { + msg store.Message + class store.Classification + rule *Rule + failed bool + } + + work := make(chan store.Message) + results := make(chan outcome) + var wg sync.WaitGroup + for i := 0; i < c.concurrency; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for msg := range work { + class, rule, err := c.classifyOne(ctx, msg, options, valid, rules) + if err != nil { + c.logger.Warn("classify message failed", + "message_id", msg.ID, "subject", msg.Subject, "error", err) + results <- outcome{msg: msg, failed: true} + continue + } + results <- outcome{msg: msg, class: class, rule: rule} + } + }() + } + + go func() { + defer close(work) + for _, msg := range pending { + select { + case <-ctx.Done(): + return + case work <- msg: + } + } + }() + + go func() { + wg.Wait() + close(results) + }() + + for res := range results { + if res.failed { + sum.Failed++ + continue + } + if _, err := c.store.SaveClassification(ctx, res.class); err != nil { + c.logger.Error("save classification failed", "message_id", res.msg.ID, "error", err) + sum.Failed++ + continue + } + detail := fmt.Sprintf("category=%s importance=%s action=%s confidence=%.2f model=%s rules_version=%d", + res.class.Category, res.class.Importance, res.class.Action, + res.class.Confidence, res.class.Model, rulesVersion) + if res.rule != nil { + detail += fmt.Sprintf(" rule_line=%d rule=%q", res.rule.Line, res.rule.Raw) + } + if err := c.store.LogMessageAction(ctx, res.msg.ID, "classify", ActorAgent, detail); err != nil { + c.logger.Warn("log classify action failed", "message_id", res.msg.ID, "error", err) + } + sum.Classified++ + } + + if cov, err := c.store.ClassificationCoverage(ctx); err == nil { + sum.Remaining = cov.Missing() + } + sum.Duration = time.Since(start) + + if err := c.store.LogAction(ctx, "classify_run", ActorAgent, + fmt.Sprintf("mode=%s considered=%d classified=%d failed=%d remaining=%d model=%s rules_version=%d", + mode, sum.Considered, sum.Classified, sum.Failed, sum.Remaining, sum.Model, rulesVersion)); err != nil { + c.logger.Warn("log classify run failed", "error", err) + } + + c.logger.Info("classification run complete", + "considered", sum.Considered, + "classified", sum.Classified, + "failed", sum.Failed, + "remaining", sum.Remaining, + "model", sum.Model, + "duration_ms", sum.Duration.Milliseconds(), + ) + return sum, nil +} + +// classifyOne asks JEV about one message and overlays the matching user rule. +func (c *Classifier) classifyOne(ctx context.Context, msg store.Message, options []CategoryOption, valid map[string]string, rules RuleSet) (store.Classification, *Rule, error) { + result, err := c.client.Classify(ctx, MessageInput{ + From: msg.FromAddr, + To: msg.ToAddr, + Subject: msg.Subject, + Source: msg.SourceAddress, + Body: msg.BodyText, + Received: receivedString(msg), + }, options) + if err != nil { + return store.Classification{}, nil, err + } + + rule := rules.Match(map[string]string{ + "from": msg.FromAddr, + "to": msg.ToAddr, + "subject": msg.Subject, + "source": msg.SourceAddress, + "body": msg.BodyText, + }) + + category := result.Category + importance := result.Importance + action := result.Action + if rule != nil { + if v, ok := rule.Effects["category"]; ok { + if name, found := valid[strings.ToLower(v)]; found { + category = name + } else { + c.logger.Warn("rule names an unknown category; ignoring effect", + "message_id", msg.ID, "rule_line", rule.Line, "category", v) + } + } + if v, ok := rule.Effects["importance"]; ok { + importance = v + } + if v, ok := rule.Effects["action"]; ok { + action = v + } + } + + // The classifier must always store a usable category, importance, and + // action (Phase 3 DoD). JEV can only pick from the supplied options, but a + // broken response or a missing category list must not leave NULLs. + if name, found := valid[strings.ToLower(category)]; found { + category = name + } else { + if category != "" { + c.logger.Warn("classifier returned an unknown category; using fallback", + "message_id", msg.ID, "category", category) + } + category = options[0].Name + } + if !allowedImportance[strings.ToLower(importance)] { + importance = "medium" + } else { + importance = strings.ToLower(importance) + } + if !allowedAction[strings.ToLower(action)] { + action = "keep" + } else { + action = strings.ToLower(action) + } + + // A rule override invalidates the JEV confidence: report the confidence of + // the answer the category actually came from. + confidence := result.CategoryConfidence + if rule != nil { + if _, ok := rule.Effects["category"]; ok { + confidence = 1 + } + } + + return store.Classification{ + MessageID: msg.ID, + Category: category, + Importance: importance, + Action: action, + Confidence: confidence, + Model: result.Model, + }, rule, nil +} + +// pending returns the messages to classify for this run. +func (c *Classifier) pending(ctx context.Context, all bool) ([]store.Message, error) { + if all { + return c.store.AllMessagesForClassification(ctx, c.maxPerRun) + } + return c.store.PendingClassifications(ctx, c.maxPerRun) +} + +func receivedString(msg store.Message) string { + t := msg.DateSent + if t.IsZero() { + t = msg.ReceivedAt + } + if t.IsZero() { + return "" + } + return t.UTC().Format(time.RFC3339) +} diff --git a/internal/classify/classify_test.go b/internal/classify/classify_test.go new file mode 100644 index 0000000..0058f9d --- /dev/null +++ b/internal/classify/classify_test.go @@ -0,0 +1,288 @@ +package classify + +import ( + "context" + "encoding/json" + "fmt" + "log/slog" + "net/http" + "net/http/httptest" + "path/filepath" + "strings" + "testing" + "time" + + "gitea.lab.audasmedia.com.au/sam/email_agent/internal/store" +) + +// stubJEV returns a TypeSafe-shaped response with a fixed category/action while +// recording how many requests it handled. +func stubJEV(t *testing.T, category, importance, action string, calls *int32) *httptest.Server { + t.Helper() + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + *calls++ + var body struct { + Questions map[string]any `json:"questions"` + } + _ = json.NewDecoder(r.Body).Decode(&body) + if len(body.Questions) != 3 { + t.Errorf("JEV stub saw %d questions, want 3", len(body.Questions)) + } + resp := map[string]any{ + "model": "jev-1.13.0", + "answers": map[string]any{ + "category": map[string]any{"type": "choice", "choice": category, "confidence": 0.8}, + "importance": map[string]any{"type": "choice", "choice": importance, "confidence": 0.7}, + "action": map[string]any{"type": "choice", "choice": action, "confidence": 0.6}, + }, + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(resp) + })) +} + +func newTestStore(t *testing.T) *store.Store { + t.Helper() + st, err := store.Open(context.Background(), filepath.Join(t.TempDir(), "email_agent.db")) + if err != nil { + t.Fatalf("store.Open() error = %v", err) + } + t.Cleanup(func() { _ = st.Close() }) + return st +} + +func seedMessages(t *testing.T, st *store.Store, from string, n int) { + t.Helper() + for i := 0; i < n; i++ { + if _, err := st.UpsertMessage(context.Background(), &store.Message{ + GmailID: fmt.Sprintf("%s-%d@test", from, i), + FromAddr: from, + ToAddr: "hello@where-woof.com", + Subject: fmt.Sprintf("message %d", i), + BodyText: "body", + SourceAddress: "hello@where-woof.com", + DateSent: time.Now().UTC(), + }); err != nil { + t.Fatalf("seed message: %v", err) + } + } +} + +func newTestClassifier(t *testing.T, st *store.Store, baseURL string) *Classifier { + t.Helper() + client := NewClient(baseURL, "test-key", "jev-latest", slog.New(slog.DiscardHandler)) + return NewClassifier(client, st, slog.New(slog.DiscardHandler), 2, 50) +} + +func TestClassifierClassifiesEveryPendingMessage(t *testing.T) { + ctx := context.Background() + st := newTestStore(t) + seedMessages(t, st, "someone@example.com", 3) + + var calls int32 + srv := stubJEV(t, "Info", "low", "keep", &calls) + defer srv.Close() + + sum, err := newTestClassifier(t, st, srv.URL).Run(ctx) + if err != nil { + t.Fatalf("Run() error = %v", err) + } + if sum.Considered != 3 || sum.Classified != 3 || sum.Failed != 0 { + t.Fatalf("summary = %+v, want 3 considered/classified, 0 failed", sum) + } + if calls != 3 { + t.Errorf("JEV calls = %d, want 3", calls) + } + + coverage, err := st.ClassificationCoverage(ctx) + if err != nil { + t.Fatalf("ClassificationCoverage() error = %v", err) + } + if coverage.Total != 3 || coverage.Classified != 3 || coverage.Missing() != 0 { + t.Fatalf("coverage = %+v, want 3/3 classified", coverage) + } + + items, err := st.ListMessages(ctx, 10) + if err != nil { + t.Fatalf("ListMessages() error = %v", err) + } + for _, it := range items { + if it.Category == "" || it.Importance == "" || it.Action == "" { + t.Errorf("message %d has a null classification: %+v", it.ID, it) + } + if !it.HasClassification { + t.Errorf("message %d HasClassification = false", it.ID) + } + } +} + +func TestClassifierRuleOverridesJEV(t *testing.T) { + ctx := context.Background() + st := newTestStore(t) + seedMessages(t, st, "billing@stripe.com", 1) + if _, err := st.SaveRules(ctx, `from contains "stripe" => category=Action, importance=high, action=reply`); err != nil { + t.Fatalf("SaveRules() error = %v", err) + } + + var calls int32 + srv := stubJEV(t, "Newsletter", "low", "keep", &calls) + defer srv.Close() + + sum, err := newTestClassifier(t, st, srv.URL).Run(ctx) + if err != nil { + t.Fatalf("Run() error = %v", err) + } + if sum.Classified != 1 { + t.Fatalf("summary = %+v", sum) + } + + items, err := st.ListMessages(ctx, 10) + if err != nil { + t.Fatalf("ListMessages() error = %v", err) + } + got := items[0] + if got.Category != "Action" || got.Importance != "high" || got.Action != "reply" { + t.Errorf("rule override not applied: %+v", got) + } + if got.Confidence != 1 { + t.Errorf("Confidence = %v, want 1 for a rule override", got.Confidence) + } + if calls != 1 { + t.Errorf("JEV calls = %d, want 1 (JEV still consulted)", calls) + } +} + +func TestClassifierIgnoresRuleWithUnknownCategory(t *testing.T) { + ctx := context.Background() + st := newTestStore(t) + seedMessages(t, st, "billing@stripe.com", 1) + if _, err := st.SaveRules(ctx, `from contains "stripe" => category=NoSuchCategory`); err != nil { + t.Fatalf("SaveRules() error = %v", err) + } + + var calls int32 + srv := stubJEV(t, "Newsletter", "medium", "keep", &calls) + defer srv.Close() + + if _, err := newTestClassifier(t, st, srv.URL).Run(ctx); err != nil { + t.Fatalf("Run() error = %v", err) + } + items, err := st.ListMessages(ctx, 10) + if err != nil { + t.Fatalf("ListMessages() error = %v", err) + } + if items[0].Category != "Newsletter" { + t.Errorf("Category = %q, want the JEV answer Newsletter", items[0].Category) + } +} + +func TestClassifierFallsBackWhenJEVCategoryUnknown(t *testing.T) { + ctx := context.Background() + st := newTestStore(t) + seedMessages(t, st, "someone@example.com", 1) + + var calls int32 + srv := stubJEV(t, "CategoryThatDoesNotExist", "medium", "keep", &calls) + defer srv.Close() + + if _, err := newTestClassifier(t, st, srv.URL).Run(ctx); err != nil { + t.Fatalf("Run() error = %v", err) + } + categories, err := st.CategoryNames(ctx) + if err != nil { + t.Fatalf("CategoryNames() error = %v", err) + } + items, err := st.ListMessages(ctx, 10) + if err != nil { + t.Fatalf("ListMessages() error = %v", err) + } + if items[0].Category != categories[0] { + t.Errorf("Category = %q, want the first configured category %q", items[0].Category, categories[0]) + } +} + +func TestClassifierRerunsOnlyPending(t *testing.T) { + ctx := context.Background() + st := newTestStore(t) + seedMessages(t, st, "someone@example.com", 2) + + var calls int32 + srv := stubJEV(t, "Info", "low", "keep", &calls) + defer srv.Close() + + classifier := newTestClassifier(t, st, srv.URL) + if _, err := classifier.Run(ctx); err != nil { + t.Fatalf("first Run() error = %v", err) + } + sum, err := classifier.Run(ctx) + if err != nil { + t.Fatalf("second Run() error = %v", err) + } + if sum.Considered != 0 { + t.Errorf("second run considered %d, want 0 (nothing pending)", sum.Considered) + } + if calls != 2 { + t.Errorf("JEV calls = %d, want 2 (only the first run)", calls) + } +} + +func TestClassifierRunAllReclassifiesEverything(t *testing.T) { + ctx := context.Background() + st := newTestStore(t) + seedMessages(t, st, "someone@example.com", 2) + + var calls int32 + srv := stubJEV(t, "Info", "low", "keep", &calls) + defer srv.Close() + + classifier := newTestClassifier(t, st, srv.URL) + if _, err := classifier.Run(ctx); err != nil { + t.Fatalf("Run() error = %v", err) + } + sum, err := classifier.RunAll(ctx) + if err != nil { + t.Fatalf("RunAll() error = %v", err) + } + if sum.Considered != 2 || sum.Classified != 2 { + t.Fatalf("RunAll() summary = %+v, want 2 considered/classified", sum) + } + if calls != 4 { + t.Errorf("JEV calls = %d, want 4 (2 initial + 2 re-classified)", calls) + } + + var n int + if err := st.DB().QueryRowContext(ctx, `SELECT COUNT(*) FROM classifications`).Scan(&n); err != nil { + t.Fatalf("count classifications: %v", err) + } + if n != 4 { + t.Errorf("classification rows = %d, want 4 (history is kept)", n) + } +} + +func TestClassifierActionLogRecordsRule(t *testing.T) { + ctx := context.Background() + st := newTestStore(t) + seedMessages(t, st, "billing@stripe.com", 1) + if _, err := st.SaveRules(ctx, `from contains "stripe" => category=Action`); err != nil { + t.Fatalf("SaveRules() error = %v", err) + } + + var calls int32 + srv := stubJEV(t, "Newsletter", "low", "keep", &calls) + defer srv.Close() + + if _, err := newTestClassifier(t, st, srv.URL).Run(ctx); err != nil { + t.Fatalf("Run() error = %v", err) + } + + var detail string + if err := st.DB().QueryRowContext(ctx, + `SELECT detail FROM actions_log WHERE action = 'classify' ORDER BY id DESC LIMIT 1`).Scan(&detail); err != nil { + t.Fatalf("read classify log: %v", err) + } + for _, want := range []string{"category=Action", "rule_line=1", "from contains"} { + if !strings.Contains(detail, want) { + t.Errorf("classify log detail %q missing %q", detail, want) + } + } +} diff --git a/internal/classify/jev.go b/internal/classify/jev.go new file mode 100644 index 0000000..22c2b00 --- /dev/null +++ b/internal/classify/jev.go @@ -0,0 +1,306 @@ +// Package classify turns stored messages into a category, an importance, a +// suggested action, and a confidence using the TypeSafe (JEV) HTTP API, then +// applies the user's rules (FR-3, FR-4). +// +// The JEV request/response shape is confirmed against the live docs +// (https://docs.typesafe.ai/api.md) and recorded in docs/phase3-classify.md: +// +// POST https://api.typesafe.ai/v1/systemone +// Authorization: Bearer +// {"state": …, "model": "jev-latest", "questions": {"‹id›": {"type": "choice", …}}} +// → {"model": "jev-1.13.0", "answers": {"‹id›": {"type": "choice", "confidence", "probabilities"}}} +package classify + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "math/rand" + "net/http" + "strings" + "time" +) + +// DefaultBaseURL is TypeSafe's documented API host. JEV_BASE_URL overrides it. +const DefaultBaseURL = "https://api.typesafe.ai" + +// DefaultModel is the flagship System One alias documented at +// https://docs.typesafe.ai/models. +const DefaultModel = "jev-latest" + +// systemOnePath is the documented evaluation endpoint. +const systemOnePath = "/v1/systemone" + +// ErrNotConfigured means JEV_API_KEY is not set. +var ErrNotConfigured = errors.New("JEV_API_KEY is not configured") + +// maxBodyChars bounds how much of a message body is sent to JEV. +const maxBodyChars = 4000 + +// MessageInput is the message text handed to JEV. +type MessageInput struct { + From string + To string + Subject string + Source string + Body string + Received string +} + +// CategoryOption is one Choice option: the live category name and its rubric. +type CategoryOption struct { + Name string + Description string +} + +// Result is one classification decision. Confidence is the Choice answer's own +// confidence, so the UI can show how sure JEV is. +type Result struct { + Category string + CategoryConfidence float64 + Importance string + ImportanceConfidence float64 + Action string + ActionConfidence float64 + Model string +} + +// Client calls the TypeSafe System One endpoint. +type Client struct { + baseURL string + apiKey string + model string + http *http.Client + logger *slog.Logger +} + +// NewClient builds a JEV client. An empty baseURL falls back to DefaultBaseURL, +// an empty model to DefaultModel. logger may be nil. +func NewClient(baseURL, apiKey, model string, logger *slog.Logger) *Client { + baseURL = strings.TrimRight(strings.TrimSpace(baseURL), "/") + if baseURL == "" { + baseURL = DefaultBaseURL + } + if strings.TrimSpace(model) == "" { + model = DefaultModel + } + if logger == nil { + logger = slog.New(slog.DiscardHandler) + } + return &Client{ + baseURL: baseURL, + apiKey: strings.TrimSpace(apiKey), + model: model, + http: &http.Client{Timeout: 60 * time.Second}, + logger: logger, + } +} + +// Configured reports whether an API key is present. +func (c *Client) Configured() bool { return c.apiKey != "" } + +// Model returns the configured model alias. +func (c *Client) Model() string { return c.model } + +// Classify sends one message and returns the category, importance, action, and +// confidences from one JEV request. +func (c *Client) Classify(ctx context.Context, msg MessageInput, categories []CategoryOption) (Result, error) { + if !c.Configured() { + return Result{}, ErrNotConfigured + } + if len(categories) == 0 { + return Result{}, errors.New("no categories configured") + } + + categoryCriteria := make(map[string]any, len(categories)) + for _, opt := range categories { + if strings.TrimSpace(opt.Name) == "" { + continue + } + if strings.TrimSpace(opt.Description) == "" { + categoryCriteria[opt.Name] = nil + } else { + categoryCriteria[opt.Name] = opt.Description + } + } + + body := map[string]any{ + "state": map[string]any{ + "email": map[string]any{ + "from": msg.From, + "to": msg.To, + "source_address": msg.Source, + "source_domain": domainOf(msg.Source), + "subject": msg.Subject, + "received": msg.Received, + "body": truncate(msg.Body, maxBodyChars), + }, + }, + "model": c.model, + "questions": map[string]any{ + "category": map[string]any{ + "type": "choice", + "instructions": "Which category best fits this email for a personal family inbox? Pick the single best match.", + "criteria": categoryCriteria, + }, + "importance": map[string]any{ + "type": "choice", + "instructions": "How important is this email to the recipient?", + "criteria": map[string]any{ + "low": "Can be ignored or removed without loss", + "medium": "Worth a quick look", + "high": "Needs the recipient's attention soon", + }, + }, + "action": map[string]any{ + "type": "choice", + "instructions": "What is the best single recommended action for this email?", + "criteria": map[string]any{ + "delete": "Junk or spam; safe to remove", + "unsubscribe": "Stop future mail from this sender", + "reply": "Send a reply to the sender", + "file": "Keep for reference; move out of the inbox", + "keep": "Leave in the inbox", + }, + }, + }, + } + + resp, err := c.post(ctx, body) + if err != nil { + return Result{}, err + } + + res := Result{Model: resp.Model} + if a, ok := resp.Answers["category"]; ok { + res.Category = a.Choice + res.CategoryConfidence = a.Confidence + } + if a, ok := resp.Answers["importance"]; ok { + res.Importance = a.Choice + res.ImportanceConfidence = a.Confidence + } + if a, ok := resp.Answers["action"]; ok { + res.Action = a.Choice + res.ActionConfidence = a.Confidence + } + if res.Model == "" { + res.Model = c.model + } + return res, nil +} + +// answer is one System One Choice answer. +type answer struct { + Type string `json:"type"` + Choice string `json:"choice"` + Confidence float64 `json:"confidence"` + Probabilities map[string]float64 `json:"probabilities"` + Noul float64 `json:"noul"` +} + +type response struct { + Model string `json:"model"` + Answers map[string]answer `json:"answers"` + Usage struct { + InputTokens int `json:"input_tokens"` + OutputTokens int `json:"output_tokens"` + } `json:"usage"` +} + +// post sends the request with bounded retries for rate limits and outages. +func (c *Client) post(ctx context.Context, body map[string]any) (response, error) { + payload, err := json.Marshal(body) + if err != nil { + return response{}, fmt.Errorf("encode JEV request: %w", err) + } + + const maxAttempts = 4 + var lastErr error + for attempt := 1; attempt <= maxAttempts; attempt++ { + resp, retryable, err := c.do(ctx, payload) + if err == nil { + return resp, nil + } + lastErr = err + if !retryable || attempt == maxAttempts { + break + } + delay := backoff(attempt) + c.logger.Warn("jev request failed; retrying", "attempt", attempt, "delay_ms", delay.Milliseconds(), "error", err) + select { + case <-ctx.Done(): + return response{}, ctx.Err() + case <-time.After(delay): + } + } + return response{}, lastErr +} + +func (c *Client) do(ctx context.Context, payload []byte) (response, bool, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+systemOnePath, bytes.NewReader(payload)) + if err != nil { + return response{}, false, fmt.Errorf("build JEV request: %w", err) + } + req.Header.Set("Authorization", "Bearer "+c.apiKey) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept", "application/json") + + httpResp, err := c.http.Do(req) + if err != nil { + return response{}, true, fmt.Errorf("call JEV: %w", err) + } + defer func() { _ = httpResp.Body.Close() }() + + raw, err := io.ReadAll(io.LimitReader(httpResp.Body, 1<<20)) + if err != nil { + return response{}, true, fmt.Errorf("read JEV response: %w", err) + } + + if httpResp.StatusCode != http.StatusOK { + retryable := httpResp.StatusCode == http.StatusTooManyRequests || + httpResp.StatusCode == http.StatusServiceUnavailable || + httpResp.StatusCode == 529 || + httpResp.StatusCode >= 500 + return response{}, retryable, + fmt.Errorf("JEV returned %d: %s", httpResp.StatusCode, strings.TrimSpace(string(raw))) + } + + var out response + if err := json.Unmarshal(raw, &out); err != nil { + return response{}, false, fmt.Errorf("decode JEV response: %w", err) + } + if len(out.Answers) == 0 { + return response{}, false, fmt.Errorf("JEV response contained no answers") + } + return out, false, nil +} + +// backoff returns the delay before the next attempt, with a little jitter so +// parallel workers do not retry in lock step. +func backoff(attempt int) time.Duration { + base := time.Duration(1<<(attempt-1)) * time.Second + if base > 8*time.Second { + base = 8 * time.Second + } + return base + time.Duration(rand.Int63n(int64(base/2+1))) +} + +func truncate(s string, max int) string { + s = strings.TrimSpace(s) + if len(s) <= max { + return s + } + return s[:max] + "…" +} + +func domainOf(addr string) string { + if i := strings.LastIndex(addr, "@"); i >= 0 && i < len(addr)-1 { + return addr[i+1:] + } + return addr +} diff --git a/internal/classify/jev_test.go b/internal/classify/jev_test.go new file mode 100644 index 0000000..9211ad0 --- /dev/null +++ b/internal/classify/jev_test.go @@ -0,0 +1,154 @@ +package classify + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + "time" +) + +func TestClientClassifyRequestAndResponse(t *testing.T) { + var ( + gotPath string + gotAuth string + gotBody map[string]any + callCount int32 + ) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + atomic.AddInt32(&callCount, 1) + gotPath = r.URL.Path + gotAuth = r.Header.Get("Authorization") + if err := json.NewDecoder(r.Body).Decode(&gotBody); err != nil { + t.Errorf("decode request: %v", err) + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{ + "model": "jev-1.13.0", + "answers": { + "category": {"type":"choice","choice":"Action","confidence":0.81,"probabilities":{"Action":0.81,"Info":0.19}}, + "importance": {"type":"choice","choice":"high","confidence":0.9,"probabilities":{"high":0.9}}, + "action": {"type":"choice","choice":"reply","confidence":0.7,"probabilities":{"reply":0.7}} + }, + "usage": {"input_tokens": 100, "output_tokens": 20} + }`)) + })) + defer srv.Close() + + client := NewClient(srv.URL, "test-key", "jev-latest", nil) + res, err := client.Classify(context.Background(), MessageInput{ + From: "favrolfe@gmail.com", To: "hello@where-woof.com", + Subject: "Quick question", Source: "hello@where-woof.com", + Body: "Can you send me the invoice?", Received: "2026-10-11T09:00:00Z", + }, []CategoryOption{ + {Name: "Action", Description: "Needs Sam to do something"}, + {Name: "Info", Description: "Informational"}, + }) + if err != nil { + t.Fatalf("Classify() error = %v", err) + } + + if gotPath != "/v1/systemone" { + t.Errorf("path = %q, want /v1/systemone", gotPath) + } + if gotAuth != "Bearer test-key" { + t.Errorf("Authorization = %q", gotAuth) + } + if gotBody["model"] != "jev-latest" { + t.Errorf("model = %v, want jev-latest", gotBody["model"]) + } + questions, _ := gotBody["questions"].(map[string]any) + if len(questions) != 3 { + t.Fatalf("questions = %d, want 3", len(questions)) + } + categoryQ, _ := questions["category"].(map[string]any) + criteria, _ := categoryQ["criteria"].(map[string]any) + if len(criteria) != 2 { + t.Errorf("category criteria = %v, want the live category list", criteria) + } + state, _ := gotBody["state"].(map[string]any) + if _, ok := state["email"]; !ok { + t.Errorf("state = %v, want an email object", state) + } + + if res.Category != "Action" || res.Importance != "high" || res.Action != "reply" { + t.Errorf("result = %+v", res) + } + if res.CategoryConfidence != 0.81 { + t.Errorf("CategoryConfidence = %v, want 0.81", res.CategoryConfidence) + } + if res.Model != "jev-1.13.0" { + t.Errorf("Model = %q, want jev-1.13.0", res.Model) + } + if atomic.LoadInt32(&callCount) != 1 { + t.Errorf("call count = %d, want 1", callCount) + } +} + +func TestClientRetriesRateLimit(t *testing.T) { + var attempts int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if atomic.AddInt32(&attempts, 1) == 1 { + w.WriteHeader(http.StatusTooManyRequests) + _, _ = w.Write([]byte(`{"error":"rate limited"}`)) + return + } + _, _ = w.Write([]byte(`{"model":"jev-1.13.0","answers":{ + "category":{"type":"choice","choice":"Info","confidence":1}, + "importance":{"type":"choice","choice":"low","confidence":1}, + "action":{"type":"choice","choice":"keep","confidence":1}}}`)) + })) + defer srv.Close() + + client := NewClient(srv.URL, "k", "", nil) + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + + res, err := client.Classify(ctx, MessageInput{Subject: "hi"}, []CategoryOption{{Name: "Info"}}) + if err != nil { + t.Fatalf("Classify() error = %v", err) + } + if res.Category != "Info" { + t.Errorf("Category = %q, want Info", res.Category) + } + if got := atomic.LoadInt32(&attempts); got != 2 { + t.Errorf("attempts = %d, want 2", got) + } +} + +func TestClientRequiresAPIKey(t *testing.T) { + client := NewClient("http://example.invalid", "", "", nil) + if client.Configured() { + t.Fatal("Configured() = true without an API key") + } + if _, err := client.Classify(context.Background(), MessageInput{}, []CategoryOption{{Name: "Info"}}); err != ErrNotConfigured { + t.Fatalf("err = %v, want ErrNotConfigured", err) + } +} + +func TestClientDefaultsBaseURL(t *testing.T) { + client := NewClient("", "k", "", nil) + if client.baseURL != DefaultBaseURL { + t.Errorf("baseURL = %q, want %q", client.baseURL, DefaultBaseURL) + } + if client.Model() != DefaultModel { + t.Errorf("model = %q, want %q", client.Model(), DefaultModel) + } +} + +func TestClientHTTPErrorSurfacesBody(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusUnauthorized) + _, _ = w.Write([]byte(`{"error":"invalid api key"}`)) + })) + defer srv.Close() + + client := NewClient(srv.URL, "bad", "", nil) + _, err := client.Classify(context.Background(), MessageInput{}, []CategoryOption{{Name: "Info"}}) + if err == nil || !strings.Contains(err.Error(), "invalid api key") { + t.Fatalf("err = %v, want the 401 body", err) + } +} diff --git a/internal/classify/rules.go b/internal/classify/rules.go new file mode 100644 index 0000000..e1992a1 --- /dev/null +++ b/internal/classify/rules.go @@ -0,0 +1,238 @@ +// User rules (FR-4). A rule overrides the JEV classification when its condition +// matches the message. +// +// Syntax, one rule per line: +// +// => +// +// condition: ` ""`, joined with " and ". +// +// fields: from, to, subject, source, body +// operators: contains, equals, starts_with, ends_with +// +// A single `*` matches every message (use it as the last line for a default). +// effects: comma-separated `category=`, `importance=`, +// `action=`. +// +// The first matching rule wins and overrides only the fields it names. The rules +// editor that writes this text lands in Phase 6; the classifier already reads it. +package classify + +import ( + "fmt" + "strings" +) + +// Rule is one parsed line. +type Rule struct { + Line int + Raw string + MatchAll bool + Conditions []Condition + Effects map[string]string +} + +// Condition is one ` ""` term. +type Condition struct { + Field string + Op string + Value string +} + +// ParseError records a line that could not be parsed. The rest of the document +// still loads; the error is surfaced in logs so a typo is visible. +type ParseError struct { + Line int + Text string + Reason string +} + +func (e ParseError) Error() string { + return fmt.Sprintf("rules line %d: %s (%q)", e.Line, e.Reason, strings.TrimSpace(e.Text)) +} + +// RuleSet is a parsed rules document. +type RuleSet struct { + Rules []Rule + Errors []ParseError +} + +// effectFields are the classification fields a rule may override. +var effectFields = map[string]bool{ + "category": true, + "importance": true, + "action": true, +} + +var allowedImportance = map[string]bool{"low": true, "medium": true, "high": true} + +var allowedAction = map[string]bool{ + "delete": true, "unsubscribe": true, "reply": true, "file": true, "keep": true, +} + +// ParseRules parses a rules document. Malformed lines are collected in Errors and +// skipped; parsing never fails outright, so one typo cannot disable the rules. +func ParseRules(text string) RuleSet { + var set RuleSet + for i, raw := range strings.Split(text, "\n") { + line := strings.TrimSpace(strings.TrimSuffix(raw, "\r")) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + rule, err := parseRule(line) + rule.Line = i + 1 + rule.Raw = line + if err != nil { + set.Errors = append(set.Errors, ParseError{Line: i + 1, Text: raw, Reason: err.Error()}) + continue + } + set.Rules = append(set.Rules, rule) + } + return set +} + +func parseRule(line string) (Rule, error) { + rule := Rule{Effects: map[string]string{}} + left, right, ok := strings.Cut(line, "=>") + if !ok { + return rule, fmt.Errorf("missing =>") + } + left, right = strings.TrimSpace(left), strings.TrimSpace(right) + if left == "" || right == "" { + return rule, fmt.Errorf("empty condition or effect") + } + + if left == "*" { + rule.MatchAll = true + } else { + for _, part := range strings.Split(left, " and ") { + cond, err := parseCondition(strings.TrimSpace(part)) + if err != nil { + return rule, err + } + rule.Conditions = append(rule.Conditions, cond) + } + } + + for _, part := range strings.Split(right, ",") { + part = strings.TrimSpace(part) + if part == "" { + continue + } + field, value, err := parseEffect(part) + if err != nil { + return rule, err + } + rule.Effects[field] = value + } + if len(rule.Effects) == 0 { + return rule, fmt.Errorf("no effects") + } + return rule, nil +} + +// parseEffect accepts `field=value` and `field: value` to be forgiving of typing. +func parseEffect(part string) (string, string, error) { + field, value, ok := strings.Cut(part, "=") + if !ok { + field, value, ok = strings.Cut(part, ":") + } + if !ok { + return "", "", fmt.Errorf("effect %q is not field=value", part) + } + field = strings.ToLower(strings.TrimSpace(field)) + value = unquote(strings.TrimSpace(value)) + if !effectFields[field] { + return "", "", fmt.Errorf("unknown effect field %q", field) + } + if value == "" { + return "", "", fmt.Errorf("effect %q has no value", field) + } + switch field { + case "importance": + value = strings.ToLower(value) + if !allowedImportance[value] { + return "", "", fmt.Errorf("importance %q must be low, medium, or high", value) + } + case "action": + value = strings.ToLower(value) + if !allowedAction[value] { + return "", "", fmt.Errorf("action %q must be delete, unsubscribe, reply, file, or keep", value) + } + } + return field, value, nil +} + +func parseCondition(part string) (Condition, error) { + fields := strings.Fields(part) + if len(fields) < 3 { + return Condition{}, fmt.Errorf("condition %q needs a field, an operator, and a value", part) + } + field := strings.ToLower(fields[0]) + switch field { + case "from", "to", "subject", "source", "body": + default: + return Condition{}, fmt.Errorf("unknown condition field %q", field) + } + op := strings.ToLower(fields[1]) + switch op { + case "contains", "equals", "starts_with", "ends_with": + default: + return Condition{}, fmt.Errorf("unknown operator %q", op) + } + value := unquote(strings.TrimSpace(strings.Join(fields[2:], " "))) + if value == "" { + return Condition{}, fmt.Errorf("condition %q has no value", part) + } + return Condition{Field: field, Op: op, Value: value}, nil +} + +func unquote(v string) string { + if len(v) >= 2 { + if (v[0] == '"' && v[len(v)-1] == '"') || (v[0] == '\'' && v[len(v)-1] == '\'') { + return v[1 : len(v)-1] + } + } + return v +} + +// Match returns the effects of the first matching rule. It reports nil when no +// rule matches. Field lookups are case-insensitive; values compare +// case-insensitively too. +func (rs RuleSet) Match(fields map[string]string) *Rule { + normalized := make(map[string]string, len(fields)) + for k, v := range fields { + normalized[strings.ToLower(k)] = strings.ToLower(v) + } + for i := range rs.Rules { + r := &rs.Rules[i] + if r.MatchAll || allMatch(r.Conditions, normalized) { + return r + } + } + return nil +} + +func allMatch(conds []Condition, fields map[string]string) bool { + for _, c := range conds { + if !matches(c, fields[c.Field]) { + return false + } + } + return true +} + +func matches(c Condition, fieldValue string) bool { + want := strings.ToLower(c.Value) + switch c.Op { + case "contains": + return strings.Contains(fieldValue, want) + case "equals": + return fieldValue == want + case "starts_with": + return strings.HasPrefix(fieldValue, want) + case "ends_with": + return strings.HasSuffix(fieldValue, want) + } + return false +} diff --git a/internal/classify/rules_test.go b/internal/classify/rules_test.go new file mode 100644 index 0000000..0c45f44 --- /dev/null +++ b/internal/classify/rules_test.go @@ -0,0 +1,119 @@ +package classify + +import "testing" + +func TestParseRulesSkipsCommentsAndBlanks(t *testing.T) { + set := ParseRules(` +# a comment + # an indented comment + +from contains "amazon.com" => category=Newsletter, action=unsubscribe +`) + if len(set.Errors) != 0 { + t.Fatalf("Errors = %v, want none", set.Errors) + } + if len(set.Rules) != 1 { + t.Fatalf("Rules len = %d, want 1", len(set.Rules)) + } + r := set.Rules[0] + if r.Line != 5 { + t.Errorf("Line = %d, want 5", r.Line) + } + if r.Effects["category"] != "Newsletter" || r.Effects["action"] != "unsubscribe" { + t.Errorf("Effects = %v", r.Effects) + } +} + +func TestParseRulesMatchAll(t *testing.T) { + set := ParseRules(`* => category=Info`) + if len(set.Rules) != 1 || !set.Rules[0].MatchAll { + t.Fatalf("expected a match-all rule, got %+v (errors %v)", set.Rules, set.Errors) + } + if got := set.Match(map[string]string{"subject": "anything"}); got == nil { + t.Fatal("match-all rule did not match") + } +} + +func TestParseRulesReportsBadLines(t *testing.T) { + set := ParseRules(` +from => category=Info +from contains "x" => importance=urgent +from contains "x" => bogus=1 +missing the arrow +`) + if len(set.Rules) != 0 { + t.Fatalf("Rules = %+v, want none", set.Rules) + } + if len(set.Errors) != 4 { + t.Fatalf("Errors len = %d, want 4: %v", len(set.Errors), set.Errors) + } +} + +func TestRuleMatchAndPrecedence(t *testing.T) { + set := ParseRules(` +from ends_with "@stripe.com" and subject contains "invoice" => category=Action, importance=high +from contains "stripe" => category=Newsletter +* => category=Info +`) + if len(set.Errors) != 0 { + t.Fatalf("unexpected errors: %v", set.Errors) + } + + // First matching rule wins. + got := set.Match(map[string]string{ + "from": "Billing@Stripe.com", + "subject": "Your invoice is ready", + }) + if got == nil || got.Effects["category"] != "Action" { + t.Fatalf("first rule did not win: %+v", got) + } + + // The second rule matches on its own terms. + got = set.Match(map[string]string{"from": "stripe-press@example.com"}) + if got == nil || got.Effects["category"] != "Newsletter" { + t.Fatalf("second rule did not match: %+v", got) + } + + // The catch-all is the fallback. + got = set.Match(map[string]string{"from": "someone@example.com"}) + if got == nil || got.Effects["category"] != "Info" { + t.Fatalf("catch-all did not match: %+v", got) + } +} + +func TestRuleOperators(t *testing.T) { + cases := []struct { + name string + rule string + fields map[string]string + wantCat string + wantNone bool + }{ + {"equals", `from equals "a@b.com" => category=Action`, map[string]string{"from": "A@B.com"}, "Action", false}, + {"starts_with", `subject starts_with "Hello" => category=Info`, map[string]string{"subject": "hello there"}, "Info", false}, + {"ends_with", `source ends_with "where-woof.com" => category=Newsletter`, map[string]string{"source": "hello@where-woof.com"}, "Newsletter", false}, + {"body", `body contains "unsubscribe here" => category=Newsletter`, map[string]string{"body": "Click UNSUBSCRIBE HERE to stop"}, "Newsletter", false}, + {"no match", `from contains "zzz" => category=Action`, map[string]string{"from": "a@b.com"}, "", true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + set := ParseRules(tc.rule) + if len(set.Errors) != 0 { + t.Fatalf("parse errors: %v", set.Errors) + } + got := set.Match(tc.fields) + if tc.wantNone { + if got != nil { + t.Fatalf("expected no match, got %+v", got) + } + return + } + if got == nil { + t.Fatal("expected a match, got none") + } + if got.Effects["category"] != tc.wantCat { + t.Errorf("category = %q, want %q", got.Effects["category"], tc.wantCat) + } + }) + } +} diff --git a/internal/config/config.go b/internal/config/config.go index 7948938..aea35aa 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -28,6 +28,16 @@ const ( DefaultIMAPPort = 993 DefaultIMAPLookbackDays = 14 DefaultIMAPMaxMessages = 100 + + // JEV (TypeSafe System One) defaults. The base URL is fixed by the TypeSafe + // docs (https://docs.typesafe.ai/api.md); JEV_BASE_URL overrides it. + DefaultJEVBaseURL = "https://api.typesafe.ai" + DefaultJEVModel = "jev-latest" + + // Classifier defaults (FR-3). Concurrency bounds in-flight JEV requests; + // max messages bounds one run so a scheduled scan stays cheap. + DefaultClassifyConcurrency = 4 + DefaultClassifyMaxMessages = 200 ) // Config is the fully resolved runtime configuration. @@ -66,6 +76,12 @@ type Config struct { JEVBaseURL string // JEVAPIKey authenticates to JEV. Secret. JEVAPIKey string + // JEVModel is the System One model alias (default jev-latest). + JEVModel string + // ClassifyConcurrency bounds in-flight JEV requests per run. + ClassifyConcurrency int + // ClassifyMaxMessages caps how many messages one classify run processes. + ClassifyMaxMessages int // OmniRouteBaseURL is the base URL of the OmniRoute gateway. OmniRouteBaseURL string @@ -99,8 +115,9 @@ func Load() (*Config, error) { Password: os.Getenv("EMAIL_AGENT_PASSWORD"), LogLevel: strings.ToLower(getenvDefault("EMAIL_AGENT_LOG_LEVEL", "info")), PublicBaseURL: strings.TrimRight(getenvDefault("EMAIL_AGENT_PUBLIC_BASE_URL", "http://localhost:8790"), "/"), - JEVBaseURL: strings.TrimRight(os.Getenv("JEV_BASE_URL"), "/"), + JEVBaseURL: strings.TrimRight(getenvDefault("JEV_BASE_URL", DefaultJEVBaseURL), "/"), JEVAPIKey: os.Getenv("JEV_API_KEY"), + JEVModel: getenvDefault("JEV_MODEL", DefaultJEVModel), OmniRouteBaseURL: strings.TrimRight(os.Getenv("OMNIROUTE_BASE_URL"), "/"), OmniRouteAPIKey: os.Getenv("OMNIROUTE_API_KEY"), GmailUser: os.Getenv("GMAIL_USER"), @@ -116,6 +133,8 @@ func Load() (*Config, error) { c.IMAPPort = imapPort c.IMAPLookbackDays = getenvDefaultInt("EMAIL_AGENT_IMAP_LOOKBACK_DAYS", DefaultIMAPLookbackDays) c.IMAPMaxMessages = getenvDefaultInt("EMAIL_AGENT_IMAP_MAX_MESSAGES", DefaultIMAPMaxMessages) + c.ClassifyConcurrency = getenvDefaultInt("EMAIL_AGENT_CLASSIFY_CONCURRENCY", DefaultClassifyConcurrency) + c.ClassifyMaxMessages = getenvDefaultInt("EMAIL_AGENT_CLASSIFY_MAX_MESSAGES", DefaultClassifyMaxMessages) port, err := resolvePort() if err != nil { @@ -219,6 +238,10 @@ func (c *Config) GmailConfigured() bool { return strings.TrimSpace(c.GmailUser) != "" && strings.TrimSpace(c.GmailAppPassword) != "" } +// JEVConfigured reports whether JEV classification can be attempted. The +// service still starts without a key; only classification is unavailable. +func (c *Config) JEVConfigured() bool { return strings.TrimSpace(c.JEVAPIKey) != "" } + // IsProduction reports whether the service runs in the production environment. func (c *Config) IsProduction() bool { return c.Env == "production" } diff --git a/internal/server/auth.go b/internal/server/auth.go index 3bcaf7c..1de91b5 100644 --- a/internal/server/auth.go +++ b/internal/server/auth.go @@ -9,6 +9,7 @@ import ( "time" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/session" + "gitea.lab.audasmedia.com.au/sam/email_agent/internal/store" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/version" ) @@ -226,36 +227,63 @@ type dashboardData struct { Folders []navItem } +// categoryCounts returns per-category message counts keyed by the canonical +// category name, plus the unclassified bucket. +func categoryCounts(items []store.MessageListItem, categories []store.Category) map[string]int { + canonical := make(map[string]string, len(categories)) + for _, c := range categories { + canonical[strings.ToLower(c.Name)] = c.Name + } + counts := map[string]int{} + for _, it := range items { + raw := strings.TrimSpace(it.Category) + if raw == "" { + counts[unclassifiedCategory]++ + continue + } + if name, ok := canonical[strings.ToLower(raw)]; ok { + counts[name]++ + } else { + counts[raw]++ + } + } + return counts +} + // handleDashboard renders the inbox shell with real category counts. func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) { sess := sessionFrom(r) - // FR-5: counts come from the stored messages. Phase 2 has no classifier - // yet, so every message lands under Unclassified (AGENTS.md §13.2). + // FR-5: counts come from the stored messages, grouped by the live category + // list (FR-14). items, err := s.store.ListMessages(r.Context(), 200) if err != nil { s.logger.Error("load dashboard counts", "error", err) items = nil } - counts := categoryCounts(items) + categories, err := s.store.ListCategories(r.Context()) + if err != nil { + s.logger.Error("load dashboard categories", "error", err) + } + counts := categoryCounts(items, categories) + + catNav := make([]navItem, 0, len(categories)+1) + for _, c := range categories { + catNav = append(catNav, navItem{Name: c.Name, Color: "var(--primary-soft, var(--surface-2))", Count: counts[c.Name]}) + } + catNav = append(catNav, navItem{ + Name: unclassifiedCategory, + Color: "var(--ink-faint)", + Count: counts[unclassifiedCategory], + }) data := dashboardData{ - Greeting: partOfDay(time.Now()), - User: sess.User, - CSRFToken: s.csrfToken(w, r), - Version: version.Version, - Total: len(items), - // Placeholders until FR-14 (editable categories) and FR-9 (folders) - // land. Colours are the design system's decorative sticker palette. - Categories: []navItem{ - {Name: "Action", Color: "var(--primary)", Count: counts["action"]}, - {Name: "Important", Color: "var(--sticker-orange)", Count: counts["important"]}, - {Name: "Info", Color: "var(--sticker-sky)", Count: counts["info"]}, - {Name: "Spam", Color: "var(--sticker-red)", Count: counts["spam"]}, - {Name: "Newsletter", Color: "var(--sticker-purple)", Count: counts["newsletter"]}, - {Name: "Auto-reply", Color: "var(--sticker-teal)", Count: counts["auto-reply"]}, - {Name: "Unclassified", Color: "var(--ink-faint)", Count: counts[unclassifiedCategory]}, - }, + Greeting: partOfDay(time.Now()), + User: sess.User, + CSRFToken: s.csrfToken(w, r), + Version: version.Version, + Total: len(items), + Categories: catNav, Folders: []navItem{ {Name: "Replied-to", Color: "var(--sticker-green)"}, {Name: "Delete", Color: "var(--sticker-red)"}, diff --git a/internal/server/categories.go b/internal/server/categories.go new file mode 100644 index 0000000..9b3d37b --- /dev/null +++ b/internal/server/categories.go @@ -0,0 +1,151 @@ +// Category management UI and handlers (FR-14). +// +// The category list is editable: add, rename, and remove. JEV uses whatever list +// is current at classification time, and renaming rewrites existing +// classifications so the triage list stays consistent. +package server + +import ( + "errors" + "fmt" + "net/http" + "net/url" + "strconv" + + "gitea.lab.audasmedia.com.au/sam/email_agent/internal/store" +) + +type categoriesData struct { + Categories []store.Category + CSRFToken string + Note string + Error string + Total int + Classified int +} + +// handleCategories renders the category editor the inbox nav links to. +func (s *Server) handleCategories(w http.ResponseWriter, r *http.Request) { + s.renderCategories(w, r, http.StatusOK, r.URL.Query().Get("note"), r.URL.Query().Get("error")) +} + +func (s *Server) renderCategories(w http.ResponseWriter, r *http.Request, code int, note, errMsg string) { + categories, err := s.store.ListCategories(r.Context()) + if err != nil { + s.logger.Error("list categories", "error", err) + http.Error(w, "could not load categories", http.StatusInternalServerError) + return + } + coverage, err := s.store.ClassificationCoverage(r.Context()) + if err != nil { + s.logger.Error("classification coverage", "error", err) + } + + data := categoriesData{ + Categories: categories, + CSRFToken: s.csrfToken(w, r), + Note: note, + Error: errMsg, + Total: coverage.Total, + Classified: coverage.Classified, + } + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.WriteHeader(code) + if err := s.tmpl.ExecuteTemplate(w, "categories.html", data); err != nil { + s.logger.Error("render categories", "error", err) + } +} + +// handleCategoryAdd creates a category (FR-14). +func (s *Server) handleCategoryAdd(w http.ResponseWriter, r *http.Request) { + if !s.csrfOK(r) { + http.Error(w, "invalid CSRF token", http.StatusForbidden) + return + } + if err := r.ParseForm(); err != nil { + s.redirectCategories(w, r, "", "Could not read the form.") + return + } + name, description := r.FormValue("name"), r.FormValue("description") + cat, err := s.store.AddCategory(r.Context(), name, description) + switch { + case err == nil: + s.logger.Info("category added", "name", cat.Name) + _ = s.store.LogAction(r.Context(), "category_add", "Sam", "name="+cat.Name) + s.redirectCategories(w, r, fmt.Sprintf("Added category %q.", cat.Name), "") + case errors.Is(err, store.ErrCategoryExists): + s.redirectCategories(w, r, "", fmt.Sprintf("A category named %q already exists.", name)) + default: + s.redirectCategories(w, r, "", err.Error()) + } +} + +// handleCategoryRename changes a category's name and updates existing +// classifications (FR-14). +func (s *Server) handleCategoryRename(w http.ResponseWriter, r *http.Request) { + if !s.csrfOK(r) { + http.Error(w, "invalid CSRF token", http.StatusForbidden) + return + } + if err := r.ParseForm(); err != nil { + s.redirectCategories(w, r, "", "Could not read the form.") + return + } + id, err := strconv.ParseInt(r.FormValue("id"), 10, 64) + if err != nil { + s.redirectCategories(w, r, "", "Unknown category.") + return + } + name := r.FormValue("name") + switch err := s.store.RenameCategory(r.Context(), id, name); { + case err == nil: + s.logger.Info("category renamed", "id", id, "name", name) + _ = s.store.LogAction(r.Context(), "category_rename", "Sam", + fmt.Sprintf("id=%d name=%s", id, name)) + s.redirectCategories(w, r, fmt.Sprintf("Renamed category to %q.", name), "") + case errors.Is(err, store.ErrCategoryExists): + s.redirectCategories(w, r, "", fmt.Sprintf("A category named %q already exists.", name)) + default: + s.redirectCategories(w, r, "", err.Error()) + } +} + +// handleCategoryRemove deletes a category and moves its messages to the first +// remaining category (FR-14). +func (s *Server) handleCategoryRemove(w http.ResponseWriter, r *http.Request) { + if !s.csrfOK(r) { + http.Error(w, "invalid CSRF token", http.StatusForbidden) + return + } + if err := r.ParseForm(); err != nil { + s.redirectCategories(w, r, "", "Could not read the form.") + return + } + id, err := strconv.ParseInt(r.FormValue("id"), 10, 64) + if err != nil { + s.redirectCategories(w, r, "", "Unknown category.") + return + } + switch err := s.store.RemoveCategory(r.Context(), id); { + case err == nil: + s.logger.Info("category removed", "id", id) + _ = s.store.LogAction(r.Context(), "category_remove", "Sam", fmt.Sprintf("id=%d", id)) + s.redirectCategories(w, r, "Removed category; its messages moved to the first remaining category.", "") + case errors.Is(err, store.ErrLastCategory): + s.redirectCategories(w, r, "", "At least one category is required.") + default: + s.redirectCategories(w, r, "", err.Error()) + } +} + +// redirectCategories is POST/redirect/GET so a refresh does not resubmit. +func (s *Server) redirectCategories(w http.ResponseWriter, r *http.Request, note, errMsg string) { + q := url.Values{} + if note != "" { + q.Set("note", note) + } + if errMsg != "" { + q.Set("error", errMsg) + } + http.Redirect(w, r, "/categories?"+q.Encode(), http.StatusSeeOther) +} diff --git a/internal/server/categories_test.go b/internal/server/categories_test.go new file mode 100644 index 0000000..4e253a1 --- /dev/null +++ b/internal/server/categories_test.go @@ -0,0 +1,213 @@ +package server + +import ( + "context" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "strings" + "testing" + "time" + + "gitea.lab.audasmedia.com.au/sam/email_agent/internal/session" + "gitea.lab.audasmedia.com.au/sam/email_agent/internal/store" +) + +// login signs in and returns the session cookie plus the CSRF cookie/token. +func login(t *testing.T, srv *Server) (*http.Cookie, *http.Cookie, string) { + t.Helper() + h := srv.Handler() + + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/login", nil)) + csrf := csrfFromBody(t, rec.Body.String()) + csrfCookie := findCookie(t, rec.Result().Cookies(), csrfCookieName) + + rec = httptest.NewRecorder() + h.ServeHTTP(rec, loginRequest(csrf, testPassword, csrfCookie)) + sessionCookie := findCookie(t, rec.Result().Cookies(), session.CookieName) + if sessionCookie == nil { + t.Fatal("login did not set the session cookie") + } + return sessionCookie, csrfCookie, csrf +} + +func postForm(t *testing.T, srv *Server, path string, session, csrfCookie *http.Cookie, form url.Values) *httptest.ResponseRecorder { + t.Helper() + req := httptest.NewRequest(http.MethodPost, path, strings.NewReader(form.Encode())) + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + req.AddCookie(session) + if csrfCookie != nil { + req.AddCookie(csrfCookie) + } + rec := httptest.NewRecorder() + srv.Handler().ServeHTTP(rec, req) + return rec +} + +func getWithSession(t *testing.T, srv *Server, path string, session *http.Cookie) *httptest.ResponseRecorder { + t.Helper() + req := httptest.NewRequest(http.MethodGet, path, nil) + req.AddCookie(session) + rec := httptest.NewRecorder() + srv.Handler().ServeHTTP(rec, req) + return rec +} + +func TestCategoriesPageRenders(t *testing.T) { + srv := newTestServer(t) + session, _, _ := login(t, srv) + + rec := getWithSession(t, srv, "/categories", session) + if rec.Code != http.StatusOK { + t.Fatalf("status = %d, want 200", rec.Code) + } + body := rec.Body.String() + for _, want := range []string{"Categories", "Action", "Spam", "Add category", "Back to inbox"} { + if !strings.Contains(body, want) { + t.Errorf("categories page missing %q", want) + } + } +} + +func TestCategoryAddRenameRemove(t *testing.T) { + srv := newTestServer(t) + session, csrfCookie, csrf := login(t, srv) + + // Add (FR-14). + rec := postForm(t, srv, "/categories/add", session, csrfCookie, url.Values{ + "csrf": {csrf}, "name": {"Finance"}, "description": {"Bills and payments"}, + }) + if rec.Code != http.StatusSeeOther { + t.Fatalf("add status = %d, want 303 (body %s)", rec.Code, rec.Body.String()) + } + if rec := getWithSession(t, srv, "/categories", session); !strings.Contains(rec.Body.String(), "Finance") { + t.Fatal("added category is not rendered") + } + + id := categoryID(t, srv, "Finance") + + // Rename (FR-14). + rec = postForm(t, srv, "/categories/rename", session, csrfCookie, url.Values{ + "csrf": {csrf}, "id": {itoa(id)}, "name": {"Money"}, + }) + if rec.Code != http.StatusSeeOther { + t.Fatalf("rename status = %d, want 303", rec.Code) + } + body := getWithSession(t, srv, "/categories", session).Body.String() + if !strings.Contains(body, "Money") || strings.Contains(body, ">Finance<") { + t.Fatalf("rename not reflected; body contains Finance=%v Money=%v", + strings.Contains(body, "Finance"), strings.Contains(body, "Money")) + } + + // Remove (FR-14). + id = categoryID(t, srv, "Money") + rec = postForm(t, srv, "/categories/remove", session, csrfCookie, url.Values{ + "csrf": {csrf}, "id": {itoa(id)}, + }) + if rec.Code != http.StatusSeeOther { + t.Fatalf("remove status = %d, want 303", rec.Code) + } + if rec := getWithSession(t, srv, "/categories", session); strings.Contains(rec.Body.String(), "Money") { + t.Fatal("removed category is still rendered") + } +} + +func TestCategoryMutationsRequireCSRF(t *testing.T) { + srv := newTestServer(t) + session, _, _ := login(t, srv) + + rec := postForm(t, srv, "/categories/add", session, nil, url.Values{"name": {"Nope"}}) + if rec.Code != http.StatusForbidden { + t.Fatalf("add without CSRF status = %d, want 403", rec.Code) + } +} + +func TestClassifyAnonymousRedirects(t *testing.T) { + srv := newTestServer(t) + for _, path := range []string{"/actions/classify", "/actions/reclassify"} { + req := httptest.NewRequest(http.MethodPost, path, nil) + rec := httptest.NewRecorder() + srv.Handler().ServeHTTP(rec, req) + if rec.Code != http.StatusSeeOther { + t.Errorf("%s status = %d, want 303 (redirect to login)", path, rec.Code) + } + } +} + +func TestClassifyWithoutJEVKeyReportsNotConfigured(t *testing.T) { + srv := newTestServer(t) + session, csrfCookie, csrf := login(t, srv) + + rec := postForm(t, srv, "/actions/classify", session, csrfCookie, url.Values{"csrf": {csrf}}) + if rec.Code != http.StatusOK { + t.Fatalf("status = %d, want 200", rec.Code) + } + if !strings.Contains(rec.Body.String(), "JEV_API_KEY") { + t.Errorf("expected a not-configured note; body: %s", rec.Body.String()) + } +} + +func TestDashboardLinksToCategoriesAndShowsClassify(t *testing.T) { + srv := newTestServer(t) + session, _, _ := login(t, srv) + + rec := getWithSession(t, srv, "/", session) + if rec.Code != http.StatusOK { + t.Fatalf("status = %d, want 200", rec.Code) + } + body := rec.Body.String() + for _, want := range []string{`href="/categories"`, "Classify", "Re-classify all", "Action", "Auto-reply"} { + if !strings.Contains(body, want) { + t.Errorf("dashboard missing %q", want) + } + } +} + +func TestInboxShowsClassificationConfidence(t *testing.T) { + srv := newTestServer(t) + ctx := context.Background() + + msgID, err := srv.store.UpsertMessage(ctx, &store.Message{ + GmailID: "cls-1", FromAddr: "a@b.com", Subject: "Classified mail", + SourceAddress: "hello@where-woof.com", DateSent: time.Now().UTC(), + }) + if err != nil { + t.Fatalf("seed message: %v", err) + } + if _, err := srv.store.SaveClassification(ctx, store.Classification{ + MessageID: msgID, Category: "Action", Importance: "high", Action: "reply", + Confidence: 0.82, Model: "jev-1.13.0", + }); err != nil { + t.Fatalf("seed classification: %v", err) + } + + session, _, _ := login(t, srv) + rec := getWithSession(t, srv, "/partials/inbox", session) + body := rec.Body.String() + for _, want := range []string{"Classified mail", "Action", "reply", "82%"} { + if !strings.Contains(body, want) { + t.Errorf("inbox partial missing %q", want) + } + } +} + +func categoryID(t *testing.T, srv *Server, name string) int64 { + t.Helper() + cats, err := srv.store.ListCategories(context.Background()) + if err != nil { + t.Fatalf("ListCategories() error = %v", err) + } + for _, c := range cats { + if c.Name == name { + return c.ID + } + } + t.Fatalf("category %q not found", name) + return 0 +} + +func itoa(v int64) string { + return strconv.FormatInt(v, 10) +} diff --git a/internal/server/inbox.go b/internal/server/inbox.go index 56c1e26..1a20bac 100644 --- a/internal/server/inbox.go +++ b/internal/server/inbox.go @@ -1,9 +1,8 @@ -// Inbox rendering and the manual mail scan. +// Inbox rendering, the manual mail scan, and the classify action. // -// Phase 2 lists real mail from the database. Classification is Phase 3, so -// messages without a classification are shown under "Unclassified" with a -// temporary "Review" recommended action; the grouping and action plumbing is -// already category-aware. +// Phase 3 groups messages by the live category list (FR-5) and shows the JEV +// recommended action plus confidence on each row. Messages without a +// classification fall under "Unclassified". package server import ( @@ -15,26 +14,13 @@ import ( "strings" "time" + "gitea.lab.audasmedia.com.au/sam/email_agent/internal/classify" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/mailfetch" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/store" ) -// unclassifiedCategory is the fallback bucket before Phase 3 adds JEV -// categories (AGENTS.md §13.2). -const unclassifiedCategory = "unclassified" - -// categoryOrder is the display order for the built-in categories. -var categoryOrder = []string{"action", "important", "info", "spam", "newsletter", "auto-reply"} - -var categoryDisplay = map[string]string{ - "action": "Action", - "important": "Important", - "info": "Info", - "spam": "Spam", - "newsletter": "Newsletter", - "auto-reply": "Auto-reply", - "unclassified": "Unclassified", -} +// unclassifiedCategory is the bucket for messages JEV has not classified yet. +const unclassifiedCategory = "Unclassified" type messageRow struct { ID int64 @@ -42,6 +28,8 @@ type messageRow struct { Subject string SourceAddr string Action string + Confidence int + Classified bool When string } @@ -57,21 +45,42 @@ type inboxData struct { Error string } -// loadInbox reads messages and groups them for rendering. +// loadInbox reads messages and groups them by the live category list. func (s *Server) loadInbox(ctx context.Context) (inboxData, error) { items, err := s.store.ListMessages(ctx, 200) if err != nil { return inboxData{}, err } - return inboxData{Groups: groupMessages(items), Total: len(items)}, nil + categories, err := s.store.ListCategories(ctx) + if err != nil { + return inboxData{}, err + } + return inboxData{Groups: groupMessages(items, categories), Total: len(items)}, nil } -// groupMessages buckets messages by category. Unknown (custom) categories get -// their own group; messages with no classification fall into "Unclassified". -func groupMessages(items []store.MessageListItem) []messageGroup { +// groupMessages buckets messages by category, ordering groups the way the +// category list is ordered in the UI. Unknown categories (for example a +// classification written before a rename) keep their own group; messages with +// no classification fall into "Unclassified" last. +func groupMessages(items []store.MessageListItem, categories []store.Category) []messageGroup { + canonical := make(map[string]string, len(categories)) + order := make([]string, 0, len(categories)) + for _, c := range categories { + canonical[strings.ToLower(c.Name)] = c.Name + order = append(order, c.Name) + } + buckets := map[string][]messageRow{} for _, it := range items { - key := strings.ToLower(strings.TrimSpace(it.Category)) + raw := strings.TrimSpace(it.Category) + key := "" + if raw != "" { + if name, ok := canonical[strings.ToLower(raw)]; ok { + key = name + } else { + key = raw + } + } if key == "" { key = unclassifiedCategory } @@ -85,62 +94,47 @@ func groupMessages(items []store.MessageListItem) []messageGroup { Subject: orDefault(it.Subject, "(no subject)"), SourceAddr: it.SourceAddress, Action: action, + Confidence: confidencePercent(it.Confidence), + Classified: it.HasClassification, When: formatWhen(it.DateSent, it.ReceivedAt), }) } - // Order: built-in categories first, then custom ones alphabetically, then - // Unclassified last. var keys []string - for _, k := range categoryOrder { - if _, ok := buckets[k]; ok { - keys = append(keys, k) + for _, name := range order { + if _, ok := buckets[name]; ok { + keys = append(keys, name) } } - var custom []string + var unknown []string for k := range buckets { - if k == unclassifiedCategory || contains(categoryOrder, k) { + if k == unclassifiedCategory || contains(order, k) { continue } - custom = append(custom, k) + unknown = append(unknown, k) } - sort.Strings(custom) - keys = append(keys, custom...) + sort.Strings(unknown) + keys = append(keys, unknown...) if _, ok := buckets[unclassifiedCategory]; ok { keys = append(keys, unclassifiedCategory) } groups := make([]messageGroup, 0, len(keys)) for _, k := range keys { - groups = append(groups, messageGroup{ - Name: displayCategory(k), - Messages: buckets[k], - }) + groups = append(groups, messageGroup{Name: k, Messages: buckets[k]}) } return groups } -func displayCategory(key string) string { - if name, ok := categoryDisplay[key]; ok { - return name +// confidencePercent turns a 0–1 confidence into a whole percentage. +func confidencePercent(v float64) int { + if v <= 0 { + return 0 } - if key == "" { - return "Unclassified" + if v > 1 { + v = 1 } - return strings.ToUpper(key[:1]) + key[1:] -} - -// categoryCounts returns per-category message counts for the left nav. -func categoryCounts(items []store.MessageListItem) map[string]int { - counts := map[string]int{} - for _, it := range items { - key := strings.ToLower(strings.TrimSpace(it.Category)) - if key == "" { - key = unclassifiedCategory - } - counts[key]++ - } - return counts + return int(v*100 + 0.5) } func (s *Server) renderInbox(w http.ResponseWriter, r *http.Request, note string) { @@ -187,6 +181,53 @@ func (s *Server) handleFetchMail(w http.ResponseWriter, r *http.Request) { } } +// handleClassify classifies pending messages (FR-3) and returns the refreshed +// list. A run can take minutes, so the write deadline is extended for this +// handler only. +func (s *Server) handleClassify(w http.ResponseWriter, r *http.Request) { + s.runClassifier(w, r, false) +} + +// handleReclassify re-runs the classifier over every stored message. It is how +// a category-list or rules edit is applied to mail that was already classified. +func (s *Server) handleReclassify(w http.ResponseWriter, r *http.Request) { + s.runClassifier(w, r, true) +} + +func (s *Server) runClassifier(w http.ResponseWriter, r *http.Request, all bool) { + if !s.csrfOK(r) { + http.Error(w, "invalid CSRF token", http.StatusForbidden) + return + } + if rc := http.NewResponseController(w); rc != nil { + _ = rc.SetWriteDeadline(time.Now().Add(15 * time.Minute)) + } + + ctx, cancel := context.WithTimeout(r.Context(), 15*time.Minute) + defer cancel() + + var ( + sum classify.Summary + err error + ) + if all { + sum, err = s.classifier.RunAll(ctx) + } else { + sum, err = s.classifier.Run(ctx) + } + switch { + case err == nil: + s.renderInbox(w, r, fmt.Sprintf( + "Classified %d of %d message(s); %d failed; %d still unclassified.", + sum.Classified, sum.Considered, sum.Failed, sum.Remaining)) + case errors.Is(err, classify.ErrNotConfigured): + s.renderInbox(w, r, "Classification is not connected yet: set JEV_API_KEY.") + default: + s.logger.Error("classification run failed", "error", err) + s.renderInbox(w, r, "Classification failed: "+err.Error()) + } +} + func orDefault(v, def string) string { if strings.TrimSpace(v) == "" { return def diff --git a/internal/server/server.go b/internal/server/server.go index 25fb6c8..7257c62 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -17,6 +17,7 @@ import ( "net/http" "time" + "gitea.lab.audasmedia.com.au/sam/email_agent/internal/classify" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/config" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/mailfetch" "gitea.lab.audasmedia.com.au/sam/email_agent/internal/session" @@ -32,13 +33,14 @@ func init() { // Server holds the HTTP dependencies. type Server struct { - cfg *config.Config - store *store.Store - logger *slog.Logger - http *http.Server - tmpl *template.Template - sessions *session.Manager - fetcher *mailfetch.Fetcher + cfg *config.Config + store *store.Store + logger *slog.Logger + http *http.Server + tmpl *template.Template + sessions *session.Manager + fetcher *mailfetch.Fetcher + classifier *classify.Classifier } // New builds a Server. It does not start listening. @@ -64,6 +66,10 @@ func New(cfg *config.Config, st *store.Store, logger *slog.Logger) (*Server, err tmpl: tmpl, sessions: sessions, fetcher: mailfetch.NewFetcher(cfg, st, logger), + classifier: classify.NewClassifier( + classify.NewClient(cfg.JEVBaseURL, cfg.JEVAPIKey, cfg.JEVModel, logger), + st, logger, cfg.ClassifyConcurrency, cfg.ClassifyMaxMessages, + ), } s.http = &http.Server{ Addr: cfg.Addr, @@ -113,6 +119,14 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc("GET /{$}", s.requireAuth(s.handleDashboard)) mux.HandleFunc("GET /partials/inbox", s.requireAuth(s.handleInboxPartial)) mux.HandleFunc("POST /actions/fetch", s.requireAuth(s.handleFetchMail)) + mux.HandleFunc("POST /actions/classify", s.requireAuth(s.handleClassify)) + mux.HandleFunc("POST /actions/reclassify", s.requireAuth(s.handleReclassify)) + + // FR-14: editable category list. + mux.HandleFunc("GET /categories", s.requireAuth(s.handleCategories)) + mux.HandleFunc("POST /categories/add", s.requireAuth(s.handleCategoryAdd)) + mux.HandleFunc("POST /categories/rename", s.requireAuth(s.handleCategoryRename)) + mux.HandleFunc("POST /categories/remove", s.requireAuth(s.handleCategoryRemove)) return s.recoverPanic(s.logRequests(mux)) } @@ -234,6 +248,10 @@ type statusRecorder struct { status int } +// Unwrap lets http.ResponseController reach the underlying writer (the classify +// handler extends its write deadline for long runs). +func (r *statusRecorder) Unwrap() http.ResponseWriter { return r.ResponseWriter } + func (r *statusRecorder) WriteHeader(code int) { r.status = code r.ResponseWriter.WriteHeader(code) diff --git a/internal/store/categories.go b/internal/store/categories.go new file mode 100644 index 0000000..8b00fb2 --- /dev/null +++ b/internal/store/categories.go @@ -0,0 +1,218 @@ +// Category management (FR-14) and the classification read/write paths (FR-3). +// +// `categories.name` is the stable key: it is stored in `classifications.category` +// and sent to JEV as the Choice option name. Renaming a category therefore +// cascades to existing classifications so the triage list never shows a category +// that no longer exists. +package store + +import ( + "context" + "errors" + "fmt" + "strings" +) + +// ErrLastCategory is returned when removing the final category would leave the +// classifier with nothing to choose from. +var ErrLastCategory = errors.New("at least one category is required") + +// ErrCategoryExists is returned when adding or renaming to a name already taken. +var ErrCategoryExists = errors.New("a category with that name already exists") + +// Category is one editable triage category. +type Category struct { + ID int64 + Name string + Description string + Position int + // MessageCount is the number of messages whose latest classification is + // this category. It is filled by ListCategories. + MessageCount int +} + +// ListCategories returns the categories in display order, each with the count of +// messages currently classified into it. +func (s *Store) ListCategories(ctx context.Context) ([]Category, error) { + const q = ` +SELECT c.id, c.name, c.description, c.position, + (SELECT COUNT(*) FROM messages m + WHERE (SELECT cc.category FROM classifications cc + WHERE cc.message_id = m.id ORDER BY cc.id DESC LIMIT 1) = c.name) +FROM categories c +ORDER BY c.position, c.name COLLATE NOCASE` + rows, err := s.db.QueryContext(ctx, q) + if err != nil { + return nil, fmt.Errorf("list categories: %w", err) + } + defer rows.Close() + + var out []Category + for rows.Next() { + var c Category + if err := rows.Scan(&c.ID, &c.Name, &c.Description, &c.Position, &c.MessageCount); err != nil { + return nil, fmt.Errorf("scan category: %w", err) + } + out = append(out, c) + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("list categories: %w", err) + } + return out, nil +} + +// CategoryNames returns just the names, in display order. +func (s *Store) CategoryNames(ctx context.Context) ([]string, error) { + cats, err := s.ListCategories(ctx) + if err != nil { + return nil, err + } + names := make([]string, 0, len(cats)) + for _, c := range cats { + names = append(names, c.Name) + } + return names, nil +} + +// AddCategory appends a category. The name is trimmed, length-capped, and must +// be unique (case-insensitively). +func (s *Store) AddCategory(ctx context.Context, name, description string) (Category, error) { + name, err := cleanCategoryName(name) + if err != nil { + return Category{}, err + } + description = strings.TrimSpace(description) + + var exists int + if err := s.db.QueryRowContext(ctx, + `SELECT COUNT(*) FROM categories WHERE name = ? COLLATE NOCASE`, name).Scan(&exists); err != nil { + return Category{}, fmt.Errorf("check category name: %w", err) + } + if exists > 0 { + return Category{}, ErrCategoryExists + } + + var position int + if err := s.db.QueryRowContext(ctx, + `SELECT COALESCE(MAX(position), 0) + 1 FROM categories`).Scan(&position); err != nil { + return Category{}, fmt.Errorf("next category position: %w", err) + } + + res, err := s.db.ExecContext(ctx, + `INSERT INTO categories (name, description, position) VALUES (?, ?, ?)`, + name, description, position) + if err != nil { + return Category{}, fmt.Errorf("insert category %q: %w", name, err) + } + id, err := res.LastInsertId() + if err != nil { + return Category{}, fmt.Errorf("read new category id: %w", err) + } + return Category{ID: id, Name: name, Description: description, Position: position}, nil +} + +// RenameCategory changes the display name and rewrites existing classifications +// so the new name is used everywhere (FR-14). +func (s *Store) RenameCategory(ctx context.Context, id int64, newName string) error { + old, err := s.categoryByID(ctx, id) + if err != nil { + return err + } + name, err := cleanCategoryName(newName) + if err != nil { + return err + } + if strings.EqualFold(name, old) { + return nil + } + + var exists int + if err := s.db.QueryRowContext(ctx, + `SELECT COUNT(*) FROM categories WHERE name = ? COLLATE NOCASE AND id <> ?`, name, id).Scan(&exists); err != nil { + return fmt.Errorf("check category name: %w", err) + } + if exists > 0 { + return ErrCategoryExists + } + + tx, err := s.db.BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("begin rename: %w", err) + } + defer func() { _ = tx.Rollback() }() + + if _, err := tx.ExecContext(ctx, `UPDATE categories SET name = ? WHERE id = ?`, name, id); err != nil { + return fmt.Errorf("rename category: %w", err) + } + if _, err := tx.ExecContext(ctx, + `UPDATE classifications SET category = ? WHERE category = ? COLLATE NOCASE`, name, old); err != nil { + return fmt.Errorf("update classifications after rename: %w", err) + } + if err := tx.Commit(); err != nil { + return fmt.Errorf("commit rename: %w", err) + } + return nil +} + +// RemoveCategory deletes a category and moves any messages still classified into +// it to the first remaining category, so no classification points at a category +// that no longer exists. +func (s *Store) RemoveCategory(ctx context.Context, id int64) error { + name, err := s.categoryByID(ctx, id) + if err != nil { + return err + } + + var count int + if err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM categories`).Scan(&count); err != nil { + return fmt.Errorf("count categories: %w", err) + } + if count <= 1 { + return ErrLastCategory + } + + var fallback string + if err := s.db.QueryRowContext(ctx, + `SELECT name FROM categories WHERE id <> ? ORDER BY position, name COLLATE NOCASE LIMIT 1`, id).Scan(&fallback); err != nil { + return fmt.Errorf("select fallback category: %w", err) + } + + tx, err := s.db.BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("begin remove: %w", err) + } + defer func() { _ = tx.Rollback() }() + + if _, err := tx.ExecContext(ctx, + `UPDATE classifications SET category = ? WHERE category = ? COLLATE NOCASE`, fallback, name); err != nil { + return fmt.Errorf("reassign classifications: %w", err) + } + if _, err := tx.ExecContext(ctx, `DELETE FROM categories WHERE id = ?`, id); err != nil { + return fmt.Errorf("delete category: %w", err) + } + if err := tx.Commit(); err != nil { + return fmt.Errorf("commit remove: %w", err) + } + return nil +} + +func (s *Store) categoryByID(ctx context.Context, id int64) (string, error) { + var name string + if err := s.db.QueryRowContext(ctx, + `SELECT name FROM categories WHERE id = ?`, id).Scan(&name); err != nil { + return "", fmt.Errorf("read category %d: %w", id, err) + } + return name, nil +} + +// cleanCategoryName trims, caps at 40 characters, and rejects an empty name. +func cleanCategoryName(name string) (string, error) { + name = strings.Join(strings.Fields(strings.TrimSpace(name)), " ") + if name == "" { + return "", errors.New("category name must not be empty") + } + if len([]rune(name)) > 40 { + return "", errors.New("category name must be 40 characters or fewer") + } + return name, nil +} diff --git a/internal/store/categories_test.go b/internal/store/categories_test.go new file mode 100644 index 0000000..fd81523 --- /dev/null +++ b/internal/store/categories_test.go @@ -0,0 +1,245 @@ +package store + +import ( + "context" + "errors" + "path/filepath" + "testing" + "time" +) + +func newCategoryStore(t *testing.T) *Store { + t.Helper() + s, err := Open(context.Background(), filepath.Join(t.TempDir(), "email_agent.db")) + if err != nil { + t.Fatalf("Open() error = %v", err) + } + t.Cleanup(func() { _ = s.Close() }) + return s +} + +func TestDefaultCategoriesSeeded(t *testing.T) { + ctx := context.Background() + s := newCategoryStore(t) + + cats, err := s.ListCategories(ctx) + if err != nil { + t.Fatalf("ListCategories() error = %v", err) + } + want := []string{"Action", "Important", "Info", "Spam", "Newsletter", "Auto-reply"} + if len(cats) != len(want) { + t.Fatalf("categories = %d, want %d: %+v", len(cats), len(want), cats) + } + for i, name := range want { + if cats[i].Name != name { + t.Errorf("category %d = %q, want %q", i, cats[i].Name, name) + } + if cats[i].Description == "" { + t.Errorf("category %q has no description (JEV needs the rubric)", name) + } + } +} + +func TestSeedCategoriesDoesNotReaddRemoved(t *testing.T) { + ctx := context.Background() + path := filepath.Join(t.TempDir(), "email_agent.db") + + s1, err := Open(ctx, path) + if err != nil { + t.Fatalf("Open() error = %v", err) + } + cats, err := s1.ListCategories(ctx) + if err != nil { + t.Fatalf("ListCategories() error = %v", err) + } + var spamID int64 + for _, c := range cats { + if c.Name == "Spam" { + spamID = c.ID + } + } + if err := s1.RemoveCategory(ctx, spamID); err != nil { + t.Fatalf("RemoveCategory() error = %v", err) + } + _ = s1.Close() + + s2, err := Open(ctx, path) + if err != nil { + t.Fatalf("second Open() error = %v", err) + } + defer s2.Close() + cats, err = s2.ListCategories(ctx) + if err != nil { + t.Fatalf("ListCategories() after restart error = %v", err) + } + for _, c := range cats { + if c.Name == "Spam" { + t.Error("removed category came back after restart") + } + } +} + +func TestAddCategoryRejectsDuplicate(t *testing.T) { + ctx := context.Background() + s := newCategoryStore(t) + + if _, err := s.AddCategory(ctx, "Finance", "Bills and payments"); err != nil { + t.Fatalf("AddCategory() error = %v", err) + } + if _, err := s.AddCategory(ctx, "finance", ""); !errors.Is(err, ErrCategoryExists) { + t.Fatalf("duplicate AddCategory() error = %v, want ErrCategoryExists", err) + } + if _, err := s.AddCategory(ctx, " ", ""); err == nil { + t.Fatal("blank AddCategory() error = nil, want an error") + } +} + +func TestRenameCategoryCascadesToClassifications(t *testing.T) { + ctx := context.Background() + s := newCategoryStore(t) + + msgID, err := s.UpsertMessage(ctx, &Message{ + GmailID: "1", Subject: "hi", FromAddr: "a@b.com", DateSent: time.Now().UTC(), + }) + if err != nil { + t.Fatalf("UpsertMessage() error = %v", err) + } + if _, err := s.SaveClassification(ctx, Classification{ + MessageID: msgID, Category: "Info", Importance: "low", Action: "keep", + }); err != nil { + t.Fatalf("SaveClassification() error = %v", err) + } + + cats, err := s.ListCategories(ctx) + if err != nil { + t.Fatalf("ListCategories() error = %v", err) + } + var infoID int64 + for _, c := range cats { + if c.Name == "Info" { + infoID = c.ID + } + } + if err := s.RenameCategory(ctx, infoID, "Updates"); err != nil { + t.Fatalf("RenameCategory() error = %v", err) + } + + items, err := s.ListMessages(ctx, 10) + if err != nil { + t.Fatalf("ListMessages() error = %v", err) + } + if items[0].Category != "Updates" { + t.Errorf("classification after rename = %q, want Updates", items[0].Category) + } +} + +func TestRemoveCategoryReassignsMessages(t *testing.T) { + ctx := context.Background() + s := newCategoryStore(t) + + msgID, err := s.UpsertMessage(ctx, &Message{GmailID: "1", Subject: "hi"}) + if err != nil { + t.Fatalf("UpsertMessage() error = %v", err) + } + if _, err := s.SaveClassification(ctx, Classification{MessageID: msgID, Category: "Spam"}); err != nil { + t.Fatalf("SaveClassification() error = %v", err) + } + + cats, err := s.ListCategories(ctx) + if err != nil { + t.Fatalf("ListCategories() error = %v", err) + } + var spamID int64 + for _, c := range cats { + if c.Name == "Spam" { + spamID = c.ID + } + } + if err := s.RemoveCategory(ctx, spamID); err != nil { + t.Fatalf("RemoveCategory() error = %v", err) + } + + items, err := s.ListMessages(ctx, 10) + if err != nil { + t.Fatalf("ListMessages() error = %v", err) + } + if items[0].Category != "Action" { + t.Errorf("reassigned category = %q, want the first remaining category Action", items[0].Category) + } +} + +func TestRemoveLastCategoryRefused(t *testing.T) { + ctx := context.Background() + s := newCategoryStore(t) + + for { + cats, err := s.ListCategories(ctx) + if err != nil { + t.Fatalf("ListCategories() error = %v", err) + } + if len(cats) == 1 { + if err := s.RemoveCategory(ctx, cats[0].ID); !errors.Is(err, ErrLastCategory) { + t.Fatalf("RemoveCategory(last) error = %v, want ErrLastCategory", err) + } + return + } + if err := s.RemoveCategory(ctx, cats[0].ID); err != nil { + t.Fatalf("RemoveCategory() error = %v", err) + } + } +} + +func TestCategoryCountsReflectClassifications(t *testing.T) { + ctx := context.Background() + s := newCategoryStore(t) + + msgID, err := s.UpsertMessage(ctx, &Message{GmailID: "1", Subject: "hi"}) + if err != nil { + t.Fatalf("UpsertMessage() error = %v", err) + } + if _, err := s.SaveClassification(ctx, Classification{MessageID: msgID, Category: "Action"}); err != nil { + t.Fatalf("SaveClassification() error = %v", err) + } + + cats, err := s.ListCategories(ctx) + if err != nil { + t.Fatalf("ListCategories() error = %v", err) + } + for _, c := range cats { + want := 0 + if c.Name == "Action" { + want = 1 + } + if c.MessageCount != want { + t.Errorf("category %q count = %d, want %d", c.Name, c.MessageCount, want) + } + } +} + +func TestRulesSeedAndVersioning(t *testing.T) { + ctx := context.Background() + s := newCategoryStore(t) + + text, version, err := s.CurrentRules(ctx) + if err != nil { + t.Fatalf("CurrentRules() error = %v", err) + } + if version != 1 || text == "" { + t.Fatalf("seeded rules = (version %d, %d bytes), want version 1 with text", version, len(text)) + } + + next, err := s.SaveRules(ctx, `from contains "x" => category=Info`) + if err != nil { + t.Fatalf("SaveRules() error = %v", err) + } + if next != 2 { + t.Errorf("SaveRules() version = %d, want 2", next) + } + text, version, err = s.CurrentRules(ctx) + if err != nil { + t.Fatalf("CurrentRules() error = %v", err) + } + if version != 2 || text != `from contains "x" => category=Info` { + t.Errorf("CurrentRules() = (%d, %q), want version 2 with the new text", version, text) + } +} diff --git a/internal/store/classify.go b/internal/store/classify.go new file mode 100644 index 0000000..18ffd10 --- /dev/null +++ b/internal/store/classify.go @@ -0,0 +1,161 @@ +// Classification persistence and the rules read/write paths (FR-3, FR-4, FR-11). +package store + +import ( + "context" + "database/sql" + "errors" + "fmt" +) + +// Classification is one classifier output for a message. +type Classification struct { + ID int64 + MessageID int64 + Category string + Importance string + Action string + Confidence float64 + Model string +} + +// SaveClassification appends a classification row. The triage list reads the +// newest row per message, so re-classifying keeps a history. +func (s *Store) SaveClassification(ctx context.Context, c Classification) (int64, error) { + const q = `INSERT INTO classifications + (message_id, category, importance, action, confidence, model) + VALUES (?, ?, ?, ?, ?, ?)` + res, err := s.db.ExecContext(ctx, q, + c.MessageID, c.Category, nullString(c.Importance), nullString(c.Action), + c.Confidence, nullString(c.Model)) + if err != nil { + return 0, fmt.Errorf("save classification for message %d: %w", c.MessageID, err) + } + id, err := res.LastInsertId() + if err != nil { + return 0, fmt.Errorf("read classification id: %w", err) + } + return id, nil +} + +// PendingClassifications returns messages whose latest classification is +// missing or points at a category that no longer exists. Oldest first so a +// bounded run makes steady progress. +func (s *Store) PendingClassifications(ctx context.Context, limit int) ([]Message, error) { + return s.messagesForClassification(ctx, limit, false) +} + +// AllMessagesForClassification returns every message, for a full re-classify +// after the category list or the rules change. +func (s *Store) AllMessagesForClassification(ctx context.Context, limit int) ([]Message, error) { + return s.messagesForClassification(ctx, limit, true) +} + +func (s *Store) messagesForClassification(ctx context.Context, limit int, all bool) ([]Message, error) { + if limit < 1 { + limit = 100 + } + filter := "WHERE c.id IS NULL OR c.category NOT IN (SELECT name FROM categories)" + if all { + filter = "" + } + q := ` +SELECT m.id, m.gmail_id, COALESCE(m.thread_id, ''), COALESCE(m.from_addr, ''), + COALESCE(m.to_addr, ''), COALESCE(m.subject, ''), COALESCE(m.body_text, ''), + COALESCE(m.source_address, ''), COALESCE(m.date_sent, ''), m.received_at, + COALESCE(m.imap_uid, 0), COALESCE(m.imap_uidvalidity, 0) +FROM messages m +LEFT JOIN classifications c ON c.id = ( + SELECT id FROM classifications WHERE message_id = m.id ORDER BY id DESC LIMIT 1 +) +` + filter + ` +ORDER BY m.id ASC +LIMIT ?` + rows, err := s.db.QueryContext(ctx, q, limit) + if err != nil { + return nil, fmt.Errorf("list messages for classification: %w", err) + } + defer rows.Close() + + var out []Message + for rows.Next() { + var ( + m Message + dateSent string + received string + ) + if err := rows.Scan( + &m.ID, &m.GmailID, &m.ThreadID, &m.FromAddr, &m.ToAddr, &m.Subject, + &m.BodyText, &m.SourceAddress, &dateSent, &received, + &m.IMAPUID, &m.IMAPUIDValidity, + ); err != nil { + return nil, fmt.Errorf("scan pending message: %w", err) + } + m.DateSent = parseSQLTime(dateSent) + m.ReceivedAt = parseSQLTime(received) + out = append(out, m) + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("list messages for classification: %w", err) + } + return out, nil +} + +// ClassificationCoverage reports how many messages have a usable classification. +type ClassificationCoverage struct { + Total int + Classified int +} + +// Missing reports how many messages still need a classification. +func (c ClassificationCoverage) Missing() int { return c.Total - c.Classified } + +// ClassificationCoverage returns the total message count and how many have a +// classification whose category still exists. +func (s *Store) ClassificationCoverage(ctx context.Context) (ClassificationCoverage, error) { + var cov ClassificationCoverage + const q = ` +SELECT + (SELECT COUNT(*) FROM messages), + (SELECT COUNT(*) FROM messages m + WHERE (SELECT cc.category FROM classifications cc + WHERE cc.message_id = m.id ORDER BY cc.id DESC LIMIT 1) + IN (SELECT name FROM categories))` + if err := s.db.QueryRowContext(ctx, q).Scan(&cov.Total, &cov.Classified); err != nil { + return cov, fmt.Errorf("classification coverage: %w", err) + } + return cov, nil +} + +// CurrentRules returns the newest rules text and its version. An empty string +// and version 0 mean no rules are stored yet. +func (s *Store) CurrentRules(ctx context.Context) (string, int, error) { + var ( + text string + version int + ) + err := s.db.QueryRowContext(ctx, + `SELECT text, version FROM rules ORDER BY version DESC LIMIT 1`).Scan(&text, &version) + if errors.Is(err, sql.ErrNoRows) { + return "", 0, nil + } + if err != nil { + return "", 0, fmt.Errorf("read rules: %w", err) + } + return text, version, nil +} + +// SaveRules appends a new rules version and returns the new version number +// (FR-11 versioning). The rules editor UI lands in Phase 6. +func (s *Store) SaveRules(ctx context.Context, text string) (int, error) { + var next int + if err := s.db.QueryRowContext(ctx, + `SELECT COALESCE(MAX(version), 0) + 1 FROM rules`).Scan(&next); err != nil { + return 0, fmt.Errorf("next rules version: %w", err) + } + if _, err := s.db.ExecContext(ctx, + `INSERT INTO rules (text, version) VALUES (?, ?)`, text, next); err != nil { + return 0, fmt.Errorf("save rules version %d: %w", next, err) + } + return next, nil +} diff --git a/internal/store/messages.go b/internal/store/messages.go index ab174d7..d3c8c03 100644 --- a/internal/store/messages.go +++ b/internal/store/messages.go @@ -159,6 +159,15 @@ func (s *Store) LogAction(ctx context.Context, action, actor, detail string) err return nil } +// LogMessageAction appends a per-message action to the actions_log (FR-13). +func (s *Store) LogMessageAction(ctx context.Context, messageID int64, action, actor, detail string) error { + const q = `INSERT INTO actions_log (message_id, action, actor, detail) VALUES (?, ?, ?, ?)` + if _, err := s.db.ExecContext(ctx, q, messageID, action, nullString(actor), nullString(detail)); err != nil { + return fmt.Errorf("log message action %q: %w", action, err) + } + return nil +} + const sqlTimeLayout = "2006-01-02 15:04:05" func nullString(v string) any { diff --git a/internal/store/schema.sql b/internal/store/schema.sql index 5ac4c56..adc679c 100644 --- a/internal/store/schema.sql +++ b/internal/store/schema.sql @@ -36,6 +36,19 @@ CREATE INDEX IF NOT EXISTS idx_messages_received_at ON messages (received_at); CREATE INDEX IF NOT EXISTS idx_messages_thread_id ON messages (thread_id); CREATE INDEX IF NOT EXISTS idx_messages_source ON messages (source_address); +-- FR-14: the editable category list. `name` is the stable key stored in +-- classifications.category and sent to JEV as the Choice option name. Renaming a +-- category cascades to existing classifications (see store.RenameCategory). +CREATE TABLE IF NOT EXISTS categories ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL COLLATE NOCASE UNIQUE, + description TEXT NOT NULL DEFAULT '', + position INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); + +CREATE INDEX IF NOT EXISTS idx_categories_position ON categories (position, name); + -- FR-3, FR-14: classifier output per message. CREATE TABLE IF NOT EXISTS classifications ( id INTEGER PRIMARY KEY AUTOINCREMENT, diff --git a/internal/store/store.go b/internal/store/store.go index c923717..181432c 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -108,6 +108,89 @@ func (s *Store) applyMigrations(ctx context.Context) error { return fmt.Errorf("record schema version 2: %w", err) } } + if version < 3 { + // FR-14: seed the default category list and a starter (comment-only) + // rules document. Seeding happens only inside this migration so that a + // category the user removes is not silently re-added on the next start. + if err := s.seedCategories(ctx); err != nil { + return err + } + if err := s.seedRules(ctx); err != nil { + return err + } + if _, err := s.db.ExecContext(ctx, + `INSERT OR IGNORE INTO schema_migrations (version) VALUES (3)`); err != nil { + return fmt.Errorf("record schema version 3: %w", err) + } + } + return nil +} + +// defaultCategories are the starter categories from AGENTS.md §13.2. The +// descriptions are shown in the UI and sent to JEV as the Choice rubrics. +var defaultCategories = []struct{ name, description string }{ + {"Action", "Needs Sam to do something"}, + {"Important", "Worth Sam's attention, but no specific task"}, + {"Info", "Informational, no action required"}, + {"Spam", "Unwanted, deceptive, or scam mail"}, + {"Newsletter", "Bulk mail, digests, and marketing"}, + {"Auto-reply", "Automated notifications and receipts"}, +} + +// seedCategories inserts the built-in categories when the table is empty. +func (s *Store) seedCategories(ctx context.Context) error { + var n int + if err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM categories`).Scan(&n); err != nil { + return fmt.Errorf("count categories: %w", err) + } + if n > 0 { + return nil + } + for i, c := range defaultCategories { + if _, err := s.db.ExecContext(ctx, + `INSERT INTO categories (name, description, position) VALUES (?, ?, ?)`, + c.name, c.description, i+1); err != nil { + return fmt.Errorf("seed category %q: %w", c.name, err) + } + } + return nil +} + +// starterRules documents the rule syntax (FR-4/FR-11). The editor that writes +// real rules lands in Phase 6; the classifier already reads whatever is stored. +const starterRules = `# Email Agent rules (FR-4). One rule per line. +# +# => +# +# condition: "", joined with " and ". +# fields: from, to, subject, source, body +# operators: contains, equals, starts_with, ends_with +# a single "*" matches every message (use it as the last line for a default). +# +# effects: comma-separated category=, importance=, +# action= +# +# The first matching rule wins and overrides the JEV classification for the +# fields it names. Category names must exist in the category list. +# +# Examples (remove the leading # to enable): +# from contains "amazon.com" => category=Newsletter, action=unsubscribe +# subject contains "invoice" => category=Action, importance=high, action=file +` + +// seedRules stores the starter rules document when no rules exist yet. +func (s *Store) seedRules(ctx context.Context) error { + var n int + if err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM rules`).Scan(&n); err != nil { + return fmt.Errorf("count rules: %w", err) + } + if n > 0 { + return nil + } + if _, err := s.db.ExecContext(ctx, + `INSERT INTO rules (text, version) VALUES (?, ?)`, starterRules, 1); err != nil { + return fmt.Errorf("seed rules: %w", err) + } return nil } diff --git a/web/static/app.css b/web/static/app.css index 5f6d0b9..d8ee88e 100644 --- a/web/static/app.css +++ b/web/static/app.css @@ -326,3 +326,52 @@ a:hover { text-decoration: underline; } /* subtle, non-decorative loading pulse (no spinners) */ .htmx-request .empty-state { opacity: .55; transition: opacity .15s ease; } +.btn.htmx-request { opacity: .55; transition: opacity .15s ease; } + +/* Phase 3: classifier confidence on each row */ +.msg-confidence { font: var(--type-caption); color: var(--ink-faint); white-space: nowrap; } + +/* Phase 3: editable category list (FR-14) */ +.nav-group-title { display: flex; align-items: center; justify-content: space-between; gap: var(--space-xs); } +.nav-edit { font: var(--type-caption); color: var(--primary); } + +.categories-panel { margin-top: var(--space-md); } +.categories-help { font: var(--type-body-sm); margin-bottom: var(--space-md); max-width: 68ch; } + +.category-add { + display: grid; + grid-template-columns: 1fr 2fr auto; + align-items: end; + gap: var(--space-sm); + padding-bottom: var(--space-md); + border-bottom: 1px solid var(--hairline); +} + +.category-list { display: flex; flex-direction: column; } +.category-row { + display: flex; + align-items: end; + gap: var(--space-sm); + padding: var(--space-sm) 0; + border-bottom: 1px solid var(--hairline); +} +.category-row:last-child { border-bottom: 0; } +.category-rename { + display: grid; + grid-template-columns: minmax(0, 1fr) minmax(0, 2fr) auto auto; + align-items: end; + gap: var(--space-sm); + flex: 1 1 auto; +} +.category-desc { min-width: 0; padding-bottom: var(--space-xs); } +.category-desc .muted { display: block; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.category-rename .count { white-space: nowrap; padding-bottom: var(--space-xs); } +.category-remove { flex: 0 0 auto; } +.btn-danger { color: var(--sticker-red); border-color: var(--sticker-red); } + +@media (max-width: 759px) { + .category-add { grid-template-columns: 1fr; } + .category-row { flex-direction: column; align-items: stretch; } + .category-rename { grid-template-columns: 1fr; } + .category-remove .btn { width: 100%; } +} diff --git a/web/templates/categories.html b/web/templates/categories.html new file mode 100644 index 0000000..ce9f95b --- /dev/null +++ b/web/templates/categories.html @@ -0,0 +1,104 @@ + + + + + + Categories · Email Agent + + + + + +
+ Email Agent +
+ Back to inbox + +
+ + +
+
+
+ +
+

Categories

+ + {{if .Note}}
{{.Note}}
{{end}} + {{if .Error}}{{end}} + +
+
+

Category list

+ {{.Classified}} of {{.Total}} message(s) classified +
+ +
+

+ JEV classifies each message against this list. Renaming a category updates + every message already filed under it. Removing one moves those messages to + the first remaining category. At least one category is required. +

+ +
+ +
+ + +
+
+ + +
+ +
+ +
+ {{range .Categories}} +
+
+ + +
+ + +
+
+ {{.Description}} +
+ {{.MessageCount}} message(s) + +
+
+ + + +
+
+ {{end}} +
+
+
+
+ + diff --git a/web/templates/dashboard.html b/web/templates/dashboard.html index c92ca5a..d74a712 100644 --- a/web/templates/dashboard.html +++ b/web/templates/dashboard.html @@ -44,7 +44,10 @@