email_agent/docs/phase3-classify.md

Phase 3 — Classify (JEV, rules, editable categories)

Status: done and deployed on .13. AUD-69, parent AUD-68. Phase 3 of AGENTS.md §11.

DoD: every message has a category, an importance, and a suggested action.

1. Confirmed JEV HTTP shape (before wiring)

Read live from the TypeSafe docs (https://docs.typesafe.ai/llms.txt → api.md, primitives/choice.md, confidence.md, models.md) and confirmed with a real call. This is the shape internal/classify/jev.go implements.

Endpoint — POST https://api.typesafe.ai/v1/systemone, bearer auth:

POST /v1/systemone HTTP/1.1
Authorization: Bearer <JEV_API_KEY>
Content-Type: application/json

Request

{
  "state": { "email": { "from": "…", "to": "…", "source_address": "…",
                        "source_domain": "…", "subject": "…", "received": "…", "body": "…" } },
  "model": "jev-latest",
  "questions": {
    "category":   { "type": "choice", "instructions": "…", "criteria": { "‹option›": "‹rubric›", … } },
    "importance": { "type": "choice", "instructions": "…", "criteria": { "low": "…", "medium": "…", "high": "…" } },
    "action":     { "type": "choice", "instructions": "…", "criteria": { "delete": "…", "unsubscribe": "…", "reply": "…", "file": "…", "keep": "…" } }
  }
}

Response — one answer per question id; Choice answers carry the selected option, the full probability distribution, and a confidence in [0,1]:

{
  "model": "jev-1.13.0",
  "answers": {
    "category":   { "type": "choice", "choice": "Action",  "confidence": 0.94, "probabilities": { … } },
    "importance": { "type": "choice", "choice": "high",    "confidence": 0.89, "probabilities": { … } },
    "action":     { "type": "choice", "choice": "file",    "confidence": 0.49, "probabilities": { … } }
  },
  "usage": { "input_tokens": 1047, "output_tokens": 146 }
}

Errors: 401 invalid key, 422 validation, 429/529 rate-limit/overloaded. The client retries 429, 529, and 5xx up to four times with exponential backoff and jitter, and surfaces any other status with the response body.

One real, read-only classification (live, message id 10)

Message 10 (InMotion Hosting Billing · "InMotion Hosting - Payment Due", arrived at samuelrolfe@gmail.com) was sent to JEV with the six live categories. Live response, verbatim:

{"model":"jev-1.13.0","answers":{
  "category":{"type":"choice","choice":"Action","confidence":0.94,
              "probabilities":{"Action":0.95,"Important":0.0,"Info":0.0,"Spam":0.03,"Newsletter":0.0,"Auto-reply":0.02}},
  "importance":{"type":"choice","choice":"high","confidence":0.89,
                "probabilities":{"low":0.03,"medium":0.04,"high":0.93}},
  "action":{"type":"choice","choice":"file","confidence":0.49,
            "probabilities":{"delete":0.24,"unsubscribe":0.01,"reply":0.01,"file":0.59,"keep":0.15}}},
 "usage":{"input_tokens":1047,"output_tokens":146}}

The stored row matches: Action / high / file / 0.94 / jev-1.13.0.

JEV_BASE_URL was not set on .13 (only JEV_API_KEY is in ~/.config/environment.d/10-secrets.conf). The TypeSafe docs fix the host, so the code defaults an unset/empty JEV_BASE_URL to the documented https://api.typesafe.ai, and JEV_MODEL defaults to jev-latest. This is recorded here rather than treated as a contradiction. A different base URL can still be supplied without a code change.

2. What was built

• internal/classify — JEV client (jev.go), rule engine (rules.go), and orchestration (classify.go). One JEV request per message, bounded concurrency (default 4), a per-run cap (default 200), and a worker pool.
• internal/store — migration v3: a categories table seeded once with the six AGENTS.md §13.2 categories (name + JEV rubric + order) and a comment-only starter rules version. Category CRUD, classification persistence, CurrentRules/SaveRules, and pending/coverage queries.
• internal/server — POST /actions/classify (pending) and POST /actions/reclassify (all), plus the FR-14 categories UI (GET /categories, POST /categories/add|rename|remove).
• FR-5 triage list — messages grouped by the live category list, newest first, with the recommended action and the classifier confidence per row.
• CLI — email-agent -classify (pending) and -classify-all (re-classify everything), the n8n/CLI entry points for FR-3.
• FR-13 — every classification is logged in actions_log with the message, the category/importance/action/confidence/model, the rules version, and the matching rule line when a rule applied. Category add/rename/remove are logged.

Rules (FR-4)

Plain text, one rule per line; the classifier reads the newest rules row. The editor that writes it lands in Phase 6, but the engine is live now.

<condition> => <effects>

from contains "amazon.com" => category=Newsletter, action=unsubscribe
subject contains "invoice" and from ends_with "@stripe.com" => category=Action, importance=high
* => category=Info

Conditions join with and over from, to, subject, source, body using contains, equals, starts_with, ends_with. Effects are category=<live name>, importance=<low|medium|high>, action=<delete|unsubscribe|reply|file|keep>. The first matching rule wins and overrides only the fields it names. A rule naming a category that no longer exists is ignored and logged; malformed lines are skipped and reported. Evidence: internal/classify/rules_test.go and internal/classify/classify_test.go (override, unknown-category fallback, rule-in-log).

Categories (FR-14)

categories.name is the key stored in classifications.category and sent to JEV as the Choice option name, so JEV always sees the current list. Renaming cascades to existing classifications; removing a category reassigns its messages to the first remaining category; the last category cannot be removed.

3. DoD evidence

Deployed: image rebuilt and email-agent recreated on .13; the Phase 1/2 volume email_agent_data was kept and migration v3 ran in place (schema_version: 3).

Check	Result
docker compose ps	email-agent up, healthy, 0.0.0.0:8790->8790/tcp
GET /health (public HTTPS)	200 {"status":"ok","env":"production","database":"ok"}
email-agent -classify (container)	considered=100 classified=100 failed=0 remaining=0 model=jev-latest rules_version=1
Every message has a classification	100 messages, 0 null/empty category, 0 importance, 0 action; 0 messages without a classification row
Model	all 100 rows from jev-1.13.0
Distribution	Newsletter 43, Auto-reply 31, Action 20, Info 4, Spam 1, Important 1
POST /actions/reclassify (HTTP, session + CSRF)	200, note Classified 100 of 100 message(s); 0 failed; 0 still unclassified.
FR-13 per-message log	100 classify actions with category/importance/action/confidence/model; classify_run summary rows
Public HTTPS	https://mail.lab.audasmedia.com.au/health → 200 ok

FR-14 HTTP/UI proof (live, over HTTP on .13)

Login (session + CSRF cookie), then:

1. Add — POST /categories/add name=Finance, description=Bills, invoices, payments, renewals, and financial account alerts → 303. GET /categories then lists … Newsletter, Auto-reply, Finance.

2. Rename — POST /categories/rename id=7 name=Money & Bills → 303. GET /categories lists Money & Bills.

3. JEV uses them — POST /actions/reclassify → 200 Classified 100 of 100 message(s). JEV then answered with the new name for 6 real messages that were not in that category before:

   id	from	subject	result
   1	Crazy Domains	ALERT: Your auto-renewal…	Money & Bills / high / delete / 0.71
   10	InMotion Hosting Billing	Payment Due	Money & Bills / high / file / 0.99
   23	api@minimaxi.com	Pay-as-you-go Balance Disconnect	Money & Bills / high / file / 0.70
   30	Crazy Domains	cmorganics.com.au has Expired	Money & Bills / high / delete / 0.45
   36	api@minimaxi.com	Re: Pay-as-you-go…	Money & Bills / high / file / 0.59
   82	Crazy Domains	cmorganics.com.au has Expired	Money & Bills / high / delete / 0.46

4. Remove — POST /categories/remove id=7 → 200 with note Removed category; its messages moved to the first remaining category.; GET /categories no longer lists it. actions_log records category_add name=Finance, category_rename id=7 name=Money & Bills, category_remove id=7. A second reclassify restored the six default categories; final state is the table above.

Tests

gofmt -l clean, go vet ./... clean, go test ./... green (in golang:1.25-alpine). New tests cover the rule parser and precedence, the JEV request/response shape and 429 retry, the classifier (every pending message, rule override, unknown-category fallback, rerun-only-pending, re-classify-all, rule-in-action-log), category CRUD and cascade, and the category HTTP handlers.

4. Configuration

Added to .env.example and docker-compose.yml (values stay in the host environment / secret file):

Variable	Default
JEV_BASE_URL	https://api.typesafe.ai
JEV_API_KEY	(required for classification)
JEV_MODEL	jev-latest
EMAIL_AGENT_CLASSIFY_CONCURRENCY	4
EMAIL_AGENT_CLASSIFY_MAX_MESSAGES	200

5. Deployment

On .13, from the repo, with the host secrets in scope:

set -a; . ~/.config/environment.d/10-secrets.conf; set +a
docker compose build
docker compose up -d          # migration v3 runs on start; data volume kept
docker compose exec -T email-agent email-agent -classify

6. Rollback

1. docker compose down on .13 (keeps email_agent_data).
2. git revert <this phase commit> and docker compose up -d --build.
3. Migration v3 is additive (new table + seeded rows) and is not undone by the revert; the Phase 2 binary ignores categories and shows every message as Unclassified.

7. Not done / deferred

• The rules editor UI is Phase 6 (FR-11). The rules table, versioning, and the classifier reading them are live.
• POST /actions/reclassify and -classify-all were added so a category or rules change can be applied to existing mail; Phase 6 will use the same path.
• Unsubscribe detection (FR-7) and the reply drafts (FR-8) remain later phases.
• The 1-month body retention job (AGENTS.md §7) is not implemented yet.