Sam Rolfe
af3b5e5f0e
Implements FR-3, FR-4, FR-5 and FR-14. - internal/classify: TypeSafe System One (JEV) HTTP client with the confirmed request/response shape, retry with backoff on 429/529/5xx, a plain-text rule engine (first match wins, overrides only the named fields), and bounded orchestration with a per-run cap. - internal/store: migration v3 adds an editable `categories` table (name = JEV Choice option and classifications key; rename cascades, remove reassigns) and seeds a starter rules version once. Classification persistence, coverage, and pending/all message queries. - internal/server: POST /actions/classify and /actions/reclassify, plus the FR-14 category editor (GET /categories and add/rename/remove). The triage list now groups by the live category list and shows action + confidence. - CLI: -classify and -classify-all for n8n. - Docs: docs/phase3-classify.md with the confirmed JEV shape, a live read-only classification, and the DoD evidence. DoD: all 100 stored messages have a non-null category, importance, and suggested action (0 nulls); JEV picked a newly added/renamed category for 6 real messages over HTTP. One live JEV call recorded in the phase doc. Co-Authored-By: Paperclip <noreply@paperclip.ing>
226 lines
10 KiB
Markdown
226 lines
10 KiB
Markdown
# Phase 3 — Classify (JEV, rules, editable categories)
|
||
|
||
Status: **done and deployed on `.13`.** [AUD-69](/AUD/issues/AUD-69), parent
|
||
[AUD-68](/AUD/issues/AUD-68). Phase 3 of AGENTS.md §11.
|
||
|
||
**DoD:** every message has a category, an importance, and a suggested action.
|
||
|
||
## 1. Confirmed JEV HTTP shape (before wiring)
|
||
|
||
Read live from the TypeSafe docs (`https://docs.typesafe.ai/llms.txt` →
|
||
`api.md`, `primitives/choice.md`, `confidence.md`, `models.md`) and confirmed with
|
||
a real call. This is the shape `internal/classify/jev.go` implements.
|
||
|
||
**Endpoint** — `POST https://api.typesafe.ai/v1/systemone`, bearer auth:
|
||
|
||
```http
|
||
POST /v1/systemone HTTP/1.1
|
||
Authorization: Bearer <JEV_API_KEY>
|
||
Content-Type: application/json
|
||
```
|
||
|
||
**Request**
|
||
|
||
```json
|
||
{
|
||
"state": { "email": { "from": "…", "to": "…", "source_address": "…",
|
||
"source_domain": "…", "subject": "…", "received": "…", "body": "…" } },
|
||
"model": "jev-latest",
|
||
"questions": {
|
||
"category": { "type": "choice", "instructions": "…", "criteria": { "‹option›": "‹rubric›", … } },
|
||
"importance": { "type": "choice", "instructions": "…", "criteria": { "low": "…", "medium": "…", "high": "…" } },
|
||
"action": { "type": "choice", "instructions": "…", "criteria": { "delete": "…", "unsubscribe": "…", "reply": "…", "file": "…", "keep": "…" } }
|
||
}
|
||
}
|
||
```
|
||
|
||
**Response** — one answer per question id; Choice answers carry the selected
|
||
option, the full probability distribution, and a confidence in `[0,1]`:
|
||
|
||
```json
|
||
{
|
||
"model": "jev-1.13.0",
|
||
"answers": {
|
||
"category": { "type": "choice", "choice": "Action", "confidence": 0.94, "probabilities": { … } },
|
||
"importance": { "type": "choice", "choice": "high", "confidence": 0.89, "probabilities": { … } },
|
||
"action": { "type": "choice", "choice": "file", "confidence": 0.49, "probabilities": { … } }
|
||
},
|
||
"usage": { "input_tokens": 1047, "output_tokens": 146 }
|
||
}
|
||
```
|
||
|
||
Errors: `401` invalid key, `422` validation, `429`/`529` rate-limit/overloaded.
|
||
The client retries `429`, `529`, and `5xx` up to four times with exponential
|
||
backoff and jitter, and surfaces any other status with the response body.
|
||
|
||
### One real, read-only classification (live, message id 10)
|
||
|
||
Message 10 (`InMotion Hosting Billing · "InMotion Hosting - Payment Due"`,
|
||
arrived at `samuelrolfe@gmail.com`) was sent to JEV with the six live categories.
|
||
Live response, verbatim:
|
||
|
||
```json
|
||
{"model":"jev-1.13.0","answers":{
|
||
"category":{"type":"choice","choice":"Action","confidence":0.94,
|
||
"probabilities":{"Action":0.95,"Important":0.0,"Info":0.0,"Spam":0.03,"Newsletter":0.0,"Auto-reply":0.02}},
|
||
"importance":{"type":"choice","choice":"high","confidence":0.89,
|
||
"probabilities":{"low":0.03,"medium":0.04,"high":0.93}},
|
||
"action":{"type":"choice","choice":"file","confidence":0.49,
|
||
"probabilities":{"delete":0.24,"unsubscribe":0.01,"reply":0.01,"file":0.59,"keep":0.15}}},
|
||
"usage":{"input_tokens":1047,"output_tokens":146}}
|
||
```
|
||
|
||
The stored row matches: `Action / high / file / 0.94 / jev-1.13.0`.
|
||
|
||
**`JEV_BASE_URL` was not set on `.13`** (only `JEV_API_KEY` is in
|
||
`~/.config/environment.d/10-secrets.conf`). The TypeSafe docs fix the host, so
|
||
the code defaults an unset/empty `JEV_BASE_URL` to the documented
|
||
`https://api.typesafe.ai`, and `JEV_MODEL` defaults to `jev-latest`. This is
|
||
recorded here rather than treated as a contradiction. A different base URL can
|
||
still be supplied without a code change.
|
||
|
||
## 2. What was built
|
||
|
||
- **`internal/classify`** — JEV client (`jev.go`), rule engine (`rules.go`),
|
||
and orchestration (`classify.go`). One JEV request per message, bounded
|
||
concurrency (default 4), a per-run cap (default 200), and a worker pool.
|
||
- **`internal/store`** — migration **v3**: a `categories` table seeded once with
|
||
the six AGENTS.md §13.2 categories (name + JEV rubric + order) and a
|
||
comment-only starter `rules` version. Category CRUD, classification
|
||
persistence, `CurrentRules`/`SaveRules`, and pending/coverage queries.
|
||
- **`internal/server`** — `POST /actions/classify` (pending) and
|
||
`POST /actions/reclassify` (all), plus the **FR-14 categories UI**
|
||
(`GET /categories`, `POST /categories/add|rename|remove`).
|
||
- **FR-5 triage list** — messages grouped by the live category list, newest
|
||
first, with the recommended action and the classifier confidence per row.
|
||
- **CLI** — `email-agent -classify` (pending) and `-classify-all` (re-classify
|
||
everything), the n8n/CLI entry points for FR-3.
|
||
- **FR-13** — every classification is logged in `actions_log` with the message,
|
||
the category/importance/action/confidence/model, the rules version, and the
|
||
matching rule line when a rule applied. Category add/rename/remove are logged.
|
||
|
||
### Rules (FR-4)
|
||
|
||
Plain text, one rule per line; the classifier reads the newest `rules` row. The
|
||
editor that writes it lands in Phase 6, but the engine is live now.
|
||
|
||
```
|
||
<condition> => <effects>
|
||
|
||
from contains "amazon.com" => category=Newsletter, action=unsubscribe
|
||
subject contains "invoice" and from ends_with "@stripe.com" => category=Action, importance=high
|
||
* => category=Info
|
||
```
|
||
|
||
Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body`
|
||
using `contains`, `equals`, `starts_with`, `ends_with`. Effects are
|
||
`category=<live name>`, `importance=<low|medium|high>`, `action=<delete|unsubscribe|reply|file|keep>`.
|
||
The **first matching rule wins** and overrides only the fields it names. A rule
|
||
naming a category that no longer exists is ignored and logged; malformed lines
|
||
are skipped and reported. Evidence: `internal/classify/rules_test.go` and
|
||
`internal/classify/classify_test.go` (override, unknown-category fallback,
|
||
rule-in-log).
|
||
|
||
### Categories (FR-14)
|
||
|
||
`categories.name` is the key stored in `classifications.category` **and** sent to
|
||
JEV as the Choice option name, so JEV always sees the current list. Renaming
|
||
cascades to existing classifications; removing a category reassigns its messages
|
||
to the first remaining category; the last category cannot be removed.
|
||
|
||
## 3. DoD evidence
|
||
|
||
Deployed: image rebuilt and `email-agent` recreated on `.13`; the Phase 1/2
|
||
volume `email_agent_data` was kept and migration v3 ran in place
|
||
(`schema_version: 3`).
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| `docker compose ps` | `email-agent` up, healthy, `0.0.0.0:8790->8790/tcp` |
|
||
| `GET /health` (public HTTPS) | `200 {"status":"ok","env":"production","database":"ok"}` |
|
||
| `email-agent -classify` (container) | `considered=100 classified=100 failed=0 remaining=0 model=jev-latest rules_version=1` |
|
||
| **Every message has a classification** | `100` messages, **`0`** null/empty category, **`0`** importance, **`0`** action; `0` messages without a classification row |
|
||
| Model | all `100` rows from `jev-1.13.0` |
|
||
| Distribution | Newsletter 43, Auto-reply 31, Action 20, Info 4, Spam 1, Important 1 |
|
||
| `POST /actions/reclassify` (HTTP, session + CSRF) | `200`, note `Classified 100 of 100 message(s); 0 failed; 0 still unclassified.` |
|
||
| FR-13 per-message log | `100` `classify` actions with category/importance/action/confidence/model; `classify_run` summary rows |
|
||
| Public HTTPS | `https://mail.lab.audasmedia.com.au/health` → `200 ok` |
|
||
|
||
### FR-14 HTTP/UI proof (live, over HTTP on `.13`)
|
||
|
||
Login (session + CSRF cookie), then:
|
||
|
||
1. **Add** — `POST /categories/add` `name=Finance`,
|
||
`description=Bills, invoices, payments, renewals, and financial account alerts`
|
||
→ `303`. `GET /categories` then lists `… Newsletter, Auto-reply, Finance`.
|
||
2. **Rename** — `POST /categories/rename` `id=7 name=Money & Bills` → `303`.
|
||
`GET /categories` lists `Money & Bills`.
|
||
3. **JEV uses them** — `POST /actions/reclassify` → `200`
|
||
`Classified 100 of 100 message(s)`. JEV then answered with the new name for
|
||
**6 real messages** that were not in that category before:
|
||
|
||
| id | from | subject | result |
|
||
|---|---|---|---|
|
||
| 1 | Crazy Domains | ALERT: Your auto-renewal… | `Money & Bills / high / delete / 0.71` |
|
||
| 10 | InMotion Hosting Billing | Payment Due | `Money & Bills / high / file / 0.99` |
|
||
| 23 | api@minimaxi.com | Pay-as-you-go Balance Disconnect | `Money & Bills / high / file / 0.70` |
|
||
| 30 | Crazy Domains | cmorganics.com.au has Expired | `Money & Bills / high / delete / 0.45` |
|
||
| 36 | api@minimaxi.com | Re: Pay-as-you-go… | `Money & Bills / high / file / 0.59` |
|
||
| 82 | Crazy Domains | cmorganics.com.au has Expired | `Money & Bills / high / delete / 0.46` |
|
||
|
||
4. **Remove** — `POST /categories/remove` `id=7` → `200` with note
|
||
`Removed category; its messages moved to the first remaining category.`;
|
||
`GET /categories` no longer lists it. `actions_log` records
|
||
`category_add name=Finance`, `category_rename id=7 name=Money & Bills`,
|
||
`category_remove id=7`. A second `reclassify` restored the six default
|
||
categories; final state is the table above.
|
||
|
||
### Tests
|
||
|
||
`gofmt -l` clean, `go vet ./...` clean, `go test ./...` green (in
|
||
`golang:1.25-alpine`). New tests cover the rule parser and precedence, the JEV
|
||
request/response shape and 429 retry, the classifier (every pending message,
|
||
rule override, unknown-category fallback, rerun-only-pending, re-classify-all,
|
||
rule-in-action-log), category CRUD and cascade, and the category HTTP handlers.
|
||
|
||
## 4. Configuration
|
||
|
||
Added to `.env.example` and `docker-compose.yml` (values stay in the host
|
||
environment / secret file):
|
||
|
||
| Variable | Default |
|
||
|---|---|
|
||
| `JEV_BASE_URL` | `https://api.typesafe.ai` |
|
||
| `JEV_API_KEY` | _(required for classification)_ |
|
||
| `JEV_MODEL` | `jev-latest` |
|
||
| `EMAIL_AGENT_CLASSIFY_CONCURRENCY` | `4` |
|
||
| `EMAIL_AGENT_CLASSIFY_MAX_MESSAGES` | `200` |
|
||
|
||
## 5. Deployment
|
||
|
||
On `.13`, from the repo, with the host secrets in scope:
|
||
|
||
```sh
|
||
set -a; . ~/.config/environment.d/10-secrets.conf; set +a
|
||
docker compose build
|
||
docker compose up -d # migration v3 runs on start; data volume kept
|
||
docker compose exec -T email-agent email-agent -classify
|
||
```
|
||
|
||
## 6. Rollback
|
||
|
||
1. `docker compose down` on `.13` (keeps `email_agent_data`).
|
||
2. `git revert <this phase commit>` and `docker compose up -d --build`.
|
||
3. Migration v3 is additive (new table + seeded rows) and is not undone by the
|
||
revert; the Phase 2 binary ignores `categories` and shows every message as
|
||
`Unclassified`.
|
||
|
||
## 7. Not done / deferred
|
||
|
||
- The rules **editor UI** is Phase 6 (FR-11). The rules table, versioning, and
|
||
the classifier reading them are live.
|
||
- `POST /actions/reclassify` and `-classify-all` were added so a category or
|
||
rules change can be applied to existing mail; Phase 6 will use the same path.
|
||
- Unsubscribe detection (FR-7) and the reply drafts (FR-8) remain later phases.
|
||
- The 1-month body retention job (AGENTS.md §7) is not implemented yet.
|