sam/email_agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 3: JEV classification, rules table, editable categories (AUD-69)

Browse Source 
Implements FR-3, FR-4, FR-5 and FR-14.

- internal/classify: TypeSafe System One (JEV) HTTP client with the confirmed
  request/response shape, retry with backoff on 429/529/5xx, a plain-text rule
  engine (first match wins, overrides only the named fields), and bounded
  orchestration with a per-run cap.
- internal/store: migration v3 adds an editable `categories` table (name =
  JEV Choice option and classifications key; rename cascades, remove reassigns)
  and seeds a starter rules version once. Classification persistence, coverage,
  and pending/all message queries.
- internal/server: POST /actions/classify and /actions/reclassify, plus the
  FR-14 category editor (GET /categories and add/rename/remove). The triage list
  now groups by the live category list and shows action + confidence.
- CLI: -classify and -classify-all for n8n.
- Docs: docs/phase3-classify.md with the confirmed JEV shape, a live read-only
  classification, and the DoD evidence.

DoD: all 100 stored messages have a non-null category, importance, and suggested
action (0 nulls); JEV picked a newly added/renamed category for 6 real messages
over HTTP. One live JEV call recorded in the phase doc.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Sam Rolfe
2026-10-11 21:37:38 +11:00
parent bf43b6bbef
commit af3b5e5f0e
27 changed files with 3235 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

80
README.md
View File

@@ -9,18 +9,23 @@ pre-planning material and is not authoritative.
## Status
**Phase 2 — Read mail (IMAP).** The Go skeleton (Phase 1.1) plus session login
and the HTMX dashboard shell (Phase 1.2), now fetching real mail from Gmail over
**IMAP with an app password, forced to IPv4** (AGENTS.md §9). Messages are stored
in SQLite and listed in the dashboard, grouped by category with a recommended
action per row. The FR-2 source address is resolved from `X-Pm-Original-To`, with
`To`, the PurelyMail `Received … for` clause, and `Return-Path` as fallbacks;
`Delivered-To` is never used. Classification and triage actions come in Phase 3+.
**Phase 3 — Classify.** The Go skeleton, session login, and HTMX dashboard
(Phase 1), real mail over **IMAP with an app password, forced to IPv4**
(Phase 2), and now **JEV classification** (Phase 3). Every message gets a
category, an importance, and a suggested action, grouped in the triage list with
the recommended action and confidence per row. The category list is editable in
the UI (add, rename, remove) and JEV uses the current list; a plain-text rules
table can override the classifier (FR-4). The FR-2 source address is resolved
from `X-Pm-Original-To`, with `To`, the PurelyMail `Received … for` clause, and
`Return-Path` as fallbacks; `Delivered-To` is never used. See
[`docs/phase3-classify.md`](docs/phase3-classify.md) for the confirmed JEV shape
and the phase evidence. Triage actions come in Phase 4.
## Layout
```
cmd/email-agent/ service entry point (`-fetch` runs one scan and exits)
cmd/email-agent/ service entry point (`-fetch`, `-classify`, `-classify-all`)
internal/classify/ JEV (TypeSafe System One) client, rule engine, orchestration
internal/config/ environment-driven configuration
internal/mailfetch/ stdlib-only IMAP client, MIME parsing, FR-2 source address
internal/session/ server-side session store + signed cookie values
@@ -57,6 +62,11 @@ Key variables:
| `EMAIL_AGENT_IMAP_PORT` | `993` | IMAP port (implicit TLS) |
| `EMAIL_AGENT_IMAP_LOOKBACK_DAYS` | `14` | First-scan window |
| `EMAIL_AGENT_IMAP_MAX_MESSAGES` | `100` | Per-scan message cap |
| `JEV_BASE_URL` | `https://api.typesafe.ai` | TypeSafe System One base URL |
| `JEV_API_KEY` | _(empty)_ | JEV API key (required to classify) |
| `JEV_MODEL` | `jev-latest` | System One model alias |
| `EMAIL_AGENT_CLASSIFY_CONCURRENCY` | `4` | In-flight JEV requests per run |
| `EMAIL_AGENT_CLASSIFY_MAX_MESSAGES` | `200` | Messages per classify run |
## Mail sync (Phase 2)
@@ -75,6 +85,45 @@ Each scan is bounded by the lookback and max-message settings. Gmail's
`X-GM-MSGID` is the stable message key; the IMAP `UID` and `UIDVALIDITY` are
stored alongside it for later label operations.
## Classification (Phase 3)
`internal/classify` sends each message to JEV (`POST /v1/systemone`) as three
Choice questions — category, importance, suggested action — with the **live**
category list as the category options, and stores the answer plus JEV's own
confidence. One request per message, bounded concurrency, retry with backoff on
`429`/`529`/`5xx`.
Trigger classification from the dashboard (**Classify** for new mail,
**Re-classify all** after a category or rules change) or from the CLI:
```sh
docker exec email-agent email-agent -classify # pending messages
docker exec email-agent email-agent -classify-all # every stored message
```
### Rules (FR-4)
Rules are plain text, stored versioned in `rules`. One rule per line:
```
<condition> => <effects>
from contains "amazon.com" => category=Newsletter, action=unsubscribe
* => category=Info
```
Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body`
using `contains`, `equals`, `starts_with`, `ends_with`. The first matching rule
wins and overrides only the fields it names. The rules editor UI is Phase 6; the
classifier already reads the stored text.
### Categories (FR-14)
`GET /categories` edits the category list. A category's name is the key stored
in `classifications.category` and the Choice option sent to JEV, so the model
always sees the current list. Renaming cascades to existing classifications;
removing a category reassigns its messages to the first remaining category.
## Run locally
```sh
@@ -94,7 +143,7 @@ The container is `email-agent`, image `email-agent:latest`, named volume
unprivileged user and has a built-in healthcheck (`email-agent -healthcheck`).
Caddy on `.35` proxies `mail.lab.audasmedia.com.au` to this port.
## Endpoints (Phase 1)
## Endpoints
| Method | Path | Auth | Purpose |
|---|---|---|---|
@@ -106,6 +155,12 @@ Caddy on `.35` proxies `mail.lab.audasmedia.com.au` to this port.
| `GET` | `/` | session | Dashboard: nav counts, real message list |
| `GET` | `/partials/inbox` | session | HTMX fragment for the centre list |
| `POST` | `/actions/fetch` | session + CSRF | Runs one IMAP scan, returns the refreshed list |
| `POST` | `/actions/classify` | session + CSRF | Classifies pending messages, returns the refreshed list |
| `POST` | `/actions/reclassify` | session + CSRF | Re-classifies every message with the current categories and rules |
| `GET` | `/categories` | session | Category editor (FR-14) |
| `POST` | `/categories/add` | session + CSRF | Adds a category |
| `POST` | `/categories/rename` | session + CSRF | Renames a category and updates existing classifications |
| `POST` | `/categories/remove` | session + CSRF | Removes a category and reassigns its messages |
| `GET` | `/static/…` | public | Embedded `tokens.css`, `app.css`, Inter fonts, `htmx.min.js` |
Unauthenticated requests to `/` and `/partials/…` redirect to `/login?next=…`.
@@ -117,6 +172,7 @@ hidden `csrf` field).
SQLite is opened with the pure-Go `modernc.org/sqlite` driver (no CGO). The
schema in `internal/store/schema.sql` creates the AGENTS.md §7 tables:
`messages`, `classifications`, `rules`, `actions_log`, `unsubscribe`, `drafts`,
`summaries`, plus `schema_migrations`. Schema init is idempotent and runs on
every start. Migration v2 adds `imap_uid` and `imap_uidvalidity` to `messages`
for databases created by Phase 1.
`summaries`, plus `categories`, `schema_migrations`. Schema init is idempotent
and runs on every start. Migration v2 adds `imap_uid` and `imap_uidvalidity` to
`messages`; migration v3 seeds the first category list and a starter rules
version once, so a category removed by the user is not silently re-added.