Sam Rolfe
af3b5e5f0e
Implements FR-3, FR-4, FR-5 and FR-14. - internal/classify: TypeSafe System One (JEV) HTTP client with the confirmed request/response shape, retry with backoff on 429/529/5xx, a plain-text rule engine (first match wins, overrides only the named fields), and bounded orchestration with a per-run cap. - internal/store: migration v3 adds an editable `categories` table (name = JEV Choice option and classifications key; rename cascades, remove reassigns) and seeds a starter rules version once. Classification persistence, coverage, and pending/all message queries. - internal/server: POST /actions/classify and /actions/reclassify, plus the FR-14 category editor (GET /categories and add/rename/remove). The triage list now groups by the live category list and shows action + confidence. - CLI: -classify and -classify-all for n8n. - Docs: docs/phase3-classify.md with the confirmed JEV shape, a live read-only classification, and the DoD evidence. DoD: all 100 stored messages have a non-null category, importance, and suggested action (0 nulls); JEV picked a newly added/renamed category for 6 real messages over HTTP. One live JEV call recorded in the phase doc. Co-Authored-By: Paperclip <noreply@paperclip.ing>
179 lines
8.2 KiB
Markdown
179 lines
8.2 KiB
Markdown
# Email Agent
|
|
|
|
Triage the family inbox from one page: all mail lands in a single Gmail account
|
|
and this service classifies it, recommends actions, performs bulk actions, drafts
|
|
replies, and answers the Hub.
|
|
|
|
The build spec is [`AGENTS.md`](AGENTS.md). `ORIGINAL_NOTES.md` is archived
|
|
pre-planning material and is not authoritative.
|
|
|
|
## Status
|
|
|
|
**Phase 3 — Classify.** The Go skeleton, session login, and HTMX dashboard
|
|
(Phase 1), real mail over **IMAP with an app password, forced to IPv4**
|
|
(Phase 2), and now **JEV classification** (Phase 3). Every message gets a
|
|
category, an importance, and a suggested action, grouped in the triage list with
|
|
the recommended action and confidence per row. The category list is editable in
|
|
the UI (add, rename, remove) and JEV uses the current list; a plain-text rules
|
|
table can override the classifier (FR-4). The FR-2 source address is resolved
|
|
from `X-Pm-Original-To`, with `To`, the PurelyMail `Received … for` clause, and
|
|
`Return-Path` as fallbacks; `Delivered-To` is never used. See
|
|
[`docs/phase3-classify.md`](docs/phase3-classify.md) for the confirmed JEV shape
|
|
and the phase evidence. Triage actions come in Phase 4.
|
|
|
|
## Layout
|
|
|
|
```
|
|
cmd/email-agent/ service entry point (`-fetch`, `-classify`, `-classify-all`)
|
|
internal/classify/ JEV (TypeSafe System One) client, rule engine, orchestration
|
|
internal/config/ environment-driven configuration
|
|
internal/mailfetch/ stdlib-only IMAP client, MIME parsing, FR-2 source address
|
|
internal/session/ server-side session store + signed cookie values
|
|
internal/store/ SQLite open + schema (internal/store/schema.sql)
|
|
internal/server/ HTTP routes, auth middleware, HTMX handlers
|
|
internal/version/ build metadata (set via -ldflags)
|
|
web/static/ embedded design tokens, Inter fonts, htmx, app.css
|
|
web/templates/ embedded server-rendered HTML
|
|
Dockerfile multi-stage, CGO-free build
|
|
docker-compose.yml this project's own container and image
|
|
```
|
|
|
|
## Configuration
|
|
|
|
All configuration is read from environment variables. Secrets live in
|
|
`~/.config/environment.d/10-secrets.conf` on the host and are passed into the
|
|
container by `docker-compose`; no secret value is ever committed. See
|
|
[`.env.example`](.env.example) for the full list of names.
|
|
|
|
Key variables:
|
|
|
|
| Variable | Default | Purpose |
|
|
|---|---|---|
|
|
| `EMAIL_AGENT_ENV` | `development` | `development`, `production`, or `test` |
|
|
| `EMAIL_AGENT_ADDR` | `:8790` | Listen address |
|
|
| `EMAIL_AGENT_PORT` | `8790` | Listen port (used when `EMAIL_AGENT_ADDR` is unset) |
|
|
| `EMAIL_AGENT_DB_PATH` | `data/email_agent.db` | SQLite file |
|
|
| `EMAIL_AGENT_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error` |
|
|
| `EMAIL_AGENT_SESSION_SECRET` | _(empty)_ | Session cookie signing key. If unset, an ephemeral key is used (sessions reset on restart) |
|
|
| `EMAIL_AGENT_PASSWORD` | _(empty)_ | Login password for Sam. If unset, `EMAIL_AGENT_SESSION_SECRET` doubles as the password |
|
|
| `GMAIL_USER` | _(empty)_ | Gmail account that receives the forwarded mail |
|
|
| `GMAIL_APP_PASSWORD` | _(empty)_ | Gmail app password for IMAP/SMTP (no OAuth) |
|
|
| `EMAIL_AGENT_IMAP_HOST` | `imap.gmail.com` | IMAP host |
|
|
| `EMAIL_AGENT_IMAP_PORT` | `993` | IMAP port (implicit TLS) |
|
|
| `EMAIL_AGENT_IMAP_LOOKBACK_DAYS` | `14` | First-scan window |
|
|
| `EMAIL_AGENT_IMAP_MAX_MESSAGES` | `100` | Per-scan message cap |
|
|
| `JEV_BASE_URL` | `https://api.typesafe.ai` | TypeSafe System One base URL |
|
|
| `JEV_API_KEY` | _(empty)_ | JEV API key (required to classify) |
|
|
| `JEV_MODEL` | `jev-latest` | System One model alias |
|
|
| `EMAIL_AGENT_CLASSIFY_CONCURRENCY` | `4` | In-flight JEV requests per run |
|
|
| `EMAIL_AGENT_CLASSIFY_MAX_MESSAGES` | `200` | Messages per classify run |
|
|
|
|
## Mail sync (Phase 2)
|
|
|
|
`internal/mailfetch` speaks the minimum IMAP needed: `LOGIN`, `SELECT INBOX`,
|
|
`UID SEARCH SINCE`, and `UID FETCH` of the full RFC822 source plus Gmail's
|
|
`X-GM-MSGID`/`X-GM-THRID`. The dialer uses `tcp4` so Gmail is only ever reached
|
|
over IPv4 — IPv6 to Gmail is broken from `.13` and previously caused a socket EOF.
|
|
|
|
Trigger a scan either from the dashboard (**Check mail**) or from the CLI:
|
|
|
|
```sh
|
|
docker exec email-agent email-agent -fetch
|
|
```
|
|
|
|
Each scan is bounded by the lookback and max-message settings. Gmail's
|
|
`X-GM-MSGID` is the stable message key; the IMAP `UID` and `UIDVALIDITY` are
|
|
stored alongside it for later label operations.
|
|
|
|
## Classification (Phase 3)
|
|
|
|
`internal/classify` sends each message to JEV (`POST /v1/systemone`) as three
|
|
Choice questions — category, importance, suggested action — with the **live**
|
|
category list as the category options, and stores the answer plus JEV's own
|
|
confidence. One request per message, bounded concurrency, retry with backoff on
|
|
`429`/`529`/`5xx`.
|
|
|
|
Trigger classification from the dashboard (**Classify** for new mail,
|
|
**Re-classify all** after a category or rules change) or from the CLI:
|
|
|
|
```sh
|
|
docker exec email-agent email-agent -classify # pending messages
|
|
docker exec email-agent email-agent -classify-all # every stored message
|
|
```
|
|
|
|
### Rules (FR-4)
|
|
|
|
Rules are plain text, stored versioned in `rules`. One rule per line:
|
|
|
|
```
|
|
<condition> => <effects>
|
|
|
|
from contains "amazon.com" => category=Newsletter, action=unsubscribe
|
|
* => category=Info
|
|
```
|
|
|
|
Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body`
|
|
using `contains`, `equals`, `starts_with`, `ends_with`. The first matching rule
|
|
wins and overrides only the fields it names. The rules editor UI is Phase 6; the
|
|
classifier already reads the stored text.
|
|
|
|
### Categories (FR-14)
|
|
|
|
`GET /categories` edits the category list. A category's name is the key stored
|
|
in `classifications.category` and the Choice option sent to JEV, so the model
|
|
always sees the current list. Renaming cascades to existing classifications;
|
|
removing a category reassigns its messages to the first remaining category.
|
|
|
|
## Run locally
|
|
|
|
```sh
|
|
go run ./cmd/email-agent
|
|
curl -s localhost:8790/health
|
|
```
|
|
|
|
## Docker
|
|
|
|
```sh
|
|
docker compose up -d --build
|
|
curl -s localhost:8790/health
|
|
```
|
|
|
|
The container is `email-agent`, image `email-agent:latest`, named volume
|
|
`email_agent_data` mounted at `/data`, listening on **8790**. It runs as an
|
|
unprivileged user and has a built-in healthcheck (`email-agent -healthcheck`).
|
|
Caddy on `.35` proxies `mail.lab.audasmedia.com.au` to this port.
|
|
|
|
## Endpoints
|
|
|
|
| Method | Path | Auth | Purpose |
|
|
|---|---|---|---|
|
|
| `GET` | `/health` | public | Liveness + database check, JSON |
|
|
| `GET` | `/readyz` | public | Readiness, JSON |
|
|
| `GET` | `/login` | public | Sign-in form (deep-indigo band) |
|
|
| `POST` | `/login` | public | CSRF-checked password sign-in, sets the session cookie |
|
|
| `POST` | `/logout` | session | Destroys the session, returns to `/login` |
|
|
| `GET` | `/` | session | Dashboard: nav counts, real message list |
|
|
| `GET` | `/partials/inbox` | session | HTMX fragment for the centre list |
|
|
| `POST` | `/actions/fetch` | session + CSRF | Runs one IMAP scan, returns the refreshed list |
|
|
| `POST` | `/actions/classify` | session + CSRF | Classifies pending messages, returns the refreshed list |
|
|
| `POST` | `/actions/reclassify` | session + CSRF | Re-classifies every message with the current categories and rules |
|
|
| `GET` | `/categories` | session | Category editor (FR-14) |
|
|
| `POST` | `/categories/add` | session + CSRF | Adds a category |
|
|
| `POST` | `/categories/rename` | session + CSRF | Renames a category and updates existing classifications |
|
|
| `POST` | `/categories/remove` | session + CSRF | Removes a category and reassigns its messages |
|
|
| `GET` | `/static/…` | public | Embedded `tokens.css`, `app.css`, Inter fonts, `htmx.min.js` |
|
|
|
|
Unauthenticated requests to `/` and `/partials/…` redirect to `/login?next=…`.
|
|
Form posts use a double-submit CSRF token (`email_agent_csrf` cookie plus a
|
|
hidden `csrf` field).
|
|
|
|
## Database
|
|
|
|
SQLite is opened with the pure-Go `modernc.org/sqlite` driver (no CGO). The
|
|
schema in `internal/store/schema.sql` creates the AGENTS.md §7 tables:
|
|
`messages`, `classifications`, `rules`, `actions_log`, `unsubscribe`, `drafts`,
|
|
`summaries`, plus `categories`, `schema_migrations`. Schema init is idempotent
|
|
and runs on every start. Migration v2 adds `imap_uid` and `imap_uidvalidity` to
|
|
`messages`; migration v3 seeds the first category list and a starter rules
|
|
version once, so a category removed by the user is not silently re-added.
|