sam/email_agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7ca888f0c6e442abe67b576e6206b614b68223e2
email_agent/README.md
Paperclip Builder 7ca888f0c6 Phase 5: reply drafts — template/LLM draft, edit, SMTP send (AUD-71) 
Implements Phase 5 of AGENTS.md §11 (FR-8, FR-15):

- Message detail pane (GET /partials/message) with body, recommended
  action, reply identity, template picker, editable draft, and history.
- internal/reply: OmniRoute (OpenAI-compatible) chat client and a
  template-based draft generator. All non-JEV models go through
  OmniRoute (§10).
- internal/mailops: SMTP send over smtp.gmail.com:587 with STARTTLS and
  the app password, forcing IPv4 (§9); FR-15 reply identity (send as the
  arrived-at X-Pm-Original-To address, else "Sam Rolfe"); threading via
  In-Reply-To/References; -check-smtp safe verification; and a
  [Gmail]/Sent Mail thread check.
- Migration v5 stores messages.message_id_header and drafts.sent_at;
  mailfetch parses Message-ID.
- Send is user-triggered only: CSRF + UI confirm + confirm=yes guard,
  no auto-send, duplicate-send refused. A sent reply files the original
  under Replied-to/<address> and logs every step (FR-13).
- Tests for reply, mailops SMTP/identity/thread check, store drafts and
  migration v5, and the server draft/edit/send guards.
- docs/phase5-replies.md, README, .env.example, docker-compose updated.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-11 22:05:35 +11:00

252 lines
12 KiB
Markdown
Raw Blame History

# Email Agent
Triage the family inbox from one page: all mail lands in a single Gmail account
and this service classifies it, recommends actions, performs bulk actions, drafts
replies, and answers the Hub.
The build spec is [`AGENTS.md`](AGENTS.md). `ORIGINAL_NOTES.md` is archived
pre-planning material and is not authoritative.
## Status
**Phase 5 — Reply drafts.** The Go skeleton, session login, and HTMX dashboard
(Phase 1), real mail over **IMAP with an app password, forced to IPv4** (Phase
2), **JEV classification** (Phase 3), **bulk triage actions** (Phase 4), and now
**reply drafts** (Phase 5). Open a message to read it, draft a reply with the LLM
through **OmniRoute** from a template, edit it, and send it over Gmail SMTP
(IPv4 + STARTTLS). A reply is sent **as the address the mail arrived at**
(`X-Pm-Original-To`, FR-15) and returns to the original sender; the reply sets
`In-Reply-To`/`References` so Gmail keeps the thread, and the original is filed
under its `Replied-to/<address>` label. Sending is a deliberate UI action with a
confirmation — there is no auto-send (§10). Every action, including drafts and
sends, is appended to `actions_log` (FR-13). See
[`docs/phase5-replies.md`](docs/phase5-replies.md) for the phase evidence and
[`docs/phase4-triage.md`](docs/phase4-triage.md) for Phase 4.
## Layout
```
cmd/email-agent/ service entry point (`-fetch`, `-classify`, `-classify-all`)
internal/classify/ JEV (TypeSafe System One) client, rule engine, orchestration
internal/config/ environment-driven configuration
internal/mailfetch/ stdlib-only IMAP client, MIME parsing, FR-2 source address
internal/mailops/ Gmail label moves and flag changes, and SMTP reply sending
internal/reply/ LLM reply drafts through OmniRoute, reply templates
internal/session/ server-side session store + signed cookie values
internal/store/ SQLite open + schema (internal/store/schema.sql)
internal/server/ HTTP routes, auth middleware, HTMX handlers
internal/version/ build metadata (set via -ldflags)
web/static/ embedded design tokens, Inter fonts, htmx, app.css
web/templates/ embedded server-rendered HTML
Dockerfile multi-stage, CGO-free build
docker-compose.yml this project's own container and image
```
## Configuration
All configuration is read from environment variables. Secrets live in
`~/.config/environment.d/10-secrets.conf` on the host and are passed into the
container by `docker-compose`; no secret value is ever committed. See
[`.env.example`](.env.example) for the full list of names.
Key variables:
| Variable | Default | Purpose |
|---|---|---|
| `EMAIL_AGENT_ENV` | `development` | `development`, `production`, or `test` |
| `EMAIL_AGENT_ADDR` | `:8790` | Listen address |
| `EMAIL_AGENT_PORT` | `8790` | Listen port (used when `EMAIL_AGENT_ADDR` is unset) |
| `EMAIL_AGENT_DB_PATH` | `data/email_agent.db` | SQLite file |
| `EMAIL_AGENT_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error` |
| `EMAIL_AGENT_SESSION_SECRET` | _(empty)_ | Session cookie signing key. If unset, an ephemeral key is used (sessions reset on restart) |
| `EMAIL_AGENT_PASSWORD` | _(empty)_ | Login password for Sam. If unset, `EMAIL_AGENT_SESSION_SECRET` doubles as the password |
| `GMAIL_USER` | _(empty)_ | Gmail account that receives the forwarded mail |
| `GMAIL_APP_PASSWORD` | _(empty)_ | Gmail app password for IMAP/SMTP (no OAuth) |
| `EMAIL_AGENT_SMTP_HOST` | `smtp.gmail.com` | SMTP submission host |
| `EMAIL_AGENT_SMTP_PORT` | `587` | SMTP submission port (STARTTLS, IPv4) |
| `EMAIL_AGENT_IMAP_HOST` | `imap.gmail.com` | IMAP host |
| `EMAIL_AGENT_IMAP_PORT` | `993` | IMAP port (implicit TLS) |
| `EMAIL_AGENT_IMAP_LOOKBACK_DAYS` | `14` | First-scan window |
| `EMAIL_AGENT_IMAP_MAX_MESSAGES` | `100` | Per-scan message cap |
| `JEV_BASE_URL` | `https://api.typesafe.ai` | TypeSafe System One base URL |
| `JEV_API_KEY` | _(empty)_ | JEV API key (required to classify) |
| `JEV_MODEL` | `jev-latest` | System One model alias |
| `EMAIL_AGENT_CLASSIFY_CONCURRENCY` | `4` | In-flight JEV requests per run |
| `EMAIL_AGENT_CLASSIFY_MAX_MESSAGES` | `200` | Messages per classify run |
| `OMNIROUTE_BASE_URL` | `http://192.168.20.13:20129/v1` | OmniRoute gateway (all non-JEV models) |
| `OMNIROUTE_API_KEY` | _(empty)_ | OmniRoute API key (required to draft replies) |
| `OMNIROUTE_MODEL` | `auto/best-chat` | Model alias for reply drafts |
## Mail sync (Phase 2)
`internal/mailfetch` speaks the minimum IMAP needed: `LOGIN`, `SELECT INBOX`,
`UID SEARCH SINCE`, and `UID FETCH` of the full RFC822 source plus Gmail's
`X-GM-MSGID`/`X-GM-THRID`. The dialer uses `tcp4` so Gmail is only ever reached
over IPv4 — IPv6 to Gmail is broken from `.13` and previously caused a socket EOF.
Trigger a scan either from the dashboard (**Check mail**) or from the CLI:
```sh
docker exec email-agent email-agent -fetch
```
Each scan is bounded by the lookback and max-message settings. Gmail's
`X-GM-MSGID` is the stable message key; the IMAP `UID` and `UIDVALIDITY` are
stored alongside it for later label operations.
## Classification (Phase 3)
`internal/classify` sends each message to JEV (`POST /v1/systemone`) as three
Choice questions — category, importance, suggested action — with the **live**
category list as the category options, and stores the answer plus JEV's own
confidence. One request per message, bounded concurrency, retry with backoff on
`429`/`529`/`5xx`.
Trigger classification from the dashboard (**Classify** for new mail,
**Re-classify all** after a category or rules change) or from the CLI:
```sh
docker exec email-agent email-agent -classify # pending messages
docker exec email-agent email-agent -classify-all # every stored message
```
### Rules (FR-4)
Rules are plain text, stored versioned in `rules`. One rule per line:
```
<condition> => <effects>
from contains "amazon.com" => category=Newsletter, action=unsubscribe
* => category=Info
```
Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body`
using `contains`, `equals`, `starts_with`, `ends_with`. The first matching rule
wins and overrides only the fields it names. The rules editor UI is Phase 6; the
classifier already reads the stored text.
### Categories (FR-14)
`GET /categories` edits the category list. A category's name is the key stored
in `classifications.category` and the Choice option sent to JEV, so the model
always sees the current list. Renaming cascades to existing classifications;
removing a category reassigns its messages to the first remaining category.
## Triage actions (Phase 4)
`internal/mailops` turns an app folder into a Gmail label and performs the move
over IMAP, forcing IPv4 exactly like the read path:
| App folder | Gmail label |
|---|---|
| `inbox` | `INBOX` |
| `delete` | `Delete` (recoverable; never a hard delete) |
| `saved` | `Saved` |
| `replied` | `Replied-to/<address the mail arrived at>` |
A move uses the IMAP `MOVE` extension (with a `COPY` + `\Deleted` + `EXPUNGE`
fallback). Because Gmail gives each label its own `UID`, a message is located by
its stable `X-GM-MSGID`, not by a stored `UID`. A message in the Delete, Saved,
or Replied-to folder has a **Restore** button that moves it back to `INBOX`.
Every action appends to `actions_log`: one `move` or `mark` row per message,
plus one `bulk_<action>` summary row.
## Reply drafts (Phase 5)
Opening a message in the inbox loads the detail pane: the body, the recommended
action, the reply identity, a template picker, and the draft. The draft is
generated by an LLM through **OmniRoute** (the only non-JEV model path, §10)
using a built-in template (`quick-ack`, `answer`, `confirm`, `decline`) plus any
note Sam types. Sam edits the draft, then sends it.
The send path uses **Gmail SMTP** (`smtp.gmail.com:587`, STARTTLS) and, like the
IMAP read path, **forces IPv4** (AGENTS.md §9).
- **Reply identity (FR-15):** the `From` header is `Sam Rolfe <address the mail
arrived at>`; when the arrived-at address is unknown it falls back to
`Sam Rolfe <GMAIL_USER>`. The SMTP envelope sender is always the authenticated
Gmail account.
- **Threading:** the reply sets `In-Reply-To` and `References` from the original
`Message-ID` (stored by Phase 5), so Gmail keeps the thread. After a send the
service checks the `[Gmail]/Sent Mail` mailbox for the thread and reports
whether it updated.
- **No auto-send (§10):** sending needs a CSRF token, an explicit UI
confirmation, and the guard field `confirm=yes`. The handler refuses a request
without them and never runs from a schedule. A draft is sent once; a second
attempt is refused.
- **Filing:** after a successful send the original moves to its
`Replied-to/<address>` label (FR-9).
Verify the SMTP path without sending mail:
```sh
# inside the container
email-agent -check-smtp # IPv4 + STARTTLS + app-password login, no message sent
```
A real send is a UI action by Sam; it is never triggered by an agent test without
explicit approval (§10, AGENTS.md §13).
## Run locally
```sh
go run ./cmd/email-agent
curl -s localhost:8790/health
```
## Docker
```sh
docker compose up -d --build
curl -s localhost:8790/health
```
The container is `email-agent`, image `email-agent:latest`, named volume
`email_agent_data` mounted at `/data`, listening on **8790**. It runs as an
unprivileged user and has a built-in healthcheck (`email-agent -healthcheck`).
Caddy on `.35` proxies `mail.lab.audasmedia.com.au` to this port.
## Endpoints
| Method | Path | Auth | Purpose |
|---|---|---|---|
| `GET` | `/health` | public | Liveness + database check, JSON |
| `GET` | `/readyz` | public | Readiness, JSON |
| `GET` | `/login` | public | Sign-in form (deep-indigo band) |
| `POST` | `/login` | public | CSRF-checked password sign-in, sets the session cookie |
| `POST` | `/logout` | session | Destroys the session, returns to `/login` |
| `GET` | `/` | session | Dashboard: nav counts, real message list |
| `GET` | `/partials/inbox` | session | HTMX fragment for the centre list |
| `POST` | `/actions/fetch` | session + CSRF | Runs one IMAP scan, returns the refreshed list |
| `POST` | `/actions/classify` | session + CSRF | Classifies pending messages, returns the refreshed list |
| `POST` | `/actions/reclassify` | session + CSRF | Re-classifies every message with the current categories and rules |
| `POST` | `/actions/bulk` | session + CSRF | Bulk delete/move/mark and restore over selected message ids (FR-6/FR-9) |
| `GET` | `/partials/message` | session | HTMX fragment for the message detail, reply draft, and history |
| `POST` | `/actions/draft` | session + CSRF | Generates and stores a reply draft with the LLM (FR-8) |
| `POST` | `/actions/draft/save` | session + CSRF | Saves Sam's edits to a draft |
| `POST` | `/actions/send` | session + CSRF + confirm | Sends a reply over SMTP (FR-8/FR-15); never automatic |
| `GET` | `/categories` | session | Category editor (FR-14) |
| `POST` | `/categories/add` | session + CSRF | Adds a category |
| `POST` | `/categories/rename` | session + CSRF | Renames a category and updates existing classifications |
| `POST` | `/categories/remove` | session + CSRF | Removes a category and reassigns its messages |
| `GET` | `/static/…` | public | Embedded `tokens.css`, `app.css`, Inter fonts, `htmx.min.js` |
Unauthenticated requests to `/` and `/partials/…` redirect to `/login?next=…`.
Form posts use a double-submit CSRF token (`email_agent_csrf` cookie plus a
hidden `csrf` field).
## Database
SQLite is opened with the pure-Go `modernc.org/sqlite` driver (no CGO). The
schema in `internal/store/schema.sql` creates the AGENTS.md §7 tables:
`messages`, `classifications`, `rules`, `actions_log`, `unsubscribe`, `drafts`,
`summaries`, plus `categories`, `schema_migrations`. Schema init is idempotent
and runs on every start. Migration v2 adds `imap_uid` and `imap_uidvalidity` to
`messages`; migration v3 seeds the first category list and a starter rules
version once, so a category removed by the user is not silently re-added;
migration v4 adds `folder` and `mailbox` to `messages` for the Phase 4 triage
folders; migration v5 adds `messages.message_id_header` (reply threading) and
`drafts.sent_at`.
Reference in New Issue View Git Blame Copy Permalink