email_agent/README.md

# Email Agent

Triage the family inbox from one page: all mail lands in a single Gmail account
and this service classifies it, recommends actions, performs bulk actions, drafts
replies, and answers the Hub.

The build spec is [`AGENTS.md`](AGENTS.md). `ORIGINAL_NOTES.md` is archived
pre-planning material and is not authoritative.

## Architecture

```mermaid
flowchart LR
    SAM["Sam<br/>phone or desktop browser"] --> CADDY["Caddy on .35<br/>mail.lab.audasmedia.com.au"]
    CADDY --> UI["HTMX dashboard<br/>login, inbox, rules, reports"]
    UI --> DB[("SQLite<br/>email_agent_data")]
    UI --> GMAIL["Gmail<br/>IMAP :993 / SMTP :587 (IPv4)"]
    UI --> JEV["JEV / TypeSafe<br/>classify"]
    UI --> OMNI["OmniRoute :20129<br/>draft reply"]
    UI --> APPRISE["Apprise :8210<br/>summary email"]
    HUB["Central AI Hub :8788"] --> API["Hub API<br/>/capabilities /ask /notify"]
    API --> DB
    N8N["n8n schedule"] --> JOBS["CLI jobs<br/>-fetch -classify -summary-send"]
    JOBS --> DB
```

Sources: [`docs/email-agent-architecture.mmd`](docs/email-agent-architecture.mmd)
and [`docs/email-agent-map.html`](docs/email-agent-map.html). Interactive map:
<https://maps.lab.audasmedia.com.au/email_agent/docs/>.

## Status

**Complete and deployed.** All eight AGENTS.md §11 phases are done. The Go
skeleton, session login, and HTMX dashboard (Phase 1), real mail over **IMAP with
an app password, forced to IPv4** (Phase 2), **JEV classification** (Phase 3),
**bulk triage actions** (Phase 4), **reply drafts** (Phase 5), **unsubscribe
detection and the rules editor** (Phase 6), and the **daily summary by email plus
the read-only Hub API** (Phase 7) run on `.13` behind Caddy. Phase 8 documents
the project across Gitea, Obsidian, Outline, Vikunja, and the architecture map.

Open a message to read it, draft a reply with the LLM through **OmniRoute** from
a template, edit it, and send it over Gmail SMTP (IPv4 + STARTTLS). A reply is
sent **as the address the mail arrived at** (`X-Pm-Original-To`, FR-15) and
returns to the original sender; the reply sets `In-Reply-To`/`References` so
Gmail keeps the thread, and the original is filed under its
`Replied-to/<address>` label. Sending is a deliberate UI action with a
confirmation — there is no auto-send (§10). The **rules editor** (`/rules`)
saves and versions the plain-text rules that JEV and the reply model read.
**Unsubscribe detection** (`/unsubscribe`) finds RFC 2369/8058 links and body
links, keeps a domain ignore list, and records what Sam marks — it **never
follows a link**. Every action, including drafts, sends, rules edits, and
unsubscribe marks, is appended to `actions_log` (FR-13). **Phase 7** adds the
daily **summary** (`-summary`, `-summary-send`; delivered by email through
Apprise) and the read-only **Hub API** (`/capabilities`, `/ask`, `/notify`).
**Retention** (AGENTS.md §7) is enforced by the bounded `-retain` job: it clears
message bodies older than one month, keeps metadata, and caps the append-only
`actions_log`; see [`docs/retention.md`](docs/retention.md). Phase evidence:
[`docs/phase7-summary-hub.md`](docs/phase7-summary-hub.md),
[`docs/phase6-unsubscribe.md`](docs/phase6-unsubscribe.md),
[`docs/phase5-replies.md`](docs/phase5-replies.md),
[`docs/phase4-triage.md`](docs/phase4-triage.md), and
[`docs/phase8-documentation.md`](docs/phase8-documentation.md).

## Layout

```
cmd/email-agent/        service entry point (`-fetch`, `-classify`, `-classify-all`, `-retain`)
internal/classify/      JEV (TypeSafe System One) client, rule engine, orchestration
internal/config/        environment-driven configuration
internal/mailfetch/      stdlib-only IMAP client, MIME parsing, FR-2 source address
internal/mailops/       Gmail label moves and flag changes, and SMTP reply sending
internal/reply/          LLM reply drafts through OmniRoute, reply templates
internal/session/       server-side session store + signed cookie values
internal/store/         SQLite open + schema (internal/store/schema.sql), retention
internal/server/        HTTP routes, auth middleware, HTMX handlers
internal/unsubscribe/   RFC 2369/8058 + body unsubscribe-link detection (no network)
internal/version/       build metadata (set via -ldflags)
web/static/             embedded design tokens, Inter fonts, htmx, app.css
web/templates/          embedded server-rendered HTML
docs/                   phase evidence, retention, AI context, map, diagrams
docs/outline/           AI-context documents (system, config, code, runbook)
docs/archify/           Archify JSON spec for the architecture map
docs/diagrams/          rendered Mermaid PNG
Dockerfile              multi-stage, CGO-free build
docker-compose.yml      this project's own container and image
```

## Configuration

All configuration is read from environment variables. Secrets live in
`~/.config/environment.d/10-secrets.conf` on the host and are passed into the
container by `docker-compose`; no secret value is ever committed. See
[`.env.example`](.env.example) for the full list of names.

Key variables:

| Variable | Default | Purpose |
|---|---|---|
| `EMAIL_AGENT_ENV` | `development` | `development`, `production`, or `test` |
| `EMAIL_AGENT_ADDR` | `:8790` | Listen address |
| `EMAIL_AGENT_PORT` | `8790` | Listen port (used when `EMAIL_AGENT_ADDR` is unset) |
| `EMAIL_AGENT_DB_PATH` | `data/email_agent.db` | SQLite file |
| `EMAIL_AGENT_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error` |
| `EMAIL_AGENT_SESSION_SECRET` | _(empty)_ | Session cookie signing key. If unset, an ephemeral key is used (sessions reset on restart) |
| `EMAIL_AGENT_PASSWORD` | _(empty)_ | Login password for Sam. If unset, `EMAIL_AGENT_SESSION_SECRET` doubles as the password |
| `GMAIL_USER` | _(empty)_ | Gmail account that receives the forwarded mail |
| `GMAIL_APP_PASSWORD` | _(empty)_ | Gmail app password for IMAP/SMTP (no OAuth) |
| `EMAIL_AGENT_SMTP_HOST` | `smtp.gmail.com` | SMTP submission host |
| `EMAIL_AGENT_SMTP_PORT` | `587` | SMTP submission port (STARTTLS, IPv4) |
| `EMAIL_AGENT_IMAP_HOST` | `imap.gmail.com` | IMAP host |
| `EMAIL_AGENT_IMAP_PORT` | `993` | IMAP port (implicit TLS) |
| `EMAIL_AGENT_IMAP_LOOKBACK_DAYS` | `14` | First-scan window |
| `EMAIL_AGENT_IMAP_MAX_MESSAGES` | `100` | Per-scan message cap |
| `JEV_BASE_URL` | `https://api.typesafe.ai` | TypeSafe System One base URL |
| `JEV_API_KEY` | _(empty)_ | JEV API key (required to classify) |
| `JEV_MODEL` | `jev-latest` | System One model alias |
| `EMAIL_AGENT_CLASSIFY_CONCURRENCY` | `4` | In-flight JEV requests per run |
| `EMAIL_AGENT_CLASSIFY_MAX_MESSAGES` | `200` | Messages per classify run |
| `OMNIROUTE_BASE_URL` | `http://192.168.20.13:20129/v1` | OmniRoute gateway (all non-JEV models) |
| `OMNIROUTE_API_KEY` | _(empty)_ | OmniRoute API key (required to draft replies) |
| `OMNIROUTE_MODEL` | `auto/best-chat` | Model alias for reply drafts |
| `EMAIL_AGENT_RETENTION_DAYS` | `30` | Message-body retention window in days (§7) |
| `EMAIL_AGENT_ACTIONS_LOG_MAX_ROWS` | `50000` | Newest `actions_log` rows kept |

## Mail sync (Phase 2)

`internal/mailfetch` speaks the minimum IMAP needed: `LOGIN`, `SELECT INBOX`,
`UID SEARCH SINCE`, and `UID FETCH` of the full RFC822 source plus Gmail's
`X-GM-MSGID`/`X-GM-THRID`. The dialer uses `tcp4` so Gmail is only ever reached
over IPv4 — IPv6 to Gmail is broken from `.13` and previously caused a socket EOF.

Trigger a scan either from the dashboard (**Check mail**) or from the CLI:

```sh
docker exec email-agent email-agent -fetch
```

Each scan is bounded by the lookback and max-message settings. Gmail's
`X-GM-MSGID` is the stable message key; the IMAP `UID` and `UIDVALIDITY` are
stored alongside it for later label operations.

## Classification (Phase 3)

`internal/classify` sends each message to JEV (`POST /v1/systemone`) as three
Choice questions — category, importance, suggested action — with the **live**
category list as the category options, and stores the answer plus JEV's own
confidence. One request per message, bounded concurrency, retry with backoff on
`429`/`529`/`5xx`.

Trigger classification from the dashboard (**Classify** for new mail,
**Re-classify all** after a category or rules change) or from the CLI:

```sh
docker exec email-agent email-agent -classify      # pending messages
docker exec email-agent email-agent -classify-all  # every stored message
```

### Rules (FR-4)

Rules are plain text, stored versioned in `rules`. One rule per line:

```
<condition> => <effects>

from contains "amazon.com" => category=Newsletter, action=unsubscribe
* => category=Info
```

Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body`
using `contains`, `equals`, `starts_with`, `ends_with`. The first matching rule
wins and overrides only the fields it names. Edit and version the rules at
`/rules` (Phase 6); the classifier and the reply model both read the newest
version.

### Categories (FR-14)

`GET /categories` edits the category list. A category's name is the key stored
in `classifications.category` and the Choice option sent to JEV, so the model
always sees the current list. Renaming cascades to existing classifications;
removing a category reassigns its messages to the first remaining category.

## Triage actions (Phase 4)

`internal/mailops` turns an app folder into a Gmail label and performs the move
over IMAP, forcing IPv4 exactly like the read path:

| App folder | Gmail label |
|---|---|
| `inbox` | `INBOX` |
| `delete` | `Delete` (recoverable; never a hard delete) |
| `saved` | `Saved` |
| `replied` | `Replied-to/<address the mail arrived at>` |

A move uses the IMAP `MOVE` extension (with a `COPY` + `\Deleted` + `EXPUNGE`
fallback). Because Gmail gives each label its own `UID`, a message is located by
its stable `X-GM-MSGID`, not by a stored `UID`. A message in the Delete, Saved,
or Replied-to folder has a **Restore** button that moves it back to `INBOX`.

Every action appends to `actions_log`: one `move` or `mark` row per message,
plus one `bulk_<action>` summary row.

## Reply drafts (Phase 5)

Opening a message in the inbox loads the detail pane: the body, the recommended
action, the reply identity, a template picker, and the draft. The draft is
generated by an LLM through **OmniRoute** (the only non-JEV model path, §10)
using a built-in template (`quick-ack`, `answer`, `confirm`, `decline`) plus any
note Sam types. Sam edits the draft, then sends it.

The send path uses **Gmail SMTP** (`smtp.gmail.com:587`, STARTTLS) and, like the
IMAP read path, **forces IPv4** (AGENTS.md §9).

- **Reply identity (FR-15):** the `From` header is `Sam Rolfe <address the mail
  arrived at>`; when the arrived-at address is unknown it falls back to
  `Sam Rolfe <GMAIL_USER>`. The SMTP envelope sender is always the authenticated
  Gmail account.
- **Threading:** the reply sets `In-Reply-To` and `References` from the original
  `Message-ID` (stored by Phase 5), so Gmail keeps the thread. After a send the
  service checks the `[Gmail]/Sent Mail` mailbox for the thread and reports
  whether it updated.
- **No auto-send (§10):** sending needs a CSRF token, an explicit UI
  confirmation, and the guard field `confirm=yes`. The handler refuses a request
  without them and never runs from a schedule. A draft is sent once; a second
  attempt is refused.
- **Filing:** after a successful send the original moves to its
  `Replied-to/<address>` label (FR-9).

Verify the SMTP path without sending mail:

```sh
# inside the container
email-agent -check-smtp   # IPv4 + STARTTLS + app-password login, no message sent
```

A real send is a UI action by Sam; it is never triggered by an agent test without
explicit approval (§10, AGENTS.md §13).

## Retention (AGENTS.md §7)

AGENTS.md §7 says: *keep message bodies for one month. Logs are append-only and
size-capped.* The bounded `-retain` job enforces both rules:

- **Message bodies.** Every message whose `received_at` is older than
  `EMAIL_AGENT_RETENTION_DAYS` (default 30) has `body_text` set to `NULL`. All
  metadata (sender, subject, source address, classification, folder, labels)
  stays. A second run is a no-op.
- **Action log.** Only the newest `EMAIL_AGENT_ACTIONS_LOG_MAX_ROWS` (default
  50000) rows of the append-only `actions_log` are kept; older rows are deleted.

The job runs in one SQLite transaction, writes one `retain` row to
`actions_log` for the audit trail, and never touches a message inside the
window. It is the CLI `-retain` flag, so it can be scheduled exactly like
`-fetch` and `-classify` and is never run by an HTTP request:

```sh
# inside the container
docker exec email-agent email-agent -retain
```

The decision is recorded in [`docs/retention.md`](docs/retention.md).

## Run locally

```sh
go run ./cmd/email-agent
curl -s localhost:8790/health
```

## Docker

```sh
docker compose up -d --build
curl -s localhost:8790/health
```

The container is `email-agent`, image `email-agent:latest`, named volume
`email_agent_data` mounted at `/data`, listening on **8790**. It runs as an
unprivileged user and has a built-in healthcheck (`email-agent -healthcheck`).
Caddy on `.35` proxies `mail.lab.audasmedia.com.au` to this port.

## Endpoints

| Method | Path | Auth | Purpose |
|---|---|---|---|
| `GET` | `/health` | public | Liveness + database check, JSON |
| `GET` | `/readyz` | public | Readiness, JSON |
| `GET` | `/login` | public | Sign-in form (deep-indigo band) |
| `POST` | `/login` | public | CSRF-checked password sign-in, sets the session cookie |
| `POST` | `/logout` | session | Destroys the session, returns to `/login` |
| `GET` | `/` | session | Dashboard: nav counts, real message list |
| `GET` | `/partials/inbox` | session | HTMX fragment for the centre list |
| `POST` | `/actions/fetch` | session + CSRF | Runs one IMAP scan, returns the refreshed list |
| `POST` | `/actions/classify` | session + CSRF | Classifies pending messages, returns the refreshed list |
| `POST` | `/actions/reclassify` | session + CSRF | Re-classifies every message with the current categories and rules |
| `POST` | `/actions/bulk` | session + CSRF | Bulk delete/move/mark and restore over selected message ids (FR-6/FR-9) |
| `GET` | `/partials/message` | session | HTMX fragment for the message detail, reply draft, and history |
| `POST` | `/actions/draft` | session + CSRF | Generates and stores a reply draft with the LLM (FR-8) |
| `POST` | `/actions/draft/save` | session + CSRF | Saves Sam's edits to a draft |
| `POST` | `/actions/send` | session + CSRF + confirm | Sends a reply over SMTP (FR-8/FR-15); never automatic |
| `GET` | `/categories` | session | Category editor (FR-14) |
| `POST` | `/categories/add` | session + CSRF | Adds a category |
| `POST` | `/categories/rename` | session + CSRF | Renames a category and updates existing classifications |
| `POST` | `/categories/remove` | session + CSRF | Removes a category and reassigns its messages |
| `GET` | `/rules` | session | Rules editor: current text, versions, parse errors (FR-11) |
| `POST` | `/rules/save` | session + CSRF | Saves the rules text as a new version |
| `POST` | `/rules/restore` | session + CSRF | Restores an older version as a new version |
| `POST` | `/rules/reclassify` | session + CSRF | Re-classifies every message with the current rules |
| `GET` | `/unsubscribe` | session | Detected unsubscribe links, ignore list, and records (FR-7) |
| `POST` | `/unsubscribe/ignore` | session + CSRF | Adds a domain to the ignore list |
| `POST` | `/unsubscribe/unignore` | session + CSRF | Removes a domain record |
| `POST` | `/unsubscribe/mark` | session + CSRF + confirm | Records a domain for unsubscribe; never contacts the link |
| `GET` | `/reports` | session | Reports screen: build, send, and delete a daily summary (FR-10) |
| `POST` | `/reports/generate` | session + CSRF | Builds and stores today's summary; sends nothing |
| `POST` | `/reports/send` | session + CSRF + confirm | Delivers today's summary by email via Apprise |
| `POST` | `/reports/delete` | session + CSRF | Deletes a stored summary |
| `GET` | `/capabilities` | Hub bearer | Hub capability list (FR-12) |
| `POST` | `/ask` | Hub bearer | Read-only short answer (FR-12); never writes |
| `GET` | `/static/…` | public | Embedded `tokens.css`, `app.css`, Inter fonts, `htmx.min.js` |

Unauthenticated requests to `/` and `/partials/…` redirect to `/login?next=…`.
Form posts use a double-submit CSRF token (`email_agent_csrf` cookie plus a
hidden `csrf` field).

## Database

SQLite is opened with the pure-Go `modernc.org/sqlite` driver (no CGO). The
schema in `internal/store/schema.sql` creates the AGENTS.md §7 tables:
`messages`, `classifications`, `rules`, `actions_log`, `unsubscribe`, `drafts`,
`summaries`, plus `categories`, `schema_migrations`. Schema init is idempotent
and runs on every start. Migration v2 adds `imap_uid` and `imap_uidvalidity` to
`messages`; migration v3 seeds the first category list and a starter rules
version once, so a category removed by the user is not silently re-added;
migration v4 adds `folder` and `mailbox` to `messages` for the Phase 4 triage
folders; migration v5 adds `messages.message_id_header` (reply threading) and
`drafts.sent_at`; migration v6 adds `messages.list_unsubscribe` and
`messages.list_unsubscribe_post` and extends `unsubscribe` with `url` and
`updated_at`.

## Documentation

Phase 8 recorded the project in five places.

| Surface | Location |
|---|---|
| Gitea | This README, `docs/phaseN-*.md`, `docs/outline/`, `docs/email-agent-architecture.mmd`, `docs/email-agent-map.html` |
| Obsidian | Note `Email Agent — Family Inbox Triage` in `010 inbox` on `.27` (the sorter files it) |
| Outline | Collection `Email Agent` (`00-index`, system overview, config, code map, runbook) |
| Vikunja | Project `Email Agent` (human milestones) |
| Map | `central_ai_llm_master/docs/hub-architecture-map.html` and this repo's map, published to <https://maps.lab.audasmedia.com.au/email_agent/docs/> |

Rebuild the map:

```sh
mmdc -i docs/email-agent-architecture.mmd -o docs/diagrams/email-agent-architecture.png --scale 2 -b white
~/.agents/skills/project-docs/scripts/publish-maps.sh /home/sam/paseo/projects/email_agent
```

See [`docs/phase8-documentation.md`](docs/phase8-documentation.md) for the evidence.