diff --git a/README.md b/README.md index 6ac3fab..6a9be51 100644 --- a/README.md +++ b/README.md @@ -9,18 +9,24 @@ pre-planning material and is not authoritative. ## Status -**Phase 5 — Reply drafts.** The Go skeleton, session login, and HTMX dashboard -(Phase 1), real mail over **IMAP with an app password, forced to IPv4** (Phase -2), **JEV classification** (Phase 3), **bulk triage actions** (Phase 4), and now -**reply drafts** (Phase 5). Open a message to read it, draft a reply with the LLM -through **OmniRoute** from a template, edit it, and send it over Gmail SMTP -(IPv4 + STARTTLS). A reply is sent **as the address the mail arrived at** -(`X-Pm-Original-To`, FR-15) and returns to the original sender; the reply sets -`In-Reply-To`/`References` so Gmail keeps the thread, and the original is filed -under its `Replied-to/
` label. Sending is a deliberate UI action with a -confirmation — there is no auto-send (§10). Every action, including drafts and -sends, is appended to `actions_log` (FR-13). See -[`docs/phase5-replies.md`](docs/phase5-replies.md) for the phase evidence and +**Phase 6 — Unsubscribe and rules editor.** The Go skeleton, session login, +and HTMX dashboard (Phase 1), real mail over **IMAP with an app password, +forced to IPv4** (Phase 2), **JEV classification** (Phase 3), **bulk triage +actions** (Phase 4), **reply drafts** (Phase 5), and now **unsubscribe +detection and the rules editor** (Phase 6). Open a message to read it, draft a +reply with the LLM through **OmniRoute** from a template, edit it, and send it +over Gmail SMTP (IPv4 + STARTTLS). A reply is sent **as the address the mail +arrived at** (`X-Pm-Original-To`, FR-15) and returns to the original sender; the +reply sets `In-Reply-To`/`References` so Gmail keeps the thread, and the +original is filed under its `Replied-to/
` label. Sending is a +deliberate UI action with a confirmation — there is no auto-send (§10). The +**rules editor** (`/rules`) saves and versions the plain-text rules that JEV and +the reply model read. **Unsubscribe detection** (`/unsubscribe`) finds RFC +2369/8058 links and body links, keeps a domain ignore list, and records what Sam +marks — it **never follows a link**. Every action, including drafts, sends, +rules edits, and unsubscribe marks, is appended to `actions_log` (FR-13). See +[`docs/phase6-unsubscribe.md`](docs/phase6-unsubscribe.md) for the phase evidence, +[`docs/phase5-replies.md`](docs/phase5-replies.md) for Phase 5, and [`docs/phase4-triage.md`](docs/phase4-triage.md) for Phase 4. ## Layout @@ -35,6 +41,7 @@ internal/reply/ LLM reply drafts through OmniRoute, reply templates internal/session/ server-side session store + signed cookie values internal/store/ SQLite open + schema (internal/store/schema.sql) internal/server/ HTTP routes, auth middleware, HTMX handlers +internal/unsubscribe/ RFC 2369/8058 + body unsubscribe-link detection (no network) internal/version/ build metadata (set via -ldflags) web/static/ embedded design tokens, Inter fonts, htmx, app.css web/templates/ embedded server-rendered HTML @@ -123,8 +130,9 @@ from contains "amazon.com" => category=Newsletter, action=unsubscribe Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body` using `contains`, `equals`, `starts_with`, `ends_with`. The first matching rule -wins and overrides only the fields it names. The rules editor UI is Phase 6; the -classifier already reads the stored text. +wins and overrides only the fields it names. Edit and version the rules at +`/rules` (Phase 6); the classifier and the reply model both read the newest +version. ### Categories (FR-14) @@ -231,6 +239,14 @@ Caddy on `.35` proxies `mail.lab.audasmedia.com.au` to this port. | `POST` | `/categories/add` | session + CSRF | Adds a category | | `POST` | `/categories/rename` | session + CSRF | Renames a category and updates existing classifications | | `POST` | `/categories/remove` | session + CSRF | Removes a category and reassigns its messages | +| `GET` | `/rules` | session | Rules editor: current text, versions, parse errors (FR-11) | +| `POST` | `/rules/save` | session + CSRF | Saves the rules text as a new version | +| `POST` | `/rules/restore` | session + CSRF | Restores an older version as a new version | +| `POST` | `/rules/reclassify` | session + CSRF | Re-classifies every message with the current rules | +| `GET` | `/unsubscribe` | session | Detected unsubscribe links, ignore list, and records (FR-7) | +| `POST` | `/unsubscribe/ignore` | session + CSRF | Adds a domain to the ignore list | +| `POST` | `/unsubscribe/unignore` | session + CSRF | Removes a domain record | +| `POST` | `/unsubscribe/mark` | session + CSRF + confirm | Records a domain for unsubscribe; never contacts the link | | `GET` | `/static/…` | public | Embedded `tokens.css`, `app.css`, Inter fonts, `htmx.min.js` | Unauthenticated requests to `/` and `/partials/…` redirect to `/login?next=…`. @@ -248,4 +264,6 @@ and runs on every start. Migration v2 adds `imap_uid` and `imap_uidvalidity` to version once, so a category removed by the user is not silently re-added; migration v4 adds `folder` and `mailbox` to `messages` for the Phase 4 triage folders; migration v5 adds `messages.message_id_header` (reply threading) and -`drafts.sent_at`. +`drafts.sent_at`; migration v6 adds `messages.list_unsubscribe` and +`messages.list_unsubscribe_post` and extends `unsubscribe` with `url` and +`updated_at`. diff --git a/docs/phase6-unsubscribe.md b/docs/phase6-unsubscribe.md new file mode 100644 index 0000000..5b7e63d --- /dev/null +++ b/docs/phase6-unsubscribe.md @@ -0,0 +1,203 @@ +# Phase 6 — Unsubscribe and rules editor + +Status: **complete.** Built, tested, deployed on `.13`, and the DoD was +demonstrated against real mail. Parent [AUD-68](/AUD/issues/AUD-68). Phase 6 of +AGENTS.md §11. Commit `00bb0e0`. + +**DoD:** a rule written in the editor changes the classification of a real +message. — met (§2.1). + +## 1. What was built + +### 1.1 Rules editor (FR-4, FR-11) + +- **`/rules`** is a plain-text editor over the `rules` table. Saving appends a + new numbered version (`store.SaveRules`); nothing is overwritten. The page + shows the current version, the active rule count, and a version list. +- **Version history** lists the newest revisions with timestamp, line count, and + a first-rule preview. **Restore** re-saves an older version as a new version, + so history stays append-only. +- **Parse errors** are surfaced on the page. `classify.ParseRules` collects a + bad line in `Errors` and skips it; the editor shows exactly which lines were + ignored and why. +- **Apply** re-runs the classifier over every message (`POST /rules/reclassify`) + and reports considered/classified/failed back on the same page. +- **JEV and the LLM read it.** The classifier already read the rules and applied + the first match as an override. The reply model now reads the same document: + `reply.Request.Rules` is added to the OmniRoute prompt as a bounded + `USER RULES` block (`internal/reply/draft.go`), and `handleDraft` passes the + current rules text. + +### 1.2 Unsubscribe detection and tracking (FR-7) + +- **Capture.** `List-Unsubscribe` (RFC 2369) and `List-Unsubscribe-Post` + (RFC 8058) are parsed by `internal/mailfetch` and stored raw on the message + (`messages.list_unsubscribe`, `messages.list_unsubscribe_post`). They are kept + raw so detection can be re-run without a refetch. +- **Detect.** `internal/unsubscribe` parses angle-bracketed and bare URIs into + `http`/`mailto` links, recognises the one-click header, de-duplicates, and + falls back to a body scan when there is no header (a URL on a line that + mentions "unsubscribe"). The ignore-list key is the sender's brand domain + when it is a real domain, with the link host as the fallback for free-mail + senders (so a mailto endpoint at `list-manage.com` does not group unrelated + senders). +- **View and act.** `/unsubscribe` lists the messages with a detected link, the + domain-level **ignore list**, and the domains **recorded for unsubscribe**. + Sam can ignore a domain, un-ignore it, or record it for unsubscribe. Every + action is a CSRF-protected POST and is written to `actions_log` (FR-13). +- **No link is ever followed.** The app makes no network call to an unsubscribe + URL and sends no mail. "Mark for unsubscribe" records the decision only; the + page states this plainly, and following a real link stays a manual step that + needs Sam's explicit approval. A pending RFC 8058 one-click POST is detected + and shown, but never sent. + +### 1.3 Schema (migration v6, additive) + +| Object | Change | +|---|---| +| `messages` | `list_unsubscribe`, `list_unsubscribe_post` (TEXT) | +| `unsubscribe` | `url` (TEXT NOT NULL DEFAULT ''), `updated_at` (TEXT) | +| `schema_migrations` | records version 6 | + +The `unsubscribe` table keeps its `status` values `done` (recorded) and +`ignored` (ignore list). Migration v6 is additive; a Phase 5 binary ignores the +new columns. + +## 2. DoD evidence + +### 2.1 A rule written in the editor changed a real message + +Real message **87**, from `Heritage Fruit Trees `. + +**Before.** Classified by JEV with no rule (rules version 1): + +| category | importance | action | confidence | rules_version | +|---|---|---|---|---| +| Newsletter | low | unsubscribe | 0.98 | 1 | + +**Rule written in the editor.** `POST /rules/save` with: + +``` +# Phase 6 DoD rule (AUD-72) +from contains "heritagefruittrees.com.au" => category=Action, importance=high, action=file +``` + +The page then showed **`version 2 · 1 active rule`**. + +**After.** `POST /rules/reclassify` returned +`Re-classified 103 of 103 message(s); 0 failed. The current rules are now applied.` +Message 87's newest classification row is now: + +| category | importance | action | confidence | model | rules_version | +|---|---|---|---|---|---| +| Action | high | file | 1.00 | jev-1.13.0 | 2 | + +The per-message action log records the override and the matching rule line: + +``` +classify category=Action importance=high action=file confidence=1.00 + model=jev-1.13.0 rules_version=2 rule_line=2 rule="from contains ..." +``` + +Message 87's history therefore shows the change: three `Newsletter/low/ +unsubscribe` rows under `rules_version=1`, then `Action/high/file` under +`rules_version=2`. + +### 2.2 Unsubscribe detection + +`docker exec email-agent email-agent -fetch` refreshed 100 messages, filling the +new header columns. **31 messages carried a `List-Unsubscribe` header.** The live +`/unsubscribe` page reported **33 messages with an unsubscribe link** — the 31 +header matches plus 2 body-fallback matches (for example `allakiyas.com`). + +Example detected domains in the live page: `google.com`, `spotify.com`, +`subaru.com.au`, `linkedin.com`, `email.centrecom.com.au`, `r.repco.com.au`, +`ozmobiles.com.au`, `heritagefruittrees.com.au`, `edm.booktopia.com.au`, +`netflix.com`, `urbnsurf.com`. + +A `mailto:` example (`Google Alerts`) and a web example (message 87, +`https://drdzlr.clicks.mlsend.com/...`) both render in the list and in the +message detail pane. + +### 2.3 Ignore list and record + +Demonstrated on the live UI with `heritagefruittrees.com.au`, then reverted (the +domain-level row is deleted; the append-only log keeps the history): + +| Step | Result | +|---|---| +| `POST /unsubscribe/ignore` | 303; page shows `Ignore list · 1 domain(s)` with an `ignored` pill and the recorded URL | +| `POST /unsubscribe/mark` `confirm=yes` | 303; page shows `Recorded for unsubscribe · 1 domain(s)` with a `recorded` pill, the URL, and a timestamp, and the note "The app has not contacted any link." | +| `POST /unsubscribe/unignore` | 303; domain removed, `unsubscribe` table back to 0 rows | + +Durable `actions_log` rows (FR-13): + +``` +rules_save version=2 rules=1 parse_errors=0 +reclassify considered=103 classified=103 failed=0 rules_version=2 +unsubscribe_ignore domain=heritagefruittrees.com.au +unsubscribe_marked domain=heritagefruittrees.com.au url=https://drdzlr.clicks.mlsend.com/... +unsubscribe_unignore domain=heritagefruittrees.com.au +``` + +No email was sent, and no unsubscribe URL was contacted. + +### 2.4 Live deployment + +| Check | Result | +|---|---| +| `GET /health` | `200 {"status":"ok","version":"phase6","commit":"00bb0e0","database":"ok"}` | +| Container log | `database ready … schema_version: 6` (migration v6 ran in place on the Phase 1–5 volume) | +| Login over public HTTPS | `https://mail.lab.audasmedia.com.au/health` → 200; `/rules` → 303 to `/login` (auth enforced) | +| Dashboard | shows `Rules` and `Unsubscribe` links | +| Message detail (87) | shows `action-pill">file` and the detected unsubscribe link | + +### 2.5 Automated tests + +`gofmt -l` clean, `go vet ./...` clean, `go test ./...` green (in +`golang:1.25-alpine`). New coverage: + +| Package | Tests | +|---|---| +| `internal/unsubscribe` | header parsing (mailto + http, bare, one-click, de-dup), body fallback, unrelated-link rejection, free-mail domain fallback, `Domain()` | +| `internal/mailfetch` | `ParseMessage` captures both List-Unsubscribe headers and the source address | +| `internal/reply` | the rules document is included in the draft prompt | +| `internal/store` | migration v6 on a hand-built v5 database; unsubscribe record upsert/list/clear and status validation; candidate selection (header + body); rules version list + restore lookup | +| `internal/server` | rules page renders; save creates a version; CSRF required; parse errors shown; restore creates a new version; re-classify redirects without JEV; unsubscribe detection renders; ignore/un-ignore; mark requires confirmation and records the URL; CSRF on every mutation; bad domain rejected; message detail shows the link | + +## 3. Configuration + +No new environment variables. The rules editor, unsubscribe detection, and +tracking work without any new credentials. Re-classify needs `JEV_API_KEY` +(already configured); reply drafting that reads the rules needs +`OMNIROUTE_API_KEY` (already configured). No secret value is committed. + +## 4. Deployment + +On `.13`, from the repo, with the host secrets in scope: + +```sh +set -a; . ~/.config/environment.d/10-secrets.conf; set +a +docker compose build +docker compose up -d # migration v6 runs on start; data volume kept +docker exec email-agent email-agent -fetch # fills the List-Unsubscribe columns +``` + +## 5. Rollback + +1. `git revert 00bb0e0 && docker compose up -d --build`. +2. Migration v6 is additive; the Phase 5 binary ignores the new columns and the + new tables. The `unsubscribe` table and `actions_log` rows are left in place. +3. Reverting the code does not un-save a rules version. Restore the intended + version from `/rules`, or re-save it. + +## 6. Not done / deferred + +- **Following an unsubscribe link is not implemented.** This is deliberate + (AUD-72): the app only marks and records. Acting on a real link — an RFC 8058 + one-click POST, a mailto, or a web form — needs Sam's explicit approval and is + a separate, manually approved step. +- The ignore-list key is the sender's domain as seen in `From:`. Different + subdomains of one brand are tracked separately; registrable-domain grouping + (a public suffix list) is not implemented. +- **1-month body retention** (§7) is still not implemented.