sam/email_agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Phase 6 unsubscribe + rules editor evidence (AUD-72)

Browse Source 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Sam Rolfe
2026-10-11 22:26:55 +11:00
parent 00bb0e05c4
commit c4213ebed1
2 changed files with 236 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
README.md
View File

@@ -9,18 +9,24 @@ pre-planning material and is not authoritative.
## Status ## Status
**Phase 5 — Reply drafts.** The Go skeleton, session login, and HTMX dashboard **Phase 6 — Unsubscribe and rules editor.** The Go skeleton, session login,
(Phase 1), real mail over **IMAP with an app password, forced to IPv4** (Phase and HTMX dashboard (Phase 1), real mail over **IMAP with an app password,
2), **JEV classification** (Phase 3), **bulk triage actions** (Phase 4), and now forced to IPv4** (Phase 2), **JEV classification** (Phase 3), **bulk triage
**reply drafts** (Phase 5). Open a message to read it, draft a reply with the LLM actions** (Phase 4), **reply drafts** (Phase 5), and now **unsubscribe
through **OmniRoute** from a template, edit it, and send it over Gmail SMTP detection and the rules editor** (Phase 6). Open a message to read it, draft a
(IPv4 + STARTTLS). A reply is sent **as the address the mail arrived at** reply with the LLM through **OmniRoute** from a template, edit it, and send it
(`X-Pm-Original-To`, FR-15) and returns to the original sender; the reply sets over Gmail SMTP (IPv4 + STARTTLS). A reply is sent **as the address the mail
`In-Reply-To`/`References` so Gmail keeps the thread, and the original is filed arrived at** (`X-Pm-Original-To`, FR-15) and returns to the original sender; the
under its `Replied-to/<address>` label. Sending is a deliberate UI action with a reply sets `In-Reply-To`/`References` so Gmail keeps the thread, and the
confirmation — there is no auto-send (§10). Every action, including drafts and original is filed under its `Replied-to/<address>` label. Sending is a
sends, is appended to `actions_log` (FR-13). See deliberate UI action with a confirmation — there is no auto-send (§10). The
[`docs/phase5-replies.md`](docs/phase5-replies.md) for the phase evidence and **rules editor** (`/rules`) saves and versions the plain-text rules that JEV and
the reply model read. **Unsubscribe detection** (`/unsubscribe`) finds RFC
2369/8058 links and body links, keeps a domain ignore list, and records what Sam
marks — it **never follows a link**. Every action, including drafts, sends,
rules edits, and unsubscribe marks, is appended to `actions_log` (FR-13). See
[`docs/phase6-unsubscribe.md`](docs/phase6-unsubscribe.md) for the phase evidence,
[`docs/phase5-replies.md`](docs/phase5-replies.md) for Phase 5, and
[`docs/phase4-triage.md`](docs/phase4-triage.md) for Phase 4. [`docs/phase4-triage.md`](docs/phase4-triage.md) for Phase 4.
## Layout ## Layout
@@ -35,6 +41,7 @@ internal/reply/ LLM reply drafts through OmniRoute, reply templates
internal/session/ server-side session store + signed cookie values internal/session/ server-side session store + signed cookie values
internal/store/ SQLite open + schema (internal/store/schema.sql) internal/store/ SQLite open + schema (internal/store/schema.sql)
internal/server/ HTTP routes, auth middleware, HTMX handlers internal/server/ HTTP routes, auth middleware, HTMX handlers
internal/unsubscribe/ RFC 2369/8058 + body unsubscribe-link detection (no network)
internal/version/ build metadata (set via -ldflags) internal/version/ build metadata (set via -ldflags)
web/static/ embedded design tokens, Inter fonts, htmx, app.css web/static/ embedded design tokens, Inter fonts, htmx, app.css
web/templates/ embedded server-rendered HTML web/templates/ embedded server-rendered HTML
@@ -123,8 +130,9 @@ from contains "amazon.com" => category=Newsletter, action=unsubscribe
Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body` Conditions join with ` and ` over `from`, `to`, `subject`, `source`, `body`
using `contains`, `equals`, `starts_with`, `ends_with`. The first matching rule using `contains`, `equals`, `starts_with`, `ends_with`. The first matching rule
wins and overrides only the fields it names. The rules editor UI is Phase 6; the wins and overrides only the fields it names. Edit and version the rules at
classifier already reads the stored text. `/rules` (Phase 6); the classifier and the reply model both read the newest
version.
### Categories (FR-14) ### Categories (FR-14)
@@ -231,6 +239,14 @@ Caddy on `.35` proxies `mail.lab.audasmedia.com.au` to this port.
| `POST` | `/categories/add` | session + CSRF | Adds a category | | `POST` | `/categories/add` | session + CSRF | Adds a category |
| `POST` | `/categories/rename` | session + CSRF | Renames a category and updates existing classifications | | `POST` | `/categories/rename` | session + CSRF | Renames a category and updates existing classifications |
| `POST` | `/categories/remove` | session + CSRF | Removes a category and reassigns its messages | | `POST` | `/categories/remove` | session + CSRF | Removes a category and reassigns its messages |
| `GET` | `/rules` | session | Rules editor: current text, versions, parse errors (FR-11) |
| `POST` | `/rules/save` | session + CSRF | Saves the rules text as a new version |
| `POST` | `/rules/restore` | session + CSRF | Restores an older version as a new version |
| `POST` | `/rules/reclassify` | session + CSRF | Re-classifies every message with the current rules |
| `GET` | `/unsubscribe` | session | Detected unsubscribe links, ignore list, and records (FR-7) |
| `POST` | `/unsubscribe/ignore` | session + CSRF | Adds a domain to the ignore list |
| `POST` | `/unsubscribe/unignore` | session + CSRF | Removes a domain record |
| `POST` | `/unsubscribe/mark` | session + CSRF + confirm | Records a domain for unsubscribe; never contacts the link |
| `GET` | `/static/…` | public | Embedded `tokens.css`, `app.css`, Inter fonts, `htmx.min.js` | | `GET` | `/static/…` | public | Embedded `tokens.css`, `app.css`, Inter fonts, `htmx.min.js` |
Unauthenticated requests to `/` and `/partials/…` redirect to `/login?next=…`. Unauthenticated requests to `/` and `/partials/…` redirect to `/login?next=…`.
@@ -248,4 +264,6 @@ and runs on every start. Migration v2 adds `imap_uid` and `imap_uidvalidity` to
version once, so a category removed by the user is not silently re-added; version once, so a category removed by the user is not silently re-added;
migration v4 adds `folder` and `mailbox` to `messages` for the Phase 4 triage migration v4 adds `folder` and `mailbox` to `messages` for the Phase 4 triage
folders; migration v5 adds `messages.message_id_header` (reply threading) and folders; migration v5 adds `messages.message_id_header` (reply threading) and
`drafts.sent_at`. `drafts.sent_at`; migration v6 adds `messages.list_unsubscribe` and
`messages.list_unsubscribe_post` and extends `unsubscribe` with `url` and
`updated_at`.

203
docs/phase6-unsubscribe.md Normal file
View File

@@ -0,0 +1,203 @@
# Phase 6 — Unsubscribe and rules editor
Status: **complete.** Built, tested, deployed on `.13`, and the DoD was
demonstrated against real mail. Parent [AUD-68](/AUD/issues/AUD-68). Phase 6 of
AGENTS.md §11. Commit `00bb0e0`.
**DoD:** a rule written in the editor changes the classification of a real
message. — met (§2.1).
## 1. What was built
### 1.1 Rules editor (FR-4, FR-11)
- **`/rules`** is a plain-text editor over the `rules` table. Saving appends a
new numbered version (`store.SaveRules`); nothing is overwritten. The page
shows the current version, the active rule count, and a version list.
- **Version history** lists the newest revisions with timestamp, line count, and
a first-rule preview. **Restore** re-saves an older version as a new version,
so history stays append-only.
- **Parse errors** are surfaced on the page. `classify.ParseRules` collects a
bad line in `Errors` and skips it; the editor shows exactly which lines were
ignored and why.
- **Apply** re-runs the classifier over every message (`POST /rules/reclassify`)
and reports considered/classified/failed back on the same page.
- **JEV and the LLM read it.** The classifier already read the rules and applied
the first match as an override. The reply model now reads the same document:
`reply.Request.Rules` is added to the OmniRoute prompt as a bounded
`USER RULES` block (`internal/reply/draft.go`), and `handleDraft` passes the
current rules text.
### 1.2 Unsubscribe detection and tracking (FR-7)
- **Capture.** `List-Unsubscribe` (RFC 2369) and `List-Unsubscribe-Post`
(RFC 8058) are parsed by `internal/mailfetch` and stored raw on the message
(`messages.list_unsubscribe`, `messages.list_unsubscribe_post`). They are kept
raw so detection can be re-run without a refetch.
- **Detect.** `internal/unsubscribe` parses angle-bracketed and bare URIs into
`http`/`mailto` links, recognises the one-click header, de-duplicates, and
falls back to a body scan when there is no header (a URL on a line that
mentions "unsubscribe"). The ignore-list key is the sender's brand domain
when it is a real domain, with the link host as the fallback for free-mail
senders (so a mailto endpoint at `list-manage.com` does not group unrelated
senders).
- **View and act.** `/unsubscribe` lists the messages with a detected link, the
domain-level **ignore list**, and the domains **recorded for unsubscribe**.
Sam can ignore a domain, un-ignore it, or record it for unsubscribe. Every
action is a CSRF-protected POST and is written to `actions_log` (FR-13).
- **No link is ever followed.** The app makes no network call to an unsubscribe
URL and sends no mail. "Mark for unsubscribe" records the decision only; the
page states this plainly, and following a real link stays a manual step that
needs Sam's explicit approval. A pending RFC 8058 one-click POST is detected
and shown, but never sent.
### 1.3 Schema (migration v6, additive)
| Object | Change |
|---|---|
| `messages` | `list_unsubscribe`, `list_unsubscribe_post` (TEXT) |
| `unsubscribe` | `url` (TEXT NOT NULL DEFAULT ''), `updated_at` (TEXT) |
| `schema_migrations` | records version 6 |
The `unsubscribe` table keeps its `status` values `done` (recorded) and
`ignored` (ignore list). Migration v6 is additive; a Phase 5 binary ignores the
new columns.
## 2. DoD evidence
### 2.1 A rule written in the editor changed a real message
Real message **87**, from `Heritage Fruit Trees <support@heritagefruittrees.com.au>`.
**Before.** Classified by JEV with no rule (rules version 1):
| category | importance | action | confidence | rules_version |
|---|---|---|---|---|
| Newsletter | low | unsubscribe | 0.98 | 1 |
**Rule written in the editor.** `POST /rules/save` with:
```
# Phase 6 DoD rule (AUD-72)
from contains "heritagefruittrees.com.au" => category=Action, importance=high, action=file
```
The page then showed **`version 2 · 1 active rule`**.
**After.** `POST /rules/reclassify` returned
`Re-classified 103 of 103 message(s); 0 failed. The current rules are now applied.`
Message 87's newest classification row is now:
| category | importance | action | confidence | model | rules_version |
|---|---|---|---|---|---|
| Action | high | file | 1.00 | jev-1.13.0 | 2 |
The per-message action log records the override and the matching rule line:
```
classify category=Action importance=high action=file confidence=1.00
model=jev-1.13.0 rules_version=2 rule_line=2 rule="from contains ..."
```
Message 87's history therefore shows the change: three `Newsletter/low/
unsubscribe` rows under `rules_version=1`, then `Action/high/file` under
`rules_version=2`.
### 2.2 Unsubscribe detection
`docker exec email-agent email-agent -fetch` refreshed 100 messages, filling the
new header columns. **31 messages carried a `List-Unsubscribe` header.** The live
`/unsubscribe` page reported **33 messages with an unsubscribe link** — the 31
header matches plus 2 body-fallback matches (for example `allakiyas.com`).
Example detected domains in the live page: `google.com`, `spotify.com`,
`subaru.com.au`, `linkedin.com`, `email.centrecom.com.au`, `r.repco.com.au`,
`ozmobiles.com.au`, `heritagefruittrees.com.au`, `edm.booktopia.com.au`,
`netflix.com`, `urbnsurf.com`.
A `mailto:` example (`Google Alerts`) and a web example (message 87,
`https://drdzlr.clicks.mlsend.com/...`) both render in the list and in the
message detail pane.
### 2.3 Ignore list and record
Demonstrated on the live UI with `heritagefruittrees.com.au`, then reverted (the
domain-level row is deleted; the append-only log keeps the history):
| Step | Result |
|---|---|
| `POST /unsubscribe/ignore` | 303; page shows `Ignore list · 1 domain(s)` with an `ignored` pill and the recorded URL |
| `POST /unsubscribe/mark` `confirm=yes` | 303; page shows `Recorded for unsubscribe · 1 domain(s)` with a `recorded` pill, the URL, and a timestamp, and the note "The app has not contacted any link." |
| `POST /unsubscribe/unignore` | 303; domain removed, `unsubscribe` table back to 0 rows |
Durable `actions_log` rows (FR-13):
```
rules_save version=2 rules=1 parse_errors=0
reclassify considered=103 classified=103 failed=0 rules_version=2
unsubscribe_ignore domain=heritagefruittrees.com.au
unsubscribe_marked domain=heritagefruittrees.com.au url=https://drdzlr.clicks.mlsend.com/...
unsubscribe_unignore domain=heritagefruittrees.com.au
```
No email was sent, and no unsubscribe URL was contacted.
### 2.4 Live deployment
| Check | Result |
|---|---|
| `GET /health` | `200 {"status":"ok","version":"phase6","commit":"00bb0e0","database":"ok"}` |
| Container log | `database ready … schema_version: 6` (migration v6 ran in place on the Phase 1–5 volume) |
| Login over public HTTPS | `https://mail.lab.audasmedia.com.au/health` → 200; `/rules` → 303 to `/login` (auth enforced) |
| Dashboard | shows `Rules` and `Unsubscribe` links |
| Message detail (87) | shows `action-pill">file` and the detected unsubscribe link |
### 2.5 Automated tests
`gofmt -l` clean, `go vet ./...` clean, `go test ./...` green (in
`golang:1.25-alpine`). New coverage:
| Package | Tests |
|---|---|
| `internal/unsubscribe` | header parsing (mailto + http, bare, one-click, de-dup), body fallback, unrelated-link rejection, free-mail domain fallback, `Domain()` |
| `internal/mailfetch` | `ParseMessage` captures both List-Unsubscribe headers and the source address |
| `internal/reply` | the rules document is included in the draft prompt |
| `internal/store` | migration v6 on a hand-built v5 database; unsubscribe record upsert/list/clear and status validation; candidate selection (header + body); rules version list + restore lookup |
| `internal/server` | rules page renders; save creates a version; CSRF required; parse errors shown; restore creates a new version; re-classify redirects without JEV; unsubscribe detection renders; ignore/un-ignore; mark requires confirmation and records the URL; CSRF on every mutation; bad domain rejected; message detail shows the link |
## 3. Configuration
No new environment variables. The rules editor, unsubscribe detection, and
tracking work without any new credentials. Re-classify needs `JEV_API_KEY`
(already configured); reply drafting that reads the rules needs
`OMNIROUTE_API_KEY` (already configured). No secret value is committed.
## 4. Deployment
On `.13`, from the repo, with the host secrets in scope:
```sh
set -a; . ~/.config/environment.d/10-secrets.conf; set +a
docker compose build
docker compose up -d # migration v6 runs on start; data volume kept
docker exec email-agent email-agent -fetch # fills the List-Unsubscribe columns
```
## 5. Rollback
1. `git revert 00bb0e0 && docker compose up -d --build`.
2. Migration v6 is additive; the Phase 5 binary ignores the new columns and the
new tables. The `unsubscribe` table and `actions_log` rows are left in place.
3. Reverting the code does not un-save a rules version. Restore the intended
version from `/rules`, or re-save it.
## 6. Not done / deferred
- **Following an unsubscribe link is not implemented.** This is deliberate
(AUD-72): the app only marks and records. Acting on a real link — an RFC 8058
one-click POST, a mailto, or a web form — needs Sam's explicit approval and is
a separate, manually approved step.
- The ignore-list key is the sender's domain as seen in `From:`. Different
subdomains of one brand are tracked separately; registrable-domain grouping
(a public suffix list) is not implemented.
- **1-month body retention** (§7) is still not implemented.