docs: Phase 6 unsubscribe + rules editor evidence (AUD-72)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
203
docs/phase6-unsubscribe.md
Normal file
203
docs/phase6-unsubscribe.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# Phase 6 — Unsubscribe and rules editor
|
||||
|
||||
Status: **complete.** Built, tested, deployed on `.13`, and the DoD was
|
||||
demonstrated against real mail. Parent [AUD-68](/AUD/issues/AUD-68). Phase 6 of
|
||||
AGENTS.md §11. Commit `00bb0e0`.
|
||||
|
||||
**DoD:** a rule written in the editor changes the classification of a real
|
||||
message. — met (§2.1).
|
||||
|
||||
## 1. What was built
|
||||
|
||||
### 1.1 Rules editor (FR-4, FR-11)
|
||||
|
||||
- **`/rules`** is a plain-text editor over the `rules` table. Saving appends a
|
||||
new numbered version (`store.SaveRules`); nothing is overwritten. The page
|
||||
shows the current version, the active rule count, and a version list.
|
||||
- **Version history** lists the newest revisions with timestamp, line count, and
|
||||
a first-rule preview. **Restore** re-saves an older version as a new version,
|
||||
so history stays append-only.
|
||||
- **Parse errors** are surfaced on the page. `classify.ParseRules` collects a
|
||||
bad line in `Errors` and skips it; the editor shows exactly which lines were
|
||||
ignored and why.
|
||||
- **Apply** re-runs the classifier over every message (`POST /rules/reclassify`)
|
||||
and reports considered/classified/failed back on the same page.
|
||||
- **JEV and the LLM read it.** The classifier already read the rules and applied
|
||||
the first match as an override. The reply model now reads the same document:
|
||||
`reply.Request.Rules` is added to the OmniRoute prompt as a bounded
|
||||
`USER RULES` block (`internal/reply/draft.go`), and `handleDraft` passes the
|
||||
current rules text.
|
||||
|
||||
### 1.2 Unsubscribe detection and tracking (FR-7)
|
||||
|
||||
- **Capture.** `List-Unsubscribe` (RFC 2369) and `List-Unsubscribe-Post`
|
||||
(RFC 8058) are parsed by `internal/mailfetch` and stored raw on the message
|
||||
(`messages.list_unsubscribe`, `messages.list_unsubscribe_post`). They are kept
|
||||
raw so detection can be re-run without a refetch.
|
||||
- **Detect.** `internal/unsubscribe` parses angle-bracketed and bare URIs into
|
||||
`http`/`mailto` links, recognises the one-click header, de-duplicates, and
|
||||
falls back to a body scan when there is no header (a URL on a line that
|
||||
mentions "unsubscribe"). The ignore-list key is the sender's brand domain
|
||||
when it is a real domain, with the link host as the fallback for free-mail
|
||||
senders (so a mailto endpoint at `list-manage.com` does not group unrelated
|
||||
senders).
|
||||
- **View and act.** `/unsubscribe` lists the messages with a detected link, the
|
||||
domain-level **ignore list**, and the domains **recorded for unsubscribe**.
|
||||
Sam can ignore a domain, un-ignore it, or record it for unsubscribe. Every
|
||||
action is a CSRF-protected POST and is written to `actions_log` (FR-13).
|
||||
- **No link is ever followed.** The app makes no network call to an unsubscribe
|
||||
URL and sends no mail. "Mark for unsubscribe" records the decision only; the
|
||||
page states this plainly, and following a real link stays a manual step that
|
||||
needs Sam's explicit approval. A pending RFC 8058 one-click POST is detected
|
||||
and shown, but never sent.
|
||||
|
||||
### 1.3 Schema (migration v6, additive)
|
||||
|
||||
| Object | Change |
|
||||
|---|---|
|
||||
| `messages` | `list_unsubscribe`, `list_unsubscribe_post` (TEXT) |
|
||||
| `unsubscribe` | `url` (TEXT NOT NULL DEFAULT ''), `updated_at` (TEXT) |
|
||||
| `schema_migrations` | records version 6 |
|
||||
|
||||
The `unsubscribe` table keeps its `status` values `done` (recorded) and
|
||||
`ignored` (ignore list). Migration v6 is additive; a Phase 5 binary ignores the
|
||||
new columns.
|
||||
|
||||
## 2. DoD evidence
|
||||
|
||||
### 2.1 A rule written in the editor changed a real message
|
||||
|
||||
Real message **87**, from `Heritage Fruit Trees <support@heritagefruittrees.com.au>`.
|
||||
|
||||
**Before.** Classified by JEV with no rule (rules version 1):
|
||||
|
||||
| category | importance | action | confidence | rules_version |
|
||||
|---|---|---|---|---|
|
||||
| Newsletter | low | unsubscribe | 0.98 | 1 |
|
||||
|
||||
**Rule written in the editor.** `POST /rules/save` with:
|
||||
|
||||
```
|
||||
# Phase 6 DoD rule (AUD-72)
|
||||
from contains "heritagefruittrees.com.au" => category=Action, importance=high, action=file
|
||||
```
|
||||
|
||||
The page then showed **`version 2 · 1 active rule`**.
|
||||
|
||||
**After.** `POST /rules/reclassify` returned
|
||||
`Re-classified 103 of 103 message(s); 0 failed. The current rules are now applied.`
|
||||
Message 87's newest classification row is now:
|
||||
|
||||
| category | importance | action | confidence | model | rules_version |
|
||||
|---|---|---|---|---|---|
|
||||
| Action | high | file | 1.00 | jev-1.13.0 | 2 |
|
||||
|
||||
The per-message action log records the override and the matching rule line:
|
||||
|
||||
```
|
||||
classify category=Action importance=high action=file confidence=1.00
|
||||
model=jev-1.13.0 rules_version=2 rule_line=2 rule="from contains ..."
|
||||
```
|
||||
|
||||
Message 87's history therefore shows the change: three `Newsletter/low/
|
||||
unsubscribe` rows under `rules_version=1`, then `Action/high/file` under
|
||||
`rules_version=2`.
|
||||
|
||||
### 2.2 Unsubscribe detection
|
||||
|
||||
`docker exec email-agent email-agent -fetch` refreshed 100 messages, filling the
|
||||
new header columns. **31 messages carried a `List-Unsubscribe` header.** The live
|
||||
`/unsubscribe` page reported **33 messages with an unsubscribe link** — the 31
|
||||
header matches plus 2 body-fallback matches (for example `allakiyas.com`).
|
||||
|
||||
Example detected domains in the live page: `google.com`, `spotify.com`,
|
||||
`subaru.com.au`, `linkedin.com`, `email.centrecom.com.au`, `r.repco.com.au`,
|
||||
`ozmobiles.com.au`, `heritagefruittrees.com.au`, `edm.booktopia.com.au`,
|
||||
`netflix.com`, `urbnsurf.com`.
|
||||
|
||||
A `mailto:` example (`Google Alerts`) and a web example (message 87,
|
||||
`https://drdzlr.clicks.mlsend.com/...`) both render in the list and in the
|
||||
message detail pane.
|
||||
|
||||
### 2.3 Ignore list and record
|
||||
|
||||
Demonstrated on the live UI with `heritagefruittrees.com.au`, then reverted (the
|
||||
domain-level row is deleted; the append-only log keeps the history):
|
||||
|
||||
| Step | Result |
|
||||
|---|---|
|
||||
| `POST /unsubscribe/ignore` | 303; page shows `Ignore list · 1 domain(s)` with an `ignored` pill and the recorded URL |
|
||||
| `POST /unsubscribe/mark` `confirm=yes` | 303; page shows `Recorded for unsubscribe · 1 domain(s)` with a `recorded` pill, the URL, and a timestamp, and the note "The app has not contacted any link." |
|
||||
| `POST /unsubscribe/unignore` | 303; domain removed, `unsubscribe` table back to 0 rows |
|
||||
|
||||
Durable `actions_log` rows (FR-13):
|
||||
|
||||
```
|
||||
rules_save version=2 rules=1 parse_errors=0
|
||||
reclassify considered=103 classified=103 failed=0 rules_version=2
|
||||
unsubscribe_ignore domain=heritagefruittrees.com.au
|
||||
unsubscribe_marked domain=heritagefruittrees.com.au url=https://drdzlr.clicks.mlsend.com/...
|
||||
unsubscribe_unignore domain=heritagefruittrees.com.au
|
||||
```
|
||||
|
||||
No email was sent, and no unsubscribe URL was contacted.
|
||||
|
||||
### 2.4 Live deployment
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `GET /health` | `200 {"status":"ok","version":"phase6","commit":"00bb0e0","database":"ok"}` |
|
||||
| Container log | `database ready … schema_version: 6` (migration v6 ran in place on the Phase 1–5 volume) |
|
||||
| Login over public HTTPS | `https://mail.lab.audasmedia.com.au/health` → 200; `/rules` → 303 to `/login` (auth enforced) |
|
||||
| Dashboard | shows `Rules` and `Unsubscribe` links |
|
||||
| Message detail (87) | shows `action-pill">file` and the detected unsubscribe link |
|
||||
|
||||
### 2.5 Automated tests
|
||||
|
||||
`gofmt -l` clean, `go vet ./...` clean, `go test ./...` green (in
|
||||
`golang:1.25-alpine`). New coverage:
|
||||
|
||||
| Package | Tests |
|
||||
|---|---|
|
||||
| `internal/unsubscribe` | header parsing (mailto + http, bare, one-click, de-dup), body fallback, unrelated-link rejection, free-mail domain fallback, `Domain()` |
|
||||
| `internal/mailfetch` | `ParseMessage` captures both List-Unsubscribe headers and the source address |
|
||||
| `internal/reply` | the rules document is included in the draft prompt |
|
||||
| `internal/store` | migration v6 on a hand-built v5 database; unsubscribe record upsert/list/clear and status validation; candidate selection (header + body); rules version list + restore lookup |
|
||||
| `internal/server` | rules page renders; save creates a version; CSRF required; parse errors shown; restore creates a new version; re-classify redirects without JEV; unsubscribe detection renders; ignore/un-ignore; mark requires confirmation and records the URL; CSRF on every mutation; bad domain rejected; message detail shows the link |
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
No new environment variables. The rules editor, unsubscribe detection, and
|
||||
tracking work without any new credentials. Re-classify needs `JEV_API_KEY`
|
||||
(already configured); reply drafting that reads the rules needs
|
||||
`OMNIROUTE_API_KEY` (already configured). No secret value is committed.
|
||||
|
||||
## 4. Deployment
|
||||
|
||||
On `.13`, from the repo, with the host secrets in scope:
|
||||
|
||||
```sh
|
||||
set -a; . ~/.config/environment.d/10-secrets.conf; set +a
|
||||
docker compose build
|
||||
docker compose up -d # migration v6 runs on start; data volume kept
|
||||
docker exec email-agent email-agent -fetch # fills the List-Unsubscribe columns
|
||||
```
|
||||
|
||||
## 5. Rollback
|
||||
|
||||
1. `git revert 00bb0e0 && docker compose up -d --build`.
|
||||
2. Migration v6 is additive; the Phase 5 binary ignores the new columns and the
|
||||
new tables. The `unsubscribe` table and `actions_log` rows are left in place.
|
||||
3. Reverting the code does not un-save a rules version. Restore the intended
|
||||
version from `/rules`, or re-save it.
|
||||
|
||||
## 6. Not done / deferred
|
||||
|
||||
- **Following an unsubscribe link is not implemented.** This is deliberate
|
||||
(AUD-72): the app only marks and records. Acting on a real link — an RFC 8058
|
||||
one-click POST, a mailto, or a web form — needs Sam's explicit approval and is
|
||||
a separate, manually approved step.
|
||||
- The ignore-list key is the sender's domain as seen in `From:`. Different
|
||||
subdomains of one brand are tracked separately; registrable-domain grouping
|
||||
(a public suffix list) is not implemented.
|
||||
- **1-month body retention** (§7) is still not implemented.
|
||||
Reference in New Issue
Block a user