Phase 5: reply drafts — template/LLM draft, edit, SMTP send (AUD-71)
Implements Phase 5 of AGENTS.md §11 (FR-8, FR-15): - Message detail pane (GET /partials/message) with body, recommended action, reply identity, template picker, editable draft, and history. - internal/reply: OmniRoute (OpenAI-compatible) chat client and a template-based draft generator. All non-JEV models go through OmniRoute (§10). - internal/mailops: SMTP send over smtp.gmail.com:587 with STARTTLS and the app password, forcing IPv4 (§9); FR-15 reply identity (send as the arrived-at X-Pm-Original-To address, else "Sam Rolfe"); threading via In-Reply-To/References; -check-smtp safe verification; and a [Gmail]/Sent Mail thread check. - Migration v5 stores messages.message_id_header and drafts.sent_at; mailfetch parses Message-ID. - Send is user-triggered only: CSRF + UI confirm + confirm=yes guard, no auto-send, duplicate-send refused. A sent reply files the original under Replied-to/<address> and logs every step (FR-13). - Tests for reply, mailops SMTP/identity/thread check, store drafts and migration v5, and the server draft/edit/send guards. - docs/phase5-replies.md, README, .env.example, docker-compose updated. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Paperclip Builder
148
docs/phase5-replies.md
Normal file
148
docs/phase5-replies.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Phase 5 — Reply drafts (templates, LLM draft, edit, send)
|
||||
|
||||
Status: **built, tested, and deployed. The one real test send is awaiting Sam's
|
||||
explicit approval** (issue interaction `080c500d-5968-49ad-9c40-0b2d3f445cc0` on
|
||||
[AUD-71](/AUD/issues/AUD-71)). Parent [AUD-68](/AUD/issues/AUD-68). Phase 5 of
|
||||
AGENTS.md §11.
|
||||
|
||||
**DoD:** a reply is drafted, edited, and sent, and the thread updates.
|
||||
|
||||
Every part of the DoD except the real send is verified below. The real send needs
|
||||
Sam's approval per the phase rules and AGENTS.md §10; it is the only step that
|
||||
cannot be tested without sending mail.
|
||||
|
||||
## 1. What was built
|
||||
|
||||
- **Message detail pane (FR-8).** Selecting a message opens the body, the
|
||||
recommended action, the reply identity, a template picker, and the draft. It is
|
||||
an HTMX fragment (`GET /partials/message`) rendered into `#message-detail`.
|
||||
- **Template-based LLM draft (FR-8).** `internal/reply` calls the **OmniRoute**
|
||||
gateway's OpenAI-compatible `/chat/completions` endpoint (the only non-JEV model
|
||||
path, §10). Four built-in templates — `quick-ack`, `answer`, `confirm`,
|
||||
`decline` — shape the prompt; the action default is `answer` for a `reply`
|
||||
recommendation and `quick-ack` otherwise. Sam can add a free-text note for the
|
||||
model. The generated body is stored in `drafts`.
|
||||
- **Edit then send.** Sam edits the draft in a textarea, can **Save draft**, then
|
||||
**Send reply**. Sending is a deliberate UI action: the handler requires a CSRF
|
||||
token, the UI confirmation, and the guard field `confirm=yes`. There is **no
|
||||
auto-send** and no scheduled send (§10).
|
||||
- **Reply identity (FR-15).** `From:` is `"Sam Rolfe" <address the mail arrived
|
||||
at>` (`X-Pm-Original-To`). When that is unknown it falls back to `"Sam Rolfe"
|
||||
<GMAIL_USER>`. The SMTP envelope sender is always the authenticated Gmail
|
||||
account. `To:` is the original sender.
|
||||
- **Threading.** The reply sets `In-Reply-To` and `References` from the original
|
||||
`Message-ID`, which Phase 5 now stores (`messages.message_id_header`, migration
|
||||
v5). After a send the service checks `[Gmail]/Sent Mail` for the original Gmail
|
||||
thread id and reports whether the thread updated.
|
||||
- **SMTP send, IPv4 forced (§9).** `internal/mailops` sends over
|
||||
`smtp.gmail.com:587` with STARTTLS and the app password, dialling `tcp4`
|
||||
exactly like the IMAP read path.
|
||||
- **Filing (FR-9).** After a successful send the original message moves to its
|
||||
`Replied-to/<address>` label.
|
||||
- **FR-13 logging.** `draft`, `draft_edit`, `reply_sent`, the `move` to the
|
||||
Replied-to label, and the summary rows are appended to `actions_log`.
|
||||
- **Duplicate guard.** A draft that is already `sent` cannot be sent again.
|
||||
|
||||
New environment variables (all optional; defaults shown): `EMAIL_AGENT_SMTP_HOST`
|
||||
(`smtp.gmail.com`), `EMAIL_AGENT_SMTP_PORT` (`587`), `OMNIROUTE_BASE_URL`
|
||||
(`http://192.168.20.13:20129/v1`), `OMNIROUTE_MODEL` (`auto/best-chat`).
|
||||
`OMNIROUTE_API_KEY` and the Gmail credentials are the only secrets.
|
||||
|
||||
## 2. DoD evidence
|
||||
|
||||
### 2.1 Automated tests
|
||||
|
||||
`gofmt -l` clean, `go vet ./...` clean, `go test ./...` green (in
|
||||
`golang:1.25-alpine`). New coverage:
|
||||
|
||||
| Package | Tests |
|
||||
|---|---|
|
||||
| `internal/reply` | chat-completions request shape (auth, `x-opencode-session`, model, non-streaming); not-configured; HTTP error; template default by action; prompt includes template/instruction/classification/message; fence and `Subject:` stripping; empty reply rejected |
|
||||
| `internal/mailops` | FR-15 identity (arrived-at vs Gmail fallback); `ReplyTo` parsing; `Re:` prefixing; RFC822 headers (From/To/Subject/In-Reply-To/References/Message-ID, CRLF); `Send` without credentials; `dialSMTP4` reaches an IPv4 listener; `VerifyThreadUpdated` selects `[Gmail]/Sent Mail` and searches the thread |
|
||||
| `internal/store` | draft round trip (save/edit/mark sent); latest-draft miss; `MessageByID` returns the Message-ID; **migration v5 on a hand-built v4 database** adds `messages.message_id_header` and `drafts.sent_at` |
|
||||
| `internal/server` | detail pane renders the reply identity and body; detail requires auth; draft requires CSRF; draft is generated and stored; edit saves; send requires confirmation; send calls the sender with the right To/From/Subject/threading headers, marks the draft sent, moves the message to Replied-to, and verifies the thread; already-sent draft refused; missing Message-ID warns |
|
||||
|
||||
A fake LLM client (an `httptest` server) and fake sender/generator are used, so
|
||||
the suite needs no network and sends no mail.
|
||||
|
||||
### 2.2 Live, deployed (schema v5, `email-agent` recreated on `.13`)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `GET /health` | `200 {"status":"ok","env":"production","database":"ok"}` |
|
||||
| Container log | `database ready … schema_version: 5` (migration v5 ran in place on the Phase 1–4 volume) |
|
||||
| `docker exec email-agent email-agent -check-smtp` | `smtp check ok: ipv4 + starttls + auth (no mail sent)` — Gmail submission path works over IPv4 with the app password |
|
||||
| OmniRoute `POST /v1/chat/completions` (model `auto/best-chat`) | `200`; a probe returned `ok` |
|
||||
| Login over public HTTPS (`https://mail.lab.audasmedia.com.au`) | `303`, session + CSRF cookies |
|
||||
| `POST /actions/draft` for message `96` | `200`; OmniRoute produced `Got it.` and the draft was stored |
|
||||
| `POST /actions/draft/save` | `200`; the edited body persisted |
|
||||
| `POST /actions/send` **without** `confirm=yes` | `200`; refused with "Sending needs confirmation"; **no mail sent** |
|
||||
| `POST /actions/send` **without** a CSRF token | `403` |
|
||||
|
||||
The draft → edit path is therefore verified end to end over the public HTTPS URL,
|
||||
and the send path is verified up to (but not including) DATA.
|
||||
|
||||
### 2.3 Gmail "Send mail as" verification (AGENTS.md §13 "Remaining")
|
||||
|
||||
AGENTS.md records send-as as "reported as already set up in Gmail … not verifiable
|
||||
from the sent folder". Gmail exposes no read-only alias list over IMAP/SMTP, so
|
||||
the only reliable check is a real send: Gmail accepts the `From` header only when
|
||||
the address is a registered send-as alias, and rejects it otherwise during DATA.
|
||||
|
||||
The SMTP connection, STARTTLS upgrade, and app-password authentication are
|
||||
verified (`-check-smtp`). The `From`-header/send-as check is **pending the
|
||||
approved test send**.
|
||||
|
||||
### 2.4 Draft generated, edited, and stored (screenshot of the live data)
|
||||
|
||||
Draft `2` for message `99` ("Test email", arrived at `hello@where-woof.com`),
|
||||
generated through the live UI path and then edited:
|
||||
|
||||
```
|
||||
Thanks, got it. This came through to hello@where-woof.com. Since it's a test, no action needed on my side. If you need anything specific, let me know.
|
||||
|
||||
Sam
|
||||
```
|
||||
|
||||
### 2.5 Thread update after a sent reply
|
||||
|
||||
Pending the approved test send. The plan is to reply to message `99` as
|
||||
`"Sam Rolfe" <hello@where-woof.com>` to `samuelrolfe@gmail.com` (Sam himself — no
|
||||
third party), then confirm the thread by finding the original Gmail thread id in
|
||||
`[Gmail]/Sent Mail` (`X-GM-THRID` search). The exact message and draft are on the
|
||||
pending approval card.
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
`OMNIROUTE_API_KEY` and `GMAIL_USER`/`GMAIL_APP_PASSWORD` come from
|
||||
`~/.config/environment.d/10-secrets.conf` and are passed in by `docker-compose`.
|
||||
No secret value is committed. The other Phase 5 settings have working defaults.
|
||||
|
||||
## 4. Deployment
|
||||
|
||||
On `.13`, from the repo, with the host secrets in scope:
|
||||
|
||||
```sh
|
||||
set -a; . ~/.config/environment.d/10-secrets.conf; set +a
|
||||
docker compose build
|
||||
docker compose up -d # migration v5 runs on start; data volume kept
|
||||
docker exec email-agent email-agent -check-smtp # safe: no mail sent
|
||||
```
|
||||
|
||||
## 5. Rollback
|
||||
|
||||
1. `git revert <phase-5 commit>` and `docker compose up -d --build`.
|
||||
2. Migration v5 is additive (`messages.message_id_header`, `drafts.sent_at`); the
|
||||
Phase 4 binary ignores the new columns.
|
||||
3. A sent reply cannot be unsent. Gmail's Sent copy and the thread stay; the
|
||||
`Replied-to/<address>` label can be removed in Gmail by hand.
|
||||
|
||||
## 6. Not done / deferred
|
||||
|
||||
- **One real send + thread confirmation** — awaiting Sam's approval (the pending
|
||||
interaction above). This is the last DoD item.
|
||||
- A **template editor** (user-defined templates) is not in the Phase 5
|
||||
requirements; templates are built in. The Phase 6 rules editor still feeds the
|
||||
classifier.
|
||||
- **1-month body retention** (§7) is still not implemented.
|
||||
- Rich-text or HTML replies are out of scope; replies are plain text.
|
||||
Reference in New Issue
Block a user