7c66e4ad31
The approved real test reply was sent and the thread updated (same X-GM-THRID as the original). The live test exposed one bug: the thread check hardcoded Gmail's sent mailbox as "[Gmail]/Sent Mail", but this account exposes it as "[Google Mail]/Sent Mail". - internal/mailfetch: LIST parser, ListMailboxes, and SentMailbox that finds the mailbox flagged \Sent (RFC 6154) with name fallbacks. - internal/mailops: VerifyThreadUpdated selects the discovered \Sent mailbox instead of a hardcoded name. - cmd/email-agent: -check-thread <id> exercises the check against real Gmail without sending mail. - Tests for LIST parsing, sent-mailbox discovery, and the discovery error path. - docs/phase5-replies.md: record send-as verification, the sent reply's thread id, the app confirmation (sent_reply_in_thread=true), and the fix. Co-Authored-By: Paperclip <noreply@paperclip.ing>
184 lines
9.8 KiB
Markdown
184 lines
9.8 KiB
Markdown
# Phase 5 — Reply drafts (templates, LLM draft, edit, send)
|
||
|
||
Status: **complete.** Built, tested, deployed, and the approved real test reply
|
||
was sent and confirmed to have updated the thread. Sam approved the test send via
|
||
issue interaction `080c500d-5968-49ad-9c40-0b2d3f445cc0` on
|
||
[AUD-71](/AUD/issues/AUD-71). Parent [AUD-68](/AUD/issues/AUD-68). Phase 5 of
|
||
AGENTS.md §11.
|
||
|
||
**DoD:** a reply is drafted, edited, and sent, and the thread updates. — met
|
||
(§2.6).
|
||
|
||
## 1. What was built
|
||
|
||
- **Message detail pane (FR-8).** Selecting a message opens the body, the
|
||
recommended action, the reply identity, a template picker, and the draft. It is
|
||
an HTMX fragment (`GET /partials/message`) rendered into `#message-detail`.
|
||
- **Template-based LLM draft (FR-8).** `internal/reply` calls the **OmniRoute**
|
||
gateway's OpenAI-compatible `/chat/completions` endpoint (the only non-JEV model
|
||
path, §10). Four built-in templates — `quick-ack`, `answer`, `confirm`,
|
||
`decline` — shape the prompt; the action default is `answer` for a `reply`
|
||
recommendation and `quick-ack` otherwise. Sam can add a free-text note for the
|
||
model. The generated body is stored in `drafts`.
|
||
- **Edit then send.** Sam edits the draft in a textarea, can **Save draft**, then
|
||
**Send reply**. Sending is a deliberate UI action: the handler requires a CSRF
|
||
token, the UI confirmation, and the guard field `confirm=yes`. There is **no
|
||
auto-send** and no scheduled send (§10).
|
||
- **Reply identity (FR-15).** `From:` is `"Sam Rolfe" <address the mail arrived
|
||
at>` (`X-Pm-Original-To`). When that is unknown it falls back to `"Sam Rolfe"
|
||
<GMAIL_USER>`. The SMTP envelope sender is always the authenticated Gmail
|
||
account. `To:` is the original sender.
|
||
- **Threading.** The reply sets `In-Reply-To` and `References` from the original
|
||
`Message-ID`, which Phase 5 now stores (`messages.message_id_header`, migration
|
||
v5). After a send the service checks `[Gmail]/Sent Mail` for the original Gmail
|
||
thread id and reports whether the thread updated.
|
||
- **SMTP send, IPv4 forced (§9).** `internal/mailops` sends over
|
||
`smtp.gmail.com:587` with STARTTLS and the app password, dialling `tcp4`
|
||
exactly like the IMAP read path.
|
||
- **Filing (FR-9).** After a successful send the original message moves to its
|
||
`Replied-to/<address>` label.
|
||
- **FR-13 logging.** `draft`, `draft_edit`, `reply_sent`, the `move` to the
|
||
Replied-to label, and the summary rows are appended to `actions_log`.
|
||
- **Duplicate guard.** A draft that is already `sent` cannot be sent again.
|
||
|
||
New environment variables (all optional; defaults shown): `EMAIL_AGENT_SMTP_HOST`
|
||
(`smtp.gmail.com`), `EMAIL_AGENT_SMTP_PORT` (`587`), `OMNIROUTE_BASE_URL`
|
||
(`http://192.168.20.13:20129/v1`), `OMNIROUTE_MODEL` (`auto/best-chat`).
|
||
`OMNIROUTE_API_KEY` and the Gmail credentials are the only secrets.
|
||
|
||
## 2. DoD evidence
|
||
|
||
### 2.1 Automated tests
|
||
|
||
`gofmt -l` clean, `go vet ./...` clean, `go test ./...` green (in
|
||
`golang:1.25-alpine`). New coverage:
|
||
|
||
| Package | Tests |
|
||
|---|---|
|
||
| `internal/reply` | chat-completions request shape (auth, `x-opencode-session`, model, non-streaming); not-configured; HTTP error; template default by action; prompt includes template/instruction/classification/message; fence and `Subject:` stripping; empty reply rejected |
|
||
| `internal/mailops` | FR-15 identity (arrived-at vs Gmail fallback); `ReplyTo` parsing; `Re:` prefixing; RFC822 headers (From/To/Subject/In-Reply-To/References/Message-ID, CRLF); `Send` without credentials; `dialSMTP4` reaches an IPv4 listener; `VerifyThreadUpdated` selects `[Gmail]/Sent Mail` and searches the thread |
|
||
| `internal/store` | draft round trip (save/edit/mark sent); latest-draft miss; `MessageByID` returns the Message-ID; **migration v5 on a hand-built v4 database** adds `messages.message_id_header` and `drafts.sent_at` |
|
||
| `internal/server` | detail pane renders the reply identity and body; detail requires auth; draft requires CSRF; draft is generated and stored; edit saves; send requires confirmation; send calls the sender with the right To/From/Subject/threading headers, marks the draft sent, moves the message to Replied-to, and verifies the thread; already-sent draft refused; missing Message-ID warns |
|
||
|
||
A fake LLM client (an `httptest` server) and fake sender/generator are used, so
|
||
the suite needs no network and sends no mail.
|
||
|
||
### 2.2 Live, deployed (schema v5, `email-agent` recreated on `.13`)
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| `GET /health` | `200 {"status":"ok","env":"production","database":"ok"}` |
|
||
| Container log | `database ready … schema_version: 5` (migration v5 ran in place on the Phase 1–4 volume) |
|
||
| `docker exec email-agent email-agent -check-smtp` | `smtp check ok: ipv4 + starttls + auth (no mail sent)` — Gmail submission path works over IPv4 with the app password |
|
||
| OmniRoute `POST /v1/chat/completions` (model `auto/best-chat`) | `200`; a probe returned `ok` |
|
||
| Login over public HTTPS (`https://mail.lab.audasmedia.com.au`) | `303`, session + CSRF cookies |
|
||
| `POST /actions/draft` for message `96` | `200`; OmniRoute produced `Got it.` and the draft was stored |
|
||
| `POST /actions/draft/save` | `200`; the edited body persisted |
|
||
| `POST /actions/send` **without** `confirm=yes` | `200`; refused with "Sending needs confirmation"; **no mail sent** |
|
||
| `POST /actions/send` **without** a CSRF token | `403` |
|
||
|
||
The draft → edit path is therefore verified end to end over the public HTTPS URL,
|
||
and the send path is verified up to (but not including) DATA.
|
||
|
||
### 2.3 Gmail "Send mail as" verification (AGENTS.md §13 "Remaining")
|
||
|
||
AGENTS.md recorded send-as as "reported as already set up in Gmail … not
|
||
verifiable from the sent folder". Gmail exposes no read-only alias list over
|
||
IMAP/SMTP, so the check is a real send: Gmail accepts the `From` header only when
|
||
the address is a registered send-as alias, and rejects it during DATA otherwise.
|
||
|
||
**Result: verified.** The approved test reply was sent as
|
||
`"Sam Rolfe" <hello@where-woof.com>` (the `X-Pm-Original-To` address) from the
|
||
`samuelrolfe@gmail.com` account. Gmail accepted it and stored it in the Sent
|
||
mailbox (`[Google Mail]/Sent Mail`) with that exact `From` header, so
|
||
`hello@where-woof.com` is a working send-as alias. The SMTP connection, STARTTLS
|
||
upgrade, and app-password authentication were already verified separately with
|
||
`-check-smtp`.
|
||
|
||
### 2.4 Draft generated, edited, and sent
|
||
|
||
Draft `2` for message `99` ("Test email", arrived at `hello@where-woof.com`),
|
||
generated through the live UI path, edited, and then sent after Sam's approval:
|
||
|
||
```
|
||
Thanks, got it. This came through to hello@where-woof.com. Since it's a test, no action needed on my side. If you need anything specific, let me know.
|
||
|
||
Sam
|
||
```
|
||
|
||
After the send the draft status is `sent`, the message left the inbox and now
|
||
lives under the `Replied-to/hello@where-woof.com` label (1 row in the Replied-to
|
||
view), and the message history shows `draft`, `reply_sent`, and the `move` rows
|
||
(FR-13).
|
||
|
||
### 2.5 Thread update after the sent reply
|
||
|
||
**Result: verified — the reply joined the original thread.** Read directly from
|
||
Gmail over IMAP (read-only):
|
||
|
||
| Message | From | To | `X-GM-THRID` | `In-Reply-To` / `References` |
|
||
|---|---|---|---|---|
|
||
| Original (`99`) | `sam rolfe <samuelrolfe@gmail.com>` | `hello@where-woof.com` | `1878738835102629816` | — |
|
||
| Sent reply | `"Sam Rolfe" <hello@where-woof.com>` | `samuelrolfe@gmail.com` | `1878738835102629816` | `<CAMAgT7S04G5E=…@mail.gmail.com>` |
|
||
|
||
The reply carries the original `Message-ID` in `In-Reply-To` and `References` and
|
||
Gmail assigned it the **same thread id**, so the thread updated. The app's own
|
||
check now agrees:
|
||
|
||
```
|
||
$ docker exec email-agent email-agent -check-thread 99
|
||
message=99 thread=1878738835102629816 sent_reply_in_thread=true
|
||
```
|
||
|
||
Status code `sent_reply_in_thread=true`. No third party received the test: the
|
||
only recipient was `samuelrolfe@gmail.com`.
|
||
|
||
### 2.6 Bug found by the live test and fixed
|
||
|
||
The first send reported "Could not yet confirm the thread update". The cause was
|
||
not threading: `VerifyThreadUpdated` hardcoded Gmail's mailbox as
|
||
`[Gmail]/Sent Mail`, but this account exposes the special-use mailboxes as
|
||
`[Google Mail]/…`. The fix discovers the mailbox flagged `\Sent` (RFC 6154) with
|
||
`LIST`, falling back to the common names. New coverage:
|
||
`internal/mailfetch` (`parseList`, `ListMailboxes`, `SentMailbox`) and
|
||
`internal/mailops` (discovered sent mailbox, discovery error). `-check-thread
|
||
<id>` was added so the check can be exercised against real Gmail without sending
|
||
mail. After the fix, `-check-thread 99` returns true (above).
|
||
|
||
## 3. Configuration
|
||
|
||
`OMNIROUTE_API_KEY` and `GMAIL_USER`/`GMAIL_APP_PASSWORD` come from
|
||
`~/.config/environment.d/10-secrets.conf` and are passed in by `docker-compose`.
|
||
No secret value is committed. The other Phase 5 settings have working defaults.
|
||
|
||
## 4. Deployment
|
||
|
||
On `.13`, from the repo, with the host secrets in scope:
|
||
|
||
```sh
|
||
set -a; . ~/.config/environment.d/10-secrets.conf; set +a
|
||
docker compose build
|
||
docker compose up -d # migration v5 runs on start; data volume kept
|
||
docker exec email-agent email-agent -check-smtp # safe: no mail sent
|
||
```
|
||
|
||
Two commits were needed: the Phase 5 implementation and the `\Sent` mailbox
|
||
discovery fix from §2.6.
|
||
|
||
## 5. Rollback
|
||
|
||
1. `git revert <sent-mailbox-fix commit> <phase-5 commit>` and
|
||
`docker compose up -d --build`.
|
||
2. Migration v5 is additive (`messages.message_id_header`, `drafts.sent_at`); the
|
||
Phase 4 binary ignores the new columns.
|
||
3. A sent reply cannot be unsent. Gmail's Sent copy and the thread stay; the
|
||
`Replied-to/<address>` label can be removed in Gmail by hand.
|
||
|
||
## 6. Not done / deferred
|
||
|
||
- A **template editor** (user-defined templates) is not in the Phase 5
|
||
requirements; templates are built in. The Phase 6 rules editor still feeds the
|
||
classifier.
|
||
- **1-month body retention** (§7) is still not implemented.
|
||
- Rich-text or HTML replies are out of scope; replies are plain text.
|