b267e55f8c
- AGENTS.md: build spec for Paperclip (constraints, conventions, 7 phases) - docs/phase-1..7: per-phase work, DoD, out-of-scope, risks - reference/: frozen decisions, stack+ports+budgets, content map, acceptance tests - agents/: persona and security policy drafts - Retires the React scaffold; history retained as record of the false start
83 lines
3.8 KiB
Markdown
83 lines
3.8 KiB
Markdown
# Phase 6 — Guardrails, cost and kill switch
|
|
|
|
**Goal:** safe to leave running in public. This is a **release gate**, not a follow-up.
|
|
|
|
**Depends on:** Phase 5.
|
|
|
|
**Policy source:** `agents/security.md` (authoritative) and `agents/persona.md`. Both are runtime files, versioned in this repo.
|
|
|
|
---
|
|
|
|
## Work
|
|
|
|
### 6.1 any-guardrail is the mechanism
|
|
|
|
Input and output validation is performed by **[`mozilla-ai/any-guardrail`](https://github.com/mozilla-ai/any-guardrail)**, pinned to an exact version.
|
|
|
|
- **Inbound:** prompt injection · jailbreak · scope violation · personal-data requests · system-prompt extraction.
|
|
- **Outbound:** policy violation · disclosure of persona/security files · personal or infrastructure detail · claims outside the corpus.
|
|
- **Fail closed.** Error, timeout or ambiguous verdict ⇒ refuse.
|
|
- Rules live in this repo, not in library defaults.
|
|
- **If a rule cannot be enforced with this library, report it — do not quietly drop the rule.**
|
|
|
|
### 6.2 The architectural guardrail
|
|
|
|
The public agent has **no tools**: no shell, no filesystem, no network fetch, no write path, no memory between visitors. This removes tool abuse, SSRF, exfiltration-by-callback and cross-visitor poisoning as attack classes rather than defending them. Add an assertion to the test suite that no tool or function definitions are registered.
|
|
|
|
### 6.3 Output safety
|
|
- Model output never reaches the DOM as raw HTML; strict allow-list.
|
|
- Citation URLs allow-listed.
|
|
- Canary string in the system prompt; any appearance ⇒ discard answer, log incident, terminate session.
|
|
|
|
### 6.4 Cost and abuse
|
|
|
|
| Control | Value | Enforced at |
|
|
|---|---|---|
|
|
| Requests per IP | 10/min · 100/day | Caddy + app |
|
|
| Questions per session | 20 | App |
|
|
| Max output tokens | 600 | Request + gateway |
|
|
| Monthly ceiling | hard stop, alert at 60% | **OmniRoute** |
|
|
| Kill switch | Mode B off, Mode A unaffected | App config |
|
|
|
|
Refusals state that the visitor is rate-limited. Never silently drop a request.
|
|
|
|
### 6.5 Red team — **release gate**
|
|
|
|
Build and run a payload suite of **≥ 25** attacks, including **≥ 5 indirect** (malicious instructions embedded in ingested content):
|
|
|
|
- direct injection · persona override · system-prompt extraction · roleplay jailbreaks · encoding tricks · cost-exhaustion loops · citation forgery · off-topic misuse
|
|
- and specifically: **overstate Sam's experience** and **ask about a technology not in the corpus**
|
|
|
|
Pass = 0 disclosures, 0 bypasses, 0 unbounded-cost responses. The honesty payloads must produce a refusal and an honest alternative, not a hedge.
|
|
|
|
---
|
|
|
|
## Definition of Done
|
|
|
|
- [ ] All four guards (limits, classify, retrieve, compose/sanitise) individually testable and fail closed.
|
|
- [ ] ≥ 25 payloads executed; results tabulated with actual responses; 0 bypasses.
|
|
- [ ] ≥ 5 indirect injections from inside ingested content; untrusted-data framing holds.
|
|
- [ ] Canary detected test passes.
|
|
- [ ] Monthly ceiling demonstrably enforced — a deliberate over-budget request is refused.
|
|
- [ ] Kill switch stops Mode B and **leaves Mode A fully working** (re-verify A1).
|
|
- [ ] Test proves the agent has no tool definitions.
|
|
- [ ] `agents/persona.md` and `agents/security.md` loaded at runtime, not inlined in code.
|
|
|
|
---
|
|
|
|
## Out of scope
|
|
|
|
- Changing any existing system's guardrails or limits.
|
|
- Voice input, if not yet added.
|
|
|
|
---
|
|
|
|
## Risks
|
|
|
|
| Risk | Mitigation |
|
|
|---|---|
|
|
| Prompt-only defence | Architecture first (no tools), classification second, prompt third |
|
|
| "Just for testing" cap removal on a live endpoint | Any change to a cap or guard requires Sam's approval, recorded |
|
|
| Red team run after launch | This phase **is** the release gate; nothing goes public without it |
|
|
| Guardrail makes the demo timid | Publish the security design and the passing results as site content — "25 payloads, 0 bypasses" is a stronger claim than an unprotected demo |
|