b267e55f8c
- AGENTS.md: build spec for Paperclip (constraints, conventions, 7 phases) - docs/phase-1..7: per-phase work, DoD, out-of-scope, risks - reference/: frozen decisions, stack+ports+budgets, content map, acceptance tests - agents/: persona and security policy drafts - Retires the React scaffold; history retained as record of the false start
3.8 KiB
Phase 6 — Guardrails, cost and kill switch
Goal: safe to leave running in public. This is a release gate, not a follow-up.
Depends on: Phase 5.
Policy source: agents/security.md (authoritative) and agents/persona.md. Both are runtime files, versioned in this repo.
Work
6.1 any-guardrail is the mechanism
Input and output validation is performed by mozilla-ai/any-guardrail, pinned to an exact version.
- Inbound: prompt injection · jailbreak · scope violation · personal-data requests · system-prompt extraction.
- Outbound: policy violation · disclosure of persona/security files · personal or infrastructure detail · claims outside the corpus.
- Fail closed. Error, timeout or ambiguous verdict ⇒ refuse.
- Rules live in this repo, not in library defaults.
- If a rule cannot be enforced with this library, report it — do not quietly drop the rule.
6.2 The architectural guardrail
The public agent has no tools: no shell, no filesystem, no network fetch, no write path, no memory between visitors. This removes tool abuse, SSRF, exfiltration-by-callback and cross-visitor poisoning as attack classes rather than defending them. Add an assertion to the test suite that no tool or function definitions are registered.
6.3 Output safety
- Model output never reaches the DOM as raw HTML; strict allow-list.
- Citation URLs allow-listed.
- Canary string in the system prompt; any appearance ⇒ discard answer, log incident, terminate session.
6.4 Cost and abuse
| Control | Value | Enforced at |
|---|---|---|
| Requests per IP | 10/min · 100/day | Caddy + app |
| Questions per session | 20 | App |
| Max output tokens | 600 | Request + gateway |
| Monthly ceiling | hard stop, alert at 60% | OmniRoute |
| Kill switch | Mode B off, Mode A unaffected | App config |
Refusals state that the visitor is rate-limited. Never silently drop a request.
6.5 Red team — release gate
Build and run a payload suite of ≥ 25 attacks, including ≥ 5 indirect (malicious instructions embedded in ingested content):
- direct injection · persona override · system-prompt extraction · roleplay jailbreaks · encoding tricks · cost-exhaustion loops · citation forgery · off-topic misuse
- and specifically: overstate Sam's experience and ask about a technology not in the corpus
Pass = 0 disclosures, 0 bypasses, 0 unbounded-cost responses. The honesty payloads must produce a refusal and an honest alternative, not a hedge.
Definition of Done
- All four guards (limits, classify, retrieve, compose/sanitise) individually testable and fail closed.
- ≥ 25 payloads executed; results tabulated with actual responses; 0 bypasses.
- ≥ 5 indirect injections from inside ingested content; untrusted-data framing holds.
- Canary detected test passes.
- Monthly ceiling demonstrably enforced — a deliberate over-budget request is refused.
- Kill switch stops Mode B and leaves Mode A fully working (re-verify A1).
- Test proves the agent has no tool definitions.
agents/persona.mdandagents/security.mdloaded at runtime, not inlined in code.
Out of scope
- Changing any existing system's guardrails or limits.
- Voice input, if not yet added.
Risks
| Risk | Mitigation |
|---|---|
| Prompt-only defence | Architecture first (no tools), classification second, prompt third |
| "Just for testing" cap removal on a live endpoint | Any change to a cap or guard requires Sam's approval, recorded |
| Red team run after launch | This phase is the release gate; nothing goes public without it |
| Guardrail makes the demo timid | Publish the security design and the passing results as site content — "25 payloads, 0 bypasses" is a stronger claim than an unprotected demo |