central_ai_llm_master/docs/outline/04-agent-runbook.md

Central AI Hub — AI Agent Runbook

Writing Style: ASD-STE100 Simplified Technical English (ste-writing).

Phase 1 — Prove the Paperclip Round Trip

This phase has no Hub code. It proves the missing link: a job that leaves and a result that returns.

Steps

1. Create a Paperclip agent API key for this project. DONE.
   • Paperclip board: http://192.168.20.13:3100.
   • Agent: Hub, id 108c6220-c9e1-4b62-9e0a-6a763e0d28fd.
   • Key name: hub-service.
   • The key is stored in /home/sam/.config/environment.d/10-secrets.conf as PAPERCLIP_API_KEY. Never commit it.
   • Verification: GET /api/agents/me with the key returns 200 and the name Hub.
   • Note: POST /api/agents/{id}/keys is board-only. A run JWT gets 403 Board access required. Mint keys with the local board key.
2. File one job with the round-trip script. The script uses the issue-create endpoint: POST /api/companies/{companyId}/issues.
   • The old POST /api/issues route does not exist. It returns 404.
   • Company id: d4921013-e5da-4d88-8016-57d5746e2d03.
   • Script: scripts/paperclip-roundtrip.sh.
3. Confirm the job appears in the Paperclip board. DONE.
   • The first job is AUD-55, id d58891ce-9934-4710-9bba-d37974251ed9.
4. Wait for the agent to close the job. DONE.
   • AUD-55 closed as done at 2026-10-10T06:21:03Z.
   • The agent comment on AUD-55: "The Hub round trip works."
5. Send the result to Jervis POST /announce so the house speaks it. DONE.
   • See the announce call and the response below.
6. Record the result in this file. DONE.

Commands

# File one job with the script. It reads PAPERCLIP_API_KEY from the
# environment, then from ~/.config/environment.d/10-secrets.conf.
scripts/paperclip-roundtrip.sh \
  --title "Hub round-trip job: confirm the return path" \
  --description "Prove the outbound Paperclip path." \
  --assignee 403de1f7-645c-4d35-bc7e-19b01d77bacc \
  --priority high \
  --project 7ae791bb-8d9a-462f-99bf-013421d20623

# Speak the result.
curl -sS -X POST http://192.168.20.13:8501/announce \
  -H "Content-Type: application/json" \
  -d '{"text":"The Hub round trip works."}'

Recorded Result — 2026-10-10

The round trip works. This table holds the proof.

Item	Value
Key source	PAPERCLIP_API_KEY in /home/sam/.config/environment.d/10-secrets.conf
Key name	hub-service
Agent	Hub, id 108c6220-c9e1-4b62-9e0a-6a763e0d28fd
Job identifier	AUD-55, id d58891ce-9934-4710-9bba-d37974251ed9
Job result	"The Hub round trip works."
Job final status	done, at 2026-10-10T06:21:03Z
Announce call	POST http://192.168.20.13:8501/announce
Announce response	HTTP 200, text "The Hub round trip works."

The script wrote this line to .state/paperclip-roundtrip-issues.log:

2026-10-10T06:20:48Z	d58891ce-9934-4710-9bba-d37974251ed9	AUD-55

Note: Jervis has no playback-telemetry endpoint. Thus the HTTP 200 code and the spoken reply are the strongest available proof of the announce. The announce ran one time.

Definition of Done

• The job appears in Paperclip. DONE.
• The job closes. DONE.
• The house speaks the result. DONE.
• This file records the result. DONE.

Phase 2 — Hub Skeleton (AUD-49)

This phase builds the Hub itself. The shape is frozen in docs/hub-api.md. The code layout is in docs/outline/03-code-map.md.

What was built

• POST /ask, POST /approve, and GET /health.
• The five steps: hear → understand → check → route → answer.
• Routing through the LLM at OmniRoute, with the capability topics and routing.md.
• The never-route check runs before the LLM. A control action never reaches a model.
• One real destination: the house speakers (Jervis /announce).
• A fake /ask project for tests and manual runs.

Test command

docker run --rm -v "$PWD/src":/src -w /src golang:1.25-alpine go test ./...

Result on 2026-10-10: all packages pass, including the fake /ask round trip.

Live round trip

The Hub ran on port 18788 with HUB_PROJECTS="compass_nhs=http://127.0.0.1:9199" and the fake project on 9199. The real OmniRoute LLM chose the project. Jervis spoke the reply on the house speakers.

Request:

{
  "text": "what does Harry have on today?",
  "asker": "sam",
  "source": "esp32",
  "device": "kitchen-esp32",
  "request_id": "01AUD49PROOF"
}

Response:

{
  "status": "ok",
  "request_id": "01AUD49PROOF",
  "answer": "Harry has maths and a music lesson today.",
  "shape": "speech",
  "speech": {"text": "Harry has maths and a music lesson today.", "target": "house"},
  "route": {"kind": "project", "project": "compass_nhs"}
}

Hub stdout (evidence):

router: model=gpt-4o-mini status=200 elapsed=1.554598547s
speakers: announced 41 chars

The routing LLM answered in 1.5 s (inside the 3 s cap). The house announcement ran out of band and Jervis returned HTTP 200. Jervis has no playback telemetry, so the HTTP 200 and the spoken reply are the strongest available proof.

Control action proof

"cut the kids internet" returned status: "refused", route.kind: "refused", route.reason: "high_consequence_control". No LLM call ran for that request. The unit test TestControlActionNeverReachesLLM asserts the zero call count.

Deploy note

The docker-compose.yml mounts routing.md and the secrets file read-only, and uses a /tmp tmpfs for the log. To deploy: build the image, then restart the container. A HUB_TOKEN_* key per client must exist in the secrets file first.

Phase 2 — Deploy the Hub (AUD-56)

The AUD-49 image is built as central-ai-hub:local. The hub container on port 8788 must serve that image, and the secrets file must hold one token per door.

Step 1 — Provision the Hub client tokens

Run the script. It adds only the missing HUB_TOKEN_* keys. It never prints a token value and never overwrites a token.

scripts/provision-hub-tokens.sh --check   # report present or missing, no change
scripts/provision-hub-tokens.sh           # add the missing keys

The script writes the keys to ~/.config/environment.d/10-secrets.conf, keeps mode 600, and leaves a mode-600 timestamped backup. The keys are: HUB_TOKEN_PHONE, HUB_TOKEN_ESP32, HUB_TOKEN_DSH, HUB_TOKEN_CONSOLE, HUB_TOKEN_ADMIN.

This step changes host credentials. It needs Sam's OK first (AGENTS.md "ask first: credentials").

This host has no openssl. The script uses openssl rand -hex 32 when it is present, and /dev/urandom (32 bytes, hex) otherwise. Both give 64 hex characters.

Step 2 — Deploy the image

cd /home/sam/paseo/projects/central_ai_llm_master
docker compose up -d --build

Step 3 — Verify

curl -s http://192.168.20.13:8788/health
docker inspect hub --format '{{.State.Health.Status}}'

Expected: ok: true, omniroute: ok, and healthy. The control action must be refused:

curl -s -X POST http://192.168.20.13:8788/ask \
  -H "Authorization: Bearer $HUB_TOKEN_ESP32" \
  -H "Content-Type: application/json" \
  -d '{"text":"cut the kids internet","asker":"sam","source":"esp32"}'

Expected: status: "refused", route.kind: "refused".

HUB_PROJECTS is empty in this phase. A project question therefore returns status: "unavailable" with route.reason: "no_client". That is correct.

Finding — the container user

The secrets file has mode 0600 and is owned by the host user. The image's distroless nonroot user (65532) cannot read that file. With nonroot, the Hub starts but reads no tokens: every /ask returns 401 and /health reports omniroute: unknown.

Fix: docker-compose.yml runs the container as the host user (user: "${HUB_UID:-1000}:${HUB_GID:-100}"). No host file change is needed.

Dry-run evidence — 2026-10-10

The AUD-49 image (sha256:eb062c638da5) ran on port 18788 against a scratch secrets file with freshly generated tokens. It did not touch the host secrets file or the running hub container.

Check	Result
GET /health	ok: true, checks.omniroute: ok, checks.log: ok
POST /ask "cut the kids internet"	status: "refused", route.kind: "refused"
POST /ask with a bad token	HTTP 401 unauthorized
POST /ask a project question (no project)	status: "unavailable", route.reason: "no_client"
docker exec <container> /app/hub -healthcheck	exit 0 (the compose healthcheck)
Container with the image's default nonroot user	401 on /ask, omniroute: unknown (the finding above)

Live deploy result — 2026-10-10

Sam accepted the approval card. The deploy ran from the project folder.

Step	Result
scripts/provision-hub-tokens.sh	Added 5 keys to the secrets file. Mode 600 kept. Backup 10-secrets.conf.bak.20261010T065304Z. Values not shown.
docker compose up -d --build	Exit 0. Container hub recreated and started.
Image	central-ai-hub:local, sha256:f6b773c8a33a
Container user	1000:100
GET /health	ok: true, omniroute: ok, log: ok
POST /ask "cut the kids internet" (HUB_TOKEN_ESP32)	status: "refused", route.kind: "refused", route.reason: "high_consequence_control"
POST /ask with a bad token	HTTP 401 unauthorized
docker inspect hub	healthy

The control refusal ran with no LLM call, as the understand/control.go phrase check runs first. The Hub now serves the AUD-49 build on port 8788.

After Phase 1

Follow the build order in AGENTS.md. Do not start a destination before the round trip works.

Health Checks

curl -s http://192.168.20.13:3100/health        # Paperclip
curl -s http://192.168.20.13:8501/health        # Jervis
curl -s -o /dev/null -w '%{http_code}\n' http://192.168.20.30:8123/   # HA