central_ai_llm_master/docs/outline/04-agent-runbook.md

# Central AI Hub — AI Agent Runbook

> **Writing Style**: ASD-STE100 Simplified Technical English (`ste-writing`).

## Phase 1 — Prove the Paperclip Round Trip

This phase has no Hub code. It proves the missing link: a job that leaves and a
result that returns.

### Steps

1. Create a Paperclip agent API key for this project. **DONE.**
   - Paperclip board: `http://192.168.20.13:3100`.
   - Agent: `Hub`, id `108c6220-c9e1-4b62-9e0a-6a763e0d28fd`.
   - Key name: `hub-service`.
   - The key is stored in `/home/sam/.config/environment.d/10-secrets.conf` as
     `PAPERCLIP_API_KEY`. Never commit it.
   - Verification: `GET /api/agents/me` with the key returns `200` and the name
     `Hub`.
   - Note: `POST /api/agents/{id}/keys` is **board-only**. A run JWT gets
     `403 Board access required`. Mint keys with the local board key.
2. File one job with the round-trip script. The script uses the issue-create
   endpoint: `POST /api/companies/{companyId}/issues`.
   - The old `POST /api/issues` route does not exist. It returns `404`.
   - Company id: `d4921013-e5da-4d88-8016-57d5746e2d03`.
   - Script: `scripts/paperclip-roundtrip.sh`.
3. Confirm the job appears in the Paperclip board. **DONE.**
   - The first job is `AUD-55`, id `d58891ce-9934-4710-9bba-d37974251ed9`.
4. Wait for the agent to close the job. **DONE.**
   - `AUD-55` closed as `done` at `2026-10-10T06:21:03Z`.
   - The agent comment on `AUD-55`: "The Hub round trip works."
5. Send the result to Jervis `POST /announce` so the house speaks it. **DONE.**
   - See the announce call and the response below.
6. Record the result in this file. **DONE.**

### Commands

```bash
# File one job with the script. It reads PAPERCLIP_API_KEY from the
# environment, then from ~/.config/environment.d/10-secrets.conf.
scripts/paperclip-roundtrip.sh \
  --title "Hub round-trip job: confirm the return path" \
  --description "Prove the outbound Paperclip path." \
  --assignee 403de1f7-645c-4d35-bc7e-19b01d77bacc \
  --priority high \
  --project 7ae791bb-8d9a-462f-99bf-013421d20623

# Speak the result.
curl -sS -X POST http://192.168.20.13:8501/announce \
  -H "Content-Type: application/json" \
  -d '{"text":"The Hub round trip works."}'
```

### Recorded Result — 2026-10-10

The round trip works. This table holds the proof.

| Item | Value |
|---|---|
| Key source | `PAPERCLIP_API_KEY` in `/home/sam/.config/environment.d/10-secrets.conf` |
| Key name | `hub-service` |
| Agent | `Hub`, id `108c6220-c9e1-4b62-9e0a-6a763e0d28fd` |
| Job identifier | `AUD-55`, id `d58891ce-9934-4710-9bba-d37974251ed9` |
| Job result | "The Hub round trip works." |
| Job final status | `done`, at `2026-10-10T06:21:03Z` |
| Announce call | `POST http://192.168.20.13:8501/announce` |
| Announce response | HTTP 200, text "The Hub round trip works." |

The script wrote this line to `.state/paperclip-roundtrip-issues.log`:

```text
2026-10-10T06:20:48Z	d58891ce-9934-4710-9bba-d37974251ed9	AUD-55
```

Note: Jervis has no playback-telemetry endpoint. Thus the HTTP 200 code and
the spoken reply are the strongest available proof of the announce. The
announce ran one time.

### Definition of Done

- The job appears in Paperclip. **DONE.**
- The job closes. **DONE.**
- The house speaks the result. **DONE.**
- This file records the result. **DONE.**

## Phase 2 — Hub Skeleton (AUD-49)

This phase builds the Hub itself. The shape is frozen in `docs/hub-api.md`. The
code layout is in `docs/outline/03-code-map.md`.

### What was built

- `POST /ask`, `POST /approve`, and `GET /health`.
- The five steps: `hear` → `understand` → `check` → `route` → `answer`.
- Routing through the LLM at OmniRoute, with the capability topics and
  `routing.md`.
- The never-route check runs before the LLM. A control action never reaches a
  model.
- One real destination: the house speakers (Jervis `/announce`).
- A fake `/ask` project for tests and manual runs.

### Test command

```bash
docker run --rm -v "$PWD/src":/src -w /src golang:1.25-alpine go test ./...
```

Result on 2026-10-10: all packages pass, including the fake `/ask` round trip.

### Live round trip

The Hub ran on port 18788 with `HUB_PROJECTS="compass_nhs=http://127.0.0.1:9199"`
and the fake project on 9199. The real OmniRoute LLM chose the project. Jervis
spoke the reply on the house speakers.

Request:

```json
{
  "text": "what does Harry have on today?",
  "asker": "sam",
  "source": "esp32",
  "device": "kitchen-esp32",
  "request_id": "01AUD49PROOF"
}
```

Response:

```json
{
  "status": "ok",
  "request_id": "01AUD49PROOF",
  "answer": "Harry has maths and a music lesson today.",
  "shape": "speech",
  "speech": {"text": "Harry has maths and a music lesson today.", "target": "house"},
  "route": {"kind": "project", "project": "compass_nhs"}
}
```

Hub stdout (evidence):

```text
router: model=gpt-4o-mini status=200 elapsed=1.554598547s
speakers: announced 41 chars
```

The routing LLM answered in 1.5 s (inside the 3 s cap). The house announcement
ran out of band and Jervis returned HTTP 200. Jervis has no playback telemetry,
so the HTTP 200 and the spoken reply are the strongest available proof.

### Control action proof

`"cut the kids internet"` returned `status: "refused"`, `route.kind: "refused"`,
`route.reason: "high_consequence_control"`. No LLM call ran for that request.
The unit test `TestControlActionNeverReachesLLM` asserts the zero call count.

### Deploy note

The `docker-compose.yml` mounts `routing.md` and the secrets file read-only, and
uses a `/tmp` tmpfs for the log. To deploy: build the image, then restart the
container. A `HUB_TOKEN_*` key per client must exist in the secrets file first.

## Phase 2 — Deploy the Hub (AUD-56)

The AUD-49 image is built as `central-ai-hub:local`. The `hub` container on port
8788 must serve that image, and the secrets file must hold one token per door.

### Step 1 — Provision the Hub client tokens

Run the script. It adds only the missing `HUB_TOKEN_*` keys. It never prints a
token value and never overwrites a token.

```bash
scripts/provision-hub-tokens.sh --check   # report present or missing, no change
scripts/provision-hub-tokens.sh           # add the missing keys
```

The script writes the keys to `~/.config/environment.d/10-secrets.conf`, keeps
mode 600, and leaves a mode-600 timestamped backup. The keys are:
`HUB_TOKEN_PHONE`, `HUB_TOKEN_ESP32`, `HUB_TOKEN_DSH`, `HUB_TOKEN_CONSOLE`,
`HUB_TOKEN_ADMIN`.

This step changes host credentials. It needs Sam's OK first
(`AGENTS.md` "ask first: credentials").

> This host has no `openssl`. The script uses `openssl rand -hex 32` when it is
> present, and `/dev/urandom` (32 bytes, hex) otherwise. Both give 64 hex
> characters.

### Step 2 — Deploy the image

```bash
cd /home/sam/paseo/projects/central_ai_llm_master
docker compose up -d --build
```

### Step 3 — Verify

```bash
curl -s http://192.168.20.13:8788/health
docker inspect hub --format '{{.State.Health.Status}}'
```

Expected: `ok: true`, `omniroute: ok`, and `healthy`. The control action must be
refused:

```bash
curl -s -X POST http://192.168.20.13:8788/ask \
  -H "Authorization: Bearer $HUB_TOKEN_ESP32" \
  -H "Content-Type: application/json" \
  -d '{"text":"cut the kids internet","asker":"sam","source":"esp32"}'
```

Expected: `status: "refused"`, `route.kind: "refused"`.

`HUB_PROJECTS` is empty in this phase. A project question therefore returns
`status: "unavailable"` with `route.reason: "no_client"`. That is correct.

### Finding — the container user

The secrets file has mode 0600 and is owned by the host user. The image's
distroless `nonroot` user (65532) cannot read that file. With `nonroot`, the Hub
starts but reads no tokens: every `/ask` returns 401 and `/health` reports
`omniroute: unknown`.

Fix: `docker-compose.yml` runs the container as the host user
(`user: "${HUB_UID:-1000}:${HUB_GID:-100}"`). No host file change is needed.

### Dry-run evidence — 2026-10-10

The AUD-49 image (`sha256:eb062c638da5`) ran on port 18788 against a scratch
secrets file with freshly generated tokens. It did not touch the host secrets
file or the running `hub` container.

| Check | Result |
|---|---|
| `GET /health` | `ok: true`, `checks.omniroute: ok`, `checks.log: ok` |
| `POST /ask` `"cut the kids internet"` | `status: "refused"`, `route.kind: "refused"` |
| `POST /ask` with a bad token | HTTP 401 `unauthorized` |
| `POST /ask` a project question (no project) | `status: "unavailable"`, `route.reason: "no_client"` |
| `docker exec <container> /app/hub -healthcheck` | exit 0 (the compose healthcheck) |
| Container with the image's default `nonroot` user | 401 on `/ask`, `omniroute: unknown` (the finding above) |

### Live deploy result — 2026-10-10

Sam accepted the approval card. The deploy ran from the project folder.

| Step | Result |
|---|---|
| `scripts/provision-hub-tokens.sh` | Added 5 keys to the secrets file. Mode 600 kept. Backup `10-secrets.conf.bak.20261010T065304Z`. Values not shown. |
| `docker compose up -d --build` | Exit 0. Container `hub` recreated and started. |
| Image | `central-ai-hub:local`, `sha256:f6b773c8a33a` |
| Container user | `1000:100` |
| `GET /health` | `ok: true`, `omniroute: ok`, `log: ok` |
| `POST /ask` `"cut the kids internet"` (`HUB_TOKEN_ESP32`) | `status: "refused"`, `route.kind: "refused"`, `route.reason: "high_consequence_control"` |
| `POST /ask` with a bad token | HTTP 401 `unauthorized` |
| `docker inspect hub` | `healthy` |

The control refusal ran with no LLM call, as the `understand/control.go` phrase
check runs first. The Hub now serves the AUD-49 build on port 8788.

## After Phase 1

Follow the build order in `AGENTS.md`. Do not start a destination before the
round trip works.

## Health Checks

```bash
curl -s http://192.168.20.13:3100/health        # Paperclip
curl -s http://192.168.20.13:8501/health        # Jervis
curl -s -o /dev/null -w '%{http_code}\n' http://192.168.20.30:8123/   # HA
```