paperclip-harness/docs/CLEAN-TESTS.md

# The two clean tests, step by step

Both tests answer one question: can a person get from a folder to a working set of panes, and
then from a task to merged work, without editing a file.

Run them in this order. Write down what you see; that is the useful part.

---

## Test 1 — a project that does not exist yet

### Step 1. Make the folder and its repository

```sh
mkdir -p ~/paseo/projects/test-alpha
cd ~/paseo/projects/test-alpha
git init -b main
printf '# test alpha\n' > README.md
git add . && git commit -m "initial commit"
```

### Step 2. Adopt it with one command

```sh
cd ~/paseo/projects/paperclip-harness
bin/office ~/paseo/projects/test-alpha --new "test-alpha"
```

`--new` creates the Paperclip project, its workspace and the worktree policy. Without `--new`,
`--project <id|name>` adopts a project that already exists.

Expected output:

```
project: test-alpha created (…)
folder:  /home/sam/paseo/projects/test-alpha
session: test-alpha
machine: nixos-desktop
wrote   …/.harness/harness.toml
wrote   …/.harness/check.sh
wrote   …/.harness/NOTES.md
session: test-alpha (created; switch to it with TUIOS's session switcher)
  shell       created  Shell
  board       created  Board
  …
workspaces in use: 8
```

### Step 3. Show the session

A program cannot move your client: TUIOS refuses that from a pane. Use TUIOS's own session
switcher, or from a terminal:

```sh
tuios attach test-alpha
```

### Step 4. Check the panes

1. `Repair>repair` → every check passes.
2. `Shell>shell` → the box at the top says the project folder is `…/test-alpha`.
3. `Board>board` → "no tasks in this project".
4. Press `R` once. All eight panes come back, still pointing at `test-alpha`.

### Step 5. One task, end to end

1. `New task>new`: title `Add a STATUS.md with one line`, description `Create STATUS.md at the
   repository root with the line: alpha is running. Change no other file. Commit it.`
   Leave the blocker and the command empty. Assign `yes`. Press `Enter` on the last field.
2. `Board>board`: the task appears and goes `todo` → `in progress` → `done`.
3. `Agents>agents`: `)` to **runs**, `↑` to the newest run, `Enter` → its log.
4. `Work>work`: `change` shows one file, one line. `review` says no review recorded yet.
5. `Work>work` → `v` starts a review; the tab shows it run.
6. `Work>work` → `m` → the dialog names the branch and `main` → `y`. The merge tab shows the
   rebase, the check, the review and `merged into main at …`.

**Pass:** the file exists on `main`, the task is `done`, cost is visible in `Project>costs`, and
nothing was typed into a terminal after step 2.

---

## Test 2 — a folder that already exists

This uses **this repository**, which is real code with real history.

### Step 1. Adopt it into its own session

```sh
cd ~/paseo/projects/paperclip-harness
bin/office . --project harness-repo --session harness-repo
```

The project `harness-repo` already exists in Paperclip and points at this folder, so `--project`
adopts it. A separate session is used on purpose: the working session `paperclip-harness` shows
the sandbox project, and this test must not rebuild it.

### Step 2. Confirm nothing was overwritten

```sh
git status --short
```

Expected: only new files under `.harness/` (the project file, `check.sh`, `NOTES.md`). Nothing
existing is changed. If you see any modified file that you did not touch, stop and say so.

### Step 3. Show the session and look

```sh
tuios attach harness-repo
```

1. `Project>project` → `projects` tab lists both `spike-2026-11-10` and `harness-repo`, each with
   its folder and machine.
2. `Board>board` → empty, because this project has no tasks yet.
3. `New task>new` → create `Add a line to README`, description `Add the line "tested by the
   harness" at the end of README.md. Change nothing else. Commit it.`, assign `yes`.
4. Watch it in `Agents>agents`, read the diff in `Work>work`. **Do not merge it** — this is the
   repository's own history. Cancel the task with `c` in `Board>board` instead, and confirm in
   `git log` that nothing landed.

**Pass:** the panes read this repository's own tasks, its own worktree is created, the diff shows
only `README.md`, and cancelling removes it from the board without touching `main`.

---

## What to write down

For each step: what you pressed, what appeared, and anything that did not match. That is worth
more than a verdict, because it says where the interface and Paperclip disagree.

---

# Results, 2026-11-12

Both tests were run on `.13`, driven with `tuios send-text`, `tuios send-keys` and
`tuios capture-pane`, so every step below is what the screen actually showed.

## Test 1 — passed, after one real defect was found and fixed

| Step | What happened |
|---|---|
| folder + `git init` | ok |
| `office --new "test-alpha"` | project, session and eight panes created |
| `New task>new` → **AUD-172** | created and handed to Builder; the Assign field takes the word `yes` |
| first three runs | **all failed in 4–5 s**, `errorCode: adapter_failed`, `logBytes: 0` |
| the fix | see "Pi 1.5.0 and the model list" below |
| the run after the fix | **succeeded in 69 s** |
| `Work>change` | worktree, branch, base, commits, `STATUS.md \| 1 +`, and the diff |
| `Work>review` (`v`) | `ocr` **selected 0 files** — it does not review Markdown |
| `Work>merge` (`m`, `y`) | rebased, no verify command, review recorded as "nothing to review", `merged into main at 36acb8d` |
| `main` | `STATUS.md` present, contents `alpha is running.` |

**What did not work as the test expected**

- **Cost is always zero.** `Project>costs` and `Board>the chosen task` report `0 cent(s)` and
  `0 tokens out` over four runs. `usageJson` on the run is `null`: the `pi_local` adapter does not
  capture token usage from Pi, so cost cannot be shown for these runs. Open item.
- **`ocr` reviews code, not Markdown.** A documentation-only change gets `status: skipped`,
  `selected: []`. `bin/merge` already treats this as "nothing to review, recorded, not hidden",
  so it does not block a merge — but a reviewer reading the tab sees "nothing reviewable" while
  the `change` tab lists one file.

## Test 2 — passed, after one real defect was found and fixed

| Step | What happened |
|---|---|
| `office . --project harness-repo --session harness-repo` | adopted the project and the panes; `.harness/` was not rewritten |
| panes after the reboot | **restored shells**, and `office` called them "adopted" |
| the fix | a pane is adopted only if its own process is the harness |
| `New task>new` → **AUD-173** | created and handed to Builder; the run succeeded in 24 s |
| the diff | `README.md \| 1 +`, nothing else, committed on the branch |
| merge | **not merged on purpose** |
| `c`, then `y` in `Board>board` | task `cancelled`; status becomes cancelled, thread and runs stay |
| `main` | untouched — `README.md` has no new line, last commit still `b81cb91` |

**What did not work as the test expected**

- **After a reboot, every pane is a bare shell and the pane keeps its name.** `board` matched the
  panes by name and reported them "adopted" while nothing was running in them. A page that looks
  built but does nothing is the worst kind of wrong. Now `open_pane` reads
  `/proc/<pid>/cmdline` and adopts a pane only when the harness is what is running; otherwise it
  closes the pane and opens a fresh one. `shell` is exempt: it is a shell either way.
- **`Board>board` `r` refreshed the task list but not the selected task's detail**, so
  "last run running" stayed on screen minutes after the run finished. `r` now reloads both.
- A cancelled task is grouped under **Done** and its row reads `gone`. That is deliberate
  (Paperclip records it as finished) but it is worth knowing before you look for it.

## Pi 1.5.0 and the model list — where names live, and what changed

Measured on `.13`:

| File | What it holds |
|---|---|
| `~/.agents/models.json` (symlinked as `~/.pi/agent/models.json`) | provider `omni` with **476 explicit model names**, and `openrouter` overrides |
| `~/.pi/agent/settings.json` | five providers with `baseUrl`, `api`, `apiKey`: `omni`, `deepseek`, `openrouter`, `opencode-go`, `google` |
| `~/.pi/agent/models-store.json` | **discovered** catalogues per provider with `etag` and `checkedAt`, e.g. `deepseek` → 2 models |
| a project's `.pi/` | no model names at all; for an agent workspace it holds only `.pi/tasks` |

So model names are global, not per project. An agent workspace's `.pi/` cannot hold a stale copy.

The names are there: `deepseek deepseek-flash`, `deepseek deepseek-v4-pro`, and through the router
`omni deepseek/deepseek-v4-flash`, `omni deepseek/deepseek-v4-pro`, `omni ds/…`,
`omni oc/deepseek-v4-flash-free`, `omni opencode-go/…`. The configured agent model
`omni/deepseek/deepseek-v4-flash` resolves.

**The change that broke Paperclip is not the names, it is the stream.** Pi 1.5.0 prints
`--list-models` on **stdout**; Pi 0.8x printed it on **stderr**. Paperclip's `pi_local` adapter
reads stderr first:

```js
// Pi outputs model list to stderr, but fall back to stdout for older versions
const output = result.stderr || result.stdout;
```

Five extension warnings now live on stderr, so the adapter parsed those, found no `provider model`
rows, and refused every run with `Pi returned no models`. The shim that fixes it,
`~/.local/share/paperclip-harness/wrappers/pi-paperclip.sh`, puts the table back on stderr for
that one flag and passes everything else through unchanged; the Builder agent's
`adapterConfig.command` points at it.

**Consequences to keep in mind**

- **omniroute** is the source of the `omni` names. A name that is missing from the router's
  `/v1/models` will not appear in Pi, and `models-store.json` caches with an `etag`, so a stale
  entry can hide a new name until the catalogue is refreshed.
- **headroom** keeps its own model registry (LiteLLM) and still has no entries for the router's
  ids, which is why it answers `502`. It is unrelated to Pi's storage, but when it is fixed it
  will need the same names.