clean tests run: pi 1.5.0 model-list shim, panes checked by process, board r refreshes detail, office . accepted

This commit is contained in:
Sam
2026-11-12 15:49:56 +11:00
parent b81cb91831
commit aa6eb7429d
18 changed files with 1060 additions and 8 deletions

View File

@@ -127,3 +127,102 @@ only `README.md`, and cancelling removes it from the board without touching `mai
For each step: what you pressed, what appeared, and anything that did not match. That is worth
more than a verdict, because it says where the interface and Paperclip disagree.
---
# Results, 2026-11-12
Both tests were run on `.13`, driven with `tuios send-text`, `tuios send-keys` and
`tuios capture-pane`, so every step below is what the screen actually showed.
## Test 1 — passed, after one real defect was found and fixed
| Step | What happened |
|---|---|
| folder + `git init` | ok |
| `office --new "test-alpha"` | project, session and eight panes created |
| `New task>new` → **AUD-172** | created and handed to Builder; the Assign field takes the word `yes` |
| first three runs | **all failed in 4–5 s**, `errorCode: adapter_failed`, `logBytes: 0` |
| the fix | see "Pi 1.5.0 and the model list" below |
| the run after the fix | **succeeded in 69 s** |
| `Work>change` | worktree, branch, base, commits, `STATUS.md \| 1 +`, and the diff |
| `Work>review` (`v`) | `ocr` **selected 0 files** — it does not review Markdown |
| `Work>merge` (`m`, `y`) | rebased, no verify command, review recorded as "nothing to review", `merged into main at 36acb8d` |
| `main` | `STATUS.md` present, contents `alpha is running.` |
**What did not work as the test expected**
- **Cost is always zero.** `Project>costs` and `Board>the chosen task` report `0 cent(s)` and
`0 tokens out` over four runs. `usageJson` on the run is `null`: the `pi_local` adapter does not
capture token usage from Pi, so cost cannot be shown for these runs. Open item.
- **`ocr` reviews code, not Markdown.** A documentation-only change gets `status: skipped`,
`selected: []`. `bin/merge` already treats this as "nothing to review, recorded, not hidden",
so it does not block a merge — but a reviewer reading the tab sees "nothing reviewable" while
the `change` tab lists one file.
## Test 2 — passed, after one real defect was found and fixed
| Step | What happened |
|---|---|
| `office . --project harness-repo --session harness-repo` | adopted the project and the panes; `.harness/` was not rewritten |
| panes after the reboot | **restored shells**, and `office` called them "adopted" |
| the fix | a pane is adopted only if its own process is the harness |
| `New task>new` → **AUD-173** | created and handed to Builder; the run succeeded in 24 s |
| the diff | `README.md \| 1 +`, nothing else, committed on the branch |
| merge | **not merged on purpose** |
| `c`, then `y` in `Board>board` | task `cancelled`; status becomes cancelled, thread and runs stay |
| `main` | untouched — `README.md` has no new line, last commit still `b81cb91` |
**What did not work as the test expected**
- **After a reboot, every pane is a bare shell and the pane keeps its name.** `board` matched the
panes by name and reported them "adopted" while nothing was running in them. A page that looks
built but does nothing is the worst kind of wrong. Now `open_pane` reads
`/proc/<pid>/cmdline` and adopts a pane only when the harness is what is running; otherwise it
closes the pane and opens a fresh one. `shell` is exempt: it is a shell either way.
- **`Board>board` `r` refreshed the task list but not the selected task's detail**, so
"last run running" stayed on screen minutes after the run finished. `r` now reloads both.
- A cancelled task is grouped under **Done** and its row reads `gone`. That is deliberate
(Paperclip records it as finished) but it is worth knowing before you look for it.
## Pi 1.5.0 and the model list — where names live, and what changed
Measured on `.13`:
| File | What it holds |
|---|---|
| `~/.agents/models.json` (symlinked as `~/.pi/agent/models.json`) | provider `omni` with **476 explicit model names**, and `openrouter` overrides |
| `~/.pi/agent/settings.json` | five providers with `baseUrl`, `api`, `apiKey`: `omni`, `deepseek`, `openrouter`, `opencode-go`, `google` |
| `~/.pi/agent/models-store.json` | **discovered** catalogues per provider with `etag` and `checkedAt`, e.g. `deepseek` → 2 models |
| a project's `.pi/` | no model names at all; for an agent workspace it holds only `.pi/tasks` |
So model names are global, not per project. An agent workspace's `.pi/` cannot hold a stale copy.
The names are there: `deepseek deepseek-flash`, `deepseek deepseek-v4-pro`, and through the router
`omni deepseek/deepseek-v4-flash`, `omni deepseek/deepseek-v4-pro`, `omni ds/…`,
`omni oc/deepseek-v4-flash-free`, `omni opencode-go/…`. The configured agent model
`omni/deepseek/deepseek-v4-flash` resolves.
**The change that broke Paperclip is not the names, it is the stream.** Pi 1.5.0 prints
`--list-models` on **stdout**; Pi 0.8x printed it on **stderr**. Paperclip's `pi_local` adapter
reads stderr first:
```js
// Pi outputs model list to stderr, but fall back to stdout for older versions
const output = result.stderr || result.stdout;
```
Five extension warnings now live on stderr, so the adapter parsed those, found no `provider model`
rows, and refused every run with `Pi returned no models`. The shim that fixes it,
`~/.local/share/paperclip-harness/wrappers/pi-paperclip.sh`, puts the table back on stderr for
that one flag and passes everything else through unchanged; the Builder agent's
`adapterConfig.command` points at it.
**Consequences to keep in mind**
- **omniroute** is the source of the `omni` names. A name that is missing from the router's
`/v1/models` will not appear in Pi, and `models-store.json` caches with an `etag`, so a stale
entry can hide a new name until the catalogue is refreshed.
- **headroom** keeps its own model registry (LiteLLM) and still has no entries for the router's
ids, which is why it answers `502`. It is unrelated to Pi's storage, but when it is fixed it
will need the same names.

View File

@@ -154,3 +154,33 @@ after 30.5 s, then `200` in 21.7 s, so the command retries exactly once and says
## Open Paperclip tasks in this project
One: `AUD-162`, a leftover test task in `todo` from the first day. Everything else is `done`.
## Done since: the two clean tests were run, 2026-11-12
Both are recorded step by step in `docs/CLEAN-TESTS.md`. Four defects came out of them and all
four are fixed: the pi 1.5.0 model-list stream (which failed **every** agent run), panes restored
by TUIOS after a reboot being "adopted" while running nothing, `r` on the board not refreshing the
chosen task's detail, and `office .` being refused as if no folder had been given.
Two open items came out of them, and neither is small:
### A. Cost is always zero for `pi_local` runs
`GET /api/issues/{id}/cost-summary` reports `costCents: 0`, `outputTokens: 0` over four runs of a
task that really ran, and the run's `usageJson` is `null`. Pi's JSON stream carries token usage;
the `pi_local` adapter is not capturing it. Until it does, `Project>costs` and the cost line on
`Board>the chosen task` are honest but useless, and the cost limit in the operating rules cannot
bite.
### B. `ocr` reviews code, not Markdown
A documentation-only diff gives `status: skipped`, `coverage.selected: []`. `bin/merge` handles
that correctly, but `Work>review` should say *why* rather than "nothing reviewable in the change
(0 files)", because the `change` tab next to it lists one file.
### C. The suite needs to survive a reboot
After a reboot TUIOS restores a session with a fresh shell in every pane and keeps the pane names.
`board` now checks the pane's process instead of trusting the name. What is still missing is a
single command that repairs this project after a reboot without a person looking at each pane —
`board` is that command now, and it does rebuild, but nothing runs it automatically.