Handover: history, plan, API measurements, checklist, open issues from the OMA attempt
This commit is contained in:
sam
193
docs/FALLBACK-OMA.md
Normal file
193
docs/FALLBACK-OMA.md
Normal file
@@ -0,0 +1,193 @@
|
||||
# Understanding OMA before planning anything else
|
||||
|
||||
**Objective set by the person, 2026-11-09:** understand `@open-multi-agent/core`
|
||||
fully, then build a plan against it. Not a rewrite. Not a second engine.
|
||||
|
||||
## What OMA is, first facts
|
||||
|
||||
- Package: `@open-multi-agent/core`, version 1.23.0, installed at
|
||||
`/home/sam/paseo/harness/phase2/node_modules/@open-multi-agent/core`.
|
||||
- Its own description: "Self-hosted TypeScript agent runtime with durable approvals
|
||||
and verifiable run records."
|
||||
- It ships a CLI, `oma`, with `run`, `task`, `dashboard`, `eval run`, `eval gate`,
|
||||
`provider`.
|
||||
- It exports, among others, `observability`, `observability/file`, `acp`, `process`,
|
||||
`mcp`, `ai-sdk`, `classifiers`, `shell`, `eval`, `eval/file`.
|
||||
|
||||
## Why this document exists
|
||||
|
||||
The harness grew a second implementation of things OMA already provides: an approval
|
||||
path, a viewer, a task starter, a card renderer. That is why faults arrived in threes
|
||||
and why the same question had two answers. This document records what OMA actually
|
||||
does, with evidence, so that the next plan puts each job in one place.
|
||||
|
||||
## Rules for the work that follows
|
||||
|
||||
1. The harness is the cockpit: pages, panes, notes, project files, the decision.
|
||||
2. OMA is the engine: coordinator, task graph, approvals, run records, the graph.
|
||||
3. Paperclip is inspiration, not a blueprint: approvals as objects, blockers as
|
||||
edges, a plan attached to each task, every action tied to its run.
|
||||
4. Nothing is deleted until its replacement has run a real task and the result can
|
||||
be shown.
|
||||
|
||||
## Findings
|
||||
|
||||
(to be filled from the research: CLI, team/tasks schemas, approval API, eval gate,
|
||||
trace store, process backend, upstream docs)
|
||||
|
||||
Read from the installed package (v1.23.0) on 2026-11-09. Evidence is
|
||||
`phase2/node_modules/@open-multi-agent/core/dist/...`.
|
||||
|
||||
### The CLI (`dist/cli/oma.js`)
|
||||
|
||||
- `oma run --goal <text> --team <team.json> [--orchestrator <orch.json>] [--coordinator <coord.json>]`
|
||||
- `oma task --file <tasks.json> [--team <team.json>]`
|
||||
- `oma dashboard --trace-store <traces.ndjson> --run-id <id> [--output <run.html>]`
|
||||
- `oma eval run --set <evalset.json> --target <target.mjs> [--scorers <s.mjs>] [--repeats n] [--concurrency n] [--tags a,b] [--report json|markdown|junit]... [--out dir] [--meta k=v]... [--gate <gate.json>] [--baseline <report.json>]`
|
||||
- `oma eval gate --report <report.json> --gate <gate.json> [--baseline <report.json>]`
|
||||
- `oma provider [list | template <provider>]`
|
||||
- Exit codes: `0` success, `1` run/eval gate failed, `2` usage/validation, `3` internal.
|
||||
- `team.json` may be a `TeamConfig` or `{ team, orchestrator }`. `tasks.json` is
|
||||
`{ team, tasks, orchestrator? }`.
|
||||
|
||||
### Task shape (`types.d.ts:1341-1377`, `RunTaskSpec`)
|
||||
|
||||
`title`, `description`, `assignee?`, `dependsOn?: string[]`,
|
||||
`memoryScope?: 'dependencies' | 'all'`, `dependencyPayload?: 'output' | 'structured' | 'both'`,
|
||||
`metadata?`, `maxRetries?`, `retryDelayMs?`, `retryBackoff?`, `role?`,
|
||||
`priority?: low|normal|high|critical`, `requires?` (requiredTools, requiredCapabilities,
|
||||
requiredBackend, requiredProvider), and **`verify?: ConsensusVerifyOptions`**.
|
||||
|
||||
`memoryScope: 'dependencies'` and `dependencyPayload` mean OMA itself decides what a
|
||||
task sees of its predecessors — we were assembling that by hand.
|
||||
|
||||
### Verification per task — `ConsensusVerifyOptions` (`types.d.ts:1883-1898`)
|
||||
|
||||
`judges: AgentConfig[]`, `mode: 'refute' | 'lens'`, `quorum` (default ceil(n/2)),
|
||||
`maxRounds` (default 2), `verdictSchema` (zod; a schema failure counts as dissent),
|
||||
`onDissent: 'revise' | 'reject' | 'keep'`, `judgePrompt`.
|
||||
Result: `verdict: 'accepted' | 'rejected'`, plus `dissent[]` and `rounds`.
|
||||
|
||||
That is a review gate already built into the runtime, per task. Our hand-written
|
||||
`gate`/`ocr` path overlaps it and is not aware of it.
|
||||
|
||||
### Approvals — durable, scoped, exact (`types.d.ts:318-390`, `approval/durable.d.ts`)
|
||||
|
||||
- `ApprovalScope = 'plan' | 'task_round' | 'task_dispatch' | 'tool_call'`.
|
||||
- `ApprovalRequest` is immutable and **content-bound**: `id`, `runId`, `scope`,
|
||||
`boundary`, `requestHash` (SHA-256 of scope+boundary+content), `content`.
|
||||
- `PlanApprovalContent` carries the tasks (`TaskSnapshot[]`) and
|
||||
`continuation: 'execute' | 'plan_only'` — the same plan can either resume
|
||||
execution or be returned for a person to read.
|
||||
- `ApprovalDecisionRecord`: `decision: 'approved' | 'rejected'`, `reviewer{id,displayName}`,
|
||||
`decidedAt`, bound to the request hash.
|
||||
- `DurableApprovalLedger` stores rows in a `MemoryStore` under
|
||||
`__oma_approval__/`, with atomic decision writes and stable error codes
|
||||
(`APPROVAL_STALE_DECISION`, `APPROVAL_CONFLICT`, `APPROVAL_INTEGRITY_ERROR`).
|
||||
|
||||
### The gates the runtime itself calls (`types.d.ts:2194-2246`)
|
||||
|
||||
- **`onPlanReady(tasks)`** — once, after the coordinator decomposes the goal, before
|
||||
anything runs. Return `true/allow` to proceed, `false/deny` to abort, or
|
||||
**`suspend`** to persist the exact plan for a later decision.
|
||||
- **`onTaskDispatch(task)`** — immediately before a ready task is dispatched.
|
||||
`deny` stops new dispatches; running tasks settle; the rest are marked `skipped`.
|
||||
- **`onApproval(completed, next)`** — between rounds (legacy executor; mutually
|
||||
exclusive with `onTaskDispatch`).
|
||||
- `requireConsequentialConfirmation` + `onToolCall` — gate consequential tool calls.
|
||||
- `PlanPatch` (`types.d.ts:1397`) — append-only repair of the not-yet-executed part
|
||||
of the graph: `addTasks`, `retargetPending`, `supersedePending`. The executed
|
||||
topology is immutable.
|
||||
|
||||
### Trace events (`types.d.ts:2017`)
|
||||
|
||||
`agent_start | agent_complete | task_start | task_complete | task_skipped | task_retry |
|
||||
approval_pending | plan_revision | recovery_decision | budget_exceeded | message |
|
||||
warning | error`, in a `FileTraceStore`, exported per run by `oma dashboard`.
|
||||
|
||||
### What this changes
|
||||
|
||||
1. **The approval we hand-built is a weaker copy of `onPlanReady` + `suspend` +
|
||||
`DurableApprovalLedger`.** OMA's version is content-bound (a changed plan cannot
|
||||
be approved by a stale decision), scoped, and recorded with a reviewer identity.
|
||||
2. **Per-task checks belong in `verify`, not in a side script.** A task can carry its
|
||||
own judges, quorum and dissent policy, and the result is
|
||||
`accepted | rejected` with the dissent text.
|
||||
3. **Dependencies and what a task sees of them are OMA's job** (`dependsOn`,
|
||||
`memoryScope`, `dependencyPayload`), not ours to re-derive — which is why the
|
||||
re-plan lost them and nothing noticed.
|
||||
4. **`oma task --file tasks.json` is the supported way to run a plan you already
|
||||
have**, which is what `bin/run-start` tried to be.
|
||||
5. **Still ours to own:** the pages, the pane layout, the notes, the project files,
|
||||
the wording of the checks, and the decision surface — plus the graph view, where
|
||||
we add per-task state and the verdict on top of OMA's data rather than drawing a
|
||||
second graph from a different source.
|
||||
|
||||
### Questions the research did not settle
|
||||
|
||||
- Which gates are available when the CLI is used, versus only through the library
|
||||
(`onPlanReady` is documented as `runTeam()` only).
|
||||
- How a `suspend` decision is later resumed from the CLI, as opposed to in-process.
|
||||
- Whether `verify` judges can run shell commands (our checks are commands), or only
|
||||
reason over the task output.
|
||||
|
||||
## Upstream documentation (researched 2026-11-09)
|
||||
|
||||
- npm `@open-multi-agent/core@1.23.0`, published 2026-10-29. MIT. Author: Shenzhen
|
||||
YuanASI Technology. One npm maintainer. ~3,100 downloads in the last month.
|
||||
- Repository `github.com/open-multi-agent/open-multi-agent`, subdirectory
|
||||
`packages/core`, tag `v1.23.0`.
|
||||
- **There is real documentation**, not just a README: 40+ guides under `docs/`, an
|
||||
examples tree (`basics/`, `cookbook/`, `patterns/`, `providers/`, `integrations/`),
|
||||
and a docs site at open-multi-agent.com.
|
||||
- The pages that matter to us:
|
||||
`docs/cli.md` (team/tasks files), `docs/coordinator.md`,
|
||||
`docs/durable-approvals.md`, `docs/evaluation-ci.md`, `docs/observability.md`,
|
||||
`docs/run-viewer.md`, `docs/external-agents.md`, `docs/glossary.md`.
|
||||
|
||||
### Planning, from the docs
|
||||
|
||||
`runTeam(team, goal)` is the only planning entry point. One coordinator model call
|
||||
produces a JSON array of task specs, validated against a Zod schema generated from
|
||||
the live roster, with one corrective retry. Caps: 200 tasks, 200 dependencies. An
|
||||
invalid plan **fails the run** (`COORDINATOR_PLAN_INVALID`, `INVALID_ASSIGNEE`,
|
||||
`INVALID_TASK_DEPENDENCIES`) and is never downgraded to a single agent. Ordering is
|
||||
then OMA's (`TaskQueue` + `Scheduler`); the coordinator does not re-engage mid-run.
|
||||
|
||||
A deterministic router short-circuits simple goals to a single agent — that is why
|
||||
`mode: 'team'` is required for the split to happen at all.
|
||||
|
||||
### Approvals
|
||||
|
||||
Four boundaries: `onPlanReady`, `onApproval`, `onTaskDispatch`, `onToolCall`; each may
|
||||
return `allow`, `deny` or `suspend`. Suspension needs checkpointing and a `MemoryStore`
|
||||
with atomic `compareAndSet`. Decisions are first-wins. The docs are explicit that the
|
||||
hash is **not** a cryptographic signature against storage administration.
|
||||
|
||||
### External agents
|
||||
|
||||
`process` (command; stdin/argv in, stdout/exit code out) and `acp` (JSON-RPC over
|
||||
stdio; Gemini CLI, Claude Code, Codex). Important limit from the docs: the tool gate,
|
||||
filesystem sandbox and LLM egress policy **do not apply** to external agents.
|
||||
|
||||
### Verifiable records — with honest limits
|
||||
|
||||
Three separate things: an execution receipt (pure function, no I/O), an opt-in run
|
||||
journal (`verifyRun()` checks sequence monotonicity and per-block hashes), and the
|
||||
trace store plus static HTML run viewer. The docs state plainly: no hash chain, no
|
||||
signature, no WORM storage — a writer with file access can rewrite an event and
|
||||
recompute its hash.
|
||||
|
||||
### Coding use
|
||||
|
||||
Generic orchestration, with coding documented as a use case ("hybrid team": LLM
|
||||
planner → external coding CLI → LLM reviewer over the diff). Pattern examples include
|
||||
`multi-perspective-code-review`, `risk-gated-bash`, `external-agent-acp`.
|
||||
|
||||
### Gaps this research settled
|
||||
|
||||
- `onApproval` and `onTaskDispatch` are **library-only**: function options cannot be
|
||||
passed through the CLI. So a CLI-driven run cannot use our gates; the coordinator
|
||||
must be our own small Node process calling the library.
|
||||
- Our earlier note about `oma eval gate --repo` is not in the upstream docs; only
|
||||
`oma eval run --gate <gate.json>`.
|
||||
Reference in New Issue
Block a user