From f063fbf2bcf2bc3e7c605451d54087d0e15290d4 Mon Sep 17 00:00:00 2001 From: Dotta Date: Mon, 28 Sep 2026 15:14:02 -0500 Subject: [PATCH] Map rich ACP surfaces to evidence and update paid qualification results Co-Authored-By: Paperclip --- .../runner-rich-acp-capabilities.md | 41 ++++++++++++++++++- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/doc/architecture/runner-rich-acp-capabilities.md b/doc/architecture/runner-rich-acp-capabilities.md index 253086f07de..d573907c9a0 100644 --- a/doc/architecture/runner-rich-acp-capabilities.md +++ b/doc/architecture/runner-rich-acp-capabilities.md @@ -1,7 +1,7 @@ # Rich ACP integration and qualification report Updated: 2026-09-28. Base: `c65fc9e3c81c41aafe421aa90a00514b84343285`. -Mainline integration: `992f7202628543749236b5a59af4dbbe7c155cce`. +Mainline integration: `18e8c121d99fee1d4038610531ab57ae3f824fdd`. Status: implementation is available as four draft PRs; deterministic review and CI verification are in progress. All three new profiles remain **pending qualification**. Authenticated paid qualification is in @@ -47,6 +47,11 @@ mean the required paid local and Daytona product cases have passed. “Not expos means the pinned interface was inspected; “unverified” is a separate finding. Codex's row is the existing app-server integration, not the Codex ACP bridge. +The provider reports above are the method/event and field inventories. In the +table below, a method that is absent from a pinned implementation is distinct +from an exposed method with no Paperclip control. Shared surfaces and their +deterministic evidence are mapped separately after the comparison. + | Capability | Codex app-server benchmark | Cursor ACP candidate | Copilot ACP candidate | Pi ACP candidate | | --- | --- | --- | --- | --- | | Exact model | Selected and reported model | Explicit ID required; exact echo, paid semantic protocol, local completion and file validation passed | Explicit ID required; exact `gpt-5.6-luna` echo, paid semantic protocol and local completion passed | Exact `openrouter/deepseek/deepseek-v4-flash-0731`; real paid response and four semantic tools observed; terminal settlement still unverified | @@ -66,6 +71,21 @@ Codex's row is the existing app-server integration, not the Codex ACP bridge. | Config/model changes | Typed configurable controls | Known modes/model interfaces researched; runtime policy cannot be changed by a display event | Model/reasoning/mode options exist; runtime policy remains authoritative | Exact qualified model; arbitrary slash commands/config/extensions disabled | | Reconnect/restart | Durable controller replay and qualified provider restoration | Live-process pending input only until exact restoration is proved | Same; history alone is not approval restoration | Same; wrapper explicitly advertises live-process-only pending-input recovery | +| Shared capability | User-visible surface | Inspectable implementation / deterministic evidence | +| --- | --- | --- | +| Text, tools and child activity | Task transcript; bounded activity details preserve provider/session/tool attribution | `src/drivers/acpx/codex-runtime-adapter.test.ts`, `runner/crates/runner-core/tests/acpx_rich_events.rs`, `ui/src/components/task-chat/TaskChatProtocolActivityRow.test.tsx` | +| Native questions and permissions | Existing question/confirmation cards; only offered decisions can be submitted | `src/protocol/permission-request.test.ts`, `src/drivers/acpx/acp-permission-adapter.test.ts`, `server/src/services/native-runtime/native-question-bridge.test.ts`, `ui/src/components/task-chat/TaskChatProtocolCard.test.tsx` | +| Native plan decisions | Full plan description followed by a revision-bound decision; no implicit plan acceptance | Cursor provider fixtures; `ui/storybook/fixtures/evidence/rich-acp-browser-proof.darwin-arm64.json` records the real renderer with a canonical fixture | +| Active steering and queued follow-up | Existing active-turn control where a bound method is negotiated; no native Pi queue selector yet | `src/drivers/acpx/turn-controls.test.ts`; provider reports state which wire method is absent or not surfaced | +| Files, diffs and images | Workspace file/artifact cards plus contained provider reference notices; raw provider diffs are still partial | `src/drivers/acpx/profile-extensions.test.ts`; provider field audits; Product `file-edit-validate` uses an independent exact-byte oracle | +| Usage and model identity | Exact configured model, per-provider token/accounting fields, explicit incomplete cost coverage | `src/drivers/acpx/usage-accounting.test.ts`, `src/cli/eval-session-contract.test.ts`, `server/src/services/native-runtime/native-session-executor.test.ts` | +| Durable input, reconnect and provider death | Pending interaction cards survive controller recovery; unsafe replacement expires unresolved requests | `src/control-plane/durable-prp-control-plane.test.ts`, `runner/crates/runner-core/tests/acpx_provider_resolutions.rs`, `src/live/runnerd-codex-transport.test.ts` | +| Session list/fork, generic configuration and commands | No added operator surface; exact owned session recovery and configured model remain available | Provider inventories identify native-only, ACP-exposed, confirmed-absent and unverified methods with follow-ups | + +Paths starting with `src/` or `runner/` in this evidence table are relative to +`packages/paperclip-runner/`; other paths are repository-relative. Deterministic +fixtures establish contract behavior, not successful paid provider execution. + ## Shared event and interaction contract The active connection, wire session, normalized turn and original request identity @@ -201,12 +221,19 @@ Separate paid Product E2E evidence now records: | Cursor / completion | `edf538e61e712dddb6b4d59045c3dcfd445686c7` | 6/6 assertions; committed finalization, one completion marker, cleanup passed | | Cursor / file edit and validation | `fe132224c2b30a8d9ce7b46cea38b8760af233fc` | 7/7 assertions; independent final file bytes, visible downloadable workspace artifact, cleanup passed | | Copilot / completion | `bcc9c638a25b91b84065f12633f083bd4f7a689f` | 6/6 assertions and cleanup passed; original accounting projection failed independently | +| Cursor / question continuation | `a7e01a0cec397dd5048f5d5b5825658dc6e91450` | UI answer Cobalt, continuation and 6/6 terminal assertions passed; two expected provider runs, no retry | +| Cursor / semantic plan approval | `a7e01a0cec397dd5048f5d5b5825658dc6e91450` | Displayed plan revision matched the confirmation target; accept/continue and 6/6 terminal assertions passed | +| Copilot / file edit and validation | `ee9536001fbe733b2386dd3379730a4e0be59488` | 7/7 assertions and cleanup passed; independent bytes validated; GitHub biller and unpriced receipt verified | +| Copilot / question continuation | `ee9536001fbe733b2386dd3379730a4e0be59488` | Question/answer and warm session reuse worked; 4/6 terminal assertions passed because the provider returned a literal placeholder instead of the required marker; failed attempt retained | The Cursor file case proves the workspace artifact surface, not complete native file/diff projection. The Copilot result incorrectly projected missing native cost as USD zero and attributed its biller to OpenAI. The original result is retained; the shared fix identifies GitHub, Cursor and OpenRouter correctly and keeps absent -candidate USD receipts unpriced. Fresh paid evidence is required to verify that fix. +candidate USD receipts unpriced. The separate Copilot file case verifies that fix. +The question case exposed a second ledger edge: zero normalized token counters +were treated as a reported cost despite no cost field. Ledger classification now +requires an explicit finite nonnegative cost; an explicit zero remains reported. Protocol evals now fail their cost gate when spend is unknown, preserving completed behavior and semantic evidence in a separate accounting-failure result. The maintained campaign stops subsequent cells on unknown accounting and never turns @@ -221,6 +248,16 @@ qualification authority. The obsolete unconditional Pi executor rejection is replaced by the same closed host authorization. Candidate active turns are bounded to 120 seconds and automatic infrastructure retries remain disabled. +Pi's next local startup attempt timed out before any prompt, with zero exclusive +key usage delta. A credential-free reproduction isolated a 37.423-second immutable +copy of its 13,827-file, 234,019,683-byte distribution, after 5.168 seconds of +verification, against the 30-second session-open deadline. Bounded parallel copying +reduced that same copy to 5.851 seconds without changing a timeout. Ten tests cover +the eight-file / 32 MiB batch bounds, unchanged per-file integrity checks, stable +digest order, mutation rejection, and draining pending copies before cleanup. +This is startup preparation evidence; a new paid Product attempt must still prove +successful inference and settlement. + The maintained Product E2E `extended-harnesses` suite covers local and Daytona completion, question/answer, semantic plan approval, pending-input restart and file edit/validation. It has no automatic retries and does not enable candidates