mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 12:59:13 +01:00
Bind candidate diagnostics and recovery to explicit runner policy
Preserve Pi pricing receipt provenance, require evaluation opt-in, and record qualification boundaries. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
84336a1b7d
commit
588dd00af1
18 files changed
+302
-45
No files matched your search
@@ -0,0 +1,150 @@
|
||||
# Rich ACP integration and qualification report
|
||||
|
||||
Updated: 2026-09-28. Base: `c65fc9e3c81c41aafe421aa90a00514b84343285`.
|
||||
Status: implementation and deterministic verification in progress; all three new
|
||||
profiles remain **pending qualification**. No paid inference or Daytona resources
|
||||
have been used. This report does not certify a provider from its ACP listing.
|
||||
|
||||
The [harness priorities report](https://pages.paperclip.ing/2026-09-25-harness-priorities/)
|
||||
recommends Cursor and Copilot, followed by Pi, using the existing qualified ACPX
|
||||
path. Codex app-server is the richness benchmark. The legacy Cursor and Pi
|
||||
adapters are outside this change.
|
||||
|
||||
## Branches and evidence ownership
|
||||
|
||||
| Branch | Deliverable |
|
||||
| --- | --- |
|
||||
| `codex/runner-rich-acp` | Shared ACPX extension boundary, durable permissions, canonical display events, provider pack infrastructure, configuration and UI |
|
||||
| `codex/runner-cursor-acp` | Cursor native distribution, questions/plans, child activity, policy admission, wire fixtures |
|
||||
| `codex/runner-copilot-acp` | Copilot native distribution, event inventory/projections, permission and settlement probes |
|
||||
| `codex/runner-pi-acp` | Patched wrapper, owned extension, MCP/tools, permissions/input, portable dependency closure |
|
||||
|
||||
Each provider branch depends on the foundation. They remain separate worktrees
|
||||
and review units. Source reports on the provider branches are
|
||||
`doc/architecture/runner-cursor-capabilities.md`,
|
||||
`doc/architecture/runner-copilot-capabilities.md`, and
|
||||
`doc/architecture/runner-pi-capabilities.md`. Those reports retain versioned
|
||||
source references, per-field dispositions, fixture paths, and narrower claims.
|
||||
The Copilot inventory enumerates all 150 pinned native event types.
|
||||
|
||||
## Capability matrix
|
||||
|
||||
“Candidate” means implemented or observed in deterministic tests. It does not
|
||||
mean the required paid local and Daytona product cases have passed. “Not exposed”
|
||||
means the pinned interface was inspected; “unverified” is a separate finding.
|
||||
Codex's row is the existing app-server integration, not the Codex ACP bridge.
|
||||
|
||||
| Capability | Codex app-server benchmark | Cursor ACP candidate | Copilot ACP candidate | Pi ACP candidate |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Exact model | Selected and reported model | Explicit ID required; authenticated discovery unverified | Explicit ID required; offline ACP status omits model metadata; admission fails if exact model cannot be proved | Exact `openrouter/deepseek/deepseek-v4-flash-0731`; paid model response unverified |
|
||||
| Text and tools | Typed thread/turn/item events | Standard ACP updates; child activity kept separate | Standard ACP plus opt-in native session events | Wrapper text/tool updates and owned tool gates |
|
||||
| Active steering | Dedicated `turn/steer` | Concurrent prompt replaces/cancels, so it is not steering | Concurrent prompt replaces/cancels, so it is not steering | Owned `pi/steer` requires handshake, exact active turn and acknowledgment |
|
||||
| Queued follow-up | Product continuation controls | Controller can schedule a later prompt; native queue not established | Native pending-message activity exists; no qualified ACP queue responder | Owned `pi/follow_up`, separately named and ordered |
|
||||
| Cancellation | Typed interrupt and process lifecycle | ACP cancel; paid command cleanup pending | ACP cancel; paid background settlement pending | Native abort; wrapper waits for `agent_settled` and treats provider errors as failure |
|
||||
| Session continuity | Read/load/history/fork and durable identity | Session load/list observed; authenticated replay pending; fork absent in tested methods | Session load plus native history events; live recovery pending | Private Pi JSONL mapping/load; unresolved UI promises cannot survive provider death |
|
||||
| Questions | Typed input requests and response correlation | `cursor/ask_question`, option identity and multiple selection preserved | Native ask-user capability exists, but pinned ACP does not wire its responder; do not display a false answerable form | `select`, `confirm`, `input`, `editor` through typed form elicitation |
|
||||
| Permissions | Durable typed approvals | Standard ACP permission options; only supported decisions shown | Standard ACP; offline deny-before-marker proof; session decision scope inspected | Native pre-tool gate; allow once, exact-operation session grant, deny; paths rechecked after wait |
|
||||
| Plans | Typed plan and collaboration mode | `cursor/create_plan` includes full plan and revision-bound accept/reject/cancel; todo activity separate | Native plan events displayed; native plan-decision callback not exposed in ACP | No native structured plan event; authenticated Paperclip planning tools available |
|
||||
| Authenticated tools | Runner bridge and governed operations | ACP HTTP MCP binding; paid semantic cases pending | ACP HTTP MCP binding; paid semantic cases pending | Owned extension registers exact bound MCP tools; no ambient servers |
|
||||
| Delegation | Typed agent roles and lifecycle | Opt-in subagent lifecycle, nested ownership and bounded child activity; never parent transcript flattening | Native delegation/session events projected with role/model/agent provenance | No built-in ACP delegation protocol; arbitrary extensions are excluded |
|
||||
| Files/diffs | Typed file changes and artifact references | Standard tool changes plus validated image references | File/workspace events retained; provider session files are not silently treated as task files | Native read/write/edit diffs; semantic artifact tools |
|
||||
| Images/artifacts | Typed references and registered work products | Existing contained files only, provenance, `registered:false` | Contained task references only; external/session-store paths become descriptive notices | Image/resource tool blocks preserved by wrapper; dedicated artifact channel absent |
|
||||
| Usage | Per-request receipt and model context | Native usage completeness unverified | ACP/native usage retained without inventing a billing charge | Assistant-message token receipts; dollar cost is a catalog pricing estimate, never authoritative billing |
|
||||
| Config/model changes | Typed configurable controls | Known modes/model interfaces researched; runtime policy cannot be changed by a display event | Model/reasoning/mode options exist; runtime policy remains authoritative | Exact qualified model; arbitrary slash commands/config/extensions disabled |
|
||||
| Reconnect/restart | Durable controller replay and qualified provider restoration | Live-process pending input only until exact restoration is proved | Same; history alone is not approval restoration | Same; wrapper explicitly advertises live-process-only pending-input recovery |
|
||||
|
||||
## Shared event and interaction contract
|
||||
|
||||
The active connection, wire session, normalized turn and original request identity
|
||||
bind every extension callback. The allowlist is per provider. An omitted wire
|
||||
session can only acquire the verified active connection's session; an explicit
|
||||
mismatch is rejected. Retired streams and inactive turns cannot emit new activity.
|
||||
|
||||
Requests enter durable runtime state before the UI presents them. The response
|
||||
must match an outstanding request and an offered action or valid typed answer.
|
||||
The sidecar validates and accepts delivery before Rust settles its durable record.
|
||||
Standard ACP does not acknowledge application of a permission reply; a lost
|
||||
transport acknowledgment is not proof of exactly-once external effects. A
|
||||
replacement provider process cannot inherit an old approval promise. No tool
|
||||
mutation or approval is automatically replayed into a replacement.
|
||||
|
||||
Full plan documents have a bounded 100,000-character description and a 196 KiB
|
||||
question-set envelope. Oversized plans fail rather than approve an unseen suffix.
|
||||
Display redaction remains visible. The rich event channel has exact canonical
|
||||
schemas and a bounded envelope. It cannot create terminal outcomes, dispatch a
|
||||
semantic tool, register an artifact, synchronize a durable plan, or supply source
|
||||
authority. Notices retain useful bounded fields and provenance in expandable UI
|
||||
details. Provider references remain unregistered until a control-plane operation
|
||||
registers them.
|
||||
|
||||
Permission labels are derived from offered option kinds. An unknown or duplicate
|
||||
option is rejected. A provider's “always” decision is not relabeled “this session”
|
||||
unless the pinned implementation proves that scope. Restrictive execution policy
|
||||
is separate from automatic approval and company governance. Fresh permission
|
||||
configuration defaults to full auto; it never overrides read-only task policy.
|
||||
|
||||
## Distribution and isolation
|
||||
|
||||
Cursor pins `2026.09.26-dd393fe`; Copilot pins `1.0.88`; Pi pins wrapper `0.0.33`,
|
||||
runtime `0.84.2`, portable Node and its full npm lock. Native distribution hashes
|
||||
cover macOS ARM64, macOS x64 and Linux x64. Source-owned closure pins remain
|
||||
separate from profile declaration digests. Native admission reads held files,
|
||||
verifies every admitted byte, creates a private immutable snapshot and retains
|
||||
the existing process guardian. A manifest cannot supply its own trusted pin.
|
||||
|
||||
Candidate credentials are only read from explicitly bound run environments:
|
||||
Cursor `CURSOR_API_KEY`/`CURSOR_AUTH_TOKEN`, Copilot `COPILOT_GITHUB_TOKEN`, Pi
|
||||
`OPENROUTER_API_KEY`. Ambient GitHub login variables, provider configuration,
|
||||
extensions and MCP discovery do not establish authority. Homes/config/cache are
|
||||
private. Updates are disabled. Pi launches only its owned extension and assigned
|
||||
skills, with a startup sentinel before a prompt can run.
|
||||
|
||||
Build candidate packs explicitly with `--candidate-providers=<name>` on that
|
||||
provider's branch. Local and remote pack verification includes the complete
|
||||
candidate asset tree. The corresponding Daytona build argument is documented in
|
||||
`docker/daytona-runner/README.md`. Candidate packaging never promotes a profile.
|
||||
|
||||
ACP is not an OS sandbox. Cursor still reads project MCP/hooks beyond its
|
||||
`--disable-project-configs` flag; the adapter rejects known ambient execution
|
||||
configuration. Mutations during a run and remote team hooks still require
|
||||
qualification. Pi's tool policy supplements the execution boundary; arbitrary
|
||||
shell commands and filesystem races require the host boundary. These are
|
||||
qualification gates, not claims that a JavaScript path check confines a shell.
|
||||
|
||||
## Explicit gaps and follow-ups
|
||||
|
||||
| Priority | Exposed but unused, partial, or unverified | Reason and next proof |
|
||||
| --- | --- | --- |
|
||||
| P0 | All paid product cases on local and Daytona | Bound provider/Daytona credentials and independently inspectable spend are absent. Prove setup, semantic tools, edit+validation, inputs, cancellation, warm continuation and restart. |
|
||||
| P0 | Copilot native ask-user and plan-decision callbacks | Pinned ACP does not install native responders. Prove no blocking request is exposed, or add a qualified responder/wrapper; never swallow the request. |
|
||||
| P0 | Copilot denial and detached background settlement | Offline deny/attached-shell tests pass only their narrow fixtures. Exercise affected native tools and detached commands against real service; failing release stays unqualified. |
|
||||
| P0 | Exact Cursor/Copilot model reporting | Authentication/entitlement required; no guessed default and no fallback. Fail admission when effective model is unverified. |
|
||||
| P0 | Cursor project and remote hooks; native shell boundaries | Configuration flags do not cover every native source. Demonstrate policy cannot be bypassed before production qualification. |
|
||||
| P1 | Native Cursor/Copilot active steering and queues | ACP prompt replacement is not steering; native SDK capabilities may be richer. Require a dedicated bound method plus acknowledgment before advertising. |
|
||||
| P1 | Child tool media/diff/raw payloads | Bounded delegation summaries preserve lifecycle and identity. Large nested payloads need a child-owned canonical item model; current omission is a visible notice and provider report entry. |
|
||||
| P1 | Copilot session-store files and export/artifact URIs | Provider paths are not task-workspace paths. Add a separately authorized export flow with validated bytes and provenance; do not resolve arbitrary URLs or auto-register. |
|
||||
| P1 | Complete usage/billing provenance | Missing cache fields remain unknown. Pi price estimates are displayed separately. Budget qualification requires actual spend coverage, not an estimate presented as a bill. |
|
||||
| P1 | Fork/history/model/mode controls not exposed by Paperclip | Research documents the native and ACP methods separately. Add governance-aware controls and durable lineage before enabling them. |
|
||||
| P1 | Exact pending-request restoration after process death | Session transcript restoration does not restore callbacks. Expire unresolved requests unless a provider proves exact restoration. |
|
||||
| P2 | Remaining Copilot native diagnostic/config/account events | The provider inventory records every event and field, its projection or reason for omission. Preserve bounded useful context; avoid credentials, raw environment or unbounded blobs. |
|
||||
| P2 | Pi native extension surfaces and unsupported slash commands | Arbitrary extensions/templates/themes may execute ambient code. Only reviewed runner-owned capabilities are admitted; native plan/fork/goals are not fabricated. |
|
||||
|
||||
## Qualification ledger
|
||||
|
||||
Combined ceiling: **$100**, including retries and infrastructure. Cursor allocation:
|
||||
$25; Copilot: $25; Pi: $25; coordinated diagnosis reserve: $25. Used: **$0**.
|
||||
No model requests or Daytona leases have been started. Registry/binary downloads,
|
||||
loopback fake-model tests and uncredentialed initialization are deterministic
|
||||
engineering evidence, not paid qualification.
|
||||
|
||||
Before each paid batch, record source SHA, executable and closure/profile digests,
|
||||
exact model, OS/architecture or Daytona image, selected cases, prior spend,
|
||||
maximum batch spend and authoritative billing coverage. Stop before the shared
|
||||
ceiling. Missing spend coverage blocks a run rather than treating unknown cost as
|
||||
zero. Retain screenshots and wire evidence without credentials. Never convert a
|
||||
provider to supported solely because a test suite or packaging check passed.
|
||||
|
||||
Verification commands and final results are recorded with the prerequisite and
|
||||
provider PRs. The full handoff requires runner checks, token gates, recursive
|
||||
typecheck, `pnpm test:run`, and `pnpm build`. Until that evidence is recorded, this
|
||||
report is an implementation report rather than a PR-ready certification.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Cursor, Copilot and Pi through rich ACP
|
||||
|
||||
Approved implementation plan, 2026-09-28. Shared base:
|
||||
`c65fc9e3c81c41aafe421aa90a00514b84343285`.
|
||||
|
||||
Keep the existing ACPX runner transport and Codex app-server richness benchmark.
|
||||
Implement a foundation prerequisite, then three separate provider branches and
|
||||
PRs. Do not merge automatically. Preserve legacy Cursor/Pi adapters.
|
||||
|
||||
1. Investigate each pinned harness, its native API, ACP adapter, observed wire,
|
||||
runner normalization and UI surface. Inventory every meaningful event field.
|
||||
2. Extend shared profile declarations, durable permissions/input, extension
|
||||
dispatch, steering/queue distinctions, artifact provenance and bounded display
|
||||
activity. Preserve existing replay and exact warm-session identities.
|
||||
3. Pin complete provider distributions for macOS ARM64/x64 and Linux x64. Use
|
||||
verified launches, isolated credentials/configuration and build-time assets.
|
||||
4. Implement Cursor native ACP and its question/plan/subagent extensions;
|
||||
Copilot ACP authentication/events/permission and settlement handling; Pi
|
||||
wrapper patch and owned extension with authenticated tools and input policy.
|
||||
5. Run deterministic, recovery, isolation and richness conformance. Qualify each
|
||||
provider locally and in Daytona with product E2E evidence before advertising
|
||||
support. Use a combined $100 ceiling: $25 each plus $25 diagnosis reserve.
|
||||
6. Run required full checks, prepare one foundation and three dependent PRs,
|
||||
retain evidence and deliver the comparative capability report. List every
|
||||
unused/partial capability with reason and prioritized follow-up.
|
||||
|
||||
Implementation and qualification status is maintained in
|
||||
[the capability report](../architecture/runner-rich-acp-capabilities.md).
|
||||
Missing credentials or spend verification leave qualification pending; they do
|
||||
not authorize a weaker supported profile or an implicit model fallback.
|
||||
@@ -54,7 +54,11 @@ leave the board showing an active run after the provider turn has already ended.
|
||||
The package also builds `paperclip-runner-acpx-sidecar`. This bounded v2
|
||||
stdin/stdout bridge admits the pinned Claude and Codex ACPX profiles. It
|
||||
validates the exact model, session identity, tool catalog, structured input,
|
||||
and terminal settlement at the process boundary. Pi remains unavailable.
|
||||
and terminal settlement at the process boundary. Cursor, Copilot and Pi have
|
||||
separate candidate branches and remain unavailable in production until local
|
||||
and Daytona qualification passes. Their verified distributions are build-owned;
|
||||
no candidate accepts an arbitrary executable. See
|
||||
[the rich ACP capability report](../../doc/architecture/runner-rich-acp-capabilities.md).
|
||||
|
||||
Native Claude skill assignments travel in the runtime-context snapshot through
|
||||
runnerd to the ACPX sidecar. After acquiring the provider lifetime lease, the
|
||||
@@ -83,12 +87,14 @@ do not change workspace isolation or grant credentials or connection access.
|
||||
task tools; `approve-reads` allows assigned reads; `deny-all` rejects requests.
|
||||
None of these restrictive modes is the default.
|
||||
|
||||
This runtime has no interactive permission handler. An operation that still
|
||||
requires approval stops the turn with `approval_required`. The server marks the
|
||||
task blocked, exposes the permission action to the operator, and disables
|
||||
automatic retry. The operator must review the operation and the agent's
|
||||
permission setting before retrying. Company access checks still run when each
|
||||
Paperclip tool executes.
|
||||
Restrictive profiles route supported permission decisions through durable runtime
|
||||
requests and the existing task interaction controls. Requests are persisted
|
||||
before presentation; answers are checked against the offered decisions and
|
||||
acknowledged by the sidecar before settlement. Unknown, stale and duplicate
|
||||
responses fail. Missing provider decision support remains a blocked disposition,
|
||||
not implicit approval. Company access checks still run for each Paperclip tool.
|
||||
Provider death expires pending promises; approvals are never replayed into a
|
||||
replacement process.
|
||||
|
||||
Runnerd selects only qualified provider profiles. Claude Managed and AWS
|
||||
AgentCore receive immutable company-profile snapshots with explicit retention,
|
||||
@@ -265,7 +271,11 @@ pnpm --filter @paperclipai/paperclip-runner report:runner-chaos-evals
|
||||
`report:runner-live-evals` is a paid, provider-backed command. Native Codex
|
||||
requires `OPENAI_API_KEY`; ACPX Claude requires
|
||||
`ANTHROPIC_API_KEY`; OpenCode candidates require `OPENROUTER_API_KEY`. The live
|
||||
matrix admits no Pi profile and does not persist credential values. Set
|
||||
matrix remains qualified-only and does not persist credential values. Candidate
|
||||
qualification uses `eval-session --candidate-profile <pi|cursor|copilot>` with an
|
||||
explicit model and a separately materialized pinned candidate pack. This option
|
||||
is a constructor-bound diagnostic opt-in; session JSON cannot enable a candidate.
|
||||
Missing credentials or unverifiable spend block paid qualification. Set
|
||||
`PAPERCLIP_EVAL_MAX_CAMPAIGN_COST_USD` to a positive finite number to bound
|
||||
additional scheduling after the observed campaign total reaches that value:
|
||||
|
||||
|
||||
@@ -74,7 +74,7 @@
|
||||
"typecheck:browser": "tsc -p tsconfig.browser.json --noEmit",
|
||||
"test": "pnpm run test:typescript && pnpm run test:rust",
|
||||
"test:typescript": "pnpm run test:typescript:prep && vitest run",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs",
|
||||
"test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs",
|
||||
"test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace",
|
||||
"test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider",
|
||||
|
||||
@@ -13,7 +13,7 @@ import {
|
||||
|
||||
export interface CodexAcpxNativeSessionBackendOptions extends Omit<
|
||||
CodexAcpxDriverOptions,
|
||||
"model" | "permissionMode" | "systemInstructions"
|
||||
"model" | "permissionMode" | "systemInstructions" | "providerPolicy"
|
||||
> {}
|
||||
|
||||
export type AcpxNativeSessionBackendOptions =
|
||||
@@ -30,11 +30,6 @@ export function createAcpxNativeSessionBackend(
|
||||
if (input.provider.kind !== "acpx") {
|
||||
throw new Error("ACPX backend requires provider kind acpx");
|
||||
}
|
||||
if (input.provider.agent === "pi") {
|
||||
throw new Error(
|
||||
"Pi ACPX backend is unavailable until descriptor-confined verified launch is implemented",
|
||||
);
|
||||
}
|
||||
const qualifiedProfile = resolveQualifiedAcpxProfile(
|
||||
input.provider.agent,
|
||||
input.provider.model,
|
||||
@@ -73,6 +68,7 @@ export function createAcpxNativeSessionBackend(
|
||||
model: input.provider.model,
|
||||
permissionMode: input.provider.permissionMode ?? "approve-reads",
|
||||
systemInstructions,
|
||||
providerPolicy: { readOnly: "executionMode" in input && input.executionMode === "plan" },
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
@@ -487,12 +487,9 @@ describe("native backend factory", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it("rejects Pi before constructing an ACPX backend", () => {
|
||||
expect(() =>
|
||||
createNativeSessionBackend(acpxExecution("pi"), {
|
||||
acpxRuntimeDirectory: "/runtime",
|
||||
}),
|
||||
).toThrow("descriptor-confined verified launch");
|
||||
it("constructs a Pi candidate descriptor while runtime installation remains separately gated", async () => {
|
||||
const backend = createNativeSessionBackend(acpxExecution("pi"), { acpxRuntimeDirectory: "/runtime" });
|
||||
await expect(backend.descriptor()).resolves.toMatchObject({ name: "acpx_runtime", version: "0.13.1" });
|
||||
});
|
||||
|
||||
it("rejects a Codex ACPX snapshot that drifts from its qualified profile", () => {
|
||||
|
||||
@@ -77,11 +77,6 @@ export function createNativeSessionBackend(
|
||||
});
|
||||
}
|
||||
if (input.provider.kind === "acpx") {
|
||||
if (input.provider.agent === "pi") {
|
||||
throw new Error(
|
||||
"Native ACPX backend for pi is unavailable until descriptor-confined verified launch is implemented",
|
||||
);
|
||||
}
|
||||
if (!options.acpxRuntimeDirectory?.trim()) {
|
||||
throw new Error("ACPX backend requires an instance runtime directory");
|
||||
}
|
||||
|
||||
@@ -11,3 +11,8 @@ export async function verifyAcpxProfileInstallation(profile: QualifiedAcpxProfil
|
||||
|
||||
/** Provider policy admission is repeated immediately before each process launch. */
|
||||
export async function assertAcpxProfileWorkspace(_agent: QualifiedAcpxAgent, _workspace: string): Promise<void> {}
|
||||
|
||||
/** Optional provider-specific classification; never changes whether admission succeeded. */
|
||||
export function classifyAcpxProfileError(_agent: QualifiedAcpxAgent, _error: unknown): Error | null {
|
||||
return null;
|
||||
}
|
||||
@@ -76,6 +76,21 @@ describe("ACPX recovery identity", () => {
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
it("rejects restoration across task execution-policy changes", async () => {
|
||||
const fixture = await recoveryFixture();
|
||||
const readonly = await createAcpxRecoveryBinding({ ...fixture.input, providerPolicy: { readOnly: true } });
|
||||
const writable = await createAcpxRecoveryBinding({ ...fixture.input, providerPolicy: { readOnly: false } });
|
||||
const record = createAcpxIdentityRecord(fixture.expected, readonly);
|
||||
expect(readonly.profileSessionKey).not.toBe(writable.profileSessionKey);
|
||||
expect(readonly.profileDigest).not.toBe(writable.profileDigest);
|
||||
expect(() => verifyExpectedAcpxIdentity(fixture.expected, writable, record)).toThrow("persisted runtime record");
|
||||
const changedRoots = await createAcpxRecoveryBinding({ ...fixture.input,
|
||||
providerPolicy: { readOnly: true, protectedPaths: ["/another-protected-root"] } });
|
||||
expect(changedRoots.profileSessionKey).not.toBe(readonly.profileSessionKey);
|
||||
await expect(createAcpxRecoveryBinding({ ...fixture.input,
|
||||
providerPolicy: { readOnly: "yes" as never } })).rejects.toThrow("valid task execution policy");
|
||||
});
|
||||
|
||||
it("uses collision-resistant roots and policy-bound provider keys", async () => {
|
||||
const fixture = await recoveryFixture();
|
||||
const otherSession = await createAcpxRecoveryBinding({
|
||||
|
||||
@@ -43,8 +43,10 @@ export async function createAcpxRecoveryBinding(input: {
|
||||
profile: QualifiedAcpxProfile;
|
||||
requestedModel: string;
|
||||
permissionMode: NativeAcpxPermissionMode;
|
||||
providerPolicy?: { readOnly: boolean; readRoots?: readonly string[]; protectedPaths?: readonly string[] };
|
||||
}): Promise<AcpxRecoveryBinding> {
|
||||
validateIdentity(input.normalizedSessionId, "normalized session");
|
||||
if (input.providerPolicy !== undefined && typeof input.providerPolicy.readOnly !== "boolean") throw new Error("ACPX recovery requires a valid task execution policy");
|
||||
if (input.requestedModel !== input.profile.qualificationModel) {
|
||||
throw new Error("ACPX recovery requested an unqualified model");
|
||||
}
|
||||
@@ -72,6 +74,11 @@ export async function createAcpxRecoveryBinding(input: {
|
||||
qualificationModel: input.profile.qualificationModel,
|
||||
reportedModelId: input.profile.reportedModelId,
|
||||
permissionPolicy: input.profile.permissionPolicy,
|
||||
...(input.providerPolicy === undefined ? {} : { executionPolicy: {
|
||||
readOnly: input.providerPolicy.readOnly,
|
||||
readRoots: input.providerPolicy.readRoots ?? [],
|
||||
protectedPaths: input.providerPolicy.protectedPaths ?? [],
|
||||
} }),
|
||||
}),
|
||||
);
|
||||
const profileSessionKey = digest(
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { dirname, join } from "node:path";
|
||||
import { assertAcpxProfileWorkspace, verifyAcpxProfileInstallation } from "./profile-installation.js";
|
||||
import { assertAcpxProfileWorkspace, classifyAcpxProfileError, verifyAcpxProfileInstallation } from "./profile-installation.js";
|
||||
import { createAcpxRuntimeSkillLease } from "./runtime-skill-lease.js";
|
||||
import { claudeNativeSkillPrompt } from "./native-skill-prompt.js";
|
||||
import { nativeMcpLaunchBinding } from "../native-mcp.js";
|
||||
@@ -328,6 +328,8 @@ export class AcpxRuntimeHost {
|
||||
profile,
|
||||
requestedModel: options.model,
|
||||
permissionMode: options.permissionMode,
|
||||
...(["cursor", "copilot", "pi"].includes(options.agent) && options.providerPolicy !== undefined
|
||||
? { providerPolicy: options.providerPolicy } : {}),
|
||||
}),
|
||||
dependencies.retainAdmissionCleanup,
|
||||
);
|
||||
@@ -605,6 +607,7 @@ export class AcpxRuntimeHost {
|
||||
: [],
|
||||
});
|
||||
} catch (error) {
|
||||
error = classifyAcpxProfileError(options.agent, error) ?? error;
|
||||
const cleanup = cleanupRuntimeResources(
|
||||
runtime,
|
||||
toolBridge,
|
||||
|
||||
@@ -55,9 +55,9 @@ describe("qualified ACPX usage", () => {
|
||||
expect(qualifiedAcpxUsageBreakdown("claude", null)).toBeNull();
|
||||
});
|
||||
|
||||
it("does not infer accounting semantics for Pi or an uninitialized agent", () => {
|
||||
it("does not double count reasoning for the exact pinned Pi model and preserves unknown profiles", () => {
|
||||
const usage = { inputTokens: 12, outputTokens: 30, thoughtTokens: 20 };
|
||||
expect(qualifiedAcpxUsageBreakdown("pi", usage)).toEqual(usage);
|
||||
expect(qualifiedAcpxUsageBreakdown("pi", usage)).toEqual({ ...usage, thoughtTokens: 0 });
|
||||
expect(qualifiedAcpxUsageBreakdown(null, usage)).toEqual(usage);
|
||||
});
|
||||
});
|
||||
@@ -157,4 +157,16 @@ describe("Pi prompt accounting authority", () => {
|
||||
expect(acpxUsageEstimateNotice(usage, "turn:pricing")).toBeNull();
|
||||
}
|
||||
});
|
||||
it("preserves compaction receipt provenance without treating estimates as billing", () => {
|
||||
const usage = persistedAcpxTurnUsage({}, { ...after, requestTokenUsage: {
|
||||
message: { input_tokens: 12, paperclip_pi: {
|
||||
provenance: "assistant_message_and_compaction_receipts", cost_usd: 0.25,
|
||||
} },
|
||||
} }, "turn", "pi")!;
|
||||
expect(usage.cost).toBeUndefined();
|
||||
expect(usage.usageProvenance).toBe("pi_assistant_message_and_compaction_receipts");
|
||||
expect(acpxUsageEstimateNotice(usage, "turn:pricing")?.payload).toMatchObject({
|
||||
details: expect.arrayContaining([{ name: "Usage source", value: "Assistant message and compaction receipts for this prompt" }]),
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -14,9 +14,10 @@ export function qualifiedAcpxUsageBreakdown(
|
||||
): unknown {
|
||||
if (value === null || value === undefined) return value;
|
||||
const breakdown = record(value);
|
||||
if (agent !== "claude" && agent !== "codex") return breakdown;
|
||||
if (agent !== "claude" && agent !== "codex" && agent !== "pi") return breakdown;
|
||||
// Claude SDK aggregate output and Codex ACP toPromptUsage.outputTokens both
|
||||
// INCLUDE reasoning. PRP folds thought into output, so its additive component
|
||||
// INCLUDE reasoning. The exact pinned Pi OpenRouter model uses
|
||||
// openai-completions, where completion_tokens also includes reasoning. PRP folds thought into output, so its additive component
|
||||
// is zero here, not the provider's diagnostic reasoning-token subset.
|
||||
return {
|
||||
...breakdown,
|
||||
@@ -50,7 +51,8 @@ export function persistedAcpxTurnUsage(
|
||||
if (added.length !== 1) return null;
|
||||
const usage = record(receipts[added[0]!]);
|
||||
const piReceipt = agent === "pi" ? record(usage.paperclip_pi) : {};
|
||||
const piReceiptVerified = piReceipt.provenance === "assistant_message_receipts";
|
||||
const piReceiptVerified = piReceipt.provenance === "assistant_message_receipts"
|
||||
|| piReceipt.provenance === "assistant_message_and_compaction_receipts";
|
||||
const estimate = piReceiptVerified && typeof piReceipt.cost_usd === "number"
|
||||
&& Number.isFinite(piReceipt.cost_usd) && piReceipt.cost_usd >= 0 ? piReceipt.cost_usd : undefined;
|
||||
return {
|
||||
@@ -60,7 +62,7 @@ export function persistedAcpxTurnUsage(
|
||||
// Pi calculates cost from catalog prices, not billing receipts. Never feed
|
||||
// this estimate into the authoritative/cumulative provider spend channel.
|
||||
cost: agent === "pi" ? undefined : current.usageCost,
|
||||
...(piReceiptVerified ? { usageProvenance: "pi_assistant_message_receipts" } : {}),
|
||||
...(piReceiptVerified ? { usageProvenance: `pi_${piReceipt.provenance}` } : {}),
|
||||
...(estimate === undefined ? {} : { pricingEstimateUsd: estimate }),
|
||||
breakdown: {
|
||||
inputTokens: usage.input_tokens,
|
||||
@@ -75,7 +77,8 @@ export function persistedAcpxTurnUsage(
|
||||
|
||||
/** Preserve the estimate for inspection while keeping billing authority separate. */
|
||||
export function acpxUsageEstimateNotice(usage: Record<string, unknown>, itemId: string): CanonicalProviderEvent | null {
|
||||
if (usage.usageProvenance !== "pi_assistant_message_receipts"
|
||||
if ((usage.usageProvenance !== "pi_assistant_message_receipts"
|
||||
&& usage.usageProvenance !== "pi_assistant_message_and_compaction_receipts")
|
||||
|| typeof usage.pricingEstimateUsd !== "number" || !Number.isFinite(usage.pricingEstimateUsd)
|
||||
|| usage.pricingEstimateUsd < 0) return null;
|
||||
return {
|
||||
@@ -86,7 +89,8 @@ export function acpxUsageEstimateNotice(usage: Record<string, unknown>, itemId:
|
||||
recoverable: true, userActionable: false,
|
||||
summary: `Pi estimates this turn at $${usage.pricingEstimateUsd.toFixed(6)} from its model prices. Billing cost is unverified.`,
|
||||
details: [{ name: "Cost source", value: "Pi model catalog pricing estimate" },
|
||||
{ name: "Usage source", value: "Assistant message receipts for this prompt" },
|
||||
{ name: "Usage source", value: usage.usageProvenance === "pi_assistant_message_and_compaction_receipts"
|
||||
? "Assistant message and compaction receipts for this prompt" : "Assistant message receipts for this prompt" },
|
||||
{ name: "Estimated USD", value: String(usage.pricingEstimateUsd) }],
|
||||
},
|
||||
};
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
import { realpathSync } from "node:fs";
|
||||
import { mkdtempSync, realpathSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll } from "vitest";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import {
|
||||
@@ -45,7 +48,10 @@ import {
|
||||
type CodexTraceInterpretation,
|
||||
} from "./app-server-transport.js";
|
||||
|
||||
export const WORKSPACE = realpathSync.native(process.cwd());
|
||||
// Tests run in managed worktrees beneath sensitive Codex state. Use a real
|
||||
// disposable workspace instead of weakening production host-path admission.
|
||||
export const WORKSPACE = realpathSync.native(mkdtempSync(join(tmpdir(), "paperclip-codex-test-workspace-")));
|
||||
afterAll(() => rmSync(WORKSPACE, { recursive: true, force: true }));
|
||||
|
||||
export class TestQueue<T> implements AsyncIterable<T> {
|
||||
values: T[] = [];
|
||||
|
||||
@@ -25,6 +25,17 @@ import { DurableCapabilityLiveSessionStore } from "./durable-live-session-store.
|
||||
import { defaultCapabilityRunnerdBinary } from "./runnerd-codex-transport.js";
|
||||
import { captureTurnRejection } from "../../test/capture-turn-rejection.js";
|
||||
|
||||
it.each(["pi", "cursor", "copilot"] as const)("requires separately bound evaluation opt-in for %s", async (acpxAgent) => {
|
||||
const service = new CapabilityLiveSessionService();
|
||||
await expect(service.create({ provider: "acpx", acpxAgent, requestedModel: "explicit-model" }))
|
||||
.rejects.toThrow("explicit evaluation opt-in");
|
||||
const mismatched = new CapabilityLiveSessionService({ transportOptions: {
|
||||
acpxCandidateProfile: acpxAgent === "pi" ? "cursor" : "pi",
|
||||
} });
|
||||
await expect(mismatched.create({ provider: "acpx", acpxAgent, requestedModel: "explicit-model" }))
|
||||
.rejects.toThrow("explicit evaluation opt-in");
|
||||
});
|
||||
|
||||
class AsyncNotifications implements AsyncIterable<CodexRpcNotification> {
|
||||
#values: CodexRpcNotification[] = [];
|
||||
#waiters: Array<(value: IteratorResult<CodexRpcNotification>) => void> = [];
|
||||
|
||||
@@ -628,7 +628,7 @@ export function assertCapabilityLiveSessionSnapshot(
|
||||
}
|
||||
if (provider === "acpx") {
|
||||
const agent = config.acpxAgent;
|
||||
if (agent !== "pi" && agent !== "claude" && agent !== "codex") {
|
||||
if (agent !== "pi" && agent !== "claude" && agent !== "codex" && agent !== "cursor" && agent !== "copilot") {
|
||||
throw new Error("capability_live_checkpoint_corrupt: invalid config.acpxAgent");
|
||||
}
|
||||
const expected = resolveQualifiedAcpxProfile(agent, text(config.requestedModel));
|
||||
@@ -861,8 +861,10 @@ export class CapabilityLiveSessionService {
|
||||
}
|
||||
|
||||
async create(input: CreateCapabilityLiveSessionInput = {}): Promise<CapabilityLiveSession> {
|
||||
if (input.provider === "acpx" && input.acpxAgent === "pi") {
|
||||
throw new Error("The Pi ACPX profile is not available");
|
||||
if (input.provider === "acpx" && input.acpxAgent !== undefined
|
||||
&& ["pi", "cursor", "copilot"].includes(input.acpxAgent)
|
||||
&& this.#transportOptions.acpxCandidateProfile !== input.acpxAgent) {
|
||||
throw new Error("The candidate ACPX profile requires explicit evaluation opt-in");
|
||||
}
|
||||
if (input.provider === "claude_managed" && !input.managedProfile) {
|
||||
throw new Error("Claude Managed live sessions require a qualified managed profile");
|
||||
|
||||
@@ -1647,6 +1647,10 @@ it("uses file-backed AWS workload identity without forwarding access keys or Pap
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ agent: "cursor" as const, allowed: ["CURSOR_API_KEY", "CURSOR_AUTH_TOKEN"],
|
||||
denied: ["COPILOT_GITHUB_TOKEN", "GH_TOKEN", "GITHUB_TOKEN", "OPENROUTER_API_KEY", "ANTHROPIC_API_KEY", "OPENAI_API_KEY"] },
|
||||
{ agent: "copilot" as const, allowed: ["COPILOT_GITHUB_TOKEN"],
|
||||
denied: ["CURSOR_API_KEY", "CURSOR_AUTH_TOKEN", "GH_TOKEN", "GITHUB_TOKEN", "OPENROUTER_API_KEY", "ANTHROPIC_API_KEY", "OPENAI_API_KEY"] },
|
||||
{
|
||||
agent: "pi" as const,
|
||||
allowed: ["OPENROUTER_API_KEY"],
|
||||
@@ -1682,6 +1686,8 @@ it.each([
|
||||
"passes only $agent ACPX credentials and the durable runtime binding",
|
||||
({ agent, allowed, denied }) => {
|
||||
const credentialEnvironment: Record<string, string> = {
|
||||
CURSOR_API_KEY: "cursor-key-canary", CURSOR_AUTH_TOKEN: "cursor-token-canary",
|
||||
COPILOT_GITHUB_TOKEN: "copilot-canary", GH_TOKEN: "ambient-gh-canary", GITHUB_TOKEN: "ambient-github-canary",
|
||||
OPENROUTER_API_KEY: "openrouter-canary",
|
||||
ANTHROPIC_API_KEY: "anthropic-canary",
|
||||
CLAUDE_CODE_OAUTH_TOKEN: "claude-oauth-canary",
|
||||
|
||||
@@ -1078,6 +1078,8 @@ export interface CapabilityRunnerdCodexTransportOptions {
|
||||
provider?: "codex" | "opencode" | "claude_managed" | "aws_agentcore" | "acpx";
|
||||
opencodePermissionMode?: NativeOpenCodePermissionMode;
|
||||
acpxAgent?: QualifiedAcpxAgent;
|
||||
/** Explicit evaluation-only candidate selection, never derived from persisted session input. */
|
||||
acpxCandidateProfile?: "pi" | "cursor" | "copilot";
|
||||
acpxPermissionMode?: NativeAcpxPermissionMode;
|
||||
acpxPermissionModePinned?: boolean;
|
||||
acpxSidecarPath?: string;
|
||||
@@ -3100,8 +3102,7 @@ export function createCapabilityRunnerdProviderEnvironment(input: {
|
||||
const providerPackageAuthority = acpxProviderPackageAuthority(sidecarPath);
|
||||
// This is the trusted runner/sidecar boundary. The provider sandbox still
|
||||
// uses createSanitizedAcpxSpawnInput and does not inherit gateway tokens.
|
||||
const assignedGateway = input.options.acpxAgent === "pi"
|
||||
? null : nativeMcpLaunchBinding(input.options.environment ?? {});
|
||||
const assignedGateway = nativeMcpLaunchBinding(input.options.environment ?? {});
|
||||
return {
|
||||
...(assignedGateway ? {
|
||||
PAPERCLIP_NATIVE_MCP_TOKEN: assignedGateway.token,
|
||||
@@ -3351,8 +3352,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
if (options.adoptExistingRunner && !options.stateDirectory?.trim()) {
|
||||
throw new Error("native_adopted_runner_state_directory_required");
|
||||
}
|
||||
if (options.provider === "acpx" && options.acpxAgent === "pi") {
|
||||
throw new Error("The Pi ACPX profile is not available");
|
||||
if (options.provider === "acpx" && options.acpxAgent !== undefined
|
||||
&& ["pi", "cursor", "copilot"].includes(options.acpxAgent)
|
||||
&& options.acpxCandidateProfile !== options.acpxAgent) {
|
||||
throw new Error("The candidate ACPX profile requires explicit evaluation opt-in");
|
||||
}
|
||||
this.#failureSignal = new Promise<never>((_resolve, reject) => {
|
||||
this.#rejectFailureSignal = reject;
|
||||
@@ -3466,6 +3469,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
? params.expectedTurnId
|
||||
: this.#turnId;
|
||||
if (!text.trim()) throw new Error("turn/steer requires a message");
|
||||
if (params.mode !== undefined && params.mode !== "steer" && params.mode !== "follow_up") throw new Error("turn/steer mode is invalid");
|
||||
if (expectedTurnId !== this.#turnId)
|
||||
throw new Error("turn/steer named a stale turn");
|
||||
const correlationId =
|
||||
@@ -3478,6 +3482,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
text,
|
||||
turnId: this.#durableTurnId,
|
||||
providerTurnId: expectedTurnId,
|
||||
...(params.mode === "follow_up" ? { mode: "follow_up" } : {}),
|
||||
...(correlationId ? { correlationId } : {}),
|
||||
},
|
||||
correlationId,
|
||||
@@ -4524,6 +4529,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
runId: identity.runId,
|
||||
cwd: String(params.cwd ?? tmpdir()),
|
||||
instructions: baseInstructions,
|
||||
providerPolicy: { readOnly: params.permissions === "paperclip-runner-workspace-read-only" },
|
||||
permissionMode: resolveRunnerdAcpxPermissionMode(
|
||||
this.options.acpxPermissionMode,
|
||||
),
|
||||
@@ -6185,7 +6191,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
if (
|
||||
descriptor.agent === "pi" ||
|
||||
descriptor.agent === "claude" ||
|
||||
descriptor.agent === "codex"
|
||||
descriptor.agent === "codex" ||
|
||||
descriptor.agent === "cursor" ||
|
||||
descriptor.agent === "copilot"
|
||||
)
|
||||
this.#evidence.acpxAgent = descriptor.agent;
|
||||
if (typeof descriptor.agentServerVersion === "string")
|
||||
|
||||
Reference in new issue
Block a user