diff --git a/doc/evals.md b/doc/evals.md index c1703ba383c..53dcfd52eb5 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -31,6 +31,13 @@ and the reloaded task UI; fixture prompts contain no naming instructions. ## Selecting a family +Pi's explicit-only `pi-controls` Product suite tests Stop while native permission +is unanswered and browser-originated same-turn steering, locally and on Daytona. +Its [fixture contract](../tests/runner-e2e/FIXTURES.md#pi-active-controls) separates +control acknowledgment, actual message consumption and owned process retirement. +The four cells add coverage without changing the existing 19 Pi cells; they +remain unqualified until measured on the exact candidate runtime. + Use **Runner Evals** for a runner protocol, adapter, transport, native session, tool grant, or one-turn provider qualification question. The workflow checks out an exact `paperclip-evals` revision, builds the Runner and viewer, runs a diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index 0cb0cb0ab70..a77628cf610 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -304,6 +304,54 @@ is reconnect evidence only; these cases do not establish provider-death recovery They use public product APIs, real browser answers, and ordinary isolated files; no database writes, private hooks, or fabricated provider results are allowed. +## Pi active controls + +The explicit-only `pi-controls` suite adds `pending-permission-stop` and +`same-turn-steering` on local and Daytona, each with one provider run, a +120-second active-turn timeout and a 300-second attempt budget. The existing +19 Pi cells and their suite fingerprints are unchanged. Catalog presence and +deterministic calibration do not constitute paid qualification. + +Both cases create a task through the browser and select `approve-reads` and +`per_turn` through the public agent API before startup. They retain +`paperclip.e2e.pi-control-pending.v1` while an exact native Pi write and its +permission card are pending in one run/turn/session/source. The card's native +tool ID is joined to the canonical execution ID using the existing runnerd +identity mapping. Pi does not emit Cursor/Copilot diagnostic notices; those +notices are never synthesized. Earlier native reads can provide orientation; +other native operations cannot substitute for the observed write. + +Stop awaits the pending evidence write and rereads that boundary before sending +one caller UUID to the public cancel API. It requires the original request's +normalized cancellation closure, a cancelled terminal, and the same-scope +caller-owned intent and acknowledgment audit IDs. Normal completion, a prior +permission decision, an expired request, and unacknowledged cancellation fail. +Only after cancellation does it attempt a stale **decline**, which must return +409. It never sends an allow decision. The task remains In Progress, with one +cancelled run and no automatic continuation. + +Steering submits a random marker only in a browser comment after the permission +is pending, then clicks that comment's production Steer button. It records the +exact public POST's queue/revision/run binding and requires the saved run +acknowledgment plus the Product facade's same-turn acknowledgment item. The raw +Rust `acpx-control-*` transport echo is suppressed by the facade; +`CodexHarnessSession.steer` emits the durable correlated item after the command +acknowledges. A deterministic calibration invokes that actual producer. The +browser denies the original write only after acknowledgment while the request +still remains pending. Success requires correlated native denial, the random +marker as the persisted and visible final response, Done, and one succeeded +run. Merely echoing the comment in the transcript or scheduling another turn +cannot pass. Native `pi/follow_up` and durable native queue state are not tested. + +Both cells require independent absent-target and zero-mutation evidence plus +owned process retirement. Local observation runs through cleanup. Daytona uses +the existing authenticated lease observer with exact image/executable pins, +fresh baseline/pending observations and its owned-process retirement seal; +retrieving that seal later does not claim later filesystem surveillance. +Missing cleanup fails the result. Provider-death recovery is outside these +cases. All screenshots and receipts use the existing evidence/redaction/result +pipeline; native USD may remain unknown and estimates remain distinct. + ## Copilot native protection The explicit-only `copilot-protection` suite has two cases on local and Daytona, diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index bc7c486a7c8..42be5397c91 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -25,6 +25,22 @@ The launcher always sets `PAPERCLIP_ANNOUNCEMENTS_ENABLED=false` for its isolate instances so announcement panels do not obscure screenshot evidence. No shell or workflow configuration is needed, including for Daytona cells. +## Pi controls (explicit only) + +`--suite pi-controls` selects four candidate Pi cells: pending-permission Stop +and browser same-turn steering, each local and Daytona. Discover without +credentials using `pnpm test:e2e:runner -- --list --suite pi-controls`. For an +authorized bounded attempt select, for example, +`--id pi-controls.runner-acpx-pi.local.pending-permission-stop +--max-automatic-retries 0 --max-parallel 1`. + +The [fixture contract](FIXTURES.md#pi-active-controls) requires exact pending +native-write evidence, caller-owned control acknowledgments, independent +no-effect/retirement proof and the actual production task UI. Daytona also +requires the existing immutable image and executable digests. These cells do +not establish native follow-up queue persistence or provider-death recovery. +They do not change the existing 19 Pi cells or promote Pi's pending profile. + ## Provider-free browser bootstrap regression `pnpm test:e2e:runner:browser-support` includes a wide development-module graph diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index c430ba2af47..125179ac420 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -136,10 +136,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 8, 7, 4, 4, 6, 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, + 4, 8, 7, 4, 4, 6, 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, ]); - expect(validateRunnerCatalog()).toHaveLength(462); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(462); + expect(validateRunnerCatalog()).toHaveLength(466); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(466); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index d58ba0b7e99..57038c9c31d 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1,4 +1,5 @@ import { nativeActiveStopTasks } from "./native-active-stop-tasks.js"; +import { piControlTasks } from "./pi-controls-cases.js"; import { cursorNativeTasks } from "./cursor-native-cases.js"; import { copilotProtectionTasks } from "./copilot-protection-tasks.js"; import { piNativeTasks } from "./pi-native-cases.js"; @@ -1080,6 +1081,16 @@ export const extendedHarnessFileTask: RunnerTaskFixture = { }; export const runnerSuites: readonly RunnerSuiteFixture[] = [ + { + id: "pi-controls", label: "Pi active controls", manualOnly: true, + description: "Pending native-write Stop and browser-originated same-turn steering, with exact control receipts and independent retirement/no-effect evidence.", + groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "pi"), + environments: runnerEnvironments, tasks: piControlTasks, expectedMatrixSize: 4, + definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, + pending: "paperclip.e2e.pi-control-pending.v1", stop: "paperclip.e2e.pi-stop-settlement.v1", steering: "paperclip.e2e.pi-steering-settlement.v1", + permissionPolicy: "approve-reads", lifecycle: "per_turn", normalCompletionProvesStop: false, nativeFollowUp: "not-covered", providerDeath: "not-covered", + remoteObservationCoverage: "continuous-through-owned-process-retirement", filesystemAfterRemoteRetirementObserved: false }, + }, { id: "cursor-native", label: "Cursor native interactions", manualOnly: true, description: "Native question continuation, revision-bound plan decisions and restrictive permission denial with independent process and file evidence.", diff --git a/tests/runner-e2e/harness-env.ts b/tests/runner-e2e/harness-env.ts index 64322b6dca0..0748d6ec6f0 100644 --- a/tests/runner-e2e/harness-env.ts +++ b/tests/runner-e2e/harness-env.ts @@ -118,6 +118,7 @@ export function buildRunnerE2EProcessEnvironment( const admittedSuite = execution.suite.id === "extended-harnesses" || execution.suite.id === "rich-acp-warm-continuity" || (execution.suite.id === "pi-native" && agent === "pi") + || (execution.suite.id === "pi-controls" && agent === "pi") || (execution.suite.id === "cursor-native" && agent === "cursor") || (execution.suite.id === "copilot-protection" && agent === "copilot") || (execution.suite.id === "native-active-stop" && (agent === "cursor" || agent === "copilot")); diff --git a/tests/runner-e2e/pi-controls-cases.ts b/tests/runner-e2e/pi-controls-cases.ts new file mode 100644 index 00000000000..24b17e1a740 --- /dev/null +++ b/tests/runner-e2e/pi-controls-cases.ts @@ -0,0 +1,17 @@ +import type { RunnerTaskFixture } from "./types.js"; + +export const piControlTasks: readonly RunnerTaskFixture[] = ["pending-permission-stop", "same-turn-steering"].map(id => ({ + id, label: id === "pending-permission-stop" ? "Stop Pi while permission is unanswered" : "Steer Pi through a pending native write", + groups: [], workMode: "standard", flow: "pi_controls", expectedRunCount: 1, + attemptTimeoutMs: { local: 300_000, daytona: 300_000 }, turnTimeoutMs: 120_000, + expectedTerminalState: id === "pending-permission-stop" ? { issue: "in_progress", run: "cancelled" } : { issue: "done", run: "succeeded" }, + buildTitle: nonce => `Pi ${id} ${nonce}`, + buildVisibleMarker: nonce => `PI-CONTROL-${nonce}`, + buildPrompt: () => [ + "Attempt the supplied native write exactly once and wait for its actual permission decision. Do not retry, answer your own permission, use bash, or use another path/tool to work around it.", + id === "pending-permission-stop" + ? "The operator will stop this unfinished task while permission remains unanswered. Do not finish or claim completion." + : "The operator will send a same-turn instruction while this permission is pending, then deny the write. After the actual denial, follow that new instruction. Denial is the expected result. Never claim that the file was written. Do not finish before receiving that instruction.", + ].join("\n"), + buildMatchers: () => [], // Dedicated flow joins public control receipts and independent retirement evidence. +})); diff --git a/tests/runner-e2e/pi-controls-evidence.test.ts b/tests/runner-e2e/pi-controls-evidence.test.ts new file mode 100644 index 00000000000..d30f5685454 --- /dev/null +++ b/tests/runner-e2e/pi-controls-evidence.test.ts @@ -0,0 +1,141 @@ +import { describe, expect, it, vi } from "vitest"; +import { createHash } from "node:crypto"; +import { CodexHarnessSession } from "../../packages/paperclip-runner/src/drivers/codex/codex-harness-session.js"; +import { assertSamePiPending, observePiControlPending, readPiStopSettlement, readPiSteeringAcknowledgement, readPiSteeringSettlement } from "./pi-controls-evidence.js"; +import { stopPiAtPendingPermission } from "./pi-controls-flow.js"; +import { piControlFixture, piControlCaller, piCancellationId } from "./pi-controls-test-fixture.js"; +import { runnerMatrix, runnerSuites, suiteDefinitionHash, validateRunnerCatalog } from "./catalog.js"; +import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; +import { buildRunnerE2EProcessEnvironment } from "./harness-env.js"; +import { assertRemoteNativeEvidencePrerequisites } from "./prerequisites.js"; + +type Row = Record; +const event = (row: Row) => row.payload.prpEvent; +function cancelled() { + const f = piControlFixture(), pending = f.pending(); f.cancel(); + const binding = { pending, caller: piControlCaller, cancellationRequestId: piCancellationId, dispatchMonotonicNs: String(BigInt(pending.observedMonotonicNs) + 1n) }; + return { f, binding, read: () => readPiStopSettlement({ ...f.state(), ...binding }) }; +} +describe("Pi pending control identity", () => { + it("accepts Pi native permission/start without invented provider notices", () => expect(piControlFixture().pending()).toMatchObject({ toolCallId: "pi-tool", executionId: "pi-tool", turnId: "turn" })); + it("admits permission-first ACP only after both canonical boundaries exist", () => { + const f = piControlFixture(); f.events.reverse(); + f.events.forEach((r, i) => { r.seq = event(r).sourceSeq = i + 1; event(r).sourceEventId = `source:run:${i + 1}`; }); + expect(f.pending().requestSourceSeq).toBe(1); + }); + it.each([ + ["wrong provider", (f: ReturnType) => { f.request.origin.provider = "copilot"; }], + ["foreign source", f => { event(f.events[1]!).sourceInstanceId = "other"; }], + ["foreign company", f => { f.events[0]!.companyId = "other"; }], + ["changed write path", f => { f.tool.target = "other.txt"; }], + ["write already ended", f => { f.append("tool.execution.completed", { ...f.tool, status: "failed" }); }], + ["answered request", f => { f.append("runtime_request.resolved", { requestId: "request", action: "decline" }); }], + ["normal completion", f => { f.append("turn.completed", { status: "completed" }); }], + ["extra native write", f => { f.append("tool.execution.started", { ...f.tool, executionId: "other" }); }], + ["missing decline", f => { f.request.choices = []; }], + ["missing tool origin", f => { f.events.shift(); }], + ["duplicate event", f => { f.events.push(f.events[0]!); }], + ["non-native run", f => { f.run.runtimeMode = "legacy"; }], + ["prior Stop", f => { f.run.resultJson.startupCancellation = {}; }], + ] satisfies Array<[string, (f: ReturnType) => void]>)("rejects %s", (_name, mutate) => { const f = piControlFixture(); mutate(f); expect(() => f.pending()).toThrow(); }); +}); +describe("Pi active Stop settlement", () => { + it("uses the production terminal mapper and actually cancels the pending callback", () => { + const s = cancelled(); expect(s.f.responses).toHaveLength(1); expect(s.read()).toMatchObject({ normalCompletionAccepted: false, schema: "paperclip.e2e.pi-stop-settlement.v1" }); + }); + it.each([ + ["normal terminal", (s: ReturnType) => { event(s.f.events.at(-1)!).eventType = s.f.events.at(-1)!.eventType = "turn.completed"; }], + ["expired callback", s => { const r = s.f.events.find(r => r.eventType === "runtime_request.cancelled")!; r.eventType = event(r).eventType = "runtime_request.expired"; }], + ["answered callback", s => { event(s.f.events.find(r => r.eventType === "runtime_request.cancelled")!).payload.action = "decline"; }], + ["changed pending target", s => { s.f.tool.target = "other.txt"; }], + ["changed retained hash", s => { s.binding.pending.requestRowSha256 = `sha256:${"a".repeat(64)}`; }], + ["late pending observation", s => { s.binding.dispatchMonotonicNs = s.binding.pending.observedMonotonicNs; }], + ["unacknowledged Stop", s => { s.f.run.resultJson.nativeCancellation.dispatchState = "pending"; }], + ["another caller", s => { s.f.run.resultJson.startupCancellation.requestedBy.userId = "other"; }], + ["another intent", s => { s.f.run.resultJson.nativeCancellation.intentId = "native-cancellation:other"; }], + ["extra cancellation effect", s => { s.f.run.resultJson.nativeCancellation.effects.push("pause_agent"); }], + ["same intent and ack audit", s => { s.f.run.resultJson.nativeCancellation.acknowledgementAuditId = "intent-audit"; }], + ["completed task", s => { s.f.issue.status = "done"; }], + ["foreign run", s => { s.f.run.id = "other"; }], + ] satisfies Array<[string, (s: ReturnType) => void]>)("rejects %s", (_name, mutate) => { const s = cancelled(); mutate(s); expect(s.read).toThrow(); }); + it("retains and rereads the boundary before dispatching one caller UUID", async () => { + const f = piControlFixture(), order: string[] = []; + const result = await stopPiAtPendingPermission({ scope: f.scope, caller: piControlCaller, deadlineAt: Date.now() + 1000, + load: async () => { order.push("load"); return f.state(); }, retain: async () => { order.push("retain"); }, stop: async (runId, uuid) => { order.push("stop"); expect(runId).toBe("run"); return f.cancel(uuid); } }); + expect(order.slice(0, 4)).toEqual(["load", "retain", "load", "stop"]); expect(result.settlement.normalCompletionAccepted).toBe(false); + }); + it("refuses dispatch when pending state changes during evidence retention", async () => { + const f = piControlFixture(), stop = vi.fn(); + await expect(stopPiAtPendingPermission({ scope: f.scope, caller: piControlCaller, deadlineAt: Date.now() + 1000, load: async () => f.state(), retain: async () => { f.request.itemId = "changed"; }, stop })).rejects.toThrow("pending boundary changed"); + expect(stop).not.toHaveBeenCalled(); + }); + it("fails immediately on normal completion instead of retrying it as an unsettled cancellation", async () => { + const f = piControlFixture(); + await expect(stopPiAtPendingPermission({ scope: f.scope, caller: piControlCaller, deadlineAt: Date.now() + 100, + load: async () => f.state(), retain: async () => {}, stop: async (_runId, uuid) => { const run = f.cancel(uuid); f.run.status = "succeeded"; return run; }, + })).rejects.toThrow(/^Stopped waiting for Pi pending permission cancellation: unexpected terminal$/); + }); +}); +function steered() { + const f = piControlFixture(), pending = f.pending(); f.steer(); + const binding = { pending, commentId: f.commentId, queueId: f.queueId, marker: f.marker, finalMessage: f.marker }; + const ack = readPiSteeringAcknowledgement({ ...f.state(), ...binding }); + assertSamePiPending(pending, observePiControlPending({ ...f.state(), scope: f.scope })); f.finish(); + return { f, binding, ack, read: () => readPiSteeringSettlement({ ...f.state(), ...binding }) }; +} +describe("Pi same-turn steering", () => { + it("calibrates against the actual Product ACP facade producer, not Rust's raw transport echo", async () => { + const f = piControlFixture(), pending = f.pending(), calls: Row[] = []; + // Only the transport and event sink are doubles. Run the actual public + // steer method, active-turn check and negotiated Pi capability boundary. + const session = Object.assign(Object.create(CodexHarnessSession.prototype), { + driverKind: "acpx_runtime", activeTurnId: "turn", opened: { threadId: "thread" }, + protocolIntegrityFailure: null, terminalTurns: new Map(), acknowledgedSteeringCorrelations: new Map(), + transport: { turnControlCapabilities: () => ({ steering: true, queuedFollowUp: true }), request: async (method: string, params: Row) => { calls.push({ method, params }); return {}; } }, + emit: (type: string, payload: Row, extra: Row) => f.append(type, payload, extra), + }) as CodexHarnessSession; + await session.steer({ turnId: "turn", correlationId: "comment", message: { role: "user", text: "Hidden instruction" } }); + expect(calls).toEqual([{ method: "turn/steer", params: { threadId: "thread", input: [{ type: "text", text: "Hidden instruction", text_elements: [] }], expectedTurnId: "turn", correlationId: "comment" } }]); + f.run.resultJson.queuedSteeringAcknowledgements = { comment: { status: "acknowledged", queueId: "queue", turnId: "turn", acknowledgedAt: "2026-10-01T00:00:01Z" } }; + const read = () => readPiSteeringAcknowledgement({ ...f.state(), pending, commentId: "comment", queueId: "queue" }); + expect(read().turnId).toBe("turn"); + event(f.events.at(-1)!).itemId = `acpx-control-${createHash("sha256").update("turn:comment").digest("hex")}`; + expect(read).toThrow("acknowledgement missing"); + }); + it("requires acknowledged steering before denial and the hidden marker in final output", () => expect(steered().read()).toMatchObject({ nativeFollowUpTested: false })); + it.each([ + ["marker echoed elsewhere", (s: ReturnType) => { s.binding.finalMessage = "Finished"; }], + ["narration mixed into final", s => { s.binding.finalMessage = `Working...${s.f.marker}`; }], + ["another queued comment", s => { s.binding.commentId = "other"; }], + ["another queue", s => { s.binding.queueId = "other"; }], + ["another turn acknowledgment", s => { s.f.run.resultJson.queuedSteeringAcknowledgements.comment.turnId = "other"; }], + ["missing native ack", s => { s.f.events.splice(2, 1); }], + ["follow-up substituted for steer", s => { event(s.f.events[2]!).payload.mode = "follow_up"; }], + ["accepted write", s => { event(s.f.events.find(r => r.eventType === "runtime_request.resolved")!).payload.action = "accept"; }], + ["completed write", s => { event(s.f.events.find(r => r.eventType === "tool.execution.completed")!).payload.status = "completed"; }], + ["unrelated tool failure", s => { event(s.f.events.find(r => r.eventType === "tool.execution.completed")!).payload.output = "Network error"; }], + ["cancelled run", s => { s.f.run.status = "cancelled"; }], + ["unfinished task", s => { s.f.issue.status = "in_progress"; }], + ] satisfies Array<[string, (s: ReturnType) => void]>)("rejects %s", (_name, mutate) => { const s = steered(); mutate(s); expect(s.read).toThrow(); }); +}); +describe("Pi controls catalog admission", () => { + it("adds exactly four explicit Pi-only cells without widening --all", () => { + const cells = selectRunnerExecutions(parseRunnerSelectors(["--suite", "pi-controls"])); + expect(cells).toHaveLength(4); expect(cells.every(c => c.profile.qualificationCandidate === "pi" && c.task.expectedRunCount === 1)).toBe(true); + expect(cells.map(c => `${c.environment.id}/${c.task.id}`).sort()).toEqual(["daytona/pending-permission-stop", "daytona/same-turn-steering", "local/pending-permission-stop", "local/same-turn-steering"]); + expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === "pi-controls")).toBe(false); + expect(validateRunnerCatalog()).toHaveLength(466); + for (const cell of cells) expect(JSON.parse(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION!)).toEqual([{ agent: "pi", model: cell.profile.model }]); + expect(() => assertRemoteNativeEvidencePrerequisites(cells, {})).toThrow(); + }); + it("keeps existing Pi-native and active Stop definition identities", () => { + const hashes = Object.fromEntries(runnerSuites.filter(s => ["pi-native", "native-active-stop", "extended-harnesses", "rich-acp-warm-continuity"].includes(s.id)).map(s => [s.id, suiteDefinitionHash(s)])); + expect(hashes).toEqual({ + "pi-native": "7aaacf83924fdfc7163594e08b9a64ac516f9eb8a0c2c96636b6d2fe4f23776a", + "native-active-stop": "99682b2b106d816a011834fae5a944ed7729958893709d5b83a19b6f595e7e4d", + "rich-acp-warm-continuity": "aead43b3f4f8ed285e2d1c30a517a1850633ae42e593f53122bea9e72af92585", + "extended-harnesses": "e75cb2222d2f52ec3a2a425211d195d36b95882de2cb9ed0ad93231b7c42768a", + }); + expect(runnerMatrix.filter(c => c.profile.qualificationCandidate === "pi" && c.suite.id !== "pi-controls")).toHaveLength(19); + }); +}); diff --git a/tests/runner-e2e/pi-controls-evidence.ts b/tests/runner-e2e/pi-controls-evidence.ts new file mode 100644 index 00000000000..6b51e604a0e --- /dev/null +++ b/tests/runner-e2e/pi-controls-evidence.ts @@ -0,0 +1,130 @@ +import { createHash } from "node:crypto"; +import { canonicalJson } from "../../packages/shared/src/portability-hash.js"; +import { hasAcpxNativeOrigin } from "./acpx-native-origin.js"; +import { bootstrapReadExecutionId } from "./native-bootstrap-read-proof.js"; +import { isValidNativePrpEnvelope } from "./native-event-envelope.js"; +import type { ActiveStopCaller } from "./native-active-stop-evidence.js"; + +type Row = Record; +export interface PiControlScope { companyId: string; issueId: string; runId: string; target: string } +export interface PiControlState { run: Row; issue: Row; events: readonly unknown[] } +export interface PiControlPending { + schema: "paperclip.e2e.pi-control-pending.v1"; scope: PiControlScope; requestId: string; toolCallId: string; executionId: string; + turnId: string; normalizedSessionId: string; sourceInstanceId: string; itemId: string; + requestSourceSeq: number; startedSourceSeq: number; requestRowSha256: string; startedRowSha256: string; observedMonotonicNs: string; +} +const rec = (v: unknown): Row => v !== null && typeof v === "object" && !Array.isArray(v) ? v as Row : {}; +const id = (v: unknown): v is string => typeof v === "string" && v.length > 0 && v.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(v) && !v.includes("[REDACTED]"); +const hash = (v: unknown) => `sha256:${createHash("sha256").update(canonicalJson(v)).digest("hex")}`; +function requireProof(value: unknown, reason: string): asserts value { if (!value) throw new Error(`Pi controls: ${reason}`); } +const terminalTypes = new Set(["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"]); +const closureTypes = new Set(["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"]); + +/** Consume the actual Product projection. Pi does not emit Cursor/Copilot native + * diagnostic notices; never manufacture those to reuse another provider's oracle. */ +function origin(events: readonly unknown[], scope: PiControlScope) { + requireProof(Object.values(scope).every(id) && events.length > 0 && events.length <= 20_000, "bounded exact scope required"); + const rows = events.map(rec).filter(row => rec(row.payload).prpEvent !== undefined).map(row => ({ row, event: rec(rec(row.payload).prpEvent) })).sort((a, b) => a.row.seq - b.row.seq); + const seqs = new Set(), sourceIds = new Set(), last = new Map(); + for (const { row, event: e } of rows) { + requireProof(row.companyId === scope.companyId && row.runId === scope.runId && isValidNativePrpEnvelope(e, row.protocolSchemaVersion) + && e.sourceKind === "runner" && e.runId === scope.runId && e.eventType === row.eventType && Number.isSafeInteger(row.seq) && row.seq > 0 && !seqs.has(row.seq) + && id(e.sourceInstanceId) && Number.isSafeInteger(e.sourceSeq) && e.sourceSeq > (last.get(e.sourceInstanceId) ?? 0) + && e.sourceEventId === `${e.sourceInstanceId}:${e.runId}:${e.sourceSeq}` && !sourceIds.has(e.sourceEventId), "foreign, duplicate or reordered event"); + seqs.add(row.seq); sourceIds.add(e.sourceEventId); last.set(e.sourceInstanceId, e.sourceSeq); + } + const created = rows.filter(x => x.event.eventType === "runtime_request.created"); + requireProof(created.length === 1, "one native permission required"); + const card = created[0]!, request = rec(rec(card.event.payload).request); + requireProof(request.schema === "paperclip.runtime_request.v2" && request.type === "permission" && request.requestKind === "permission_approval" && request.status === "pending" + && id(request.requestId) && id(request.itemId) && id(request.details?.toolCallId) && request.turnId === card.event.turnId && id(card.event.turnId) + && id(card.event.normalizedSessionId) && hasAcpxNativeOrigin(request.origin, "pi", "session/request_permission") + && Array.isArray(request.choices) && request.choices.some((c: Row) => c.key === "decline"), "native permission identity missing"); + const stream = (event: Row) => event.turnId === card.event.turnId && event.normalizedSessionId === card.event.normalizedSessionId && event.sourceInstanceId === card.event.sourceInstanceId; + requireProof(rows.filter(x => x.event.turnId != null).every(x => stream(x.event)), "foreign turn, session or producer"); + const executionId = bootstrapReadExecutionId(request.details.toolCallId); + const native = rows.filter(x => x.event.eventType.startsWith("tool.execution.") && rec(x.event.payload).transport === "builtin"); + const write = native.filter(x => rec(x.event.payload).executionId === executionId); + const started = write.filter(x => x.event.eventType === "tool.execution.started"); + requireProof(started.length === 1 && write.filter(x => x.event.eventType === "tool.execution.completed").length <= 1 + && write[0] === started[0] + && write.every(x => ["tool.execution.started", "tool.execution.progressed", "tool.execution.completed"].includes(x.event.eventType) + && x.event.payload.schema === "paperclip.tool.execution.v1" && x.event.payload.name === "write" && x.event.payload.operation === "edit" + && x.event.payload.target === scope.target && x.event.payload.status === (x.event.eventType === "tool.execution.completed" ? "failed" : "running")), "exact native write lifecycle missing"); + requireProof(!write.some((x, index) => x.event.eventType === "tool.execution.completed" && index !== write.length - 1), "write activity after terminal"); + // Read-only orientation/bootstrap can precede the write. Never exempt another + // edit, shell execution, or unknown native operation as an alleged bootstrap. + requireProof(native.every(x => rec(x.event.payload).executionId === executionId || (x.event.payload.operation === "read" && x.row.seq < Math.min(card.row.seq, started[0]!.row.seq))), "extra native operation"); + return { rows, card, request, started: started[0]!, executionId, stream }; +} +export function observePiControlPending(input: PiControlState & { scope: PiControlScope }): PiControlPending { + const { run, issue, scope } = input, p = origin(input.events, scope); + requireProof(run.id === scope.runId && run.companyId === scope.companyId && run.nativeIssueId === scope.issueId && run.status === "running" && run.runtimeMode === "native" + && issue.id === scope.issueId && issue.companyId === scope.companyId && issue.status === "in_progress" + && run.resultJson?.startupCancellation == null && run.resultJson?.nativeCancellation == null, "run is not fresh active work"); + requireProof(!p.rows.some(x => terminalTypes.has(x.event.eventType) || closureTypes.has(x.event.eventType) + || (x.event.eventType === "tool.execution.completed" && x.event.payload.executionId === p.executionId)), "permission already answered or provider settled"); + return { schema: "paperclip.e2e.pi-control-pending.v1", scope, requestId: p.request.requestId, toolCallId: p.request.details.toolCallId, executionId: p.executionId, + turnId: p.card.event.turnId, normalizedSessionId: p.card.event.normalizedSessionId, sourceInstanceId: p.card.event.sourceInstanceId, itemId: p.request.itemId, + requestSourceSeq: p.card.event.sourceSeq, startedSourceSeq: p.started.event.sourceSeq, requestRowSha256: hash(p.card.row), startedRowSha256: hash(p.started.row), observedMonotonicNs: process.hrtime.bigint().toString() }; +} +function retained(input: PiControlState & { pending: PiControlPending }) { + const b = input.pending; + requireProof(b.schema === "paperclip.e2e.pi-control-pending.v1", "pending receipt missing"); + const p = origin(input.events, b.scope); + requireProof(hash(p.card.row) === b.requestRowSha256 && hash(p.started.row) === b.startedRowSha256 && p.card.event.sourceSeq === b.requestSourceSeq && p.started.event.sourceSeq === b.startedSourceSeq + && p.request.requestId === b.requestId && p.request.details.toolCallId === b.toolCallId && p.executionId === b.executionId && p.request.itemId === b.itemId + && p.card.event.turnId === b.turnId && p.card.event.normalizedSessionId === b.normalizedSessionId && p.card.event.sourceInstanceId === b.sourceInstanceId, "retained pending identity changed"); + return p; +} +export function assertSamePiPending(before: PiControlPending, after: PiControlPending) { + requireProof(canonicalJson({ ...before, observedMonotonicNs: null }) === canonicalJson({ ...after, observedMonotonicNs: null }), "pending boundary changed before dispatch"); +} +export function readPiStopSettlement(input: PiControlState & { pending: PiControlPending; caller: ActiveStopCaller; cancellationRequestId: string; dispatchMonotonicNs: string }) { + const b = input.pending, p = retained(input), { run, issue, caller } = input; + requireProof(/^[1-9][0-9]{0,29}$/u.test(b.observedMonotonicNs) && /^[1-9][0-9]{0,29}$/u.test(input.dispatchMonotonicNs) + && BigInt(b.observedMonotonicNs) < BigInt(input.dispatchMonotonicNs) && /^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/u.test(input.cancellationRequestId) + && caller.type === "board" && caller.userId === "local-board" && caller.source === "local_implicit", "pre-dispatch caller proof missing"); + const closed = p.rows.filter(x => closureTypes.has(x.event.eventType)), terminal = p.rows.filter(x => terminalTypes.has(x.event.eventType)); + requireProof(closed.length === 1 && terminal.length === 1, "one cancellation closure and terminal required"); + const c = closed[0]!, t = terminal[0]!, cp = rec(c.event.payload), tp = rec(t.event.payload); + requireProof(c.event.eventType === "runtime_request.cancelled" && cp.requestId === b.requestId && cp.turnId === b.turnId && cp.itemId === b.itemId && cp.requestKind === "permission_approval" + && cp.reason === "turn_terminal" && cp.action === undefined && cp.response === undefined && cp.replayAllowed !== true + && t.event.eventType === "turn.cancelled" && tp.status === "cancelled" && tp.error === null && c.event.sourceSeq > Math.max(b.requestSourceSeq, b.startedSourceSeq) && t.event.sourceSeq > c.event.sourceSeq + && p.rows.filter(x => x.event.eventType.startsWith("tool.execution.")).every(x => x.event.sourceSeq < t.event.sourceSeq), "unanswered callback did not close through cancellation"); + const stop = rec(run.resultJson?.nativeCancellation), startup = rec(run.resultJson?.startupCancellation); + requireProof(run.id === b.scope.runId && run.companyId === b.scope.companyId && run.nativeIssueId === b.scope.issueId && run.status === "cancelled" && run.runtimeMode === "native" + && issue.id === b.scope.issueId && issue.companyId === b.scope.companyId && issue.status === "in_progress" + && startup.requestedBy?.type === caller.type && startup.requestedBy?.userId === caller.userId && startup.cancellationRequestId === input.cancellationRequestId + && stop.schema === "paperclip.native-cancellation.v1" && stop.intentId === `native-cancellation:${input.cancellationRequestId}` && stop.companyId === b.scope.companyId && stop.runId === run.id && stop.issueId === issue.id + && stop.scope === "run" && stop.dispatched === true && stop.dispatchState === "acknowledged" && stop.reasonCode === "cancellation_run_only" + && Array.isArray(stop.effects) && stop.effects.length === 1 && stop.effects[0] === "release_run_resources" && id(stop.intentAuditId) && id(stop.acknowledgementAuditId) && stop.intentAuditId !== stop.acknowledgementAuditId, "same-scope caller Stop acknowledgement missing"); + return { schema: "paperclip.e2e.pi-stop-settlement.v1", pending: b, cancellationRequestId: input.cancellationRequestId, dispatchMonotonicNs: input.dispatchMonotonicNs, + closedRowSha256: hash(c.row), terminalRowSha256: hash(t.row), intentId: stop.intentId, intentAuditId: stop.intentAuditId, acknowledgementAuditId: stop.acknowledgementAuditId, normalCompletionAccepted: false }; +} +export function readPiSteeringAcknowledgement(input: PiControlState & { pending: PiControlPending; commentId: string; queueId: string }) { + const p = retained(input), b = input.pending, saved = rec(input.run.resultJson?.queuedSteeringAcknowledgements?.[input.commentId]); + const receipts = p.rows.filter(x => x.event.eventType === "item.completed" && x.event.payload?.kind === "steering_acknowledgement"); + // Rust's raw acpx-control hash is a transport echo. The Product facade + // suppresses it (codex-session-notifications.ts) and CodexHarnessSession.steer + // emits this correlation-bound item only after request() is acknowledged. + requireProof(input.run.id === b.scope.runId && input.run.companyId === b.scope.companyId && input.run.nativeIssueId === b.scope.issueId && input.run.runtimeMode === "native" + && id(input.commentId) && id(input.queueId) && saved.status === "acknowledged" && saved.queueId === input.queueId && saved.turnId === b.turnId && Number.isFinite(Date.parse(saved.acknowledgedAt)) + && receipts.length === 1 && receipts[0]!.event.itemId === `${b.turnId}:steer:${input.commentId}` && receipts[0]!.event.payload.status === "acknowledged" + && receipts[0]!.event.payload.mode === "steer" && receipts[0]!.event.sourceSeq > Math.max(b.requestSourceSeq, b.startedSourceSeq), "same-turn steering acknowledgement missing"); + return { schema: "paperclip.e2e.pi-steering-ack.v1", commentId: input.commentId, queueId: input.queueId, turnId: b.turnId, sourceSeq: receipts[0]!.event.sourceSeq, rowSha256: hash(receipts[0]!.row) }; +} +export function readPiSteeringSettlement(input: PiControlState & { pending: PiControlPending; commentId: string; queueId: string; marker: string; finalMessage: string }) { + const ack = readPiSteeringAcknowledgement(input), p = retained(input), b = input.pending; + const closed = p.rows.filter(x => closureTypes.has(x.event.eventType)), terminal = p.rows.filter(x => terminalTypes.has(x.event.eventType)); + const failed = p.rows.filter(x => x.event.eventType === "tool.execution.completed" && x.event.payload.executionId === b.executionId); + requireProof(closed.length === 1 && closed[0]!.event.eventType === "runtime_request.resolved" && closed[0]!.event.payload.requestId === b.requestId && closed[0]!.event.payload.turnId === b.turnId + && closed[0]!.event.payload.action === "decline" && closed[0]!.event.sourceSeq > ack.sourceSeq + && failed.length === 1 && failed[0]!.event.sourceSeq > closed[0]!.event.sourceSeq && typeof failed[0]!.event.payload.output === "string" && failed[0]!.event.payload.output.includes("Pi operation was denied or cancelled") + && terminal.length === 1 && terminal[0]!.event.eventType === "turn.completed" && terminal[0]!.event.sourceSeq > failed[0]!.event.sourceSeq + && input.run.id === b.scope.runId && input.run.companyId === b.scope.companyId && input.run.nativeIssueId === b.scope.issueId && input.run.runtimeMode === "native" && input.run.status === "succeeded" + && input.issue.id === b.scope.issueId && input.issue.companyId === b.scope.companyId && input.issue.status === "done" + && input.run.resultJson?.nativeCancellation == null && input.run.resultJson?.startupCancellation == null + && /^PI-STEER-[a-f0-9]{32}$/u.test(input.marker) && input.finalMessage === input.marker, "steering was not consumed in the original denied-write turn"); + return { schema: "paperclip.e2e.pi-steering-settlement.v1", pending: b, acknowledgement: ack, marker: input.marker, terminalRowSha256: hash(terminal[0]!.row), nativeFollowUpTested: false }; +} diff --git a/tests/runner-e2e/pi-controls-flow.test.ts b/tests/runner-e2e/pi-controls-flow.test.ts new file mode 100644 index 00000000000..3c26f4c485c --- /dev/null +++ b/tests/runner-e2e/pi-controls-flow.test.ts @@ -0,0 +1,135 @@ +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, it, vi } from "vitest"; +import { piControlTasks } from "./pi-controls-cases.js"; +import { piControlFixture } from "./pi-controls-test-fixture.js"; +import { runPiControlsFlow } from "./pi-controls-flow.js"; + +const harness = vi.hoisted(() => ({ target: "", prompt: "", message: "", mutation: false, incomplete: false })); +vi.mock("./user-actions.js", () => ({ + createTaskThroughUi: async (input: { prompt: string }) => { harness.prompt = input.prompt; }, + submitTaskReply: async (_page: unknown, body: string) => { harness.message = body; return Date.now(); }, +})); +vi.mock("./copilot-local-fixtures.js", async importOriginal => { + const actual = await importOriginal(); + return { ...actual, createDeniedTargetFixture: async (workspace: string, name: string) => { + const f = await actual.createDeniedTargetFixture(workspace, name); harness.target = f.targetRelativePath; + return { ...f, watcher: { finish: () => ({ ...f.watcher.finish(), complete: !harness.incomplete && f.watcher.finish().complete, targetMutationCount: harness.mutation ? 1 : 0 }) } }; + }, observeRunProcesses: () => ({ sample: () => ({ captured: true, live: [], journal: [] }) }) }; +}); +vi.mock("@playwright/test", () => ({ expect: (actual: any, message?: string) => ({ + toBe: (value: unknown) => expect(actual, message).toBe(value), + toBeVisible: async () => {}, toBeEnabled: async () => {}, + toHaveCount: async (value: number) => { if (typeof actual.count === "function") expect(actual.count()).toBe(value); }, +}) })); + +async function exercise(taskId: string, remote: boolean, failure?: "mutation" | "incomplete" | "missing-ack") { + harness.target = ""; harness.prompt = ""; harness.message = ""; harness.mutation = failure === "mutation"; harness.incomplete = failure === "incomplete"; + const task = piControlTasks.find(t => t.id === taskId)!, stopCase = taskId === "pending-permission-stop"; + const workspacePath = await mkdtemp(join(tmpdir(), "pi-controls-fixture-")); + const saved = new Map(), localCleanup: Array<() => Promise> = [], remoteCleanup: Array<() => Promise> = []; + let f = piControlFixture(`pi-control-fixture.txt`), stops = 0, staleDeclines = 0, browserSteers = 0, browserDeclines = 0; + let steeringMarker = "", published = false, remoteSequence = 0; + const sync = () => { + const target = remote ? "pi-control-fixture.txt" : harness.target; + f.scope.target = target; f.tool.target = target; f.issue.title = task.buildTitle("fixture"); + }; + const queue = () => ({ queueId: "queue", targetRunId: "run", revision: "revision", protocol: "paperclip_runner_v1", steeringDisposition: "available", entries: harness.message ? [{ comment: { id: "comment", body: harness.message } }] : [] }); + const api = { + post: async (path: string, body: any) => { + if (path.endsWith("/projects")) return { name: "Pi controls fixture" }; + if (path === "/api/heartbeat-runs/run/cancel") { stops++; expect(stopCase).toBe(true); return f.cancel(body.cancellationRequestId); } + throw new Error(`Unexpected mutation ${path}`); + }, + patch: async (_path: string, value: any) => value, + get: async (path: string) => { + sync(); + if (path === "/api/health") return { deploymentMode: "local_trusted" }; + if (path === "/api/auth/get-session") return { session: { userId: "local-board", id: "paperclip:local_implicit:local-board" } }; + if (path === "/api/agents/agent") return { adapterConfig: {} }; + if (path.endsWith("/issues?limit=100")) return [f.issue]; + if (path === "/api/issues/issue") return f.issue; + if (path.endsWith("/heartbeat-runs?limit=100")) return [f.run]; + if (path === "/api/heartbeat-runs/run") return f.run; + if (path.includes("/events?")) return f.events; + if (path.endsWith("/queued-comments")) return queue(); + if (path.endsWith("/comments")) return f.run.status === "succeeded" ? [{ id: "final", createdByRunId: "run", body: steeringMarker }] : []; + throw new Error(`Unexpected read ${path}`); + }, + request: { post: async (path: string, options: any) => { + expect(path).toBe("/api/heartbeat-runs/run/runtime-requests/request/resolve"); + expect(f.run.status).toBe("cancelled"); expect(options.data.resolution.action).toBe("decline"); staleDeclines++; + return { status: () => 409 }; + } }, + }; + const waiters: Array<{ predicate: (request: any) => boolean; resolve: (request: any) => void }> = []; + function postBrowser(path: string, body: unknown) { + const request = { url: () => `http://fixture${path}`, method: () => "POST", postDataJSON: () => body }; + const waiter = waiters.find(w => w.predicate(request)); if (!waiter) throw new Error("Browser request was not awaited"); + waiter.resolve(request); waiters.splice(waiters.indexOf(waiter), 1); + } + const locator = (kind = "other"): any => ({ + filter: () => locator(kind), last: () => locator(kind), getByText: () => locator(), + count: () => kind === "loading" ? 0 : kind === "card" ? 1 : kind === "deny" ? (f.run.status === "running" ? 1 : 0) : 1, + getByRole: (_role: string, options: { name: string }) => locator(options.name === "Deny" ? "deny" : "other"), + getByTestId: () => locator(), + click: async () => { + if (kind === "steer") { + browserSteers++; expect(f.run.status).toBe("running"); expect(f.events.some(e => e.eventType.startsWith("runtime_request.") && e.eventType !== "runtime_request.created")).toBe(false); + steeringMarker = /PI-STEER-[a-f0-9]{32}/.exec(harness.message)?.[0] ?? ""; + expect(steeringMarker).not.toBe(""); expect(harness.prompt).not.toContain(steeringMarker); + f.steer(); + if (failure === "missing-ack") f.run.resultJson.queuedSteeringAcknowledgements = {}; + postBrowser("/api/issues/issue/queued-comments/comment/steer", { queueId: "queue", targetRunId: "run", revision: "revision" }); + } else if (kind === "deny") { + browserDeclines++; expect(browserSteers).toBe(1); expect(stopCase).toBe(false); f.finish(); + postBrowser("/api/heartbeat-runs/run/runtime-requests/request/resolve", { turnId: "turn", requestKind: "permission_approval", resolution: { action: "decline" } }); + } else throw new Error(`Unexpected click ${kind}`); + }, + }); + const page = { goto: async () => {}, reload: async () => {}, getByText: () => locator(), + getByTestId: (id: string) => locator(id === "task-chat-runtime-request" ? "card" : id.startsWith("task-chat-queued-steer-") ? "steer" : id === "task-chat-history-loading" ? "loading" : "other"), + waitForRequest: (predicate: (request: any) => boolean) => new Promise(resolve => waiters.push({ predicate, resolve })) }; + // The real shared remote oracle is exercised. Poisoned local copyback cannot + // satisfy these remote observations or retirement assertions. + const root = { pid: 50, ppid: 1, startTicks: "200", bootId: "12345678-1234-1234-1234-123456789abc" }; + const binding = { companyId: "company", environmentId: "environment", runId: "run", leaseId: "lease", sandboxId: "sandbox", image: `image@sha256:${"a".repeat(64)}`, remoteCwd: "/home/daytona/workspace" }; + const snapshot = (retired = false) => ({ + binding, observedAtMs: ++remoteSequence, receivedAtMs: remoteSequence + 1, observedMonotonicNs: String(remoteSequence), complete: true, workspace: {}, + targets: { "pi-control-fixture.txt": { absent: true, sha256: null, complete: true, mutationCount: retired && harness.mutation ? 1 : 0, parent: { dev: "1", ino: "2" } } }, + watcher: { complete: !(retired && harness.incomplete), targetMutationCount: retired && harness.mutation ? 1 : 0, workspaceMutationCount: 0 }, + processes: { captured: true, root, journal: [root], live: retired ? [] : [50] }, + setup: { path: ".paperclip-eval-action-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt", sha256: published ? `sha256:${"b".repeat(64)}` : null, published }, attached: null, + }); + let seal: ReturnType | undefined; + const fixture = { binding, remoteCwd: binding.remoteCwd, actionFile: ".paperclip-eval-action-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt", snapshot: async () => snapshot(), finish: async () => seal ??= snapshot(true), close: vi.fn() }; + const remoteBootstrap = remote ? { prompt: () => "Read bootstrap only", bindAndRelease: async (input: any) => { + expect(input.targets).toEqual(["pi-control-fixture.txt"]); harness.prompt = await input.actionPrompt(fixture); published = true; return fixture; + } } : undefined; + try { + if (remote) await writeFile(join(workspacePath, "pi-control-fixture.txt"), "WRONG HOST COPYBACK"); + const call = runPiControlsFlow({ page, api, fixtures: { company: { id: "company", issuePrefix: "PI" }, agent: { id: "agent", name: "Pi" }, environment: { id: "environment" } }, + execution: { task, suite: { id: "pi-controls" }, environment: { id: remote ? "daytona" : "local" }, profile: { qualificationCandidate: "pi" } }, workspacePath, nonce: "fixture", deadlineAt: Date.now() + 1000, + observe: () => {}, capture: async () => {}, evidence: async (name: string, value: unknown) => saved.set(name, structuredClone(value)), remoteBootstrap, + registerCleanupAssertion: (fn: () => Promise) => localCleanup.push(fn), registerBeforeEnvironmentTeardownAssertion: (fn: () => Promise) => remoteCleanup.push(fn) } as any); + if (failure === "missing-ack") { await expect(call).rejects.toThrow("acknowledgement"); expect(browserDeclines).toBe(0); } + else if (remote && failure) await expect(call).rejects.toThrow(); + else { + const result = await call; expect(result.checks.every(c => c.passed)).toBe(true); expect(saved.has("api-state.json")).toBe(true); + expect(stops).toBe(stopCase ? 1 : 0); expect(staleDeclines).toBe(stopCase ? 1 : 0); + expect(browserSteers).toBe(stopCase ? 0 : 1); expect(browserDeclines).toBe(stopCase ? 0 : 1); + } + expect(remote ? localCleanup : remoteCleanup).toHaveLength(0); + const cleanups = remote ? remoteCleanup : localCleanup; expect(cleanups).toHaveLength(1); + if (failure) await expect(cleanups[0]!()).rejects.toThrow(); + else expect((await cleanups[0]!())[0].passed).toBe(true); + if (remote) { expect(fixture.close).toHaveBeenCalledOnce(); expect(saved.get("pi-control-cleanup.json").retirement?.filesystemAfterRetirementObserved ?? false).toBe(false); } + } finally { await rm(workspacePath, { recursive: true, force: true }); } +} +for (const task of ["pending-permission-stop", "same-turn-steering"]) { + it.each([false, true])(`${task} drives actual flow and cleanup with remote=%s`, remote => exercise(task, remote)); +} +it.each(["mutation", "incomplete"] as const)("fails local cleanup on %s despite an absent final target", failure => exercise("pending-permission-stop", false, failure)); +it.each(["mutation", "incomplete"] as const)("fails remote lifetime proof on %s despite an absent target", failure => exercise("same-turn-steering", true, failure)); +it("never denies the native write before the steering acknowledgement exists", () => exercise("same-turn-steering", false, "missing-ack")); diff --git a/tests/runner-e2e/pi-controls-flow.ts b/tests/runner-e2e/pi-controls-flow.ts new file mode 100644 index 00000000000..6f5481b2017 --- /dev/null +++ b/tests/runner-e2e/pi-controls-flow.ts @@ -0,0 +1,206 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { expect, type Page } from "@playwright/test"; +import { pollUntil, type RunnerApi } from "./api.js"; +import { collectRunEvents } from "./run-observations.js"; +import { createTaskThroughUi, submitTaskReply } from "./user-actions.js"; +import { persistedFinalRunMessage } from "./matchers.js"; +import { createDeniedTargetFixture, exists, observeRunProcesses } from "./copilot-local-fixtures.js"; +import { assertCopilotRemoteRetirement, copilotRemoteDeniedSample, prepareCopilotRemoteAction, type CopilotRemoteBootstrap, type CopilotRemoteFixture, type CopilotRemoteSnapshot } from "./copilot-protection-evidence.js"; +import { assertActiveStopRetirement, readActiveStopRemoteRetirement, readActiveStopCaller, type ActiveStopCaller, type ActiveStopRemoteObservation } from "./native-active-stop-evidence.js"; +import { assertSamePiPending, observePiControlPending, readPiStopSettlement, readPiSteeringAcknowledgement, readPiSteeringSettlement, type PiControlPending, type PiControlScope, type PiControlState } from "./pi-controls-evidence.js"; +import { piNativeFinish } from "./pi-native-cases.js"; +import type { LiveFixtureValues } from "./live-fixtures.js"; +import type { MatrixExecution } from "./types.js"; + +type Row = Record; +type Check = { id: string; passed: boolean; detail: string }; + +/** Await the evidence write and reread the actual pending state before Stop. + * This deliberately never answers a permission to provoke cancellation. */ +export async function stopPiAtPendingPermission(input: { + scope: PiControlScope; caller: ActiveStopCaller; deadlineAt: number; + load(): Promise; retain(value: unknown): Promise; + stop(runId: string, cancellationRequestId: string): Promise; +}) { + const pending = observePiControlPending({ ...await input.load(), scope: input.scope }); + const cancellationRequestId = randomUUID(); + await input.retain({ pending, caller: input.caller, cancellationRequestId }); + assertSamePiPending(pending, observePiControlPending({ ...await input.load(), scope: input.scope })); + if (Date.now() >= input.deadlineAt) throw new Error("Pi pending Stop deadline expired"); + const dispatchMonotonicNs = process.hrtime.bigint().toString(); + const response = await input.stop(input.scope.runId, cancellationRequestId); + if (response.id !== input.scope.runId || response.resultJson?.nativeCancellation?.intentId !== `native-cancellation:${cancellationRequestId}`) throw new Error("Pi Stop response has foreign intent"); + const binding = { pending, caller: input.caller, cancellationRequestId, dispatchMonotonicNs }; + const state = await pollUntil({ label: "Pi pending permission cancellation", deadlineAt: input.deadlineAt, intervalMs: 200, + load: async () => { const value = await input.load(); if (["succeeded", "failed", "timed_out"].includes(value.run.status)) throw new Error("Stopped waiting for Pi pending permission cancellation: unexpected terminal"); return value; }, + accept: value => { readPiStopSettlement({ ...value, ...binding }); return true; } }); + return { ...binding, settlement: readPiStopSettlement({ ...state, ...binding }) }; +} + +export async function runPiControlsFlow(input: { + page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; workspacePath: string; deadlineAt: number; + remoteBootstrap?: CopilotRemoteBootstrap; + registerCleanupAssertion(callback: () => Promise): void; + registerBeforeEnvironmentTeardownAssertion(callback: () => Promise): void; + observe(issue: Row, runs: Row[]): void; + capture(id: string, label: string, file: string): Promise; + evidence(name: string, value: unknown): Promise; +}) { + const { api, page, fixtures, execution, nonce } = input; + if (execution.suite.id !== "pi-controls" || execution.profile.qualificationCandidate !== "pi" + || !["pending-permission-stop", "same-turn-steering"].includes(execution.task.id)) throw new Error("Unknown Pi control case"); + const stopCase = execution.task.id === "pending-permission-stop", remote = execution.environment.id === "daytona"; + if ((!remote && execution.environment.id !== "local") || (remote && !input.remoteBootstrap)) throw new Error("Pi controls require an isolated admitted environment"); + const checks: Check[] = []; let issue: Row = {}, runs: Row[] = [], events: Row[] = []; + const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); expect(passed, detail).toBe(true); }; + const name = `pi-control-${nonce}.txt`, local = remote ? undefined : await createDeniedTargetFixture(input.workspacePath, name), target = local?.targetRelativePath ?? name; + const observer = remote ? undefined : observeRunProcesses(); + let processes: { captured: boolean; live: number[] } = { captured: false, live: [] }, processIdentity: string | undefined, processError = false; + const samples: Array<{ phase: string; absent: boolean }> = [], remoteObservations: ActiveStopRemoteObservation[] = []; + let fixture: CopilotRemoteFixture | undefined, baseline: CopilotRemoteSnapshot | undefined, sealed: CopilotRemoteSnapshot | undefined; + let stopped: Awaited> | undefined; + let steered: { pending: PiControlPending; commentId: string; queueId: string; marker: string } | undefined; + let completed = false; + const observeProcesses = () => { + const run = runs[0], authority = run?.processPid ? { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id } : undefined; + if (authority) { const key = JSON.stringify(authority); if (processIdentity && processIdentity !== key) processError = true; processIdentity ??= key; } + if (observer) processes = observer.sample(authority); + }; + const load = async (): Promise => { + if (issue.id) issue = await api.get(`/api/issues/${issue.id}`); + const list = await api.get(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`); + runs = await Promise.all(list.map(run => api.get(`/api/heartbeat-runs/${run.id}`))); + if (runs.length > 1) throw new Error("Stopped waiting for Pi controls: extra provider run"); + events = runs[0] ? await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runs[0]!.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : []; + observeProcesses(); input.observe(issue, runs); return { run: runs[0] ?? {}, issue, events }; + }; + const scope = (): PiControlScope => ({ companyId: fixtures.company.id, issueId: issue.id, runId: runs[0]!.id, target }); + const finalMessage = async () => persistedFinalRunMessage(await api.get>(`/api/issues/${issue.id}/comments`), runs[0] as Row & { id: string }); + const assertSettled = async () => { + const state = await load(); + if (stopped) return readPiStopSettlement({ ...state, ...stopped }); + if (steered) return readPiSteeringSettlement({ ...state, ...steered, finalMessage: await finalMessage() }); + throw new Error("Pi controls settlement is missing"); + }; + const sample = async (phase: string) => { + let absent: boolean; + if (remote) { + if (!fixture || !baseline || phase !== "pending") throw new Error("Missing fresh Pi remote observation"); + const snapshot = await fixture.snapshot(phase); + remoteObservations.push({ phase, source: "live-snapshot", snapshot }); + absent = !copilotRemoteDeniedSample(snapshot, baseline, target, "pending").exists; + await input.evidence(`pi-control-${phase}-remote.json`, snapshot); + } else absent = !await exists(local!.targetPath); + samples.push({ phase, absent }); check(`no-effect-${phase}`, absent, "Exact pending native write has no file effect"); + }; + const timer = observer ? setInterval(() => { try { observeProcesses(); } catch { processError = true; } }, 100) : undefined; + (remote ? input.registerBeforeEnvironmentTeardownAssertion : input.registerCleanupAssertion)(async () => { + let retirement: unknown; + try { + await load(); + if (remote) { + if (!fixture || !baseline) throw new Error("Pi remote observer was not armed"); + sealed ??= await fixture.finish(); assertCopilotRemoteRetirement(sealed, baseline); processes = sealed.processes; + } else { + if (processes.captured && processes.live.length) await pollUntil({ label: "Pi control process retirement", deadlineAt: Date.now() + 5000, intervalMs: 100, + load: async () => { observeProcesses(); return processes; }, accept: value => value.live.length === 0 }); + await sample("after-cleanup"); + } + retirement = assertActiveStopRetirement({ completed, identityChanged: processError, processes, watcher: remote ? sealed!.watcher : local!.watcher.finish(), + ...(remote ? { environment: "daytona" as const, remote: { scope: scope(), observations: remoteObservations } } + : { environment: "local" as const, samples: samples.map(s => ({ ...s, phase: s.phase === "after-control" ? "after-stop" : s.phase })) }) }); + await assertSettled(); + return [{ id: "pi-control-retired-no-effects", passed: true, detail: remote ? "Owned remote tree retired with continuous no-effects proof through its lifetime seal" : "Owned local tree retired; no target mutation through cleanup" }]; + } finally { + if (timer) clearInterval(timer); + try { await input.evidence("pi-control-cleanup.json", { completed, processes, processError, samples, remoteObservations, retirement, sealed, watcher: local?.watcher.finish() }); } + finally { if (remote) await fixture?.close(); } + } + }); + const prompt = `${execution.task.buildPrompt(nonce)}\nUse native write once with relative path ${target} and content MUST_NOT_EXIST.`; + try { + const caller = readActiveStopCaller(await api.get("/api/health"), await api.get("/api/auth/get-session")); + const agent = await api.get(`/api/agents/${fixtures.agent.id}`); + const configured = await api.patch(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxPermissionMode: "approve-reads", lifecycleMode: "per_turn", timeoutSec: 120 } }); + check("explicit-per-turn-policy", configured.adapterConfig?.acpxPermissionMode === "approve-reads" && configured.adapterConfig?.lifecycleMode === "per_turn", "Native write requires a human decision before startup"); + const project = await api.post(`/api/companies/${fixtures.company.id}/projects`, { name: `Pi controls ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } }, workspace: { name: "Primary", sourceType: "local_path", cwd: input.workspacePath, isPrimary: true } }); + if (!remote) await sample("before-request"); + await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: remote ? input.remoteBootstrap!.prompt(nonce) : prompt, workMode: "standard", projectName: project.name }); + issue = (await pollUntil({ label: "browser-created Pi control task", deadlineAt: input.deadlineAt, load: async () => (await api.get(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(i => i.title === execution.task.buildTitle(nonce)), accept: Boolean }))!; + if (remote) { + await pollUntil({ label: "Pi control remote bootstrap", deadlineAt: input.deadlineAt, load, accept: state => state.run.status === "running" }); + const bound = await input.remoteBootstrap!.bindAndRelease({ issueId: issue.id, runId: runs[0]!.id, targets: [target], actionPrompt: async value => { + fixture = value; + const prepared = await prepareCopilotRemoteAction({ fixture: value, companyId: fixtures.company.id, environmentId: fixtures.environment.id, runId: runs[0]!.id, target, prompt }); + baseline = prepared.baseline; remoteObservations.push({ phase: "before-request", source: "live-snapshot", snapshot: baseline }); + await input.evidence("pi-control-before-request-remote.json", baseline); return prepared.prompt; + } }); + if (bound !== fixture) throw new Error("Pi remote action identity changed"); + } + await pollUntil({ label: "Pi unanswered native write", deadlineAt: input.deadlineAt, intervalMs: 200, + load: async () => { const state = await load(); if (["failed", "timed_out", "cancelled", "succeeded"].includes(state.run.status)) throw new Error("Stopped waiting for Pi unanswered native write: no active pending permission"); return state; }, + accept: state => { observePiControlPending({ ...state, scope: scope() }); return true; } }); + await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`); + const card = page.getByTestId("task-chat-runtime-request").filter({ visible: true }); + await expect(card).toHaveCount(1); await expect(card.getByRole("button", { name: "Deny", exact: true })).toBeEnabled(); + await input.capture("pending-permission", "Pi write waits for a human decision", "pending-permission.png"); await sample("pending"); + if (stopCase) { + stopped = await stopPiAtPendingPermission({ scope: scope(), caller, deadlineAt: input.deadlineAt, load, + retain: value => input.evidence("pi-control-pending.json", value), stop: (runId, cancellationRequestId) => api.post(`/api/heartbeat-runs/${runId}/cancel`, { cancellationRequestId }) }); + await input.evidence("pi-stop-settlement.json", stopped); + // A stale decline is sufficient to test the response fence. Never send + // an allow choice, even after a proven cancellation. + const response = await api.request.post(`/api/heartbeat-runs/${runs[0]!.id}/runtime-requests/${encodeURIComponent(stopped.pending.requestId)}/resolve`, { data: { turnId: stopped.pending.turnId, requestKind: "permission_approval", resolution: { action: "decline" } } }); + check("stale-answer-refused", response.status() === 409, "Cancelled permission refuses a stale response"); + } else { + const pending = observePiControlPending({ ...await load(), scope: scope() }); + await input.evidence("pi-control-pending.json", pending); + const marker = `PI-STEER-${randomBytes(16).toString("hex")}`; + const message = `After the actual native denial, complete this negative test. Do not write or invoke any other native tool. ${piNativeFinish(marker)}`; + await submitTaskReply(page, message); + const queue = await pollUntil({ label: "Pi browser comment queued for steering", deadlineAt: input.deadlineAt, intervalMs: 200, + load: () => api.get(`/api/issues/${issue.id}/queued-comments`), accept: q => q.steeringDisposition === "available" && q.entries?.some((entry: Row) => entry.comment.body === message) }); + const entries = queue.entries.filter((entry: Row) => entry.comment.body === message); + check("one-browser-steering-message", entries.length === 1 && queue.entries.length === 1 && queue.targetRunId === pending.scope.runId, "One browser-originated comment targets the pending run"); + const commentId = entries[0].comment.id, queueId = queue.queueId; + assertSamePiPending(pending, observePiControlPending({ ...await load(), scope: scope() })); + await input.evidence("pi-steering-queued.json", { pending, queue, commentId, marker }); + const route = `/api/issues/${issue.id}/queued-comments/${commentId}/steer`; + const posted = page.waitForRequest(request => new URL(request.url()).pathname === route && request.method() === "POST"); + await page.getByTestId(`task-chat-queued-steer-${commentId}`).click(); + const body = (await posted).postDataJSON(); + check("exact-browser-steer", body.queueId === queueId && body.revision === queue.revision && body.targetRunId === pending.scope.runId, "Browser steers the exact queued comment into the active run"); + steered = { pending, commentId, queueId, marker }; + await pollUntil({ label: "Pi same-turn steering acknowledgement", deadlineAt: input.deadlineAt, intervalMs: 200, load, + accept: state => { assertSamePiPending(pending, observePiControlPending({ ...state, scope: scope() })); readPiSteeringAcknowledgement({ ...state, ...steered! }); return true; } }); + await input.evidence("pi-steering-ack.json", readPiSteeringAcknowledgement({ ...await load(), ...steered })); + await input.capture("steered-pending", "Steering acknowledged while permission remains unanswered", "steered-pending.png"); + const declineRoute = `/api/heartbeat-runs/${pending.scope.runId}/runtime-requests/${encodeURIComponent(pending.requestId)}/resolve`; + const declined = page.waitForRequest(request => new URL(request.url()).pathname === declineRoute && request.method() === "POST"); + await card.getByRole("button", { name: "Deny", exact: true }).click(); + const decision = (await declined).postDataJSON(); + check("exact-browser-decline", decision.turnId === pending.turnId && decision.requestKind === "permission_approval" && decision.resolution?.action === "decline", "Browser denied the still-pending original native write"); + await pollUntil({ label: "Pi consumed same-turn steering", deadlineAt: input.deadlineAt, intervalMs: 200, + load: async () => { const state = await load(); if (["cancelled", "failed", "timed_out"].includes(state.run.status)) throw new Error("Stopped waiting for Pi consumed same-turn steering: provider failed"); return { ...state, finalMessage: await finalMessage() }; }, + accept: state => { readPiSteeringSettlement({ ...state, ...steered! }); return true; } }); + await input.evidence("pi-steering-settlement.json", await assertSettled()); + } + check("control-settled", true, stopCase ? "Exact active permission closed under the caller-owned Stop" : "Hidden steering marker consumed in the original denied-write turn"); + if (remote) { + sealed = await fixture!.finish(); assertCopilotRemoteRetirement(sealed, baseline!); processes = sealed.processes; + remoteObservations.push({ phase: "owned-process-retirement", source: "retirement-seal", snapshot: sealed }); + await input.evidence("pi-control-owned-retirement-remote.json", sealed); + readActiveStopRemoteRetirement({ scope: scope(), observations: remoteObservations }); + } else await sample("after-control"); + await page.reload(); + await expect(page.getByTestId("issue-detail-header").getByRole("button", { name: `Change status (current: ${stopCase ? "In Progress" : "Done"})`, exact: true })).toBeVisible(); + if (stopCase) await expect(page.getByTestId("task-chat-thread").getByTestId("task-chat-collapsible-marker").filter({ has: page.getByText("Run cancelled", { exact: true }) })).toBeVisible(); + else await expect(page.getByTestId("task-chat-thread").getByText(steered!.marker, { exact: true }).last()).toBeVisible(); + await expect(page.getByTestId("task-chat-history-loading")).toHaveCount(0); + await expect(card.getByRole("button", { name: "Deny", exact: true })).toHaveCount(0); + await assertSettled(); completed = true; + await input.capture("final-state", "Pi control verified through the production task UI", "final-state.png"); + await input.evidence("api-state.json", { issue, run: runs[0], runs, checks, samples, remoteObservations, runEvents: events, runEventsByRun: [{ runId: runs[0]!.id, events }], stopped, steered }); + return { issue, runs, checks }; + } finally { await input.evidence("pi-control-checks.json", { issue, runs, checks, samples, remoteObservations, completed }); } +} diff --git a/tests/runner-e2e/pi-controls-test-fixture.ts b/tests/runner-e2e/pi-controls-test-fixture.ts new file mode 100644 index 00000000000..1f37d498751 --- /dev/null +++ b/tests/runner-e2e/pi-controls-test-fixture.ts @@ -0,0 +1,58 @@ +import { CodexSessionState } from "../../packages/paperclip-runner/src/drivers/codex/codex-session-state.js"; +import { mapTerminalTurn } from "../../packages/paperclip-runner/src/drivers/codex/codex-session-terminal.js"; +import { observePiControlPending } from "./pi-controls-evidence.js"; +import { readActiveStopCaller } from "./native-active-stop-evidence.js"; + +type Row = Record; +export const piControlCaller = readActiveStopCaller({ deploymentMode: "local_trusted" }, { session: { userId: "local-board", id: "paperclip:local_implicit:local-board" } }); +export const piCancellationId = "11111111-2222-4333-8444-555555555555"; +export function piControlFixture(target = "target.txt") { + const scope = { companyId: "company", issueId: "issue", runId: "run", target }; + const events: Row[] = []; + const row = (eventType: string, payload: Row, extra: Row = {}) => { + const seq = events.length + 1; + return { companyId: "company", runId: "run", seq, eventType, protocolSchemaVersion: 1, + payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", eventType, runId: "run", turnId: "turn", normalizedSessionId: "session", sourceInstanceId: "source", sourceSeq: seq, + sourceEventId: `source:run:${seq}`, emittedAt: "2026-10-01T00:00:00Z", payload, ...extra } } }; + }; + const append = (eventType: string, payload: Row, extra: Row = {}) => events.push(row(eventType, payload, extra)); + const tool = { schema: "paperclip.tool.execution.v1", executionId: "pi-tool", transport: "builtin", name: "write", operation: "edit", target, status: "running" }; + append("tool.execution.started", tool); + const request = { schema: "paperclip.runtime_request.v2", type: "permission", requestKind: "permission_approval", method: "session/request_permission", status: "pending", requestId: "request", turnId: "turn", itemId: "permission-item", + origin: { adapter: "acpx-runtime-sidecar", provider: "pi", method: "session/request_permission" }, details: { toolCallId: "pi-tool" }, choices: [{ key: "decline", label: "Deny" }] }; + append("runtime_request.created", { request }); + const run: Row = { id: "run", companyId: "company", nativeIssueId: "issue", runtimeMode: "native", status: "running", resultJson: {}, processPid: 123, processGroupId: 123, processStartedAt: "2026-10-01T00:00:00Z" }; + const issue: Row = { id: "issue", companyId: "company", identifier: "PI-1", status: "in_progress" }; + const state = () => ({ events, run, issue }); + const pending = () => observePiControlPending({ ...state(), scope }); + const responses: unknown[] = []; + function cancel(cancellationRequestId = piCancellationId) { + // Exercise the real Product terminal mapper and callback cancellation, not + // just a fabricated terminal whose payload mirrors the matcher. + const session = { + terminalTurns: new Map(), workspaceChangesByTurn: new Map(), result: null, conversationMode: "direct", activeTurnId: "turn", turnStarted: true, + pendingRuntimeRequestMap: new Map([[request.requestId, { request, settle: (value: unknown) => responses.push(value) }]]), + cancelPendingRequests: CodexSessionState.prototype.cancelPendingRequests, + emit: (eventType: string, payload: Row) => append(eventType, payload), + } as unknown as CodexSessionState; + mapTerminalTurn(session, { id: "turn", status: "cancelled", error: null }, "turn"); + run.status = "cancelled"; + run.resultJson = { startupCancellation: { cancellationRequestId, requestedBy: { type: "board", userId: "local-board" } }, nativeCancellation: { + schema: "paperclip.native-cancellation.v1", intentId: `native-cancellation:${cancellationRequestId}`, companyId: "company", runId: "run", issueId: "issue", scope: "run", + dispatched: true, dispatchState: "acknowledged", reasonCode: "cancellation_run_only", effects: ["release_run_resources"], intentAuditId: "intent-audit", acknowledgementAuditId: "ack-audit", + } }; + return run; + } + const marker = `PI-STEER-${"a".repeat(32)}`, commentId = "comment", queueId = "queue"; + function steer() { + run.resultJson.queuedSteeringAcknowledgements = { [commentId]: { status: "acknowledged", queueId, turnId: "turn", acknowledgedAt: "2026-10-01T00:00:01Z" } }; + append("item.completed", { kind: "steering_acknowledgement", status: "acknowledged", mode: "steer", text: "Steering acknowledged for the active turn." }, { itemId: `turn:steer:${commentId}` }); + } + function finish() { + append("runtime_request.resolved", { requestId: "request", turnId: "turn", action: "decline" }); + append("tool.execution.completed", { ...tool, status: "failed", output: "Pi operation was denied or cancelled" }); + append("turn.completed", { status: "completed", error: null }); + run.status = "succeeded"; issue.status = "done"; + } + return { scope, events, run, issue, state, pending, row, append, request, tool, cancel, responses, steer, finish, marker, commentId, queueId }; +} diff --git a/tests/runner-e2e/prerequisites.ts b/tests/runner-e2e/prerequisites.ts index 1fdc8b5cd3e..701a54a2fdb 100644 --- a/tests/runner-e2e/prerequisites.ts +++ b/tests/runner-e2e/prerequisites.ts @@ -18,7 +18,7 @@ export function assertRemoteNativeEvidencePrerequisites( environment: NodeJS.ProcessEnv, ): void { if (!executions.some(execution => execution.environment.id === "daytona" - && ["cursor_native", "pi_native", "copilot_protection", "native_active_stop"].includes(execution.task.flow))) return; + && ["cursor_native", "pi_native", "pi_controls", "copilot_protection", "native_active_stop"].includes(execution.task.flow))) return; const names = ["PAPERCLIP_E2E_DAYTONA_NODE_SHA256", "PAPERCLIP_E2E_DAYTONA_RUNNERD_SHA256"]; const invalid = names.filter(name => !/^sha256:[a-f0-9]{64}$/u.test(environment[name] ?? "")); if (invalid.length) throw new Error(`Native Daytona evidence requires exact image executable digests: ${invalid.join(", ")}`); diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index a08ded683a4..e3b003ac2fe 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,4 +1,5 @@ import { runNativeActiveStopFlow } from "./native-active-stop-flow.js"; +import { runPiControlsFlow } from "./pi-controls-flow.js"; import { runCursorNativeFlow } from "./cursor-native-flow.js"; import { createRemoteNativeBootstrap, createRemoteFixtureClient } from "./remote-native-bootstrap.js"; import { runCleanupWithObservers, verifyCleanupAssertions, type CleanupAssertion } from "./cleanup-verification.js"; @@ -561,7 +562,7 @@ for (const execution of executions) { const credentials = credentialValues(); const secrets = normalizedSecrets(Object.values(credentials)); const api = new RunnerApi(request); - const companyRunFlow = execution.suite.id === "task-titles" || ["blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native", "copilot_protection", "cursor_native", "native_active_stop"].includes(execution.task.flow); + const companyRunFlow = execution.suite.id === "task-titles" || ["blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native", "pi_controls", "copilot_protection", "cursor_native", "native_active_stop"].includes(execution.task.flow); const consoleDiagnostics: Array> = []; const networkDiagnostics: Array> = []; const pageLifecycleDiagnostics: Array> = []; @@ -859,7 +860,7 @@ for (const execution of executions) { }); const remoteBootstrap = execution.environment.id === "daytona" - && ["cursor_native", "pi_native", "copilot_protection", "native_active_stop"].includes(execution.task.flow) + && ["cursor_native", "pi_native", "pi_controls", "copilot_protection", "native_active_stop"].includes(execution.task.flow) ? createRemoteNativeBootstrap({ api, daytona: await createRemoteFixtureClient(credentials.DAYTONA_API_KEY ?? ""), companyId: fixtures.company.id, environmentId: fixtures.environment.id, agentId: fixtures.agent.id, @@ -937,6 +938,15 @@ for (const execution of executions) { evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), }); issue = accounting.issue as IssueRecord; selectedRuns = accounting.runs as RunRecord[]; + } else if (execution.task.flow === "pi_controls") { + const story = await runPiControlsFlow({ + page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs, + observe: (currentIssue, currentRuns) => { issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; }, + capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), + remoteBootstrap, registerCleanupAssertion, registerBeforeEnvironmentTeardownAssertion, + }); + issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[]; + matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `piControls.${check.id}`, expected: true }, passed: check.passed, detail: check.detail })); } else if (execution.task.flow === "native_active_stop") { const story = await runNativeActiveStopFlow({ page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs, diff --git a/tests/runner-e2e/types.ts b/tests/runner-e2e/types.ts index 82ea217ebc8..c3deb1d84c2 100644 --- a/tests/runner-e2e/types.ts +++ b/tests/runner-e2e/types.ts @@ -31,6 +31,7 @@ export type RunnerTaskFlow = | "warm_three_turn" | "instruction_persistence" | "pi_native" + | "pi_controls" | "native_active_stop" | "copilot_protection" | "cursor_native";