test(runner-e2e): qualify Stop at pending native permission

Add explicit Cursor and Copilot local/Daytona cells that retain an unanswered callback before a caller-correlated Stop, require cancelled provider settlement and stale-answer rejection, and independently verify no effects through owned retirement. Normal completion and provider death do not satisfy this active-work oracle.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-09-30 14:42:10 -05:00
1 parent e4bf459e31
commit a6a5417cd9
11 files changed
+589 -6

No files matched your search

+31
View File
@@ -391,3 +391,34 @@ the response and final `paperclip.e2e.copilot-denial-settlement.v3` receipt.
It refuses a non-running controller or existing Stop marker before dispatch.
The completed-provider branch still requires a running controller that this
request can stop; it does not accept a no-op Stop of an already terminal run.
## Definitely-active native Stop
`native-active-stop` / `pending-permission-stop` has four explicit-only cells:
Cursor and Copilot, each local and Daytona. Its `native_active_stop` flow retains
`paperclip.e2e.native-active-stop-pending.v1` from the public API while exactly one
native permission remains pending, the exact controller run is running, and no
Stop or answer marker exists. The receipt independently binds native session,
normalized session, turn, source instance, request/tool IDs, source sequences and
canonical row hashes. An awaited artifact write and fresh pending reread precede
Stop dispatch; process-monotonic timestamps describe only these local observer
boundaries. Remote provider clocks and database transaction timestamps never
establish that ordering. The atomic caller UUID fence rejects an earlier racing
Stop instead of borrowing its acknowledgement.
`paperclip.e2e.native-active-stop-settlement.v1` accepts only
`pending_permission_cancelled`: explicit-cancellation request closure with
`replayAllowed:false`, then one exact `turn.cancelled`, plus the same scoped
caller-owned native Stop acknowledgement. Missing, duplicate, foreign, failed,
interrupted or normal terminal evidence fails. Only after this proof does the
fixture attempt a stale public answer, requiring HTTP409 and an unanswerable
browser card. It retains one cancelled run, issue `in_progress`, exact target
absence through continuous observation, and all observed owned descendants
retired. Remote proof is sealed before environment teardown; unproven cleanup
fails independently. Only fully attested bootstrap reads may precede the one
tested operation; alternate operations or attempts are rejected.
The existing `copilot-protection` denial remains distinct: rejecting a permission
before Stop does not exercise this pending-callback boundary. This new suite has
pure calibration and wiring tests, not a paid qualification result. Provider
process death is not simulated by substituting the chat runner-worker crash hook.
+26
View File
@@ -1447,3 +1447,29 @@ pre-Stop observation to the public cancel request and its exact durable native
intent. It rejects prior startup/Stop markers before dispatch and a competing
request that wins after the last read. The server atomically reserves the caller
identity; the fixture does not infer ownership from timestamps or HTTP success.
## Stop during an unanswered native permission (explicit only)
`--suite native-active-stop --task pending-permission-stop` selects Cursor and
Copilot on local or Daytona, one run per cell. These four cells are excluded
from `--all`. They use `approve-reads`, per-turn lifecycle and Cursor Agent mode;
provider timeout is 120s and the attempt budget is 300s. The fixture asks for one
exact native write, observes its unanswered permission card, then sends the
public Stop request with a freshly retained caller UUID. It never denies or
approves that callback before Stop.
Passage requires a canonical cancelled request and cancelled provider turn,
exact caller-intent acknowledgement, an unfinished task, rejection of a later
stale answer, no follow-up run, continuous target no-effect observation and
owned process retirement. Normal completion and interrupted/failed turns do
not qualify this case. Native permission cards use runtime requests; ordinary
issue-interaction rows are not substituted for their authority. Pending and final
screenshots, `native-active-stop-pending.json`, `native-active-stop-settlement.json`
and cleanup evidence retain the boundaries. Daytona also requires the exact
owned lease and sealed remote observer proof before sandbox deletion.
This suite adds an active-work cancellation oracle; it does not reinterpret
older denial or cancellation results. Current live qualification is pending.
Provider death during a pending callback remains a separate uncovered case:
the existing chat worker-crash hook targets a runner worker and cannot establish
safe ownership of the native provider process in both environments.
+3 -3
View File
@@ -91,10 +91,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
8, 7, 4, 6, 30, 3, 16, 16, 2, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 10, 48, 16, 10, 2,
8, 7, 4, 4, 6, 30, 3, 16, 16, 2, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 10, 48, 16, 10, 2,
]);
expect(validateRunnerCatalog()).toHaveLength(432);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(432);
expect(validateRunnerCatalog()).toHaveLength(436);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(436);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
+8
View File
@@ -1,3 +1,4 @@
import { nativeActiveStopTasks } from "./native-active-stop-tasks.js";
import { cursorNativeTasks } from "./cursor-native-cases.js";
import { copilotProtectionTasks } from "./copilot-protection-tasks.js";
import { piNativeTasks } from "./pi-native-cases.js";
@@ -1046,6 +1047,13 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
excludedExecutionIds: ["pi-native.runner-acpx-pi.daytona.restrictive-denial"],
definitionMetadata: { version: 2, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, remoteDenyAll: "unsupported-native-bootstrap-read-is-denied", remoteEvidence: "owned-lease-sealed-observer" },
},
{
id: "native-active-stop", label: "Stop an unanswered native permission", manualOnly: true,
description: "Stop while one exact Cursor or Copilot native permission remains unanswered; require cancelled provider settlement, caller-owned acknowledgement, stale-answer refusal and independent retirement/no effects.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => ["cursor", "copilot"].includes(profile.qualificationCandidate ?? "")),
environments: runnerEnvironments, tasks: nativeActiveStopTasks, expectedMatrixSize: 4,
definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", evidence: "paperclip.e2e.native-active-stop-settlement.v1", pendingObservation: "retained-api-before-caller-uuid-stop", normalCompletionAccepted: false, permissionPolicy: "approve-reads", lifecycle: "per_turn", remoteEvidence: "owned-lease-sealed-observer", providerDeath: "not-covered" },
},
{
id: "copilot-protection", label: "Copilot native protection", manualOnly: true,
description: "Exact native denial with independently correlated provider settlement and acknowledged run Stop, plus attached command settlement with independent process evidence.",
@@ -0,0 +1,161 @@
import { createHash } from "node:crypto";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { hasAcpxNativeOrigin } from "./acpx-native-origin.js";
import { readCopilotToolEvidence } from "./copilot-evidence.js";
import { readCursorToolEvidence } from "./cursor-native-evidence.js";
import { bootstrapReadExecutionId, withoutProvenBootstrapReads, type BootstrapReadProof } from "./native-bootstrap-read-proof.js";
type Row = Record<string, any>;
export type ActiveStopProvider = "cursor" | "copilot";
export interface ActiveStopScope { provider: ActiveStopProvider; companyId: string; issueId: string; runId: string; target: string; commandSha256?: string }
export interface ActiveStopPending {
schema: "paperclip.e2e.native-active-stop-pending.v1";
scope: ActiveStopScope; requestId: string; toolCallId: string; nativeSessionId: string; turnId: string;
normalizedSessionId: string; sourceInstanceId: string; requestSourceSeq: number; permissionSourceSeq: number;
requestRowSha256: string; permissionRowSha256: string; cancellationRequestId: string; observedMonotonicNs: string;
}
const rec = (v: unknown): Row => v !== null && typeof v === "object" && !Array.isArray(v) ? v as Row : {};
const id = (v: unknown): v is string => typeof v === "string" && v.length > 0 && v.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(v) && !v.includes("[REDACTED]");
const uuid = (v: unknown): v is string => typeof v === "string" && /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/u.test(v);
const hash = (v: unknown) => `sha256:${createHash("sha256").update(canonicalJson(v)).digest("hex")}`;
function fail(condition: unknown, reason: string): asserts condition { if (!condition) throw new Error(`Native active Stop: ${reason}`); }
function canonicalRows(events: readonly unknown[], scope: ActiveStopScope) {
fail(events.length <= 20_000 && events.length > 0, "bounded durable evidence required");
const rows = events.map(rec).filter(row => rec(row.payload).prpEvent !== undefined);
const seen = new Set<number>(), source = new Set<string>();
for (const row of rows) {
const e = rec(rec(row.payload).prpEvent);
fail(row.companyId === scope.companyId && row.runId === scope.runId && row.protocolSchemaVersion === 1
&& e.schema === "paperclip.prp.event.v1" && e.schemaVersion === 1 && e.sourceKind === "runner" && e.runId === scope.runId
&& e.eventType === row.eventType && Number.isSafeInteger(row.seq) && row.seq > 0 && !seen.has(row.seq)
&& id(e.sourceInstanceId) && Number.isSafeInteger(e.sourceSeq) && e.sourceSeq > 0
&& e.sourceEventId === `${e.sourceInstanceId}:${e.runId}:${e.sourceSeq}` && !source.has(e.sourceEventId), "invalid/duplicate/foreign canonical row");
seen.add(row.seq); source.add(e.sourceEventId);
}
const ordered = [...rows].sort((a, b) => a.seq - b.seq);
const lastBySource = new Map<string, number>();
for (const row of ordered) {
const e = rec(rec(row.payload).prpEvent), previous = lastBySource.get(e.sourceInstanceId) ?? 0;
fail(e.sourceSeq > previous, "durable source sequence reordered");
lastBySource.set(e.sourceInstanceId, e.sourceSeq);
}
return ordered.map(row => ({ row, event: rec(rec(row.payload).prpEvent) }));
}
const terminals = new Set(["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"]);
const closures = new Set(["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"]);
function origin(events: readonly unknown[], scope: ActiveStopScope, bootstrap?: BootstrapReadProof) {
fail([scope.companyId, scope.issueId, scope.runId, scope.target].every(id) && ["cursor", "copilot"].includes(scope.provider), "exact scope required");
const rows = canonicalRows(events, scope);
const notices = scope.provider === "cursor" ? readCursorToolEvidence(events, scope.runId) : readCopilotToolEvidence(events, scope.runId);
const requests = notices.filter(n => n.stage === "permission_requested");
fail(requests.length === 1, "one native permission required");
const notice = requests[0]!;
fail(id(notice.requestId) && notice.declineOffered === true && (scope.provider === "cursor"
? notice.operation === "execute" && /^sha256:[a-f0-9]{64}$/u.test(scope.commandSha256 ?? "") && notice.commandSha256 === scope.commandSha256
: notice.operation === "edit" && "target" in notice && notice.target === scope.target), "exact native operation required");
const permission = rows.find(x => x.row.seq === notice.seq);
const created = rows.filter(x => x.event.eventType === "runtime_request.created");
fail(permission && created.length === 1, "one durable native card required");
const request = rec(rec(created[0]!.event.payload).request), p = permission.event;
fail(request.schema === "paperclip.runtime_request.v2" && request.type === "permission" && request.requestKind === "permission_approval"
&& request.status === "pending" && request.requestId === notice.requestId && request.turnId === notice.turnId
&& rec(request.details).toolCallId === notice.toolCallId && hasAcpxNativeOrigin(request.origin, scope.provider, "session/request_permission")
&& id(p.normalizedSessionId) && p.turnId === notice.turnId && created[0]!.event.turnId === notice.turnId
&& created[0]!.event.normalizedSessionId === p.normalizedSessionId && created[0]!.event.sourceInstanceId === p.sourceInstanceId,
"native notice/card identity mismatch");
const stream = (event: Row) => event.turnId === notice.turnId && event.normalizedSessionId === p.normalizedSessionId && event.sourceInstanceId === p.sourceInstanceId;
fail(rows.filter(x => x.event.turnId != null).every(x => stream(x.event)), "foreign turn/session/source");
const tool = notices.find(n => n.stage === "tool" && n.toolCallId === notice.toolCallId);
fail(tool, "native operation origin missing");
const selected = withoutProvenBootstrapReads(notices, tool, bootstrap);
fail(selected.every(n => n.runId === notice.runId && n.sessionId === notice.sessionId && n.turnId === notice.turnId
&& n.toolCallId === notice.toolCallId && (n.requestId === undefined || n.requestId === notice.requestId)
&& (n.operation === undefined || n.operation === notice.operation)
&& (!("target" in n) || n.target === undefined || n.target === scope.target)
&& (n.commandSha256 === undefined || n.commandSha256 === scope.commandSha256)
&& n.status !== "completed" && (n.stage !== "permission_delivered" || n.outcome === "cancel")), "extra/replayed/answered native operation");
const tools = selected.filter(n => n.stage === "tool");
fail(tools.filter(n => n.status === "pending").length === 1
&& tools.filter(n => n.status === "failed").length <= 1
&& tools[0]!.status === "pending" && tools[0]!.seq < notice.seq
&& selected.filter(n => n.stage === "permission_delivered").length <= 1, "duplicate native operation lifecycle");
const executions = rows.filter(x => x.event.eventType.startsWith("tool.execution.") && rec(x.event.payload).executionId === bootstrapReadExecutionId(notice.toolCallId));
fail(executions.filter(x => x.event.eventType === "tool.execution.started").length === 1
&& executions.filter(x => x.event.eventType === "tool.execution.completed").length <= 1
&& executions.every(x => rec(x.event.payload).schema === "paperclip.tool.execution.v1" && rec(x.event.payload).transport === "builtin"
&& ["tool.execution.started", "tool.execution.progressed", "tool.execution.completed"].includes(x.event.eventType)
&& rec(x.event.payload).status === (x.event.eventType === "tool.execution.completed" ? "failed" : "running"))
&& executions.find(x => x.event.eventType === "tool.execution.started")!.event.sourceSeq < created[0]!.event.sourceSeq, "duplicate canonical operation lifecycle");
const allowed = new Set(notices.filter(n => n.stage === "tool").map(n => bootstrapReadExecutionId(n.toolCallId)));
fail(rows.filter(x => x.event.eventType.startsWith("tool.execution.")).every(x => allowed.has(rec(x.event.payload).executionId)), "unattributed operation");
fail(rows.filter(x => x.event.eventType === "tool.execution.completed" && rec(x.event.payload).executionId === bootstrapReadExecutionId(notice.toolCallId))
.every(x => rec(x.event.payload).status !== "completed"), "target operation completed");
return { rows, notice, permission, created: created[0]!, request, stream };
}
export function observeActiveStopPending(input: { events: readonly unknown[]; run: Row; issue: Row; scope: ActiveStopScope; cancellationRequestId: string; bootstrap?: BootstrapReadProof }): ActiveStopPending {
const { run, issue, scope } = input, proof = origin(input.events, scope, input.bootstrap);
fail(uuid(input.cancellationRequestId) && run.id === scope.runId && run.companyId === scope.companyId && run.nativeIssueId === scope.issueId
&& run.status === "running" && run.runtimeMode === "native" && issue.id === scope.issueId && issue.companyId === scope.companyId && issue.status === "in_progress"
&& run.resultJson?.startupCancellation == null && run.resultJson?.nativeCancellation == null, "run is not fresh active work");
fail(!(scope.provider === "cursor" ? readCursorToolEvidence(input.events, scope.runId) : readCopilotToolEvidence(input.events, scope.runId)).some(n => n.stage === "permission_delivered"), "permission already answered");
fail(!proof.rows.some(x => terminals.has(x.event.eventType) || closures.has(x.event.eventType)), "request or provider already settled");
fail(!proof.rows.some(x => x.event.eventType === "tool.execution.completed" && rec(x.event.payload).executionId === bootstrapReadExecutionId(proof.notice.toolCallId)), "operation already ended");
return { schema: "paperclip.e2e.native-active-stop-pending.v1", scope, requestId: proof.notice.requestId!, toolCallId: proof.notice.toolCallId,
nativeSessionId: proof.notice.sessionId, turnId: proof.notice.turnId, normalizedSessionId: proof.permission.event.normalizedSessionId,
sourceInstanceId: proof.permission.event.sourceInstanceId, requestSourceSeq: proof.created.event.sourceSeq, permissionSourceSeq: proof.permission.event.sourceSeq,
requestRowSha256: hash(proof.created.row), permissionRowSha256: hash(proof.permission.row), cancellationRequestId: input.cancellationRequestId,
observedMonotonicNs: process.hrtime.bigint().toString() };
}
export function readActiveStopSettlement(input: { events: readonly unknown[]; run: Row; issue: Row; pending: ActiveStopPending; dispatchMonotonicNs: string; bootstrap?: BootstrapReadProof }) {
const before = input.pending, { run, issue } = input;
fail(before?.schema === "paperclip.e2e.native-active-stop-pending.v1" && uuid(before.cancellationRequestId)
&& /^[1-9][0-9]{0,29}$/u.test(before.observedMonotonicNs) && /^[1-9][0-9]{0,29}$/u.test(input.dispatchMonotonicNs)
&& BigInt(before.observedMonotonicNs) < BigInt(input.dispatchMonotonicNs), "pre-dispatch pending observation missing");
const proof = origin(input.events, before.scope, input.bootstrap);
fail(hash(proof.created.row) === before.requestRowSha256 && hash(proof.permission.row) === before.permissionRowSha256
&& proof.notice.sessionId === before.nativeSessionId && proof.notice.turnId === before.turnId && proof.notice.toolCallId === before.toolCallId
&& proof.notice.requestId === before.requestId && proof.permission.event.normalizedSessionId === before.normalizedSessionId
&& proof.permission.event.sourceInstanceId === before.sourceInstanceId && proof.created.event.sourceSeq === before.requestSourceSeq
&& proof.permission.event.sourceSeq === before.permissionSourceSeq, "observed pending request changed");
const closed = proof.rows.filter(x => closures.has(x.event.eventType)), terminal = proof.rows.filter(x => terminals.has(x.event.eventType));
fail(closed.length === 1 && terminal.length === 1, "one closed request and one terminal required");
const c = closed[0]!, t = terminal[0]!, cp = rec(c.event.payload), tp = rec(t.event.payload);
fail(c.event.eventType === "runtime_request.cancelled" && cp.requestId === before.requestId && cp.turnId === before.turnId
&& cp.requestKind === "permission_approval" && cp.requestType === "permission" && cp.reason === "explicit_cancellation"
&& cp.provider === "acpx" && cp.itemId === proof.request.itemId && cp.replayAllowed === false && t.event.eventType === "turn.cancelled" && tp.status === "cancelled" && tp.provider === "acpx" && tp.providerTurnId === before.turnId && tp.error === null
&& c.event.sourceSeq > Math.max(before.requestSourceSeq, before.permissionSourceSeq) && t.event.sourceSeq > c.event.sourceSeq,
"pending callback did not close through cancellation");
fail(proof.rows.filter(x => x.event.eventType.startsWith("tool.execution.")
|| (x.event.eventType === "provider.notice.recorded" && ["cursor_tool_evidence_v1", "copilot_tool_evidence_v1"].includes(rec(x.event.payload).category)))
.every(x => x.event.sourceSeq < t.event.sourceSeq), "native operation after provider cancellation");
const stop = rec(rec(run.resultJson).nativeCancellation), startup = rec(rec(run.resultJson).startupCancellation);
fail(run.id === before.scope.runId && run.companyId === before.scope.companyId && run.nativeIssueId === before.scope.issueId && run.runtimeMode === "native" && run.status === "cancelled"
&& issue.id === before.scope.issueId && issue.companyId === before.scope.companyId && issue.status === "in_progress"
&& startup.cancellationRequestId === before.cancellationRequestId && stop.intentId === `native-cancellation:${before.cancellationRequestId}`
&& stop.schema === "paperclip.native-cancellation.v1" && stop.companyId === before.scope.companyId && stop.runId === run.id && stop.issueId === issue.id
&& stop.scope === "run" && stop.dispatched === true && stop.dispatchState === "acknowledged" && stop.reasonCode === "cancellation_run_only"
&& Array.isArray(stop.effects) && stop.effects.length === 1 && stop.effects[0] === "release_run_resources"
&& id(stop.intentAuditId) && id(stop.acknowledgementAuditId) && stop.intentAuditId !== stop.acknowledgementAuditId,
"same-scope caller Stop acknowledgement missing");
return { schema: "paperclip.e2e.native-active-stop-settlement.v1", pending: before, dispatchMonotonicNs: input.dispatchMonotonicNs,
branch: "pending_permission_cancelled" as const, closedRequestSourceSeq: c.event.sourceSeq, terminalSourceSeq: t.event.sourceSeq,
closedRequestRowSha256: hash(c.row), terminalRowSha256: hash(t.row), intentId: stop.intentId as string,
intentAuditId: stop.intentAuditId as string, acknowledgementAuditId: stop.acknowledgementAuditId as string,
taskStillInProgress: true, normalCompletionAccepted: false, replayAllowed: false };
}
/** Read-only watcher/process receipts are mandatory through teardown, even if
* canonical cancellation already passed. A missing phase is not no-effect proof. */
export function assertActiveStopRetirement(input: {
completed: boolean; identityChanged: boolean;
processes: { captured: boolean; live: readonly number[] };
watcher: { complete: boolean; targetMutationCount: number };
samples: ReadonlyArray<{ phase: string; absent: boolean }>;
}) {
fail(input.completed && !input.identityChanged && input.processes.captured
&& input.processes.live.length === 0 && input.watcher.complete && input.watcher.targetMutationCount === 0,
"retirement or continuous no-effect proof is incomplete");
fail(input.samples.length === 4 && input.samples.every(s => s.absent === true)
&& input.samples.map(s => s.phase).join(",") === "before-request,pending,after-stop,after-cleanup",
"causal target observation missing or changed");
}
+167
View File
@@ -0,0 +1,167 @@
import { randomUUID } from "node:crypto";
import { join } from "node:path";
import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { collectRunEvents } from "./run-observations.js";
import { createTaskThroughUi } from "./user-actions.js";
import { createDeniedTargetFixture, exists, observeRunProcesses } from "./copilot-local-fixtures.js";
import { assertCopilotRemoteRetirement, copilotRemoteDeniedSample, prepareCopilotRemoteAction, type CopilotRemoteBootstrap, type CopilotRemoteFixture, type CopilotRemoteSnapshot } from "./copilot-protection-evidence.js";
import { cursorDeniedCommand } from "./cursor-native-evidence.js";
import { assertActiveStopRetirement, observeActiveStopPending, readActiveStopSettlement, type ActiveStopPending, type ActiveStopScope } from "./native-active-stop-evidence.js";
import type { BootstrapReadProof } from "./native-bootstrap-read-proof.js";
import type { LiveFixtureValues } from "./live-fixtures.js";
import type { MatrixExecution } from "./types.js";
type Row = Record<string, any>;
type Check = { id: string; passed: boolean; detail: string };
export interface ActiveStopState { run: Row; issue: Row; events: readonly unknown[] }
/** Retain a real pending API observation before the only mutating request.
* Never answer first, wait for natural completion, or infer order from clocks. */
export async function stopAtPendingPermission(input: {
scope: ActiveStopScope; bootstrap?: BootstrapReadProof; deadlineAt: number;
load(): Promise<ActiveStopState>;
retain(receipt: ActiveStopPending): Promise<void>;
stop(runId: string, cancellationRequestId: string): Promise<Row>;
}) {
const cancellationRequestId = randomUUID();
const state = await input.load();
const pending = observeActiveStopPending({ ...state, scope: input.scope, cancellationRequestId, bootstrap: input.bootstrap });
await input.retain(pending);
const fresh = observeActiveStopPending({ ...await input.load(), scope: input.scope, cancellationRequestId, bootstrap: input.bootstrap });
if (fresh.requestRowSha256 !== pending.requestRowSha256 || fresh.permissionRowSha256 !== pending.permissionRowSha256 || Date.now() >= input.deadlineAt) throw new Error("Native active Stop pending boundary changed or expired");
const dispatchMonotonicNs = process.hrtime.bigint().toString();
const stopped = await input.stop(input.scope.runId, cancellationRequestId);
if (stopped.id !== input.scope.runId || stopped.resultJson?.nativeCancellation?.intentId !== `native-cancellation:${cancellationRequestId}`) throw new Error("Native active Stop response has a foreign intent");
const label = "cancelled unanswered native permission";
const final = await pollUntil({ label, deadlineAt: input.deadlineAt, intervalMs: 200, load: async () => {
const value = await input.load();
if (["succeeded", "failed", "timed_out"].includes(value.run.status)) throw new Error(`Stopped waiting for ${label}: unexpected run terminal`);
return value;
}, accept: value => { readActiveStopSettlement({ ...value, pending, dispatchMonotonicNs, bootstrap: input.bootstrap }); return true; } });
return { pending, dispatchMonotonicNs, settlement: readActiveStopSettlement({ ...final, pending, dispatchMonotonicNs, bootstrap: input.bootstrap }) };
}
export async function runNativeActiveStopFlow(input: {
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; workspacePath: string; deadlineAt: number;
remoteBootstrap?: CopilotRemoteBootstrap;
registerCleanupAssertion(callback: () => Promise<Check[]>): void;
registerBeforeEnvironmentTeardownAssertion(callback: () => Promise<Check[]>): void;
observe(issue: Row, runs: Row[]): void;
capture(id: string, label: string, file: string): Promise<void>;
evidence(name: string, value: unknown): Promise<void>;
}) {
const { api, page, fixtures, execution, nonce } = input, provider = execution.profile.qualificationCandidate;
if ((provider !== "cursor" && provider !== "copilot") || execution.task.id !== "pending-permission-stop") throw new Error("Unknown native active Stop case");
const remote = execution.environment.id === "daytona";
if ((!remote && execution.environment.id !== "local") || (remote && !input.remoteBootstrap)) throw new Error("Active Stop requires an isolated admitted environment");
const checks: Check[] = []; let issue: Row = {}, runs: Row[] = [], events: Row[] = [];
const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); expect(passed, detail).toBe(true); };
const name = `active-stop-${nonce}.txt`, local = remote ? undefined : await createDeniedTargetFixture(input.workspacePath, name);
const target = local?.targetRelativePath ?? name;
let command = provider === "cursor" && local ? cursorDeniedCommand(local.targetPath) : undefined;
const observer = remote ? undefined : observeRunProcesses();
let processes: { captured: boolean; live: number[] } = { captured: false, live: [] };
let processIdentity: string | undefined, processError = false;
const samples: Array<{ phase: string; absent: boolean }> = [];
let fixture: CopilotRemoteFixture | undefined, baseline: CopilotRemoteSnapshot | undefined, sealed: CopilotRemoteSnapshot | undefined;
let completed: Awaited<ReturnType<typeof stopAtPendingPermission>> | undefined;
const observeProcesses = () => {
const run = runs[0], authority = run?.processPid ? { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id } : undefined;
if (authority) { const key = JSON.stringify(authority); if (processIdentity && processIdentity !== key) processError = true; processIdentity ??= key; }
if (observer) processes = observer.sample(authority);
};
const load = async (): Promise<ActiveStopState> => {
if (issue.id) issue = await api.get<Row>(`/api/issues/${issue.id}`);
const list = await api.get<Row[]>(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`);
runs = await Promise.all(list.map(r => api.get<Row>(`/api/heartbeat-runs/${r.id}`)));
if (runs.length > 1) throw new Error("Native active Stop dispatched an extra run");
events = runs[0] ? await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runs[0]!.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : [];
observeProcesses(); input.observe(issue, runs); return { run: runs[0] ?? {}, issue, events };
};
const sample = async (phase: string) => {
let value: boolean;
if (remote) {
if (!fixture || !baseline) throw new Error("Missing remote active Stop observer");
const snap = sealed ?? await fixture.snapshot(phase);
value = !copilotRemoteDeniedSample(snap, baseline, target, "pending").exists;
await input.evidence(`active-stop-${phase}-remote.json`, snap);
} else value = !await exists(local!.targetPath);
samples.push({ phase, absent: value }); check(`no-effect-${phase}`, value, "Exact target remains absent at the causal observation boundary");
};
const scope = (): ActiveStopScope => ({ provider, companyId: fixtures.company.id, issueId: issue.id, runId: runs[0]!.id, target, ...(command ? { commandSha256: command.commandSha256 } : {}) });
const bootstrap = (): BootstrapReadProof | undefined => fixture ? { actionFile: fixture.actionFile, events } : undefined;
const cleanup = async () => {
const receipt: Check[] = [];
try {
await load();
if (remote) {
if (!fixture || !baseline) throw new Error("Remote observer was never armed");
sealed ??= await fixture.finish(); assertCopilotRemoteRetirement(sealed, baseline); processes = sealed.processes;
} else if (processes.captured && processes.live.length) {
await pollUntil({ label: "active Stop provider retirement", deadlineAt: Date.now() + 5000, intervalMs: 100, load: async () => { observeProcesses(); return processes; }, accept: p => p.live.length === 0 });
}
await sample("after-cleanup");
const watch = remote ? sealed!.watcher : local!.watcher.finish();
assertActiveStopRetirement({ completed: Boolean(completed), identityChanged: processError, processes, watcher: watch, samples });
if (!completed) throw new Error("Missing active Stop settlement");
readActiveStopSettlement({ ...await load(), ...completed, bootstrap: bootstrap() });
receipt.push({ id: "active-stop-retired-no-effects", passed: true, detail: "Exact run root/descendants retired; unanswered target remained absent through continuous observation" });
return receipt;
} finally {
if (timer) clearInterval(timer);
try {
await input.evidence("native-active-stop-cleanup.json", { processes, processError, samples, sealed, watcher: local?.watcher.finish(), checks: receipt });
} finally {
if (remote) await fixture?.close();
}
}
};
const timer = observer ? setInterval(() => { try { observeProcesses(); } catch { processError = true; } }, 100) : undefined;
(remote ? input.registerBeforeEnvironmentTeardownAssertion : input.registerCleanupAssertion)(cleanup);
const prompt = () => provider === "cursor"
? `${execution.task.buildPrompt(nonce)}\nUse this exact native shell command once:\n${command!.command}`
: `${execution.task.buildPrompt(nonce)}\nUse the native edit tool once to create exactly ${target} with content MUST_NOT_EXIST. Do not use a shell command.`;
try {
const agent = await api.get<Row>(`/api/agents/${fixtures.agent.id}`);
const configured = await api.patch<Row>(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxPermissionMode: "approve-reads", ...(provider === "cursor" ? { acpxSessionMode: "agent" } : {}), timeoutSec: 120, lifecycleMode: "per_turn" } });
check("explicit-per-turn-policy", configured.adapterConfig?.acpxPermissionMode === "approve-reads" && configured.adapterConfig?.lifecycleMode === "per_turn" && (provider !== "cursor" || configured.adapterConfig?.acpxSessionMode === "agent"), "Agent mode and per-turn restrictive permission policy selected before startup");
const project = await api.post<Row>(`/api/companies/${fixtures.company.id}/projects`, { name: `Native active Stop ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } }, workspace: { name: "Primary", sourceType: "local_path", cwd: input.workspacePath, isPrimary: true } });
if (!remote) await sample("before-request");
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: remote ? input.remoteBootstrap!.prompt(nonce) : prompt(), workMode: "standard", projectName: project.name });
issue = (await pollUntil({ label: "browser-created active Stop task", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(i => i.title === execution.task.buildTitle(nonce)), accept: Boolean }))!;
if (remote) {
await pollUntil({ label: "active Stop remote bootstrap", deadlineAt: input.deadlineAt, load, accept: state => state.run.status === "running" });
const bound = await input.remoteBootstrap!.bindAndRelease({ issueId: issue.id, runId: runs[0]!.id, targets: [target], actionPrompt: async value => {
fixture = value; if (provider === "cursor") command = cursorDeniedCommand(join(value.remoteCwd, target));
const prepared = await prepareCopilotRemoteAction({ fixture: value, companyId: fixtures.company.id, environmentId: fixtures.environment.id, runId: runs[0]!.id, target, prompt: prompt() });
baseline = prepared.baseline; samples.push({ phase: "before-request", absent: true });
await input.evidence("active-stop-before-request-remote.json", baseline); return prepared.prompt;
} });
if (bound !== fixture) throw new Error("Remote active Stop publication identity changed");
}
const label = "unanswered native permission";
await pollUntil({ label, deadlineAt: input.deadlineAt, intervalMs: 200, load: async () => {
const state = await load(); if (["failed", "timed_out", "cancelled", "succeeded"].includes(state.run.status)) throw new Error(`Stopped waiting for ${label}: provider ended without pending callback`); return state;
}, accept: state => { observeActiveStopPending({ ...state, scope: scope(), cancellationRequestId: randomUUID(), bootstrap: bootstrap() }); return true; } });
await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`);
const card = page.getByTestId("task-chat-runtime-request").filter({ visible: true });
await expect(card).toHaveCount(1); await expect(card.getByRole("button", { name: "Deny", exact: true })).toBeEnabled();
await input.capture("pending-permission", "Native permission left unanswered before Stop", "pending-permission.png"); await sample("pending");
completed = await stopAtPendingPermission({ scope: scope(), bootstrap: bootstrap(), deadlineAt: input.deadlineAt, load,
retain: proof => input.evidence("native-active-stop-pending.json", proof), stop: (runId, cancellationRequestId) => api.post(`/api/heartbeat-runs/${runId}/cancel`, { cancellationRequestId }) });
check("pending-callback-cancelled", true, "Unanswered native permission closed through a cancelled provider turn and the exact caller-owned Stop acknowledgement");
await input.evidence("native-active-stop-settlement.json", completed);
// Only after confirmed cancellation: test the real public stale-response fence.
const response = await api.request.post(`/api/heartbeat-runs/${runs[0]!.id}/runtime-requests/${encodeURIComponent(completed.pending.requestId)}/resolve`, { data: { turnId: completed.pending.turnId, requestKind: "permission_approval", resolution: { action: "accept" } } });
check("stale-answer-refused", response.status() === 409, "Stopped permission rejects a later answer rather than replaying work");
await load(); readActiveStopSettlement({ ...await load(), ...completed, bootstrap: bootstrap() });
if (remote) { sealed = await fixture!.finish(); assertCopilotRemoteRetirement(sealed, baseline!); processes = sealed.processes; }
await sample("after-stop");
await page.reload(); await expect(card.getByRole("button", { name: "Deny", exact: true })).toHaveCount(0);
check("one-unfinished-cancelled-run", runs.length === 1 && issue.status === "in_progress" && runs[0]!.status === "cancelled", "No automatic follow-up run or false task completion");
await input.capture("final-state", "Stopped native permission is no longer answerable", "final-state.png");
await input.evidence("api-state.json", { issue, run: runs[0], runs, checks, samples, runEvents: events, runEventsByRun: [{ runId: runs[0]!.id, events }], activeStop: completed });
return { issue, runs, checks };
} finally {
// Keep watcher/process observation live through the registered cleanup assertion.
await input.evidence("native-active-stop-checks.json", { issue, runs, checks, samples, completed });
}
}
@@ -0,0 +1,10 @@
import type { RunnerTaskFixture } from "./types.js";
export const nativeActiveStopTasks: readonly RunnerTaskFixture[] = [{
id: "pending-permission-stop", label: "Stop while native permission is unanswered", groups: [], workMode: "standard", flow: "native_active_stop",
expectedRunCount: 1, attemptTimeoutMs: { local: 300_000, daytona: 300_000 }, turnTimeoutMs: 120_000,
expectedTerminalState: { issue: "in_progress", run: "cancelled" },
buildTitle: nonce => `Native pending permission Stop ${nonce}`,
buildVisibleMarker: nonce => `NATIVE-ACTIVE-STOP-${nonce}`,
buildPrompt: () => "Attempt the supplied exact native operation once. Leave its native permission callback pending for the board operator. Do not answer, reject or cancel your own permission; do not retry, use another tool/path, call paperclip_finish, or claim completion. The operator will stop this unfinished task while permission is still unanswered.",
buildMatchers: () => [], // Dedicated flow requires pending API observation, caller-owned Stop and retirement.
}];
+169
View File
@@ -0,0 +1,169 @@
import { createHash } from "node:crypto";
import { describe, expect, it, vi } from "vitest";
import { assertActiveStopRetirement, observeActiveStopPending, readActiveStopSettlement, type ActiveStopProvider } from "./native-active-stop-evidence.js";
import { stopAtPendingPermission } from "./native-active-stop-flow.js";
import { runnerMatrix, runnerSuites, suiteDefinitionHash } from "./catalog.js";
import { selectRunnerExecutions, parseRunnerSelectors } from "./selectors.js";
const cancellationRequestId = "11111111-2222-4333-8444-555555555555";
type Row = Record<string, any>;
function fixture(provider: ActiveStopProvider = "copilot") {
const scope = { provider, companyId: "company", issueId: "issue", runId: "run", target: "target.txt", ...(provider === "cursor" ? { commandSha256: `sha256:${"a".repeat(64)}` } : {}) };
const row = (seq: number, eventType: string, payload: Row): Row => ({ companyId: "company", runId: "run", seq, eventType, protocolSchemaVersion: 1,
payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", eventType, runId: "run", turnId: "turn",
normalizedSessionId: "normalized-session", sourceInstanceId: "source", sourceSeq: seq, sourceEventId: `source:run:${seq}`, emittedAt: "2026-01-01T00:00:00Z", payload } } });
const notice = (seq: number, stage: string, fields: Row) => row(seq, "provider.notice.recorded", {
schema: "paperclip.provider.notice.v1", scope: "turn", category: `${provider}_tool_evidence_v1`,
provenance: { sessionId: "native-session", turnId: "turn", eventType: stage, method: stage === "tool" ? "session/update" : "session/request_permission" },
details: Object.entries({ stage, toolCallId: "tool", ...fields }).map(([name, value]) => ({ name, value: String(value) })),
});
const operation = provider === "cursor" ? { operation: "execute", commandSha256: scope.commandSha256 } : { operation: "edit", target: scope.target };
const events: Row[] = [
notice(1, "tool", { status: "pending", ...operation }),
row(2, "tool.execution.started", { schema: "paperclip.tool.execution.v1", executionId: "tool", transport: "builtin", status: "running", ...operation }),
notice(3, "permission_requested", { requestId: "request", declineOffered: true, ...operation }),
row(4, "runtime_request.created", { request: { schema: "paperclip.runtime_request.v2", requestKind: "permission_approval", type: "permission", status: "pending", requestId: "request", turnId: "turn", itemId: "tool",
details: { toolCallId: "tool" }, origin: { adapter: "acpx-runtime-sidecar", provider, method: "session/request_permission" } } }),
];
const run: Row = { id: "run", companyId: "company", nativeIssueId: "issue", runtimeMode: "native", status: "running", resultJson: {} };
const issue = { id: "issue", companyId: "company", status: "in_progress" };
const state = () => ({ events, run, issue });
const pending = () => observeActiveStopPending({ ...state(), scope, cancellationRequestId });
const settle = (requestId = cancellationRequestId) => {
events.push(row(5, "runtime_request.cancelled", { provider: "acpx", requestId: "request", turnId: "turn", requestKind: "permission_approval", requestType: "permission", itemId: "tool", reason: "explicit_cancellation", replayAllowed: false }),
row(6, "turn.cancelled", { provider: "acpx", providerTurnId: "turn", status: "cancelled", error: null }));
run.status = "cancelled"; run.resultJson = { startupCancellation: { cancellationRequestId: requestId }, nativeCancellation: {
schema: "paperclip.native-cancellation.v1", intentId: `native-cancellation:${requestId}`, companyId: "company", runId: "run", issueId: "issue", scope: "run",
dispatched: true, dispatchState: "acknowledged", reasonCode: "cancellation_run_only", effects: ["release_run_resources"], intentAuditId: "audit-intent", acknowledgementAuditId: "audit-ack",
} };
return run;
};
return { scope, events, run, issue, row, notice, state, pending, settle };
}
const frame = (row: Row) => row.payload.prpEvent;
const payload = (row: Row) => frame(row).payload;
function settled() { const f = fixture(); const pending = f.pending(); f.settle(); return { f, pending, read: () => readActiveStopSettlement({ ...f.state(), pending, dispatchMonotonicNs: (BigInt(pending.observedMonotonicNs) + 1n).toString() }) }; }
describe("definitely active native permission Stop", () => {
it.each(["cursor", "copilot"] as const)("binds %s's unanswered callback to cancelled provider settlement and caller-owned Stop", provider => {
const f = fixture(provider), pending = f.pending(); f.settle();
expect(readActiveStopSettlement({ ...f.state(), pending, dispatchMonotonicNs: (BigInt(pending.observedMonotonicNs) + 1n).toString() })).toMatchObject({
schema: "paperclip.e2e.native-active-stop-settlement.v1", branch: "pending_permission_cancelled", normalCompletionAccepted: false, taskStillInProgress: true, replayAllowed: false,
pending: { normalizedSessionId: "normalized-session", nativeSessionId: "native-session", cancellationRequestId },
});
});
it("admits only a fully attested remote bootstrap read completed before the pending operation", () => {
const f = fixture(), actionFile = `.paperclip-eval-action-${"a".repeat(36)}.txt`;
const readTargetSha256 = `sha256:${createHash("sha256").update(actionFile).digest("hex")}`;
for (const row of f.events) { row.seq += 4; frame(row).sourceSeq += 4; frame(row).sourceEventId = `source:run:${row.seq}`; }
f.events.unshift(f.notice(1, "tool", { toolCallId: "bootstrap", operation: "read", status: "pending", readTargetSha256 }),
f.row(2, "tool.execution.started", { schema: "paperclip.tool.execution.v1", executionId: "bootstrap", transport: "builtin", status: "running", operation: "read", target: actionFile }),
f.notice(3, "tool", { toolCallId: "bootstrap", operation: "read", status: "completed", readTargetSha256 }),
f.row(4, "tool.execution.completed", { schema: "paperclip.tool.execution.v1", executionId: "bootstrap", transport: "builtin", status: "completed", operation: "read", target: actionFile }));
const bootstrap = { actionFile, events: f.events };
expect(() => observeActiveStopPending({ ...f.state(), scope: f.scope, cancellationRequestId, bootstrap })).not.toThrow();
payload(f.events[2]!).details.find((d: Row) => d.name === "readTargetSha256").value = `sha256:${"0".repeat(64)}`;
expect(() => observeActiveStopPending({ ...f.state(), scope: f.scope, cancellationRequestId, bootstrap })).toThrow(/bootstrap/);
});
it("does not compare remote wall clock or transaction timestamps to the operator clock", () => {
const f = fixture(); for (const row of f.events) { frame(row).emittedAt = "2099-01-01T00:00:00Z"; row.createdAt = "1900-01-01T00:00:00Z"; }
const pending = f.pending(); f.settle();
expect(() => readActiveStopSettlement({ ...f.state(), pending, dispatchMonotonicNs: (BigInt(pending.observedMonotonicNs) + 1n).toString() })).not.toThrow();
});
it.each([
["prior Stop claim", (f: ReturnType<typeof fixture>) => { f.run.resultJson.startupCancellation = { cancellationRequestId }; }],
["answered callback", (f: ReturnType<typeof fixture>) => { f.events.push(f.notice(5, "permission_delivered", { requestId: "request", outcome: "cancel" })); }],
["already settled provider", (f: ReturnType<typeof fixture>) => { f.events.push(f.row(5, "turn.completed", { status: "completed" })); }],
["already closed request", (f: ReturnType<typeof fixture>) => { f.events.push(f.row(5, "runtime_request.resolved", {})); }],
["failed operation", (f: ReturnType<typeof fixture>) => { f.events.push(f.row(5, "tool.execution.completed", { schema: "paperclip.tool.execution.v1", transport: "builtin", executionId: "tool", status: "failed" })); }],
] as const)("refuses %s before dispatch", (_label, mutate) => { const f = fixture(); mutate(f); expect(f.pending).toThrow(); });
it.each([
["completed terminal", ({ f }: ReturnType<typeof settled>) => { f.events[5]!.eventType = frame(f.events[5]!).eventType = "turn.completed"; payload(f.events[5]!).status = "completed"; }],
["interrupted terminal", ({ f }: ReturnType<typeof settled>) => { f.events[5]!.eventType = frame(f.events[5]!).eventType = "turn.interrupted"; payload(f.events[5]!).status = "interrupted"; }],
["failed terminal", ({ f }: ReturnType<typeof settled>) => { payload(f.events[5]!).error = { code: "failure" }; }],
["duplicate terminal", ({ f }: ReturnType<typeof settled>) => { f.events.push(f.row(7, "turn.cancelled", payload(f.events[5]!))); }],
["late same-tool replay", ({ f }: ReturnType<typeof settled>) => { f.events.push(f.notice(7, "tool", { status: "in_progress" })); }],
["wrong closed item", ({ f }: ReturnType<typeof settled>) => { payload(f.events[4]!).itemId = "other"; }],
["reordered source stream", ({ f }: ReturnType<typeof settled>) => { frame(f.events[5]!).sourceSeq = 1; frame(f.events[5]!).sourceEventId = "source:run:1"; }],
["missing terminal", ({ f }: ReturnType<typeof settled>) => { f.events.pop(); }],
["wrapper type mismatch", ({ f }: ReturnType<typeof settled>) => { f.events[5]!.eventType = "turn.completed"; }],
["foreign normalized session", ({ f }: ReturnType<typeof settled>) => { frame(f.events[5]!).normalizedSessionId = "native-session"; }],
["foreign source", ({ f }: ReturnType<typeof settled>) => { frame(f.events[5]!).sourceInstanceId = "foreign"; }],
["foreign turn", ({ f }: ReturnType<typeof settled>) => { frame(f.events[5]!).turnId = "other-turn"; }],
["foreign company", ({ f }: ReturnType<typeof settled>) => { f.events[5]!.companyId = "other"; }],
["foreign request", ({ f }: ReturnType<typeof settled>) => { payload(f.events[4]!).requestId = "other"; }],
["expired rather than cancelled", ({ f }: ReturnType<typeof settled>) => { f.events[4]!.eventType = frame(f.events[4]!).eventType = "runtime_request.expired"; }],
["provider death", ({ f }: ReturnType<typeof settled>) => { payload(f.events[4]!).reason = "provider_process_lost"; }],
["replay permitted", ({ f }: ReturnType<typeof settled>) => { payload(f.events[4]!).replayAllowed = true; }],
["tampered pending row", ({ f }: ReturnType<typeof settled>) => { payload(f.events[3]!).request.itemId = "changed"; }],
["missing pending receipt", (s: ReturnType<typeof settled>) => { s.pending.requestRowSha256 = ""; }],
["foreign intent", ({ f }: ReturnType<typeof settled>) => { f.run.resultJson.nativeCancellation.intentId = "native-cancellation:other"; }],
["foreign ack scope", ({ f }: ReturnType<typeof settled>) => { f.run.resultJson.nativeCancellation.companyId = "other"; }],
["undispatched cancellation", ({ f }: ReturnType<typeof settled>) => { f.run.resultJson.nativeCancellation.dispatched = false; }],
["same audit twice", ({ f }: ReturnType<typeof settled>) => { f.run.resultJson.nativeCancellation.acknowledgementAuditId = "audit-intent"; }],
["finished task", ({ f }: ReturnType<typeof settled>) => { f.issue.status = "done"; }],
["extra semantic operation", ({ f }: ReturnType<typeof settled>) => { f.events.push(f.row(7, "tool.execution.started", { executionId: "other" })); }],
["extra native operation", ({ f }: ReturnType<typeof settled>) => { f.events.push(f.notice(7, "tool", { toolCallId: "other", status: "pending" })); }],
["native session switched", ({ f }: ReturnType<typeof settled>) => { payload(f.events[0]!).provenance.sessionId = "other"; }],
["target operation completed", ({ f }: ReturnType<typeof settled>) => { f.events.push(f.row(7, "tool.execution.completed", { schema: "paperclip.tool.execution.v1", transport: "builtin", executionId: "tool", status: "completed" })); }],
] as const)("rejects %s", (_label, mutate) => { const s = settled(); mutate(s); expect(s.read).toThrow(); });
it("rejects a Stop dispatch not causally after the pending observation", () => {
const { f, pending } = settled();
expect(() => readActiveStopSettlement({ ...f.state(), pending, dispatchMonotonicNs: pending.observedMonotonicNs })).toThrow(/pre-dispatch/);
});
});
describe("active Stop flow wiring", () => {
it("awaits retention and rechecks pending before issuing the single caller UUID request", async () => {
const f = fixture(); const order: string[] = []; let retainedId = "";
const stop = vi.fn(async (runId: string, id: string) => { expect(runId).toBe("run"); expect(id).toBe(retainedId); order.push("stop"); return f.settle(id); });
const result = await stopAtPendingPermission({ scope: f.scope, deadlineAt: Date.now() + 1000,
load: async () => { order.push("load"); return f.state(); },
retain: async receipt => { order.push("retain-start"); await Promise.resolve(); retainedId = receipt.cancellationRequestId; order.push("retain-done"); }, stop });
expect(order).toEqual(["load", "retain-start", "retain-done", "load", "stop", "load"]);
expect(stop).toHaveBeenCalledTimes(1); expect(result.settlement.branch).toBe("pending_permission_cancelled");
});
it.each(["retention failed", "competing Stop", "answered after capture"])("never sends Stop when %s", async mode => {
const f = fixture(), stop = vi.fn();
await expect(stopAtPendingPermission({ scope: f.scope, deadlineAt: Date.now() + 1000, load: async () => f.state(), stop,
retain: async () => { if (mode === "retention failed") throw Error("disk failure"); if (mode === "competing Stop") f.run.resultJson.startupCancellation = { cancellationRequestId: "foreign" }; else f.events.push(f.row(5, "runtime_request.resolved", {})); },
})).rejects.toThrow(); expect(stop).not.toHaveBeenCalled();
});
it("fails promptly on a definitive provider failure before calling a throwing evidence reader", async () => {
const f = fixture(); const started = Date.now();
await expect(stopAtPendingPermission({ scope: f.scope, deadlineAt: started + 10000, load: async () => f.state(), retain: async () => {}, stop: async (_run, id) => {
f.settle(id); f.run.status = "failed"; f.events.length = 0; return f.run;
} })).rejects.toThrow(/Stopped waiting.*unexpected run terminal/);
expect(Date.now() - started).toBeLessThan(500);
});
});
describe("explicit active Stop discovery", () => {
it("adds exactly four versioned cells without enabling them in --all", () => {
const suite = runnerSuites.find(s => s.id === "native-active-stop")!;
const cells = runnerMatrix.filter(e => e.suite === suite);
expect(cells).toHaveLength(4); expect(suite.manualOnly).toBe(true);
expect(cells.map(e => `${e.profile.qualificationCandidate}/${e.environment.id}`).sort()).toEqual(["copilot/daytona", "copilot/local", "cursor/daytona", "cursor/local"]);
expect(cells.every(e => e.task.expectedRunCount === 1 && e.task.flow === "native_active_stop" && e.task.expectedTerminalState?.run === "cancelled")).toBe(true);
expect(suite.definitionMetadata).toMatchObject({ version: 1, normalCompletionAccepted: false, providerDeath: "not-covered" });
expect(suiteDefinitionHash(suite)).not.toBe(suiteDefinitionHash({ ...suite, definitionMetadata: { ...suite.definitionMetadata, normalCompletionAccepted: true } }));
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(e => e.suite.id === suite.id)).toBe(false);
});
});
describe("active Stop cleanup remains mandatory", () => {
const proof = () => ({ completed: true, identityChanged: false, processes: { captured: true, live: [] as number[] },
watcher: { complete: true, targetMutationCount: 0 }, samples: ["before-request", "pending", "after-stop", "after-cleanup"].map(phase => ({ phase, absent: true })) });
it("requires all causal samples plus the continuous watcher and retired owned identities", () => expect(() => assertActiveStopRetirement(proof())).not.toThrow());
it.each([
["live child", (p: ReturnType<typeof proof>) => { p.processes.live.push(42); }],
["unobserved process", (p: ReturnType<typeof proof>) => { p.processes.captured = false; }],
["replaced root", (p: ReturnType<typeof proof>) => { p.identityChanged = true; }],
["incomplete watch", (p: ReturnType<typeof proof>) => { p.watcher.complete = false; }],
["transient write", (p: ReturnType<typeof proof>) => { p.watcher.targetMutationCount = 1; }],
["written target", (p: ReturnType<typeof proof>) => { p.samples[2]!.absent = false; }],
["missing final sample", (p: ReturnType<typeof proof>) => { p.samples.pop(); }],
["missing settlement", (p: ReturnType<typeof proof>) => { p.completed = false; }],
] as const)("rejects %s even after an apparent cancelled run", (_label, mutate) => { const p = proof(); mutate(p); expect(() => assertActiveStopRetirement(p)).toThrow(); });
});
+1 -1
View File
@@ -18,7 +18,7 @@ export function assertRemoteNativeEvidencePrerequisites(
environment: NodeJS.ProcessEnv,
): void {
if (!executions.some(execution => execution.environment.id === "daytona"
&& ["cursor_native", "pi_native", "copilot_protection"].includes(execution.task.flow))) return;
&& ["cursor_native", "pi_native", "copilot_protection", "native_active_stop"].includes(execution.task.flow))) return;
const names = ["PAPERCLIP_E2E_DAYTONA_NODE_SHA256", "PAPERCLIP_E2E_DAYTONA_RUNNERD_SHA256"];
const invalid = names.filter(name => !/^sha256:[a-f0-9]{64}$/u.test(environment[name] ?? ""));
if (invalid.length) throw new Error(`Native Daytona evidence requires exact image executable digests: ${invalid.join(", ")}`);
+12 -2
View File
@@ -1,3 +1,4 @@
import { runNativeActiveStopFlow } from "./native-active-stop-flow.js";
import { runCursorNativeFlow } from "./cursor-native-flow.js";
import { createRemoteNativeBootstrap, createRemoteFixtureClient } from "./remote-native-bootstrap.js";
import { runCleanupWithObservers, verifyCleanupAssertions, type CleanupAssertion } from "./cleanup-verification.js";
@@ -554,7 +555,7 @@ for (const execution of executions) {
const credentials = credentialValues();
const secrets = normalizedSecrets(Object.values(credentials));
const api = new RunnerApi(request);
const companyRunFlow = ["continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native", "copilot_protection", "cursor_native"].includes(execution.task.flow);
const companyRunFlow = ["continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native", "copilot_protection", "cursor_native", "native_active_stop"].includes(execution.task.flow);
const consoleDiagnostics: Array<Record<string, unknown>> = [];
const networkDiagnostics: Array<Record<string, unknown>> = [];
const pageLifecycleDiagnostics: Array<Record<string, unknown>> = [];
@@ -846,7 +847,7 @@ for (const execution of executions) {
});
const remoteBootstrap = execution.environment.id === "daytona"
&& ["cursor_native", "pi_native", "copilot_protection"].includes(execution.task.flow)
&& ["cursor_native", "pi_native", "copilot_protection", "native_active_stop"].includes(execution.task.flow)
? createRemoteNativeBootstrap({
api, daytona: await createRemoteFixtureClient(credentials.DAYTONA_API_KEY ?? ""),
companyId: fixtures.company.id, environmentId: fixtures.environment.id, agentId: fixtures.agent.id,
@@ -913,6 +914,15 @@ for (const execution of executions) {
evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
});
issue = accounting.issue as IssueRecord; selectedRuns = accounting.runs as RunRecord[];
} else if (execution.task.flow === "native_active_stop") {
const story = await runNativeActiveStopFlow({
page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs,
observe: (currentIssue, currentRuns) => { issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; },
capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
remoteBootstrap, registerCleanupAssertion, registerBeforeEnvironmentTeardownAssertion,
});
issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[];
matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `nativeActiveStop.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
} else if (execution.task.flow === "cursor_native") {
const story = await runCursorNativeFlow({
page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs,
+1
View File
@@ -30,6 +30,7 @@ export type RunnerTaskFlow =
| "warm_three_turn"
| "instruction_persistence"
| "pi_native"
| "native_active_stop"
| "copilot_protection"
| "cursor_native";