From 76254ead22294cc70b5f27d19934fbe42374137b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 9 Nov 2026 12:11:47 -0600 Subject: [PATCH] fix(runner): allow Codex interruption to settle governed waits (#20330) ## Thinking Path > - Paperclip manages AI agents and their work. > - A native run must stop provider work before it waits for a human response. > - The Codex transport allows 30 seconds for a control command. > - The governed-wait loop used its 15-second default for Codex. > - A valid interruption terminal at 18 seconds therefore failed the run. > - This change gives Codex the existing 30-second terminal drain allowance. > - The run still fails if it cannot prove that the provider turn ended. ## Linked Issues or Issue Description **What happened?** A Codex run can save a human wait, accept an interruption, and fail before the provider sends its terminal. Its saved wait remains intact, but normal finalization does not occur. **Expected behavior:** Allow terminal delivery through the transport's bounded command window. Keep missing-terminal cases closed. **Steps to reproduce:** Run the new protocol regression against the old backend. The real Runner binary uses a controlled Codex process that delays its interruption terminal for 18 seconds. The run fails at the 15-second drain deadline. The same process passes with this change. **Paperclip version or commit:** Reproduced against `1b7fe33432`. **Deployment mode:** Local test of the native Runner path. The incident was observed in a hosted sandbox. **Installation method:** Source checkout. **Operating system:** macOS arm64 for local verification. Hosted incident runtime was Linux. **Additional context:** No matching repair was found. This addresses a reproducible deadline mismatch. The hosted log ends terminal observation at 15 seconds, so it cannot prove whether that provider would have settled within 30 seconds. The approved local live-provider check passed. This PR does not claim recovery of the hosted task. ## What Changed - Give Codex a 30-second governed-wait terminal drain through the existing backend option. - Keep other providers and the caller's shorter execution bound unchanged. - Add real Runner protocol cases for prompt, delayed, and absent terminal delivery. Check retained wait state and one provider turn. - Document the bounded wait and terminal-proof requirement. ## Verification - Red: the 18-second controlled-provider case failed with `governed_wait_provider_terminal_unproven` on the old backend. The prompt case passed. - Green: 282 tests passed across the native runtime, harness backend, and new Runner protocol suite, including a repeat on the rebased PR revision. `pnpm verify --explain` returned exit 2 because the execution-contract document requires broad CI; the [required CI run](https://github.com/paperclipai/paperclip/actions/runs/42668235142) passed on the final head. All 58 applicable checks passed; the three skipped checks were the two unrelated Storybook jobs and the selected-test planner replaced by broad coverage. - The missing-terminal case still fails without `completeRun` or a successful run terminal. The saved governed wait remains available. Only one provider turn and one tool call occur. - Existing runtime tests cover shorter deadlines, foreign or unproven terminals, complete usage without a terminal, and checkpoint recovery without another model turn. - Live acceptance at `f3da2b87425d4fd45c649bfc98785c61f82abbe0`: real Runner binary, Codex CLI 0.165.0, gpt-6.1-sol, Node 26.4.0, macOS arm64, disposable local workspace. One `request_human_input` tool call returned a pending interaction fixture. The real provider emitted `turn.interrupted`; the controller accepted the yielded wait and its run terminal. The check passed in 8.14 seconds. The tool callback and persistence port were controlled; no hosted server permission or Notebook publication claim is made. - Initial local probe setup failed before model execution: first the explicit launch lacked `app-server`, then it omitted the Runner permission configuration. The successful check used the Runner's normal launch arguments and isolated home with the approved existing account. - Hosted sandbox acceptance and recovery remain unverified. The 18-second delayed shutdown is a controlled protocol regression; the live model check proves normal shutdown, not that specific latency. ## Risks - A Codex provider that never stops can occupy the settlement window for up to 15 additional seconds. It still fails closed and is quarantined. - No persisted schema, provider identity, permission, or recovery rule changes. Previous-release checkpoints use the same format and existing recovery checks. - This change cannot fix a provider that never emits a terminal. It also does not grant missing setup permissions or decide pending approvals. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, tool calls, code edits, and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/execution-semantics.md | 3 + .../src/backends/codex-native-backend.ts | 5 +- .../src/live/runnerd-governed-wait.test.ts | 113 ++++++++++++++++++ 3 files changed, 120 insertions(+), 1 deletion(-) create mode 100644 packages/paperclip-runner/src/live/runnerd-governed-wait.test.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 27a3999f5e0..defd7e6b3e2 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -4081,6 +4081,9 @@ are retained during shutdown so recovery can replay a contiguous journal. They carry no permission to start work. Stream closure, stream failure or drain timeout after complete usage leaves the saved wait unfinalized until a provider terminal is retained; complete token counters alone cannot certify a stopped provider. +Codex governed-wait settlement allows up to 30 seconds for its interruption +terminal, matching the transport command window. The caller's execution deadline +can shorten that window. An interrupt acknowledgement alone is not terminal proof. A timeout, missing accounting, or an unproven provider terminal remains a failure; the saved wait does not certify successful provider execution. diff --git a/packages/paperclip-runner/src/backends/codex-native-backend.ts b/packages/paperclip-runner/src/backends/codex-native-backend.ts index 27188fb6f0f..630c14f07c3 100644 --- a/packages/paperclip-runner/src/backends/codex-native-backend.ts +++ b/packages/paperclip-runner/src/backends/codex-native-backend.ts @@ -223,7 +223,10 @@ function createTransportBackedNativeSessionBackend( requireProviderSessionIdentity: options.transportFactory !== undefined, }), preparedContext ? constraints : undefined, - input.provider.kind === "cursor_cloud" ? CURSOR_CLOUD_TERMINAL_DRAIN_TIMEOUT_MS : undefined, + // Codex interruption uses a bounded 30-second transport command. Keep + // listening for its terminal through that window before quarantining a wait. + input.provider.kind === "cursor_cloud" ? CURSOR_CLOUD_TERMINAL_DRAIN_TIMEOUT_MS + : isCodex ? 30_000 : undefined, ); if (input.provider.kind === "cursor_cloud") { backend.descriptor = async () => { diff --git a/packages/paperclip-runner/src/live/runnerd-governed-wait.test.ts b/packages/paperclip-runner/src/live/runnerd-governed-wait.test.ts new file mode 100644 index 00000000000..e0413ee00b3 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-governed-wait.test.ts @@ -0,0 +1,113 @@ +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, it, vi } from "vitest"; +import { createRunnerdNativeSessionBackend } from "../backends/codex-native-backend.js"; +import type { NativeExecutionInputV1 } from "../contracts/native-execution.js"; +import type { PersistedNativeSession } from "../contracts/native-session-backend.js"; +import type { PrpEvent, PrpStructuredRunResult } from "../protocol/replay-contract.js"; +import { executeNativeSession } from "../native-session-runtime.js"; +import { createRunnerdCodexTransport } from "./runnerd-codex-transport.js"; +import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js"; + +it.each([0, 18_000, null])("requires a proven Codex terminal when parking a wait (delay: %s)", async terminalDelayMs => { + const root = await mkdtemp(join(tmpdir(), "runner-governed-wait-")); + const input: NativeExecutionInputV1 = { + schema: "paperclip.native-execution-input.v1", + binding: { + runId: "run-wait", companyId: "company-wait", issueId: "issue-wait", + agentId: "agent-wait", executionWorkspaceId: "workspace-wait", + }, + session: { + normalizedSessionId: "session-wait", driverKind: "codex_app_server", protocolVersion: 1, + lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 }, + }, + task: { + identifier: "TEST-1", title: "Ask for access", description: null, + prompt: "Ask for required access.", workMode: "standard", + }, + workspace: { cwd: root, repoUrl: null, repoRef: null, branchName: null }, + provider: { kind: "codex", model: null }, + completionContract: { + id: "contract-wait", sha256: "digest", schemaVersion: "paperclip.completion-contract.v1", + contract: { + revision: "1", objective: "Ask for access", + criteria: [{ id: "access", requirement: "Obtain access" }], + }, + }, + interactionResponses: [], credentialBindings: [], + }; + const bundle = createRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(root, "--hold-turn", "--emit-tool-call", "--call-log", join(root, "calls.log"), + ...(terminalDelayMs === null + ? ["--accept-interrupt-without-terminal"] + : ["--interrupt-terminal-delay-ms", String(terminalDelayMs)])), + stateDirectory: root, + prpIdentity: { + runnerInstanceId: "runner-wait", environmentLeaseId: "lease-wait", runId: "run-wait", + normalizedSessionId: "session-wait", turnId: "turn-wait", itemId: "item-wait", + }, + }); + // The controlled provider calls its standard dynamic-tool fixture. The port + // below supplies the server's already-resolved governed-wait disposition. + const tool = vi.fn(async () => ({ ok: true, task: { id: "task-1" } })); + const backend = createRunnerdNativeSessionBackend(input, { + runnerInstanceId: "runner-wait", transportFactory: () => bundle.transport, + dynamicTools: [{ + name: "get_task_context", description: "Read the task.", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + }], + dynamicToolHandler: tool, + }); + const yielded: PrpStructuredRunResult = { + schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", summary: "Waiting for access.", + completionClaim: { + contractRevision: "1", objectiveSatisfied: false, + criteria: [{ criterionId: "access", status: "unknown", evidenceRefs: [] }], + remainingWork: [{ description: "Await access.", blocksCompletion: true }], + }, + evidence: [], verification: [], attentionRequests: [], artifacts: [], + continuation: { kind: "response_wake", summary: "Continue after access.", idempotencyKey: "question-1" }, + }; + const history: PrpEvent[] = []; + const completeRun = vi.fn(async () => undefined); + let checkpoint: PersistedNativeSession | undefined; + try { + const execution = executeNativeSession({ + input, backend, runnerInstanceId: "runner-wait", controlPlaneInstanceId: "controller-wait", timeoutMs: 120_000, + controlPlane: { + async openRun() {}, + async checkpointSession(value) { checkpoint = structuredClone(value); }, + completeRun, + async appendEvent(event: PrpEvent) { + history.push(event); + return { cursor: history.length, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; + }, + async replayEvents(request) { + return { + events: history.filter(event => event.sourceInstanceId === request.sourceInstanceId && event.sourceSeq > request.afterSourceSeq), + highestContiguousSourceSeq: 0, + }; + }, + }, + resolveGovernedWait: ({ event }) => event.eventType === "item.completed" && event.payload.kind === "dynamicToolCall" ? yielded : null, + }); + if (terminalDelayMs === null) { + await expect(execution).rejects.toMatchObject({ providerCode: "governed_wait_provider_terminal_unproven" }); + expect(completeRun).not.toHaveBeenCalled(); + expect(history.some(event => event.eventType === "run.terminal")).toBe(false); + } else { + await expect(execution).resolves.toMatchObject({ result: yielded }); + expect(completeRun).toHaveBeenCalledOnce(); + expect(history.some(event => event.eventType === "turn.interrupted")).toBe(true); + } + expect(checkpoint?.governedWait?.result).toEqual(yielded); + expect(tool).toHaveBeenCalledOnce(); + const calls = (await readFile(join(root, "calls.log"), "utf8")).split("\n"); + expect(calls.filter(call => call === "turn/start")).toHaveLength(1); + } finally { + await bundle.transport.close().catch(() => undefined); + await rm(root, { recursive: true, force: true }); + } +}, 45_000);