From cb941084844600f2e3b11a112c10647cc113f862 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 03:43:41 -0600 Subject: [PATCH] fix(native): finish the run from a provider turn a timed-out start left running (#21338) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run talks to a remote Runner through a controller. A deploy or a same-run retry can reattach a new controller to the live Runner. > - A slow turn start can make the first controller give up with "runnerd did not report the provider turn identity". Codex still runs that turn. > - The retry reattaches the Runner and adopts that turn, because the Runner accepted it for this run's latest turn.start. > - If the adopted turn finishes while recovery still runs, the saved checkpoint has no active turn. The runtime then tries to start a fresh turn, and the run fails with "session cannot start another turn". The Runner is lost and the task needs a Retry. > - This pull request makes the runtime treat a terminal that the recovered session newly settled as the adopted provider terminal when the saved checkpoint bound no turn. > - The benefit is that a turn which a timed-out start left running finishes the same run on the reattached Runner, with its session context, and with no second turn/start. ## Linked Issues or Issue Description **What happened?** On staging, Codex runs on a remote Runner lost the Runner after a same-run reattachment. Each chain started with a turn.start that timed out ("runnerd did not report the provider turn identity") while Codex ran the turn anyway. After the reattachment, the run failed with a protocol integrity fault (thread_binding_mismatch, tool_binding_mismatch, or native_event_replay_conflict "Provider authoritative notification item/started did not name its execution owner"). The task then fell back to a fresh session and lost its context. A local reproduction with the real runnerd and the fake Codex app server shows the same trigger also fails when the abandoned turn finishes during the reattachment: the runtime tries a new turn and fails with "session cannot start another turn". **Expected behavior** A turn that a timed-out turn.start left running belongs to the same run. The reattached run adopts it, binds its late notifications, and finishes from its outcome. Recovery does not kill the Runner and does not need a manual Retry. A real cross-run or cross-task binding violation still fails closed. **Steps to reproduce** 1. Start a native Codex run on a remote Runner whose provider is slow to emit turn/started after it accepts turn/start. 2. Let the controller give up with "runnerd did not report the provider turn identity" while the Runner stays alive. 3. Let the same-run retry reattach the Runner. 4. Let the provider turn start and finish after the reattachment. 5. See the run fail instead of finishing from the adopted turn. **Paperclip version or commit** master at 22c9300dac (after #21321 and #21329) **Deployment mode** Paperclip Cloud staging, remote Runner in a Daytona sandbox **Agent adapter(s) involved** Codex (codex_app_server through paperclip-runnerd) ## What Changed - `native-session-runtime.ts`: a terminal that the recovered session newly settled now counts as the adopted provider terminal when the saved checkpoint had no active turn. The runtime consumes that terminal instead of starting a fresh turn. - New `runnerd-abandoned-turn-start-reattach.test.ts` with the real runnerd and the fake Codex app server. Each attempt runs in a fresh module graph, as each attempt runs in its own executor process in production. The first attempt times out on turn.start against an unreachable remote Runner and leaves it alive. - Red/green: the provider turn starts and finishes only after the reattachment. The run now succeeds from `provider-turn-1` with one turn/start and no duplicate Runner launch. Before the fix it failed with "session cannot start another turn". - Guard: output the abandoned turn reported while no controller was attached binds to the reattached run's adopted turn with no `session.failed`, and the Runner stays alive. ## Verification - `npx vitest run src/live/runnerd-abandoned-turn-start-reattach.test.ts` in `packages/paperclip-runner`: red without the runtime change ("session cannot start another turn"), green with it. - `npx vitest run src/native-session-runtime.test.ts src/live/runnerd-restart-continuation.test.ts src/live/runnerd-abandoned-turn-start-reattach.test.ts src/live/runnerd-turn-start-timeout-retention.test.ts src/live/runnerd-codex-transport-cut-turn-start.test.ts src/drivers/codex/codex-app-server-driver.terminal-replay.test.ts src/drivers/codex/codex-app-server-driver.recovery.test.ts`: 7 files, 386 tests passed. - `tsc -p tsconfig.test.json --noEmit` in `packages/paperclip-runner`: no new errors (the existing `src/eval/workflow-harness.ts` errors need the eval kernel build). ## Risks - The runtime now adopts a newly settled terminal when the saved checkpoint bound no turn. The driver records a turn terminal only for a turn that it bound as active (the persisted active turn, the uncheckpointed disposition turn, or this run's own accepted turn). A turn of another run or task is never bound, so it still fails closed in the driver. - The staging chains failed with binding-mismatch codes. The local reproduction reaches the same trigger and the adoption path, but the fake provider cannot hold a live turn while it answers other requests, so the exact staging codes did not reproduce locally. Late items that reach a reattached controller before it knows the provider turn identity can still fail closed. That path needs a separate fix if it recurs after this change. ## Model Used - Claude Opus 5.5 (claude-opus-5-5) in Claude Code, with tool use (shell, file edits, browser reads of staging run events). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #21321, #21329 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...nerd-abandoned-turn-start-reattach.test.ts | 245 ++++++++++++++++++ .../src/native-session-runtime.ts | 1 + 2 files changed, 246 insertions(+) create mode 100644 packages/paperclip-runner/src/live/runnerd-abandoned-turn-start-reattach.test.ts diff --git a/packages/paperclip-runner/src/live/runnerd-abandoned-turn-start-reattach.test.ts b/packages/paperclip-runner/src/live/runnerd-abandoned-turn-start-reattach.test.ts new file mode 100644 index 00000000000..c4494f8f958 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-abandoned-turn-start-reattach.test.ts @@ -0,0 +1,245 @@ +import { execFileSync, spawn } from "node:child_process"; +import { createServer } from "node:http"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, it, vi } from "vitest"; +import type { NativeExecutionInputV1 } from "../contracts/native-execution.js"; +import type { PersistedNativeSession } from "../contracts/native-session-backend.js"; +import type * as durableControlPlane from "../control-plane/durable-prp-control-plane.js"; +import type { PrpEvent } from "../protocol/replay-contract.js"; +import type { executeNativeSession } from "../native-session-runtime.js"; +import type { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js"; +import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js"; + +type ControlPlaneModule = typeof import("../control-plane/durable-prp-control-plane.js"); +type Transport = ReturnType; + +function alive(pid: number | null | undefined): boolean { + if (!pid) return false; + try { process.kill(pid, 0); } catch { return false; } + try { + return !execFileSync("ps", ["-o", "stat=", "-p", String(pid)], { encoding: "utf8" }).trim().startsWith("Z"); + } catch { return false; } +} + +const kill = (pid: number | null | undefined) => { + if (!pid) return; + try { process.kill(pid, "SIGKILL"); } catch (error) { if (!["ESRCH", "EPERM"].includes(String((error as NodeJS.ErrnoException).code))) throw error; } +}; + +async function freshExecutorProcess() { + vi.resetModules(); + const [backend, control, runtime, transport] = await Promise.all([ + import("../backends/codex-native-backend.js"), + import("../control-plane/durable-prp-control-plane.js"), + import("../native-session-runtime.js"), + import("./runnerd-codex-transport.js"), + ]); + return { + createBackend: backend.createRunnerdNativeSessionBackend, + control, + execute: runtime.executeNativeSession, + createTransport: transport.createCapabilityRunnerdCodexTransport, + }; +} + +function withUnreachableRemoteRunner(control: ControlPlaneModule) { + let unreachable = true; + const OriginalCore = control.DurablePrpControlPlane; + const spy = vi.spyOn(control, "DurablePrpControlPlane") + .mockImplementation(function(options: ConstructorParameters[0]) { + const core = new OriginalCore(options); + const queueCommand = core.queueCommand.bind(core); + core.queueCommand = ((type: string, ...rest: unknown[]) => { + if (unreachable && type === "session.snapshot") throw new Error("runner snapshot unavailable"); + if (unreachable && (type === "runner.suspend" || type === "turn.stop")) return undefined; + return (queueCommand as (...args: unknown[]) => unknown)(type, ...rest); + }) as typeof core.queueCommand; + return core; + } as unknown as typeof OriginalCore); + return { restore: () => { unreachable = false; spy.mockRestore(); } }; +} + +async function abandonTurnStart(label: string, providerArgs: string[]) { + const root = await mkdtemp(join(tmpdir(), `runnerd-abandoned-${label}-`)); + const gate = join(root, "turn-started-gate"); + const callLog = join(root, "calls.log"); + const identity = { runId: `run-${label}`, sessionId: `session-${label}`, companyId: `company-${label}`, + issueId: `issue-${label}`, agentId: `agent-${label}` }; + const execution: NativeExecutionInputV1 = { + schema: "paperclip.native-execution-input.v1", + binding: { runId: identity.runId, companyId: identity.companyId, issueId: identity.issueId, agentId: identity.agentId, + executionWorkspaceId: `workspace-${label}` }, + session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + task: { identifier: "TEST-21", title: "Finish checks", description: null, prompt: "Finish the checks.", workMode: "standard" }, + workspace: { cwd: root, repoUrl: null, repoRef: null, branchName: null }, + provider: { kind: "codex", model: null }, + completionContract: { id: `contract-${label}`, sha256: "digest", schemaVersion: "paperclip.completion-contract.v1", contract: { + revision: "1", objective: "Finish checks", criteria: [{ id: "checks", requirement: "Check the outcome" }], + } }, + interactionResponses: [], credentialBindings: [], + }; + const children: ReturnType[] = []; + const launch = vi.fn((spec: durableControlPlane.RunnerProcessLaunchSpec) => { + const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore" }); + children.push(child); + return { + child, + completion: new Promise((resolveExit, rejectExit) => { + child.once("error", rejectExit); + child.once("exit", (code, exitSignal) => resolveExit({ code, signal: exitSignal, stdout: "", stderr: "" })); + }), + isExitConfirmed: () => child.exitCode !== null || child.signalCode !== null, + }; + }); + const server = createServer(); + let authority: durableControlPlane.DurablePrpControlPlane | null = null; + server.on("upgrade", (request, socket, head) => { + if (!authority) { socket.destroy(); return; } + authority.handleUpgrade(request, socket, "/runner", head); + }); + await new Promise(resolveListen => server.listen(0, "127.0.0.1", resolveListen)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Expected a control-plane listener"); + const options = { + controlPlaneRegistration: async (next: durableControlPlane.DurablePrpControlPlane) => { + authority = next; + return { connectUrl: `ws://127.0.0.1:${address.port}/runner`, release: async () => { if (authority === next) authority = null; } }; + }, + runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(root, "--turn-started-gate", gate, "--call-log", callLog, ...providerArgs), + stateDirectory: root, runnerReconnectGraceMs: 60_000, closeGraceMs: 600, turnStartTimeoutMs: 2_000, + prpIdentity: { runnerInstanceId: `runner-${label}`, environmentLeaseId: `lease-${label}`, runId: identity.runId, + normalizedSessionId: identity.sessionId, turnId: `turn-${label}`, itemId: `item-${label}` }, + }; + const events: PrpEvent[] = []; + let checkpoint: PersistedNativeSession | null = null; + const controlPlane: Parameters[0]["controlPlane"] = { + async openRun() {}, + async checkpointSession(snapshot) { checkpoint = structuredClone(snapshot); }, + async appendEvent(event) { + if (!("sourceSeq" in event)) throw new Error("Expected a durable PRP event"); + events.push(event); + return { cursor: events.length, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; + }, + async replayEvents(request) { + const source = events.filter(event => event.sourceInstanceId === request.sourceInstanceId); + return { events: source.filter(event => event.sourceSeq > request.afterSourceSeq), + highestContiguousSourceSeq: source.at(-1)?.sourceSeq ?? 0 }; + }, + async completeRun() {}, + }; + const transports: Transport[] = []; + const cleanup = async () => { + await rm(gate, { force: true }); + for (const bundle of [...transports].reverse()) await bundle.transport.close().catch(() => undefined); + for (const child of children) kill(child.pid); + for (const bundle of transports) { kill(bundle.evidence().runnerPid); kill(bundle.evidence().providerPid); } + await new Promise(resolveClose => server.close(() => resolveClose())); + await rm(root, { recursive: true, force: true }); + }; + try { + const firstProcess = await freshExecutorProcess(); + const remote = withUnreachableRemoteRunner(firstProcess.control); + try { + const first = firstProcess.createTransport({ ...options, runnerProcessLauncher: launch }); + transports.push(first); + await expect(firstProcess.execute({ + input: execution, runnerInstanceId: options.prpIdentity.runnerInstanceId, controlPlaneInstanceId: `controller-${label}`, + backend: firstProcess.createBackend(execution, { runnerInstanceId: options.prpIdentity.runnerInstanceId, transportFactory: () => first.transport }), + timeoutMs: 30_000, requireSessionCloseBeforeReturn: true, controlPlane, + })).rejects.toThrow("runnerd did not report the provider turn identity"); + } finally { + remote.restore(); + } + const runner = children[0]!; + expect(alive(runner.pid)).toBe(true); + const startedAt = transports[0]!.transport.processInfo!().startedAt; + const saved = checkpoint as PersistedNativeSession | null; + if (saved === null) throw new Error("Expected the first attempt to checkpoint its session"); + expect(saved.activeTurnId ?? null).toBeNull(); + const reattach = async () => { + const secondProcess = await freshExecutorProcess(); + const duplicateLauncher = vi.fn(() => { throw new Error("duplicate runner spawn attempted"); }); + const adopted = secondProcess.createTransport({ + ...options, + resumeDynamicTools: [], + resumeActiveTurnId: saved.activeTurnId ?? null, + resumeProviderSession: { driverSessionId: saved.sessionId, providerSessionId: saved.providerSessionId ?? null }, + runnerProcessLauncher: duplicateLauncher, + adoptExistingRunner: { + pid: runner.pid!, processGroupId: runner.pid!, startedAt, signal: () => true, + isAlive: () => alive(runner.pid), + }, + }); + transports.push(adopted); + const backend = secondProcess.createBackend(execution, { + runnerInstanceId: options.prpIdentity.runnerInstanceId, transportFactory: () => adopted.transport, + }); + return { + adopted, duplicateLauncher, backend, + execute: (persistedSession: PersistedNativeSession) => secondProcess.execute({ + input: execution, persistedSession, resumeInterruptedTurn: true, runnerInstanceId: options.prpIdentity.runnerInstanceId, + controlPlaneInstanceId: `controller-${label}-2`, timeoutMs: 30_000, controlPlane, backend, + }), + }; + }; + return { root, gate, callLog, runner, saved, events, reattach, cleanup }; + } catch (error) { + await cleanup(); + throw error; + } +} + +it("finishes the run from the provider turn a timed-out turn.start left running once it completes after the reattachment", async () => { + const attempt = await abandonTurnStart("completed-start", []); + let release: ReturnType | undefined; + try { + const second = await attempt.reattach(); + release = setInterval(() => { + if (!second.adopted.evidence().diagnostics.some(line => line.includes("restored its durable PRP session"))) return; + clearInterval(release); + void writeFile(attempt.gate, "start"); + }, 20); + release.unref(); + const completion = await second.execute(attempt.saved); + expect(completion).toMatchObject({ turnId: "provider-turn-1", terminal: { runTerminalState: "succeeded" } }); + expect(second.duplicateLauncher).not.toHaveBeenCalled(); + expect(attempt.events.some(event => event.eventType === "session.failed")).toBe(false); + const calls = (await readFile(attempt.callLog, "utf8")).trim().split(/\r?\n/); + expect(calls.filter(call => call === "turn/start")).toHaveLength(1); + } finally { + clearInterval(release); + await attempt.cleanup(); + } +}, 90_000); + +it("binds output the abandoned provider turn reported while no controller was attached to the reattached run", async () => { + const attempt = await abandonTurnStart("buffered-output", ["--hold-turn", "--held-final-message"]); + try { + await writeFile(attempt.gate, "start"); + await vi.waitFor(async () => { + expect(await readFile(join(attempt.root, "runner", "codex-provider-state.json"), "utf8")).toContain("Durable Cloud final"); + }, { timeout: 15_000, interval: 50 }); + const second = await attempt.reattach(); + const recovery = await second.backend.recoverSession!(attempt.saved, { signal: new AbortController().signal }); + expect(recovery).toMatchObject({ recovered: true }); + const session = recovery.session!; + const received: PrpEvent[] = []; + const reading = (async () => { for await (const event of session.events()) received.push(event); })().catch(() => undefined); + await vi.waitFor(() => { + expect(received.find(event => event.eventType === "session.failed")).toBeUndefined(); + expect(received.some(event => event.eventType === "item.completed" && event.turnId === "provider-turn-1" + && JSON.stringify(event.payload).includes("Durable Cloud final"))).toBe(true); + }, { timeout: 15_000, interval: 50 }); + expect((await session.snapshot()).activeTurnId).toBe("provider-turn-1"); + expect(alive(attempt.runner.pid)).toBe(true); + expect(second.duplicateLauncher).not.toHaveBeenCalled(); + await session.close({ reason: "test complete" }).catch(() => undefined); + await reading; + } finally { + await attempt.cleanup(); + } +}, 90_000); diff --git a/packages/paperclip-runner/src/native-session-runtime.ts b/packages/paperclip-runner/src/native-session-runtime.ts index 182f8e8b13f..4bbd09856cb 100644 --- a/packages/paperclip-runner/src/native-session-runtime.ts +++ b/packages/paperclip-runner/src/native-session-runtime.ts @@ -2987,6 +2987,7 @@ export async function executeNativeSession( (persistedTerminal) => persistedTerminal.turnId === terminal.turnId, ) && (terminal.turnId === persistedSession?.activeTurnId || + !persistedSession?.activeTurnId || recoveredSnapshot.dispositionOnlyRecoveryConsumed), ), );