From 46e5e4450ffada4826775a8336ef6f33ae8ef345 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 15 Nov 2026 10:27:04 -0600 Subject: [PATCH] fix(runner): leave an undispatched tool call to the next controller on restart detach (#21786) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner keeps an agent session alive while its controller restarts, for example during a deploy, and the next controller adopts it > - A Paperclip tool call can be between its committed input and its handler when the controller detaches > - In that window the old controller journaled "The turn stopped before this operation was dispatched" as the final, non-retryable result, but the turn did not stop > - The agent then got a false failure for a call that never ran, and the next controller could not recover the call. A restart must never fail work > - This pull request leaves that call unsettled for the next controller, the same as a call that was already in its handler > - The benefit is that a controller restart no longer turns an agent's pending tool call into a permanent error, and a CI test that failed on four pull requests now passes deterministically ## Linked Issues or Issue Description **What happened?** `src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` › `settles an earlier generation's in-flight tool call at attach even when that attach never finishes startup` failed in "ci / Verify Paperclip Runner (vitest package (3/3))" on 2026-11-15. In each job the first try and the CI retry both failed: - PR #21757, head `d04825d812`, job 132824209377 - PR #21713, head `7758d56719`, job 132681892394 - PR #21729, head `c63a05a013`, job 132732839971 Every failure is the same, at line 74, right after `detachControllerForRestart()`: ``` AssertionError: expected [ { …(8) } ] to deeply equal [] + "type": "semantic_tool.result", "status": "pending", + "result": { "error": { "code": "semantic_tool_not_dispatched", + "message": "The turn stopped before this operation was dispatched.", "retryable": false } } ``` **Expected behavior** When a controller detaches for restart, a tool call that it has not dispatched stays unsettled. The next controller recovers it at attach, the same as a call that is in its handler. **Steps to reproduce** 1. Hold the semantic tool dispatch at a gate after the Runner commits the tool input. 2. Call `detachControllerForRestart()`, then release the gate. 3. On `master`, the control-plane state has the `semantic_tool_not_dispatched` result above. It is byte-identical to CI (same command id and payload). The new test case `(still before dispatch at detach)` does this. **Root cause** This is a Runner defect, not a test timing problem. `#detachControllerOnce` sets `#closed`, resolves the closed signal and resolves the turn-start admission with `false`. A call in `#handleSemanticToolInput` that has not reached the handler then throws `SemanticToolNotDispatchedError`. The durable control plane turns that error into a journaled `semantic_tool_not_dispatched` result. That result is correct when the turn stops. It is false when the controller only detaches for restart, because the Runner and the turn continue. The old test waited only for the committed input, so on a loaded CI runner the detach sometimes came before the handler call. **Production impact** Yes. During a controller restart or deploy, an agent tool call in this window got a non-retryable "turn stopped" error after the Runner was adopted, and the next controller did not recover the call. ## What Changed - `durable-prp-control-plane.ts`: new `SemanticToolHandedOffError`. When the dispatcher rejects with it, the control plane journals no result. The input stays unsettled for the next controller. - `runnerd-codex-transport.ts`: the semantic tool callback goes through `#dispatchSemanticToolInput`. If the call ends as not dispatched after the transport detached for restart, it throws `SemanticToolHandedOffError`. A normal close still journals `semantic_tool_not_dispatched`. - `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`: the test waits for the exact stage instead of the committed input. It runs two cases: the call is in its handler at detach, and the call is held before dispatch across the detach. Both then check the existing attach recovery and the final `completed` result. ## Verification - Red on `master`: the `(still before dispatch at detach)` case fails every run with the CI assertion and payload. - Green on this branch: `npx vitest run src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` passes 2/2. - Loop on macOS with 4 extra busy-loop processes (load average about 13 before them), 50 runs each: this branch 50/50 (100 test cases). The old `master` test was 50/50 locally too, so the local loop does not show the CI interleaving. The gate does. - All 36 runner test files that use `detachControllerForRestart` or `SemanticToolNotDispatchedError`: 1180/1182 pass with 3 workers under load. The 2 failures (`runnerd-codex-transport-settlement.test.ts` and `runnerd-acpx-source-reconciliation.test.ts`, `kill EPERM`) pass 36/36 when run alone. - `npx tsc -p tsconfig.json --noEmit` and `npx tsc -p tsconfig.test.json --noEmit` pass in `packages/paperclip-runner`. - The other two static-check flakes from the same day are test-only and are fixed in a separate pull request. ## Risks - Medium-low risk. This changes Runner restart behavior for one window. A call that was not dispatched now goes to the next controller's recovery (`recoverSemanticToolInput`) instead of getting a final error. That is the same path that already handles a call in its handler, so no new recovery code is involved. - If no successor ever attaches, the input stays unsettled. That is the same state as for a call that was in its handler at detach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../durable-prp-control-plane.ts | 7 ++ ...-transport-orphaned-tool-at-attach.test.ts | 89 +++++++++++++++++-- .../src/live/runnerd-codex-transport.ts | 14 ++- 3 files changed, 101 insertions(+), 9 deletions(-) diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index 492e7534800..2ddf31f66b5 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -293,6 +293,12 @@ export class SemanticToolNotDispatchedError extends Error { } } +export class SemanticToolHandedOffError extends Error { + constructor() { + super("semantic_tool_handed_off"); + } +} + export interface DurablePrpControlPlaneOptions { stateDirectory: string; identity: DurableRecoveryIdentity; @@ -4798,6 +4804,7 @@ export class DurablePrpControlPlane { queueResult(outcome.result, outcome.isError === true), ) .catch((error: unknown) => { + if (error instanceof SemanticToolHandedOffError) return; if (error instanceof SemanticToolNotDispatchedError) { queueResult({ error: { code: "semantic_tool_not_dispatched", diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts index c946377453d..080305097ad 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts @@ -3,7 +3,8 @@ import { createServer } from "node:http"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { expect, it, vi } from "vitest"; -import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import * as durableControlPlane from "../control-plane/durable-prp-control-plane.js"; +import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; import { recoverToolCallback, TOOL_CALLBACK_INTERRUPTED_CODE } from "../control-plane/orphaned-tool-callback.js"; import { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js"; import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js"; @@ -12,7 +13,7 @@ const taskTool = { name: "get_task_context", description: "Read current task", i type SavedCommand = { commandId: string; type: string; status: string; payload: Record }; -it("settles an earlier generation's in-flight tool call at attach even when that attach never finishes startup", async () => { +it.each(["dispatched to its handler", "still before dispatch"] as const)("settles an earlier generation's in-flight tool call at attach even when that attach never finishes startup (%s at detach)", async stage => { const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-orphaned-tool-at-attach-")); const controlPlaneStatePath = join(stateDirectory, "control-plane", "control-plane-state.json"); const server = createServer(); @@ -55,8 +56,28 @@ it("settles an earlier generation's in-flight tool call at attach even when that const recovered = vi.fn(async (call: { operationId: string; callId: string; input: unknown; admission?: "this_controller" | "earlier_controller" }) => recoverToolCallback({ runId: identity.runId, call: { tool: call.operationId, callId: call.callId, arguments: call.input }, admission: call.admission, recover: async () => null })); + let reached!: () => void; + const callReached = new Promise(resolveReached => { reached = resolveReached; }); + let releaseDispatch!: () => void; + const dispatchGate = new Promise(resolveGate => { releaseDispatch = resolveGate; }); + let gatedOutcome: Promise | null = null; + const OriginalCore = durableControlPlane.DurablePrpControlPlane; + const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane").mockImplementation(function (coreOptions) { + const dispatch = coreOptions.onSemanticToolInput; + if (stage === "still before dispatch" && dispatch && !gatedOutcome) { + coreOptions = { ...coreOptions, onSemanticToolInput: call => { + reached(); + gatedOutcome = dispatchGate.then(() => dispatch(call)); + return gatedOutcome as ReturnType; + } }; + } + return new OriginalCore(coreOptions); + }); const first = createCapabilityRunnerdCodexTransport(options); - first.transport.setServerRequestHandler(() => new Promise>(() => undefined)); + first.transport.setServerRequestHandler(() => { + reached(); + return new Promise>(() => undefined); + }); const generations: Array> = []; let runnerPid: number | null = null; const savedToolResults = async () => (JSON.parse(await readFile(controlPlaneStatePath, "utf8")) as { commands: SavedCommand[] }) @@ -66,11 +87,11 @@ it("settles an earlier generation's in-flight tool call at attach even when that runnerPid = first.evidence().runnerPid; const runnerStartedAt = first.transport.processInfo!().startedAt; await first.transport.request("turn/start", { input: [{ type: "text", text: "Read current task" }] }); - await vi.waitFor(async () => { - const saved = JSON.parse(await readFile(controlPlaneStatePath, "utf8")) as { committedEvents: Array<{ eventType: string }> }; - expect(saved.committedEvents.some(event => event.eventType === "semantic_tool.input")).toBe(true); - }, { timeout: 10_000 }); + await callReached; await first.detachControllerForRestart(); + releaseDispatch(); + await (gatedOutcome as Promise | null)?.catch(() => undefined); + await new Promise(resolveTurn => setImmediate(resolveTurn)); expect(await savedToolResults()).toEqual([]); const adopt = (graceMs?: number) => { @@ -110,6 +131,8 @@ it("settles an earlier generation's in-flight tool call at attach even when that }, { timeout: 10_000 }); expect(recovered).toHaveBeenCalledTimes(1); } finally { + releaseDispatch(); + coreSpy.mockRestore(); for (const generation of generations.reverse()) await generation.transport.close().catch(() => undefined); await first.transport.close().catch(() => undefined); if (runnerPid) { @@ -120,3 +143,55 @@ it("settles an earlier generation's in-flight tool call at attach even when that await rm(stateDirectory, { recursive: true, force: true }); } }, 120_000); + +it("still settles a call stopped before dispatch as not dispatched when the controller closes without a restart", async () => { + const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-not-dispatched-at-close-")); + const controlPlaneStatePath = join(stateDirectory, "control-plane", "control-plane-state.json"); + let reached!: () => void; + const callReached = new Promise(resolveReached => { reached = resolveReached; }); + let releaseDispatch!: () => void; + const dispatchGate = new Promise(resolveGate => { releaseDispatch = resolveGate; }); + let gatedOutcome: Promise | null = null; + const OriginalCore = durableControlPlane.DurablePrpControlPlane; + const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane").mockImplementation(function (coreOptions) { + const dispatch = coreOptions.onSemanticToolInput; + if (dispatch && !gatedOutcome) { + coreOptions = { ...coreOptions, onSemanticToolInput: call => { + reached(); + gatedOutcome = dispatchGate.then(() => dispatch(call)); + return gatedOutcome as ReturnType; + } }; + } + return new OriginalCore(coreOptions); + }); + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(stateDirectory, "--emit-tool-call"), + stateDirectory, + }); + bundle.transport.setServerRequestHandler(() => new Promise>(() => undefined)); + let runnerPid: number | null = null; + try { + await bundle.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [taskTool] }); + runnerPid = bundle.evidence().runnerPid; + await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Read current task" }] }); + await callReached; + const closing = bundle.transport.close().catch(() => undefined); + releaseDispatch(); + await (gatedOutcome as Promise | null)?.catch(() => undefined); + await closing; + const results = (JSON.parse(await readFile(controlPlaneStatePath, "utf8")) as { commands: SavedCommand[] }) + .commands.filter(command => command.type === "semantic_tool.result"); + expect(results).toHaveLength(1); + expect(JSON.stringify(results[0]!.payload)).toContain("semantic_tool_not_dispatched"); + } finally { + releaseDispatch(); + coreSpy.mockRestore(); + await bundle.transport.close().catch(() => undefined); + if (runnerPid) { + try { process.kill(-runnerPid, "SIGKILL"); } catch { runnerPid = null; } + } + await rm(stateDirectory, { recursive: true, force: true }); + } +}, 120_000); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index a9e56ff8f2e..338eedbcd6f 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -76,6 +76,7 @@ import { NativeRunnerResumeStateUnavailableError, NativeRunnerSourceReconciliati import { DurablePrpControlPlane, inspectStagedRunAttachment, + SemanticToolHandedOffError, SemanticToolNotDispatchedError, durableRecoveryInternals, inspectWarmRunTransition, @@ -6518,7 +6519,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { onAuthenticatedConnection: () => this.#startupSteps?.advance("runner.process.launch", "provider.session.load"), get authorityRunId() { return authorityRunId(); }, semanticInputMode: this.#retainsSourceInputs() || this.options.recoveryPurpose === "stop_cleanup" ? "retain_only" : "dispatch", - onSemanticToolInput: this.options.recoveryPurpose === "stop_cleanup" ? undefined : (call) => this.#handleSemanticToolInput(call), + onSemanticToolInput: this.options.recoveryPurpose === "stop_cleanup" ? undefined : (call) => this.#dispatchSemanticToolInput(call), recoverSemanticToolInput: this.options.recoveryPurpose === "stop_cleanup" ? undefined : call => this.#recoverSemanticToolInput(call), onCommittedEvent: async (event) => { const source = this.#sourceAccountingIdentity(); @@ -7079,7 +7080,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { onAuthenticatedConnection: () => this.#startupSteps?.advance("runner.process.launch", "provider.session.load"), get authorityRunId() { return authorityRunId(); }, semanticInputMode: this.#retainsSourceInputs() || this.options.recoveryPurpose === "stop_cleanup" ? "retain_only" : "dispatch", - onSemanticToolInput: this.options.recoveryPurpose === "stop_cleanup" ? undefined : (call) => this.#handleSemanticToolInput(call), + onSemanticToolInput: this.options.recoveryPurpose === "stop_cleanup" ? undefined : (call) => this.#dispatchSemanticToolInput(call), recoverSemanticToolInput: this.options.recoveryPurpose === "stop_cleanup" ? undefined : call => this.#recoverSemanticToolInput(call), onCommittedEvent: async (event) => { const source = this.#sourceAccountingIdentity(); @@ -7680,6 +7681,15 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { return outcome ? this.#bindSemanticToolOutcome(call, outcome) : null; } + async #dispatchSemanticToolInput(call: Parameters>[0]) { + try { + return await this.#handleSemanticToolInput(call); + } catch (error) { + if (error instanceof SemanticToolNotDispatchedError && this.#controllerDetachedForRestart) throw new SemanticToolHandedOffError(); + throw error; + } + } + async #handleSemanticToolInput( call: Parameters< NonNullable