From c1edb9ec02f6d80127d70fa137a307bd5804e2cf Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 14:17:25 -0600 Subject: [PATCH] fix(native): keep an unreachable remote Runner alive when a failed run closes (#20923) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) start a detached `paperclip-runnerd`. After a recoverable failure, the server resumes the same run and reattaches to that Runner > - On staging, `runner_remote_process_identity_unavailable` was the most frequent failure. The tasks stayed blocked until someone clicked Retry > - In 31 of the 34 failed runs the reattachment found the Runner gone with an unsettled `ready` journal (`runnerSettled: false`). The Runner was not lost by accident. The controller's own close stopped it with `SIGKILL`: close could not prove suspension within its 10 s budget, so it force-stopped the Runner in the middle of a turn > - The server then scheduled a same-run reattachment that could never succeed, because the close had just removed the Runner it needed > - This pull request keeps an unreachable remote Runner alive when a failed run closes. A Stop, a cancelled turn and a `stop_cleanup` recovery still stop it > - The benefit is that the same-run reattachment finds the live Runner. The Runner then settles into a `suspended` journal, which the existing relaunch path resumes. The task continues without Retry, and no provider work is replayed ## Linked Issues or Issue Description Refs #20853, #20907 **What happened?** Remote native runs failed with `runner_remote_process_identity_unavailable` and error code `native_continuation_requires_reconciliation`. The tasks were left blocked. In the failed runs the first attempt failed with a recoverable transport error, for example `provider_transport_failed: PRP command turn.start timed out`, `native_session_interrupted` or a server restart. The run log then showed `runner did not prove durable suspension before checkpoint` and `runner shutdown failed: Error: Durable recovery runner timed out.` The second string comes from `waitForProcess`, which sends `SIGKILL` when its deadline passes. The same-run reattachment ran 20–60 s later and reported `process_verification` / `process_not_running` with `runnerLifecycle: ready` and `runnerSettled: false`. Most of these runs had done provider work before the failure (turns and tool calls), so the automatic Retry correctly refused them (`provider_turn_unsettled`). **What did you expect to happen?** When a run fails for a reason that the server resumes on the same run, the close does not destroy the Runner that the reattachment needs. The reattachment adopts it, and the run continues without Retry and without a repeated tool effect. **Steps to reproduce** 1. Start a remote native run and let a turn start. 2. Make the Runner unreachable (a broken transport or a slow event pump), so that the turn fails and the close cannot prove suspension within its budget. 3. The close sends `SIGKILL` to the Runner. The same-run reattachment then fails with `runner_remote_process_identity_unavailable`, and the task is blocked. **Relation to open PRs** - #20853 covers the 3 runs whose journal was a settled `ready` cut (`runnerSettled: true`). This PR does not change that path. - #20907 makes the Runner's exit reason visible. It does not change recovery. This PR does not overlap with it. ## What Changed - `runnerd-codex-transport.ts`: `close(reason, { retainUnsettledRemoteRunner })` accepts a new option. When the option is set, the Runner is remote (a process launcher or an adopted Runner), the close is not a `stop_cleanup`, the Runner is still alive and suspension is not proven, the close does not stop the Runner. It skips the `waitForProcess` kill, the adopted-runner `SIGKILL` and `forceKill`. It still releases the controller route, writes the unsettled checkpoint and throws `NativeSessionCloseUnrecoverableError`, so the session stays quarantined and is never reused. The settlement diagnostic records `runnerRetained: true`, and a new diagnostic records `remote runner left alive unsettled for same-run reattachment`. - `HarnessNativeSession.close` asks for retention only when the session was not explicitly cancelled. After `cancel()` (Stop), it asks for the old behavior, so Stop still stops the Runner. - `CodexHarnessSession.close` passes the option to the transport. `HarnessSession.close` and `CodexAppServerTransport.close` accept the optional field. Callers that do not set it keep the old behavior. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md` describe the rule and the `runnerRetained` field. How the run continues: the reattaching controller adopts the live Runner. The Runner then completes the `runner.drain` and `runner.suspend` that the failed close queued, and exits with a settled `suspended` journal. That attempt fails as `native_runner_process_exited`, which the server resumes on the same run. The next reattachment finds the process gone with a settled `suspended` journal and relaunches the Runner over it (the existing path from #19745). If the Runner was only slow, it suspends by itself and the first reattachment relaunches. ## Verification - Red, on `master` with only the tests added: `runnerd-codex-transport-remote-retention.test.ts` fails 2 of 5 (`expected 'gone' to be 'alive'`): the close killed the Runner. `harness-driver-backend.test.ts -t "retain an unsettled"` fails 2 of 2. - Green with the fix: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-remote-retention.test.ts src/backends/harness-driver-backend.test.ts`. - The staging shape: a real `paperclip-runnerd` started through a process launcher, a turn in progress, the Runner frozen with `SIGSTOP` during close. The close throws `NativeSessionCloseUnrecoverableError` with `runnerRetained: true`, the Runner stays alive, and its journal stays `ready`. - A second controller attaches through `adoptExistingRunner` (no duplicate launch). The Runner settles into `lifecycle: suspended` with an empty outbox and no pending terminal delivery, then exits. - An explicit cancellation, a caller that does not ask for retention, and a `stop_cleanup` close still stop the Runner. - Also run: `src/backends`, `src/drivers/codex`, `src/drivers/acpx/codex-acpx-driver.test.ts`, `src/drivers/acpx/codex-runtime-adapter.test.ts`, `src/native-session-runtime.test.ts` (1045 tests pass; 2 `spawnSync ETIMEDOUT` under full-parallel load pass alone), and all of `src/live/` (files that timed out under full-parallel load pass at `--maxWorkers=2`). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is clean. - `pnpm verify` selects a broad plan (exit 2) because `doc/execution-semantics.md` is a global CI input. CI supplies that coverage. - Not run: a live Daytona run. The fix is in the controller library that runs on the server, so it needs no new Runner image. ## Risks - Medium. A failed close now leaves an unreachable remote Runner alive instead of killing it. This is the same state a restart detach leaves today, and the same-run reattachment or a Stop cleanup owns it. - If the server does not resume the run on the same Runner, the Runner stays alive until something reattaches or the sandbox is released. Remote Runners have no reconnect deadline (`runnerProcessReconnectGraceMs: null`). Before this change they were killed. The close still quarantines the session, so it is never released for reuse. - Stop is unchanged: an explicit cancellation, `settleCancelledTurn` and `stop_cleanup` close without retention, so they still stop the Runner. - Previous-release story: this change adds and changes no persisted state. State that an older release left (a killed Runner with an unsettled `ready` journal) keeps its current handling: the named refusal `runner_remote_process_identity_unavailable` and a working Retry. Tasks already blocked on staging keep their Retry button. A deploy during a run changes nothing, because the decision is made in the closing process. - No wait is added before `turn.start`. The change only affects the failure path of close. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 10 + doc/run-log-events.md | 5 +- .../backends/harness-driver-backend.test.ts | 16 ++ .../src/backends/harness-driver-backend.ts | 2 +- .../src/contracts/harness-driver.ts | 2 +- .../src/drivers/codex/app-server-transport.ts | 2 +- .../drivers/codex/codex-harness-session.ts | 4 +- ...d-codex-transport-remote-retention.test.ts | 188 ++++++++++++++++++ .../src/live/runnerd-codex-transport.ts | 17 +- 9 files changed, 237 insertions(+), 9 deletions(-) create mode 100644 packages/paperclip-runner/src/live/runnerd-codex-transport-remote-retention.test.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 62622d23111..a7f91e7ba13 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1715,6 +1715,16 @@ starts by itself once the gate clears. A Retry accepted during the pre-deploy drain stays saved and is dispatched by the deferred-receipt worker after the restart. +When a remote native run fails and its close cannot prove that the Runner +suspended (for example, the Runner is unreachable after `provider_transport_failed` +or a `turn.start` timeout), the controller no longer force-stops that Runner. It +leaves the Runner and its saved state running, keeps the session quarantined, and +the same-run reattachment adopts the live Runner with its real turn state. Before +this, the close stopped the Runner mid-turn, so the reattachment always found it +gone with an unsettled journal (`runner_remote_process_identity_unavailable`) and +the task waited for Retry. A Stop, a cancelled turn and a `stop_cleanup` recovery +still stop the Runner, and a Runner that settled or exited is handled as before. + A remote native run can lose its original runner or sandbox before reattachment. This happens after a server restart, and for a same-run retry after a turn ends without a semantic result. The reattachment then fails with diff --git a/doc/run-log-events.md b/doc/run-log-events.md index ecca5d5a430..cbf3b62079b 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -358,7 +358,10 @@ runner diagnostic. Its evidence includes runner suspension, provider drain, final provider state, pending call/operation/source-event IDs and input digests, and incomplete result-delivery command IDs and statuses. If execution and cleanup both fail, execution retains its original error identity and cleanup is -attached as `cleanupError`. +attached as `cleanupError`. `runnerRetained: true` means a remote Runner that +could not prove its suspension was left running for the same-run +reattachment instead of being force-stopped; the diagnostic +`remote runner left alive unsettled for same-run reattachment` records it. Semantic settlement also includes up to 20 content-free failure records, with the call ID, operation ID, stage (`dispatch` or `persist_result`), and cause. diff --git a/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts b/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts index f9af4a42915..cd2d6975156 100644 --- a/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts +++ b/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts @@ -124,6 +124,22 @@ describe("HarnessDriverBackend", () => { expect(startTurn).toHaveBeenCalledExactlyOnceWith({ message }); }); + it.each([ + { cancelled: false, retainUnsettledRemoteRunner: true }, + { cancelled: true, retainUnsettledRemoteRunner: false }, + ])("asks to retain an unsettled remote Runner on close only when the run was not cancelled (cancelled=$cancelled)", + async ({ cancelled, retainUnsettledRemoteRunner }) => { + const close = vi.fn(async () => {}); + const session = Object.assign(new FakeHarnessSession(), { close }); + const backend = new HarnessDriverBackend({ ...driver, openSession: async () => session }); + const opened = await backend.openSession({ identity: { + runId: "run-1", sessionId: "session-1", companyId: "company-1", issueId: "issue-1", agentId: "agent-1", + } }); + if (cancelled) await cancelSession(opened, { reason: "Stop requested", signal: new AbortController().signal }).cleanup; + await opened.close({ reason: "native session execution complete" }); + expect(close).toHaveBeenCalledExactlyOnceWith({ reason: "native session execution complete", retainUnsettledRemoteRunner }); + }); + it("forwards reasoning when the provider explicitly supports it", async () => { const startTurn = vi.fn(async () => ({ turnId: "turn-1" })); const session = Object.assign(new FakeHarnessSession(), { startTurn, supportsTurnReasoning: () => true }); diff --git a/packages/paperclip-runner/src/backends/harness-driver-backend.ts b/packages/paperclip-runner/src/backends/harness-driver-backend.ts index e6623853776..d736206396c 100644 --- a/packages/paperclip-runner/src/backends/harness-driver-backend.ts +++ b/packages/paperclip-runner/src/backends/harness-driver-backend.ts @@ -917,7 +917,7 @@ class HarnessNativeSession implements NativeSession { } close(input: { reason: string }) { - return this.#session.close(input); + return this.#session.close({ ...input, retainUnsettledRemoteRunner: !this.#explicitlyCancelled }); } } diff --git a/packages/paperclip-runner/src/contracts/harness-driver.ts b/packages/paperclip-runner/src/contracts/harness-driver.ts index 7a9396eac64..922fbd929ff 100644 --- a/packages/paperclip-runner/src/contracts/harness-driver.ts +++ b/packages/paperclip-runner/src/contracts/harness-driver.ts @@ -660,7 +660,7 @@ export interface HarnessSession { release?(): Promise; /** Relinquish controller authority without semantically closing the session. */ detachControllerForRestart?(): Promise; - close(input: { reason: string; force?: boolean }): Promise; + close(input: { reason: string; force?: boolean; retainUnsettledRemoteRunner?: boolean }): Promise; } /** A local harness implementation hidden behind the runner daemon boundary. */ diff --git a/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts b/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts index 2502927f526..c24c3825526 100644 --- a/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts +++ b/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts @@ -75,7 +75,7 @@ export interface CodexAppServerTransport { * Close the provider transport. The optional reason is controller-owned * diagnostic context; transports must not forward it to the provider. */ - close(reason?: string): Promise; + close(reason?: string, options?: { retainUnsettledRemoteRunner?: boolean }): Promise; /** After close: the runner settled and its process exit was observed. */ runnerExitConfirmed?(): boolean; /** Return the confirmed released source, or false when active use was not released. */ diff --git a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts index 6de9c319e24..e3bb904288b 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts @@ -933,11 +933,11 @@ export class CodexHarnessSession } // `force` is part of the contract. Codex closes the same way with or without it. - async close(input: { reason: string; force?: boolean }): Promise { + async close(input: { reason: string; force?: boolean; retainUnsettledRemoteRunner?: boolean }): Promise { this.cancelPendingRequests("session_closed"); this.eventQueue.close(); // Untyped callers (for example generated scripts) may still omit the input. - await this.transport.close(input?.reason); + await this.transport.close(input?.reason, { retainUnsettledRemoteRunner: input?.retainUnsettledRemoteRunner === true }); } async release(): Promise { diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport-remote-retention.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport-remote-retention.test.ts new file mode 100644 index 00000000000..c68c0f131e5 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport-remote-retention.test.ts @@ -0,0 +1,188 @@ +import { execFileSync, spawn } from "node:child_process"; +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { createServer } from "node:http"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import { NativeSessionCloseUnrecoverableError } from "../contracts/native-session-backend.js"; +import type * as durableControlPlane from "../control-plane/durable-prp-control-plane.js"; +import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import { codexSemanticToolSpecs } from "../drivers/codex/codex-app-server-driver.js"; +import { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js"; +import { defaultCapabilityRunnerdBinary, expectTurnStarted, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js"; + +function processState(pid: number): "alive" | "gone" { + try { process.kill(pid, 0); } catch { return "gone"; } + try { + return execFileSync("ps", ["-o", "stat=", "-p", String(pid)], { encoding: "utf8" }).trim().startsWith("Z") ? "gone" : "alive"; + } catch { return "gone"; } +} + +const signal = (pid: number | null | undefined, name: NodeJS.Signals) => { + if (!pid) return; + try { process.kill(pid, name); } catch (error) { if (!["ESRCH", "EPERM"].includes(String((error as NodeJS.ErrnoException).code))) throw error; } +}; + +async function unreachableRemoteRunnerClose(input: { + retainUnsettledRemoteRunner?: boolean; + recoveryPurpose?: "execution" | "stop_cleanup"; +}) { + const directory = await mkdtemp(join(tmpdir(), "runnerd-remote-retention-")); + const children: ReturnType[] = []; + const launch = vi.fn((spec: durableControlPlane.RunnerProcessLaunchSpec) => { + const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore" }); + children.push(child); + return { + child, + completion: new Promise((resolveExit, rejectExit) => { + child.once("error", rejectExit); + child.once("exit", (code, exitSignal) => resolveExit({ code, signal: exitSignal, stdout: "", stderr: "" })); + }), + isExitConfirmed: () => child.exitCode !== null || child.signalCode !== null, + }; + }); + const diagnostics: string[] = []; + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(directory, "--linger-after-turn-start"), stateDirectory: directory, + runnerReconnectGraceMs: 60_000, runnerProcessLauncher: launch, closeGraceMs: 600, + onDiagnostic: message => diagnostics.push(message), + ...(input.recoveryPurpose ? { recoveryPurpose: input.recoveryPurpose } : {}), + }); + bundle.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] })); + let providerPid: number | null = null; + try { + await bundle.transport.request("initialize", {}); + await bundle.transport.request("thread/start", { cwd: directory, dynamicTools: [] }); + await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Keep working." }] }); + await expectTurnStarted(bundle.transport.notifications()[Symbol.asyncIterator]()); + providerPid = bundle.evidence().codexPid; + const runner = children[0]!; + signal(runner.pid, "SIGSTOP"); + const closed = await bundle.transport.close("native session execution complete", + input.retainUnsettledRemoteRunner === undefined ? undefined + : { retainUnsettledRemoteRunner: input.retainUnsettledRemoteRunner }) + .then(() => null, (error: unknown) => error); + const runnerState = processState(runner.pid!); + signal(runner.pid, "SIGCONT"); + const runnerJournal = JSON.parse(await readFile(join(directory, "runner", "runner-state.json"), "utf8")) as Record; + return { closed, runnerState, runnerJournal, launch, diagnostics }; + } finally { + for (const child of children) { signal(child.pid, "SIGCONT"); signal(child.pid, "SIGKILL"); } + signal(providerPid, "SIGKILL"); + await rm(directory, { recursive: true, force: true }); + } +} + +it("leaves an unreachable remote Runner alive and unsettled when a failed run closes, so the same-run reattach can adopt it", async () => { + const result = await unreachableRemoteRunnerClose({ retainUnsettledRemoteRunner: true }); + expect(result.closed).toBeInstanceOf(NativeSessionCloseUnrecoverableError); + expect(result.runnerState).toBe("alive"); + expect(result.runnerJournal.lifecycle).toBe("ready"); + expect(result.launch).toHaveBeenCalledTimes(1); + const settlement = result.diagnostics.find(message => message.startsWith("native_session_settlement_incomplete ")); + expect(JSON.parse(settlement!.slice("native_session_settlement_incomplete ".length))) + .toMatchObject({ runnerSuspended: false, runnerRetained: true }); +}, 30_000); + +it.each([ + { title: "an explicit cancellation", options: { retainUnsettledRemoteRunner: false } }, + { title: "a caller that does not ask to retain it", options: {} }, + { title: "a Stop cleanup", options: { retainUnsettledRemoteRunner: true, recoveryPurpose: "stop_cleanup" as const } }, +])("still stops an unreachable remote Runner on $title", async ({ options }) => { + const result = await unreachableRemoteRunnerClose(options); + expect(result.closed).toBeInstanceOf(Error); + expect(result.runnerState).toBe("gone"); +}, 30_000); + +it("keeps a retained remote Runner for the reattaching controller, which settles it into a journal the next attempt can relaunch", async () => { + const root = await mkdtemp(join(tmpdir(), "runnerd-remote-retention-adopt-")); + const server = createServer(); + let authority: DurablePrpControlPlane | null = null; + server.on("upgrade", (request, socket, head) => { + if (!authority) { socket.destroy(); return; } + authority.handleUpgrade(request, socket, "/runner", head); + }); + await new Promise(resolveListen => server.listen(0, "127.0.0.1", resolveListen)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Missing retention test listener"); + const registration = async (next: DurablePrpControlPlane) => { + authority = next; + return { connectUrl: `ws://127.0.0.1:${address.port}/runner`, release: () => { if (authority === next) authority = null; } }; + }; + const children: ReturnType[] = []; + const launch = vi.fn((spec: durableControlPlane.RunnerProcessLaunchSpec) => { + const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore", detached: true }); + children.push(child); + return { + child, + completion: new Promise((resolveExit, rejectExit) => { + child.once("error", rejectExit); + child.once("exit", (code, exitSignal) => resolveExit({ code, signal: exitSignal, stdout: "", stderr: "" })); + }), + isExitConfirmed: () => child.exitCode !== null || child.signalCode !== null, + }; + }); + const shared = { + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(root, "provider-state.json"), "--hold-first-durable-turn"], + stateDirectory: root, + prpIdentity: { + runnerInstanceId: "runner-retained", environmentLeaseId: "lease-retained", runId: "run-retained", + normalizedSessionId: "session-retained", turnId: "turn-retained", itemId: "item-retained", + }, + runnerReconnectGraceMs: 5_000, + runnerProcessReconnectGraceMs: null, + controlPlaneRegistration: registration, + }; + const first = createCapabilityRunnerdCodexTransport({ ...shared, runnerProcessLauncher: launch, closeGraceMs: 600 }); + let adopted: ReturnType | undefined; + let pid: number | null = null; + let providerPid: number | null = null; + try { + const opened = await first.transport.request("thread/start", { cwd: root, dynamicTools: codexSemanticToolSpecs() }); + await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep this task active." }] }); + for await (const notification of first.transport.notifications()) { + if (notification.method === "turn/started") break; + } + pid = children[0]!.pid!; + providerPid = first.evidence().codexPid; + signal(pid, "SIGSTOP"); + await expect(first.transport.close("native session execution complete", { retainUnsettledRemoteRunner: true })) + .rejects.toBeInstanceOf(NativeSessionCloseUnrecoverableError); + expect(processState(pid)).toBe("alive"); + signal(pid, "SIGCONT"); + const journal = async () => JSON.parse(await readFile(join(root, "runner", "runner-state.json"), "utf8")) as Record; + await new Promise(resolveWait => setTimeout(resolveWait, 500)); + expect(processState(pid)).toBe("alive"); + expect((await journal()).lifecycle).toBe("ready"); + const duplicateLaunch = vi.fn(() => { throw new Error("Must adopt the existing runner"); }); + const thread = opened.thread as Record; + const adoptedPid = pid; + adopted = createCapabilityRunnerdCodexTransport({ + ...shared, + resumeDynamicTools: codexSemanticToolSpecs(), + resumeProviderSession: { driverSessionId: String(thread.id), providerSessionId: String(thread.sessionId) }, + runnerProcessLauncher: duplicateLaunch, + adoptExistingRunner: { + pid: adoptedPid, processGroupId: adoptedPid, startedAt: new Date().toISOString(), + isAlive: () => processState(adoptedPid) === "alive", + signal: name => { try { process.kill(-adoptedPid, name); return true; } catch { return false; } }, + }, + }); + adopted.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] })); + await adopted.transport.request("thread/read", {}).catch(() => undefined); + await vi.waitFor(() => expect(processState(adoptedPid)).toBe("gone"), { timeout: 10_000, interval: 50 }); + expect(await journal()).toMatchObject({ lifecycle: "suspended", outbox: [], pendingTerminalDelivery: null }); + expect(duplicateLaunch).not.toHaveBeenCalled(); + expect(launch).toHaveBeenCalledTimes(1); + } finally { + await adopted?.transport.close().catch(() => undefined); + for (const child of children) { signal(child.pid, "SIGCONT"); signal(child.pid ? -child.pid : null, "SIGKILL"); } + signal(providerPid, "SIGKILL"); + server.closeAllConnections(); + if (server.listening) await new Promise(resolveClose => server.close(() => resolveClose())); + await rm(root, { recursive: true, force: true }); + } +}, 30_000); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 791f0afda6c..58329d15648 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -4049,6 +4049,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { readonly #closedSignal: Promise; #resolveClosedSignal!: () => void; #closePromise: Promise | null = null; + #retainUnsettledRemoteRunner = false; #pendingCloseCheckpointRetry: CloseCheckpointRetry | null = null; #failedCloseCheckpoint: CloseCheckpointRetry | null = null; #shutdownSettled = false; @@ -5304,11 +5305,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { return false; } - close(reason?: string): Promise { + close(reason?: string, options?: { retainUnsettledRemoteRunner?: boolean }): Promise { // Detachment relinquishes process ownership. A late execution finalizer // must not suspend or signal the runner now owned by the next controller. if (this.#controllerDetachedForRestart) return this.#detachPromise ?? Promise.resolve(); + if (this.#closePromise === null) this.#retainUnsettledRemoteRunner = options?.retainUnsettledRemoteRunner === true; if (reason) { this.#diagnostic( `runner transport close requested: ${reason.replaceAll(/[\r\n]/g, " ").slice(0, 1_000)}`, @@ -5516,6 +5518,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { // runnerd exited on its own, unsuspended, while the session was open: it // was lost (killed or crashed), not stopped or broken by this close. let runnerLost = false; + let runnerRetained = false; + const remoteRunnerRetainable = this.#retainUnsettledRemoteRunner + && this.options.recoveryPurpose !== "stop_cleanup" + && (this.options.runnerProcessLauncher !== undefined || adoptedRunner !== undefined); let lastSuspensionState: Record | null = null; let suspensionState: { commandStatus: string | null; @@ -5604,9 +5610,13 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#diagnostic( "runner did not prove durable suspension before checkpoint", ); + runnerRetained = remoteRunnerRetainable && !runnerLost && !(await this.#runnerHasExited()); + if (runnerRetained) this.#diagnostic("remote runner left alive unsettled for same-run reattachment"); } try { - if (this.#handle) { + if (runnerRetained) { + this.#evidence.runnerExited = false; + } else if (this.#handle) { const result = await waitForProcess( this.#handle, Math.max(0, closeDeadline - Date.now()), @@ -5690,6 +5700,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { if (suspensionRequired && !runnerSettled && !runnerLossReaped) { const settlement = { runnerSuspended, + runnerRetained, providerDrained, suspensionState, semanticTools: this.#core?.semanticToolSettlementDiagnostics(), @@ -5738,7 +5749,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { }), forceKill: async () => { const handle = this.#handle; - if (!handle || this.#evidence.runnerExited) return; + if (!handle || this.#evidence.runnerExited || runnerRetained) return; handle.child.kill("SIGKILL"); // A remote kill dispatches an asynchronous, ownership-fenced RPC. // Do not release the session for reuse until its process monitor has