fix(native): keep an unreachable remote Runner alive when a failed run closes (#20923)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - On staging, `runner_remote_process_identity_unavailable` was the
most frequent failure. The tasks stayed blocked until someone clicked
Retry
> - In 31 of the 34 failed runs the reattachment found the Runner gone
with an unsettled `ready` journal (`runnerSettled: false`). The Runner
was not lost by accident. The controller's own close stopped it with
`SIGKILL`: close could not prove suspension within its 10 s budget, so
it force-stopped the Runner in the middle of a turn
> - The server then scheduled a same-run reattachment that could never
succeed, because the close had just removed the Runner it needed
> - This pull request keeps an unreachable remote Runner alive when a
failed run closes. A Stop, a cancelled turn and a `stop_cleanup`
recovery still stop it
> - The benefit is that the same-run reattachment finds the live Runner.
The Runner then settles into a `suspended` journal, which the existing
relaunch path resumes. The task continues without Retry, and no provider
work is replayed

## Linked Issues or Issue Description

Refs #20853, #20907

**What happened?**
Remote native runs failed with
`runner_remote_process_identity_unavailable` and error code
`native_continuation_requires_reconciliation`. The tasks were left
blocked. In the failed runs the first attempt failed with a recoverable
transport error, for example `provider_transport_failed: PRP command
turn.start timed out`, `native_session_interrupted` or a server restart.
The run log then showed `runner did not prove durable suspension before
checkpoint` and `runner shutdown failed: Error: Durable recovery runner
timed out.` The second string comes from `waitForProcess`, which sends
`SIGKILL` when its deadline passes. The same-run reattachment ran 20–60
s later and reported `process_verification` / `process_not_running` with
`runnerLifecycle: ready` and `runnerSettled: false`. Most of these runs
had done provider work before the failure (turns and tool calls), so the
automatic Retry correctly refused them (`provider_turn_unsettled`).

**What did you expect to happen?**
When a run fails for a reason that the server resumes on the same run,
the close does not destroy the Runner that the reattachment needs. The
reattachment adopts it, and the run continues without Retry and without
a repeated tool effect.

**Steps to reproduce**
1. Start a remote native run and let a turn start.
2. Make the Runner unreachable (a broken transport or a slow event
pump), so that the turn fails and the close cannot prove suspension
within its budget.
3. The close sends `SIGKILL` to the Runner. The same-run reattachment
then fails with `runner_remote_process_identity_unavailable`, and the
task is blocked.

**Relation to open PRs**
- #20853 covers the 3 runs whose journal was a settled `ready` cut
(`runnerSettled: true`). This PR does not change that path.
- #20907 makes the Runner's exit reason visible. It does not change
recovery. This PR does not overlap with it.

## What Changed

- `runnerd-codex-transport.ts`: `close(reason, {
retainUnsettledRemoteRunner })` accepts a new option. When the option is
set, the Runner is remote (a process launcher or an adopted Runner), the
close is not a `stop_cleanup`, the Runner is still alive and suspension
is not proven, the close does not stop the Runner. It skips the
`waitForProcess` kill, the adopted-runner `SIGKILL` and `forceKill`. It
still releases the controller route, writes the unsettled checkpoint and
throws `NativeSessionCloseUnrecoverableError`, so the session stays
quarantined and is never reused. The settlement diagnostic records
`runnerRetained: true`, and a new diagnostic records `remote runner left
alive unsettled for same-run reattachment`.
- `HarnessNativeSession.close` asks for retention only when the session
was not explicitly cancelled. After `cancel()` (Stop), it asks for the
old behavior, so Stop still stops the Runner.
- `CodexHarnessSession.close` passes the option to the transport.
`HarnessSession.close` and `CodexAppServerTransport.close` accept the
optional field. Callers that do not set it keep the old behavior.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`
describe the rule and the `runnerRetained` field.

How the run continues: the reattaching controller adopts the live
Runner. The Runner then completes the `runner.drain` and
`runner.suspend` that the failed close queued, and exits with a settled
`suspended` journal. That attempt fails as
`native_runner_process_exited`, which the server resumes on the same
run. The next reattachment finds the process gone with a settled
`suspended` journal and relaunches the Runner over it (the existing path
from #19745). If the Runner was only slow, it suspends by itself and the
first reattachment relaunches.

## Verification

- Red, on `master` with only the tests added:
`runnerd-codex-transport-remote-retention.test.ts` fails 2 of 5
(`expected 'gone' to be 'alive'`): the close killed the Runner.
`harness-driver-backend.test.ts -t "retain an unsettled"` fails 2 of 2.
- Green with the fix: `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/backends/harness-driver-backend.test.ts`.
- The staging shape: a real `paperclip-runnerd` started through a
process launcher, a turn in progress, the Runner frozen with `SIGSTOP`
during close. The close throws `NativeSessionCloseUnrecoverableError`
with `runnerRetained: true`, the Runner stays alive, and its journal
stays `ready`.
- A second controller attaches through `adoptExistingRunner` (no
duplicate launch). The Runner settles into `lifecycle: suspended` with
an empty outbox and no pending terminal delivery, then exits.
- An explicit cancellation, a caller that does not ask for retention,
and a `stop_cleanup` close still stop the Runner.
- Also run: `src/backends`, `src/drivers/codex`,
`src/drivers/acpx/codex-acpx-driver.test.ts`,
`src/drivers/acpx/codex-runtime-adapter.test.ts`,
`src/native-session-runtime.test.ts` (1045 tests pass; 2 `spawnSync
ETIMEDOUT` under full-parallel load pass alone), and all of `src/live/`
(files that timed out under full-parallel load pass at
`--maxWorkers=2`).
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is
clean.
- `pnpm verify` selects a broad plan (exit 2) because
`doc/execution-semantics.md` is a global CI input. CI supplies that
coverage.
- Not run: a live Daytona run. The fix is in the controller library that
runs on the server, so it needs no new Runner image.

## Risks

- Medium. A failed close now leaves an unreachable remote Runner alive
instead of killing it. This is the same state a restart detach leaves
today, and the same-run reattachment or a Stop cleanup owns it.
- If the server does not resume the run on the same Runner, the Runner
stays alive until something reattaches or the sandbox is released.
Remote Runners have no reconnect deadline
(`runnerProcessReconnectGraceMs: null`). Before this change they were
killed. The close still quarantines the session, so it is never released
for reuse.
- Stop is unchanged: an explicit cancellation, `settleCancelledTurn` and
`stop_cleanup` close without retention, so they still stop the Runner.
- Previous-release story: this change adds and changes no persisted
state. State that an older release left (a killed Runner with an
unsettled `ready` journal) keeps its current handling: the named refusal
`runner_remote_process_identity_unavailable` and a working Retry. Tasks
already blocked on staging keep their Retry button. A deploy during a
run changes nothing, because the decision is made in the closing
process.
- No wait is added before `turn.start`. The change only affects the
failure path of close.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-11 14:17:25 -06:00
1 parent e8e548e856
commit c1edb9ec02
9 files changed
+237 -9

No files matched your search

+10
View File
@@ -1715,6 +1715,16 @@ starts by itself once the gate clears. A Retry accepted during the pre-deploy
drain stays saved and is dispatched by the deferred-receipt worker after the drain stays saved and is dispatched by the deferred-receipt worker after the
restart. restart.
When a remote native run fails and its close cannot prove that the Runner
suspended (for example, the Runner is unreachable after `provider_transport_failed`
or a `turn.start` timeout), the controller no longer force-stops that Runner. It
leaves the Runner and its saved state running, keeps the session quarantined, and
the same-run reattachment adopts the live Runner with its real turn state. Before
this, the close stopped the Runner mid-turn, so the reattachment always found it
gone with an unsettled journal (`runner_remote_process_identity_unavailable`) and
the task waited for Retry. A Stop, a cancelled turn and a `stop_cleanup` recovery
still stop the Runner, and a Runner that settled or exited is handled as before.
A remote native run can lose its original runner or sandbox before reattachment. A remote native run can lose its original runner or sandbox before reattachment.
This happens after a server restart, and for a same-run retry after a turn ends This happens after a server restart, and for a same-run retry after a turn ends
without a semantic result. The reattachment then fails with without a semantic result. The reattachment then fails with
+4 -1
View File
@@ -358,7 +358,10 @@ runner diagnostic. Its evidence includes runner suspension, provider drain,
final provider state, pending call/operation/source-event IDs and input digests, final provider state, pending call/operation/source-event IDs and input digests,
and incomplete result-delivery command IDs and statuses. If execution and and incomplete result-delivery command IDs and statuses. If execution and
cleanup both fail, execution retains its original error identity and cleanup is cleanup both fail, execution retains its original error identity and cleanup is
attached as `cleanupError`. attached as `cleanupError`. `runnerRetained: true` means a remote Runner that
could not prove its suspension was left running for the same-run
reattachment instead of being force-stopped; the diagnostic
`remote runner left alive unsettled for same-run reattachment` records it.
Semantic settlement also includes up to 20 content-free failure records, with Semantic settlement also includes up to 20 content-free failure records, with
the call ID, operation ID, stage (`dispatch` or `persist_result`), and cause. the call ID, operation ID, stage (`dispatch` or `persist_result`), and cause.
@@ -124,6 +124,22 @@ describe("HarnessDriverBackend", () => {
expect(startTurn).toHaveBeenCalledExactlyOnceWith({ message }); expect(startTurn).toHaveBeenCalledExactlyOnceWith({ message });
}); });
it.each([
{ cancelled: false, retainUnsettledRemoteRunner: true },
{ cancelled: true, retainUnsettledRemoteRunner: false },
])("asks to retain an unsettled remote Runner on close only when the run was not cancelled (cancelled=$cancelled)",
async ({ cancelled, retainUnsettledRemoteRunner }) => {
const close = vi.fn(async () => {});
const session = Object.assign(new FakeHarnessSession(), { close });
const backend = new HarnessDriverBackend({ ...driver, openSession: async () => session });
const opened = await backend.openSession({ identity: {
runId: "run-1", sessionId: "session-1", companyId: "company-1", issueId: "issue-1", agentId: "agent-1",
} });
if (cancelled) await cancelSession(opened, { reason: "Stop requested", signal: new AbortController().signal }).cleanup;
await opened.close({ reason: "native session execution complete" });
expect(close).toHaveBeenCalledExactlyOnceWith({ reason: "native session execution complete", retainUnsettledRemoteRunner });
});
it("forwards reasoning when the provider explicitly supports it", async () => { it("forwards reasoning when the provider explicitly supports it", async () => {
const startTurn = vi.fn(async () => ({ turnId: "turn-1" })); const startTurn = vi.fn(async () => ({ turnId: "turn-1" }));
const session = Object.assign(new FakeHarnessSession(), { startTurn, supportsTurnReasoning: () => true }); const session = Object.assign(new FakeHarnessSession(), { startTurn, supportsTurnReasoning: () => true });
@@ -917,7 +917,7 @@ class HarnessNativeSession implements NativeSession {
} }
close(input: { reason: string }) { close(input: { reason: string }) {
return this.#session.close(input); return this.#session.close({ ...input, retainUnsettledRemoteRunner: !this.#explicitlyCancelled });
} }
} }
@@ -660,7 +660,7 @@ export interface HarnessSession {
release?(): Promise<DurableRecoveryIdentity | false>; release?(): Promise<DurableRecoveryIdentity | false>;
/** Relinquish controller authority without semantically closing the session. */ /** Relinquish controller authority without semantically closing the session. */
detachControllerForRestart?(): Promise<void>; detachControllerForRestart?(): Promise<void>;
close(input: { reason: string; force?: boolean }): Promise<void>; close(input: { reason: string; force?: boolean; retainUnsettledRemoteRunner?: boolean }): Promise<void>;
} }
/** A local harness implementation hidden behind the runner daemon boundary. */ /** A local harness implementation hidden behind the runner daemon boundary. */
@@ -75,7 +75,7 @@ export interface CodexAppServerTransport {
* Close the provider transport. The optional reason is controller-owned * Close the provider transport. The optional reason is controller-owned
* diagnostic context; transports must not forward it to the provider. * diagnostic context; transports must not forward it to the provider.
*/ */
close(reason?: string): Promise<void>; close(reason?: string, options?: { retainUnsettledRemoteRunner?: boolean }): Promise<void>;
/** After close: the runner settled and its process exit was observed. */ /** After close: the runner settled and its process exit was observed. */
runnerExitConfirmed?(): boolean; runnerExitConfirmed?(): boolean;
/** Return the confirmed released source, or false when active use was not released. */ /** Return the confirmed released source, or false when active use was not released. */
@@ -933,11 +933,11 @@ export class CodexHarnessSession
} }
// `force` is part of the contract. Codex closes the same way with or without it. // `force` is part of the contract. Codex closes the same way with or without it.
async close(input: { reason: string; force?: boolean }): Promise<void> { async close(input: { reason: string; force?: boolean; retainUnsettledRemoteRunner?: boolean }): Promise<void> {
this.cancelPendingRequests("session_closed"); this.cancelPendingRequests("session_closed");
this.eventQueue.close(); this.eventQueue.close();
// Untyped callers (for example generated scripts) may still omit the input. // Untyped callers (for example generated scripts) may still omit the input.
await this.transport.close(input?.reason); await this.transport.close(input?.reason, { retainUnsettledRemoteRunner: input?.retainUnsettledRemoteRunner === true });
} }
async release(): Promise<DurableRecoveryIdentity | false> { async release(): Promise<DurableRecoveryIdentity | false> {
@@ -0,0 +1,188 @@
import { execFileSync, spawn } from "node:child_process";
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { createServer } from "node:http";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { expect, it, vi } from "vitest";
import { NativeSessionCloseUnrecoverableError } from "../contracts/native-session-backend.js";
import type * as durableControlPlane from "../control-plane/durable-prp-control-plane.js";
import type { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";
import { codexSemanticToolSpecs } from "../drivers/codex/codex-app-server-driver.js";
import { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js";
import { defaultCapabilityRunnerdBinary, expectTurnStarted, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js";
function processState(pid: number): "alive" | "gone" {
try { process.kill(pid, 0); } catch { return "gone"; }
try {
return execFileSync("ps", ["-o", "stat=", "-p", String(pid)], { encoding: "utf8" }).trim().startsWith("Z") ? "gone" : "alive";
} catch { return "gone"; }
}
const signal = (pid: number | null | undefined, name: NodeJS.Signals) => {
if (!pid) return;
try { process.kill(pid, name); } catch (error) { if (!["ESRCH", "EPERM"].includes(String((error as NodeJS.ErrnoException).code))) throw error; }
};
async function unreachableRemoteRunnerClose(input: {
retainUnsettledRemoteRunner?: boolean;
recoveryPurpose?: "execution" | "stop_cleanup";
}) {
const directory = await mkdtemp(join(tmpdir(), "runnerd-remote-retention-"));
const children: ReturnType<typeof spawn>[] = [];
const launch = vi.fn((spec: durableControlPlane.RunnerProcessLaunchSpec) => {
const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore" });
children.push(child);
return {
child,
completion: new Promise<durableControlPlane.RunnerProcessResult>((resolveExit, rejectExit) => {
child.once("error", rejectExit);
child.once("exit", (code, exitSignal) => resolveExit({ code, signal: exitSignal, stdout: "", stderr: "" }));
}),
isExitConfirmed: () => child.exitCode !== null || child.signalCode !== null,
};
});
const diagnostics: string[] = [];
const bundle = createCapabilityRunnerdCodexTransport({
runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex,
codexArgs: fakeCodexArgs(directory, "--linger-after-turn-start"), stateDirectory: directory,
runnerReconnectGraceMs: 60_000, runnerProcessLauncher: launch, closeGraceMs: 600,
onDiagnostic: message => diagnostics.push(message),
...(input.recoveryPurpose ? { recoveryPurpose: input.recoveryPurpose } : {}),
});
bundle.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
let providerPid: number | null = null;
try {
await bundle.transport.request("initialize", {});
await bundle.transport.request("thread/start", { cwd: directory, dynamicTools: [] });
await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Keep working." }] });
await expectTurnStarted(bundle.transport.notifications()[Symbol.asyncIterator]());
providerPid = bundle.evidence().codexPid;
const runner = children[0]!;
signal(runner.pid, "SIGSTOP");
const closed = await bundle.transport.close("native session execution complete",
input.retainUnsettledRemoteRunner === undefined ? undefined
: { retainUnsettledRemoteRunner: input.retainUnsettledRemoteRunner })
.then(() => null, (error: unknown) => error);
const runnerState = processState(runner.pid!);
signal(runner.pid, "SIGCONT");
const runnerJournal = JSON.parse(await readFile(join(directory, "runner", "runner-state.json"), "utf8")) as Record<string, unknown>;
return { closed, runnerState, runnerJournal, launch, diagnostics };
} finally {
for (const child of children) { signal(child.pid, "SIGCONT"); signal(child.pid, "SIGKILL"); }
signal(providerPid, "SIGKILL");
await rm(directory, { recursive: true, force: true });
}
}
it("leaves an unreachable remote Runner alive and unsettled when a failed run closes, so the same-run reattach can adopt it", async () => {
const result = await unreachableRemoteRunnerClose({ retainUnsettledRemoteRunner: true });
expect(result.closed).toBeInstanceOf(NativeSessionCloseUnrecoverableError);
expect(result.runnerState).toBe("alive");
expect(result.runnerJournal.lifecycle).toBe("ready");
expect(result.launch).toHaveBeenCalledTimes(1);
const settlement = result.diagnostics.find(message => message.startsWith("native_session_settlement_incomplete "));
expect(JSON.parse(settlement!.slice("native_session_settlement_incomplete ".length)))
.toMatchObject({ runnerSuspended: false, runnerRetained: true });
}, 30_000);
it.each([
{ title: "an explicit cancellation", options: { retainUnsettledRemoteRunner: false } },
{ title: "a caller that does not ask to retain it", options: {} },
{ title: "a Stop cleanup", options: { retainUnsettledRemoteRunner: true, recoveryPurpose: "stop_cleanup" as const } },
])("still stops an unreachable remote Runner on $title", async ({ options }) => {
const result = await unreachableRemoteRunnerClose(options);
expect(result.closed).toBeInstanceOf(Error);
expect(result.runnerState).toBe("gone");
}, 30_000);
it("keeps a retained remote Runner for the reattaching controller, which settles it into a journal the next attempt can relaunch", async () => {
const root = await mkdtemp(join(tmpdir(), "runnerd-remote-retention-adopt-"));
const server = createServer();
let authority: DurablePrpControlPlane | null = null;
server.on("upgrade", (request, socket, head) => {
if (!authority) { socket.destroy(); return; }
authority.handleUpgrade(request, socket, "/runner", head);
});
await new Promise<void>(resolveListen => server.listen(0, "127.0.0.1", resolveListen));
const address = server.address();
if (!address || typeof address === "string") throw new Error("Missing retention test listener");
const registration = async (next: DurablePrpControlPlane) => {
authority = next;
return { connectUrl: `ws://127.0.0.1:${address.port}/runner`, release: () => { if (authority === next) authority = null; } };
};
const children: ReturnType<typeof spawn>[] = [];
const launch = vi.fn((spec: durableControlPlane.RunnerProcessLaunchSpec) => {
const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore", detached: true });
children.push(child);
return {
child,
completion: new Promise<durableControlPlane.RunnerProcessResult>((resolveExit, rejectExit) => {
child.once("error", rejectExit);
child.once("exit", (code, exitSignal) => resolveExit({ code, signal: exitSignal, stdout: "", stderr: "" }));
}),
isExitConfirmed: () => child.exitCode !== null || child.signalCode !== null,
};
});
const shared = {
runnerBinary: defaultCapabilityRunnerdBinary(),
codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
codexArgs: ["--state-file", join(root, "provider-state.json"), "--hold-first-durable-turn"],
stateDirectory: root,
prpIdentity: {
runnerInstanceId: "runner-retained", environmentLeaseId: "lease-retained", runId: "run-retained",
normalizedSessionId: "session-retained", turnId: "turn-retained", itemId: "item-retained",
},
runnerReconnectGraceMs: 5_000,
runnerProcessReconnectGraceMs: null,
controlPlaneRegistration: registration,
};
const first = createCapabilityRunnerdCodexTransport({ ...shared, runnerProcessLauncher: launch, closeGraceMs: 600 });
let adopted: ReturnType<typeof createCapabilityRunnerdCodexTransport> | undefined;
let pid: number | null = null;
let providerPid: number | null = null;
try {
const opened = await first.transport.request("thread/start", { cwd: root, dynamicTools: codexSemanticToolSpecs() });
await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep this task active." }] });
for await (const notification of first.transport.notifications()) {
if (notification.method === "turn/started") break;
}
pid = children[0]!.pid!;
providerPid = first.evidence().codexPid;
signal(pid, "SIGSTOP");
await expect(first.transport.close("native session execution complete", { retainUnsettledRemoteRunner: true }))
.rejects.toBeInstanceOf(NativeSessionCloseUnrecoverableError);
expect(processState(pid)).toBe("alive");
signal(pid, "SIGCONT");
const journal = async () => JSON.parse(await readFile(join(root, "runner", "runner-state.json"), "utf8")) as Record<string, unknown>;
await new Promise(resolveWait => setTimeout(resolveWait, 500));
expect(processState(pid)).toBe("alive");
expect((await journal()).lifecycle).toBe("ready");
const duplicateLaunch = vi.fn(() => { throw new Error("Must adopt the existing runner"); });
const thread = opened.thread as Record<string, unknown>;
const adoptedPid = pid;
adopted = createCapabilityRunnerdCodexTransport({
...shared,
resumeDynamicTools: codexSemanticToolSpecs(),
resumeProviderSession: { driverSessionId: String(thread.id), providerSessionId: String(thread.sessionId) },
runnerProcessLauncher: duplicateLaunch,
adoptExistingRunner: {
pid: adoptedPid, processGroupId: adoptedPid, startedAt: new Date().toISOString(),
isAlive: () => processState(adoptedPid) === "alive",
signal: name => { try { process.kill(-adoptedPid, name); return true; } catch { return false; } },
},
});
adopted.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
await adopted.transport.request("thread/read", {}).catch(() => undefined);
await vi.waitFor(() => expect(processState(adoptedPid)).toBe("gone"), { timeout: 10_000, interval: 50 });
expect(await journal()).toMatchObject({ lifecycle: "suspended", outbox: [], pendingTerminalDelivery: null });
expect(duplicateLaunch).not.toHaveBeenCalled();
expect(launch).toHaveBeenCalledTimes(1);
} finally {
await adopted?.transport.close().catch(() => undefined);
for (const child of children) { signal(child.pid, "SIGCONT"); signal(child.pid ? -child.pid : null, "SIGKILL"); }
signal(providerPid, "SIGKILL");
server.closeAllConnections();
if (server.listening) await new Promise<void>(resolveClose => server.close(() => resolveClose()));
await rm(root, { recursive: true, force: true });
}
}, 30_000);
@@ -4049,6 +4049,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
readonly #closedSignal: Promise<void>; readonly #closedSignal: Promise<void>;
#resolveClosedSignal!: () => void; #resolveClosedSignal!: () => void;
#closePromise: Promise<void> | null = null; #closePromise: Promise<void> | null = null;
#retainUnsettledRemoteRunner = false;
#pendingCloseCheckpointRetry: CloseCheckpointRetry | null = null; #pendingCloseCheckpointRetry: CloseCheckpointRetry | null = null;
#failedCloseCheckpoint: CloseCheckpointRetry | null = null; #failedCloseCheckpoint: CloseCheckpointRetry | null = null;
#shutdownSettled = false; #shutdownSettled = false;
@@ -5304,11 +5305,12 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
return false; return false;
} }
close(reason?: string): Promise<void> { close(reason?: string, options?: { retainUnsettledRemoteRunner?: boolean }): Promise<void> {
// Detachment relinquishes process ownership. A late execution finalizer // Detachment relinquishes process ownership. A late execution finalizer
// must not suspend or signal the runner now owned by the next controller. // must not suspend or signal the runner now owned by the next controller.
if (this.#controllerDetachedForRestart) if (this.#controllerDetachedForRestart)
return this.#detachPromise ?? Promise.resolve(); return this.#detachPromise ?? Promise.resolve();
if (this.#closePromise === null) this.#retainUnsettledRemoteRunner = options?.retainUnsettledRemoteRunner === true;
if (reason) { if (reason) {
this.#diagnostic( this.#diagnostic(
`runner transport close requested: ${reason.replaceAll(/[\r\n]/g, " ").slice(0, 1_000)}`, `runner transport close requested: ${reason.replaceAll(/[\r\n]/g, " ").slice(0, 1_000)}`,
@@ -5516,6 +5518,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
// runnerd exited on its own, unsuspended, while the session was open: it // runnerd exited on its own, unsuspended, while the session was open: it
// was lost (killed or crashed), not stopped or broken by this close. // was lost (killed or crashed), not stopped or broken by this close.
let runnerLost = false; let runnerLost = false;
let runnerRetained = false;
const remoteRunnerRetainable = this.#retainUnsettledRemoteRunner
&& this.options.recoveryPurpose !== "stop_cleanup"
&& (this.options.runnerProcessLauncher !== undefined || adoptedRunner !== undefined);
let lastSuspensionState: Record<string, unknown> | null = null; let lastSuspensionState: Record<string, unknown> | null = null;
let suspensionState: { let suspensionState: {
commandStatus: string | null; commandStatus: string | null;
@@ -5604,9 +5610,13 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
this.#diagnostic( this.#diagnostic(
"runner did not prove durable suspension before checkpoint", "runner did not prove durable suspension before checkpoint",
); );
runnerRetained = remoteRunnerRetainable && !runnerLost && !(await this.#runnerHasExited());
if (runnerRetained) this.#diagnostic("remote runner left alive unsettled for same-run reattachment");
} }
try { try {
if (this.#handle) { if (runnerRetained) {
this.#evidence.runnerExited = false;
} else if (this.#handle) {
const result = await waitForProcess( const result = await waitForProcess(
this.#handle, this.#handle,
Math.max(0, closeDeadline - Date.now()), Math.max(0, closeDeadline - Date.now()),
@@ -5690,6 +5700,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
if (suspensionRequired && !runnerSettled && !runnerLossReaped) { if (suspensionRequired && !runnerSettled && !runnerLossReaped) {
const settlement = { const settlement = {
runnerSuspended, runnerSuspended,
runnerRetained,
providerDrained, providerDrained,
suspensionState, suspensionState,
semanticTools: this.#core?.semanticToolSettlementDiagnostics(), semanticTools: this.#core?.semanticToolSettlementDiagnostics(),
@@ -5738,7 +5749,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
}), }),
forceKill: async () => { forceKill: async () => {
const handle = this.#handle; const handle = this.#handle;
if (!handle || this.#evidence.runnerExited) return; if (!handle || this.#evidence.runnerExited || runnerRetained) return;
handle.child.kill("SIGKILL"); handle.child.kill("SIGKILL");
// A remote kill dispatches an asynchronous, ownership-fenced RPC. // A remote kill dispatches an asynchronous, ownership-fenced RPC.
// Do not release the session for reuse until its process monitor has // Do not release the session for reuse until its process monitor has