fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 22:06:24 -06:00
1 parent 9b613562b7
commit 497ccba8f1
8 files changed
+389 -45

No files matched your search

+31 -1
View File
@@ -1782,6 +1782,35 @@ in-flight turn stay for the next same-run reattachment. An abort (a Stop or the
recovery deadline), any identity or authority refusal, any other recovery
failure, and a protocol integrity fault still stop it.
A kept Runner can answer its WebSocket pings but no recovery command. On
staging, a reattached Codex Runner authenticated and restored its session, then
left `session.snapshot` unanswered (`pending ahead 0; last runner event seq 338
76s ago`), and each later attempt adopted the same Runner and failed the same
way until the attempt budget ended the run. When a same-run reattachment fails
because its Runner left a recovery command unanswered (`PRP command <type> timed
out`, or `runnerd exited while waiting for <type>`), and the failure is not an
identity, authority or integrity refusal, the run records
`adoptedRunnerUnanswered` on its failure and the next claim carries
`replaceUnansweredRunner`. That attempt does not adopt the Runner again. After
the usual identity checks it launches a fresh Runner from the Runner's saved
journal and resumes the provider session from the saved checkpoint. This applies
only while the old Runner is alive: the launch fence stops that
`paperclip-runnerd` on the state folder and its whole process lineage, and fails
the launch closed when it cannot prove them gone. A Runner that already exited
mid-turn can leave provider or tool processes that the fence cannot find, so it
keeps the fail-closed path below, and the task continues in a fresh session from
the saved work. The fresh Runner
restores the provider thread; its in-flight turn is reconciled from history and
never submitted again. The provider attempt budget still bounds these attempts.
A paginated Codex history read (`thread/turns/list`, `thread/items/list`) that
fails without an answer is reported as `codex_history_read_failed: <method> read
failed`, with the underlying error. Only a provider answer that refuses the method
on its first page means the build lacks paginated history. Then the driver reads
the complete thread once (`thread/read` with `includeTurns: true`) and uses its
turns and items, failing closed when that read names another thread or omits
turns, items or turn identities. A refusal on a later page still fails closed.
A failed close can leave its `runner.suspend` (or `runner.shutdown`) pending in
the saved journal of a Runner it kept alive. The controller sends one command at
a time, so neither that command nor any command queued after it reached the
@@ -5896,7 +5925,8 @@ lifecycle `suspended`, an empty outbox and no pending terminal delivery, provide
cleanup or warm transition, the retry launches a Runner from that state instead
of adopting one. This matches a settled staged source. Any other exit stays
fail-closed for reconciliation, and the provider attempt budget still bounds the
retries.
retries. A live Runner that left the previous attempt's recovery read unanswered
is replaced, not adopted (see above).
The final admitted provider attempt may use this path; attempts above the limit
cannot. Process verification and the final state read share one recovery
deadline. A timeout, command failure, transport failure or invalid saved JSON on
@@ -1,10 +1,13 @@
import { describe, it, expect, vi } from "vitest";
import { CODEX_METHOD_NOT_FOUND, CodexRpcError } from "./app-server-transport.js";
import {
readCodexThreadState,
readCodexTurnMetadata,
readCodexTurnItems,
} from "./codex-history.js";
const unsupported = (method: string) => new CodexRpcError(`${method} is not supported`, CODEX_METHOD_NOT_FOUND);
describe("Codex paginated history", () => {
it("reads state without hydrating history", async () => {
const request = vi.fn().mockResolvedValue({ thread: { id: "thread" } });
@@ -79,14 +82,67 @@ describe("Codex paginated history", () => {
"codex_history_incomplete",
);
});
it("does not turn an interrupted page or unsupported method into idle history", async () => {
it("does not turn an unsupported later page into idle history", async () => {
const request = vi
.fn()
.mockResolvedValueOnce({ data: [], nextCursor: "next" })
.mockRejectedValueOnce(new Error("method not found"));
.mockRejectedValueOnce(unsupported("thread/turns/list"));
await expect(readCodexTurnMetadata({ request }, "thread")).rejects.toThrow(
"supported paginated history",
);
expect(request.mock.calls.map(call => call[0])).toEqual(["thread/turns/list", "thread/turns/list"]);
});
it("names a failed history read as a failed read, not as unsupported history", async () => {
const timeout = new Error("provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0)");
const request = vi.fn().mockRejectedValueOnce(timeout);
await expect(readCodexTurnMetadata({ request }, "thread")).rejects.toThrow(
`codex_history_read_failed: thread/turns/list read failed; Error: ${timeout.message}`,
);
expect(request).toHaveBeenCalledTimes(1);
});
it("reads complete turn history from the thread when paginated history is unsupported", async () => {
const request = vi.fn(async (method: string) => {
if (method === "thread/turns/list") throw unsupported(method);
return { thread: { id: "thread", turns: [
{ id: "one", status: "completed", items: [{ id: "answer", type: "agentMessage", text: "done" }] },
{ id: "two", status: "inProgress", items: [] },
] } };
});
expect(await readCodexTurnMetadata({ request }, "thread")).toEqual([
{ id: "one", status: "completed", items: [], itemsView: "notLoaded" },
{ id: "two", status: "inProgress", items: [], itemsView: "notLoaded" },
]);
expect(request).toHaveBeenLastCalledWith("thread/read", { threadId: "thread", includeTurns: true });
});
it("reads a turn's items from the thread when paginated items are unsupported", async () => {
const answer = { id: "answer", type: "agentMessage", text: "done" };
const request = vi.fn(async (method: string) => {
if (method === "thread/items/list") throw unsupported(method);
return { thread: { id: "thread", turns: [
{ id: "one", status: "completed", items: [answer] },
{ id: "two", status: "inProgress", items: [{ id: "other" }] },
] } };
});
expect(await readCodexTurnItems({ request }, "thread", "one")).toEqual([answer]);
});
it.each([
["another thread", { thread: { id: "other", turns: [] } }, "different driver session"],
["no turns", { thread: { id: "thread" } }, "supported paginated history"],
["a turn without a status", { thread: { id: "thread", turns: [{ id: "one" }] } }, "codex_history_incomplete"],
["a turn without an id", { thread: { id: "thread", turns: [{ status: "completed" }] } }, "codex_history_incomplete"],
])("fails closed when the unpaginated thread read returns %s", async (_case, thread, message) => {
const request = vi.fn(async (method: string) => {
if (method === "thread/turns/list") throw unsupported(method);
return thread;
});
await expect(readCodexTurnMetadata({ request }, "thread")).rejects.toThrow(message);
});
it("fails closed when the unpaginated thread read omits the requested turn's items", async () => {
const request = vi.fn(async (method: string) => {
if (method === "thread/items/list") throw unsupported(method);
return { thread: { id: "thread", turns: [{ id: "one", status: "completed" }] } };
});
await expect(readCodexTurnItems({ request }, "thread", "one")).rejects.toThrow("codex_history_incomplete");
});
it("rejects another thread or turn", async () => {
await expect(
@@ -1,9 +1,13 @@
import type { CodexAppServerTransport } from "./app-server-transport.js";
import { isCodexMethodUnavailable, type CodexAppServerTransport } from "./app-server-transport.js";
import { HarnessReconciliationError } from "../../contracts/harness-driver.js";
import { record, text } from "./codex-driver-values.js";
type Requester = Pick<CodexAppServerTransport, "request">;
const TURN_STATUSES = ["inProgress", "completed", "failed", "interrupted", "cancelled"];
class CodexPaginatedHistoryUnsupported extends HarnessReconciliationError {}
/** A partial or unsupported provider read is never evidence that work is idle. */
async function pages(
transport: Requester,
@@ -23,9 +27,14 @@ async function pages(
...(cursor ? { cursor } : {}),
});
} catch (error) {
throw new HarnessReconciliationError(
`codex_history_read_failed: ${method} requires supported paginated history; ${String(error)}`,
);
if (!isCodexMethodUnavailable(error))
throw new HarnessReconciliationError(
`codex_history_read_failed: ${method} read failed; ${String(error)}`,
);
const unsupported = `codex_history_read_failed: ${method} requires supported paginated history; ${String(error)}`;
throw page === 0
? new CodexPaginatedHistoryUnsupported(unsupported)
: new HarnessReconciliationError(unsupported);
}
if (!Array.isArray(response.data))
throw new HarnessReconciliationError(
@@ -71,31 +80,57 @@ export async function readCodexThreadState(
return snapshot;
}
export function readCodexTurnMetadata(
function turnIdentity(value: Record<string, unknown>): string {
if (!TURN_STATUSES.includes(text(value.status))) {
throw new HarnessReconciliationError(
"codex_history_incomplete: invalid turn status",
);
}
return text(value.id);
}
async function readCompleteThreadTurns(
transport: Requester,
threadId: string,
method: string,
cause: unknown,
): Promise<Record<string, unknown>[]> {
if (!(cause instanceof CodexPaginatedHistoryUnsupported)) throw cause;
const snapshot = await transport.request("thread/read", { threadId, includeTurns: true });
const thread = record(snapshot.thread);
if (text(thread.id) !== threadId)
throw new HarnessReconciliationError(
"thread/read returned a different driver session",
);
if (!Array.isArray(thread.turns))
throw new HarnessReconciliationError(
`codex_history_read_failed: ${method} requires supported paginated history; thread/read omitted turns`,
);
const turns = thread.turns.map(record);
for (const turn of turns) {
if (!turnIdentity(turn))
throw new HarnessReconciliationError(
"codex_history_incomplete: thread/read omitted an identity",
);
}
return turns;
}
export async function readCodexTurnMetadata(
transport: Requester,
threadId: string,
): Promise<Record<string, unknown>[]> {
return pages(
transport,
"thread/turns/list",
{ threadId, sortDirection: "asc", itemsView: "notLoaded" },
(value) => {
if (
![
"inProgress",
"completed",
"failed",
"interrupted",
"cancelled",
].includes(text(value.status))
) {
throw new HarnessReconciliationError(
"codex_history_incomplete: invalid turn status",
);
}
return text(value.id);
},
);
try {
return await pages(
transport,
"thread/turns/list",
{ threadId, sortDirection: "asc", itemsView: "notLoaded" },
turnIdentity,
);
} catch (error) {
const turns = await readCompleteThreadTurns(transport, threadId, "thread/turns/list", error);
return turns.map((turn) => ({ ...turn, items: [], itemsView: "notLoaded" }));
}
}
export async function readCodexTurnItems(
@@ -103,17 +138,28 @@ export async function readCodexTurnItems(
threadId: string,
turnId: string,
): Promise<Record<string, unknown>[]> {
const entries = await pages(
transport,
"thread/items/list",
{ threadId, turnId, sortDirection: "asc" },
(value) => {
if (text(value.turnId) !== turnId)
throw new HarnessReconciliationError(
"thread/items/list returned a different turn",
);
return text(record(value.item).id);
},
);
let entries: Record<string, unknown>[];
try {
entries = await pages(
transport,
"thread/items/list",
{ threadId, turnId, sortDirection: "asc" },
(value) => {
if (text(value.turnId) !== turnId)
throw new HarnessReconciliationError(
"thread/items/list returned a different turn",
);
return text(record(value.item).id);
},
);
} catch (error) {
const turns = await readCompleteThreadTurns(transport, threadId, "thread/items/list", error);
const items = turns.find((turn) => text(turn.id) === turnId)?.items;
if (!Array.isArray(items))
throw new HarnessReconciliationError(
"codex_history_incomplete: thread/read omitted the turn's items",
);
return items.map(record);
}
return entries.map((entry) => record(entry.item));
}
@@ -189,3 +189,111 @@ it("keeps a retained remote Runner for the reattaching controller, which continu
await rm(root, { recursive: true, force: true });
}
}, 30_000);
it("continues the same provider thread from a fresh Runner over the journal of a retained Runner that stopped answering, without replaying its turn", async () => {
const root = await mkdtemp(join(tmpdir(), "runnerd-remote-retention-replace-"));
const callLog = join(root, "provider-calls.log");
const server = createServer();
let authority: DurablePrpControlPlane | null = null;
server.on("upgrade", (request, socket, head) => {
if (!authority) { socket.destroy(); return; }
authority.handleUpgrade(request, socket, "/runner", head);
});
await new Promise<void>(resolveListen => server.listen(0, "127.0.0.1", resolveListen));
const address = server.address();
if (!address || typeof address === "string") throw new Error("Missing replacement test listener");
const registration = async (next: DurablePrpControlPlane) => {
authority = next;
return { connectUrl: `ws://127.0.0.1:${address.port}/runner`, release: () => { if (authority === next) authority = null; } };
};
const children: ReturnType<typeof spawn>[] = [];
const launch = vi.fn((spec: durableControlPlane.RunnerProcessLaunchSpec) => {
const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: "ignore", detached: true });
children.push(child);
return {
child,
completion: new Promise<durableControlPlane.RunnerProcessResult>((resolveExit, rejectExit) => {
child.once("error", rejectExit);
child.once("exit", (code, exitSignal) => resolveExit({ code, signal: exitSignal, stdout: "", stderr: "" }));
}),
isExitConfirmed: () => child.exitCode !== null || child.signalCode !== null,
};
});
const shared = {
runnerBinary: defaultCapabilityRunnerdBinary(),
codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
codexArgs: ["--state-file", join(root, "provider-state.json"), "--hold-first-durable-turn", "--call-log", callLog],
stateDirectory: root,
prpIdentity: {
runnerInstanceId: "runner-replaced", environmentLeaseId: "lease-replaced", runId: "run-replaced",
normalizedSessionId: "session-replaced", turnId: "turn-replaced", itemId: "item-replaced",
},
runnerReconnectGraceMs: 5_000,
runnerProcessReconnectGraceMs: null,
controlPlaneRegistration: registration,
runnerProcessLauncher: launch,
};
const first = createCapabilityRunnerdCodexTransport({ ...shared, closeGraceMs: 600 });
let adopted: ReturnType<typeof createCapabilityRunnerdCodexTransport> | undefined;
let replacement: ReturnType<typeof createCapabilityRunnerdCodexTransport> | undefined;
const providerPids: number[] = [];
try {
const opened = await first.transport.request("thread/start", { cwd: root, dynamicTools: codexSemanticToolSpecs() });
await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep this task active." }] });
for await (const notification of first.transport.notifications()) {
if (notification.method === "turn/started") break;
}
const wedged = children[0]!.pid!;
providerPids.push(first.evidence().codexPid!);
signal(wedged, "SIGSTOP");
await expect(first.transport.close("native session execution complete", { retainUnsettledRemoteRunner: true }))
.rejects.toBeInstanceOf(NativeSessionCloseUnrecoverableError);
expect(processState(wedged)).toBe("alive");
signal(wedged, "SIGCONT");
const thread = opened.thread as Record<string, unknown>;
const resume = {
resumeDynamicTools: codexSemanticToolSpecs(),
resumeProviderSession: { driverSessionId: String(thread.id), providerSessionId: String(thread.sessionId) },
};
adopted = createCapabilityRunnerdCodexTransport({
...shared, ...resume,
runnerProcessLauncher: vi.fn(() => { throw new Error("Must adopt the existing runner"); }),
adoptExistingRunner: {
pid: wedged, processGroupId: wedged, startedAt: new Date().toISOString(),
isAlive: () => processState(wedged) === "alive",
signal: name => { try { process.kill(-wedged, name); return true; } catch { return false; } },
},
});
adopted.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
await expect(adopted.transport.request("thread/read", {})).resolves.toEqual(
expect.objectContaining({ thread: expect.objectContaining({ id: thread.id }) }));
signal(wedged, "SIGSTOP");
await adopted.transport.detachControllerForRestart!();
expect(processState(wedged)).toBe("alive");
const journal = JSON.parse(await readFile(join(root, "runner", "runner-state.json"), "utf8")) as Record<string, unknown>;
expect(journal.lifecycle).not.toBe("suspended");
signal(-wedged, "SIGKILL");
for (const pid of providerPids) signal(pid, "SIGKILL");
await vi.waitFor(() => expect(processState(wedged)).toBe("gone"), { timeout: 5_000 });
replacement = createCapabilityRunnerdCodexTransport({
...shared, ...resume,
});
replacement.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] }));
await expect(replacement.transport.request("thread/read", {})).resolves.toEqual(
expect.objectContaining({ thread: expect.objectContaining({ id: thread.id }) }));
if (replacement.evidence().codexPid) providerPids.push(replacement.evidence().codexPid!);
expect(launch).toHaveBeenCalledTimes(2);
expect(replacement.evidence().runnerPid).toBe(children[1]!.pid);
const calls = (await readFile(callLog, "utf8")).trim().split("\n");
expect(calls.filter(call => call === "thread/start")).toHaveLength(1);
expect(calls.filter(call => call === "turn/start")).toHaveLength(1);
} finally {
await replacement?.transport.close().catch(() => undefined);
await adopted?.transport.close().catch(() => undefined);
for (const child of children) { signal(child.pid, "SIGCONT"); signal(child.pid ? -child.pid : null, "SIGKILL"); }
for (const pid of providerPids) signal(pid, "SIGKILL");
server.closeAllConnections();
if (server.listening) await new Promise<void>(resolveClose => server.close(() => resolveClose()));
await rm(root, { recursive: true, force: true });
}
}, 60_000);
@@ -103,6 +103,7 @@ export type NativeRestartRecoveryClaim =
/** Discovery evidence only; executor revalidates the encrypted header. */
stagedAttachment?: Pick<NativeStagedAttachmentRecovery, "sourceIdentity" | "targetIdentity" | "templateDigest">;
exitedBeforeSessionOpen?: true;
replaceUnansweredRunner?: true;
}
| {
kind: "resume_dead_runner";
@@ -2121,6 +2122,8 @@ export async function claimNativeRestartRecoveries(input: {
},
...(!pluginTask && row.coordinator.failureDetail?.runnerExitedBeforeSessionOpen === true
? { exitedBeforeSessionOpen: true as const } : {}),
...(!pluginTask && row.coordinator.failureDetail?.adoptedRunnerUnanswered === true
? { replaceUnansweredRunner: true as const } : {}),
} satisfies NativeRestartRecoveryClaim;
}
if (claimKind === "reattach_existing_runner") {
@@ -1573,6 +1573,39 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
expect(run?.processPid).toBe(remotePid);
});
it.each([true, false])("asks the same-run reattachment to replace a Runner that left its recovery read unanswered: %s", async (unanswered) => {
const fixture = await seedRun(`remote-unanswered-${unanswered}`);
await fixture.db.update(heartbeatRuns).set({
processPid: null,
processStartedAt: new Date("2020-01-01T00:00:00Z"),
runnerProfileJson: {
nativeWorkspaceSync: {
schema: "paperclip.native-workspace-sync/v1", state: "prepared",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64),
workspaceId: "workspace", leaseId: "lease", providerLeaseId: "sandbox",
remoteCwd: "/workspace",
},
sessionCheckpoint: {
identity: { companyId, agentId, issueId: fixture.issueId, runId: fixture.runId,
sessionId: fixture.native.normalizedSessionId },
providerSessionId: "provider-session",
process: { providerPid: process.pid },
},
},
}).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ attempt: 2, failureDetail: {
message: "native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 337 111s ago)",
recoveryMode: "exact_checkpoint_resume", checkpointExists: true,
...(unanswered ? { adoptedRunnerUnanswered: true } : {}),
} }).where(eq(nativeRunFinalizations.runId, fixture.runId));
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "graceful", runIds: [fixture.runId] });
expect(claim).toMatchObject({ kind: "reattach_remote_runner", runId: fixture.runId, providerAttempt: 2,
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } });
expect(claim?.kind === "reattach_remote_runner" ? claim.replaceUnansweredRunner : "not remote")
.toBe(unanswered ? true : undefined);
});
it("keeps a restart recovery's attempt credit while the agent directory is busy, then resumes at attempt 3", async () => {
const fixture = await seedRun("remote-busy-directory");
await fixture.db.update(heartbeatRuns).set({
@@ -476,6 +476,7 @@ import {
verifyRemoteRunnerReattachment,
stopStaleRemoteRunnerBeforeBootstrap,
verifyRemoteRunnerOrSettledExit,
nativeRecoveryCommandUnanswered,
readRemoteProviderPackManifest,
verifyRemoteProviderPackManifest,
prepareRemoteProviderPackOnTarget,
@@ -1141,6 +1142,54 @@ describe("remote controller restart adoption", () => {
target: exitedRunner(state) as never, identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught);
expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" });
});
describe("a same-run reattachment whose adopted Runner never answered its recovery read", () => {
const unanswered = { ...claim, replaceUnansweredRunner: true as const };
const midTurn = { lifecycle: "turn_active", outbox: [{ id: 338 }], pendingTerminalDelivery: null, pendingProviderCleanup: null };
it.each([2, 3])("launches a fresh Runner instead of adopting the live one again on attempt %i", async providerAttempt => {
const { target, execute } = fixture(midTurn);
await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity,
runId: "run", normalizedSessionId: "session", providerAttempt })).resolves.toBeUndefined();
expect(execute.mock.calls.some(([request]) => /kill -/.test(request.args.join(" ")))).toBe(false);
});
it("adopts the live Runner when the earlier attempt did not report an unanswered read", async () => {
const { target } = fixture(midTurn);
const adopted = await verifyRemoteRunnerOrSettledExit({ claim, target: target as never, identity,
runId: "run", normalizedSessionId: "session", providerAttempt: 2 });
expect(adopted?.pid).toBe(123);
});
it.each([
["a mid-turn exit whose provider and tools may survive it", midTurn, 3],
["a pending provider cleanup", { ...midTurn, pendingProviderCleanup: { pid: 7 } }, 3],
["an exhausted attempt budget", midTurn, 4],
] as const)("never resumes beside an exited unanswered Runner's lineage: %s", async (_case, state, providerAttempt) => {
const target = exitedRunner(state);
const error = await verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never,
identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught);
expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" });
});
it("keeps the live Runner's identity checks before replacing it", async () => {
const { target } = fixture({ ...midTurn, runnerInstanceId: "different" });
await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity,
runId: "run", normalizedSessionId: "session", providerAttempt: 2 })).rejects.toThrow("runner_remote_recovery_identity_mismatch");
});
it.each([
"native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 338 76s ago)",
"native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: runnerd exited while waiting for session.snapshot",
"native_session_recovery_failed: Error: runner_ingress_unavailable: PRP command runner.drain timed out (runner connected; pending ahead 0; last runner event seq 93 391s ago)",
])("recognizes the staging recovery read that got no answer: %s", message => {
expect(nativeRecoveryCommandUnanswered(new Error(message))).toBe(true);
expect(nativeRecoveryCommandUnanswered(new Error("wrapped", { cause: new Error(message) }))).toBe(true);
});
it.each([
"runnerd did not report the provider turn identity",
"provider_transport_failed: PRP command session.snapshot timed out",
"native_session_recovery_failed: provider resumed a different session",
"native_session_recovery_failed: provider_transport_failed: PRP command session.snapshot timed out; native_adopted_provider_identity_mismatch",
"native_session_recovery_failed: restart interruption history is incomplete",
])("does not replace a Runner for another failure: %s", message => {
expect(nativeRecoveryCommandUnanswered(new Error(message))).toBe(false);
});
});
it.each([
["an unsettled suspension", { ...settledSuspension, outbox: [{ id: 1 }] }, 2],
["a pending provider cleanup", { ...settledSuspension, pendingProviderCleanup: { pid: 7 } }, 2],
@@ -8860,6 +8860,16 @@ export function nativeRunnerExitedBeforeSessionOpen(error: unknown): boolean {
return nativeRunnerExitConfirmed(error) && runnerExitBeforeSessionOpenMessage(error);
}
export function nativeRecoveryCommandUnanswered(error: unknown, depth = 0): boolean {
if (!error || typeof error !== "object" || depth > 5) return false;
const value = error as Record<string, unknown>;
const message = typeof value.message === "string" ? value.message : "";
if (message.startsWith("native_session_recovery_failed:")
&& /PRP command [\w.]+ timed out|runnerd exited while waiting for [\w.]+/.test(message)
&& !/mismatch|different|identity|authority|integrity|incompatible/i.test(message)) return true;
return nativeRecoveryCommandUnanswered(value.cause, depth + 1);
}
/** Same provider, category and message (numbers ignored): a retry would meet the same fault. */
export function nativeStartupFailureSignature(failure: unknown): string | null {
const value = record(failure);
@@ -13054,6 +13064,9 @@ async function executePaperclipNativeSessionAttempt(
? { runnerExitConfirmed: true } : {}),
...(input.runnerExecutionTarget?.kind === "remote" && nativeRunnerExitedBeforeSessionOpen(error)
? { runnerExitedBeforeSessionOpen: true } : {}),
...(input.runnerExecutionTarget?.kind === "remote" && input.restartRecovery?.kind === "reattach_remote_runner"
&& input.restartRecovery.remote.kind !== "plugin_task" && nativeRecoveryCommandUnanswered(error)
? { adoptedRunnerUnanswered: true } : {}),
...(providerCooldownAt ? { providerCooldownAt: providerCooldownAt.toISOString() } : {}),
...(providerLimit?.plan.resetAt ? { providerLimitResetAt: providerLimit.plan.resetAt.toISOString() } : {}),
recoveryOwner: recoveryProjection.recoveryOwner,
@@ -14987,7 +15000,8 @@ const REMOTE_RUNNER_PROCESS_STDERR_TRIM_SCRIPT =
* A launch therefore stops a `paperclip-runnerd` that is still serving this
* exact `--state-dir`. A launch only happens after the server chose not to
* adopt a Runner: a same-run reattachment proves the marker's Runner gone
* first, and a transient reattachment failure never reaches a launch.
* first, or replaces a Runner that left the previous attempt's recovery read
* unanswered. Any other transient reattachment failure never reaches a launch.
*
* The stop runs in node ($1 the runnerd binary, $2 the state directory, then
* candidate PIDs):
@@ -15270,16 +15284,20 @@ async function readRemoteRunnerExitEvidence(runner: CommandManagedRuntimeRunner,
export async function verifyRemoteRunnerOrSettledExit(
input: Parameters<typeof verifyRemoteRunnerReattachment>[0] & { providerAttempt: number;
claim: { exitedBeforeSessionOpen?: true } },
claim: { exitedBeforeSessionOpen?: true; replaceUnansweredRunner?: true } },
) {
const recoveryDeadline = Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY,
Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS);
const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline);
const target = input.target;
const replaceUnanswered = input.claim.replaceUnansweredRunner === true
&& input.providerAttempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT
&& target?.kind === "remote" && target.transport === "sandbox" && Boolean(target.runner);
let verified: Awaited<ReturnType<typeof verifyRemoteRunnerReattachment>>;
try {
return await verifyRemoteRunnerReattachment({ ...input, recoveryDeadline, stall });
verified = await verifyRemoteRunnerReattachment({ ...input, recoveryDeadline, stall });
} catch (error) {
const diagnostic = readRemoteRunnerRecoveryError(error);
const target = input.target;
const exitedBeforeSessionOpen = input.claim.exitedBeforeSessionOpen === true;
if (input.providerAttempt > NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT || diagnostic?.stage !== "process_verification" ||
diagnostic.reason !== "process_not_running" ||
@@ -15292,6 +15310,7 @@ export async function verifyRemoteRunnerOrSettledExit(
: state.lifecycle !== "suspended" || !describeExitedRemoteRunner(state).runnerSettled) throw error;
return undefined;
}
return replaceUnanswered ? undefined : verified;
}
function runnerStateAllowsFreshStart(state: Record<string, unknown>): boolean {