fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 22:06:24 -06:00
1 parent 9b613562b7
commit 497ccba8f1
8 files changed
+389 -45

No files matched your search

@@ -103,6 +103,7 @@ export type NativeRestartRecoveryClaim =
/** Discovery evidence only; executor revalidates the encrypted header. */
stagedAttachment?: Pick<NativeStagedAttachmentRecovery, "sourceIdentity" | "targetIdentity" | "templateDigest">;
exitedBeforeSessionOpen?: true;
replaceUnansweredRunner?: true;
}
| {
kind: "resume_dead_runner";
@@ -2121,6 +2122,8 @@ export async function claimNativeRestartRecoveries(input: {
},
...(!pluginTask && row.coordinator.failureDetail?.runnerExitedBeforeSessionOpen === true
? { exitedBeforeSessionOpen: true as const } : {}),
...(!pluginTask && row.coordinator.failureDetail?.adoptedRunnerUnanswered === true
? { replaceUnansweredRunner: true as const } : {}),
} satisfies NativeRestartRecoveryClaim;
}
if (claimKind === "reattach_existing_runner") {
@@ -1573,6 +1573,39 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
expect(run?.processPid).toBe(remotePid);
});
it.each([true, false])("asks the same-run reattachment to replace a Runner that left its recovery read unanswered: %s", async (unanswered) => {
const fixture = await seedRun(`remote-unanswered-${unanswered}`);
await fixture.db.update(heartbeatRuns).set({
processPid: null,
processStartedAt: new Date("2020-01-01T00:00:00Z"),
runnerProfileJson: {
nativeWorkspaceSync: {
schema: "paperclip.native-workspace-sync/v1", state: "prepared",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64),
workspaceId: "workspace", leaseId: "lease", providerLeaseId: "sandbox",
remoteCwd: "/workspace",
},
sessionCheckpoint: {
identity: { companyId, agentId, issueId: fixture.issueId, runId: fixture.runId,
sessionId: fixture.native.normalizedSessionId },
providerSessionId: "provider-session",
process: { providerPid: process.pid },
},
},
}).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ attempt: 2, failureDetail: {
message: "native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 337 111s ago)",
recoveryMode: "exact_checkpoint_resume", checkpointExists: true,
...(unanswered ? { adoptedRunnerUnanswered: true } : {}),
} }).where(eq(nativeRunFinalizations.runId, fixture.runId));
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "graceful", runIds: [fixture.runId] });
expect(claim).toMatchObject({ kind: "reattach_remote_runner", runId: fixture.runId, providerAttempt: 2,
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } });
expect(claim?.kind === "reattach_remote_runner" ? claim.replaceUnansweredRunner : "not remote")
.toBe(unanswered ? true : undefined);
});
it("keeps a restart recovery's attempt credit while the agent directory is busy, then resumes at attempt 3", async () => {
const fixture = await seedRun("remote-busy-directory");
await fixture.db.update(heartbeatRuns).set({
@@ -476,6 +476,7 @@ import {
verifyRemoteRunnerReattachment,
stopStaleRemoteRunnerBeforeBootstrap,
verifyRemoteRunnerOrSettledExit,
nativeRecoveryCommandUnanswered,
readRemoteProviderPackManifest,
verifyRemoteProviderPackManifest,
prepareRemoteProviderPackOnTarget,
@@ -1141,6 +1142,54 @@ describe("remote controller restart adoption", () => {
target: exitedRunner(state) as never, identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught);
expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" });
});
describe("a same-run reattachment whose adopted Runner never answered its recovery read", () => {
const unanswered = { ...claim, replaceUnansweredRunner: true as const };
const midTurn = { lifecycle: "turn_active", outbox: [{ id: 338 }], pendingTerminalDelivery: null, pendingProviderCleanup: null };
it.each([2, 3])("launches a fresh Runner instead of adopting the live one again on attempt %i", async providerAttempt => {
const { target, execute } = fixture(midTurn);
await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity,
runId: "run", normalizedSessionId: "session", providerAttempt })).resolves.toBeUndefined();
expect(execute.mock.calls.some(([request]) => /kill -/.test(request.args.join(" ")))).toBe(false);
});
it("adopts the live Runner when the earlier attempt did not report an unanswered read", async () => {
const { target } = fixture(midTurn);
const adopted = await verifyRemoteRunnerOrSettledExit({ claim, target: target as never, identity,
runId: "run", normalizedSessionId: "session", providerAttempt: 2 });
expect(adopted?.pid).toBe(123);
});
it.each([
["a mid-turn exit whose provider and tools may survive it", midTurn, 3],
["a pending provider cleanup", { ...midTurn, pendingProviderCleanup: { pid: 7 } }, 3],
["an exhausted attempt budget", midTurn, 4],
] as const)("never resumes beside an exited unanswered Runner's lineage: %s", async (_case, state, providerAttempt) => {
const target = exitedRunner(state);
const error = await verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never,
identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught);
expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" });
});
it("keeps the live Runner's identity checks before replacing it", async () => {
const { target } = fixture({ ...midTurn, runnerInstanceId: "different" });
await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity,
runId: "run", normalizedSessionId: "session", providerAttempt: 2 })).rejects.toThrow("runner_remote_recovery_identity_mismatch");
});
it.each([
"native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 338 76s ago)",
"native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: runnerd exited while waiting for session.snapshot",
"native_session_recovery_failed: Error: runner_ingress_unavailable: PRP command runner.drain timed out (runner connected; pending ahead 0; last runner event seq 93 391s ago)",
])("recognizes the staging recovery read that got no answer: %s", message => {
expect(nativeRecoveryCommandUnanswered(new Error(message))).toBe(true);
expect(nativeRecoveryCommandUnanswered(new Error("wrapped", { cause: new Error(message) }))).toBe(true);
});
it.each([
"runnerd did not report the provider turn identity",
"provider_transport_failed: PRP command session.snapshot timed out",
"native_session_recovery_failed: provider resumed a different session",
"native_session_recovery_failed: provider_transport_failed: PRP command session.snapshot timed out; native_adopted_provider_identity_mismatch",
"native_session_recovery_failed: restart interruption history is incomplete",
])("does not replace a Runner for another failure: %s", message => {
expect(nativeRecoveryCommandUnanswered(new Error(message))).toBe(false);
});
});
it.each([
["an unsettled suspension", { ...settledSuspension, outbox: [{ id: 1 }] }, 2],
["a pending provider cleanup", { ...settledSuspension, pendingProviderCleanup: { pid: 7 } }, 2],
@@ -8860,6 +8860,16 @@ export function nativeRunnerExitedBeforeSessionOpen(error: unknown): boolean {
return nativeRunnerExitConfirmed(error) && runnerExitBeforeSessionOpenMessage(error);
}
export function nativeRecoveryCommandUnanswered(error: unknown, depth = 0): boolean {
if (!error || typeof error !== "object" || depth > 5) return false;
const value = error as Record<string, unknown>;
const message = typeof value.message === "string" ? value.message : "";
if (message.startsWith("native_session_recovery_failed:")
&& /PRP command [\w.]+ timed out|runnerd exited while waiting for [\w.]+/.test(message)
&& !/mismatch|different|identity|authority|integrity|incompatible/i.test(message)) return true;
return nativeRecoveryCommandUnanswered(value.cause, depth + 1);
}
/** Same provider, category and message (numbers ignored): a retry would meet the same fault. */
export function nativeStartupFailureSignature(failure: unknown): string | null {
const value = record(failure);
@@ -13054,6 +13064,9 @@ async function executePaperclipNativeSessionAttempt(
? { runnerExitConfirmed: true } : {}),
...(input.runnerExecutionTarget?.kind === "remote" && nativeRunnerExitedBeforeSessionOpen(error)
? { runnerExitedBeforeSessionOpen: true } : {}),
...(input.runnerExecutionTarget?.kind === "remote" && input.restartRecovery?.kind === "reattach_remote_runner"
&& input.restartRecovery.remote.kind !== "plugin_task" && nativeRecoveryCommandUnanswered(error)
? { adoptedRunnerUnanswered: true } : {}),
...(providerCooldownAt ? { providerCooldownAt: providerCooldownAt.toISOString() } : {}),
...(providerLimit?.plan.resetAt ? { providerLimitResetAt: providerLimit.plan.resetAt.toISOString() } : {}),
recoveryOwner: recoveryProjection.recoveryOwner,
@@ -14987,7 +15000,8 @@ const REMOTE_RUNNER_PROCESS_STDERR_TRIM_SCRIPT =
* A launch therefore stops a `paperclip-runnerd` that is still serving this
* exact `--state-dir`. A launch only happens after the server chose not to
* adopt a Runner: a same-run reattachment proves the marker's Runner gone
* first, and a transient reattachment failure never reaches a launch.
* first, or replaces a Runner that left the previous attempt's recovery read
* unanswered. Any other transient reattachment failure never reaches a launch.
*
* The stop runs in node ($1 the runnerd binary, $2 the state directory, then
* candidate PIDs):
@@ -15270,16 +15284,20 @@ async function readRemoteRunnerExitEvidence(runner: CommandManagedRuntimeRunner,
export async function verifyRemoteRunnerOrSettledExit(
input: Parameters<typeof verifyRemoteRunnerReattachment>[0] & { providerAttempt: number;
claim: { exitedBeforeSessionOpen?: true } },
claim: { exitedBeforeSessionOpen?: true; replaceUnansweredRunner?: true } },
) {
const recoveryDeadline = Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY,
Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS);
const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline);
const target = input.target;
const replaceUnanswered = input.claim.replaceUnansweredRunner === true
&& input.providerAttempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT
&& target?.kind === "remote" && target.transport === "sandbox" && Boolean(target.runner);
let verified: Awaited<ReturnType<typeof verifyRemoteRunnerReattachment>>;
try {
return await verifyRemoteRunnerReattachment({ ...input, recoveryDeadline, stall });
verified = await verifyRemoteRunnerReattachment({ ...input, recoveryDeadline, stall });
} catch (error) {
const diagnostic = readRemoteRunnerRecoveryError(error);
const target = input.target;
const exitedBeforeSessionOpen = input.claim.exitedBeforeSessionOpen === true;
if (input.providerAttempt > NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT || diagnostic?.stage !== "process_verification" ||
diagnostic.reason !== "process_not_running" ||
@@ -15292,6 +15310,7 @@ export async function verifyRemoteRunnerOrSettledExit(
: state.lifecycle !== "suspended" || !describeExitedRemoteRunner(state).runnerSettled) throw error;
return undefined;
}
return replaceUnanswered ? undefined : verified;
}
function runnerStateAllowsFreshStart(state: Record<string, unknown>): boolean {