fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner, and a same-run retry
reattaches that Runner after an interruption
> - #21272 made the next attempt replace a Runner that left its recovery
read unanswered, instead of adopting it again
> - On staging, three runs after #21272 went live still failed: their
first unanswered read came on the third and last attempt, so no next
attempt existed
> - This pull request gives such a run one more attempt, used only for
that replacement
> - The benefit is that the replacement from #21272 can run when a
restart used up the middle attempt

## Linked Issues or Issue Description

Refs #21272, Refs #21161

**What happened?**

After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex
runs still failed with `native_session_recovery_failed:
HarnessCapabilityUnavailableError: read is unavailable: Error:
provider_transport_failed: PRP command session.snapshot timed out
(runner connected; pending ahead 0; last runner event seq … 55–84s
ago)`. The run events of all three show the same sequence:

1. Attempt 1 fails at 06:24 with `runnerd did not report the provider
turn identity`.
2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends
it with `local_runtime_api_listener_unavailable`. This attempt sends no
recovery read.
3. Attempt 3 adopts the same Runner. Its first recovery read gets no
answer. Attempt 3 is the last of three, so the run ends.

`adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt
read it. The replacement from #21272 never ran.

**Expected behavior**

A Runner whose first unanswered read comes on the last attempt is still
replaced once by a fresh Runner from the saved journal. The provider
turn is not sent again. The run does not end only because a restart used
the middle attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at
turn start, so the Runner stays alive.
2. Restart the server during attempt 2, so that attempt fails without a
recovery read.
3. Let the Runner leave its recovery read unanswered on attempt 3.
4. See the run end with `session.snapshot timed out` and no replacement.

**Paperclip version or commit**

Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `nativeUnansweredRunnerReplacementDisposition` (executor): when
attempt 3 fails because its adopted live Runner left a recovery read
unanswered, and that attempt was not itself a replacement, the failure
stays `retryable_failure` (`native_session_interrupted`, retry in 30 s)
instead of `native_session_retry_exhausted`.
- `claimNativeRestartRecoveries`: a failed remote retry at attempt 3
with `adoptedRunnerUnanswered` is not treated as an exhausted budget.
The claim is `reattach_remote_runner` at provider attempt 4 with
`replaceUnansweredRunner`.
- `nativeAdmittedProviderAttemptLimit` (executor admission gate): a
`reattach_remote_runner` claim with `replaceUnansweredRunner` is
admitted up to provider attempt 4. Every other claim keeps the limit of
3.
- `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is
allowed up to attempt 4. An exited Runner keeps the fail-closed path.
- Bound: the extra attempt is a replacement (`replaceUnansweredRunner`),
so it never gets another one. Attempt 4 is exhausted by the existing
rule.
- `doc/execution-semantics.md` describes the extra attempt.

## Verification

- End to end (embedded Postgres):
`native-remote-retry-admission.test.ts` › "executes the one replacement
attempt after the last attempt's first unanswered recovery read, and no
further one". It seeds attempt 3 with the staging failure, claims it,
and executes the claim through `executePaperclipNativeSession`. The
provider is reached once, the run then ends at attempt 4, and no further
claim is made. With the old `nextAttempt > 3` gate this test fails
before the provider is reached.
- Red on `master`, green with the fix (7 tests, including "admits
provider attempt 4 at execution only for the replacement claim"):
- `native-runner-restart-recovery.integration.test.ts` › "gives a Runner
whose first unanswered read came on the last attempt one replacement
attempt: true" (embedded Postgres; the coordinator is seeded at attempt
3 with the staging message). On `master` no reattach claim was made.
- `native-session-executor.test.ts` › "bounds the replacement after …"
(4 cases: the last attempt's first unanswered read is retryable; a
replacement that also went unanswered, another failure, and attempt 4
stay terminal) and "replaces the live Runner on the extra replacement
attempt".
- Focused suites: `native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped.
- `pnpm typecheck` in `server` passes.
- Not run locally: the container-gated remote Runner suites and the full
suite (CI covers it). Why the remote Runner stops answering is a
separate investigation and is not changed here.

## Risks

- Low to medium. This adds at most one provider attempt per run, and
only after an unanswered recovery read on a live adopted Runner. The
attempt replaces the Runner through the launch fence, which proves the
old Runner and its lineage gone or fails closed. No provider turn is
sent again.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events),
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 01:27:35 -06:00
1 parent 49f0030f06
commit 86f9555695
6 files changed
+143 -9

No files matched your search

+7 -1
View File
@@ -1799,7 +1799,13 @@ only while the old Runner is alive: the launch fence stops that
the launch closed when it cannot prove them gone. A Runner that already exited
mid-turn can leave provider or tool processes that the fence cannot find, so it
keeps the fail-closed path below, and the task continues in a fresh session from
the saved work. The fresh Runner
the saved work.
The first unanswered read can come on the last of the three provider attempts,
for example when a server restart used the second attempt. The run then gets one
more attempt for the replacement only: the failure stays retryable, the claim
admits attempt 4 with `replaceUnansweredRunner`, and a live Runner is replaced as
above. That attempt never gets another one; any failure on it ends the run. The fresh Runner
restores the provider thread; its in-flight turn is reconciled from history and
never submitted again. The provider attempt budget still bounds these attempts.
@@ -461,6 +461,46 @@ describeEmbeddedPostgres("remote native retry execution admission", () => {
processPid: 1386, runnerProfileJson: { sessionCheckpoint: f.checkpoint } });
});
it("executes the one replacement attempt after the last attempt's first unanswered recovery read, and no further one", async () => {
const f = await failedRemoteRun("remote");
await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId));
const [issue] = await f.db.select().from(issues).where(eq(issues.id, f.issueId));
const [contract] = await f.db.select().from(completionContracts).where(eq(completionContracts.issueId, f.issueId));
const execution: NativeExecutionInputV1 = {
schema: "paperclip.native-execution-input.v1",
binding: { companyId: f.companyId, runId: f.runId, issueId: f.issueId, agentId: f.agentId, executionWorkspaceId: f.runId },
provider: { kind: "codex", model: "gpt-5.6-luna" },
task: { identifier: issue.identifier!, title: issue.title, description: null, prompt: "Continue retained work.", workMode: "standard" },
workspace: { cwd: environment.current.runtimeRoot, repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: f.native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1,
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
interactionResponses: [], credentialBindings: [],
};
await f.db.update(heartbeatRuns).set({ runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb)
|| ${JSON.stringify({ nativeExecutionInput: execution })}::jsonb` }).where(eq(heartbeatRuns.id, f.runId));
await f.db.update(nativeRunFinalizations).set({ attempt: 3, failureDetail: {
message: "native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 412 84s ago)",
recoveryMode: "exact_checkpoint_resume", providerSessionEstablished: true, checkpointExists: true,
providerEventsExist: true, recoverable: true, adoptedRunnerUnanswered: true,
} }).where(eq(nativeRunFinalizations.runId, f.runId));
provider.execute.mockReset().mockRejectedValue(new Error("replacement attempt reached the provider session"));
const controller = await currentNativeControllerIdentity();
const [claim] = await claimNativeRestartRecoveries({ db: f.db, controller, restartKind: "hard", runIds: [f.runId] });
expect(claim).toMatchObject({ kind: "reattach_remote_runner", providerAttempt: 4, replaceUnansweredRunner: true });
if (claim.kind !== "reattach_remote_runner") throw new Error("Expected the replacement claim");
await expect(executePaperclipNativeSession({ db: f.db, execution, runnerInstanceId: f.native.runnerInstanceId,
leaseOwner: claim.leaseOwner, restartRecovery: claim,
backend: { descriptor: async () => { throw new Error("Unexpected descriptor call"); },
openSession: async () => { throw new Error("Unexpected session launch"); } },
})).rejects.toThrow("replacement attempt reached the provider session");
expect(provider.execute).toHaveBeenCalledTimes(1);
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
.toMatchObject({ attempt: 4, phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null });
expect(await claimNativeRestartRecoveries({ db: f.db, controller, restartKind: "hard", runIds: [f.runId] })).toEqual([]);
});
it.each((["remote", "plugin"] as const).flatMap(storage => (["released", "retained"] as const).map(lock => [storage, lock] as const)))
("does not admit another failed %s retry when the persisted provider budget is already exhausted with a %s task lock", async (storage, lock) => {
const f = await failedRemoteRun(storage);
@@ -1252,7 +1252,9 @@ export async function claimNativeRestartRecoveries(input: {
const lockLostBootstrapRetry = row.run.status === "failed" && row.coordinator.phase === "retryable_failure"
&& row.coordinator.failureDetail?.recoveryMode === "bootstrap_retry" && row.issueExecutionRunId !== row.run.id;
const releasedRemoteRetry = failedRemoteRetry && row.issueExecutionRunId === null;
const exhaustedRemoteRetry = failedRemoteRetry && row.coordinator.attempt >= 3;
const unansweredRunnerReplacement = row.coordinator.failureDetail?.adoptedRunnerUnanswered === true
&& row.coordinator.attempt === NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT;
const exhaustedRemoteRetry = failedRemoteRetry && row.coordinator.attempt >= 3 && !unansweredRunnerReplacement;
const needsResumptionAdmission = releasedRemoteRetry || exhaustedRemoteRetry ||
(failedRemoteRetry && ["done", "cancelled"].includes(issue.status));
const admission = { exhausted: null as { reason: string; policy: SystemRecoveryState } | null };
@@ -1575,6 +1575,42 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
expect(run?.processPid).toBe(remotePid);
});
it.each([true, false])("gives a Runner whose first unanswered read came on the last attempt one replacement attempt: %s", async (unanswered) => {
const fixture = await seedRun(`remote-final-unanswered-${unanswered}`);
const now = new Date();
await fixture.db.update(heartbeatRuns).set({ status: "failed", finishedAt: now,
nativePhase: "retryable_failure", runnerProfileJson: { ...fixture.preparedProfile,
nativeWorkspaceSync: remotePreparedWorkspace(),
sessionCheckpoint: { identity: { companyId, agentId, issueId: fixture.issueId, runId: fixture.runId,
sessionId: fixture.native.normalizedSessionId }, providerSessionId: "final-unanswered-session" },
} }).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId,
leaseOwner: "predecessor-final-unanswered", leaseExpiresAt: new Date(now.getTime() + 60_000) })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
const authority = new DurablePrpControlPlane({ stateDirectory: resolve(runtimeRoot, fixture.runId),
identity: transportOptions(fixture, runtimeRoot).prpIdentity,
expectedRunnerVersion: "final-unanswered-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
try {
await registerNativeRunnerCheckpoint({ db: fixture.db,
binding: { companyId, issueId: fixture.issueId, runId: fixture.runId }, bootId: successor.bootId, authority });
} finally { await authority.stop(); }
await fixture.db.update(nativeRunFinalizations).set({ phase: "retryable_failure", attempt: 3,
leaseOwner: null, leaseExpiresAt: null, nextAttemptAt: new Date(0), failureCode: "native_session_interrupted",
failureDetail: { message: "native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 412 84s ago)",
recoveryMode: "exact_checkpoint_resume", checkpointExists: true, recoverable: true,
providerSessionEstablished: true, providerEventsExist: true,
...(unanswered ? { adoptedRunnerUnanswered: true } : {}) } })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId], now });
if (unanswered) {
expect(claim).toMatchObject({ kind: "reattach_remote_runner", runId: fixture.runId, providerAttempt: 4,
replaceUnansweredRunner: true });
} else {
expect(claim?.kind).not.toBe("reattach_remote_runner");
}
});
it.each([true, false])("asks the same-run reattachment to replace a Runner that left its recovery read unanswered: %s", async (unanswered) => {
const fixture = await seedRun(`remote-unanswered-${unanswered}`);
await fixture.db.update(heartbeatRuns).set({
@@ -477,6 +477,8 @@ import {
stopStaleRemoteRunnerBeforeBootstrap,
verifyRemoteRunnerOrSettledExit,
nativeRecoveryCommandUnanswered,
nativeUnansweredRunnerReplacementDisposition,
nativeAdmittedProviderAttemptLimit,
readRemoteProviderPackManifest,
verifyRemoteProviderPackManifest,
prepareRemoteProviderPackOnTarget,
@@ -1172,6 +1174,35 @@ describe("remote controller restart adoption", () => {
await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity,
runId: "run", normalizedSessionId: "session", providerAttempt: 2 })).rejects.toThrow("runner_remote_recovery_identity_mismatch");
});
it.each([
["the last attempt's first unanswered read", { adoptedRunnerUnanswered: true, replacing: false }, 3, "retryable_failure"],
["a replacement that also went unanswered", { adoptedRunnerUnanswered: true, replacing: true }, 3, "terminal_failure"],
["another failure on the last attempt", { adoptedRunnerUnanswered: false, replacing: false }, 3, "terminal_failure"],
["the extra replacement attempt", { adoptedRunnerUnanswered: true, replacing: false }, 4, "terminal_failure"],
] as const)("bounds the replacement after %s", (_case, input, attempt, phase) => {
const now = new Date("2026-11-13T06:37:26.000Z");
const disposition = nativeUnansweredRunnerReplacementDisposition(
nativeSessionFailureDisposition(attempt, now, "provider_transport_failed"), { ...input, attempt, now });
expect(disposition.phase).toBe(phase);
if (phase === "retryable_failure") {
expect(disposition).toEqual({ phase, failureCode: "native_session_interrupted", nextAttemptAt: new Date(now.getTime() + 30_000) });
}
});
it("admits provider attempt 4 at execution only for the replacement claim", () => {
const claim = { kind: "reattach_remote_runner" as const, runId: "run", leaseOwner: "owner", controllerGeneration: 6,
providerAttempt: 4, restartKind: "hard" as const, recoveryRequestId: null,
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } };
expect(nativeAdmittedProviderAttemptLimit({ ...claim, replaceUnansweredRunner: true })).toBe(4);
expect(nativeAdmittedProviderAttemptLimit(claim)).toBe(3);
expect(nativeAdmittedProviderAttemptLimit({ kind: "resume_dead_runner", runId: "run", leaseOwner: "owner",
controllerGeneration: 6, providerAttempt: 4, restartKind: "hard", recoveryRequestId: null })).toBe(3);
expect(nativeAdmittedProviderAttemptLimit(null)).toBe(3);
});
it("replaces the live Runner on the extra replacement attempt", async () => {
const { target } = fixture(midTurn);
await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity,
runId: "run", normalizedSessionId: "session", providerAttempt: 4 })).resolves.toBeUndefined();
});
it.each([
"native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 338 76s ago)",
"native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: runnerd exited while waiting for session.snapshot",
@@ -8797,6 +8797,21 @@ export function nativeSessionFailureDisposition(
};
}
export function nativeAdmittedProviderAttemptLimit(claim: NativeRestartRecoveryClaim | null | undefined): number {
return claim?.kind === "reattach_remote_runner" && claim.replaceUnansweredRunner === true
? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT;
}
export function nativeUnansweredRunnerReplacementDisposition(
disposition: ReturnType<typeof nativeSessionFailureDisposition>,
input: { attempt: number; now: Date; adoptedRunnerUnanswered: boolean; replacing: boolean },
): ReturnType<typeof nativeSessionFailureDisposition> {
if (!input.adoptedRunnerUnanswered || input.replacing || input.attempt !== NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT
|| disposition.failureCode !== "native_session_retry_exhausted") return disposition;
return { phase: "retryable_failure", failureCode: "native_session_interrupted",
nextAttemptAt: new Date(input.now.getTime() + 30_000) };
}
export function nativeSessionRecoveryProjection(input: {
phase: "retryable_failure" | "terminal_failure";
failureCode: string;
@@ -11134,7 +11149,7 @@ async function executePaperclipNativeSessionAttempt(
recovering?.kind,
!recovering || !("chargesProviderAttempt" in recovering) || recovering.chargesProviderAttempt !== false,
);
if (nextAttempt > 3)
if (nextAttempt > nativeAdmittedProviderAttemptLimit(recovering))
throw new Error("native_session_retry_exhausted");
const scheduledResumeDecision = await scheduledResumeContinuationAllowed(tx as unknown as Db, {
companyId: input.execution.binding.companyId,
@@ -12937,11 +12952,17 @@ async function executePaperclipNativeSessionAttempt(
sourceFailureCode,
bootstrap,
});
const disposition = nativeSessionFailureDisposition(
const adoptedRunnerUnanswered = input.runnerExecutionTarget?.kind === "remote"
&& input.restartRecovery?.kind === "reattach_remote_runner"
&& input.restartRecovery.remote.kind !== "plugin_task" && nativeRecoveryCommandUnanswered(error);
const disposition = nativeUnansweredRunnerReplacementDisposition(nativeSessionFailureDisposition(
attempt,
now,
sourceFailureCode,
);
), {
attempt, now, adoptedRunnerUnanswered,
replacing: input.restartRecovery?.kind === "reattach_remote_runner" && input.restartRecovery.replaceUnansweredRunner === true,
});
const phase =
recoveryEvidence.recoveryMode === "ambiguous_state"
? ("terminal_failure" as const)
@@ -13075,9 +13096,7 @@ async function executePaperclipNativeSessionAttempt(
? { runnerExitConfirmed: true } : {}),
...(input.runnerExecutionTarget?.kind === "remote" && nativeRunnerExitedBeforeSessionOpen(error)
? { runnerExitedBeforeSessionOpen: true } : {}),
...(input.runnerExecutionTarget?.kind === "remote" && input.restartRecovery?.kind === "reattach_remote_runner"
&& input.restartRecovery.remote.kind !== "plugin_task" && nativeRecoveryCommandUnanswered(error)
? { adoptedRunnerUnanswered: true } : {}),
...(adoptedRunnerUnanswered ? { adoptedRunnerUnanswered: true } : {}),
...(providerCooldownAt ? { providerCooldownAt: providerCooldownAt.toISOString() } : {}),
...(providerLimit?.plan.resetAt ? { providerLimitResetAt: providerLimit.plan.resetAt.toISOString() } : {}),
recoveryOwner: recoveryProjection.recoveryOwner,
@@ -15304,7 +15323,7 @@ export async function verifyRemoteRunnerOrSettledExit(
const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline);
const target = input.target;
const replaceUnanswered = input.claim.replaceUnansweredRunner === true
&& input.providerAttempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT
&& input.providerAttempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1
&& target?.kind === "remote" && target.transport === "sandbox" && Boolean(target.runner);
let verified: Awaited<ReturnType<typeof verifyRemoteRunnerReattachment>>;
try {