From 86f9555695ef02ce3f09882c041fb2b6dd5199d7 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 01:27:35 -0600 Subject: [PATCH] fix(native): give a Runner whose first unanswered read came on the last attempt one replacement attempt (#21322) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs execute in a remote Runner, and a same-run retry reattaches that Runner after an interruption > - #21272 made the next attempt replace a Runner that left its recovery read unanswered, instead of adopting it again > - On staging, three runs after #21272 went live still failed: their first unanswered read came on the third and last attempt, so no next attempt existed > - This pull request gives such a run one more attempt, used only for that replacement > - The benefit is that the replacement from #21272 can run when a restart used up the middle attempt ## Linked Issues or Issue Description Refs #21272, Refs #21161 **What happened?** After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex runs still failed with `native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq … 55–84s ago)`. The run events of all three show the same sequence: 1. Attempt 1 fails at 06:24 with `runnerd did not report the provider turn identity`. 2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends it with `local_runtime_api_listener_unavailable`. This attempt sends no recovery read. 3. Attempt 3 adopts the same Runner. Its first recovery read gets no answer. Attempt 3 is the last of three, so the run ends. `adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt read it. The replacement from #21272 never ran. **Expected behavior** A Runner whose first unanswered read comes on the last attempt is still replaced once by a fresh Runner from the saved journal. The provider turn is not sent again. The run does not end only because a restart used the middle attempt. **Steps to reproduce** 1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at turn start, so the Runner stays alive. 2. Restart the server during attempt 2, so that attempt fails without a recovery read. 3. Let the Runner leave its recovery read unanswered on attempt 3. 4. See the run end with `session.snapshot timed out` and no replacement. **Paperclip version or commit** Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`. **Deployment mode** Hosted (staging), remote Daytona sandbox Runner. **Agent adapter(s) involved** Codex (native Runner, `paperclip-runnerd`). ## What Changed - `nativeUnansweredRunnerReplacementDisposition` (executor): when attempt 3 fails because its adopted live Runner left a recovery read unanswered, and that attempt was not itself a replacement, the failure stays `retryable_failure` (`native_session_interrupted`, retry in 30 s) instead of `native_session_retry_exhausted`. - `claimNativeRestartRecoveries`: a failed remote retry at attempt 3 with `adoptedRunnerUnanswered` is not treated as an exhausted budget. The claim is `reattach_remote_runner` at provider attempt 4 with `replaceUnansweredRunner`. - `nativeAdmittedProviderAttemptLimit` (executor admission gate): a `reattach_remote_runner` claim with `replaceUnansweredRunner` is admitted up to provider attempt 4. Every other claim keeps the limit of 3. - `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is allowed up to attempt 4. An exited Runner keeps the fail-closed path. - Bound: the extra attempt is a replacement (`replaceUnansweredRunner`), so it never gets another one. Attempt 4 is exhausted by the existing rule. - `doc/execution-semantics.md` describes the extra attempt. ## Verification - End to end (embedded Postgres): `native-remote-retry-admission.test.ts` › "executes the one replacement attempt after the last attempt's first unanswered recovery read, and no further one". It seeds attempt 3 with the staging failure, claims it, and executes the claim through `executePaperclipNativeSession`. The provider is reached once, the run then ends at attempt 4, and no further claim is made. With the old `nextAttempt > 3` gate this test fails before the provider is reached. - Red on `master`, green with the fix (7 tests, including "admits provider attempt 4 at execution only for the replacement claim"): - `native-runner-restart-recovery.integration.test.ts` › "gives a Runner whose first unanswered read came on the last attempt one replacement attempt: true" (embedded Postgres; the coordinator is seeded at attempt 3 with the staging message). On `master` no reattach claim was made. - `native-session-executor.test.ts` › "bounds the replacement after …" (4 cases: the last attempt's first unanswered read is retryable; a replacement that also went unanswered, another failure, and attempt 4 stay terminal) and "replaces the live Runner on the extra replacement attempt". - Focused suites: `native-session-executor`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped. - `pnpm typecheck` in `server` passes. - Not run locally: the container-gated remote Runner suites and the full suite (CI covers it). Why the remote Runner stops answering is a separate investigation and is not changed here. ## Risks - Low to medium. This adds at most one provider attempt per run, and only after an unanswered recovery read on a live adopted Runner. The attempt replaces the Runner through the launch fence, which proves the old Runner and its lineage gone or fails closed. No provider turn is sent again. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run events), extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 8 +++- .../native-remote-retry-admission.test.ts | 40 +++++++++++++++++++ .../native-runtime/native-restart-recovery.ts | 4 +- ...unner-restart-recovery.integration.test.ts | 36 +++++++++++++++++ .../native-session-executor.test.ts | 31 ++++++++++++++ .../native-runtime/native-session-executor.ts | 33 +++++++++++---- 6 files changed, 143 insertions(+), 9 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 50df9a08fcd..83f248ee413 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1799,7 +1799,13 @@ only while the old Runner is alive: the launch fence stops that the launch closed when it cannot prove them gone. A Runner that already exited mid-turn can leave provider or tool processes that the fence cannot find, so it keeps the fail-closed path below, and the task continues in a fresh session from -the saved work. The fresh Runner +the saved work. + +The first unanswered read can come on the last of the three provider attempts, +for example when a server restart used the second attempt. The run then gets one +more attempt for the replacement only: the failure stays retryable, the claim +admits attempt 4 with `replaceUnansweredRunner`, and a live Runner is replaced as +above. That attempt never gets another one; any failure on it ends the run. The fresh Runner restores the provider thread; its in-flight turn is reconciled from history and never submitted again. The provider attempt budget still bounds these attempts. diff --git a/server/src/services/native-runtime/native-remote-retry-admission.test.ts b/server/src/services/native-runtime/native-remote-retry-admission.test.ts index f34349e5ab1..0a67db1d02f 100644 --- a/server/src/services/native-runtime/native-remote-retry-admission.test.ts +++ b/server/src/services/native-runtime/native-remote-retry-admission.test.ts @@ -461,6 +461,46 @@ describeEmbeddedPostgres("remote native retry execution admission", () => { processPid: 1386, runnerProfileJson: { sessionCheckpoint: f.checkpoint } }); }); + it("executes the one replacement attempt after the last attempt's first unanswered recovery read, and no further one", async () => { + const f = await failedRemoteRun("remote"); + await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId)); + const [issue] = await f.db.select().from(issues).where(eq(issues.id, f.issueId)); + const [contract] = await f.db.select().from(completionContracts).where(eq(completionContracts.issueId, f.issueId)); + const execution: NativeExecutionInputV1 = { + schema: "paperclip.native-execution-input.v1", + binding: { companyId: f.companyId, runId: f.runId, issueId: f.issueId, agentId: f.agentId, executionWorkspaceId: f.runId }, + provider: { kind: "codex", model: "gpt-5.6-luna" }, + task: { identifier: issue.identifier!, title: issue.title, description: null, prompt: "Continue retained work.", workMode: "standard" }, + workspace: { cwd: environment.current.runtimeRoot, repoUrl: null, repoRef: null, branchName: null }, + session: { normalizedSessionId: f.native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", + contract: buildNativeCompletionContract(issue, { revision: contract.revision }) }, + interactionResponses: [], credentialBindings: [], + }; + await f.db.update(heartbeatRuns).set({ runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) + || ${JSON.stringify({ nativeExecutionInput: execution })}::jsonb` }).where(eq(heartbeatRuns.id, f.runId)); + await f.db.update(nativeRunFinalizations).set({ attempt: 3, failureDetail: { + message: "native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 412 84s ago)", + recoveryMode: "exact_checkpoint_resume", providerSessionEstablished: true, checkpointExists: true, + providerEventsExist: true, recoverable: true, adoptedRunnerUnanswered: true, + } }).where(eq(nativeRunFinalizations.runId, f.runId)); + provider.execute.mockReset().mockRejectedValue(new Error("replacement attempt reached the provider session")); + const controller = await currentNativeControllerIdentity(); + const [claim] = await claimNativeRestartRecoveries({ db: f.db, controller, restartKind: "hard", runIds: [f.runId] }); + expect(claim).toMatchObject({ kind: "reattach_remote_runner", providerAttempt: 4, replaceUnansweredRunner: true }); + if (claim.kind !== "reattach_remote_runner") throw new Error("Expected the replacement claim"); + await expect(executePaperclipNativeSession({ db: f.db, execution, runnerInstanceId: f.native.runnerInstanceId, + leaseOwner: claim.leaseOwner, restartRecovery: claim, + backend: { descriptor: async () => { throw new Error("Unexpected descriptor call"); }, + openSession: async () => { throw new Error("Unexpected session launch"); } }, + })).rejects.toThrow("replacement attempt reached the provider session"); + expect(provider.execute).toHaveBeenCalledTimes(1); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]) + .toMatchObject({ attempt: 4, phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null }); + expect(await claimNativeRestartRecoveries({ db: f.db, controller, restartKind: "hard", runIds: [f.runId] })).toEqual([]); + }); + it.each((["remote", "plugin"] as const).flatMap(storage => (["released", "retained"] as const).map(lock => [storage, lock] as const))) ("does not admit another failed %s retry when the persisted provider budget is already exhausted with a %s task lock", async (storage, lock) => { const f = await failedRemoteRun(storage); diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index 76ea996d941..cd63032c87a 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -1252,7 +1252,9 @@ export async function claimNativeRestartRecoveries(input: { const lockLostBootstrapRetry = row.run.status === "failed" && row.coordinator.phase === "retryable_failure" && row.coordinator.failureDetail?.recoveryMode === "bootstrap_retry" && row.issueExecutionRunId !== row.run.id; const releasedRemoteRetry = failedRemoteRetry && row.issueExecutionRunId === null; - const exhaustedRemoteRetry = failedRemoteRetry && row.coordinator.attempt >= 3; + const unansweredRunnerReplacement = row.coordinator.failureDetail?.adoptedRunnerUnanswered === true + && row.coordinator.attempt === NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT; + const exhaustedRemoteRetry = failedRemoteRetry && row.coordinator.attempt >= 3 && !unansweredRunnerReplacement; const needsResumptionAdmission = releasedRemoteRetry || exhaustedRemoteRetry || (failedRemoteRetry && ["done", "cancelled"].includes(issue.status)); const admission = { exhausted: null as { reason: string; policy: SystemRecoveryState } | null }; diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 625ea711774..cbbe0422a00 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -1575,6 +1575,42 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); expect(run?.processPid).toBe(remotePid); }); + it.each([true, false])("gives a Runner whose first unanswered read came on the last attempt one replacement attempt: %s", async (unanswered) => { + const fixture = await seedRun(`remote-final-unanswered-${unanswered}`); + const now = new Date(); + await fixture.db.update(heartbeatRuns).set({ status: "failed", finishedAt: now, + nativePhase: "retryable_failure", runnerProfileJson: { ...fixture.preparedProfile, + nativeWorkspaceSync: remotePreparedWorkspace(), + sessionCheckpoint: { identity: { companyId, agentId, issueId: fixture.issueId, runId: fixture.runId, + sessionId: fixture.native.normalizedSessionId }, providerSessionId: "final-unanswered-session" }, + } }).where(eq(heartbeatRuns.id, fixture.runId)); + await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId, + leaseOwner: "predecessor-final-unanswered", leaseExpiresAt: new Date(now.getTime() + 60_000) }) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + const authority = new DurablePrpControlPlane({ stateDirectory: resolve(runtimeRoot, fixture.runId), + identity: transportOptions(fixture, runtimeRoot).prpIdentity, + expectedRunnerVersion: "final-unanswered-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + try { + await registerNativeRunnerCheckpoint({ db: fixture.db, + binding: { companyId, issueId: fixture.issueId, runId: fixture.runId }, bootId: successor.bootId, authority }); + } finally { await authority.stop(); } + await fixture.db.update(nativeRunFinalizations).set({ phase: "retryable_failure", attempt: 3, + leaseOwner: null, leaseExpiresAt: null, nextAttemptAt: new Date(0), failureCode: "native_session_interrupted", + failureDetail: { message: "native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 412 84s ago)", + recoveryMode: "exact_checkpoint_resume", checkpointExists: true, recoverable: true, + providerSessionEstablished: true, providerEventsExist: true, + ...(unanswered ? { adoptedRunnerUnanswered: true } : {}) } }) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId], now }); + if (unanswered) { + expect(claim).toMatchObject({ kind: "reattach_remote_runner", runId: fixture.runId, providerAttempt: 4, + replaceUnansweredRunner: true }); + } else { + expect(claim?.kind).not.toBe("reattach_remote_runner"); + } + }); + it.each([true, false])("asks the same-run reattachment to replace a Runner that left its recovery read unanswered: %s", async (unanswered) => { const fixture = await seedRun(`remote-unanswered-${unanswered}`); await fixture.db.update(heartbeatRuns).set({ diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 1f05f6623b9..52518b89559 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -477,6 +477,8 @@ import { stopStaleRemoteRunnerBeforeBootstrap, verifyRemoteRunnerOrSettledExit, nativeRecoveryCommandUnanswered, + nativeUnansweredRunnerReplacementDisposition, + nativeAdmittedProviderAttemptLimit, readRemoteProviderPackManifest, verifyRemoteProviderPackManifest, prepareRemoteProviderPackOnTarget, @@ -1172,6 +1174,35 @@ describe("remote controller restart adoption", () => { await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity, runId: "run", normalizedSessionId: "session", providerAttempt: 2 })).rejects.toThrow("runner_remote_recovery_identity_mismatch"); }); + it.each([ + ["the last attempt's first unanswered read", { adoptedRunnerUnanswered: true, replacing: false }, 3, "retryable_failure"], + ["a replacement that also went unanswered", { adoptedRunnerUnanswered: true, replacing: true }, 3, "terminal_failure"], + ["another failure on the last attempt", { adoptedRunnerUnanswered: false, replacing: false }, 3, "terminal_failure"], + ["the extra replacement attempt", { adoptedRunnerUnanswered: true, replacing: false }, 4, "terminal_failure"], + ] as const)("bounds the replacement after %s", (_case, input, attempt, phase) => { + const now = new Date("2026-11-13T06:37:26.000Z"); + const disposition = nativeUnansweredRunnerReplacementDisposition( + nativeSessionFailureDisposition(attempt, now, "provider_transport_failed"), { ...input, attempt, now }); + expect(disposition.phase).toBe(phase); + if (phase === "retryable_failure") { + expect(disposition).toEqual({ phase, failureCode: "native_session_interrupted", nextAttemptAt: new Date(now.getTime() + 30_000) }); + } + }); + it("admits provider attempt 4 at execution only for the replacement claim", () => { + const claim = { kind: "reattach_remote_runner" as const, runId: "run", leaseOwner: "owner", controllerGeneration: 6, + providerAttempt: 4, restartKind: "hard" as const, recoveryRequestId: null, + remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } }; + expect(nativeAdmittedProviderAttemptLimit({ ...claim, replaceUnansweredRunner: true })).toBe(4); + expect(nativeAdmittedProviderAttemptLimit(claim)).toBe(3); + expect(nativeAdmittedProviderAttemptLimit({ kind: "resume_dead_runner", runId: "run", leaseOwner: "owner", + controllerGeneration: 6, providerAttempt: 4, restartKind: "hard", recoveryRequestId: null })).toBe(3); + expect(nativeAdmittedProviderAttemptLimit(null)).toBe(3); + }); + it("replaces the live Runner on the extra replacement attempt", async () => { + const { target } = fixture(midTurn); + await expect(verifyRemoteRunnerOrSettledExit({ claim: unanswered, target: target as never, identity, + runId: "run", normalizedSessionId: "session", providerAttempt: 4 })).resolves.toBeUndefined(); + }); it.each([ "native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 338 76s ago)", "native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: runnerd exited while waiting for session.snapshot", diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 00ac5f87600..f74002866cd 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -8797,6 +8797,21 @@ export function nativeSessionFailureDisposition( }; } +export function nativeAdmittedProviderAttemptLimit(claim: NativeRestartRecoveryClaim | null | undefined): number { + return claim?.kind === "reattach_remote_runner" && claim.replaceUnansweredRunner === true + ? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT; +} + +export function nativeUnansweredRunnerReplacementDisposition( + disposition: ReturnType, + input: { attempt: number; now: Date; adoptedRunnerUnanswered: boolean; replacing: boolean }, +): ReturnType { + if (!input.adoptedRunnerUnanswered || input.replacing || input.attempt !== NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + || disposition.failureCode !== "native_session_retry_exhausted") return disposition; + return { phase: "retryable_failure", failureCode: "native_session_interrupted", + nextAttemptAt: new Date(input.now.getTime() + 30_000) }; +} + export function nativeSessionRecoveryProjection(input: { phase: "retryable_failure" | "terminal_failure"; failureCode: string; @@ -11134,7 +11149,7 @@ async function executePaperclipNativeSessionAttempt( recovering?.kind, !recovering || !("chargesProviderAttempt" in recovering) || recovering.chargesProviderAttempt !== false, ); - if (nextAttempt > 3) + if (nextAttempt > nativeAdmittedProviderAttemptLimit(recovering)) throw new Error("native_session_retry_exhausted"); const scheduledResumeDecision = await scheduledResumeContinuationAllowed(tx as unknown as Db, { companyId: input.execution.binding.companyId, @@ -12937,11 +12952,17 @@ async function executePaperclipNativeSessionAttempt( sourceFailureCode, bootstrap, }); - const disposition = nativeSessionFailureDisposition( + const adoptedRunnerUnanswered = input.runnerExecutionTarget?.kind === "remote" + && input.restartRecovery?.kind === "reattach_remote_runner" + && input.restartRecovery.remote.kind !== "plugin_task" && nativeRecoveryCommandUnanswered(error); + const disposition = nativeUnansweredRunnerReplacementDisposition(nativeSessionFailureDisposition( attempt, now, sourceFailureCode, - ); + ), { + attempt, now, adoptedRunnerUnanswered, + replacing: input.restartRecovery?.kind === "reattach_remote_runner" && input.restartRecovery.replaceUnansweredRunner === true, + }); const phase = recoveryEvidence.recoveryMode === "ambiguous_state" ? ("terminal_failure" as const) @@ -13075,9 +13096,7 @@ async function executePaperclipNativeSessionAttempt( ? { runnerExitConfirmed: true } : {}), ...(input.runnerExecutionTarget?.kind === "remote" && nativeRunnerExitedBeforeSessionOpen(error) ? { runnerExitedBeforeSessionOpen: true } : {}), - ...(input.runnerExecutionTarget?.kind === "remote" && input.restartRecovery?.kind === "reattach_remote_runner" - && input.restartRecovery.remote.kind !== "plugin_task" && nativeRecoveryCommandUnanswered(error) - ? { adoptedRunnerUnanswered: true } : {}), + ...(adoptedRunnerUnanswered ? { adoptedRunnerUnanswered: true } : {}), ...(providerCooldownAt ? { providerCooldownAt: providerCooldownAt.toISOString() } : {}), ...(providerLimit?.plan.resetAt ? { providerLimitResetAt: providerLimit.plan.resetAt.toISOString() } : {}), recoveryOwner: recoveryProjection.recoveryOwner, @@ -15304,7 +15323,7 @@ export async function verifyRemoteRunnerOrSettledExit( const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline); const target = input.target; const replaceUnanswered = input.claim.replaceUnansweredRunner === true - && input.providerAttempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + && input.providerAttempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 && target?.kind === "remote" && target.transport === "sandbox" && Boolean(target.runner); let verified: Awaited>; try {