From eb54831d2891cdee7aca9db1e9b6f344ef5b74bd Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 06:53:38 -0600 Subject: [PATCH] fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs often use a remote Runner in a sandbox. After a restart, a deploy or a same-run retry, the server reattaches to that Runner or starts a new Runner over its saved state > - A Runner that no controller reaches before its connection lease expires stops its provider, saves its state as `recoverable_failure` with `lease_expired_requires_bootstrap`, and exits. The Runner protocol says that only a new process with a fresh bootstrap ticket can reconnect, and that the new process continues from the same cursor > - Reattachment started a new Runner over saved state only for a settled `suspended` exit (or an exit before `session.open`). So it refused this exit, and the run ended with `runner_remote_process_identity_unavailable` (`native_continuation_requires_reconciliation`) > - This pull request lets reattachment launch a Runner over the exact saved state of a Runner that exited to ask for a fresh bootstrap > - The benefit is that the run continues by itself after a long controller gap, with no Retry and no manual sandbox repair ## Linked Issues or Issue Description Refs #21329, #21362, #20853 **What happened?** On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on remote Daytona Runners ended with `native_continuation_requires_reconciliation` and the error `runner_remote_process_identity_unavailable`, about 5 per hour, after #21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df, dd595934, 50c6e6e6) ended the same way. The last `native.remote_runner_recovery_failed` event recorded `process_verification` / `process_not_running`, `runnerLifecycle: recoverable_failure`, `runnerRecoverableFailure: lease_expired_requires_bootstrap`, and the Runner diagnostic "connection lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the 15 runs that ended with this error had this exact exit. Timeline of run c6fabe9a: - 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded sandbox. - 10:54:02: the controller sent `turn.start`. At 10:54:47 the command timed out: "PRP command turn.start timed out (runner connected; pending ahead 0; last runner event seq 2 91s ago)". The run resumed in the same run and the turn then ran (tools completed from 11:14). - 11:39: a graceful server shutdown suspended the controller. At 11:43 the next controller started the reattachment, but it did not reconnect to the Runner until the server restarted again at 11:58. In that gap the Runner's lease expired and the Runner exited. - 11:58:53: the reattachment proved the process gone and refused the `recoverable_failure` state. The run failed at 11:59. Why `turn.start` timed out while the Runner was connected: the Runner answers `turn.start` only after Codex answers its `turn/start` JSON-RPC request. While it waits, the Runner buffers Codex notifications and commits no events, so the controller sees a quiet connected Runner with nothing pending ahead. The controller's bound for the `turn.start` command result is a fixed 30 s. #21329 changed only the later wait for `turn/started`. In a sandbox that needed 8 minutes to start, Codex answered later than 30 s. That timeout is recoverable, and the run did recover from it. This pull request does not change it (see Risks). The failure that ended the runs is the refused reattachment. **Expected behavior** When a remote Runner exits because its lease expired (or its reconnect grace or deadline ended) and its saved state is intact, reattachment starts a new Runner from that state and the run continues. **Steps to reproduce** 1. Start a native Codex run on a remote sandbox Runner. 2. Keep every controller from reaching the Runner for longer than its connection lease (on staging: a deploy that stalled the reattachment for about 15 minutes). 3. The Runner saves `recoverable_failure` / `lease_expired_requires_bootstrap` and exits. 4. The next reattachment fails with `runner_remote_process_identity_unavailable`, and the run ends. **Agent adapter(s) involved** `paperclip_runner` with Codex on a Daytona remote sandbox. The rule is not provider-specific. ## What Changed - `remote-runner-recovery-diagnostics.ts`: new `exitedRunnerRequestedFreshBootstrap` (diagnostic) and `runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle `recoverable_failure` with `lease_expired_requires_bootstrap`, `transport_reconnect_grace_exceeded` or `transport_reconnect_deadline_exceeded`. These are the three codes the Rust Runner records after `shutdown_preserving_cleanup`. The state check also requires no pending terminal delivery and no warm transition. - `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when the exact Runner process is proven gone and its diagnostic names one of these exits, the server reads the saved state again. If the state has the same six identity fields and still names that exit, the reattachment launches a Runner from it instead of failing. The outbox does not need to be empty and a turn may be active: the new Runner replays from the same cursor. The attempt bound is the one the reattachment already admits (3, or 4 for an unanswered-Runner replacement claim). Other exits are unchanged. - `native-session-executor.ts` (host preparation in `createRunnerdBackend`): a resumed sandbox for a reattachment that did not adopt a process re-reads the Runner state. When it still has the same six identity fields and names the fresh-bootstrap exit, preparation keeps it in place, as for an adopted live Runner. It no longer fails with `runner_harness_state_mismatch: resumed_not_suspended` (found by Greptile). - `doc/execution-semantics.md`: one paragraph describes the rule and the staging evidence. ## Verification - Red on `origin/master`: the 5 new "launches a new Runner over the saved state after ..." cases failed (the reattachment threw `runner_remote_process_identity_unavailable`). The 7 new fail-closed cases passed. - Green: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote controller restart adoption"` passes 118 tests. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm owner before refreshing run authority", warm-session supervision, not touched here) passes when run alone. It is a timing failure under full-file load. - Launch path: "keeps the saved mid-turn state of a remote Runner that exited for lease_expired_requires_bootstrap when the reattachment prepares its sandbox" runs `createRunnerdBackend` and `prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red before the host-preparation change (`resumed_not_suspended`), green after. An unknown recoverable failure stays fail-closed. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts` on the final revision: 1494 passed, 7 skipped. - `cd server && node scripts/typecheck-layers.mjs` (includes test programs): passes. - Fail-closed cases covered: an unknown recoverable failure, a warm transition, a pending terminal delivery, a state that changed after the exit, a different Runner instance, attempt 4 on an ordinary claim, and attempt 5 on a replacement claim. - Staging read (read-only): the 5 sample runs and 69 runs with this error in the last 9 hours were classified from their `native.remote_runner_recovery_failed` events. ## Risks - Medium. Reattachment now starts a new Runner over a saved state that can be mid-turn. The Runner itself stops its provider before it saves this state. The new Runner restores the session. It continues the turn, or it reports the lost turn as recoverable only after it proves that the old provider process ended. A recoverable lost turn on a remote reattachment ends the run as `native_provider_terminal_failed` until #21362 merges; the replacement lane then continues the task. That is still better than the current reconciliation hold. - The attempt bound does not grow. A Runner that exits this way again on a later reattachment is bounded by the same provider attempt budget. - This does not change the 30 s `turn.start` command bound or the deploy gaps that let the lease expire. Those are separate follow-ups. 57 older runs with this error had a different exit (`ready`, not settled). This pull request keeps that case fail-closed; #20853 covers the settled-`ready` part. - No schema, API, UI or Runner image change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edit, code execution, read-only browser reads of staging run data). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 19 +++++ .../native-session-executor.test.ts | 85 +++++++++++++++++++ .../native-runtime/native-session-executor.ts | 24 +++++- .../remote-runner-recovery-diagnostics.ts | 23 +++++ 4 files changed, 147 insertions(+), 4 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index bea7c10ddec..0e802f1d336 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -6016,6 +6016,25 @@ cannot. Process verification and the final state read share one recovery deadline. A timeout, command failure, transport failure or invalid saved JSON on that final read keeps its closed recovery stage and reason in the diagnostic. +A Runner that no controller reaches before its connection lease expires, or +before its reconnect grace or deadline ends, stops its provider, saves its state +with lifecycle `recoverable_failure` and `lease_expired_requires_bootstrap`, +`transport_reconnect_grace_exceeded` or `transport_reconnect_deadline_exceeded`, +and exits. Only a new process with a fresh bootstrap ticket can reconnect, and +that process continues from the same cursor. A reattachment that finds this exit +re-reads the state after the process is proven gone. When it has the same six +identity fields, the same lifecycle and code, and no pending terminal delivery +or warm transition, the reattachment launches a Runner from that state, even +mid-turn and with events still in the outbox. Sandbox preparation reads that +state once more and keeps it in place, as it does for an adopted live Runner; +it does not require a `suspended` state or an idle provider checkpoint for this +exit. The new Runner restores the +conversation: it continues the turn, or reports the lost turn as recoverable +after it proves that the old provider process ended. This path admits the same +attempts as a reattachment, including the extra attempt of an unanswered-Runner +replacement. On staging, a deploy that kept a reattachment from reconnecting for +more than the lease ended these runs with `runner_remote_process_identity_unavailable`. + Reattachment keeps the complete durable PRP binding, including its turn and item attachment IDs. A healthy warm handoff can create new attachment IDs within the retained Runner. Replacing that run's controller must preserve those IDs; deriving diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index fbf92cb97cd..113c0e724f6 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1146,6 +1146,34 @@ describe("remote controller restart adoption", () => { target: exitedRunner(state) as never, identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught); expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" }); }); + describe("a reattachment whose Runner exited on its own and asked for a fresh bootstrap", () => { + const leaseExpired = { lifecycle: "recoverable_failure", recoverableFailure: "lease_expired_requires_bootstrap", + outbox: [{ id: 412 }], pendingTerminalDelivery: null, pendingProviderCleanup: null, warmTransition: null }; + const unanswered = { ...claim, replaceUnansweredRunner: true as const }; + it.each([ + ["a lease expiry on attempt 2", claim, leaseExpired, 2], + ["a lease expiry on the last attempt", claim, leaseExpired, 3], + ["a lease expiry on the admitted replacement attempt", unanswered, leaseExpired, 4], + ["an elapsed reconnect deadline", claim, { ...leaseExpired, recoverableFailure: "transport_reconnect_deadline_exceeded" }, 2], + ["an elapsed reconnect grace", claim, { ...leaseExpired, recoverableFailure: "transport_reconnect_grace_exceeded" }, 2], + ] as const)("launches a new Runner over the saved state after %s", async (_case, recoveryClaim, state, providerAttempt) => { + await expect(verifyRemoteRunnerOrSettledExit({ claim: recoveryClaim, target: exitedRunner(state) as never, identity, + runId: "run", normalizedSessionId: "session", providerAttempt })).resolves.toBeUndefined(); + }); + it.each([ + ["an unknown recoverable failure", claim, leaseExpired, { ...leaseExpired, recoverableFailure: "provider_failed" }, 2], + ["a warm transition", claim, leaseExpired, { ...leaseExpired, warmTransition: { transitionId: "warm" } }, 2], + ["a pending terminal delivery", claim, leaseExpired, { ...leaseExpired, pendingTerminalDelivery: { eventId: "event" } }, 2], + ["a state that changed after the exit", claim, leaseExpired, { ...leaseExpired, lifecycle: "turn_active" }, 2], + ["a different Runner after the exit", claim, leaseExpired, { ...leaseExpired, runnerInstanceId: "other" }, 2], + ["an attempt beyond the budget", claim, leaseExpired, leaseExpired, 4], + ["an attempt beyond the replacement budget", unanswered, leaseExpired, leaseExpired, 5], + ] as const)("keeps the reattachment fail-closed for %s", async (_case, recoveryClaim, state, afterExit, providerAttempt) => { + const error = await verifyRemoteRunnerOrSettledExit({ claim: recoveryClaim, target: exitedRunner(state, afterExit) as never, + identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught); + expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" }); + }); + }); describe("a same-run reattachment whose adopted Runner never answered its recovery read", () => { const unanswered = { ...claim, replaceUnansweredRunner: true as const }; const midTurn = { lifecycle: "turn_active", outbox: [{ id: 338 }], pendingTerminalDelivery: null, pendingProviderCleanup: null }; @@ -17767,6 +17795,63 @@ describe("runnerd provider runtime wiring", () => { || request.args.includes("paperclip-runner-claim-unstarted-session"))).toBe(false); }); + it.each([ + ["keeps", "lease_expired_requires_bootstrap"], + ["refuses", "provider_failed"], + ] as const)("%s the saved mid-turn state of a remote Runner that exited for %s when the reattachment prepares its sandbox", async (outcome, recoverableFailure) => { + const current = scopedNativeExecution(`fresh-bootstrap-relaunch-${recoverableFailure}`); + const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!, + runnerInstanceId: "lease-expired-runner", environmentLeaseId: "lease-expired-lease", + turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` }; + const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId); + await mkdir(join(root, "control-plane"), { recursive: true }); + await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity))); + const runnerState = { ...identity, lifecycle: "recoverable_failure", recoverableFailure, outbox: [{ id: 412 }], + pendingTerminalDelivery: null, pendingProviderCleanup: null, warmTransition: null }; + const providerState = { schema: "paperclip.runner.codex-provider-state.v1", lifecycle: "session_open", + threadId: "lease-expired-thread", providerSessionId: null, activeProviderTurnId: "provider-turn-1", + config: { provider: "codex", driver: "codex_app_server" } }; + const execute = vi.fn(async (request: { args: string[] }) => { + const script = request.args[1] ?? ""; + if (request.args[2] === "paperclip-runner-recovery-check") return { exitCode: 45, timedOut: false, stderr: "", stdout: "" }; + const stdout = request.args[0] === "--build-metadata" ? JSON.stringify({ + schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", + packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2, + durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"], + }) : request.args[0] === "--version" ? "codex-cli 0.160.0" + : script.includes("base64") && script.includes("runner-state.json") ? Buffer.from(JSON.stringify(runnerState)).toString("base64") + : script.includes("base64") ? Buffer.from(JSON.stringify(providerState)).toString("base64") + : script.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : ""; + return { exitCode: 0, timedOut: false, stderr: "", stdout }; + }); + const syncIn = vi.fn(async () => undefined); + state.createBackend.mockClear(); state.createTransport.mockClear(); + const created = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId, + runnerIngressAuthorized: true, + controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-version", digest: `sha256:${"a".repeat(64)}` } }, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", + environmentId: "environment", leaseId: identity.environmentLeaseId, providerKey: "daytona", runner: { execute, syncIn }, + effectiveCapabilities: { runnerWebSocketIngress: true }, + sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never, + restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", + controllerGeneration: 2, providerAttempt: 2, restartKind: "hard", recoveryRequestId: null, + remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } }, + }); + if (outcome === "refuses") { + await expect(created).rejects.toThrow("runner_remote_process_identity_unavailable"); + return; + } + await created; + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise }; + expect(options).toMatchObject({ prpIdentity: identity }); + expect((options as { adoptExistingRunner?: unknown }).adoptExistingRunner).toBeUndefined(); + await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined(); + expect(syncIn).not.toHaveBeenCalled(); + expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-claim-unstarted-session") + || request.args.some(argument => argument.includes(".continuity-break-") || argument.includes("rm -")))).toBe(false); + }); + it.each(["claude", "pi"] as const)("relays the assigned gateway when a deploy lands mid-run and the same ACPX %s run reattaches its surviving runner", async agent => { const base = scopedNativeExecution(`relay-mid-run-${agent}`); const current = { ...base, provider: { kind: "acpx", agent, model: null }, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 59fb391c5e6..3b6d3662c47 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -291,7 +291,8 @@ import { reportRunFailure } from "../run-failure-report.js"; import { REMOTE_RECOVERY_MARKER_READ_SCRIPT, REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT, REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT, RemoteRunnerRecoveryError, attachRemoteRunnerRecoveryDiagnostic, boundRemoteRunnerExitEvidence, - describeExitedRemoteRunner, describeExitedRemoteRunnerReason, readRemoteRunnerRecoveryError, remoteRecoveryProbeReason, + describeExitedRemoteRunner, describeExitedRemoteRunnerReason, exitedRunnerRequestedFreshBootstrap, readRemoteRunnerRecoveryError, + remoteRecoveryProbeReason, runnerStateRequestsFreshBootstrap, } from "./remote-runner-recovery-diagnostics.js"; import { isPluginRpcTimeout, readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS, remoteRunnerPluginStall, type RemoteRunnerPluginStall } from "./remote-runner-recovery-probe.js"; import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery, unsettledControllerCommands } from "./remote-warm-session-recovery.js"; @@ -15372,14 +15373,18 @@ export async function verifyRemoteRunnerOrSettledExit( } catch (error) { const diagnostic = readRemoteRunnerRecoveryError(error); const exitedBeforeSessionOpen = input.claim.exitedBeforeSessionOpen === true; - if (input.providerAttempt > NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT || diagnostic?.stage !== "process_verification" || + const freshBootstrap = diagnostic !== null && exitedRunnerRequestedFreshBootstrap(diagnostic); + const attemptLimit = freshBootstrap && input.claim.replaceUnansweredRunner === true + ? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT; + if (input.providerAttempt > attemptLimit || diagnostic?.stage !== "process_verification" || diagnostic.reason !== "process_not_running" || - (!exitedBeforeSessionOpen && (diagnostic.runnerLifecycle !== "suspended" || diagnostic.runnerSettled !== true)) || + (!exitedBeforeSessionOpen && !freshBootstrap && + (diagnostic.runnerLifecycle !== "suspended" || diagnostic.runnerSettled !== true)) || target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner) throw error; const state = await readRemoteRunnerState({ runner: target.runner, stateDirectory: remoteRunnerStateDirectory(target.remoteCwd, input.normalizedSessionId), recoveryDeadline, stall }); if (REMOTE_RUNNER_IDENTITY_FIELDS.some(field => state[field] !== input.identity[field])) throw error; - if (exitedBeforeSessionOpen ? !runnerStateAllowsFreshStart(state) + if (freshBootstrap ? !runnerStateRequestsFreshBootstrap(state) : exitedBeforeSessionOpen ? !runnerStateAllowsFreshStart(state) : state.lifecycle !== "suspended" || !describeExitedRemoteRunner(state).runnerSettled) throw error; return undefined; } @@ -17177,6 +17182,13 @@ async function createRunnerdBackendWithinSessionClaim( }; }; + const retainsFreshBootstrapRunnerState = async (): Promise => { + if (!remoteCommandRunner || !remoteStateDirectory || !durableIdentity) return false; + const runnerState = await readRemoteRunnerState({ runner: remoteCommandRunner, stateDirectory: remoteStateDirectory }); + return runnerStateRequestsFreshBootstrap(runnerState) + && REMOTE_RUNNER_IDENTITY_FIELDS.every(field => runnerState[field] === durableIdentity[field]); + }; + const claimUntouchedRemoteSession = async (): Promise => { if (!remoteCommandRunner || !remoteRuntimeRoot || !remoteSessionRoot) return false; const controllerState = runnerControlPlane?.store.state; @@ -17470,6 +17482,10 @@ async function createRunnerdBackendWithinSessionClaim( throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "process_not_running" }); } }); + } else if (sandboxLeaseAcquisition?.outcome === "resumed" && !adoptedProcess && !stagedRecovery && + input.restartRecovery?.kind === "reattach_remote_runner" && + await measureNativeRunnerSpan(input.trace, "sandbox.lease.resume", retainsFreshBootstrapRunnerState)) { + await input.onLog?.("stdout", "[paperclip-runner] Starting a new runner from the saved state of the runner whose connection lease ended.\n"); } else if (sandboxLeaseAcquisition?.outcome === "resumed") { const reuseStartedAtMs = Date.now(); const state = await measureNativeRunnerSpan( diff --git a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts index c35cee2a9db..8cc70adc72d 100644 --- a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts +++ b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts @@ -90,6 +90,29 @@ export function describeExitedRemoteRunner(state: Record): Pick }; } +/** + * The Runner exits with these codes after it stops its provider and saves its + * state, because only a new process with a fresh bootstrap ticket can + * reconnect. The replacement continues from the same cursor. + */ +const FRESH_BOOTSTRAP_RECOVERABLE_FAILURES = new Set([ + "lease_expired_requires_bootstrap", "transport_reconnect_deadline_exceeded", "transport_reconnect_grace_exceeded", +]); + +export function exitedRunnerRequestedFreshBootstrap( + diagnostic: Pick, +): boolean { + return diagnostic.runnerLifecycle === "recoverable_failure" + && FRESH_BOOTSTRAP_RECOVERABLE_FAILURES.has(diagnostic.runnerRecoverableFailure ?? ""); +} + +/** The saved state of an exited Runner that asked for a fresh bootstrap, re-read after its exit. */ +export function runnerStateRequestsFreshBootstrap(state: Record): boolean { + return exitedRunnerRequestedFreshBootstrap({ ...describeExitedRemoteRunner(state), + ...describeExitedRemoteRunnerReason(state, text => text) }) + && state.pendingTerminalDelivery == null && state.warmTransition == null; +} + const transportDiagnostics = new WeakMap(); export function attachRemoteRunnerRecoveryDiagnostic(error: unknown, diagnostic: RemoteRunnerRecoveryDiagnostic): unknown {