From 15ee6a25a601e9ac2faf1a0372fdbeb8dc36e475 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 17:17:14 -0600 Subject: [PATCH] fix(native): re-bootstrap a claimed Daytona run killed before runner launch (#19616) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. Restart recovery decides what to do with each live run after the control plane restarts. > - #19583 lets recovery start the same-run bootstrap again when the Postgres controller checkpoint proves that the runner received only bootstrap commands. > - A live replay found an earlier window. The control plane died after the executor claimed the run and staged the sandbox workspace, but before it asked for a runner process. No runner checkpoint existed, so #19583 did not apply. > - Recovery classified that run as `ambiguous_remote_recovery_evidence` and blocked the task. The task also showed "Workspace repair required", which did not describe the real cause. > - This pull request proves the "nothing reached a runner" case from durable evidence and starts the bootstrap again. It also names the restart in the settled hold note. > - The benefit is that a restart at any point of Daytona startup before runner launch continues automatically. ## Linked Issues or Issue Description **What happened?** A chaos replay killed the control plane about 2 s after a claimed ACPX Claude run started on Daytona. The kill came after `stage.workspace` and environment acquisition, before `native.process_start_requested`. The restarted server took over because the previous controller process was dead. It then returned `ambiguous_remote_recovery_evidence` with disposition `blocked`. The task rested blocked as "Workspace repair required". Evidence in the saved database at that point: coordinator `attempt: 1`, `controllerGeneration: 1`, previous controller identity recorded and dead. Native workspace sync was `prepared`. There were no runner checkpoint rows, no session checkpoint, no process PID or start time, no provider session, and no execution-start or runner-sourced event. The local session root was an empty directory. **Expected behavior** When durable evidence proves that no runner process was requested and no controller checkpoint exists, recovery continues the run with a fresh bootstrap. Any evidence that a runner may have started keeps the hold. **Steps to reproduce** 1. Start a native run on a Daytona environment. 2. Kill the control plane after workspace staging and before the executor requests the runner process. 3. Restart the control plane. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: `server/src/services/native-runtime/native-restart-recovery.ts`, function `classifyNativeRunnerRecoveryEvidence`, remote branch. The only no-checkpoint remote rule required an unclaimed coordinator (`attempt: 0`, no controller identity). A claimed run with no launch evidence fell through to "ambiguous". Refs #19583, #16349 ## What Changed - `native-restart-recovery.ts`: add the `remote_runner_never_launched` proof, which leads to `bootstrap_incomplete`. It applies only to a classic sandbox run with a prepared workspace and the preparation fencing version. The run must have a real executor claim (controller identity and lease owner) whose controller is gone (takeover allowed). It needs no durable checkpoint, no session checkpoint, no process fields, no provider session, and no provider evidence. The run must also have no `native.process_start_requested`, process identity, `adapter.invoke`, `tool.execution.started`, or runner-sourced event. The same three-attempt budget applies. The shared event query moved into `hasNativeExecutionStartEvidence`. - `native-session-executor.ts`: `restoreRemoteBootstrapCheckpointForRecovery` removes an empty session root that preparation created. Without that step, the fresh bootstrap read it as `durable_identity_unreadable`. - `execution-recovery-resolution.ts`: the new `executionRecoverySettlementNote` names a settled `native_restart_recovery_blocked` hold as a stop after a server restart. It no longer shows "Workspace repair required" because a later workspace export was retried. Other causes keep their notes. - `doc/run-log-events.md`: document the new reason. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "unlaunched|never requested|Postgres checkpoint"`. Before the fix, the proven case (lease live with a dead owner, and lease expired) and the budget case returned `blocked` / `ambiguous_remote_recovery_evidence`. The empty-root executor case failed. - Fail-closed cases: these keep the hold: `native.process_start_requested`, a process identity event, `adapter.invoke`, a runner-sourced event, a recorded PID, a recorded provider session, and a spent budget. A live claiming controller returns `awaiting_evidence` / `controller_still_alive`. The existing "contradictory evidence" cases (`claimed` with no controller identity, `launch`, `process`, and others) stay blocked. - Wording: `server/src/services/execution-recovery-resolution.test.ts`. - Regression set (10 files, including `workspace-recovery`, `native-safe-replacement`, `heartbeat/recovery`, `native-session-executor`, and `native-restart-recovery`): 2035 passed, 14 skipped. - Live Daytona replay of the failing chaos task, 3 runs, sequential, ACPX Claude, double SIGKILL, current master plus this change. Image: content-matched `paperclip-daytona-runner@sha256:5a29a797…` built from c6eecac1e0, with runner inputs pinned to it, because the image-only build sat in the Actions queue. Result: **3/3 PASS** (R1 pass; R2, R4, R5, R6 and T pass). Each run's faults landed after the runner launched. Recovery waited on the live remote controller lease, then continued without a Retry. Run 3 continued through `reattach_remote_runner`. The saved logs for runs 1 and 2 do not name their claim kind. The earlier pre-launch window (the subject of this PR) is not hit reliably by replay timing. The integration tests cover that window with the same evidence shape as the saved failing database. ## Risks - The proof depends on the executor writing `native.process_start_requested` before a backend may spawn a runner (`onSessionAdmission` in `executeNativeSession`). A future change that spawns a runner earlier would weaken this proof. The fail-closed tests pin the event as the gate. - A fresh bootstrap acquires a new sandbox. The staged sandbox from the dead attempt keeps its lease until run settlement releases all run leases. It never ran a runner. - The note change affects only settled holds with the cause `native_restart_recovery_blocked`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/run-log-events.md | 6 ++ .../execution-recovery-resolution.test.ts | 20 +++++ .../services/execution-recovery-resolution.ts | 24 ++++-- .../native-runtime/native-restart-recovery.ts | 71 ++++++++++++----- ...unner-restart-recovery.integration.test.ts | 78 ++++++++++++++++++- .../native-runtime/native-session-executor.ts | 19 ++++- 6 files changed, 189 insertions(+), 29 deletions(-) create mode 100644 server/src/services/execution-recovery-resolution.test.ts diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 9ac6842d450..26d6e9782b3 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -355,6 +355,12 @@ unfinished `session.open`, with no committed or acknowledged event. The executor restores that checkpoint and checks the same proof before it bootstraps again. After three provider attempts it records `execution_recovery_budget_exhausted` instead. +A claimed remote run whose previous controller is gone, with no +`native.process_start_requested`, process identity, adapter invocation, +runner-sourced event or controller checkpoint, records +`remote_runner_never_launched` with the same disposition and budget. The +executor records `native.process_start_requested` before a backend may spawn a +runner, so its absence (with no checkpoint) proves nothing reached a runner. For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed` event retains `error.code: "provider_turn_lost_on_restore"` and diff --git a/server/src/services/execution-recovery-resolution.test.ts b/server/src/services/execution-recovery-resolution.test.ts new file mode 100644 index 00000000000..a7c0fc57143 --- /dev/null +++ b/server/src/services/execution-recovery-resolution.test.ts @@ -0,0 +1,20 @@ +import { describe, expect, it } from "vitest"; +import { executionRecoverySettlementNote } from "./execution-recovery-resolution.js"; + +describe("executionRecoverySettlementNote", () => { + it("names a restart-recovery hold instead of asking for a workspace repair", () => { + const note = executionRecoverySettlementNote({ current: true, cause: "native_restart_recovery_blocked", + workspaceRestoreFailed: true }); + expect(note).not.toContain("Workspace repair required"); + expect(note).toContain("after a server restart"); + }); + + it("keeps the workspace repair note for other holds with a restore failure", () => { + expect(executionRecoverySettlementNote({ current: true, cause: "execution_finalization_deadline_exceeded", + workspaceRestoreFailed: true })).toContain("Workspace repair required"); + expect(executionRecoverySettlementNote({ current: true, cause: "execution_finalization_deadline_exceeded", + workspaceRestoreFailed: false })).toContain("Automatic recovery stopped. Recorded work is preserved"); + expect(executionRecoverySettlementNote({ current: false, cause: "native_restart_recovery_blocked", + workspaceRestoreFailed: true })).toContain("Recovery closed"); + }); +}); diff --git a/server/src/services/execution-recovery-resolution.ts b/server/src/services/execution-recovery-resolution.ts index f579d908d14..7c1ed06d034 100644 --- a/server/src/services/execution-recovery-resolution.ts +++ b/server/src/services/execution-recovery-resolution.ts @@ -36,6 +36,23 @@ import { readSystemRecovery } from "./system-recovery-state.js"; import { alias } from "drizzle-orm/pg-core"; /** An operator records observed outcomes; this is not permission to blindly retry. */ +/** + * The user-facing note for a settled recovery hold. A restart-recovery hold + * names its own cause: a workspace export retried after that hold does not + * mean the user must repair the workspace. + */ +export function executionRecoverySettlementNote(input: { + current: boolean; cause: string; workspaceRestoreFailed: boolean; +}): string { + if (!input.current) return "Recovery closed because the task's owner, execution, or status changed. No work was replayed."; + if (input.cause === "native_restart_recovery_blocked") { + return "Automatic recovery stopped after a server restart because the previous execution could not be verified. Saved work is preserved; actions with unverified outcomes will not be repeated."; + } + return input.workspaceRestoreFailed + ? "Workspace repair required. Verify safe staging or repair before continuing. Saved work and approval decisions remain in force." + : "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated."; +} + export async function validateExecutionReconciliation(input: { db: Db; companyId: string; @@ -626,11 +643,8 @@ export async function settleUnrecoverableExecutions( !["done", "cancelled"].includes(task.status) && (!task.executionRunId || task.executionRunId === run.id) && (!task.checkoutRunId || task.checkoutRunId === run.id); - const note = current - ? hasWorkspaceRestoreFailure(run.resultJson) - ? "Workspace repair required. Verify safe staging or repair before continuing. Saved work and approval decisions remain in force." - : "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated." - : "Recovery closed because the task's owner, execution, or status changed. No work was replayed."; + const note = executionRecoverySettlementNote({ current, cause: action.cause, + workspaceRestoreFailed: hasWorkspaceRestoreFailure(run.resultJson) }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: task.companyId, issueId: task.id }); let nativeFailureBlock = action.evidence.nativeFailureBlock; let nativeBootstrapFailureBlock = action.evidence.nativeBootstrapFailureBlock; diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index 089d421450e..d46a0ffe566 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -453,8 +453,9 @@ export function classifyNativeRunnerRecoveryEvidence(input: { runnerGroupAlive: boolean; remoteSandbox?: boolean; unclaimedBootstrap?: boolean; - /** The durable controller checkpoint of a claimed remote bootstrap proves no provider work. */ + /** Durable evidence proves a claimed remote bootstrap reached no provider work. */ remoteBootstrapProvenIncomplete?: boolean; + remoteBootstrapIncompleteReason?: "remote_runner_never_launched"; processStartMatches: boolean; knownProviderProcessAlive?: boolean; knownProviderProcessIdentityAmbiguous?: boolean; @@ -487,7 +488,8 @@ export function classifyNativeRunnerRecoveryEvidence(input: { // Each same-run bootstrap is another provider attempt; keep restarts bounded. if ((input.providerAttempt ?? 0) >= 3) return { claimKind: null, reason: "execution_recovery_budget_exhausted" }; - return { claimKind: "bootstrap_incomplete", reason: "remote_bootstrap_proven_incomplete" }; + return { claimKind: "bootstrap_incomplete", + reason: input.remoteBootstrapIncompleteReason ?? "remote_bootstrap_proven_incomplete" }; } return input.hasCheckpoint && input.checkpointIdentityMatches === true && @@ -547,6 +549,31 @@ export function classifyNativeRunnerRecoveryEvidence(input: { return { claimKind: null, reason: "ambiguous_provider_recovery_evidence" }; } +/** Any record that execution reached a runner: a requested or identified + * process, an adapter invocation, a tool start, or a runner-sourced event. */ +async function hasNativeExecutionStartEvidence( + tx: Pick, run: { companyId: string; id: string }, +): Promise { + const [executionEvent] = await tx + .select({ id: heartbeatRunEvents.id }) + .from(heartbeatRunEvents) + .where(and( + eq(heartbeatRunEvents.companyId, run.companyId), + eq(heartbeatRunEvents.runId, run.id), + or( + isNotNull(heartbeatRunEvents.sourceEventId), + inArray(heartbeatRunEvents.eventType, [ + "adapter.invoke", + PROCESS_START_REQUESTED, + PROCESS_IDENTITY_RECORDED, + "tool.execution.started", + ]), + ), + )) + .limit(1); + return executionEvent !== undefined; +} + function historyEntry(input: { now: Date; restartKind: NativeRestartKind; @@ -1051,26 +1078,27 @@ export async function claimNativeRestartRecoveries(input: { controller.bootId === (await currentNativeControllerIdentity()).bootId) { return { kind: "awaiting_evidence", runId: row.run.id, reason: "current_controller_preparing" } as const; } - const [executionEvent] = await tx - .select({ id: heartbeatRunEvents.id }) - .from(heartbeatRunEvents) - .where(and( - eq(heartbeatRunEvents.companyId, row.run.companyId), - eq(heartbeatRunEvents.runId, row.run.id), - or( - isNotNull(heartbeatRunEvents.sourceEventId), - inArray(heartbeatRunEvents.eventType, [ - "adapter.invoke", - PROCESS_START_REQUESTED, - PROCESS_IDENTITY_RECORDED, - "tool.execution.started", - ]), - ), - )) - .limit(1); - unclaimedBootstrap = !executionEvent; + unclaimedBootstrap = !(await hasNativeExecutionStartEvidence(tx, row.run)); } + // A claimed remote run whose executor died while still preparing. The + // executor durably records native.process_start_requested before any + // backend may spawn a runner, and a remote runner exchanges commands only + // through a Postgres controller checkpoint. With neither, nothing reached + // a runner. The previous controller is gone (takeover was allowed above). + const remoteRunnerNeverLaunched = !unclaimedBootstrap && remoteWorkspace?.state === "prepared" && !pluginTask + && !durableAuthority && stagedAttachment === null + && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION + && row.coordinator.phase === "observed" && row.coordinator.attempt >= 1 && row.coordinator.controllerGeneration >= 1 + && Boolean(row.coordinator.controllerBootId && row.coordinator.controllerPid + && row.coordinator.controllerProcessStartedAt && row.coordinator.leaseOwner) + && !row.coordinator.failureDetail?.successorRunId + && !row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt + && !row.run.sessionIdBefore && !row.run.sessionIdAfter + && Boolean(row.run.nativeSessionId && row.run.runnerInstanceId) + && !hasCheckpoint && !hasProviderEvidence + && !(await hasNativeExecutionStartEvidence(tx, row.run)); + // A claimed remote bootstrap (runner launched, perhaps connected) has no // provider evidence when its controller dies before session.open // completes. The owner's lease has expired above, and the controller @@ -1087,7 +1115,8 @@ export async function claimNativeRestartRecoveries(input: { const classification = classifyNativeRunnerRecoveryEvidence({ remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority), unclaimedBootstrap, - remoteBootstrapProvenIncomplete, + remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched, + remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched ? "remote_runner_never_launched" : undefined, runnerPidAlive, runnerGroupAlive, processStartMatches: exactRunnerIdentity, diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index a9cc8f325cc..29881952bc1 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -2310,7 +2310,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( contract: { revision: "1", objective: "Restart bootstrap", criteria: [{ id: "result", requirement: "Complete work" }] } }, }); const root = resolve(runtimeRoot, `marker-${fixture.runId}`); - await mkdir(resolve(root, "control-plane"), { recursive: true, mode: 0o700 }); + await mkdir(root, { recursive: true, mode: 0o700 }); const writeMarker = (runId: string) => writeFile(resolve(root, "control-plane", "control-plane-state.json"), JSON.stringify({ schema: "paperclip.runner.external-control-plane-state.v1", identity: { runnerInstanceId: fixture.native.runnerInstanceId, environmentLeaseId: fixture.runId, runId, @@ -2320,6 +2320,15 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( controllerGeneration: 2, providerAttempt: 1, restartKind: "hard" as const, recoveryRequestId: null }; const target = { kind: "remote", transport: "sandbox", remoteCwd: "/workspace" } as AdapterExecutionTarget; try { + // RC-29b: preparation may leave an empty session root before any + // controller state exists. It carries no evidence and must not block. + await restoreRemoteBootstrapCheckpointForRecovery({ db: fixture.db, execution, root, + restartRecovery: undefined, runnerExecutionTarget: target }); + expect(existsSync(root)).toBe(true); + await restoreRemoteBootstrapCheckpointForRecovery({ db: fixture.db, execution, root, + restartRecovery: claim, runnerExecutionTarget: target }); + expect(existsSync(root)).toBe(false); + await mkdir(resolve(root, "control-plane"), { recursive: true, mode: 0o700 }); await writeMarker(fixture.runId); expect(runnerdStateProvesIncompleteBootstrap(root)).toBe(false); // Only a bootstrap recovery on a classic sandbox restores the checkpoint. @@ -2357,6 +2366,73 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( restartKind: "hard", runIds: [fixture.runId] })).toEqual([{ kind: "blocked", runId: fixture.runId, reason }]); }); + // RC-29b: a restart while the executor was still preparing a claimed Daytona + // run, before it asked for a runner process. The executor records + // native.process_start_requested (and only then may a backend spawn or a + // controller checkpoint exist), so its absence proves nothing reached a runner. + async function seedUnlaunchedRemoteClaim(label: string, options: { + attempt?: number; leaseExpired?: boolean; liveController?: boolean; + event?: { eventType: string; sourceEventId?: string }; + run?: Partial; + } = {}) { + const fixture = await seedRun(`REMOTE-UNLAUNCHED-${label}`); + await fixture.db.update(heartbeatRuns).set({ + runnerProfileJson: { ...fixture.preparedProfile, nativeWorkspaceSync: remotePreparedWorkspace() }, + ...options.run, + }).where(eq(heartbeatRuns.id, fixture.runId)); + if (options.event) await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId, + seq: 900, stream: "system", level: "info", ...options.event }); + const current = await currentNativeControllerIdentity(); + await fixture.db.update(nativeRunFinalizations).set({ + attempt: options.attempt ?? 1, controllerGeneration: 1, + controllerBootId: options.liveController ? current.bootId : randomUUID(), + controllerPid: options.liveController ? current.pid : 2_000_000_001, + controllerProcessStartedAt: options.liveController ? current.processStartedAt : new Date(Date.now() - 60_000), + leaseOwner: `previous:${randomUUID()}`, + leaseExpiresAt: new Date(Date.now() + (options.leaseExpired ? -1_000 : 20 * 60_000)), + }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + return fixture; + } + + it.each([false, true])("restarts a claimed remote run that never requested a runner (lease expired=%s)", async (leaseExpired) => { + const fixture = await seedUnlaunchedRemoteClaim(`PROVEN-${leaseExpired}`, { leaseExpired }); + expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ + kind: "bootstrap_incomplete", runId: fixture.runId, providerAttempt: 1, controllerGeneration: 2 })]); + const [coordinator] = await fixture.db.select().from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(coordinator).toMatchObject({ phase: "observed", recoveryState: "bootstrap_incomplete", failureCode: null }); + expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ disposition: "bootstrap_incomplete", + reason: "remote_runner_never_launched" }); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(run).toMatchObject({ status: "running", errorCode: null }); + expect(await fixture.db.select().from(issueRecoveryActions) + .where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId))).toHaveLength(0); + }); + + it.each<[string, Parameters[1], { kind: string; reason: string }]>([ + ["a runner process start was requested", { event: { eventType: "native.process_start_requested" } }, + { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], + ["a runner process identity was recorded", { event: { eventType: "native.process_identity_recorded" } }, + { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], + ["the adapter was invoked", { event: { eventType: "adapter.invoke" } }, + { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], + ["a runner event was committed", { event: { eventType: "item.updated", sourceEventId: "runner-event-1" } }, + { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], + ["a process pid is recorded", { run: { processPid: 443 } }, + { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], + ["a provider session is recorded", { run: { sessionIdAfter: "provider-session" } }, + { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], + ["the bootstrap budget is spent", { attempt: 3 }, + { kind: "blocked", reason: "execution_recovery_budget_exhausted" }], + ["the claiming controller is still alive", { liveController: true }, + { kind: "awaiting_evidence", reason: "controller_still_alive" }], + ])("keeps a claimed unlaunched remote run held when %s", async (label, options, expected) => { + const fixture = await seedUnlaunchedRemoteClaim(label.replace(/\W+/g, "-"), options); + expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ runId: fixture.runId, ...expected })]); + }); + it.each([ { attempt: 2, driverKind: "codex_app_server" }, { attempt: 3, driverKind: "codex_app_server" }, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index acc9ace7e60..d0243073dd5 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -126,6 +126,7 @@ import { opendirSync, readFileSync, readdirSync, + rmdirSync, readlinkSync, readSync, renameSync, @@ -6044,8 +6045,22 @@ export async function restoreRemoteBootstrapCheckpointForRecovery(input: { }): Promise { const target = input.runnerExecutionTarget; if (input.restartRecovery?.kind !== "bootstrap_incomplete" || target?.kind !== "remote" - || target.transport !== "sandbox" || target.runnerTask || !existsSync(input.root) - || readRunnerdDurableIdentity(input.root) !== null) return; + || target.transport !== "sandbox" || target.runnerTask || !isSafeNativeStateDirectory(input.root)) return; + // Preparation can create the session root before the executor opens any + // controller state. An empty root holds no evidence; remove it so the fresh + // bootstrap does not read it as an unreadable identity. + if (readdirSync(input.root).length === 0) { + try { + rmdirSync(input.root); + return; + } catch (error) { + // An entry that appeared after the read goes through the checks below. + const code = (error as NodeJS.ErrnoException).code; + if (code === "ENOENT") return; + if (code !== "ENOTEMPTY" && code !== "EEXIST") throw error; + } + } + if (readRunnerdDurableIdentity(input.root) !== null) return; const marker = readRunnerdDurableIdentity(input.root, true); if (!marker || marker.runId !== input.execution.binding.runId) return; await restoreNativeRunnerCheckpoint(input.db, input.execution.binding, input.root, nativeSessionKey(input.execution));