From 5504151eaed4ad9856de3b4ecaea8515a55e0dbd Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 06:12:35 -0600 Subject: [PATCH] fix(recovery): restart a remote bootstrap again after it waited for a busy agent directory (#21370) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A server restart hands each interrupted native run to restart recovery. Recovery decides if the run can continue safely. > - For a remote sandbox run that reached no provider work, recovery restarts the bootstrap on the same run (`bootstrap_incomplete`). > - The restarted executor can meet a busy agent directory while it prepares. It then defers the run for a few seconds and asks restart recovery to look again. > - That deferral saves the coordinator as `retryable_failure` while the run stays `running`. No remote proof accepted this shape, so recovery blocked the run with `ambiguous_remote_recovery_evidence`. > - This pull request lets restart recovery restart the bootstrap again when the previous claim was a bootstrap restart and nothing reached execution since that claim. > - The benefit is that a deploy plus agent-directory contention no longer ends the task as failed. No provider turn is replayed. ## Linked Issues or Issue Description **What happened?** On staging, native runs of a Codex agent on Daytona failed about 3 times an hour with `native_restart_recovery_blocked` and error `ambiguous_remote_recovery_evidence`. The same failure was seen on the two days before. I read 12 failed runs. 11 of them have the same event sequence: 1. The first executor prepares the remote workspace. A deploy restarts the server. 2. Restart recovery waits for the old controller lease, then records "Restarting an incomplete native runner bootstrap on the same heartbeat run" (`bootstrap_incomplete`, controller generation 2). 3. The new executor starts the run again and records "Another run of this agent is still using its files" (`agent_directory_operation_busy`). 4. About 15 seconds later, restart recovery records "Native restart recovery blocked ambiguous or conflicting ownership" (`ambiguous_remote_recovery_evidence`). The run fails. The busy-directory deferral writes `phase: retryable_failure`, `failureCode: agent_directory_operation_busy`, `recoveryMode: bootstrap_retry`, no lease owner, and keeps the run `running`. Every remote proof in restart recovery needs either `phase: observed` with a lease owner, or a `failed` run with a specific saved failure code. So none matched. With no checkpoint and no provider evidence, the classifier returned `ambiguous_remote_recovery_evidence`. The other result fields on the failed run (workspace restore failure, fresh-continuation supersede) were written after this block. They are consequences, not the cause. **Expected behavior** The deferral happens while the executor prepares, before it requests a runner. The previous recovery claim already proved that the run reached no provider work. If nothing reached execution after that claim, recovery must restart the bootstrap again without manual Retry. If a runner start, runner identity, provider event or checkpoint appears after the claim, recovery must keep the run held and not replay provider work. **Steps to reproduce** 1. Start a native remote-sandbox run and stop the server before the runner receives a command. 2. Let restart recovery claim the run as `bootstrap_incomplete`. 3. Make the agent directory lock busy when the new executor prepares instructions, so it defers with `agent_directory_operation_busy`. 4. Let the scheduled restart recovery run at the retry time. The run is blocked with `ambiguous_remote_recovery_evidence`. The new integration test reproduces these steps with the real recovery and deferral functions. **Paperclip version or commit** Staging Core `70be453c38`. Fix based on `e23549cd9c`. **Deployment mode** Hosted staging (cloud workspace), remote Daytona sandbox, frequent auto-deploys (every 20 to 45 minutes). **Agent adapter(s) involved** `paperclip_runner` with the `codex_app_server` backend on a Daytona sandbox. ## What Changed - Restart recovery accepts a new remote proof, `remote_bootstrap_restart_deferred_before_launch`. It applies when all of these are true: - The most recent recovery claim was `bootstrap_incomplete`, with a recorded event watermark. - The coordinator is `retryable_failure` with `failureCode: agent_directory_operation_busy`, `recoveryMode: bootstrap_retry`, no lease owner, and no successor run. The run is still `running`. - No `native.process_start_requested`, `native.process_identity_recorded`, `adapter.invoke`, tool start or runner-sourced event exists after the claim watermark. No process identity was ever recorded for the run. - No provider event exists. A checkpoint, if present, is only the continued session with no turn, result or request. - The workspace is a prepared remote workspace with no durable controller checkpoint and no staged attachment. - The run then goes through the same resumption admission as the other settled remote retries (agent invokable, task not paused, dependencies ready, no competing run). - Added red/green integration tests built from the staging shape: the run restarts again (generation 3), and a runner identity, a provider event, or a checkpoint saved during the wait keeps it held. ## Verification - Red: on the unchanged source, the new test returns `{ kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }`, the same error as staging. - Green: `npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/native-controller-recovery.integration.test.ts` passes (251 passed, 14 skipped). - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. The change only adds a narrow proof. It requires that nothing reached execution since a claim that already proved no provider work. Any runner start, runner identity, provider event or new turn keeps the run held, so a provider turn is never replayed. - The restart counts against the existing provider attempt budget and the no-progress deadline, so a loop of deferrals stays bounded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, code editing, browser read-only API fetches) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Refs #20829 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-runtime/native-restart-recovery.ts | 37 +++++++++++++-- ...unner-restart-recovery.integration.test.ts | 46 +++++++++++++++++++ 2 files changed, 78 insertions(+), 5 deletions(-) diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index d2ae6c18e23..544940c183a 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -5,7 +5,7 @@ import { CONTROLLER_LEASE_RENEW_MS } from "../controller-instance-lease.js"; import { currentControllerProcessStamp, UNKNOWN_PROCESS_SPACE } from "../controller-process-identity.js"; import { readNativeSessionResumptionAgent, restoreNativeSessionExecution } from "./native-session-resumption-admission.js"; import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js"; -import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js"; +import { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js"; import { notifyNativeEnvironmentSettlement } from "../native-environment-settlement-notifications.js"; import { remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared"; import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js"; @@ -223,6 +223,12 @@ function previousNoProgressRestarts(recoveryHistory: ReadonlyArray): number | null { + const last = [...recoveryHistory].reverse().find(entry => RECOVERY_CLAIM_DISPOSITIONS.has(entry?.disposition)); + const previous = previousRecoveryClaim(recoveryHistory); + return last?.disposition === "bootstrap_incomplete" && typeof previous === "object" ? previous.watermark : null; +} + /** * The most recent recovery claim's event-log position and the provider attempt * its resume runs as. Entries from releases that did not record them give none. @@ -759,7 +765,7 @@ export function classifyNativeRunnerRecoveryEvidence(input: { remoteBootstrapProvenIncomplete?: boolean; remoteBootstrapIncompleteReason?: "remote_runner_never_launched" | "remote_session_opened_without_turn" | "remote_runner_exited_before_provider_work" | "remote_runner_never_received_commands" - | "remote_follow_up_never_attached"; + | "remote_follow_up_never_attached" | "remote_bootstrap_restart_deferred_before_launch"; processStartMatches: boolean; knownProviderProcessAlive?: boolean; knownProviderProcessIdentityAmbiguous?: boolean; @@ -1735,7 +1741,25 @@ export async function claimNativeRestartRecoveries(input: { && await carriesSettledPredecessorSession(tx, row.run, checkpointRecord.providerSessionId) && !(await hasNativeRunnerOriginatedEvidence(tx, row.run)) && !(await hasNativeRunEvent(tx, row.run, PROCESS_IDENTITY_RECORDED)); - if ((remoteRunnerNeverLaunched || remoteRunnerExitedBeforeProviderWork || remoteFollowUpNeverAttached) + const bootstrapRestartWatermark = lastBootstrapRestartWatermark(row.coordinator.recoveryHistory); + const remoteBootstrapRestartDeferredBeforeLaunch = !unclaimedBootstrap && !remoteRunnerNeverLaunched + && !remoteRunnerExitedBeforeProviderWork && !remoteFollowUpNeverAttached + && remoteWorkspace?.state === "prepared" && !pluginTask && !durableAuthority && stagedAttachment === null + && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION + && row.coordinator.phase === "retryable_failure" && row.run.status === "running" && !row.coordinator.leaseOwner + && row.coordinator.failureCode === NATIVE_AGENT_DIRECTORY_BUSY_WAIT + && settledFailure?.recoveryMode === "bootstrap_retry" + && settledFailure?.providerSessionEstablished === false && settledFailure?.providerEventsExist === false + && !settledFailure?.successorRunId + && bootstrapRestartWatermark !== null + && providerEvents.length === 0 && (!hasCheckpoint || continuedSessionOnly) + && (inheritedRunnerIdentity || (!row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt)) + && (!row.run.sessionIdBefore || continuedSessionOnly) && !row.run.sessionIdAfter + && Boolean(row.run.nativeSessionId && row.run.runnerInstanceId) + && !(await nativeExecutionStartedSince(tx, row.run.id, bootstrapRestartWatermark)) + && !(await hasNativeRunEvent(tx, row.run, PROCESS_IDENTITY_RECORDED)); + if ((remoteRunnerNeverLaunched || remoteRunnerExitedBeforeProviderWork || remoteFollowUpNeverAttached + || remoteBootstrapRestartDeferredBeforeLaunch) && row.coordinator.phase === "retryable_failure" && !admission.exhausted && !await readNativeSessionResumptionAgent(tx as unknown as Db, { issue, run: row.run, coordinator: row.coordinator })) { const deferred = await deferRemoteRetry("native_retry_admission_ineligible", ["done", "cancelled"].includes(issue.status)); @@ -1809,11 +1833,13 @@ export async function claimNativeRestartRecoveries(input: { remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority) || portablePreparation, unclaimedBootstrap: unclaimedBootstrap || portablePreparation && row.coordinator.attempt === 0, remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched - || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached || portablePreparation, + || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached || portablePreparation + || remoteBootstrapRestartDeferredBeforeLaunch, remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched || portablePreparation ? "remote_runner_never_launched" : remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work" : remoteRunnerNeverReceivedCommands ? "remote_runner_never_received_commands" : remoteFollowUpNeverAttached ? "remote_follow_up_never_attached" + : remoteBootstrapRestartDeferredBeforeLaunch ? "remote_bootstrap_restart_deferred_before_launch" : remoteBootstrapProof === "session_opened_without_turn" ? "remote_session_opened_without_turn" : undefined, runnerPidAlive, runnerGroupAlive, @@ -1822,7 +1848,8 @@ export async function claimNativeRestartRecoveries(input: { knownProviderProcessIdentityAmbiguous: providerProcesses.ambiguousLivePids.length > 0, hasCheckpoint: hasCheckpoint && !portablePreparation && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly) - && !remoteRunnerNeverLaunched && !remoteFollowUpNeverAttached && !(remoteBootstrapProvenIncomplete && reboundSessionOnly), + && !remoteRunnerNeverLaunched && !remoteFollowUpNeverAttached && !(remoteBootstrapProvenIncomplete && reboundSessionOnly) + && !remoteBootstrapRestartDeferredBeforeLaunch, stagedAttachmentVerified: stagedAttachment !== null, checkpointIdentityMatches: checkpointIdentityMatches && diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 12f75658633..150cccd0938 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -3147,6 +3147,52 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_received_commands" }); }); + it.each([ + { label: "restarts it again", evidence: "none" }, + { label: "keeps a runner process recorded during the wait held", evidence: "native.process_identity_recorded" }, + { label: "keeps a provider event recorded during the wait held", evidence: "turn.started" }, + { label: "keeps a checkpoint saved during the wait held", evidence: "checkpoint" }, + ] as const)("$label when a restarted remote bootstrap waited for a busy agent directory", async ({ evidence }) => { + const fixture = await seedUnlaunchedRemoteClaim(`BUSY-DIRECTORY-BOOTSTRAP-${evidence}`, + { event: { eventType: "native.process_start_requested" } }); + const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] }); + expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 }); + if (claim?.kind !== "bootstrap_incomplete") throw new Error("restart recovery did not restart the bootstrap"); + const schedule = vi.fn(); + const deferral = await prepareWithAgentDirectoryDeferral(async () => { throw new AgentDirectoryOperationBusyError(); }) + .catch(error => error); + expect(await deferRunForBusyAgentDirectory(fixture.db, { run: { id: fixture.runId, companyId }, error: deferral, + restartRecovery: claim, schedule, record: async () => undefined, random: () => 0 })).toBe("deferred"); + const [, retryAt, mode] = schedule.mock.calls[0]! as [string, Date, string | undefined]; + expect(mode).toBe("restart"); + if (evidence === "checkpoint") { + const [seeded] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...seeded.runnerProfileJson, sessionCheckpoint: { + identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, sessionId: fixture.native.normalizedSessionId }, + providerSessionId: "provider-thread", activeTurnId: "turn-1", cursor: null, terminal: null, terminalTurns: [], + semanticResult: null, pendingRuntimeRequests: [], + } } }).where(eq(heartbeatRuns.id, fixture.runId)); + } else if (evidence !== "none") { + await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId, seq: 950, + stream: "system", level: "info", eventType: evidence }); + } + const [resumed] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId], now: retryAt }); + if (evidence !== "none") { + expect(resumed).not.toMatchObject({ kind: "bootstrap_incomplete" }); + return; + } + expect(resumed).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 3 }); + if (resumed?.kind !== "bootstrap_incomplete") throw new Error("deferred restart recovery did not restart the bootstrap"); + const [coordinator] = await fixture.db.select().from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(coordinator).toMatchObject({ phase: "observed", leaseOwner: resumed.leaseOwner }); + expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_bootstrap_restart_deferred_before_launch" }); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(run).toMatchObject({ status: "running", errorCode: null }); + }); + // Daytona waves s9202 t02 (B1) and s9205 t04 (B2): a follow-up run on a // continued session was killed after native.process_start_requested but // before its remote runner launched. Admission rebinds the previous turn's