fix(recovery): restart a remote bootstrap again after it waited for a busy agent directory (#21370)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each interrupted native run to restart
recovery. Recovery decides if the run can continue safely.
> - For a remote sandbox run that reached no provider work, recovery
restarts the bootstrap on the same run (`bootstrap_incomplete`).
> - The restarted executor can meet a busy agent directory while it
prepares. It then defers the run for a few seconds and asks restart
recovery to look again.
> - That deferral saves the coordinator as `retryable_failure` while the
run stays `running`. No remote proof accepted this shape, so recovery
blocked the run with `ambiguous_remote_recovery_evidence`.
> - This pull request lets restart recovery restart the bootstrap again
when the previous claim was a bootstrap restart and nothing reached
execution since that claim.
> - The benefit is that a deploy plus agent-directory contention no
longer ends the task as failed. No provider turn is replayed.

## Linked Issues or Issue Description

**What happened?**

On staging, native runs of a Codex agent on Daytona failed about 3 times
an hour with `native_restart_recovery_blocked` and error
`ambiguous_remote_recovery_evidence`. The same failure was seen on the
two days before. I read 12 failed runs. 11 of them have the same event
sequence:

1. The first executor prepares the remote workspace. A deploy restarts
the server.
2. Restart recovery waits for the old controller lease, then records
"Restarting an incomplete native runner bootstrap on the same heartbeat
run" (`bootstrap_incomplete`, controller generation 2).
3. The new executor starts the run again and records "Another run of
this agent is still using its files" (`agent_directory_operation_busy`).
4. About 15 seconds later, restart recovery records "Native restart
recovery blocked ambiguous or conflicting ownership"
(`ambiguous_remote_recovery_evidence`). The run fails.

The busy-directory deferral writes `phase: retryable_failure`,
`failureCode: agent_directory_operation_busy`, `recoveryMode:
bootstrap_retry`, no lease owner, and keeps the run `running`. Every
remote proof in restart recovery needs either `phase: observed` with a
lease owner, or a `failed` run with a specific saved failure code. So
none matched. With no checkpoint and no provider evidence, the
classifier returned `ambiguous_remote_recovery_evidence`. The other
result fields on the failed run (workspace restore failure,
fresh-continuation supersede) were written after this block. They are
consequences, not the cause.

**Expected behavior**

The deferral happens while the executor prepares, before it requests a
runner. The previous recovery claim already proved that the run reached
no provider work. If nothing reached execution after that claim,
recovery must restart the bootstrap again without manual Retry. If a
runner start, runner identity, provider event or checkpoint appears
after the claim, recovery must keep the run held and not replay provider
work.

**Steps to reproduce**

1. Start a native remote-sandbox run and stop the server before the
runner receives a command.
2. Let restart recovery claim the run as `bootstrap_incomplete`.
3. Make the agent directory lock busy when the new executor prepares
instructions, so it defers with `agent_directory_operation_busy`.
4. Let the scheduled restart recovery run at the retry time. The run is
blocked with `ambiguous_remote_recovery_evidence`.

The new integration test reproduces these steps with the real recovery
and deferral functions.

**Paperclip version or commit**

Staging Core `70be453c38`. Fix based on `e23549cd9c`.

**Deployment mode**

Hosted staging (cloud workspace), remote Daytona sandbox, frequent
auto-deploys (every 20 to 45 minutes).

**Agent adapter(s) involved**

`paperclip_runner` with the `codex_app_server` backend on a Daytona
sandbox.

## What Changed

- Restart recovery accepts a new remote proof,
`remote_bootstrap_restart_deferred_before_launch`. It applies when all
of these are true:
- The most recent recovery claim was `bootstrap_incomplete`, with a
recorded event watermark.
- The coordinator is `retryable_failure` with `failureCode:
agent_directory_operation_busy`, `recoveryMode: bootstrap_retry`, no
lease owner, and no successor run. The run is still `running`.
- No `native.process_start_requested`,
`native.process_identity_recorded`, `adapter.invoke`, tool start or
runner-sourced event exists after the claim watermark. No process
identity was ever recorded for the run.
- No provider event exists. A checkpoint, if present, is only the
continued session with no turn, result or request.
- The workspace is a prepared remote workspace with no durable
controller checkpoint and no staged attachment.
- The run then goes through the same resumption admission as the other
settled remote retries (agent invokable, task not paused, dependencies
ready, no competing run).
- Added red/green integration tests built from the staging shape: the
run restarts again (generation 3), and a runner identity, a provider
event, or a checkpoint saved during the wait keeps it held.

## Verification

- Red: on the unchanged source, the new test returns `{ kind: "blocked",
reason: "ambiguous_remote_recovery_evidence" }`, the same error as
staging.
- Green: `npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-controller-recovery.integration.test.ts`
passes (251 passed, 14 skipped).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change only adds a narrow proof. It requires that
nothing reached execution since a claim that already proved no provider
work. Any runner start, runner identity, provider event or new turn
keeps the run held, so a provider turn is never replayed.
- The restart counts against the existing provider attempt budget and
the no-progress deadline, so a loop of deferrals stays bounded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser read-only API fetches) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #20829

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 06:12:35 -06:00
1 parent 0bb16ace54
commit 5504151eae
2 files changed
+78 -5

No files matched your search

@@ -5,7 +5,7 @@ import { CONTROLLER_LEASE_RENEW_MS } from "../controller-instance-lease.js";
import { currentControllerProcessStamp, UNKNOWN_PROCESS_SPACE } from "../controller-process-identity.js"; import { currentControllerProcessStamp, UNKNOWN_PROCESS_SPACE } from "../controller-process-identity.js";
import { readNativeSessionResumptionAgent, restoreNativeSessionExecution } from "./native-session-resumption-admission.js"; import { readNativeSessionResumptionAgent, restoreNativeSessionExecution } from "./native-session-resumption-admission.js";
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js"; import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js"; import { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js";
import { notifyNativeEnvironmentSettlement } from "../native-environment-settlement-notifications.js"; import { notifyNativeEnvironmentSettlement } from "../native-environment-settlement-notifications.js";
import { remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared"; import { remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js"; import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js";
@@ -223,6 +223,12 @@ function previousNoProgressRestarts(recoveryHistory: ReadonlyArray<RecoveryClaim
return 0; return 0;
} }
function lastBootstrapRestartWatermark(recoveryHistory: ReadonlyArray<RecoveryClaimProgressEntry>): number | null {
const last = [...recoveryHistory].reverse().find(entry => RECOVERY_CLAIM_DISPOSITIONS.has(entry?.disposition));
const previous = previousRecoveryClaim(recoveryHistory);
return last?.disposition === "bootstrap_incomplete" && typeof previous === "object" ? previous.watermark : null;
}
/** /**
* The most recent recovery claim's event-log position and the provider attempt * The most recent recovery claim's event-log position and the provider attempt
* its resume runs as. Entries from releases that did not record them give none. * its resume runs as. Entries from releases that did not record them give none.
@@ -759,7 +765,7 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
remoteBootstrapProvenIncomplete?: boolean; remoteBootstrapProvenIncomplete?: boolean;
remoteBootstrapIncompleteReason?: "remote_runner_never_launched" | "remote_session_opened_without_turn" remoteBootstrapIncompleteReason?: "remote_runner_never_launched" | "remote_session_opened_without_turn"
| "remote_runner_exited_before_provider_work" | "remote_runner_never_received_commands" | "remote_runner_exited_before_provider_work" | "remote_runner_never_received_commands"
| "remote_follow_up_never_attached"; | "remote_follow_up_never_attached" | "remote_bootstrap_restart_deferred_before_launch";
processStartMatches: boolean; processStartMatches: boolean;
knownProviderProcessAlive?: boolean; knownProviderProcessAlive?: boolean;
knownProviderProcessIdentityAmbiguous?: boolean; knownProviderProcessIdentityAmbiguous?: boolean;
@@ -1735,7 +1741,25 @@ export async function claimNativeRestartRecoveries(input: {
&& await carriesSettledPredecessorSession(tx, row.run, checkpointRecord.providerSessionId) && await carriesSettledPredecessorSession(tx, row.run, checkpointRecord.providerSessionId)
&& !(await hasNativeRunnerOriginatedEvidence(tx, row.run)) && !(await hasNativeRunnerOriginatedEvidence(tx, row.run))
&& !(await hasNativeRunEvent(tx, row.run, PROCESS_IDENTITY_RECORDED)); && !(await hasNativeRunEvent(tx, row.run, PROCESS_IDENTITY_RECORDED));
if ((remoteRunnerNeverLaunched || remoteRunnerExitedBeforeProviderWork || remoteFollowUpNeverAttached) const bootstrapRestartWatermark = lastBootstrapRestartWatermark(row.coordinator.recoveryHistory);
const remoteBootstrapRestartDeferredBeforeLaunch = !unclaimedBootstrap && !remoteRunnerNeverLaunched
&& !remoteRunnerExitedBeforeProviderWork && !remoteFollowUpNeverAttached
&& remoteWorkspace?.state === "prepared" && !pluginTask && !durableAuthority && stagedAttachment === null
&& row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION
&& row.coordinator.phase === "retryable_failure" && row.run.status === "running" && !row.coordinator.leaseOwner
&& row.coordinator.failureCode === NATIVE_AGENT_DIRECTORY_BUSY_WAIT
&& settledFailure?.recoveryMode === "bootstrap_retry"
&& settledFailure?.providerSessionEstablished === false && settledFailure?.providerEventsExist === false
&& !settledFailure?.successorRunId
&& bootstrapRestartWatermark !== null
&& providerEvents.length === 0 && (!hasCheckpoint || continuedSessionOnly)
&& (inheritedRunnerIdentity || (!row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt))
&& (!row.run.sessionIdBefore || continuedSessionOnly) && !row.run.sessionIdAfter
&& Boolean(row.run.nativeSessionId && row.run.runnerInstanceId)
&& !(await nativeExecutionStartedSince(tx, row.run.id, bootstrapRestartWatermark))
&& !(await hasNativeRunEvent(tx, row.run, PROCESS_IDENTITY_RECORDED));
if ((remoteRunnerNeverLaunched || remoteRunnerExitedBeforeProviderWork || remoteFollowUpNeverAttached
|| remoteBootstrapRestartDeferredBeforeLaunch)
&& row.coordinator.phase === "retryable_failure" && !admission.exhausted && row.coordinator.phase === "retryable_failure" && !admission.exhausted
&& !await readNativeSessionResumptionAgent(tx as unknown as Db, { issue, run: row.run, coordinator: row.coordinator })) { && !await readNativeSessionResumptionAgent(tx as unknown as Db, { issue, run: row.run, coordinator: row.coordinator })) {
const deferred = await deferRemoteRetry("native_retry_admission_ineligible", ["done", "cancelled"].includes(issue.status)); const deferred = await deferRemoteRetry("native_retry_admission_ineligible", ["done", "cancelled"].includes(issue.status));
@@ -1809,11 +1833,13 @@ export async function claimNativeRestartRecoveries(input: {
remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority) || portablePreparation, remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority) || portablePreparation,
unclaimedBootstrap: unclaimedBootstrap || portablePreparation && row.coordinator.attempt === 0, unclaimedBootstrap: unclaimedBootstrap || portablePreparation && row.coordinator.attempt === 0,
remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched
|| remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached || portablePreparation, || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached || portablePreparation
|| remoteBootstrapRestartDeferredBeforeLaunch,
remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched || portablePreparation ? "remote_runner_never_launched" remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched || portablePreparation ? "remote_runner_never_launched"
: remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work" : remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work"
: remoteRunnerNeverReceivedCommands ? "remote_runner_never_received_commands" : remoteRunnerNeverReceivedCommands ? "remote_runner_never_received_commands"
: remoteFollowUpNeverAttached ? "remote_follow_up_never_attached" : remoteFollowUpNeverAttached ? "remote_follow_up_never_attached"
: remoteBootstrapRestartDeferredBeforeLaunch ? "remote_bootstrap_restart_deferred_before_launch"
: remoteBootstrapProof === "session_opened_without_turn" ? "remote_session_opened_without_turn" : undefined, : remoteBootstrapProof === "session_opened_without_turn" ? "remote_session_opened_without_turn" : undefined,
runnerPidAlive, runnerPidAlive,
runnerGroupAlive, runnerGroupAlive,
@@ -1822,7 +1848,8 @@ export async function claimNativeRestartRecoveries(input: {
knownProviderProcessIdentityAmbiguous: knownProviderProcessIdentityAmbiguous:
providerProcesses.ambiguousLivePids.length > 0, providerProcesses.ambiguousLivePids.length > 0,
hasCheckpoint: hasCheckpoint && !portablePreparation && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly) hasCheckpoint: hasCheckpoint && !portablePreparation && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly)
&& !remoteRunnerNeverLaunched && !remoteFollowUpNeverAttached && !(remoteBootstrapProvenIncomplete && reboundSessionOnly), && !remoteRunnerNeverLaunched && !remoteFollowUpNeverAttached && !(remoteBootstrapProvenIncomplete && reboundSessionOnly)
&& !remoteBootstrapRestartDeferredBeforeLaunch,
stagedAttachmentVerified: stagedAttachment !== null, stagedAttachmentVerified: stagedAttachment !== null,
checkpointIdentityMatches: checkpointIdentityMatches:
checkpointIdentityMatches && checkpointIdentityMatches &&
@@ -3147,6 +3147,52 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_received_commands" }); expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_received_commands" });
}); });
it.each([
{ label: "restarts it again", evidence: "none" },
{ label: "keeps a runner process recorded during the wait held", evidence: "native.process_identity_recorded" },
{ label: "keeps a provider event recorded during the wait held", evidence: "turn.started" },
{ label: "keeps a checkpoint saved during the wait held", evidence: "checkpoint" },
] as const)("$label when a restarted remote bootstrap waited for a busy agent directory", async ({ evidence }) => {
const fixture = await seedUnlaunchedRemoteClaim(`BUSY-DIRECTORY-BOOTSTRAP-${evidence}`,
{ event: { eventType: "native.process_start_requested" } });
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId] });
expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 });
if (claim?.kind !== "bootstrap_incomplete") throw new Error("restart recovery did not restart the bootstrap");
const schedule = vi.fn();
const deferral = await prepareWithAgentDirectoryDeferral(async () => { throw new AgentDirectoryOperationBusyError(); })
.catch(error => error);
expect(await deferRunForBusyAgentDirectory(fixture.db, { run: { id: fixture.runId, companyId }, error: deferral,
restartRecovery: claim, schedule, record: async () => undefined, random: () => 0 })).toBe("deferred");
const [, retryAt, mode] = schedule.mock.calls[0]! as [string, Date, string | undefined];
expect(mode).toBe("restart");
if (evidence === "checkpoint") {
const [seeded] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...seeded.runnerProfileJson, sessionCheckpoint: {
identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, sessionId: fixture.native.normalizedSessionId },
providerSessionId: "provider-thread", activeTurnId: "turn-1", cursor: null, terminal: null, terminalTurns: [],
semanticResult: null, pendingRuntimeRequests: [],
} } }).where(eq(heartbeatRuns.id, fixture.runId));
} else if (evidence !== "none") {
await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId, seq: 950,
stream: "system", level: "info", eventType: evidence });
}
const [resumed] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId], now: retryAt });
if (evidence !== "none") {
expect(resumed).not.toMatchObject({ kind: "bootstrap_incomplete" });
return;
}
expect(resumed).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 3 });
if (resumed?.kind !== "bootstrap_incomplete") throw new Error("deferred restart recovery did not restart the bootstrap");
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(coordinator).toMatchObject({ phase: "observed", leaseOwner: resumed.leaseOwner });
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_bootstrap_restart_deferred_before_launch" });
const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(run).toMatchObject({ status: "running", errorCode: null });
});
// Daytona waves s9202 t02 (B1) and s9205 t04 (B2): a follow-up run on a // Daytona waves s9202 t02 (B1) and s9205 t04 (B2): a follow-up run on a
// continued session was killed after native.process_start_requested but // continued session was killed after native.process_start_requested but
// before its remote runner launched. Admission rebinds the previous turn's // before its remote runner launched. Admission rebinds the previous turn's