From 11c07c622c52c0c8bea2427b0b8ff94f2ea6eb21 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 19:54:04 -0600 Subject: [PATCH] fix(native): restart a remote bootstrap retry that shutdown cut before runner launch (#19684) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. Restart recovery decides what to do with each live or due run after the control plane restarts. > - #19616 added the `remote_runner_never_launched` proof. A claimed remote run with no runner process request and no controller checkpoint starts its bootstrap again. > - A graceful shutdown closes the local runtime listener before its drain. A remote run that is still starting then fails with `local_runtime_api_listener_unavailable`. The executor settles it as a same-run `bootstrap_retry` and releases its lease. #19596 treats this as a restart detach only for local runners. > - When the server restarted with that retry due, restart recovery claimed the row. The proof accepted only rows that still held a lease, so recovery blocked the task as ambiguous. > - This pull request lets the same proof also admit that settled `bootstrap_retry` row. > - The benefit is that a graceful restart during Daytona startup continues automatically. ## Linked Issues or Issue Description **What happened?** A chaos wave restarted the control plane gracefully (SIGTERM) while an ACPX Claude run on Daytona was starting. At 2 s after `stage.workspace`, the run failed with `local_runtime_api_listener_unavailable`, and the executor persisted "same-run resume persisted" with `recoveryMode: bootstrap_retry`. After the restart, restart recovery returned `ambiguous_remote_recovery_evidence` with disposition `blocked`. The task rested blocked. Evidence in the saved database at that point: - Coordinator `phase: retryable_failure`, `attempt: 1`, `controllerGeneration: 1`, controller identity recorded, and no lease owner. - `failureDetail` recorded `recoveryMode: bootstrap_retry`, `checkpointExists: false`, `providerEventsExist: false` and `providerSessionEstablished: false`. - The run had no runner checkpoint rows, no `native.process_start_requested`, no process identity, and no runner-sourced event. It also had no PID and no provider session. **Expected behavior** Restart recovery starts the bootstrap again, as it does for a claimed run that holds a lease (#19616) and for a local run. **Steps to reproduce** 1. Start a native run on a Daytona environment. 2. Send SIGTERM to the control plane after workspace staging and before the runner is requested. The run settles as a same-run `bootstrap_retry`. 3. Restart the control plane while that retry is due. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: in `server/src/services/native-runtime/native-restart-recovery.ts`, the `remoteRunnerNeverLaunched` condition required `phase === "observed"` and a lease owner. The settled `retryable_failure` / `bootstrap_retry` row failed that check. The remote classifier then returned `ambiguous_remote_recovery_evidence`. Refs #19616, #19596, #19583 ## What Changed - `native-restart-recovery.ts`: `remoteRunnerNeverLaunched` now accepts one of two coordinator shapes: - an `observed` claim that holds a lease (unchanged); - a `retryable_failure` with `recoveryMode: bootstrap_retry`, run status `failed`, and no lease owner. - All other proof conditions are unchanged. The run must still have no process start, process identity, adapter, tool or runner-sourced event. It must still have no process fields, no provider session, no provider evidence and no checkpoint. The same three-attempt budget applies. - An `exact_checkpoint_resume` retry keeps its own admission path. - Admission: before claiming a settled retry, restart recovery applies the same admission as the resumption reconciler (`readNativeSessionResumptionAgent`): task pause hold, agent status, dependencies, ownership, and competing runs. An ineligible retry is deferred with `native_retry_admission_ineligible` and keeps waiting. - `doc/run-log-events.md`: note the settled `bootstrap_retry` case. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "bootstrap retry"`. Before the fix, the proven case and the budget case returned `blocked` / `ambiguous_remote_recovery_evidence`. - New cases: the proven case gets `bootstrap_incomplete`. The coordinator returns to `observed`, `nextAttemptAt` is cleared, the run returns to `running`, and no recovery action is created. A requested process start or a committed runner event keeps the hold. A spent budget blocks with `execution_recovery_budget_exhausted`. An `exact_checkpoint_resume` retry does not take this path. - Regression: `native-runner-restart-recovery.integration`, `heartbeat/recovery`, `native-restart-recovery`, `native-bootstrap-retry.integration` and `native-remote-retry-admission`: 337 passed, 14 skipped. - Admission: a paused agent's due `bootstrap_retry` is deferred with `native_retry_admission_ineligible`. It is not started (red before the follow-up commit, green after). - Live Daytona replay of `w-20261106T234719-s12/t02`, 3 runs, ACPX Claude B1, using the recorded K4, K2 and K1 sequence. Image: the wave's own `paperclip-daytona-runner@sha256:2767e91c…`, with runner inputs pinned to its revision e482f0f632 locally. The replays ran the first commit of this PR. - Runs 1 and 3 passed R1. Run 2 passed R3: one Retry worked on the first click. - Runs 1 and 2 failed R6 because the wave-end sweep found a run lease still in `pending_cleanup`. In run 1 the environment delete also timed out. These failures are not in this change's path. - The restarts landed in other startup windows, so none hit this exact `bootstrap_retry` state. The integration tests cover that state with the same evidence as the saved failing database. - Run 2 also exposed a separate gap. A graceful drain that suspended a remote run after the runner launched (`awaiting_runner_reattach`) was blocked as ambiguous on the next boot. The Retry recovered it. That case is outside this PR. ## Risks - The proof still depends on the executor writing `native.process_start_requested` before a backend may spawn a runner. The fail-closed tests pin that gate. - Restart recovery and the same-run resumption reconciler can both see a due row. Both claim with a conditional update on phase and attempt, so only one can win. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/run-log-events.md | 3 + .../native-runtime/native-restart-recovery.ts | 20 +++++- ...unner-restart-recovery.integration.test.ts | 71 +++++++++++++++++++ 3 files changed, 91 insertions(+), 3 deletions(-) diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 26d6e9782b3..c21734be604 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -361,6 +361,9 @@ runner-sourced event or controller checkpoint, records `remote_runner_never_launched` with the same disposition and budget. The executor records `native.process_start_requested` before a backend may spawn a runner, so its absence (with no checkpoint) proves nothing reached a runner. +The same rule covers a remote run that its executor already settled as a +same-run `bootstrap_retry` (for example when a shutdown closed the local +runtime listener during startup) and that is due when the server restarts. For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed` event retains `error.code: "provider_turn_lost_on_restore"` and diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index b58403e4803..b72a164a1cf 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -1128,15 +1128,29 @@ export async function claimNativeRestartRecoveries(input: { const remoteRunnerNeverLaunched = !unclaimedBootstrap && remoteWorkspace?.state === "prepared" && !pluginTask && !durableAuthority && stagedAttachment === null && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION - && row.coordinator.phase === "observed" && row.coordinator.attempt >= 1 && row.coordinator.controllerGeneration >= 1 - && Boolean(row.coordinator.controllerBootId && row.coordinator.controllerPid - && row.coordinator.controllerProcessStartedAt && row.coordinator.leaseOwner) + && row.coordinator.attempt >= 1 && row.coordinator.controllerGeneration >= 1 + && Boolean(row.coordinator.controllerBootId && row.coordinator.controllerPid && row.coordinator.controllerProcessStartedAt) + // Either the executor still held the claim, or it settled a same-run + // bootstrap retry itself (for example after a shutdown closed its local + // runtime listener) and released the lease. That retry already proved + // no provider session, event or checkpoint; the checks below re-prove it. + && (row.coordinator.phase === "observed" ? Boolean(row.coordinator.leaseOwner) + : row.coordinator.phase === "retryable_failure" && row.run.status === "failed" + && row.coordinator.failureDetail?.recoveryMode === "bootstrap_retry" && !row.coordinator.leaseOwner) && !row.coordinator.failureDetail?.successorRunId && !row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt && !row.run.sessionIdBefore && !row.run.sessionIdAfter && Boolean(row.run.nativeSessionId && row.run.runnerInstanceId) && !hasCheckpoint && !hasProviderEvidence && !(await hasNativeExecutionStartEvidence(tx, row.run)); + // A settled retry starts new work, so it needs the same current + // admission as the resumption reconciler: task pause, agent status, + // dependencies, ownership and competing runs. Keep it waiting otherwise. + if (remoteRunnerNeverLaunched && row.coordinator.phase === "retryable_failure" && !admission.exhausted + && !await readNativeSessionResumptionAgent(tx as unknown as Db, { issue, run: row.run, coordinator: row.coordinator })) { + const deferred = await deferRemoteRetry("native_retry_admission_ineligible", ["done", "cancelled"].includes(issue.status)); + if (deferred) return deferred; + } // A claimed remote bootstrap (runner launched, perhaps connected) has no // provider evidence when its controller dies before session.open diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 29881952bc1..92e8c5517a2 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -2374,6 +2374,8 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( attempt?: number; leaseExpired?: boolean; liveController?: boolean; event?: { eventType: string; sourceEventId?: string }; run?: Partial; + /** RC-29c: the executor settled a same-run bootstrap retry before the restart. */ + bootstrapRetry?: boolean; } = {}) { const fixture = await seedRun(`REMOTE-UNLAUNCHED-${label}`); await fixture.db.update(heartbeatRuns).set({ @@ -2391,9 +2393,78 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( leaseOwner: `previous:${randomUUID()}`, leaseExpiresAt: new Date(Date.now() + (options.leaseExpired ? -1_000 : 20 * 60_000)), }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + if (options.bootstrapRetry) { + // Shape persisted by the executor when shutdown closed the local runtime + // listener during remote startup (local_runtime_api_listener_unavailable). + await fixture.db.update(nativeRunFinalizations).set({ + phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, + nextAttemptAt: new Date(Date.now() - 1_000), failureCode: "local_runtime_api_listener_unavailable", + failureDetail: { message: "local_runtime_api_listener_unavailable", recoverable: true, + recoveryMode: "bootstrap_retry", checkpointExists: false, providerEventsExist: false, + providerSessionEstablished: false, originalFailureCode: "local_runtime_api_listener_unavailable" }, + }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + await fixture.db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), + errorCode: "local_runtime_api_listener_unavailable" }).where(eq(heartbeatRuns.id, fixture.runId)); + } return fixture; } + it("restarts a remote bootstrap retry that shutdown cut before any runner was requested", async () => { + const fixture = await seedUnlaunchedRemoteClaim("BOOTSTRAP-RETRY", { bootstrapRetry: true }); + expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ + kind: "bootstrap_incomplete", runId: fixture.runId, providerAttempt: 1, controllerGeneration: 2 })]); + const [coordinator] = await fixture.db.select().from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(coordinator).toMatchObject({ phase: "observed", recoveryState: "bootstrap_incomplete", failureCode: null, + nextAttemptAt: null }); + expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_launched" }); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(run).toMatchObject({ status: "running", errorCode: null, finishedAt: null }); + expect(await fixture.db.select().from(issueRecoveryActions) + .where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId))).toHaveLength(0); + }); + + it.each<[string, Parameters[1], { kind: string; reason: string }]>([ + ["a runner process start was requested", { bootstrapRetry: true, event: { eventType: "native.process_start_requested" } }, + { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], + ["a runner event was committed", { bootstrapRetry: true, event: { eventType: "item.updated", sourceEventId: "runner-event-1" } }, + { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }], + ["the bootstrap budget is spent", { bootstrapRetry: true, attempt: 3 }, + { kind: "blocked", reason: "execution_recovery_budget_exhausted" }], + ])("keeps a remote bootstrap retry held when %s", async (label, options, expected) => { + const fixture = await seedUnlaunchedRemoteClaim(`RETRY-${label.replace(/\W+/g, "-")}`, options); + expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ runId: fixture.runId, ...expected })]); + }); + + it("keeps a paused agent's remote bootstrap retry waiting instead of starting it", async () => { + const fixture = await seedUnlaunchedRemoteClaim("PAUSED-RETRY", { bootstrapRetry: true }); + await fixture.db.update(agents).set({ status: "paused" }).where(eq(agents.id, agentId)); + try { + expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ + kind: "blocked", runId: fixture.runId, reason: "native_retry_admission_ineligible" })]); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(run.status).toBe("failed"); + const [coordinator] = await fixture.db.select().from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(coordinator).toMatchObject({ phase: "retryable_failure", leaseOwner: null }); + expect(coordinator.nextAttemptAt!.getTime()).toBeGreaterThan(Date.now()); + } finally { + await fixture.db.update(agents).set({ status: "active" }).where(eq(agents.id, agentId)); + } + }); + + it("leaves a remote exact-checkpoint retry to its own admission path", async () => { + const fixture = await seedUnlaunchedRemoteClaim("EXACT-RETRY", { bootstrapRetry: true }); + await fixture.db.update(nativeRunFinalizations).set({ failureDetail: { recoveryMode: "exact_checkpoint_resume" } }) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + const [disposition] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] }); + expect(disposition.kind).not.toBe("bootstrap_incomplete"); + }); + it.each([false, true])("restarts a claimed remote run that never requested a runner (lease expired=%s)", async (leaseExpired) => { const fixture = await seedUnlaunchedRemoteClaim(`PROVEN-${leaseExpired}`, { leaseExpired }); expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,