From dfd2a6efcab84bc19106a2dbf720eb53cb2282d3 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 18:36:45 -0600 Subject: [PATCH] fix(native): restart a follow-up the closing server cut before it attached, instead of reattaching to its predecessor's Runner (#20979) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs on a remote sandbox keep a warm Runner between turns. A follow-up run takes over its settled predecessor's provider session and Runner. > - When a deploy stops the server, a follow-up that is still preparing fails with `local_runtime_api_listener_unavailable`. The executor saves this failure as a same-run bootstrap retry (#20786). The saved failure proves that the run did no provider work of its own. > - After the restart, restart recovery saw a checkpoint and a provider session, and chose `reattach_remote_runner`. But the checkpoint and the Runner identity belong to the predecessor. The Runner identity check then failed with `native_remote_recovery_lease_mismatch: stage=runner_verification field=identity_run_id`. The run ended terminal and the task needed Retry. > - This pull request recognizes that durable shape in restart recovery and restarts the run as a fresh bootstrap on its warm session. It does not try to reattach to a Runner that belongs to a different run. > - The pull request also stops the queue from claiming a queued run after shutdown closed run admission during the queue scan. > - The benefit is that a deploy during a follow-up's preparation no longer ends the run terminal with a Retry. ## Linked Issues or Issue Description Refs #20786, #20928, #20852, #20904 **What happened?** In three Codex runs across two Daytona restart-chaos waves (warm sandboxes, CR1 and K15 with overlap), a warm follow-up run (`automation`, `retryOfRunId` = its predecessor) started 0–2 s before its server got SIGTERM. Its bootstrap failed with `local_runtime_api_listener_unavailable` and saved a `bootstrap_retry` with `providerSessionEstablished: false`, `providerEventsExist: false`. After the restart, `native.recovery.transition` chose `reattach_remote_runner`. `verifyRemoteRunnerReattachment` then threw `identity_run_id`, because the warm Runner's identity still named the predecessor. The run ended with `native_continuation_requires_reconciliation`. In every sandbox capture the Runner was cleanly suspended, with no stderr. **Expected behavior** The run did no provider work and never wrote its own Runner identity. The next server must restart it without a Retry. **Steps to reproduce** 1. Finish a remote warm native run, so that its provider session and Runner identity are saved. 2. Start a follow-up run. Admission carries the predecessor's provider session and Runner process identity into the follow-up. 3. Send SIGTERM while the follow-up prepares, so that it fails with `local_runtime_api_listener_unavailable`. 4. Restart the server. Restart recovery chooses `reattach_remote_runner`, and the run fails with `identity_run_id`. ## What Changed - `native-restart-recovery.ts` (`claimNativeRestartRecoveries`): add the `remoteFollowUpNeverAttached` proof. It is true only when all of these are true: - the run is a failed, settled `bootstrap_retry` whose original failure is `local_runtime_api_listener_unavailable`, with `providerSessionEstablished: false`, `providerEventsExist: false`, no successor and no lease owner; - the run is on a prepared remote sandbox workspace with the current preparation fencing version, it is not a plugin task, and it has no Runner controller checkpoint and no staged attachment of its own; - its session checkpoint is only the carried admission checkpoint (no cursor, turn, result, terminal or pending request), and a settled earlier run of the same session saved that exact provider session; - its Runner process identity is absent or is exactly the identity of a settled predecessor; - it has no provider event, no Runner-originated event, and no `native.process_identity_recorded` event of its own. With that proof, the run gets a `bootstrap_incomplete` claim with the reason `remote_follow_up_never_attached`, and it passes the same current-admission check as the other settled bootstrap retries. - `verifyRemoteRunnerReattachment` is unchanged. It still refuses a Runner that names a different run. - `heartbeat/queue.ts` (`startNextQueuedRunForAgent`): check run admission again before each claim. A run that a scan reaches after shutdown closed admission stays queued for the next server. - Tests: - `native-runner-restart-recovery.integration.test.ts`: eight cases for the staging shape. Three restart (inherited Runner identity; inherited identity plus a Runner request; a Runner request with no identity). Five stay held (own recorded Runner identity; a Runner-originated event; a carried active turn; a provider session that no settled run saved; a saved failure that established a provider session). - `queue.test.ts`: a queued run stays queued and is not executed when admission closes during the scan. ### How this relates to #20852 and #20904 - #20852 hands a preparing run to the next controller and persists launch intents, so that preparation itself survives a deploy. It does not change restart recovery classification for a settled `bootstrap_retry`, and it does not touch the run-admission gate. This PR covers runs that already failed and saved a `bootstrap_retry` (including state that older releases saved), so the two are complementary. - #20904 changes startup readiness for native-controller preservation obligations. It does not touch either code path in this PR. ### Finding about the "draining server" admission In both waves the follow-up was claimed before the process observed shutdown. In CR1 (seed 9511) the run was created at 21:54:04.695Z and started at 21:54:04.854Z; the server logged "shutdown signal received" at 21:54:05 (the harness sent SIGTERM at 21:54:03.256Z, under load average 10). In the K15 overlap (seed 9513) the runs started at 23:31:10.9Z and 23:31:11.5Z, and the old server got SIGTERM at 23:31:13Z. During the overlap the old server is fully live and has no signal that a newer deploy serves traffic. So the existing admission gate (`closeProcessRunAdmission`) was not bypassed. The only gap in the gate was the check-then-claim window inside `startNextQueuedRunForAgent`, which this PR closes. Holding work on a superseded but not yet signaled server needs a new "superseded deploy" signal; this PR does not add one. ## Verification - Red on master (`23909077ad`): `node cli/node_modules/tsx/dist/cli.mjs packages/db/src/test-postgres-command.ts -- node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "closed the runtime listener before it attached"`: the three restart cases failed with `"kind": "reattach_remote_runner"` (the staging disposition); the five held cases passed. - Red on master: `queue.test.ts -t "closes admission during the queue scan"` failed (`expected [ { …(81) } ] to deeply equal []`). - Green with the fix: both filters pass. - `native-runner-restart-recovery`, `native-runner-restart-preparation`, `native-startup-retry` and `native-controller-recovery` integration files: 267 passed, 14 skipped. - `queue.test.ts`: 89 passed. - `cd server && pnpm typecheck`: passed. - Chaos waves: see the PR comments for the reruns of seeds 9511 (CR1) and 9513 (K15 overlap). ## Risks - The new proof applies only to the closed-listener `bootstrap_retry` that #20786 writes, on a remote sandbox, with a carried session that a settled predecessor saved. Every other shape keeps its earlier classification. - A `bootstrap_incomplete` claim sends the run through the existing bootstrap path, which verifies the prior owner of the session root before it starts. The s9205 follow-ups already use this path with an inherited Runner identity. - Upgrade safety: no persisted shape changes. The proof reads only fields that the current and recent releases write. A row saved without these facts keeps the earlier verdict. - No wait or file work is added before `turn.start`. The new queries run only in restart recovery. - The queue change only stops claiming after admission closed. Runs stay queued for the next server, the same as the existing entry check. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- server/src/services/heartbeat/queue.test.ts | 13 +++++ server/src/services/heartbeat/queue.ts | 2 +- .../native-runtime/native-restart-recovery.ts | 27 ++++++++-- ...unner-restart-recovery.integration.test.ts | 49 +++++++++++++++++++ 4 files changed, 85 insertions(+), 6 deletions(-) diff --git a/server/src/services/heartbeat/queue.test.ts b/server/src/services/heartbeat/queue.test.ts index b3f0d0e97b7..6dee76c29f3 100644 --- a/server/src/services/heartbeat/queue.test.ts +++ b/server/src/services/heartbeat/queue.test.ts @@ -1026,6 +1026,19 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => { expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id); }); + it("leaves a queued run for the next server when shutdown closes admission during the queue scan", async () => { + const { run, agent, wake } = await fixture("queued"); + await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id)); + let admissionClosed = false; + const deps = callbacks(db); + deps.isRunAdmissionClosed = () => admissionClosed; + deps.getAgent.mockImplementation(async () => { admissionClosed = true; return agent; }); + expect(await createHeartbeatQueue(db, deps).startNextQueuedRunForAgent(agent.id)).toEqual([]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toMatchObject({ status: "queued", startedAt: null }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0].status).toBe(wake.status); + expect(deps.executeRun).not.toHaveBeenCalled(); + }); + it("rechecks the connection hold when Stop races the queue scan", async () => { const { run, agent } = await fixture("queued"); const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) }; diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 509b8c6bed7..030a3abbf18 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -2284,7 +2284,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende const claimedRuns: Array = []; for (const queuedRun of prioritizedRuns) { - if (claimedRuns.length >= availableSlots) break; + if (claimedRuns.length >= availableSlots || isRunAdmissionClosed()) break; let claimed: typeof heartbeatRuns.$inferSelect | null; try { claimed = await claimQueuedRun(queuedRun, companyAgents, deferredPostCommitEffects, deferredCancellations); diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index f84352cd845..d71af628d3e 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -745,7 +745,8 @@ export function classifyNativeRunnerRecoveryEvidence(input: { /** Durable evidence proves a claimed remote bootstrap reached no provider work. */ remoteBootstrapProvenIncomplete?: boolean; remoteBootstrapIncompleteReason?: "remote_runner_never_launched" | "remote_session_opened_without_turn" - | "remote_runner_exited_before_provider_work" | "remote_runner_never_received_commands"; + | "remote_runner_exited_before_provider_work" | "remote_runner_never_received_commands" + | "remote_follow_up_never_attached"; processStartMatches: boolean; knownProviderProcessAlive?: boolean; knownProviderProcessIdentityAmbiguous?: boolean; @@ -1592,7 +1593,22 @@ export async function claimNativeRestartRecoveries(input: { // Fresh evidence wins over the saved failure: any event the runner or // provider sent contradicts "no provider work". && !(await hasNativeRunnerOriginatedEvidence(tx, row.run)); - if ((remoteRunnerNeverLaunched || remoteRunnerExitedBeforeProviderWork) && row.coordinator.phase === "retryable_failure" && !admission.exhausted + const remoteFollowUpNeverAttached = !unclaimedBootstrap && !remoteRunnerNeverLaunched && !remoteRunnerExitedBeforeProviderWork + && remoteWorkspace?.state === "prepared" && !pluginTask && !durableAuthority && stagedAttachment === null + && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION + && row.coordinator.phase === "retryable_failure" && row.run.status === "failed" && !row.coordinator.leaseOwner + && settledFailure?.recoveryMode === "bootstrap_retry" + && settledFailure?.originalFailureCode === "local_runtime_api_listener_unavailable" + && settledFailure?.providerSessionEstablished === false && settledFailure?.providerEventsExist === false + && !settledFailure?.successorRunId + && continuedSessionOnly && inheritedCheckpointOnly && !row.run.sessionIdAfter + && Boolean(row.run.nativeSessionId && row.run.runnerInstanceId) + && (inheritedRunnerIdentity || (!row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt)) + && await carriesSettledPredecessorSession(tx, row.run, checkpointRecord.providerSessionId) + && !(await hasNativeRunnerOriginatedEvidence(tx, row.run)) + && !(await hasNativeRunEvent(tx, row.run, PROCESS_IDENTITY_RECORDED)); + if ((remoteRunnerNeverLaunched || remoteRunnerExitedBeforeProviderWork || remoteFollowUpNeverAttached) + && row.coordinator.phase === "retryable_failure" && !admission.exhausted && !await readNativeSessionResumptionAgent(tx as unknown as Db, { issue, run: row.run, coordinator: row.coordinator })) { const deferred = await deferRemoteRetry("native_retry_admission_ineligible", ["done", "cancelled"].includes(issue.status)); if (deferred) return deferred; @@ -1633,7 +1649,7 @@ export async function claimNativeRestartRecoveries(input: { && (!ownProviderEvidence || remoteBootstrapProof === "session_opened_without_turn"); const remoteRunnerNeverReceivedCommands = !unclaimedBootstrap && !remoteRunnerNeverLaunched - && !remoteRunnerExitedBeforeProviderWork && !remoteBootstrapProvenIncomplete + && !remoteRunnerExitedBeforeProviderWork && !remoteBootstrapProvenIncomplete && !remoteFollowUpNeverAttached && remoteSandbox && remoteWorkspace !== null && !pluginTask && !durableAuthority && stagedAttachment === null && admissionCheckpointOnly && providerEvents.length === 0 && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION @@ -1659,10 +1675,11 @@ export async function claimNativeRestartRecoveries(input: { remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority), unclaimedBootstrap, remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched - || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands, + || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached, remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched ? "remote_runner_never_launched" : remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work" : remoteRunnerNeverReceivedCommands ? "remote_runner_never_received_commands" + : remoteFollowUpNeverAttached ? "remote_follow_up_never_attached" : remoteBootstrapProof === "session_opened_without_turn" ? "remote_session_opened_without_turn" : undefined, runnerPidAlive, runnerGroupAlive, @@ -1671,7 +1688,7 @@ export async function claimNativeRestartRecoveries(input: { knownProviderProcessIdentityAmbiguous: providerProcesses.ambiguousLivePids.length > 0, hasCheckpoint: hasCheckpoint && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly) - && !remoteRunnerNeverLaunched && !(remoteBootstrapProvenIncomplete && reboundSessionOnly), + && !remoteRunnerNeverLaunched && !remoteFollowUpNeverAttached && !(remoteBootstrapProvenIncomplete && reboundSessionOnly), stagedAttachmentVerified: stagedAttachment !== null, checkpointIdentityMatches: checkpointIdentityMatches && diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 5b1251bb68d..cf50f4d54b1 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -3105,6 +3105,55 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_execution_never_claimed" }); }); + it.each([ + { label: "bootstraps a follow-up carrying its predecessor's Runner identity", mode: "inherited", restarts: true }, + { label: "bootstraps a follow-up that requested its warm Runner on its predecessor's identity", mode: "inherited_requested", restarts: true }, + { label: "bootstraps a follow-up with no Runner identity that requested a Runner", mode: "requested", restarts: true }, + { label: "keeps a follow-up with its own recorded Runner identity held", mode: "own_identity", restarts: false }, + { label: "keeps a follow-up with a Runner-originated event held", mode: "runner_event", restarts: false }, + { label: "keeps a follow-up with a carried active turn held", mode: "active_turn", restarts: false }, + { label: "keeps a follow-up whose provider session no settled run saved held", mode: "no_predecessor", restarts: false }, + { label: "keeps a follow-up whose saved failure established a provider session held", mode: "session_established", restarts: false }, + ] as const)("$label after its own server closed the runtime listener before it attached (s9511, s9513)", async ({ mode, restarts }) => { + const fixture = await seedUnlaunchedRemoteClaim(`FOLLOW-UP-LISTENER-${mode}`, { bootstrapRetry: true, + ...(["inherited_requested", "requested", "own_identity", "runner_event"].includes(mode) + ? { event: { eventType: "native.process_start_requested" } } : {}) }); + await reboundFollowUpCheckpoint(fixture, mode === "active_turn" ? { activeTurnId: "turn-in-flight" } : {}, mode !== "no_predecessor"); + await fixture.db.update(nativeRunFinalizations).set({ failureDetail: { + message: "local_runtime_api_listener_unavailable", recoverable: true, recoveryMode: "bootstrap_retry", + checkpointExists: true, providerEventsExist: false, providerSessionEstablished: mode === "session_established", + originalFailureCode: "local_runtime_api_listener_unavailable", + } }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + const [current] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + if (mode !== "requested") { + const processStartedAt = new Date(Date.now() - 120_000); + await fixture.db.update(heartbeatRuns).set({ processPid: 596, processGroupId: null, processStartedAt, processHostHash: "remote" }) + .where(eq(heartbeatRuns.id, fixture.runId)); + if (mode !== "own_identity") await fixture.db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", + runtimeMode: "native", nativeIssueId: fixture.issueId, nativeSessionId: current.nativeSessionId, + runnerInstanceId: current.runnerInstanceId, createdAt: new Date(current.createdAt.getTime() - 60_000), + finishedAt: new Date(current.createdAt.getTime() - 5_000), processPid: 596, processGroupId: null, processStartedAt, + processHostHash: "remote", contextSnapshot: { issueId: fixture.issueId } }); + } + if (mode === "own_identity") await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId, + seq: 901, stream: "system", level: "info", eventType: "native.process_identity_recorded" }); + if (mode === "runner_event") await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId, + seq: 902, stream: "system", level: "info", eventType: "item.updated", sourceEventId: "runner-event-after-failure" }); + const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] }); + if (!restarts) { + expect(claim).not.toMatchObject({ kind: "bootstrap_incomplete" }); + return; + } + expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 }); + const [coordinator] = await fixture.db.select().from(nativeRunFinalizations) + .where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(coordinator).toMatchObject({ phase: "observed", recoveryState: "bootstrap_incomplete", failureCode: null }); + expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_follow_up_never_attached" }); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(run).toMatchObject({ status: "running", errorCode: null, finishedAt: null }); + }); + it("restarts a remote bootstrap retry that shutdown cut before any runner was requested", async () => { const fixture = await seedUnlaunchedRemoteClaim("BOOTSTRAP-RETRY", { bootstrapRetry: true }); expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,