From c44a422607f9f08c39ad98f9c830fe6e3dba8dc2 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 04:22:49 -0600 Subject: [PATCH] fix(native): retry a follow-up whose own server stopped serving before its remote runner (#20786) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs each agent task through a Runner. On a graceful shutdown (SIGTERM), the server closes its runtime API listener first, then drains runs. > - A run that is still preparing can reach the runtime API lookup after the listener closed. It then fails with `local_runtime_api_listener_unavailable` before it reaches its Runner. > - For a remote (sandbox) Runner, the executor already settles this failure as a same-run bootstrap retry, so the next server restarts it with no manual Retry. > - A follow-up run is different. Admission rebinds the settled predecessor's provider session into the follow-up's run as its checkpoint. The recovery evidence saw "a checkpoint exists" and called the failure `ambiguous_state`. That blocked the task and the user had to click Retry. > - This pull request recognizes that carried checkpoint. When this server's own listener closed, the run has no Runner checkpoint and no provider event of its own, and a settled predecessor saved that exact provider session, the failure is a same-run bootstrap retry. > - The benefit is that a graceful restart during a follow-up's preparation no longer needs a manual Retry. ## Linked Issues or Issue Description **What happened?** In a Daytona restart-chaos wave (codex, remote sandbox Runner), a follow-up run was created. About 2 s later a graceful SIGTERM drain began. The draining server kept preparing the run. `resolveLocalRuntimeApiUrl()` in `createRunnerdBackendWithinSessionClaim` threw `LocalRuntimeApiListenerClosedError`. The run failed with `local_runtime_api_listener_unavailable`. The task went to `blocked` with a board-owned recovery action, and it needed a manual Retry. The manual Retry worked. **Expected behavior** No provider work started for the run. The next server must pick it up automatically. **Steps to reproduce** 1. Finish a remote native run for a task, so that its provider session is saved. 2. Comment on the task to start a follow-up run. The follow-up carries the saved provider session as its admission checkpoint. 3. Send SIGTERM while the follow-up is in preparation, before it reaches its Runner. 4. The follow-up fails with `ambiguous_state` and the task is blocked. **Paperclip version or commit** `master` at `62b7733161` (wave) and `eb75c9bb85` (this branch base). Refs #20514, #20721 ## What Changed - `native-session-executor.ts` (`nativeProviderRecoveryEvidence`): add one narrow case before the generic checks. When all of these are true, return `bootstrap_retry` (with `checkpointExists: true` and `providerSessionEstablished: false`): - the source failure is `local_runtime_api_listener_unavailable` (this server stopped serving); - the target is remote, the run has the current preparation fencing version, and the run has no Runner controller checkpoint; - the run has no provider event; - the run checkpoint has no cursor, turn, terminal, result, or pending request of its own; - a settled earlier run of the same native session saved that exact provider session. - `native-restart-recovery.ts`: export the existing `carriesSettledPredecessorSession` proof, so that the executor uses the same proof that restart recovery uses for rebound sessions. - `native-controller-recovery.integration.test.ts`: add a regression test. It also shows that the verdict stays `ambiguous_state` for a different failure code, for an unknown provider session, for a checkpoint with its own turn, and for a run with a provider event. ## Verification - Red on master: `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "own server stopped serving"` returned `recoveryMode: "ambiguous_state", providerSessionEstablished: true`. - Green with the fix: the same file passes (25 tests). - `npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts src/services/native-runtime/native-runner-restart-preparation.integration.test.ts`: 43 passed (the existing closed-listener bootstrap-retry and ambiguous cases are unchanged). - `cd server && pnpm typecheck`: passed. - Not run: a live restart-chaos wave. ## Risks - Low risk. The new case applies only to the closed-listener failure on a remote target. It needs durable proof that the run has no Runner checkpoint and no provider event, and that the provider session belongs to a settled predecessor. All other cases keep their earlier verdict. - If the retry later finds different evidence, the existing restart recovery and resumption checks re-prove it and fail closed. The task then keeps a working Retry. - Upgrade safety: no persisted shape changes. The proof reads only rows that older servers already write (`sessionCheckpoint`, `nativePreparationFencingVersion`, Runner checkpoints, run events). A run that an older server prepared without the fencing version keeps its earlier verdict. - No waits are added before `turn.start`. The new queries run only on the failure path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude --- ...ve-controller-recovery.integration.test.ts | 57 ++++++++++++++++++- .../native-runtime/native-restart-recovery.ts | 2 +- .../native-runtime/native-session-executor.ts | 33 ++++++++++- 3 files changed, 88 insertions(+), 4 deletions(-) diff --git a/server/src/services/native-runtime/native-controller-recovery.integration.test.ts b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts index c40cb4dd51e..9129ea6374f 100644 --- a/server/src/services/native-runtime/native-controller-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts @@ -9,7 +9,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { agentTaskSessions, agents, companies, createDb, environmentLeases, environments, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, nativeRunnerCheckpoints } from "@paperclipai/db"; +import { agentTaskSessions, agents, companies, createDb, environmentLeases, environments, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, nativeRunnerCheckpoints } from "@paperclipai/db"; import { DurablePrpControlPlane, parseNativeExecutionInput, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; import { startRecoveryPostgres, recoveryPostgresSupport } from "../../__tests__/helpers/recovery-postgres.js"; import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js"; @@ -192,6 +192,61 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d executionTargetKind: "local" })).toMatchObject({ recoveryMode: "exact_checkpoint_resume", providerSessionEstablished: true }); }); + it("retries a remote follow-up whose own server stopped serving before its runner, carrying only the predecessor session", async () => { + // Daytona wave s9203: a graceful drain closed the runtime API listener while + // a follow-up was still preparing. Its only checkpoint was the settled + // predecessor's provider session, rebound at admission. + const fixture = await seed(), runId = randomUUID(); + const checkpoint: PersistedNativeSession = { ...fixture.snapshot, backendKind: "runner", driverKind: "codex_app_server", + providerSessionId: "predecessor-thread", sessionId: "predecessor-thread", cursor: "predecessor-cursor", + activeTurnId: null, terminal: null, terminalTurns: [], pendingRuntimeRequests: [] }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...fixture.profile, sessionCheckpoint: checkpoint } }) + .where(eq(heartbeatRuns.id, fixture.scope.runId)); + const rebound = { ...checkpoint, cursor: null, identity: { ...checkpoint.identity, runId } }; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: fixture.scope.issueId, + nativeSessionId: fixture.scope.normalizedSessionId, runnerInstanceId: randomUUID(), status: "running", runtimeMode: "native", + createdAt: new Date(Date.now() + 1_000), + runnerProfileJson: { sessionCheckpoint: rebound, nativePreparationFencingVersion: 1 } }); + const evidence = (sourceFailureCode: Parameters[0]["sourceFailureCode"], + executionTargetKind: "remote" | "local" = "remote") => + nativeProviderRecoveryEvidence({ db, runId, executionTargetKind, sourceFailureCode }); + expect(await evidence("local_runtime_api_listener_unavailable")).toMatchObject({ recoveryMode: "bootstrap_retry", + providerSessionEstablished: false, providerEventsExist: false, checkpointExists: true }); + // Only this server's own shutdown proves the runner was never reached. + expect((await evidence("provider_process_exited")).recoveryMode).toBe("ambiguous_state"); + // A provider session no settled predecessor saved could be this run's own. + await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: { ...rebound, providerSessionId: "unknown-thread" }, + nativePreparationFencingVersion: 1 } }).where(eq(heartbeatRuns.id, runId)); + expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("ambiguous_state"); + // A turn of this run is provider work, whatever stopped it. + await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: { ...rebound, cursor: "own-turn" }, + nativePreparationFencingVersion: 1 } }).where(eq(heartbeatRuns.id, runId)); + expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("ambiguous_state"); + await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: rebound, nativePreparationFencingVersion: 1 } }) + .where(eq(heartbeatRuns.id, runId)); + // Any Runner controller checkpoint of this run, even one whose identity + // no longer matches the run, proves a Runner was reached. + const core = new DurablePrpControlPlane({ stateDirectory: join(root, runId, "reached-runner"), + identity: { runId, normalizedSessionId: fixture.scope.normalizedSessionId, runnerInstanceId: randomUUID(), + environmentLeaseId: fixture.native.environmentLeaseId, turnId: fixture.native.turnId, itemId: fixture.native.itemId }, + expectedRunnerVersion: "discovery-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + const runnerOwner = { companyId, issueId: fixture.scope.issueId, runId, authorityRunId: runId, + bootId: randomUUID(), generation: 1, leaseOwner: randomUUID() }; + await db.insert(nativeRunFinalizations).values({ companyId, issueId: fixture.scope.issueId, runId, phase: "observed", + controllerBootId: runnerOwner.bootId, controllerGeneration: 1, leaseOwner: runnerOwner.leaseOwner, + leaseExpiresAt: new Date(Date.now() + 60_000) }); + core.setCheckpointPersistence(nativeRunnerCheckpointWriter(db, runnerOwner)); + try { + await core.flushCheckpoint(); + expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("ambiguous_state"); + } finally { await core.stop(); } + await db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, runId)); + expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("bootstrap_retry"); + await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 1, stream: "system", level: "info", + eventType: "session.resumed" }); + expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("ambiguous_state"); + }); + it("uses the database clock when an import finds an expired owner", async () => { const fixture = await seed(); const now = Date.now(); diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index bd81b559f24..bc7a277b294 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -861,7 +861,7 @@ async function hasNativeRunnerOriginatedEvidence( /** A settled earlier run of the same native session saved this exact provider * session. Admission rebinds that checkpoint to a follow-up run, so the * follow-up's copy of the provider session is carried, not its own work. */ -async function carriesSettledPredecessorSession( +export async function carriesSettledPredecessorSession( tx: Pick, run: { companyId: string; id: string; nativeSessionId: string | null; createdAt: Date }, providerSessionId: unknown, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 3b37d6c3145..2029f7c7442 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -347,6 +347,7 @@ import { NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT, type NativeControllerIdentity, type NativeRestartRecoveryClaim, + carriesSettledPredecessorSession, } from "./native-restart-recovery.js"; type ActiveNativeSession = { @@ -8731,7 +8732,7 @@ export async function nativeProviderRecoveryEvidence(input: { const run = await input.db .select({ runnerProfileJson: heartbeatRuns.runnerProfileJson, companyId: heartbeatRuns.companyId, nativeIssueId: heartbeatRuns.nativeIssueId, nativeSessionId: heartbeatRuns.nativeSessionId, - runnerInstanceId: heartbeatRuns.runnerInstanceId }) + runnerInstanceId: heartbeatRuns.runnerInstanceId, id: heartbeatRuns.id, createdAt: heartbeatRuns.createdAt }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.runId)) .limit(1) @@ -8765,6 +8766,7 @@ export async function nativeProviderRecoveryEvidence(input: { ? await loadNativeUnclaimedBootstrapCheckpoint(input.db, input.bootstrap.binding, input.bootstrap.identity) : null; let controllerCheckpointMatches = true; + let controllerCheckpointAbsent = false; if (input.executionTargetKind === "remote" && checkpointRecord.backendKind === "runner" && record(run?.runnerProfileJson).nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION) { controllerCheckpointMatches = false; @@ -8772,6 +8774,7 @@ export async function nativeProviderRecoveryEvidence(input: { const header = await loadNativeRunnerCheckpointHeader(input.db, { companyId: run.companyId, issueId: run.nativeIssueId, runId: input.runId, }); + controllerCheckpointAbsent = !header; if (header && Buffer.byteLength(header) <= NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES) { const identity = inspectDurablePrpCheckpointHeader(header); const staged = inspectStagedRunAttachmentHeader(header); @@ -8811,6 +8814,30 @@ export async function nativeProviderRecoveryEvidence(input: { "provider_session_open_timeout", "local_runtime_api_listener_unavailable", ]).has(input.sourceFailureCode); + // A closed runtime API listener means this server stopped serving (a + // graceful drain) before the attempt could reach its remote runner. A + // follow-up may still carry its settled predecessor's provider session, + // rebound at admission with no cursor, turn, result or request of its own. + // With no runner checkpoint and no provider event for this run, that + // carried session is not this run's provider work: retry its startup on the + // same run instead of holding the task for a manual Retry. + const carriedAdmissionCheckpointOnly = checkpointExists && !providerEventsExist + && input.sourceFailureCode === "local_runtime_api_listener_unavailable" + && controllerCheckpointAbsent && !!run + && checkpointRecord.cursor == null && checkpointRecord.activeTurnId == null + && checkpointRecord.semanticResult == null && checkpointRecord.terminal == null + && (!Array.isArray(checkpointRecord.terminalTurns) || checkpointRecord.terminalTurns.length === 0) + && (checkpointRecord.pendingRuntimeRequests == null || (Array.isArray(checkpointRecord.pendingRuntimeRequests) + && checkpointRecord.pendingRuntimeRequests.length === 0)) + && await carriesSettledPredecessorSession(input.db, run, checkpointRecord.providerSessionId); + if (carriedAdmissionCheckpointOnly) { + return { + recoveryMode: "bootstrap_retry", + providerSessionEstablished: false, + providerEventsExist: false, + checkpointExists: true, + }; + } if (!checkpointExists && !providerEventsExist && (definitelyPreSession || unclaimedBootstrap && input.bootstrap?.transientIngressFailure)) { return { @@ -12463,7 +12490,9 @@ async function executePaperclipNativeSessionAttempt( ? "Inspect the persisted native session after its bounded resume budget was exhausted." : recoveryEvidence.recoveryMode === "bootstrap_retry" - ? `Retry provider bootstrap on this same run${nativeStartupRetryWait(providerCooldownAt)}; durable evidence proves no provider session or provider event was created.` + ? recoveryEvidence.checkpointExists + ? `Retry provider bootstrap on this same run${nativeStartupRetryWait(providerCooldownAt)}; durable evidence proves this run did no provider work of its own, and it resumes the provider session carried from its settled predecessor.` + : `Retry provider bootstrap on this same run${nativeStartupRetryWait(providerCooldownAt)}; durable evidence proves no provider session or provider event was created.` : "Resume this same run from its exact persisted native provider checkpoint after the retry delay.", }), nextAttemptAt,