fix(native): retry a follow-up whose own server stopped serving before its remote runner (#20786)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs each agent task through a Runner. On a
graceful shutdown (SIGTERM), the server closes its runtime API listener
first, then drains runs.
> - A run that is still preparing can reach the runtime API lookup after
the listener closed. It then fails with
`local_runtime_api_listener_unavailable` before it reaches its Runner.
> - For a remote (sandbox) Runner, the executor already settles this
failure as a same-run bootstrap retry, so the next server restarts it
with no manual Retry.
> - A follow-up run is different. Admission rebinds the settled
predecessor's provider session into the follow-up's run as its
checkpoint. The recovery evidence saw "a checkpoint exists" and called
the failure `ambiguous_state`. That blocked the task and the user had to
click Retry.
> - This pull request recognizes that carried checkpoint. When this
server's own listener closed, the run has no Runner checkpoint and no
provider event of its own, and a settled predecessor saved that exact
provider session, the failure is a same-run bootstrap retry.
> - The benefit is that a graceful restart during a follow-up's
preparation no longer needs a manual Retry.

## Linked Issues or Issue Description

**What happened?**
In a Daytona restart-chaos wave (codex, remote sandbox Runner), a
follow-up run was created. About 2 s later a graceful SIGTERM drain
began. The draining server kept preparing the run.
`resolveLocalRuntimeApiUrl()` in
`createRunnerdBackendWithinSessionClaim` threw
`LocalRuntimeApiListenerClosedError`. The run failed with
`local_runtime_api_listener_unavailable`. The task went to `blocked`
with a board-owned recovery action, and it needed a manual Retry. The
manual Retry worked.

**Expected behavior**
No provider work started for the run. The next server must pick it up
automatically.

**Steps to reproduce**
1. Finish a remote native run for a task, so that its provider session
is saved.
2. Comment on the task to start a follow-up run. The follow-up carries
the saved provider session as its admission checkpoint.
3. Send SIGTERM while the follow-up is in preparation, before it reaches
its Runner.
4. The follow-up fails with `ambiguous_state` and the task is blocked.

**Paperclip version or commit**
`master` at `62b7733161` (wave) and `eb75c9bb85` (this branch base).

Refs #20514, #20721

## What Changed

- `native-session-executor.ts` (`nativeProviderRecoveryEvidence`): add
one narrow case before the generic checks. When all of these are true,
return `bootstrap_retry` (with `checkpointExists: true` and
`providerSessionEstablished: false`):
- the source failure is `local_runtime_api_listener_unavailable` (this
server stopped serving);
- the target is remote, the run has the current preparation fencing
version, and the run has no Runner controller checkpoint;
  - the run has no provider event;
- the run checkpoint has no cursor, turn, terminal, result, or pending
request of its own;
- a settled earlier run of the same native session saved that exact
provider session.
- `native-restart-recovery.ts`: export the existing
`carriesSettledPredecessorSession` proof, so that the executor uses the
same proof that restart recovery uses for rebound sessions.
- `native-controller-recovery.integration.test.ts`: add a regression
test. It also shows that the verdict stays `ambiguous_state` for a
different failure code, for an unknown provider session, for a
checkpoint with its own turn, and for a run with a provider event.

## Verification

- Red on master: `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
-t "own server stopped serving"` returned `recoveryMode:
"ambiguous_state", providerSessionEstablished: true`.
- Green with the fix: the same file passes (25 tests).
- `npx vitest run
src/services/native-runtime/native-startup-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-preparation.integration.test.ts`:
43 passed (the existing closed-listener bootstrap-retry and ambiguous
cases are unchanged).
- `cd server && pnpm typecheck`: passed.
- Not run: a live restart-chaos wave.

## Risks

- Low risk. The new case applies only to the closed-listener failure on
a remote target. It needs durable proof that the run has no Runner
checkpoint and no provider event, and that the provider session belongs
to a settled predecessor. All other cases keep their earlier verdict.
- If the retry later finds different evidence, the existing restart
recovery and resumption checks re-prove it and fail closed. The task
then keeps a working Retry.
- Upgrade safety: no persisted shape changes. The proof reads only rows
that older servers already write (`sessionCheckpoint`,
`nativePreparationFencingVersion`, Runner checkpoints, run events). A
run that an older server prepared without the fencing version keeps its
earlier verdict.
- No waits are added before `turn.start`. The new queries run only on
the failure path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

server/src/services/native-runtime/native-controller-recovery.integration.test.ts

@@ -9,7 +9,7 @@ import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { eq } from "drizzle-orm";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
-import { agentTaskSessions, agents, companies, createDb, environmentLeases, environments, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, nativeRunnerCheckpoints } from "@paperclipai/db";
+import { agentTaskSessions, agents, companies, createDb, environmentLeases, environments, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, nativeRunnerCheckpoints } from "@paperclipai/db";
 import { DurablePrpControlPlane, parseNativeExecutionInput, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
 import { startRecoveryPostgres, recoveryPostgresSupport } from "../../__tests__/helpers/recovery-postgres.js";
 import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js";
@@ -192,6 +192,61 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d
       executionTargetKind: "local" })).toMatchObject({ recoveryMode: "exact_checkpoint_resume", providerSessionEstablished: true });
   });
 
+  it("retries a remote follow-up whose own server stopped serving before its runner, carrying only the predecessor session", async () => {
+    // Daytona wave s9203: a graceful drain closed the runtime API listener while
+    // a follow-up was still preparing. Its only checkpoint was the settled
+    // predecessor's provider session, rebound at admission.
+    const fixture = await seed(), runId = randomUUID();
+    const checkpoint: PersistedNativeSession = { ...fixture.snapshot, backendKind: "runner", driverKind: "codex_app_server",
+      providerSessionId: "predecessor-thread", sessionId: "predecessor-thread", cursor: "predecessor-cursor",
+      activeTurnId: null, terminal: null, terminalTurns: [], pendingRuntimeRequests: [] };
+    await db.update(heartbeatRuns).set({ runnerProfileJson: { ...fixture.profile, sessionCheckpoint: checkpoint } })
+      .where(eq(heartbeatRuns.id, fixture.scope.runId));
+    const rebound = { ...checkpoint, cursor: null, identity: { ...checkpoint.identity, runId } };
+    await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: fixture.scope.issueId,
+      nativeSessionId: fixture.scope.normalizedSessionId, runnerInstanceId: randomUUID(), status: "running", runtimeMode: "native",
+      createdAt: new Date(Date.now() + 1_000),
+      runnerProfileJson: { sessionCheckpoint: rebound, nativePreparationFencingVersion: 1 } });
+    const evidence = (sourceFailureCode: Parameters<typeof nativeProviderRecoveryEvidence>[0]["sourceFailureCode"],
+      executionTargetKind: "remote" | "local" = "remote") =>
+      nativeProviderRecoveryEvidence({ db, runId, executionTargetKind, sourceFailureCode });
+    expect(await evidence("local_runtime_api_listener_unavailable")).toMatchObject({ recoveryMode: "bootstrap_retry",
+      providerSessionEstablished: false, providerEventsExist: false, checkpointExists: true });
+    // Only this server's own shutdown proves the runner was never reached.
+    expect((await evidence("provider_process_exited")).recoveryMode).toBe("ambiguous_state");
+    // A provider session no settled predecessor saved could be this run's own.
+    await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: { ...rebound, providerSessionId: "unknown-thread" },
+      nativePreparationFencingVersion: 1 } }).where(eq(heartbeatRuns.id, runId));
+    expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("ambiguous_state");
+    // A turn of this run is provider work, whatever stopped it.
+    await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: { ...rebound, cursor: "own-turn" },
+      nativePreparationFencingVersion: 1 } }).where(eq(heartbeatRuns.id, runId));
+    expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("ambiguous_state");
+    await db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: rebound, nativePreparationFencingVersion: 1 } })
+      .where(eq(heartbeatRuns.id, runId));
+    // Any Runner controller checkpoint of this run, even one whose identity
+    // no longer matches the run, proves a Runner was reached.
+    const core = new DurablePrpControlPlane({ stateDirectory: join(root, runId, "reached-runner"),
+      identity: { runId, normalizedSessionId: fixture.scope.normalizedSessionId, runnerInstanceId: randomUUID(),
+        environmentLeaseId: fixture.native.environmentLeaseId, turnId: fixture.native.turnId, itemId: fixture.native.itemId },
+      expectedRunnerVersion: "discovery-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
+    const runnerOwner = { companyId, issueId: fixture.scope.issueId, runId, authorityRunId: runId,
+      bootId: randomUUID(), generation: 1, leaseOwner: randomUUID() };
+    await db.insert(nativeRunFinalizations).values({ companyId, issueId: fixture.scope.issueId, runId, phase: "observed",
+      controllerBootId: runnerOwner.bootId, controllerGeneration: 1, leaseOwner: runnerOwner.leaseOwner,
+      leaseExpiresAt: new Date(Date.now() + 60_000) });
+    core.setCheckpointPersistence(nativeRunnerCheckpointWriter(db, runnerOwner));
+    try {
+      await core.flushCheckpoint();
+      expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("ambiguous_state");
+    } finally { await core.stop(); }
+    await db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, runId));
+    expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("bootstrap_retry");
+    await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 1, stream: "system", level: "info",
+      eventType: "session.resumed" });
+    expect((await evidence("local_runtime_api_listener_unavailable")).recoveryMode).toBe("ambiguous_state");
+  });
+
   it("uses the database clock when an import finds an expired owner", async () => {
     const fixture = await seed();
     const now = Date.now();

server/src/services/native-runtime/native-restart-recovery.ts

@@ -861,7 +861,7 @@ async function hasNativeRunnerOriginatedEvidence(
 /** A settled earlier run of the same native session saved this exact provider
  * session. Admission rebinds that checkpoint to a follow-up run, so the
  * follow-up's copy of the provider session is carried, not its own work. */
-async function carriesSettledPredecessorSession(
+export async function carriesSettledPredecessorSession(
   tx: Pick<Db, "select">,
   run: { companyId: string; id: string; nativeSessionId: string | null; createdAt: Date },
   providerSessionId: unknown,

server/src/services/native-runtime/native-session-executor.ts

@@ -347,6 +347,7 @@ import {
   NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT,
   type NativeControllerIdentity,
   type NativeRestartRecoveryClaim,
+  carriesSettledPredecessorSession,
 } from "./native-restart-recovery.js";
 
 type ActiveNativeSession = {
@@ -8731,7 +8732,7 @@ export async function nativeProviderRecoveryEvidence(input: {
   const run = await input.db
     .select({ runnerProfileJson: heartbeatRuns.runnerProfileJson, companyId: heartbeatRuns.companyId,
       nativeIssueId: heartbeatRuns.nativeIssueId, nativeSessionId: heartbeatRuns.nativeSessionId,
-      runnerInstanceId: heartbeatRuns.runnerInstanceId })
+      runnerInstanceId: heartbeatRuns.runnerInstanceId, id: heartbeatRuns.id, createdAt: heartbeatRuns.createdAt })
     .from(heartbeatRuns)
     .where(eq(heartbeatRuns.id, input.runId))
     .limit(1)
@@ -8765,6 +8766,7 @@ export async function nativeProviderRecoveryEvidence(input: {
     ? await loadNativeUnclaimedBootstrapCheckpoint(input.db, input.bootstrap.binding, input.bootstrap.identity)
     : null;
   let controllerCheckpointMatches = true;
+  let controllerCheckpointAbsent = false;
   if (input.executionTargetKind === "remote" && checkpointRecord.backendKind === "runner"
     && record(run?.runnerProfileJson).nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION) {
     controllerCheckpointMatches = false;
@@ -8772,6 +8774,7 @@ export async function nativeProviderRecoveryEvidence(input: {
       const header = await loadNativeRunnerCheckpointHeader(input.db, {
         companyId: run.companyId, issueId: run.nativeIssueId, runId: input.runId,
       });
+      controllerCheckpointAbsent = !header;
       if (header && Buffer.byteLength(header) <= NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES) {
         const identity = inspectDurablePrpCheckpointHeader(header);
         const staged = inspectStagedRunAttachmentHeader(header);
@@ -8811,6 +8814,30 @@ export async function nativeProviderRecoveryEvidence(input: {
     "provider_session_open_timeout",
     "local_runtime_api_listener_unavailable",
   ]).has(input.sourceFailureCode);
+  // A closed runtime API listener means this server stopped serving (a
+  // graceful drain) before the attempt could reach its remote runner. A
+  // follow-up may still carry its settled predecessor's provider session,
+  // rebound at admission with no cursor, turn, result or request of its own.
+  // With no runner checkpoint and no provider event for this run, that
+  // carried session is not this run's provider work: retry its startup on the
+  // same run instead of holding the task for a manual Retry.
+  const carriedAdmissionCheckpointOnly = checkpointExists && !providerEventsExist
+    && input.sourceFailureCode === "local_runtime_api_listener_unavailable"
+    && controllerCheckpointAbsent && !!run
+    && checkpointRecord.cursor == null && checkpointRecord.activeTurnId == null
+    && checkpointRecord.semanticResult == null && checkpointRecord.terminal == null
+    && (!Array.isArray(checkpointRecord.terminalTurns) || checkpointRecord.terminalTurns.length === 0)
+    && (checkpointRecord.pendingRuntimeRequests == null || (Array.isArray(checkpointRecord.pendingRuntimeRequests)
+      && checkpointRecord.pendingRuntimeRequests.length === 0))
+    && await carriesSettledPredecessorSession(input.db, run, checkpointRecord.providerSessionId);
+  if (carriedAdmissionCheckpointOnly) {
+    return {
+      recoveryMode: "bootstrap_retry",
+      providerSessionEstablished: false,
+      providerEventsExist: false,
+      checkpointExists: true,
+    };
+  }
   if (!checkpointExists && !providerEventsExist && (definitelyPreSession
     || unclaimedBootstrap && input.bootstrap?.transientIngressFailure)) {
     return {
@@ -12463,7 +12490,9 @@ async function executePaperclipNativeSessionAttempt(
                               ? "Inspect the persisted native session after its bounded resume budget was exhausted."
                               : recoveryEvidence.recoveryMode ===
                                   "bootstrap_retry"
-                                ? `Retry provider bootstrap on this same run${nativeStartupRetryWait(providerCooldownAt)}; durable evidence proves no provider session or provider event was created.`
+                                ? recoveryEvidence.checkpointExists
+                                  ? `Retry provider bootstrap on this same run${nativeStartupRetryWait(providerCooldownAt)}; durable evidence proves this run did no provider work of its own, and it resumes the provider session carried from its settled predecessor.`
+                                  : `Retry provider bootstrap on this same run${nativeStartupRetryWait(providerCooldownAt)}; durable evidence proves no provider session or provider event was created.`
                                 : "Resume this same run from its exact persisted native provider checkpoint after the retry delay.",
             }),
             nextAttemptAt,