fix(native): restart a follow-up the closing server cut before it attached, instead of reattaching to its predecessor's Runner (#20979)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs on a remote sandbox keep a warm Runner between turns. A
follow-up run takes over its settled predecessor's provider session and
Runner.
> - When a deploy stops the server, a follow-up that is still preparing
fails with `local_runtime_api_listener_unavailable`. The executor saves
this failure as a same-run bootstrap retry (#20786). The saved failure
proves that the run did no provider work of its own.
> - After the restart, restart recovery saw a checkpoint and a provider
session, and chose `reattach_remote_runner`. But the checkpoint and the
Runner identity belong to the predecessor. The Runner identity check
then failed with `native_remote_recovery_lease_mismatch:
stage=runner_verification field=identity_run_id`. The run ended terminal
and the task needed Retry.
> - This pull request recognizes that durable shape in restart recovery
and restarts the run as a fresh bootstrap on its warm session. It does
not try to reattach to a Runner that belongs to a different run.
> - The pull request also stops the queue from claiming a queued run
after shutdown closed run admission during the queue scan.
> - The benefit is that a deploy during a follow-up's preparation no
longer ends the run terminal with a Retry.

## Linked Issues or Issue Description

Refs #20786, #20928, #20852, #20904

**What happened?**
In three Codex runs across two Daytona restart-chaos waves (warm
sandboxes, CR1 and K15 with overlap), a warm follow-up run
(`automation`, `retryOfRunId` = its predecessor) started 0–2 s before
its server got SIGTERM. Its bootstrap failed with
`local_runtime_api_listener_unavailable` and saved a `bootstrap_retry`
with `providerSessionEstablished: false`, `providerEventsExist: false`.
After the restart, `native.recovery.transition` chose
`reattach_remote_runner`. `verifyRemoteRunnerReattachment` then threw
`identity_run_id`, because the warm Runner's identity still named the
predecessor. The run ended with
`native_continuation_requires_reconciliation`. In every sandbox capture
the Runner was cleanly suspended, with no stderr.

**Expected behavior**
The run did no provider work and never wrote its own Runner identity.
The next server must restart it without a Retry.

**Steps to reproduce**
1. Finish a remote warm native run, so that its provider session and
Runner identity are saved.
2. Start a follow-up run. Admission carries the predecessor's provider
session and Runner process identity into the follow-up.
3. Send SIGTERM while the follow-up prepares, so that it fails with
`local_runtime_api_listener_unavailable`.
4. Restart the server. Restart recovery chooses
`reattach_remote_runner`, and the run fails with `identity_run_id`.

## What Changed

- `native-restart-recovery.ts` (`claimNativeRestartRecoveries`): add the
`remoteFollowUpNeverAttached` proof. It is true only when all of these
are true:
- the run is a failed, settled `bootstrap_retry` whose original failure
is `local_runtime_api_listener_unavailable`, with
`providerSessionEstablished: false`, `providerEventsExist: false`, no
successor and no lease owner;
- the run is on a prepared remote sandbox workspace with the current
preparation fencing version, it is not a plugin task, and it has no
Runner controller checkpoint and no staged attachment of its own;
- its session checkpoint is only the carried admission checkpoint (no
cursor, turn, result, terminal or pending request), and a settled
earlier run of the same session saved that exact provider session;
- its Runner process identity is absent or is exactly the identity of a
settled predecessor;
- it has no provider event, no Runner-originated event, and no
`native.process_identity_recorded` event of its own.
With that proof, the run gets a `bootstrap_incomplete` claim with the
reason `remote_follow_up_never_attached`, and it passes the same
current-admission check as the other settled bootstrap retries.
- `verifyRemoteRunnerReattachment` is unchanged. It still refuses a
Runner that names a different run.
- `heartbeat/queue.ts` (`startNextQueuedRunForAgent`): check run
admission again before each claim. A run that a scan reaches after
shutdown closed admission stays queued for the next server.
- Tests:
- `native-runner-restart-recovery.integration.test.ts`: eight cases for
the staging shape. Three restart (inherited Runner identity; inherited
identity plus a Runner request; a Runner request with no identity). Five
stay held (own recorded Runner identity; a Runner-originated event; a
carried active turn; a provider session that no settled run saved; a
saved failure that established a provider session).
- `queue.test.ts`: a queued run stays queued and is not executed when
admission closes during the scan.

### How this relates to #20852 and #20904

- #20852 hands a preparing run to the next controller and persists
launch intents, so that preparation itself survives a deploy. It does
not change restart recovery classification for a settled
`bootstrap_retry`, and it does not touch the run-admission gate. This PR
covers runs that already failed and saved a `bootstrap_retry` (including
state that older releases saved), so the two are complementary.
- #20904 changes startup readiness for native-controller preservation
obligations. It does not touch either code path in this PR.

### Finding about the "draining server" admission

In both waves the follow-up was claimed before the process observed
shutdown. In CR1 (seed 9511) the run was created at 21:54:04.695Z and
started at 21:54:04.854Z; the server logged "shutdown signal received"
at 21:54:05 (the harness sent SIGTERM at 21:54:03.256Z, under load
average 10). In the K15 overlap (seed 9513) the runs started at
23:31:10.9Z and 23:31:11.5Z, and the old server got SIGTERM at
23:31:13Z. During the overlap the old server is fully live and has no
signal that a newer deploy serves traffic. So the existing admission
gate (`closeProcessRunAdmission`) was not bypassed. The only gap in the
gate was the check-then-claim window inside
`startNextQueuedRunForAgent`, which this PR closes. Holding work on a
superseded but not yet signaled server needs a new "superseded deploy"
signal; this PR does not add one.

## Verification

- Red on master (`23909077ad`): `node cli/node_modules/tsx/dist/cli.mjs
packages/db/src/test-postgres-command.ts -- node
node_modules/vitest/vitest.mjs run --project @paperclipai/server
server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "closed the runtime listener before it attached"`: the three restart
cases failed with `"kind": "reattach_remote_runner"` (the staging
disposition); the five held cases passed.
- Red on master: `queue.test.ts -t "closes admission during the queue
scan"` failed (`expected [ { …(81) } ] to deeply equal []`).
- Green with the fix: both filters pass.
- `native-runner-restart-recovery`, `native-runner-restart-preparation`,
`native-startup-retry` and `native-controller-recovery` integration
files: 267 passed, 14 skipped.
- `queue.test.ts`: 89 passed.
- `cd server && pnpm typecheck`: passed.
- Chaos waves: see the PR comments for the reruns of seeds 9511 (CR1)
and 9513 (K15 overlap).

## Risks

- The new proof applies only to the closed-listener `bootstrap_retry`
that #20786 writes, on a remote sandbox, with a carried session that a
settled predecessor saved. Every other shape keeps its earlier
classification.
- A `bootstrap_incomplete` claim sends the run through the existing
bootstrap path, which verifies the prior owner of the session root
before it starts. The s9205 follow-ups already use this path with an
inherited Runner identity.
- Upgrade safety: no persisted shape changes. The proof reads only
fields that the current and recent releases write. A row saved without
these facts keeps the earlier verdict.
- No wait or file work is added before `turn.start`. The new queries run
only in restart recovery.
- The queue change only stops claiming after admission closed. Runs stay
queued for the next server, the same as the existing entry check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/heartbeat/queue.test.ts

@@ -1026,6 +1026,19 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
     expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id);
   });
 
+  it("leaves a queued run for the next server when shutdown closes admission during the queue scan", async () => {
+    const { run, agent, wake } = await fixture("queued");
+    await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id));
+    let admissionClosed = false;
+    const deps = callbacks(db);
+    deps.isRunAdmissionClosed = () => admissionClosed;
+    deps.getAgent.mockImplementation(async () => { admissionClosed = true; return agent; });
+    expect(await createHeartbeatQueue(db, deps).startNextQueuedRunForAgent(agent.id)).toEqual([]);
+    expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toMatchObject({ status: "queued", startedAt: null });
+    expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0].status).toBe(wake.status);
+    expect(deps.executeRun).not.toHaveBeenCalled();
+  });
+
   it("rechecks the connection hold when Stop races the queue scan", async () => {
     const { run, agent } = await fixture("queued");
     const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) };

server/src/services/heartbeat/queue.ts

@@ -2284,7 +2284,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
 
       const claimedRuns: Array<typeof heartbeatRuns.$inferSelect> = [];
       for (const queuedRun of prioritizedRuns) {
-        if (claimedRuns.length >= availableSlots) break;
+        if (claimedRuns.length >= availableSlots || isRunAdmissionClosed()) break;
         let claimed: typeof heartbeatRuns.$inferSelect | null;
         try {
           claimed = await claimQueuedRun(queuedRun, companyAgents, deferredPostCommitEffects, deferredCancellations);

server/src/services/native-runtime/native-restart-recovery.ts

@@ -745,7 +745,8 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
   /** Durable evidence proves a claimed remote bootstrap reached no provider work. */
   remoteBootstrapProvenIncomplete?: boolean;
   remoteBootstrapIncompleteReason?: "remote_runner_never_launched" | "remote_session_opened_without_turn"
-    | "remote_runner_exited_before_provider_work" | "remote_runner_never_received_commands";
+    | "remote_runner_exited_before_provider_work" | "remote_runner_never_received_commands"
+    | "remote_follow_up_never_attached";
   processStartMatches: boolean;
   knownProviderProcessAlive?: boolean;
   knownProviderProcessIdentityAmbiguous?: boolean;
@@ -1592,7 +1593,22 @@ export async function claimNativeRestartRecoveries(input: {
         // Fresh evidence wins over the saved failure: any event the runner or
         // provider sent contradicts "no provider work".
         && !(await hasNativeRunnerOriginatedEvidence(tx, row.run));
-      if ((remoteRunnerNeverLaunched || remoteRunnerExitedBeforeProviderWork) && row.coordinator.phase === "retryable_failure" && !admission.exhausted
+      const remoteFollowUpNeverAttached = !unclaimedBootstrap && !remoteRunnerNeverLaunched && !remoteRunnerExitedBeforeProviderWork
+        && remoteWorkspace?.state === "prepared" && !pluginTask && !durableAuthority && stagedAttachment === null
+        && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION
+        && row.coordinator.phase === "retryable_failure" && row.run.status === "failed" && !row.coordinator.leaseOwner
+        && settledFailure?.recoveryMode === "bootstrap_retry"
+        && settledFailure?.originalFailureCode === "local_runtime_api_listener_unavailable"
+        && settledFailure?.providerSessionEstablished === false && settledFailure?.providerEventsExist === false
+        && !settledFailure?.successorRunId
+        && continuedSessionOnly && inheritedCheckpointOnly && !row.run.sessionIdAfter
+        && Boolean(row.run.nativeSessionId && row.run.runnerInstanceId)
+        && (inheritedRunnerIdentity || (!row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt))
+        && await carriesSettledPredecessorSession(tx, row.run, checkpointRecord.providerSessionId)
+        && !(await hasNativeRunnerOriginatedEvidence(tx, row.run))
+        && !(await hasNativeRunEvent(tx, row.run, PROCESS_IDENTITY_RECORDED));
+      if ((remoteRunnerNeverLaunched || remoteRunnerExitedBeforeProviderWork || remoteFollowUpNeverAttached)
+        && row.coordinator.phase === "retryable_failure" && !admission.exhausted
         && !await readNativeSessionResumptionAgent(tx as unknown as Db, { issue, run: row.run, coordinator: row.coordinator })) {
         const deferred = await deferRemoteRetry("native_retry_admission_ineligible", ["done", "cancelled"].includes(issue.status));
         if (deferred) return deferred;
@@ -1633,7 +1649,7 @@ export async function claimNativeRestartRecoveries(input: {
         && (!ownProviderEvidence || remoteBootstrapProof === "session_opened_without_turn");
 
       const remoteRunnerNeverReceivedCommands = !unclaimedBootstrap && !remoteRunnerNeverLaunched
-        && !remoteRunnerExitedBeforeProviderWork && !remoteBootstrapProvenIncomplete
+        && !remoteRunnerExitedBeforeProviderWork && !remoteBootstrapProvenIncomplete && !remoteFollowUpNeverAttached
         && remoteSandbox && remoteWorkspace !== null && !pluginTask && !durableAuthority && stagedAttachment === null
         && admissionCheckpointOnly && providerEvents.length === 0
         && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION
@@ -1659,10 +1675,11 @@ export async function claimNativeRestartRecoveries(input: {
         remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority),
         unclaimedBootstrap,
         remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched
-          || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands,
+          || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached,
         remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched ? "remote_runner_never_launched"
           : remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work"
           : remoteRunnerNeverReceivedCommands ? "remote_runner_never_received_commands"
+          : remoteFollowUpNeverAttached ? "remote_follow_up_never_attached"
           : remoteBootstrapProof === "session_opened_without_turn" ? "remote_session_opened_without_turn" : undefined,
         runnerPidAlive,
         runnerGroupAlive,
@@ -1671,7 +1688,7 @@ export async function claimNativeRestartRecoveries(input: {
         knownProviderProcessIdentityAmbiguous:
           providerProcesses.ambiguousLivePids.length > 0,
         hasCheckpoint: hasCheckpoint && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly)
-          && !remoteRunnerNeverLaunched && !(remoteBootstrapProvenIncomplete && reboundSessionOnly),
+          && !remoteRunnerNeverLaunched && !remoteFollowUpNeverAttached && !(remoteBootstrapProvenIncomplete && reboundSessionOnly),
         stagedAttachmentVerified: stagedAttachment !== null,
         checkpointIdentityMatches:
           checkpointIdentityMatches &&

server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts

@@ -3105,6 +3105,55 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
     expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_execution_never_claimed" });
   });
 
+  it.each([
+    { label: "bootstraps a follow-up carrying its predecessor's Runner identity", mode: "inherited", restarts: true },
+    { label: "bootstraps a follow-up that requested its warm Runner on its predecessor's identity", mode: "inherited_requested", restarts: true },
+    { label: "bootstraps a follow-up with no Runner identity that requested a Runner", mode: "requested", restarts: true },
+    { label: "keeps a follow-up with its own recorded Runner identity held", mode: "own_identity", restarts: false },
+    { label: "keeps a follow-up with a Runner-originated event held", mode: "runner_event", restarts: false },
+    { label: "keeps a follow-up with a carried active turn held", mode: "active_turn", restarts: false },
+    { label: "keeps a follow-up whose provider session no settled run saved held", mode: "no_predecessor", restarts: false },
+    { label: "keeps a follow-up whose saved failure established a provider session held", mode: "session_established", restarts: false },
+  ] as const)("$label after its own server closed the runtime listener before it attached (s9511, s9513)", async ({ mode, restarts }) => {
+    const fixture = await seedUnlaunchedRemoteClaim(`FOLLOW-UP-LISTENER-${mode}`, { bootstrapRetry: true,
+      ...(["inherited_requested", "requested", "own_identity", "runner_event"].includes(mode)
+        ? { event: { eventType: "native.process_start_requested" } } : {}) });
+    await reboundFollowUpCheckpoint(fixture, mode === "active_turn" ? { activeTurnId: "turn-in-flight" } : {}, mode !== "no_predecessor");
+    await fixture.db.update(nativeRunFinalizations).set({ failureDetail: {
+      message: "local_runtime_api_listener_unavailable", recoverable: true, recoveryMode: "bootstrap_retry",
+      checkpointExists: true, providerEventsExist: false, providerSessionEstablished: mode === "session_established",
+      originalFailureCode: "local_runtime_api_listener_unavailable",
+    } }).where(eq(nativeRunFinalizations.runId, fixture.runId));
+    const [current] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
+    if (mode !== "requested") {
+      const processStartedAt = new Date(Date.now() - 120_000);
+      await fixture.db.update(heartbeatRuns).set({ processPid: 596, processGroupId: null, processStartedAt, processHostHash: "remote" })
+        .where(eq(heartbeatRuns.id, fixture.runId));
+      if (mode !== "own_identity") await fixture.db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded",
+        runtimeMode: "native", nativeIssueId: fixture.issueId, nativeSessionId: current.nativeSessionId,
+        runnerInstanceId: current.runnerInstanceId, createdAt: new Date(current.createdAt.getTime() - 60_000),
+        finishedAt: new Date(current.createdAt.getTime() - 5_000), processPid: 596, processGroupId: null, processStartedAt,
+        processHostHash: "remote", contextSnapshot: { issueId: fixture.issueId } });
+    }
+    if (mode === "own_identity") await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId,
+      seq: 901, stream: "system", level: "info", eventType: "native.process_identity_recorded" });
+    if (mode === "runner_event") await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId,
+      seq: 902, stream: "system", level: "info", eventType: "item.updated", sourceEventId: "runner-event-after-failure" });
+    const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] });
+    if (!restarts) {
+      expect(claim).not.toMatchObject({ kind: "bootstrap_incomplete" });
+      return;
+    }
+    expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 });
+    const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
+      .where(eq(nativeRunFinalizations.runId, fixture.runId));
+    expect(coordinator).toMatchObject({ phase: "observed", recoveryState: "bootstrap_incomplete", failureCode: null });
+    expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_follow_up_never_attached" });
+    const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
+    expect(run).toMatchObject({ status: "running", errorCode: null, finishedAt: null });
+  });
+
   it("restarts a remote bootstrap retry that shutdown cut before any runner was requested", async () => {
     const fixture = await seedUnlaunchedRemoteClaim("BOOTSTRAP-RETRY", { bootstrapRetry: true });
     expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,