fix(native): restart a remote bootstrap retry that shutdown cut before runner launch (#19684)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. Restart recovery decides what to do with each live or due run
after the control plane restarts.
> - #19616 added the `remote_runner_never_launched` proof. A claimed
remote run with no runner process request and no controller checkpoint
starts its bootstrap again.
> - A graceful shutdown closes the local runtime listener before its
drain. A remote run that is still starting then fails with
`local_runtime_api_listener_unavailable`. The executor settles it as a
same-run `bootstrap_retry` and releases its lease. #19596 treats this as
a restart detach only for local runners.
> - When the server restarted with that retry due, restart recovery
claimed the row. The proof accepted only rows that still held a lease,
so recovery blocked the task as ambiguous.
> - This pull request lets the same proof also admit that settled
`bootstrap_retry` row.
> - The benefit is that a graceful restart during Daytona startup
continues automatically.

## Linked Issues or Issue Description

**What happened?**

A chaos wave restarted the control plane gracefully (SIGTERM) while an
ACPX Claude run on Daytona was starting. At 2 s after `stage.workspace`,
the run failed with `local_runtime_api_listener_unavailable`, and the
executor persisted "same-run resume persisted" with `recoveryMode:
bootstrap_retry`. After the restart, restart recovery returned
`ambiguous_remote_recovery_evidence` with disposition `blocked`. The
task rested blocked.

Evidence in the saved database at that point:
- Coordinator `phase: retryable_failure`, `attempt: 1`,
`controllerGeneration: 1`, controller identity recorded, and no lease
owner.
- `failureDetail` recorded `recoveryMode: bootstrap_retry`,
`checkpointExists: false`, `providerEventsExist: false` and
`providerSessionEstablished: false`.
- The run had no runner checkpoint rows, no
`native.process_start_requested`, no process identity, and no
runner-sourced event. It also had no PID and no provider session.

**Expected behavior**

Restart recovery starts the bootstrap again, as it does for a claimed
run that holds a lease (#19616) and for a local run.

**Steps to reproduce**

1. Start a native run on a Daytona environment.
2. Send SIGTERM to the control plane after workspace staging and before
the runner is requested. The run settles as a same-run
`bootstrap_retry`.
3. Restart the control plane while that retry is due.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.

Root cause: in
`server/src/services/native-runtime/native-restart-recovery.ts`, the
`remoteRunnerNeverLaunched` condition required `phase === "observed"`
and a lease owner. The settled `retryable_failure` / `bootstrap_retry`
row failed that check. The remote classifier then returned
`ambiguous_remote_recovery_evidence`.

Refs #19616, #19596, #19583

## What Changed

- `native-restart-recovery.ts`: `remoteRunnerNeverLaunched` now accepts
one of two coordinator shapes:
  - an `observed` claim that holds a lease (unchanged);
- a `retryable_failure` with `recoveryMode: bootstrap_retry`, run status
`failed`, and no lease owner.
- All other proof conditions are unchanged. The run must still have no
process start, process identity, adapter, tool or runner-sourced event.
It must still have no process fields, no provider session, no provider
evidence and no checkpoint. The same three-attempt budget applies.
- An `exact_checkpoint_resume` retry keeps its own admission path.
- Admission: before claiming a settled retry, restart recovery applies
the same admission as the resumption reconciler
(`readNativeSessionResumptionAgent`): task pause hold, agent status,
dependencies, ownership, and competing runs. An ineligible retry is
deferred with `native_retry_admission_ineligible` and keeps waiting.
- `doc/run-log-events.md`: note the settled `bootstrap_retry` case.

## Verification

- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "bootstrap retry"`. Before the fix, the proven case and the budget
case returned `blocked` / `ambiguous_remote_recovery_evidence`.
- New cases: the proven case gets `bootstrap_incomplete`. The
coordinator returns to `observed`, `nextAttemptAt` is cleared, the run
returns to `running`, and no recovery action is created. A requested
process start or a committed runner event keeps the hold. A spent budget
blocks with `execution_recovery_budget_exhausted`. An
`exact_checkpoint_resume` retry does not take this path.
- Regression: `native-runner-restart-recovery.integration`,
`heartbeat/recovery`, `native-restart-recovery`,
`native-bootstrap-retry.integration` and
`native-remote-retry-admission`: 337 passed, 14 skipped.
- Admission: a paused agent's due `bootstrap_retry` is deferred with
`native_retry_admission_ineligible`. It is not started (red before the
follow-up commit, green after).
- Live Daytona replay of `w-20261106T234719-s12/t02`, 3 runs, ACPX
Claude B1, using the recorded K4, K2 and K1 sequence. Image: the wave's
own `paperclip-daytona-runner@sha256:2767e91c…`, with runner inputs
pinned to its revision e482f0f632 locally. The replays ran the first
commit of this PR.
- Runs 1 and 3 passed R1. Run 2 passed R3: one Retry worked on the first
click.
- Runs 1 and 2 failed R6 because the wave-end sweep found a run lease
still in `pending_cleanup`. In run 1 the environment delete also timed
out. These failures are not in this change's path.
- The restarts landed in other startup windows, so none hit this exact
`bootstrap_retry` state. The integration tests cover that state with the
same evidence as the saved failing database.
- Run 2 also exposed a separate gap. A graceful drain that suspended a
remote run after the runner launched (`awaiting_runner_reattach`) was
blocked as ambiguous on the next boot. The Retry recovered it. That case
is outside this PR.

## Risks

- The proof still depends on the executor writing
`native.process_start_requested` before a backend may spawn a runner.
The fail-closed tests pin that gate.
- Restart recovery and the same-run resumption reconciler can both see a
due row. Both claim with a conditional update on phase and attempt, so
only one can win.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/run-log-events.md

@@ -361,6 +361,9 @@ runner-sourced event or controller checkpoint, records
 `remote_runner_never_launched` with the same disposition and budget. The
 executor records `native.process_start_requested` before a backend may spawn a
 runner, so its absence (with no checkpoint) proves nothing reached a runner.
+The same rule covers a remote run that its executor already settled as a
+same-run `bootstrap_retry` (for example when a shutdown closed the local
+runtime listener during startup) and that is due when the server restarts.
 
 For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed`
 event retains `error.code: "provider_turn_lost_on_restore"` and

server/src/services/native-runtime/native-restart-recovery.ts

@@ -1128,15 +1128,29 @@ export async function claimNativeRestartRecoveries(input: {
       const remoteRunnerNeverLaunched = !unclaimedBootstrap && remoteWorkspace?.state === "prepared" && !pluginTask
         && !durableAuthority && stagedAttachment === null
         && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION
-        && row.coordinator.phase === "observed" && row.coordinator.attempt >= 1 && row.coordinator.controllerGeneration >= 1
-        && Boolean(row.coordinator.controllerBootId && row.coordinator.controllerPid
-          && row.coordinator.controllerProcessStartedAt && row.coordinator.leaseOwner)
+        && row.coordinator.attempt >= 1 && row.coordinator.controllerGeneration >= 1
+        && Boolean(row.coordinator.controllerBootId && row.coordinator.controllerPid && row.coordinator.controllerProcessStartedAt)
+        // Either the executor still held the claim, or it settled a same-run
+        // bootstrap retry itself (for example after a shutdown closed its local
+        // runtime listener) and released the lease. That retry already proved
+        // no provider session, event or checkpoint; the checks below re-prove it.
+        && (row.coordinator.phase === "observed" ? Boolean(row.coordinator.leaseOwner)
+          : row.coordinator.phase === "retryable_failure" && row.run.status === "failed"
+            && row.coordinator.failureDetail?.recoveryMode === "bootstrap_retry" && !row.coordinator.leaseOwner)
         && !row.coordinator.failureDetail?.successorRunId
         && !row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt
         && !row.run.sessionIdBefore && !row.run.sessionIdAfter
         && Boolean(row.run.nativeSessionId && row.run.runnerInstanceId)
         && !hasCheckpoint && !hasProviderEvidence
         && !(await hasNativeExecutionStartEvidence(tx, row.run));
+      // A settled retry starts new work, so it needs the same current
+      // admission as the resumption reconciler: task pause, agent status,
+      // dependencies, ownership and competing runs. Keep it waiting otherwise.
+      if (remoteRunnerNeverLaunched && row.coordinator.phase === "retryable_failure" && !admission.exhausted
+        && !await readNativeSessionResumptionAgent(tx as unknown as Db, { issue, run: row.run, coordinator: row.coordinator })) {
+        const deferred = await deferRemoteRetry("native_retry_admission_ineligible", ["done", "cancelled"].includes(issue.status));
+        if (deferred) return deferred;
+      }
 
       // A claimed remote bootstrap (runner launched, perhaps connected) has no
       // provider evidence when its controller dies before session.open

server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts

@@ -2374,6 +2374,8 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
     attempt?: number; leaseExpired?: boolean; liveController?: boolean;
     event?: { eventType: string; sourceEventId?: string };
     run?: Partial<typeof heartbeatRuns.$inferInsert>;
+    /** RC-29c: the executor settled a same-run bootstrap retry before the restart. */
+    bootstrapRetry?: boolean;
   } = {}) {
     const fixture = await seedRun(`REMOTE-UNLAUNCHED-${label}`);
     await fixture.db.update(heartbeatRuns).set({
@@ -2391,9 +2393,78 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
       leaseOwner: `previous:${randomUUID()}`,
       leaseExpiresAt: new Date(Date.now() + (options.leaseExpired ? -1_000 : 20 * 60_000)),
     }).where(eq(nativeRunFinalizations.runId, fixture.runId));
+    if (options.bootstrapRetry) {
+      // Shape persisted by the executor when shutdown closed the local runtime
+      // listener during remote startup (local_runtime_api_listener_unavailable).
+      await fixture.db.update(nativeRunFinalizations).set({
+        phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null,
+        nextAttemptAt: new Date(Date.now() - 1_000), failureCode: "local_runtime_api_listener_unavailable",
+        failureDetail: { message: "local_runtime_api_listener_unavailable", recoverable: true,
+          recoveryMode: "bootstrap_retry", checkpointExists: false, providerEventsExist: false,
+          providerSessionEstablished: false, originalFailureCode: "local_runtime_api_listener_unavailable" },
+      }).where(eq(nativeRunFinalizations.runId, fixture.runId));
+      await fixture.db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(),
+        errorCode: "local_runtime_api_listener_unavailable" }).where(eq(heartbeatRuns.id, fixture.runId));
+    }
     return fixture;
   }
 
+  it("restarts a remote bootstrap retry that shutdown cut before any runner was requested", async () => {
+    const fixture = await seedUnlaunchedRemoteClaim("BOOTSTRAP-RETRY", { bootstrapRetry: true });
+    expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
+      kind: "bootstrap_incomplete", runId: fixture.runId, providerAttempt: 1, controllerGeneration: 2 })]);
+    const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
+      .where(eq(nativeRunFinalizations.runId, fixture.runId));
+    expect(coordinator).toMatchObject({ phase: "observed", recoveryState: "bootstrap_incomplete", failureCode: null,
+      nextAttemptAt: null });
+    expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_launched" });
+    const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
+    expect(run).toMatchObject({ status: "running", errorCode: null, finishedAt: null });
+    expect(await fixture.db.select().from(issueRecoveryActions)
+      .where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId))).toHaveLength(0);
+  });
+
+  it.each<[string, Parameters<typeof seedUnlaunchedRemoteClaim>[1], { kind: string; reason: string }]>([
+    ["a runner process start was requested", { bootstrapRetry: true, event: { eventType: "native.process_start_requested" } },
+      { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }],
+    ["a runner event was committed", { bootstrapRetry: true, event: { eventType: "item.updated", sourceEventId: "runner-event-1" } },
+      { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }],
+    ["the bootstrap budget is spent", { bootstrapRetry: true, attempt: 3 },
+      { kind: "blocked", reason: "execution_recovery_budget_exhausted" }],
+  ])("keeps a remote bootstrap retry held when %s", async (label, options, expected) => {
+    const fixture = await seedUnlaunchedRemoteClaim(`RETRY-${label.replace(/\W+/g, "-")}`, options);
+    expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ runId: fixture.runId, ...expected })]);
+  });
+
+  it("keeps a paused agent's remote bootstrap retry waiting instead of starting it", async () => {
+    const fixture = await seedUnlaunchedRemoteClaim("PAUSED-RETRY", { bootstrapRetry: true });
+    await fixture.db.update(agents).set({ status: "paused" }).where(eq(agents.id, agentId));
+    try {
+      expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+        restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
+        kind: "blocked", runId: fixture.runId, reason: "native_retry_admission_ineligible" })]);
+      const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
+      expect(run.status).toBe("failed");
+      const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
+        .where(eq(nativeRunFinalizations.runId, fixture.runId));
+      expect(coordinator).toMatchObject({ phase: "retryable_failure", leaseOwner: null });
+      expect(coordinator.nextAttemptAt!.getTime()).toBeGreaterThan(Date.now());
+    } finally {
+      await fixture.db.update(agents).set({ status: "active" }).where(eq(agents.id, agentId));
+    }
+  });
+
+  it("leaves a remote exact-checkpoint retry to its own admission path", async () => {
+    const fixture = await seedUnlaunchedRemoteClaim("EXACT-RETRY", { bootstrapRetry: true });
+    await fixture.db.update(nativeRunFinalizations).set({ failureDetail: { recoveryMode: "exact_checkpoint_resume" } })
+      .where(eq(nativeRunFinalizations.runId, fixture.runId));
+    const [disposition] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] });
+    expect(disposition.kind).not.toBe("bootstrap_incomplete");
+  });
+
   it.each([false, true])("restarts a claimed remote run that never requested a runner (lease expired=%s)", async (leaseExpired) => {
     const fixture = await seedUnlaunchedRemoteClaim(`PROVEN-${leaseExpired}`, { leaseExpired });
     expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,