fix(native): bootstrap again when a relaunched remote runner exited before provider work (#19951)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Restart recovery decides whether an interrupted native run can
resume, bootstrap again, or must wait for a person.
> - A remote run is blocked as "ambiguous" when recovery cannot prove
what reached the sandbox runner.
> - In a Daytona chaos wave (#19947), two runs were blocked even though
the controller had recorded that the relaunched runner exited before any
provider work.
> - This pull request accepts that recorded runner exit as proof and
bootstraps the run again, under the existing attempt budget.
> - The benefit is that a restart no longer turns a run with no provider
work into a failed, blocked run.

## Linked Issues or Issue Description

Refs #19947 (Daytona data), #19831 (durable-ownership plan), #19934 (the
stuck claimed wake from the same runs).

**What happened?**
Wave `w-20261108T012041-s9301`, runs `450f277d…` and `9b93d40c…`, went
through this sequence:
1. Restart recovery proved the remote bootstrap incomplete and
relaunched it.
2. The new runner exited: "runnerd exited while waiting for
run.prepare". The executor settled `bootstrap_retry` with
`providerSessionEstablished: false`, `providerEventsExist: false` and
`checkpointExists: false`.
3. The next restart recovery saw the recorded runner PID and blocked the
run as `ambiguous_remote_recovery_evidence`.

**What did you expect?**
Bootstrap again, because the runner is proven gone and no provider work
happened.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "relaunched remote runner"` fails on master with `blocked /
ambiguous_remote_recovery_evidence`.

## What Changed

- `native-restart-recovery.ts`: new proof
`remoteRunnerExitedBeforeProviderWork`. It applies to a remote sandbox
with a prepared workspace and a settled `bootstrap_retry` failure whose
original code is `native_runner_process_exited`, with no provider
session, events or checkpoint and no lease or successor. Such a run is
classified as `bootstrap_incomplete`
(`remote_runner_exited_before_provider_work`). It passes the same retry
admission check as `remote_runner_never_launched`, and the same
three-attempt budget.
- Tests: the s9301-shaped retry plus five counter-cases.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "exited remote runner|relaunched remote runner|remote bootstrap
retry|never requested a runner|exact-checkpoint retry"`: 14 pass.
- Red/green: the new bootstrap test and the budget counter-case failed
on master (both returned `blocked / ambiguous_remote_recovery_evidence`,
the same as the s9301 rows) and pass here. The other four counter-cases
(not a runner exit, session established, provider events, checkpoint)
block on both.
- Loop: focused vitest with one embedded Postgres, 26–50 s.
- Not run locally: the server typecheck and the full suite (memory
limits on this host). Required CI covers them.

## Risks

- Fail-open risk: the proof needs the executor's own settled record of a
runner exit with no provider work, plus a fresh check that no checkpoint
or provider evidence exists. A recorded PID without the exit record
still blocks, as before.
- Previous-release story (rules/upgrade-safe-run-state.mdc): no
persisted shape changes. Rows without these failure fields keep today's
outcome.
- Nothing runs before `turn.start`. This is restart recovery only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/native-runtime/native-restart-recovery.ts

@@ -668,7 +668,8 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
   unclaimedBootstrap?: boolean;
   /** Durable evidence proves a claimed remote bootstrap reached no provider work. */
   remoteBootstrapProvenIncomplete?: boolean;
-  remoteBootstrapIncompleteReason?: "remote_runner_never_launched" | "remote_session_opened_without_turn";
+  remoteBootstrapIncompleteReason?: "remote_runner_never_launched" | "remote_session_opened_without_turn"
+    | "remote_runner_exited_before_provider_work";
   processStartMatches: boolean;
   knownProviderProcessAlive?: boolean;
   knownProviderProcessIdentityAmbiguous?: boolean;
@@ -768,6 +769,21 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
 
 /** Any record that execution reached a runner: a requested or identified
  * process, an adapter invocation, a tool start, or a runner-sourced event. */
+/** Whether a runner or provider produced any event for this run. Unlike
+ * hasNativeExecutionStartEvidence, the controller's own process-start and
+ * identity records do not count: a relaunched runner that exited before run.prepare
+ * has those, but nothing the runner itself sent. */
+async function hasNativeRunnerOriginatedEvidence(
+  tx: Pick<Db, "select">, run: { companyId: string; id: string },
+): Promise<boolean> {
+  const [event] = await tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents)
+    .where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
+      or(isNotNull(heartbeatRunEvents.sourceEventId),
+        inArray(heartbeatRunEvents.eventType, ["adapter.invoke", "tool.execution.started"]))))
+    .limit(1);
+  return event !== undefined;
+}
+
 export async function hasNativeExecutionStartEvidence(
   tx: Pick<Db, "select">, run: { companyId: string; id: string },
 ): Promise<boolean> {
@@ -1333,7 +1349,28 @@ export async function claimNativeRestartRecoveries(input: {
       // A settled retry starts new work, so it needs the same current
       // admission as the resumption reconciler: task pause, agent status,
       // dependencies, ownership and competing runs. Keep it waiting otherwise.
-      if (remoteRunnerNeverLaunched && row.coordinator.phase === "retryable_failure" && !admission.exhausted
+      // The executor relaunched a remote runner for a same-run bootstrap
+      // retry, and that runner exited (observed by its transport) before any
+      // provider work: the executor's settled failure durably records no
+      // provider session, no provider event and no checkpoint, and none exists
+      // now. The runner's own exit is the remote proof that it is gone; a
+      // recorded runner PID alone would not be (Daytona wave s9301).
+      const settledFailure = row.coordinator.failureDetail;
+      const remoteRunnerExitedBeforeProviderWork = !unclaimedBootstrap && !remoteRunnerNeverLaunched
+        && remoteWorkspace?.state === "prepared" && !pluginTask && stagedAttachment === null
+        && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION
+        && row.coordinator.phase === "retryable_failure" && row.run.status === "failed" && !row.coordinator.leaseOwner
+        && settledFailure?.recoveryMode === "bootstrap_retry"
+        && settledFailure?.originalFailureCode === "native_runner_process_exited"
+        && settledFailure?.providerSessionEstablished === false && settledFailure?.providerEventsExist === false
+        && settledFailure?.checkpointExists === false && !settledFailure?.successorRunId
+        && !row.run.sessionIdBefore && !row.run.sessionIdAfter
+        && Boolean(row.run.nativeSessionId && row.run.runnerInstanceId)
+        && !hasCheckpoint && !hasProviderEvidence
+        // Fresh evidence wins over the saved failure: any event the runner or
+        // provider sent contradicts "no provider work".
+        && !(await hasNativeRunnerOriginatedEvidence(tx, row.run));
+      if ((remoteRunnerNeverLaunched || remoteRunnerExitedBeforeProviderWork) && row.coordinator.phase === "retryable_failure" && !admission.exhausted
         && !await readNativeSessionResumptionAgent(tx as unknown as Db, { issue, run: row.run, coordinator: row.coordinator })) {
         const deferred = await deferRemoteRetry("native_retry_admission_ineligible", ["done", "cancelled"].includes(issue.status));
         if (deferred) return deferred;
@@ -1370,8 +1407,10 @@ export async function claimNativeRestartRecoveries(input: {
       const classification = classifyNativeRunnerRecoveryEvidence({
         remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority),
         unclaimedBootstrap,
-        remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched,
+        remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched
+          || remoteRunnerExitedBeforeProviderWork,
         remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched ? "remote_runner_never_launched"
+          : remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work"
           : remoteBootstrapProof === "session_opened_without_turn" ? "remote_session_opened_without_turn" : undefined,
         runnerPidAlive,
         runnerGroupAlive,

server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts

@@ -2678,6 +2678,66 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
       .where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId))).toHaveLength(0);
   });
 
+  /** Daytona wave s9301: the relaunched remote runner exited before run.prepare. */
+  async function seedExitedRemoteRunner(label: string, failure: Record<string, unknown> = {}, attempt = 2) {
+    const fixture = await seedUnlaunchedRemoteClaim(label, { bootstrapRetry: true, attempt,
+      event: { eventType: "native.process_start_requested" }, run: { processPid: 955, processStartedAt: new Date(Date.now() - 30_000) } });
+    await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId, seq: 901, stream: "system",
+      level: "info", eventType: "native.process_identity_recorded" });
+    await fixture.db.update(nativeRunFinalizations).set({ failureCode: "native_runner_process_exited",
+      failureDetail: { message: "runnerd exited while waiting for run.prepare", recoverable: true, recoveryMode: "bootstrap_retry",
+        checkpointExists: false, providerEventsExist: false, providerSessionEstablished: false,
+        originalFailureCode: "native_runner_process_exited", ...failure } })
+      .where(eq(nativeRunFinalizations.runId, fixture.runId));
+    await fixture.db.update(heartbeatRuns).set({ errorCode: "native_runner_process_exited" }).where(eq(heartbeatRuns.id, fixture.runId));
+    return fixture;
+  }
+
+  it("bootstraps again when the relaunched remote runner exited before any provider work", async () => {
+    // The runner's own exit, observed and recorded by the controller with no
+    // provider session, events or checkpoint, proves the remote runner gone.
+    const fixture = await seedExitedRemoteRunner("EXITED-RUNNER");
+    expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
+      kind: "bootstrap_incomplete", runId: fixture.runId, providerAttempt: 2 })]);
+    const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
+      .where(eq(nativeRunFinalizations.runId, fixture.runId));
+    expect(coordinator).toMatchObject({ phase: "observed", recoveryState: "bootstrap_incomplete", failureCode: null });
+    expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_exited_before_provider_work" });
+    const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
+    expect(run).toMatchObject({ status: "running", errorCode: null });
+  });
+
+  it.each(["a provider event committed after the failure", "a session checkpoint saved after the failure"])(
+    "keeps an exited remote runner's retry held when %s contradicts the saved failure", async (label) => {
+      // The saved failure still says no provider work; fresh evidence wins.
+      const fixture = await seedExitedRemoteRunner(`EXITED-FRESH-${label.split(" ")[1]}`);
+      if (label.includes("event")) {
+        await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId, seq: 902,
+          stream: "system", level: "info", eventType: "item.updated", sourceEventId: "runner-event-after-failure" });
+      } else {
+        const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
+        await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, sessionCheckpoint: {} } })
+          .where(eq(heartbeatRuns.id, fixture.runId));
+      }
+      expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+        restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
+        kind: "blocked", runId: fixture.runId, reason: "ambiguous_remote_recovery_evidence" })]);
+    });
+
+  it.each<[string, Record<string, unknown>, number, string]>([
+    ["the failure was not a runner exit", { originalFailureCode: "local_runtime_api_listener_unavailable" }, 2, "ambiguous_remote_recovery_evidence"],
+    ["the provider session was established", { providerSessionEstablished: true }, 2, "ambiguous_remote_recovery_evidence"],
+    ["provider events exist", { providerEventsExist: true }, 2, "ambiguous_remote_recovery_evidence"],
+    ["a checkpoint exists", { checkpointExists: true }, 2, "ambiguous_remote_recovery_evidence"],
+    ["the bootstrap budget is spent", {}, 3, "execution_recovery_budget_exhausted"],
+  ])("keeps an exited remote runner's retry held when %s", async (label, failure, attempt, reason) => {
+    const fixture = await seedExitedRemoteRunner(`EXITED-${label.replace(/\W+/g, "-")}`, failure, attempt);
+    expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
+      kind: "blocked", runId: fixture.runId, reason })]);
+  });
+
   it.each<[string, Parameters<typeof seedUnlaunchedRemoteClaim>[1], { kind: string; reason: string }]>([
     ["a runner process start was requested", { bootstrapRetry: true, event: { eventType: "native.process_start_requested" } },
       { kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }],