fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error

## Linked Issues or Issue Description

Refs #20808
Refs #20832
Refs #20820

**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.

**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.

**Paperclip version or commit**
master at 3ded7bbd91.

## What Changed

- Runner transport (`runnerd-codex-transport.ts`): new
`runnerExitConfirmed()`. It is true only after a completed close settled
the runner (provider stopped, runner suspended) and the runner process
exit was observed.
- Codex driver (`codex-app-server-driver-impl.ts`): after it closes the
transport for a failed session open, it sets `runnerExitConfirmed` on
the error when the transport proves the exit.
- Executor (`native-session-executor.ts`):
- A remote `provider_session_open_rejected` with a startup
`providerFailure` keeps the same-run `bootstrap_retry` only with that
proof. The saved failure records `runnerExitConfirmed: true`.
- Each failed attempt's recovery history entry records a startup failure
signature: provider, category and message, with numbers ignored. When
the next attempt fails with the same signature, it stops as
`native_provider_startup_rejected`. Its next action names the startup
error and keeps "Retry starts a new run".
- Deterministic rejections (`requires_action` authentication, permission
or invalid request) stop at once as before.
  - Local targets are unchanged.
- Restart recovery (`native-restart-recovery.ts`):
`remoteRunnerExitedBeforeProviderWork` also accepts a settled
`provider_session_open_rejected` with `runnerExitConfirmed: true`. All
its other checks still apply: no provider session, no provider event, no
checkpoint, no runner-sent event, no successor.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

- Red on master, green here:
- `native-startup-rejected-retry.integration.test.ts`: proven remote
rejection → `retryable_failure` with `runnerExitConfirmed`; unproven →
terminal; repeat → terminal, the next action names `bwrap: loopback`,
and Retry is offered.
- `native-runner-restart-recovery.integration.test.ts`: proven failure →
`bootstrap_incomplete` (`remote_runner_exited_before_provider_work`); an
older release's failure without the proof stays held.
- `codex-app-server-driver.lifecycle.test.ts`: the driver records the
proof only when the transport reports it after close.
- `native-session-executor.test.ts`: unit cases for the rule, the
`cause` chain, the repeat signature and deterministic rejections.
- Local runs: the 3 server files above pass (1380 tests); restart
recovery `-t remote` 75 pass; driver lifecycle 25 pass.
- `tsc --noEmit` clean for `server` and `packages/paperclip-runner`.
- Not run: a Daytona chaos wave. There is no in-sandbox startup fault
seam yet (see Risks).

## Risks

- The proof depends on the transport's close settling the runner after a
failed `session.open`. The Rust runner keeps serving after a failed
`session.open` (lifecycle `Continue`) and accepts `runner.suspend`, so
close should settle. If it does not settle, there is no proof and the
run ends at once with Retry, which is today's behaviour. It is never
less safe than today.
- Retries stay bounded: the startup retry budget (3 attempts) and the
repeat rule both apply. A repeated identical error stops after one
automatic retry.
- Saved state: failures saved by older releases have no
`runnerExitConfirmed` and keep the old rule. History entries without a
signature never match a repeat. A mid-run deploy is safe: an old
executor never writes the proof.
- Not covered by chaos: a remote "Codex fails at startup N times" fault
needs a hook inside the sandbox Runner or Codex. DF1's seam is
server-side only, and no open chaos PR adds a seam inside the sandbox.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/execution-semantics.md

@@ -1358,8 +1358,12 @@ same-run bootstrap retry, and leaves an active board recovery action that asks
 for a supported model. A definite rejection whose startup `providerFailure`
 is `requires_action` for `authentication`, `permission` or `invalid_request`
 stops on its first attempt as `native_provider_startup_rejected`. On a remote
-execution target, any definite rejection with a startup `providerFailure` stops
-the same way, because a remote same-run bootstrap cannot prove ownership. No
+execution target, another definite rejection with a startup `providerFailure`
+retries on the same run only when the Runner transport proved, at close, that
+the failed runner settled and its process exited (`runnerExitConfirmed` in the
+saved failure). Restart recovery then launches a new runner. A rejection
+without that proof, or one that repeats the previous attempt's startup error,
+stops as `native_provider_startup_rejected`; the repeat names that error. No
 provider work began, so the board offers Retry, which starts a new run with a
 fresh provider session. Other local `session.open` rejections keep the bounded
 startup retry and end with Retry when it is spent. An

packages/paperclip-runner/docs/durable-recovery.md

@@ -159,10 +159,13 @@ which proves that the sidecar and the provider are gone.
   `authentication`, `permission` or `invalid_request` records
   `native_provider_startup_rejected`. No provider work began, so the
   controller stops same-run bootstrap retries and the board offers Retry at once
-  with the provider message. On a remote execution target, every definite
-  `session.open` failure with a startup `providerFailure` records the same code,
-  because a remote same-run bootstrap after the runner launched cannot prove
-  ownership and would stop on `ambiguous_remote_recovery_evidence`. On a local
+  with the provider message. On a remote execution target, a definite
+  `session.open` failure with a startup `providerFailure` keeps a same-run
+  bootstrap retry only when the transport's close settled the runner and
+  observed its exit (`runnerExitConfirmed`); restart recovery accepts that proof
+  as it accepts a runner's own exit. Without it, or when the failure repeats the
+  previous attempt's startup error, it records the same code, because a remote
+  same-run bootstrap cannot then prove ownership. On a local
   target, other startup failures (unknown, transient, rate limit, quota, model)
   keep the bounded same-run startup retries and end as
   `native_session_retry_exhausted` with Retry. A run an older release held on

packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs

@@ -1327,6 +1327,12 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
                 if let Some(delay) = argument(&args, "--thread-start-delay-ms") {
                     thread::sleep(Duration::from_millis(delay.parse()?));
                 }
+                // A definite provider startup rejection, as Codex answers when
+                // its sandbox helper cannot start (no thread is created).
+                if let Some(reason) = argument(&args, "--reject-thread-start") {
+                    send(json!({"id": id, "error": {"code": -32603, "message": reason}}))?;
+                    continue;
+                }
                 if require_external_sandbox
                     && (message.pointer("/params/sandbox") != Some(&json!("danger-full-access"))
                         || message.pointer("/params/permissions").is_some())

packages/paperclip-runner/src/drivers/codex/app-server-transport.ts

@@ -76,6 +76,8 @@ export interface CodexAppServerTransport {
    * diagnostic context; transports must not forward it to the provider.
    */
   close(reason?: string): Promise<void>;
+  /** After close: the runner settled and its process exit was observed. */
+  runnerExitConfirmed?(): boolean;
   /** Return the confirmed released source, or false when active use was not released. */
   release?(): Promise<DurableRecoveryIdentity | false>;
   /** Relinquish controller authority while leaving durable runner work alive. */

packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts

@@ -373,6 +373,11 @@ export class CodexAppServerDriver implements HarnessDriver {
       await cancellation.close().catch(() => {});
       if (error instanceof NativeSessionProtocolIntegrityError) throw error;
       if (input.signal?.aborted) input.signal.throwIfAborted();
+      // The controller may retry this bootstrap on the same run only when the
+      // failed runner is proven gone; record that proof on the failure.
+      if (error && typeof error === "object" && transport.runnerExitConfirmed?.() === true) {
+        Object.defineProperty(error, "runnerExitConfirmed", { value: true, enumerable: false });
+      }
       if (error instanceof CodexRpcError && (this.#options.driverIdentity?.kind ?? DRIVER_KIND) === DRIVER_KIND) {
         const providerFailure = codexProviderFailure(error, "startup", this.#options.environment);
         if (providerFailure) Object.defineProperty(error, "providerFailure", { value: providerFailure, enumerable: false });

packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts

@@ -126,6 +126,21 @@ describe("Codex app-server Codex driver", () => {
     else expect(error.providerFailure).toBeUndefined();
   });
 
+  it.each([true, false])("records the transport's runner exit proof (%s) on a failed session open after close", async (proven) => {
+    class ExitProofTransport extends FakeCodexTransport {
+      closed = false;
+      override async close(): Promise<void> { await super.close(); this.closed = true; }
+      // Proof is read only after the driver's close completed.
+      runnerExitConfirmed(): boolean { return proven && this.closed; }
+    }
+    const transport = new ExitProofTransport();
+    transport.rejectMethods.set("thread/start", new CodexRpcError("error creating thread: bwrap: loopback: Failed RTM_NEWADDR", -32603));
+    const error = await makeDriver([transport]).openSession({ runId: "exit-proof", normalizedSessionId: "exit-proof",
+      workingDirectory: WORKSPACE }).catch(error => error);
+    expect(transport.closed).toBe(true);
+    expect(error.runnerExitConfirmed === true).toBe(proven);
+  });
+
   it("accepts runner-proven warm attachment when the host active-turn reducer is stale", async () => {
     const transport = new WarmAttachTransport();
     const driver = makeDriver([transport]);

packages/paperclip-runner/src/live/runnerd-codex-transport.ts

@@ -4045,6 +4045,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
   #resolveClosedSignal!: () => void;
   #closePromise: Promise<void> | null = null;
   #shutdownSettled = false;
+  #closeFinished = false;
   #probeRetirement: Promise<void> | null = null;
   readonly prepareConversationResume?: CodexAppServerTransport["prepareConversationResume"];
   #detachPromise: Promise<void> | null = null;
@@ -4275,6 +4276,20 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
     return structuredClone(this.#evidence);
   }
 
+  /**
+   * True only when this transport's startup failed before any provider
+   * identity existed, its close has finished, and the runner process's exit
+   * was observed (it suspended and exited, or close killed it and waited).
+   * A runner that failed session.open cannot always suspend, and its
+   * controller checkpoint can be refused for lack of a provider identity, so
+   * neither is required. A remote same-run bootstrap retry may then launch a
+   * new runner: the old one is gone, and its provider never opened a thread.
+   */
+  runnerExitConfirmed(): boolean {
+    return this.#closeFinished && !this.#startupComplete && this.#providerIdentity === null
+      && this.#evidence.runnerExited === true && this.#handle?.isExitConfirmed?.() !== false;
+  }
+
   async request(
     method: string,
     params: Record<string, unknown>,
@@ -5421,7 +5436,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
     const deadline = Date.now() + runnerCloseGraceMs(this.options);
     try {
       await this.#closeAndSettle();
+      this.#closeFinished = true;
     } catch (error) {
+      this.#closeFinished = true;
       this.#queue.close();
       try { await this.#retireControllerForRecovery(deadline); }
       catch (retirementError) {

packages/paperclip-runner/src/live/runnerd-startup-rejection-exit.test.ts

@@ -0,0 +1,81 @@
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+import { expect, it } from "vitest";
+import { createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "./runnerd-codex-transport.js";
+
+const REJECTION = "error creating thread: fs sandbox helper failed: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted";
+
+function processAbsent(pid: number): boolean {
+  try { process.kill(pid, 0); return false; } catch { return true; }
+}
+
+it.each([
+  ["saves an unsettled checkpoint", false],
+  // The controller's real remote checkpoint refuses a session with no provider
+  // identity, which is exactly the state a rejected session.open leaves.
+  ["refuses the checkpoint for lack of a provider identity", true],
+])("proves a runner whose provider rejected session.open stopped, when the controller %s", async (_label, refuseCheckpoint) => {
+  const root = await mkdtemp(join(tmpdir(), "runner-startup-rejection-"));
+  const settlements: string[] = [];
+  const bundle = createCapabilityRunnerdCodexTransport({
+    runnerBinary: defaultCapabilityRunnerdBinary(),
+    codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
+    codexArgs: ["--state-file", join(root, "provider.json"), "--reject-thread-start", REJECTION],
+    stateDirectory: root,
+    controlPlaneRegistration: async authority => {
+      await authority.start();
+      return {
+        release: () => undefined,
+        checkpoint: (settlement: "settled" | "unsettled") => {
+          settlements.push(settlement);
+          if (refuseCheckpoint) throw new Error("runnerd did not report its provider identity");
+        },
+      };
+    },
+  });
+  let pid: number | null = null;
+  try {
+    const failure = await bundle.transport.request("thread/start", { cwd: root }).then(() => null, (error: unknown) => error);
+    expect(String(failure)).toContain("session.open");
+    pid = bundle.evidence().runnerPid;
+    expect(pid).not.toBeNull();
+    expect(bundle.transport.runnerExitConfirmed?.()).toBe(false);
+    // The runner cannot suspend without a session, so close kills it and waits
+    // for its exit; the checkpoint is "unsettled" and may be refused.
+    await bundle.transport.close().catch(() => undefined);
+    expect(settlements).toEqual(["unsettled"]);
+    expect(bundle.transport.runnerExitConfirmed?.()).toBe(true);
+    expect(processAbsent(pid!)).toBe(true);
+  } finally {
+    await bundle.transport.close().catch(() => undefined);
+    if (pid) { try { process.kill(-pid, "SIGKILL"); } catch { /* already stopped */ } }
+    await rm(root, { recursive: true, force: true });
+  }
+}, 30_000);
+
+it("gives no exit proof for a runner whose session opened", async () => {
+  const root = await mkdtemp(join(tmpdir(), "runner-startup-opened-"));
+  const bundle = createCapabilityRunnerdCodexTransport({
+    runnerBinary: defaultCapabilityRunnerdBinary(),
+    codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"),
+    codexArgs: ["--state-file", join(root, "provider.json")],
+    stateDirectory: root,
+    controlPlaneRegistration: async authority => {
+      await authority.start();
+      return { release: () => undefined };
+    },
+  });
+  let pid: number | null = null;
+  try {
+    await bundle.transport.request("thread/start", { cwd: root });
+    pid = bundle.evidence().runnerPid;
+    await bundle.transport.close().catch(() => undefined);
+    // Provider work could follow an opened session; this proof never covers it.
+    expect(bundle.transport.runnerExitConfirmed?.()).toBe(false);
+  } finally {
+    await bundle.transport.close().catch(() => undefined);
+    if (pid) { try { process.kill(-pid, "SIGKILL"); } catch { /* already stopped */ } }
+    await rm(root, { recursive: true, force: true });
+  }
+}, 30_000);

server/src/services/native-runtime/native-restart-recovery.ts

@@ -1571,7 +1571,12 @@ export async function claimNativeRestartRecoveries(input: {
         && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION
         && row.coordinator.phase === "retryable_failure" && row.run.status === "failed" && !row.coordinator.leaseOwner
         && settledFailure?.recoveryMode === "bootstrap_retry"
-        && settledFailure?.originalFailureCode === "native_runner_process_exited"
+        // The runner exited on its own, or it rejected session.open and the
+        // controller's close then proved it settled and exited. A failure
+        // saved without that proof (older releases) stays held.
+        && (settledFailure?.originalFailureCode === "native_runner_process_exited"
+          || (settledFailure?.originalFailureCode === "provider_session_open_rejected"
+            && settledFailure?.runnerExitConfirmed === true))
         && settledFailure?.providerSessionEstablished === false && settledFailure?.providerEventsExist === false
         && settledFailure?.checkpointExists === false && !settledFailure?.successorRunId
         && !row.run.sessionIdBefore && !row.run.sessionIdAfter

server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts

@@ -3157,6 +3157,28 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
       kind: "blocked", runId: fixture.runId, reason })]);
   });
 
+  it("bootstraps again after a remote session.open rejection whose runner exit the controller proved", async () => {
+    // Staging run 61d2e06e with the new proof: the runner answered session.open
+    // with a definite startup failure, then the controller's close settled it
+    // and observed its exit. No provider session, event or checkpoint exists.
+    const fixture = await seedExitedRemoteRunner("STARTUP-REJECTED-EXIT-PROVEN", {
+      originalFailureCode: "provider_session_open_rejected", runnerExitConfirmed: true }, 1);
+    expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
+      kind: "bootstrap_incomplete", runId: fixture.runId })]);
+    const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
+      .where(eq(nativeRunFinalizations.runId, fixture.runId));
+    expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_exited_before_provider_work" });
+  });
+
+  it("keeps an older release's remote session.open rejection held without the runner exit proof", async () => {
+    const fixture = await seedExitedRemoteRunner("STARTUP-REJECTED-NO-PROOF", {
+      originalFailureCode: "provider_session_open_rejected" }, 1);
+    expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
+      kind: "blocked", runId: fixture.runId, reason: "ambiguous_remote_recovery_evidence" })]);
+  });
+
   it("records that a held same-run retry was not a server restart", async () => {
     // Staging run 61d2e06e: a remote session.open rejection saved by an older
     // release, held when its due retry re-entered restart recovery. No server

server/src/services/native-runtime/native-session-executor.test.ts

@@ -446,6 +446,9 @@ import {
   NATIVE_PROVIDER_SESSION_CLOSED,
   NATIVE_PROVIDER_STARTUP_REJECTED,
   nativeRemoteStartupRejectionCode,
+  nativeRepeatedStartupFailureAction,
+  nativePreviousStartupFailureSignature,
+  nativeStartupFailureSignature,
   nativeSessionRecoveryProjection,
   nativeGovernedWaitResult,
   nativeConversationReplyResult,
@@ -14169,6 +14172,33 @@ describe("native session bounded recovery", () => {
         .toBe("provider_session_open_rejected");
       expect(nativeRemoteStartupRejectionCode("native_session_interrupted", failure, true)).toBe("native_session_interrupted");
     });
+
+    it("retries a remote startup rejection on the same run once the runner's exit is proven, but not a repeat", () => {
+      const confirmed = (fields: Record<string, unknown> = {}) =>
+        Object.defineProperty(sessionOpenFailure(startupFailure(fields)), "runnerExitConfirmed", { value: true, enumerable: false });
+      const first = confirmed();
+      expect(nativeRemoteStartupRejectionCode(nativeSessionFailureSourceCode(first), first, true))
+        .toBe("provider_session_open_rejected");
+      // Wrapped by a caller: the proof still reads through `cause`.
+      const wrapped = new Error("native session bootstrap failed", { cause: first });
+      expect(nativeRemoteStartupRejectionCode("provider_session_open_rejected", wrapped, true))
+        .toBe("provider_session_open_rejected");
+      // The same startup error as the previous attempt stops, with the message named.
+      const previous = nativePreviousStartupFailureSignature([
+        { disposition: "retryable_failure", startupFailureSignature: nativeStartupFailureSignature(startupFailure({})) },
+        { disposition: "bootstrap_incomplete", reason: "remote_runner_exited_before_provider_work" },
+      ]);
+      expect(previous).not.toBeNull();
+      expect(nativeRemoteStartupRejectionCode("provider_session_open_rejected", confirmed(), true, previous))
+        .toBe(NATIVE_PROVIDER_STARTUP_REJECTED);
+      expect(nativeRepeatedStartupFailureAction(startupFailure({}).message as string)).toContain("bwrap: loopback");
+      // A different startup error is not a repeat.
+      expect(nativeRemoteStartupRejectionCode("provider_session_open_rejected",
+        confirmed({ message: "Query closed before response received" }), true, previous)).toBe("provider_session_open_rejected");
+      // A deterministic rejection never retries, even with exit proof.
+      const deterministic = confirmed({ category: "authentication", retryability: "requires_action" });
+      expect(nativeSessionFailureSourceCode(deterministic)).toBe(NATIVE_PROVIDER_STARTUP_REJECTED);
+    });
   });
 
   it("stops same-run retries for an operator while a saved side-effecting call has no result", () => {

server/src/services/native-runtime/native-session-executor.ts

@@ -8517,14 +8517,62 @@ function deterministicProviderStartupFailure(error: unknown): boolean {
     && ["authentication", "permission", "invalid_request"].includes(failure.category);
 }
 
+/** The Runner transport proved the failed runner settled and its process exited. */
+export function nativeRunnerExitConfirmed(error: unknown, depth = 0): boolean {
+  if (!error || typeof error !== "object" || depth > 5) return false;
+  const value = error as Record<string, unknown>;
+  if (value.runnerExitConfirmed === true) return true;
+  return nativeRunnerExitConfirmed(value.cause, depth + 1)
+    || (Array.isArray(value.errors) && value.errors.slice(0, 3).some(item => nativeRunnerExitConfirmed(item, depth + 1)));
+}
+
+/** Same provider, category and message (numbers ignored): a retry would meet the same fault. */
+export function nativeStartupFailureSignature(failure: unknown): string | null {
+  const value = record(failure);
+  if (value.phase !== "startup" || typeof value.message !== "string" || !value.message.trim()) return null;
+  return JSON.stringify([value.provider ?? null, value.category ?? null, value.message.replace(/\d+/g, "#").trim()]);
+}
+
+/**
+ * On a remote target, a definite startup rejection retries on the same run
+ * only when the Runner proved its failed runner exited (so no second runner
+ * can own the sandbox) and the previous attempt did not fail with the same
+ * startup error. Otherwise it ends at once with Retry. A deterministic
+ * rejection never reaches here: it is already NATIVE_PROVIDER_STARTUP_REJECTED.
+ */
 export function nativeRemoteStartupRejectionCode<Code extends string>(
   sourceFailureCode: Code,
   error: unknown,
   remoteTarget: boolean,
+  previousStartupSignature?: string | null,
 ): Code | typeof NATIVE_PROVIDER_STARTUP_REJECTED {
-  return remoteTarget && sourceFailureCode === "provider_session_open_rejected"
-    && providerFailureFromError(error)?.phase === "startup"
-    ? NATIVE_PROVIDER_STARTUP_REJECTED : sourceFailureCode;
+  if (!remoteTarget || sourceFailureCode !== "provider_session_open_rejected") return sourceFailureCode;
+  const failure = providerFailureFromError(error);
+  if (failure?.phase !== "startup") return sourceFailureCode;
+  return nativeRunnerExitConfirmed(error) && !nativeRepeatedStartupFailure(error, previousStartupSignature)
+    ? sourceFailureCode : NATIVE_PROVIDER_STARTUP_REJECTED;
+}
+
+export function nativeRepeatedStartupFailure(error: unknown, previousStartupSignature: string | null | undefined): boolean {
+  const current = nativeStartupFailureSignature(providerFailureFromError(error));
+  return current !== null && current === previousStartupSignature;
+}
+
+/** The startup failure signature this run's previous failed attempt recorded. The
+ * execution claim clears the saved failure, but its history entry remains. */
+export function nativePreviousStartupFailureSignature(recoveryHistory: unknown): string | null {
+  if (!Array.isArray(recoveryHistory)) return null;
+  for (let index = recoveryHistory.length - 1; index >= 0; index -= 1) {
+    const entry = record(recoveryHistory[index]);
+    if (entry.disposition === "retryable_failure" || entry.disposition === "terminal_failure") {
+      return typeof entry.startupFailureSignature === "string" ? entry.startupFailureSignature : null;
+    }
+  }
+  return null;
+}
+
+export function nativeRepeatedStartupFailureAction(message: string): string {
+  return `Provider startup failed the same way on consecutive attempts, so automatic retries stopped: "${message.slice(0, 500)}". No provider work began. Fix the cause in the agent's environment if needed, then Retry. Retry starts a new run with a fresh provider session.`;
 }
 
 /** Reads the Runner's closed class from an authenticated turn.start failure. */
@@ -12330,12 +12378,27 @@ async function executePaperclipNativeSessionAttempt(
         !prepareNativeSessionOpenTimeoutRetry(input.execution, input.runnerExecutionTarget?.kind !== "remote")
         ? "native_session_interrupted" as const
         : startupClassifiedFailureCode;
+      // The previous attempt's saved failure survives the retry claim; a
+      // remote startup rejection compares against it to stop a repeat.
+      const remoteStartupRejection = classifiedFailureCode === "provider_session_open_rejected"
+        && input.runnerExecutionTarget?.kind === "remote";
+      const previousStartupSignature = remoteStartupRejection
+        ? await input.db.select({ recoveryHistory: nativeRunFinalizations.recoveryHistory }).from(nativeRunFinalizations)
+          .where(and(eq(nativeRunFinalizations.runId, input.execution.binding.runId),
+            eq(nativeRunFinalizations.companyId, input.execution.binding.companyId)))
+          .limit(1).then(rows => nativePreviousStartupFailureSignature(rows[0]?.recoveryHistory))
+          // Without the history no repeat is provable; the attempt budget still bounds retries.
+          .catch(() => null)
+        : null;
+      const repeatedRemoteStartupFailure = remoteStartupRejection
+        && nativeRepeatedStartupFailure(error, previousStartupSignature);
       const sourceFailureCode =
         classifiedFailureCode === "native_event_replay_conflict"
           ? classifiedFailureCode
           : providerUsageLimitObserved
             ? "native_provider_usage_limit"
-            : nativeRemoteStartupRejectionCode(classifiedFailureCode, error, input.runnerExecutionTarget?.kind === "remote");
+            : nativeRemoteStartupRejectionCode(classifiedFailureCode, error, input.runnerExecutionTarget?.kind === "remote",
+              previousStartupSignature);
       const bootstrap = input.useRunnerd && input.runnerExecutionTarget?.kind === "remote"
           && !input.restartRecovery && !retainedTransition && !remoteWarmRecovery
           ? {
@@ -12368,6 +12431,10 @@ async function executePaperclipNativeSessionAttempt(
           ? sourceFailureCode
           : disposition.failureCode;
       const providerFailure = providerFailureFromError(error);
+      const startupFailureSignature = recoveryEvidence.recoveryMode === "bootstrap_retry"
+        ? nativeStartupFailureSignature(providerFailure) : null;
+      const startupRejectedAction = repeatedRemoteStartupFailure && providerFailure
+        ? nativeRepeatedStartupFailureAction(providerFailure.message) : NATIVE_PROVIDER_STARTUP_REJECTED_ACTION;
       // Pre-provider startup retries back off and wait out a known provider
       // limit; their exhaustion explains the startup failure.
       const startupRetryExhausted =
@@ -12482,6 +12549,10 @@ async function executePaperclipNativeSessionAttempt(
                 recoveryEvidence.providerSessionEstablished,
               providerEventsExist: recoveryEvidence.providerEventsExist,
               checkpointExists: recoveryEvidence.checkpointExists,
+              // Restart recovery relaunches a remote runner for this retry only
+              // with this proof that the failed runner is gone.
+              ...(recoveryEvidence.recoveryMode === "bootstrap_retry" && nativeRunnerExitConfirmed(error)
+                ? { runnerExitConfirmed: true } : {}),
               ...(providerCooldownAt ? { providerCooldownAt: providerCooldownAt.toISOString() } : {}),
               ...(providerLimit?.plan.resetAt ? { providerLimitResetAt: providerLimit.plan.resetAt.toISOString() } : {}),
               recoveryOwner: recoveryProjection.recoveryOwner,
@@ -12495,7 +12566,7 @@ async function executePaperclipNativeSessionAttempt(
                   : sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED
                   ? NATIVE_PROVIDER_SESSION_CLOSED_ACTION
                   : sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED
-                  ? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION
+                  ? startupRejectedAction
                   : sourceFailureCode === "native_session_cleanup_quarantined"
                   ? nativeCleanupRefusalAction(completedTurnCleanup)
                   : sourceFailureCode === "runner_remote_provider_verification_unavailable"
@@ -12552,6 +12623,7 @@ async function executePaperclipNativeSessionAttempt(
                   providerSessionEstablished:
                     recoveryEvidence.providerSessionEstablished,
                   checkpointExists: recoveryEvidence.checkpointExists,
+                  ...(startupFailureSignature ? { startupFailureSignature } : {}),
                 })}::jsonb)
               ) with ordinality as history(item, ordinal)
               where ordinal > greatest(
@@ -12569,6 +12641,7 @@ async function executePaperclipNativeSessionAttempt(
                     providerSessionEstablished:
                       recoveryEvidence.providerSessionEstablished,
                     checkpointExists: recoveryEvidence.checkpointExists,
+                    ...(startupFailureSignature ? { startupFailureSignature } : {}),
                   })}::jsonb)
                 ) - 20,
                 0
@@ -12772,7 +12845,7 @@ async function executePaperclipNativeSessionAttempt(
               : sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED
               ? NATIVE_PROVIDER_SESSION_CLOSED_ACTION
               : sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED
-              ? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION
+              ? startupRejectedAction
               : sourceFailureCode === "native_session_cleanup_quarantined"
               ? nativeCleanupRefusalAction(completedTurnCleanup)
               : sourceFailureCode === "runner_task_admission_rejected"

server/src/services/native-runtime/native-startup-rejected-retry.integration.test.ts

@@ -46,7 +46,7 @@ describe("native session.open the provider rejected before any provider work", (
     message: "error creating thread: failed to load AGENTS.md instructions for environment `local`: fs sandbox helper failed with status exit status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted",
     ...fields });
 
-  async function rejectSessionOpen(failure: Error, attempt = 0) {
+  async function rejectSessionOpen(failure: Error, attempt = 0, options: { remote?: boolean; recoveryHistory?: Record<string, unknown>[] } = {}) {
     const issueId = randomUUID();
     await db.insert(issues).values({ id: issueId, companyId, title: "Plan the change", status: "in_progress", workMode: "standard",
       assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" });
@@ -58,7 +58,8 @@ describe("native session.open the provider rejected before any provider work", (
     await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
     const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
     const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
-    await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt });
+    await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt,
+      ...(options.recoveryHistory ? { recoveryHistory: options.recoveryHistory } : {}) });
     const execution: NativeExecutionInputV1 = {
       schema: "paperclip.native-execution-input.v1",
       binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
@@ -72,7 +73,9 @@ describe("native session.open the provider rejected before any provider work", (
     };
     await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
     provider.execute.mockReset().mockRejectedValue(failure);
-    await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: {
+    await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId,
+      ...(options.remote ? { runnerExecutionTarget: { kind: "remote", transport: "sandbox", providerKey: "test", remoteCwd: "/workspace",
+        runner: { execute: vi.fn() } } as never } : {}), backend: {
       descriptor: async () => { throw new Error("Unexpected provider call"); },
       openSession: async () => { throw new Error("Unexpected provider call"); },
     } }).catch(() => undefined);
@@ -118,4 +121,32 @@ describe("native session.open the provider rejected before any provider work", (
     const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
     await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true });
   }, 120_000);
+
+  const exitProven = (error: Error) => Object.defineProperty(error, "runnerExitConfirmed", { value: true, enumerable: false });
+
+  it("ends a remote startup rejection at once when the runner exit is not proven", async () => {
+    const { coordinator } = await rejectSessionOpen(rejection(startupFailure()), 0, { remote: true });
+    expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED });
+    expect(coordinator.failureDetail).not.toHaveProperty("runnerExitConfirmed");
+  }, 120_000);
+
+  it("retries a remote startup rejection on the same run when the runner exit is proven", async () => {
+    const { coordinator } = await rejectSessionOpen(exitProven(rejection(startupFailure())), 0, { remote: true });
+    expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "native_session_interrupted",
+      failureDetail: expect.objectContaining({ recoveryMode: "bootstrap_retry", originalFailureCode: "provider_session_open_rejected",
+        runnerExitConfirmed: true, providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false }) });
+  }, 120_000);
+
+  it("stops a remote startup rejection that repeats the previous attempt's error, naming it, with Retry", async () => {
+    // The first attempt's own failure write records the signature the second compares with.
+    const first = await rejectSessionOpen(exitProven(rejection(startupFailure())), 0, { remote: true });
+    const { issueId, coordinator } = await rejectSessionOpen(exitProven(rejection(startupFailure())), 1, { remote: true,
+      recoveryHistory: [...(first.coordinator.recoveryHistory as Record<string, unknown>[]),
+        { disposition: "bootstrap_incomplete", reason: "remote_runner_exited_before_provider_work" }] });
+    expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED, nextAttemptAt: null,
+      failureDetail: expect.objectContaining({ nextAction: expect.stringContaining("bwrap: loopback: Failed RTM_NEWADDR") }) });
+    expect(coordinator.failureDetail?.nextAction).toContain("Retry starts a new run");
+    const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
+    await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true });
+  }, 120_000);
 });