fix(native): re-bootstrap a Daytona runner restarted before session.open (#19583)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. The local control plane keeps the runner's controller state in an encrypted Postgres checkpoint. > - When the control plane restarts, restart recovery classifies each live native run and decides to reattach, resume, bootstrap again, or hold the task. > - A restart chaos wave killed the control plane twice while a Daytona runner was connecting. The run was claimed and the sandbox runner was launched, but `session.open` never completed. No provider evidence existed. > - Recovery waited correctly for the old 60 s controller lease to expire. Then the classifier had no rule for a claimed remote bootstrap. It returned `ambiguous_remote_recovery_evidence` and blocked the task as "Workspace repair required". > - The controller writes each command to the Postgres checkpoint before it sends it. After the lease expires, that checkpoint is the complete record of what reached the sandbox runner. > - This pull request lets recovery prove from that checkpoint that no provider work exists, and then start the same-run bootstrap again. > - The benefit is that a restart during Daytona runner startup continues automatically instead of stopping the task. ## Linked Issues or Issue Description **What happened?** A restart during Daytona remote runner startup (native ACPX Claude, two SIGKILLs about 20 s apart) left the task blocked. The run history showed `remote_controller_lease_live` twice, then `ambiguous_remote_recovery_evidence` with disposition `blocked`. The task showed "Workspace repair required" (`native_restart_recovery_blocked`) and did not continue. **Expected behavior** When the durable controller record proves that the runner received only bootstrap commands, recovery continues the run with a fresh bootstrap. Any evidence of provider work keeps the hold. **Steps to reproduce** 1. Start a native run on a Daytona environment. 2. Kill the control plane after the sandbox runner connects and before `session.open` completes. 3. Restart the control plane and wait for the old controller lease to expire. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: `server/src/services/native-runtime/native-restart-recovery.ts` `classifyNativeRunnerRecoveryEvidence`, remote branch. It admitted only an unclaimed bootstrap (`remote_execution_never_claimed`, #16349) or a run with a provider checkpoint. A claimed bootstrap with no provider evidence fell through to `ambiguous_remote_recovery_evidence`. The decrypted checkpoint of the failed run held `connectionCount: 1`, `run.prepare` delivered once and pending, `session.open` pending and never delivered, no committed events, and `ackedSourceSeq: 0`. Refs #16349, #19524 ## What Changed - `native-bootstrap-proof.ts`: move the incomplete-bootstrap predicates from the executor into a small pure module. Restart recovery and the executor now use the same proof. No behavior change. - `native-runner-checkpoint.ts`: add `nativeRunnerCheckpointProvesIncompleteBootstrap`. It reads only a checkpoint with the bootstrap parts (header, cursor, `command:1`, `command:2`), checks the run, runner instance, and session identity, and applies the shared proof. A remote runner journal is not on this host, so the proof uses the controller record alone. - `native-restart-recovery.ts`: for a classic sandbox run whose owner lease has expired, with a durable checkpoint, no session checkpoint, no provider evidence, and no staged attachment, claim `bootstrap_incomplete` with the reason `remote_bootstrap_proven_incomplete` when the checkpoint proof holds. At three provider attempts, block with `execution_recovery_budget_exhausted`. - `native-session-executor.ts`: add `restoreRemoteBootstrapCheckpointForRecovery`. For a `bootstrap_incomplete` recovery on a sandbox target, it restores the Postgres checkpoint over the local external-state marker. The existing incomplete-bootstrap proof then checks the restored state before it quarantines the root and bootstraps again. - `doc/run-log-events.md`: document the new disposition reason. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "remote bootstrap"`. Before the fix, the proven case and the budget case returned `blocked` / `ambiguous_remote_recovery_evidence`. - New cases: the proven claimed bootstrap gets `bootstrap_incomplete` (attempt kept, generation +1, run still `running`, no recovery action). The hold stays for a queued `turn.start`, an acknowledged event, adopted state, no checkpoint, and a spent budget. The executor helper restores the checkpoint only for a bootstrap recovery on a classic sandbox and only over a marker for the same run. - Regression set: `native-runner-restart-recovery.integration`, `native-session-executor`, `native-restart-recovery`, `native-bootstrap-retry.integration`, `heartbeat/recovery`: 1383 passed, 14 skipped. - Live Daytona replay of the failing chaos task (double SIGKILL during remote runner startup), 3 runs: REPLAY_RESULTS ## Risks - A fresh bootstrap acquires a new sandbox. The old sandbox keeps its lease until run settlement releases all run leases. The old runner gets no turn command, because only the controller issues one and the new controller never connects to it. - If `session.open` reached the old runner, the old provider session can exist without a turn. No provider work or tool call can happen in it. - Any checkpoint shape other than the bootstrap-only shape keeps the previous fail-closed hold. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 16:02:36 +01:00 · 2026-11-06 14:43:22 -06:00
1 parent dc1adb7209
commit d728be8fbf
@@ -347,6 +347,14 @@ For a prepared remote workspace interrupted before its first executor claim,
 that ledger records `remote_execution_never_claimed` with a
 `bootstrap_incomplete` disposition. The provider attempt remains zero during
 classification; the executor increments it before starting provider work.
+A claimed remote bootstrap (the sandbox runner was launched and may have
+connected) records `remote_bootstrap_proven_incomplete` with the same
+disposition. Recovery admits it only after the previous controller lease has
+expired and the Postgres controller checkpoint holds just `run.prepare` and an
+unfinished `session.open`, with no committed or acknowledged event. The
+executor restores that checkpoint and checks the same proof before it
+bootstraps again. After three provider attempts it records
+`execution_recovery_budget_exhausted` instead.

 For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed`
 event retains `error.code: "provider_turn_lost_on_restore"` and
@@ -0,0 +1,68 @@
+/**
+ * Pure proofs over durable runner state that a bootstrap never reached
+ * provider work. Restart recovery and the executor share them, so the claim
+ * and the executor's own check accept exactly the same shapes.
+ */
+
+export const RUNNERD_CONTROL_PLANE_STATE_SCHEMAS: ReadonlySet<string> = new Set([
+  "paperclip.runner.durable.control-plane-state.v1",
+  "paperclip.runner.durable.control-plane-state.v2",
+]);
+
+const RUNNERD_BOOTSTRAP_COMMAND_TYPES = new Set(["run.prepare", "session.open"]);
+
+function record(value: unknown): Record<string, unknown> {
+  return value && typeof value === "object" && !Array.isArray(value)
+    ? (value as Record<string, unknown>)
+    : {};
+}
+
+/** The runner never connected, so every bootstrap command is still undelivered. */
+export function runnerdStateIsUnconnectedBootstrap(state: Record<string, unknown>): boolean {
+  const commands = Array.isArray(state.commands) ? state.commands.map(record) : [];
+  const committedEvents = Array.isArray(state.committedEvents) ? state.committedEvents : [];
+  return RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema)) &&
+    state.connectionCount === 0 && committedEvents.length === 0 &&
+    commands.every(command => command.status === "pending" &&
+      RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)));
+}
+
+/**
+ * A provider turn can start only after session.open completes and the
+ * controller issues a turn command. A root whose controller journal stops
+ * before that point (run.prepare may be acknowledged, session.open never is,
+ * no event was committed or acknowledged) and whose runner journal emitted no
+ * event either holds no provider work, so restart recovery may quarantine it
+ * and bootstrap again. Any other shape stays fail-closed.
+ *
+ * `runnerJournal` is `undefined` when the runner journal is not on this host
+ * (never written locally, or a remote runner), `null` when it is unreadable.
+ */
+export function runnerdStateIsIncompleteBootstrap(
+  state: Record<string, unknown>,
+  runnerJournal?: Record<string, unknown> | null,
+): boolean {
+  if (runnerdStateIsUnconnectedBootstrap(state)) return true;
+  if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema))) return false;
+  if (typeof state.connectionCount !== "number" || !Number.isSafeInteger(state.connectionCount) ||
+      state.connectionCount < 0) return false;
+  if (!Array.isArray(state.committedEvents) || state.committedEvents.length !== 0) return false;
+  if (state.ackedSourceSeq !== 0) return false;
+  if (state.warmTransition != null || state.completedWarmTransition != null ||
+      state.stagedRunAttachment != null) return false;
+  if (record(state.bootstrapHistory).adopted === true) return false;
+  if (!Array.isArray(state.commands)) return false;
+  const commands = state.commands.map(record);
+  if (!commands.every(command => RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)) &&
+      (command.status === "pending" ||
+        (command.type === "run.prepare" && command.status === "completed")))) return false;
+  // The controller cannot see runner events that were emitted but never
+  // committed. When the runner journal exists it must agree that none were.
+  if (runnerJournal === undefined) return true;
+  if (runnerJournal === null) return false;
+  const processed = Object.values(record(runnerJournal.processedCommands)).map(record);
+  return Array.isArray(runnerJournal.outbox) && runnerJournal.outbox.length === 0 &&
+    runnerJournal.nextSourceSeq === 1 && runnerJournal.ackedSourceSeq === 0 &&
+    processed.every(command => RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.commandType)) &&
+      (command.commandType === "run.prepare" || command.status !== "completed"));
+}
@@ -11,7 +11,8 @@ import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-pre
 import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
 import { notifyExecutionRunWork } from "../execution-control-notifications.js";
 import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js";
-import { hasNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader } from "./native-runner-checkpoint.js";
+import { hasNativeRunnerCheckpoint, loadNativeRunnerCheckpointHeader,
+  nativeRunnerCheckpointProvesIncompleteBootstrap } from "./native-runner-checkpoint.js";
 import { resolveNativeStagedAttachmentRecovery, type NativeStagedAttachmentRecovery } from "./native-staged-attachment-recovery.js";
 import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
 import { activeNativeExecutionRunIds, NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js";
@@ -340,6 +341,8 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
  runnerGroupAlive: boolean;
  remoteSandbox?: boolean;
  unclaimedBootstrap?: boolean;
+  /** The durable controller checkpoint of a claimed remote bootstrap proves no provider work. */
+  remoteBootstrapProvenIncomplete?: boolean;
  processStartMatches: boolean;
  knownProviderProcessAlive?: boolean;
  knownProviderProcessIdentityAmbiguous?: boolean;
@@ -368,6 +371,12 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
    if (input.unclaimedBootstrap === true && !input.hasCheckpoint && !input.hasProviderEvidence) {
      return { claimKind: "bootstrap_incomplete", reason: "remote_execution_never_claimed" };
    }
+    if (input.remoteBootstrapProvenIncomplete === true && !input.hasCheckpoint && !input.hasProviderEvidence) {
+      // Each same-run bootstrap is another provider attempt; keep restarts bounded.
+      if ((input.providerAttempt ?? 0) >= 3)
+        return { claimKind: null, reason: "execution_recovery_budget_exhausted" };
+      return { claimKind: "bootstrap_incomplete", reason: "remote_bootstrap_proven_incomplete" };
+    }
    return input.hasCheckpoint &&
      input.checkpointIdentityMatches === true &&
      input.hasProviderEvidence
@@ -1017,9 +1026,23 @@ export async function claimNativeRestartRecoveries(input: {
        unclaimedBootstrap = !executionEvent;
      }

+      // A claimed remote bootstrap (runner launched, perhaps connected) has no
+      // provider evidence when its controller dies before session.open
+      // completes. The owner's lease has expired above, and the controller
+      // writes every command to Postgres before sending it, so that checkpoint
+      // is the complete record of what reached the sandbox runner. When it
+      // proves no provider work, bootstrap again instead of holding the task.
+      const remoteBootstrapProvenIncomplete = !unclaimedBootstrap && remoteWorkspace !== null && !pluginTask
+        && Boolean(durableAuthority) && stagedAttachment === null && !hasCheckpoint && !hasProviderEvidence
+        && Boolean(row.run.runnerInstanceId && row.run.nativeSessionId)
+        && await nativeRunnerCheckpointProvesIncompleteBootstrap(tx, {
+          companyId: row.run.companyId, issueId: row.coordinator.issueId, runId: row.run.id,
+        }, { runnerInstanceId: row.run.runnerInstanceId!, normalizedSessionId: row.run.nativeSessionId! });
+
      const classification = classifyNativeRunnerRecoveryEvidence({
        remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority),
        unclaimedBootstrap,
+        remoteBootstrapProvenIncomplete,
        runnerPidAlive,
        runnerGroupAlive,
        processStartMatches: exactRunnerIdentity,
@@ -5,6 +5,7 @@ import { inspectDurablePrpCheckpointHeader, inspectUnclaimedBootstrapCheckpoint,
 import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provider.js";
 import { legacyControllerBootId } from "../legacy-controller-lease.js";
 import { assertNativeSourceCheckpointConsumer, lockSourceRecoveryCoordinators } from "./native-source-recovery-ownership.js";
+import { runnerdStateIsIncompleteBootstrap } from "./native-bootstrap-proof.js";

 type Binding = { companyId: string; issueId: string; runId: string };
 export const NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES = 1024 * 1024;
@@ -248,6 +249,53 @@ export async function loadNativeUnclaimedBootstrapCheckpoint(
  } catch { return null; }
 }

+const BOOTSTRAP_CHECKPOINT_PARTS = new Set(["header", "cursor", "command:1", "command:2"]);
+
+/**
+ * A remote runner's controller state lives only in this checkpoint, and every
+ * command is written here before the controller sends it. Once the previous
+ * controller's lease has expired, the checkpoint is therefore a complete record
+ * of what reached the sandbox runner. Returns true only when that record holds
+ * the bootstrap commands alone (run.prepare, an unfinished session.open) with
+ * no committed or acknowledged event, so no provider turn can exist. The
+ * sandbox runner journal is not on this host; the proof uses the controller
+ * record alone, as for a local root whose runner never journaled.
+ */
+export async function nativeRunnerCheckpointProvesIncompleteBootstrap(
+  db: Pick<Db, "select">, binding: Binding, expected: { runnerInstanceId: string; normalizedSessionId: string },
+): Promise<boolean> {
+  try {
+    const parts = await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints).where(and(
+      eq(nativeRunnerCheckpoints.companyId, binding.companyId), eq(nativeRunnerCheckpoints.issueId, binding.issueId),
+      eq(nativeRunnerCheckpoints.runId, binding.runId))).limit(BOOTSTRAP_CHECKPOINT_PARTS.size + 1);
+    if (!parts.length || parts.length > BOOTSTRAP_CHECKPOINT_PARTS.size
+      || parts.some(row => !BOOTSTRAP_CHECKPOINT_PARTS.has(row.part))) return false;
+    const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
+    if (!checkpoint || checkpoint.predecessor !== null
+      || checkpoint.recoveryEvidence.runnerInstanceId !== expected.runnerInstanceId
+      || checkpoint.recoveryEvidence.normalizedSessionId !== expected.normalizedSessionId) return false;
+    // The decoder above validated these canonical v2 records. Rebuild the
+    // controller state in memory; the part check admits no event records.
+    const read = (part: string) => {
+      const value = checkpoint.records.get(part);
+      return value === undefined ? undefined : JSON.parse(value) as Record<string, unknown>;
+    };
+    const header = read("header");
+    const cursor = read("cursor");
+    if (header?.schema !== "paperclip.runner.controller-checkpoint.v2" || !cursor) return false;
+    const state: Record<string, unknown> = {
+      ...(header.state as Record<string, unknown>),
+      ackedSourceSeq: cursor.ackedSourceSeq,
+      commands: ["command:1", "command:2"].map(read).filter(command => command !== undefined),
+      committedEvents: [],
+    };
+    return (state.identity as { runId?: unknown } | undefined)?.runId === binding.runId
+      && runnerdStateIsIncompleteBootstrap(state);
+  } catch {
+    return false;
+  }
+}
+
 export async function restoreNativeRunnerCheckpoint(db: Db, binding: Binding, root: string, normalizedSessionId?: string) {
  const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
  if (!checkpoint) return null;
@@ -60,7 +60,10 @@ import {
  detachNativeSessionsForRestart,
  executePaperclipNativeSession,
  NativeControllerDetachedForRestartError,
+  restoreRemoteBootstrapCheckpointForRecovery,
+  runnerdStateProvesIncompleteBootstrap,
 } from "./native-session-executor.js";
+import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
 import {
  listExpiredNativeRunnerAuthorities,
  loadNativeRunnerCheckpoint,
@@ -2220,6 +2223,140 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
    },
  );

+  // RC-29: a restart while a Daytona runner was connecting. The executor had
+  // claimed the run and launched the sandbox runner, which authenticated, but
+  // session.open never completed. The controller's Postgres checkpoint is the
+  // only record of what the runner received.
+  async function seedClaimedRemoteBootstrap(label: string, options: {
+    attempt?: number;
+    mutate?: (core: DurablePrpControlPlane) => void;
+    checkpoint?: boolean;
+  } = {}) {
+    const fixture = await seedRun(`REMOTE-CLAIMED-${label}`);
+    const { runId, issueId, native } = fixture;
+    const remote = remotePreparedWorkspace();
+    await fixture.db.update(heartbeatRuns).set({
+      runnerProfileJson: { ...fixture.preparedProfile, nativeWorkspaceSync: remote },
+      processPid: 443, processStartedAt: new Date(Date.now() - 30_000),
+    }).where(eq(heartbeatRuns.id, runId));
+    await fixture.db.insert(heartbeatRunEvents).values([
+      { companyId, runId, agentId, seq: 900, stream: "system", level: "info", eventType: "native.process_start_requested" },
+      { companyId, runId, agentId, seq: 901, stream: "system", level: "info", eventType: "native.process_identity_recorded",
+        payload: { targetKind: "remote", processPid: 443 } },
+    ]);
+    const bootId = randomUUID();
+    const leaseOwner = `${bootId}:1:${randomUUID()}`;
+    await fixture.db.update(nativeRunFinalizations).set({
+      attempt: options.attempt ?? 1, controllerGeneration: 1, controllerBootId: bootId, controllerPid: 2_000_000_001,
+      controllerProcessStartedAt: new Date(Date.now() - 60_000), leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000),
+    }).where(eq(nativeRunFinalizations.runId, runId));
+    if (options.checkpoint !== false) {
+      const identity = { runnerInstanceId: native.runnerInstanceId, environmentLeaseId: runId, runId,
+        normalizedSessionId: native.normalizedSessionId, turnId: `turn-${runId}`, itemId: `item-${runId}` };
+      const core = new DurablePrpControlPlane({ stateDirectory: resolve(runtimeRoot, runId, "control-plane"), identity,
+        expectedRunnerVersion: "0.3.0", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
+      try {
+        const template = { provider: { provider: "acpx", command: "fixture" }, authorizedTools: {} };
+        core.persistRunAttachTemplate(template);
+        core.queueCommand("run.prepare", template);
+        core.queueCommand("session.open", { reuse: "same_session" });
+        core.issueBootstrapTicket();
+        // The runner authenticated once and received run.prepare.
+        core.store.state.connectionCount = 1;
+        core.store.state.freshBootstraps = 1;
+        core.store.state.commandDeliveryCounts = { [core.store.state.commands[0]!.commandId]: 1 };
+        options.mutate?.(core);
+        core.setCheckpointPersistence(nativeRunnerCheckpointWriter(fixture.db, {
+          companyId, issueId, runId, authorityRunId: runId, bootId, leaseOwner, generation: 1,
+        }));
+        await core.flushCheckpoint();
+      } finally {
+        await core.stop();
+      }
+    }
+    // The killed controller can no longer renew its lease.
+    await fixture.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) })
+      .where(eq(nativeRunFinalizations.runId, runId));
+    return fixture;
+  }
+
+  it("restarts a claimed remote bootstrap whose durable checkpoint proves no provider work", async () => {
+    const fixture = await seedClaimedRemoteBootstrap("PROVEN");
+    const dispositions = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] });
+    expect(dispositions).toEqual([expect.objectContaining({ kind: "bootstrap_incomplete", runId: fixture.runId,
+      providerAttempt: 1, controllerGeneration: 2 })]);
+    const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
+      .where(eq(nativeRunFinalizations.runId, fixture.runId));
+    expect(coordinator).toMatchObject({ phase: "observed", recoveryState: "bootstrap_incomplete", failureCode: null });
+    expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ disposition: "bootstrap_incomplete",
+      reason: "remote_bootstrap_proven_incomplete" });
+    const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
+    expect(run).toMatchObject({ status: "running", errorCode: null, processPid: null });
+    expect(await fixture.db.select().from(issueRecoveryActions)
+      .where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId))).toHaveLength(0);
+  });
+
+  it("lets the executor prove the remote bootstrap from its Postgres checkpoint, not the local marker", async () => {
+    const fixture = await seedClaimedRemoteBootstrap("EXECUTOR");
+    const execution = buildNativeExecutionInput({
+      companyId, agentId, runId: fixture.runId,
+      issue: { id: fixture.issueId, identifier: null, title: "Restart bootstrap", description: null, workMode: "standard" },
+      taskPrompt: "Restart bootstrap", provider: "codex",
+      workspace: { id: randomUUID(), cwd: runtimeRoot, repoUrl: null, repoRef: null, branchName: null },
+      normalizedSessionId: fixture.native.normalizedSessionId,
+      runtimeContext: nativeRuntimeContextFixture(),
+      completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1",
+        contract: { revision: "1", objective: "Restart bootstrap", criteria: [{ id: "result", requirement: "Complete work" }] } },
+    });
+    const root = resolve(runtimeRoot, `marker-${fixture.runId}`);
+    await mkdir(resolve(root, "control-plane"), { recursive: true, mode: 0o700 });
+    const writeMarker = (runId: string) => writeFile(resolve(root, "control-plane", "control-plane-state.json"), JSON.stringify({
+      schema: "paperclip.runner.external-control-plane-state.v1",
+      identity: { runnerInstanceId: fixture.native.runnerInstanceId, environmentLeaseId: fixture.runId, runId,
+        normalizedSessionId: fixture.native.normalizedSessionId, turnId: `turn-${runId}`, itemId: `item-${runId}` },
+    }));
+    const claim = { kind: "bootstrap_incomplete" as const, runId: fixture.runId, leaseOwner: "successor",
+      controllerGeneration: 2, providerAttempt: 1, restartKind: "hard" as const, recoveryRequestId: null };
+    const target = { kind: "remote", transport: "sandbox", remoteCwd: "/workspace" } as AdapterExecutionTarget;
+    try {
+      await writeMarker(fixture.runId);
+      expect(runnerdStateProvesIncompleteBootstrap(root)).toBe(false);
+      // Only a bootstrap recovery on a classic sandbox restores the checkpoint.
+      for (const other of [{ restartRecovery: undefined, runnerExecutionTarget: target },
+        { restartRecovery: claim, runnerExecutionTarget: { ...target, runnerTask: vi.fn() } as AdapterExecutionTarget }]) {
+        await restoreRemoteBootstrapCheckpointForRecovery({ db: fixture.db, execution, root, ...other });
+        expect(runnerdStateProvesIncompleteBootstrap(root)).toBe(false);
+      }
+      // A marker for another run never receives this run's checkpoint.
+      await writeMarker(randomUUID());
+      await restoreRemoteBootstrapCheckpointForRecovery({ db: fixture.db, execution, root,
+        restartRecovery: claim, runnerExecutionTarget: target });
+      expect(runnerdStateProvesIncompleteBootstrap(root)).toBe(false);
+      await writeMarker(fixture.runId);
+      await restoreRemoteBootstrapCheckpointForRecovery({ db: fixture.db, execution, root,
+        restartRecovery: claim, runnerExecutionTarget: target });
+      expect(runnerdStateProvesIncompleteBootstrap(root)).toBe(true);
+    } finally {
+      await rm(root, { recursive: true, force: true });
+    }
+  });
+
+  it.each<[string, Parameters<typeof seedClaimedRemoteBootstrap>[1], string]>([
+    ["a turn was issued", { mutate: core => { core.queueCommand("turn.start", { prompt: "Prior work must stay held." }); } },
+      "ambiguous_remote_recovery_evidence"],
+    ["an event was acknowledged", { mutate: core => { core.store.state.ackedSourceSeq = 1; } },
+      "ambiguous_remote_recovery_evidence"],
+    ["the state was adopted", { mutate: core => { core.store.state.bootstrapHistory!.adopted = true; } },
+      "ambiguous_remote_recovery_evidence"],
+    ["no checkpoint exists", { checkpoint: false }, "ambiguous_remote_recovery_evidence"],
+    ["the bootstrap budget is spent", { attempt: 3 }, "execution_recovery_budget_exhausted"],
+  ])("keeps a claimed remote bootstrap held when %s", async (label, options, reason) => {
+    const fixture = await seedClaimedRemoteBootstrap(label.replace(/\W+/g, "-"), options);
+    expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
+      restartKind: "hard", runIds: [fixture.runId] })).toEqual([{ kind: "blocked", runId: fixture.runId, reason }]);
+  });
+
  it.each([
    { attempt: 2, driverKind: "codex_app_server" },
    { attempt: 3, driverKind: "codex_app_server" },
@@ -85,6 +85,8 @@ import { createNativeToolTrace, type NativeToolTrace } from "./native-tool-trace
 import { createNativeProviderFailureObservation } from "./native-provider-failure.js";
 import { nativeStartupRetryExhaustedAction, nativeStartupRetryWait, planNativeStartupRetry } from "./native-startup-retry.js";
 import { admitUnclaimedNativeBootstrapRetry } from "./native-bootstrap-retry.js";
+import { RUNNERD_CONTROL_PLANE_STATE_SCHEMAS, runnerdStateIsIncompleteBootstrap,
+  runnerdStateIsUnconnectedBootstrap } from "./native-bootstrap-proof.js";
 import { createNativeDeliverableTextObservation } from "./native-deliverable-text.js";
 import { createNativeGitHubAccess, type NativeGitHubAccess } from "./native-github-access.js";
 import { resolveGitHubOperationCredentials } from "../github-operation-credentials.js";
@@ -522,10 +524,6 @@ const ACPX_RUNTIME_CHECKPOINT_EXCLUDED_ENTRIES = [
  "home/.npm",
  "data/pnpm/store",
 ] as const;
-const RUNNERD_CONTROL_PLANE_STATE_SCHEMAS = new Set([
-  "paperclip.runner.durable.control-plane-state.v1",
-  "paperclip.runner.durable.control-plane-state.v2",
-]);
 const RUNNERD_STATE_SCHEMA = "paperclip.runner.durable.state.v1";
 const CODEX_PROVIDER_STATE_SCHEMA = "paperclip.runner.codex-provider-state.v1";
 const ACPX_PROVIDER_STATE_SCHEMA = "paperclip.runner.acpx-provider-state.v3";
@@ -5486,6 +5484,7 @@ async function migrateRunnerdStateRootForExecution(input: {
  if (input.restartRecovery?.kind === "reattach_remote_runner" && !existsSync(scoped)) {
    throw new Error("runner_state_identity_mismatch: remote_reattach_root_missing");
  }
+  await restoreRemoteBootstrapCheckpointForRecovery({ ...input, root: scoped });
  if (existsSync(scoped)) {
    if (!isSafeNativeStateDirectory(scoped)) {
      throw new Error("runner_state_directory_unsafe");
@@ -5952,55 +5951,6 @@ function localProcessDefinitelyGone(value: unknown, group = false): boolean {
  }
 }

-const RUNNERD_BOOTSTRAP_COMMAND_TYPES = new Set(["run.prepare", "session.open"]);
-
-/** The runner never connected, so every bootstrap command is still undelivered. */
-function runnerdStateIsUnconnectedBootstrap(state: Record<string, unknown>): boolean {
-  const commands = Array.isArray(state.commands) ? state.commands.map(record) : [];
-  const committedEvents = Array.isArray(state.committedEvents) ? state.committedEvents : [];
-  return RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema)) &&
-    state.connectionCount === 0 && committedEvents.length === 0 &&
-    commands.every(command => command.status === "pending" &&
-      RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)));
-}
-
-/**
- * A provider turn can start only after session.open completes and the
- * controller issues a turn command. A root whose controller journal stops
- * before that point (run.prepare may be acknowledged, session.open never is,
- * no event was committed or acknowledged) and whose runner journal emitted no
- * event either holds no provider work, so restart recovery may quarantine it
- * and bootstrap again. Any other shape stays fail-closed.
- */
-function runnerdStateIsIncompleteBootstrap(
-  state: Record<string, unknown>,
-  runnerJournal?: Record<string, unknown> | null,
-): boolean {
-  if (runnerdStateIsUnconnectedBootstrap(state)) return true;
-  if (!RUNNERD_CONTROL_PLANE_STATE_SCHEMAS.has(String(state.schema))) return false;
-  if (typeof state.connectionCount !== "number" || !Number.isSafeInteger(state.connectionCount) ||
-      state.connectionCount < 0) return false;
-  if (!Array.isArray(state.committedEvents) || state.committedEvents.length !== 0) return false;
-  if (state.ackedSourceSeq !== 0) return false;
-  if (state.warmTransition != null || state.completedWarmTransition != null ||
-      state.stagedRunAttachment != null) return false;
-  if (record(state.bootstrapHistory).adopted === true) return false;
-  if (!Array.isArray(state.commands)) return false;
-  const commands = state.commands.map(record);
-  if (!commands.every(command => RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.type)) &&
-      (command.status === "pending" ||
-        (command.type === "run.prepare" && command.status === "completed")))) return false;
-  // The controller cannot see runner events that were emitted but never
-  // committed. When the runner journal exists it must agree that none were.
-  if (runnerJournal === undefined) return true;
-  if (runnerJournal === null) return false;
-  const processed = Object.values(record(runnerJournal.processedCommands)).map(record);
-  return Array.isArray(runnerJournal.outbox) && runnerJournal.outbox.length === 0 &&
-    runnerJournal.nextSourceSeq === 1 && runnerJournal.ackedSourceSeq === 0 &&
-    processed.every(command => RUNNERD_BOOTSTRAP_COMMAND_TYPES.has(String(command.commandType)) &&
-      (command.commandType === "run.prepare" || command.status !== "completed"));
-}
-
 /** `undefined` when the runner never journaled, `null` when unreadable. */
 function readRunnerdBootstrapJournal(root: string): Record<string, unknown> | null | undefined {
  const runnerRoot = resolve(root, "runner");
@@ -6029,6 +5979,26 @@ function readRunnerdControlPlaneState(root: string): Record<string, unknown> {
  );
 }

+/**
+ * A remote runner's controller state lives in Postgres; its local root holds
+ * only a fail-closed marker. Before a `bootstrap_incomplete` recovery checks
+ * that root, restore the exact checkpoint so the same incomplete-bootstrap
+ * proof applies. A missing checkpoint leaves the marker, which stays fail-closed.
+ */
+export async function restoreRemoteBootstrapCheckpointForRecovery(input: {
+  db: Db; execution: NativeExecutionInput; root: string;
+  restartRecovery?: NativeRestartRecoveryClaim;
+  runnerExecutionTarget?: AdapterExecutionTarget | null;
+}): Promise<void> {
+  const target = input.runnerExecutionTarget;
+  if (input.restartRecovery?.kind !== "bootstrap_incomplete" || target?.kind !== "remote"
+    || target.transport !== "sandbox" || target.runnerTask || !existsSync(input.root)
+    || readRunnerdDurableIdentity(input.root) !== null) return;
+  const marker = readRunnerdDurableIdentity(input.root, true);
+  if (!marker || marker.runId !== input.execution.binding.runId) return;
+  await restoreNativeRunnerCheckpoint(input.db, input.execution.binding, input.root, nativeSessionKey(input.execution));
+}
+
 export function runnerdStateProvesIncompleteBootstrap(root: string): boolean {
  try {
    return runnerdStateIsIncompleteBootstrap(readRunnerdControlPlaneState(root), readRunnerdBootstrapJournal(root));