fix(native): read the scheduled-resume evidence the resume reconciler claimed (#20656)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20645 lets a local same-run scheduled resume send the one
same-conversation continuation for a turn lost on restore. It applies
only when durable records prove three facts: the previous runner
stopped, the attempt was admitted within the budget, and it is the same
checkpointed conversation.
> - It read the third proof from the coordinator's saved
`failure_detail` while the phase was `retryable_failure`.
> - In production, the resume reconciler
(`claimNativeSessionResumptions`) claims the run first. It moves the
coordinator to `observed` and clears `failure_detail`. So the executor
never saw that evidence, and #20645 refused every real scheduled resume.
> - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the
exact path (`runner.drain` timeout, scheduled resume,
`provider_turn_lost_on_restore`), sent no continuation, and needed
Retry.
> - This pull request makes the reconciler record that evidence durably
when it claims the run, and makes the executor read it from there.
> - The benefit is that #20645 works on the real path. The proof rule is
unchanged.

## Linked Issues or Issue Description

Refs #20645

**What happened?**
Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus
#20645), task codex B5 TM2:
- `native_session_recovery_failed: runner_local_connect_failed: PRP
command runner.drain timed out`;
- `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`,
`recoverable`, `providerSessionEstablished` and `checkpointExists` all
set;
- `native.local_process_stopped`;
- attempt 2 started;
- `turn.failed` with `provider_turn_lost_on_restore` for the same turn
ID that the checkpoint was running.

No continuation was sent. The run failed with
`native_provider_terminal_failed` and was preserved with
`uncertain_provider_action`.

**Expected behavior**
The proven scheduled resume sends the single continuation.

**Steps to reproduce**
The new integration test below. The live path is timing-dependent (the
L7 command in #20645).

## What Changed

- `native-finalization-reconciler.ts`: when it claims a same-run resume,
the reconciler appends one `same_run_resume` entry to
`recovery_history`. The append is bounded like other history writes, and
it runs in the same update that clears the failure. The entry copies
`recoveryMode`, `recoverable`, `providerSessionEstablished`,
`checkpointExists`, `successorRunId` and `replacementDenied` from the
saved failure, with the controller boot and provider attempt.
- `scheduled-resume-continuation.ts`: the evidence for the third proof
now comes from either of two durable records:
- the saved failure, when the executor claims a `retryable_failure`
coordinator directly (unchanged); or
- a reconciler claim: an `observed` coordinator whose last history entry
is a `same_run_resume` entry naming the executor's own controller boot.
The two other proofs are unchanged: the stop receipt, and an admitted
attempt from 1 to 3.
- `native-session-executor.ts`: passes the coordinator's last history
entry and the controller boot to the decision.
- `doc/execution-semantics.md`: describes the recorded evidence.

Readers of `recovery_history` that look at the last entry do so only for
restart claims, Stop recovery and shutdown suspension. Restart
recovery's previous-claim lookup skips non-claim dispositions, so a
`same_run_resume` entry does not change those paths.

**Still refused** (tested):
- every #20645 case;
- the last history entry is not the resume claim, or there is no
history;
- the resume claim was recorded by another boot;
- the claimed evidence is a bootstrap retry, has no checkpoint, or
records a replacement;
- a reconciler-claimed resume without a stop receipt.

## Verification

- `native-scheduled-resume-continuation.integration.test.ts` now also
runs through the real reconciler claim (`claimNativeSessionResumptions`)
before `executePaperclipNativeSession`. On master the reconciler-claimed
allowed case fails (`resumeInterruptedTurn` is false). With this change
it is true, and the claim admits attempt 2. A reconciler-claimed resume
without a stop receipt still passes false.
- `scheduled-resume-continuation.test.ts`: decision cases for the
reconciler-claim entry.
- `native-session-resumption.test.ts` passes. 54 tests pass across the
three files.
- `cd server && npx tsc --noEmit` passes.
- Live: not yet. The failing wave above is the reproduction. A rerun is
timing-dependent.

## Risks

- One more `recovery_history` entry per scheduled resume, inside the
existing 20-entry bound.
- The proof rule itself is unchanged. This only reads the same evidence
from the record where it now survives.
- No schema change. A run claimed by an earlier build has no
`same_run_resume` entry, so it keeps the existing failure: refused, with
Retry offered as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

doc/execution-semantics.md

@@ -3349,9 +3349,14 @@ that process is gone. Second, the resume claimed a provider attempt within the
 three-attempt budget from a `retryable_failure` coordinator. Third, that failure
 saved `recoveryMode: "exact_checkpoint_resume"`, `recoverable`,
 `providerSessionEstablished` and `checkpointExists`, with no replacement recorded,
-and the runner restores that same checkpointed conversation. A remote run, a
-bootstrap retry, an ambiguous state, a missing or superseded stop record, an
-exhausted budget, or a replacement session keeps the existing failure.
+and the runner restores that same checkpointed conversation. The resume reconciler
+clears the saved failure when it claims the run. So it first appends a
+`same_run_resume` entry to `recovery_history` that copies those fields, together
+with its controller boot. The executor then reads that entry. It must be the last
+entry, and it must name the executor's boot. A remote run, a bootstrap retry, an
+ambiguous state, a missing or superseded stop record, an exhausted budget, a
+replacement session, or a claim entry from another boot or followed by another
+entry keeps the existing failure.
 
 A restart can also land after the agent proposed its result but before the
 provider turn ended. When the restored runner proves that turn ended with the

server/src/services/native-runtime/native-exact-resume-failure.test.ts

@@ -50,7 +50,18 @@ describe("interruptedExactResumeCause", () => {
     });
   });
 
+  it("finds the interruption behind the resume reconciler's claim entry for the same attempt", () => {
+    const resumeClaim = { at: "2099-01-01T00:01:00.000Z", disposition: "same_run_resume", controllerBootId: "boot-1",
+      providerAttempt: 1, evidence: { recoveryMode: "exact_checkpoint_resume" } };
+    expect(interruptedExactResumeCause({ ...claimed, recoveryHistory: [interruption, resumeClaim] })).toEqual({
+      originalFailureCode: "native_session_interrupted",
+      originalMessage: "PRP command turn.start failed: classification=session_resume_required",
+      interruptedAttempt: 1,
+    });
+  });
+
   it.each([
+    ["a resume claim entry for a later provider attempt", { disposition: "same_run_resume", providerAttempt: 2 }],
     ["a wait for a later provider attempt", { disposition: "blocked", reason: "held", providerAttempt: 2 }],
     ["a later recovery entry that is not a wait", { disposition: "resume_dead_runner", reason: "restart", providerAttempt: 1 }],
   ])("does not reach past %s", (_name, later) => {

server/src/services/native-runtime/native-exact-resume-failure.ts

@@ -16,7 +16,7 @@ const FAILURE_DETAIL_MAX_CHARS = 2_000;
 const REFUSAL_MAX_CHARS = 480;
 const REFUSAL_SEPARATOR = "; exact checkpoint resume refused: ";
 /** Restart-recovery waits that keep the interrupted provider attempt. */
-const RECOVERY_WAIT_DISPOSITIONS: ReadonlySet<unknown> = new Set(["blocked", "awaiting_evidence"]);
+const RECOVERY_WAIT_DISPOSITIONS: ReadonlySet<unknown> = new Set(["blocked", "awaiting_evidence", "same_run_resume"]);
 
 /**
  * Cuts to at most `max` UTF-16 code units without splitting a surrogate pair.

server/src/services/native-runtime/native-finalization-reconciler.ts

@@ -1,3 +1,5 @@
+import { sameRunResumeClaimEntry } from "./scheduled-resume-continuation.js";
+import { appendBoundedRecoveryHistory } from "./native-restart-recovery.js";
 import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
 import { currentControllerProcessStamp } from "../controller-process-identity.js";
 import { NATIVE_SESSION_RESUMPTION_ISSUE_STATUSES } from "./native-session-resumption-statuses.js";
@@ -309,15 +311,16 @@ export async function claimNativeSessionResumptions(input: {
 
       const leaseExpiresAt = new Date(now.getTime() + 20 * 60_000);
       await notifyLiveRunRecovery(tx, { companyId: row.run.companyId, runId: row.run.id, agentId: row.run.agentId });
+      const stamp = currentControllerProcessStamp();
+      const resumeClaim = sameRunResumeClaimEntry({ at: now, controllerBootId: stamp.bootId,
+        providerAttempt: row.coordinator.attempt, failureDetail: row.coordinator.failureDetail });
       const updated = await tx.update(nativeRunFinalizations).set({
         phase: "observed",
         leaseOwner,
         leaseExpiresAt,
-        ...(() => {
-          const stamp = currentControllerProcessStamp();
-          return { controllerBootId: stamp.bootId, controllerPid: stamp.pid, controllerHostHash: stamp.hostHash,
-            controllerProcessStartedAt: stamp.processStartedAt ? new Date(stamp.processStartedAt) : null };
-        })(),
+        controllerBootId: stamp.bootId, controllerPid: stamp.pid, controllerHostHash: stamp.hostHash,
+        controllerProcessStartedAt: stamp.processStartedAt ? new Date(stamp.processStartedAt) : null,
+        recoveryHistory: appendBoundedRecoveryHistory(resumeClaim),
         failureCode: null,
         failureDetail: null,
         nextAttemptAt: null,

server/src/services/native-runtime/native-restart-recovery.ts

@@ -963,7 +963,7 @@ function historyEntry(input: {
   } satisfies Record<string, unknown>;
 }
 
-function appendBoundedRecoveryHistory(entry: Record<string, unknown>) {
+export function appendBoundedRecoveryHistory(entry: Record<string, unknown>) {
   const encoded = JSON.stringify(entry);
   return sql`(
     select coalesce(jsonb_agg(item order by ordinal), '[]'::jsonb)

server/src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts

@@ -14,6 +14,7 @@ vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
 import { executePaperclipNativeSession } from "./native-session-executor.js";
 import { buildNativeCompletionContract } from "./completion-contracts.js";
 import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
+import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
 
 const EXITED_PID = 4_194_304;
 
@@ -35,7 +36,7 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
   const exactResume = { recoveryMode: "exact_checkpoint_resume", recoverable: true, providerSessionEstablished: true,
     checkpointExists: true, originalFailureCode: "native_session_recovery_failed" };
 
-  async function resumeOnce(options: { failureDetail: Record<string, unknown>; stopRecorded: boolean }) {
+  async function resumeOnce(options: { failureDetail: Record<string, unknown>; stopRecorded: boolean; viaReconciler?: boolean }) {
     const issueId = randomUUID();
     const runId = randomUUID();
     const [issue] = await db.insert(issues).values({ id: issueId, companyId, title: "Resume the lost turn",
@@ -58,7 +59,12 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
         contract: buildNativeCompletionContract(issue!, { revision: contract!.revision }) },
       interactionResponses: [], credentialBindings: [],
     };
-    await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
+    await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution,
+      sessionCheckpoint: { backendKind: "runner", driverKind: "codex_app_server", sessionId: native.normalizedSessionId,
+        identity: { companyId, issueId, runId, agentId, sessionId: native.normalizedSessionId },
+        providerSessionId: "provider-thread", cursor: "1", activeTurnId: "lost-turn", pendingRuntimeRequests: [], lineage: [] } },
+      ...(options.viaReconciler ? { status: "failed" as const, nativePhase: "retryable_failure" } : {}) })
+      .where(eq(heartbeatRuns.id, runId));
     await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "retryable_failure", attempt: 1,
       failureCode: "native_session_interrupted", failureDetail: options.failureDetail });
     const identity = { processPid: EXITED_PID, processGroupId: EXITED_PID, processStartedAt: new Date().toISOString() };
@@ -80,7 +86,13 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
       passed.push(input.resumeInterruptedTurn);
       throw new Error("scheduled_resume_fixture_stop");
     });
-    await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: {
+    let leaseOwner: string | undefined;
+    if (options.viaReconciler) {
+      const claims = await claimNativeSessionResumptions({ db, runnerInstanceId: native.runnerInstanceId, runIds: [runId] });
+      expect(claims).toEqual([expect.objectContaining({ runId })]);
+      leaseOwner = claims[0]!.leaseOwner;
+    }
+    await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, leaseOwner, backend: {
       descriptor: async () => { throw new Error("Unexpected provider call"); },
       openSession: async () => { throw new Error("Unexpected provider call"); },
     } }).catch(() => undefined);
@@ -88,12 +100,17 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
     return { passed, coordinator: coordinator! };
   }
 
-  it("lets the runtime continue a lost turn after a verified stop and an admitted attempt", async () => {
-    const { passed, coordinator } = await resumeOnce({ failureDetail: exactResume, stopRecorded: true });
+  it.each([false, true])("lets the runtime continue a lost turn after a verified stop and an admitted attempt (claimed by the resume reconciler: %s)", async (viaReconciler) => {
+    const { passed, coordinator } = await resumeOnce({ failureDetail: exactResume, stopRecorded: true, viaReconciler });
     expect(passed).toEqual([true]);
     expect(coordinator.attempt).toBe(2);
   });
 
+  it("keeps the lost turn failing when the reconciler claimed a resume without a stop receipt", async () => {
+    const { passed } = await resumeOnce({ failureDetail: exactResume, stopRecorded: false, viaReconciler: true });
+    expect(passed).toEqual([false]);
+  });
+
   it.each([
     ["no stop of the previous runner was recorded", { failureDetail: exactResume, stopRecorded: false }],
     ["the saved failure is a bootstrap retry", { failureDetail: { ...exactResume, recoveryMode: "bootstrap_retry" }, stopRecorded: true }],

server/src/services/native-runtime/native-session-executor.ts

@@ -10371,6 +10371,9 @@ async function executePaperclipNativeSessionAttempt(
             targetKind: input.runnerExecutionTarget?.kind === "remote" ? "remote" : "local",
             coordinatorPhase: coordinator.phase,
             failureDetail: coordinator.failureDetail,
+            resumeClaim: Array.isArray(coordinator.recoveryHistory)
+              ? (coordinator.recoveryHistory.at(-1) as Record<string, unknown> | undefined) ?? null : null,
+            controllerBootId: controller.bootId,
             admittedAttempt: nextAttempt,
           })).allowed;
           const nextControllerGeneration = recovering

server/src/services/native-runtime/scheduled-resume-continuation.test.ts

@@ -8,6 +8,7 @@ import {
 } from "../../__tests__/helpers/embedded-postgres.js";
 import {
   decideScheduledResumeContinuation,
+  sameRunResumeClaimEntry,
   scheduledResumeContinuationAllowed,
   type ScheduledResumeContinuationEvidence,
 } from "./scheduled-resume-continuation.js";
@@ -53,6 +54,27 @@ describe("same-run scheduled resume continuation of a turn lost on restore", ()
   ])("refuses when %s", (_name, evidence, reason) => {
     expect(decideScheduledResumeContinuation(evidence)).toEqual({ allowed: false, reason });
   });
+
+  const resumeClaim = (patch: Record<string, unknown> = {}, detail: Record<string, unknown> = {}) =>
+    ({ ...sameRunResumeClaimEntry({ at: new Date(), controllerBootId: "boot-1", providerAttempt: 1,
+      failureDetail: { ...proven.failureDetail, ...detail } }), ...patch });
+  const claimed = (claim: Record<string, unknown> | null, patch: Partial<ScheduledResumeContinuationEvidence> = {}) =>
+    ({ ...proven, coordinatorPhase: "observed", failureDetail: null, resumeClaim: claim, controllerBootId: "boot-1", ...patch });
+
+  it("reads the evidence the resume reconciler recorded when it claimed the run", () => {
+    expect(decideScheduledResumeContinuation(claimed(resumeClaim()))).toEqual({ allowed: true });
+  });
+
+  it.each<[string, ScheduledResumeContinuationEvidence, string]>([
+    ["the last history entry is not the resume claim", claimed(resumeClaim({ disposition: "retryable_failure" })), "not_a_scheduled_same_run_resume"],
+    ["no history entry exists", claimed(null), "not_a_scheduled_same_run_resume"],
+    ["another boot recorded the resume claim", claimed(resumeClaim(), { controllerBootId: "boot-2" }), "not_a_scheduled_same_run_resume"],
+    ["the claimed failure was a bootstrap retry", claimed(resumeClaim({}, { recoveryMode: "bootstrap_retry" })), "not_an_exact_checkpoint_resume"],
+    ["the claimed failure had no checkpoint", claimed(resumeClaim({}, { checkpointExists: false })), "checkpoint_missing"],
+    ["the claimed failure recorded a replacement", claimed(resumeClaim({}, { successorRunId: "run-2" })), "replacement_recorded"],
+  ])("refuses a reconciler-claimed resume when %s", (_name, evidence, reason) => {
+    expect(decideScheduledResumeContinuation(evidence)).toEqual({ allowed: false, reason });
+  });
 });
 
 const support = await getEmbeddedPostgresTestSupport();

server/src/services/native-runtime/scheduled-resume-continuation.ts

@@ -2,6 +2,28 @@ import type { Db } from "@paperclipai/db";
 import { readNativeLocalProcessStop } from "../native-local-process-stop.js";
 import { NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT } from "./native-restart-recovery.js";
 
+export const SAME_RUN_RESUME_DISPOSITION = "same_run_resume";
+
+export function sameRunResumeClaimEntry(input: {
+  at: Date; controllerBootId: string; providerAttempt: number; failureDetail: Record<string, unknown> | null | undefined;
+}) {
+  const detail = input.failureDetail ?? {};
+  return {
+    at: input.at.toISOString(),
+    disposition: SAME_RUN_RESUME_DISPOSITION,
+    controllerBootId: input.controllerBootId,
+    providerAttempt: input.providerAttempt,
+    evidence: {
+      recoveryMode: detail.recoveryMode ?? null,
+      recoverable: detail.recoverable ?? null,
+      providerSessionEstablished: detail.providerSessionEstablished ?? null,
+      checkpointExists: detail.checkpointExists ?? null,
+      successorRunId: detail.successorRunId ?? null,
+      replacementDenied: detail.replacementDenied ?? null,
+    },
+  };
+}
+
 export type ScheduledResumeContinuationRefusal =
   | "restart_recovery_claim"
   | "remote_target"
@@ -19,6 +41,8 @@ export interface ScheduledResumeContinuationEvidence {
   targetKind: "local" | "remote";
   coordinatorPhase: string | null;
   failureDetail: Record<string, unknown> | null | undefined;
+  resumeClaim?: Record<string, unknown> | null;
+  controllerBootId?: string;
   admittedAttempt: number;
   localStop: { processPid: number } | null;
 }
@@ -33,8 +57,10 @@ export function decideScheduledResumeContinuation(
   const refuse = (reason: ScheduledResumeContinuationRefusal) => ({ allowed: false as const, reason });
   if (evidence.restartRecovery) return refuse("restart_recovery_claim");
   if (evidence.targetKind !== "local") return refuse("remote_target");
-  if (evidence.coordinatorPhase !== "retryable_failure") return refuse("not_a_scheduled_same_run_resume");
-  const detail = evidence.failureDetail ?? {};
+  const claimed = evidence.coordinatorPhase === "observed" && evidence.resumeClaim?.disposition === SAME_RUN_RESUME_DISPOSITION
+    && typeof evidence.controllerBootId === "string" && evidence.resumeClaim.controllerBootId === evidence.controllerBootId;
+  if (evidence.coordinatorPhase !== "retryable_failure" && !claimed) return refuse("not_a_scheduled_same_run_resume");
+  const detail = (claimed ? evidence.resumeClaim?.evidence as Record<string, unknown> | undefined : evidence.failureDetail) ?? {};
   if (detail.recoveryMode !== "exact_checkpoint_resume") return refuse("not_an_exact_checkpoint_resume");
   if (detail.recoverable !== true) return refuse("failure_not_recoverable");
   if (detail.providerSessionEstablished !== true) return refuse("provider_session_not_established");