fix(native): take back a lapsed lease held under the run's own restart claim (#20625)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run holds an execution lease on its
`native_run_finalizations` row. The owner renews it every 5 minutes.
After a host sleep (or the restart-chaos time fault TM2), the stored
lease looks lapsed although the owner is alive.
> - #20474 lets the same owner take back a lapsed lease when no other
actor recorded anything on the row. One of the checks is
"`recovery_state` is null".
> - Restart recovery writes `recovery_state` (`resuming_session`,
`awaiting_runner_reattach` or `bootstrap_incomplete`) in the same update
that grants the lease. The executor keeps that value until a result is
accepted.
> - So a run that a restart resumed can never take back its own lapsed
lease. Renewal interrupts the turn, and failure settlement is refused
with `native_session_lease_lost`. The run fails with `adapter_failed`,
recovery holds it with `preserve_without_replay_v1`, and the user must
click Retry.
> - This pull request counts the claim's own record as untouched when
the row still shows exactly that claim and nothing recorded since.
> - The benefit is that a resumed run survives a later host sleep with
no Retry. Every real takeover still fences the late owner out.

## Linked Issues or Issue Description

Refs #20474

**What happened?**
The zero-retry restart-chaos matrix (local wave L7, seed 9107: TM1, TM2,
TM3 with 1 h aging) had 3 of 6 tasks that needed Retry. All three were
Codex runs. In each run, a TM3 kill made restart recovery resume the run
(`resume_dead_runner`, generation 2). The runner restored the
conversation, reported `provider_turn_lost_on_restore`, and the
same-conversation restart continuation started as designed. Then the TM2
fault aged every database time by one hour. The execution lease lapsed
while its owner was alive. The run then failed with
`native_session_lease_lost` (`adapter_failed`), and recovery stopped
with `native_continuation_requires_reconciliation` /
`preserve_without_replay_v1`. One run had already proposed its result
and completed its turn. The Retry click worked every time.

**Expected behavior**
A run whose lease lapsed only because time jumped, and that no other
actor claimed, continues with no Retry. This must also be true after a
restart resumed that run.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`

## What Changed

- `server/src/services/native-runtime/native-execution-lease.ts`: the
"untouched" part of `nativeExecutionLeaseHeldCondition` now also accepts
the run's own local restart-recovery claim. All of these must be true:
- the last `recovery_history` entry is that claim: same controller boot
and generation as the row;
- its disposition matches the current `recovery_state`
(`resume_dead_runner` → `resuming_session`, `reattach_existing_runner` →
`awaiting_runner_reattach`, `bootstrap_incomplete` →
`bootstrap_incomplete`);
  - `recovery_request_id` equals the request that claim recorded.
Any later actor appends a history entry (shutdown suspension, failure
settlement), moves the state (`awaiting_evidence`, `blocked`), or
replaces the owner, boot or generation. Each of these still fences the
late owner out. All other untouched checks are unchanged.
- `native-execution-lease-reacquire.test.ts`: new cases for each claim
kind, a requested claim, failure settlement under the claim, and eight
fenced variants.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1
lists the new rule.

## Verification

- Red/green: `cd server && npx vitest run
src/services/native-runtime/native-execution-lease-reacquire.test.ts`.
Without the source change, 5 new cases fail (the three claim kinds, the
requested claim, and failure settlement). With it, all 37 pass.
- Related suites pass: `native-execution-lease.test.ts`,
`heartbeat-restart-claim-lapse.test.ts`,
`heartbeat-native-orphan-sweep-lease-race.test.ts`,
`native-runner-restart-lease-renewal.integration.test.ts`,
`native-runner-restart-preparation.integration.test.ts`.
- `cd server && npx tsc --noEmit` passes.
- Live: the same L7 wave (seed 9107, local) on this branch. The server
log has 0 `native_session_lease_lost` (the baseline had 3). The two
tasks whose cells failed this way in the baseline (codex C1 TM1, codex
C1 TM3) now pass with no Retry. Two other tasks needed Retry for
different causes, which this pull request does not change: codex B5 TM2
(a `runner.drain` timeout, then a same-run retry that may not continue a
turn lost on restore) and claude C1 TM2 (`ACPX session is not open` on
`turn.start` until the retry budget ran out). The L6 wave (seed 9106)
also ran on this branch: 5 of 6 passed. Its one failure is
`execution_recovery_budget_exhausted` and is not related to leases.

## Risks

- The rule widens when an owner may take back its own lapsed lease. It
applies only to local runs, before a result, with the same owner token,
attempt, boot, pid, process start and generation. A claim that any other
actor changed afterwards does not match.
- Rows saved by earlier releases carry the same history entry shape, so
a run claimed before a deploy is handled the same way. A deploy changes
the boot, so the new process never takes back the old process's lease.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md

@@ -22,7 +22,14 @@ succeeds only when all of these hold:
   process start are unchanged;
 - the phase is `observed` and no result was accepted;
 - `recovery_state`, `recovery_request_id`, `failure_code` and `next_attempt_at`
-  are null;
+  are null, or (2026-11-10) they still show exactly the local restart-recovery
+  claim that granted this lease: the last `recovery_history` entry names this
+  boot and generation, its disposition pairs with `recovery_state`
+  (`resume_dead_runner`/`resuming_session`, `reattach_existing_runner`/
+  `awaiting_runner_reattach`, `bootstrap_incomplete`/`bootstrap_incomplete`),
+  and `recovery_request_id` is the request that claim recorded. A resumed run
+  keeps that state until a result is accepted, so without this rule it could
+  never survive a later host sleep;
 - `controller_host_hash` is not the board's attested-stopped stamp;
 - the run is still `running`, without the ownership-unverified error code and
   without a `nativeStartupCleanup` marker;

server/src/services/native-runtime/native-execution-lease-reacquire.test.ts

@@ -192,6 +192,104 @@ describeEmbeddedPostgres("native execution lease taken back after a host sleep",
     });
   }
 
+  function restartClaimEntry(overrides: Record<string, unknown> = {}) {
+    return { at: new Date().toISOString(), restartKind: "hard", controllerBootId: controller.bootId,
+      controllerPid: controller.pid, controllerProcessStartedAt: controller.processStartedAt.toISOString(),
+      controllerGeneration: GENERATION, disposition: "resume_dead_runner", reason: "dead_runner_with_exact_provider_checkpoint",
+      stateRootAction: "reuse_exact_root", recoveryRequestId: null, providerAttempt: 1, ...overrides };
+  }
+
+  async function seedResumedSleptRun(label: string, state = "resuming_session",
+    history: Array<Record<string, unknown>> = [restartClaimEntry()], recoveryRequestId: string | null = null) {
+    const run = await seedSleptRun(label);
+    await db.update(nativeRunFinalizations).set({ recoveryState: state, recoveryRequestId, recoveryHistory: history })
+      .where(eq(nativeRunFinalizations.runId, run.runId));
+    return run;
+  }
+
+  it.each([
+    ["a dead Runner resumed", "resuming_session", "resume_dead_runner"],
+    ["a live Runner reattached", "awaiting_runner_reattach", "reattach_existing_runner"],
+    ["an incomplete bootstrap restarted", "bootstrap_incomplete", "bootstrap_incomplete"],
+  ])("takes back a lease held under its own restart claim (%s)", async (_name, state, disposition) => {
+    const run = await seedResumedSleptRun(`resumed-${disposition}`, state, [
+      restartClaimEntry({ controllerGeneration: GENERATION - 1, controllerBootId: `boot-${randomUUID()}` }),
+      restartClaimEntry({ disposition }),
+    ]);
+    await expect(renewNativeSessionExecutionLease(run.renewal)).resolves.toBeUndefined();
+    const saved = await row(run.runId);
+    expect(saved.leaseOwner).toBe(run.leaseOwner);
+    expect(saved.recoveryState).toBe(state);
+    expect(saved.leaseExpiresAt!.getTime()).toBeGreaterThan(Date.now() + 15 * 60_000);
+  });
+
+  it("takes back a lease held under a requested restart claim with the same request", async () => {
+    const requestId = randomUUID();
+    const run = await seedResumedSleptRun("requested", "resuming_session",
+      [restartClaimEntry({ recoveryRequestId: requestId })], requestId);
+    await expect(renewNativeSessionExecutionLease(run.renewal)).resolves.toBeUndefined();
+  });
+
+  it("lets the owner release its own lapsed lease under its own restart claim", async () => {
+    const run = await seedResumedSleptRun("resumed-release");
+    expect(await releaseAsOwner(run)).toHaveLength(1);
+  });
+
+  async function settleFailureAsOwner(run: SleptRun) {
+    return owner.transaction(async tx => {
+      await lockNativeExecutionLeaseForReacquire(tx, { ...run, preResultOnly: true });
+      return tx.update(nativeRunFinalizations).set({
+        phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, recoveryState: "resuming_session",
+        failureCode: "native_session_lease_lost", nextAttemptAt: new Date(),
+        recoveryHistory: sql`coalesce(${nativeRunFinalizations.recoveryHistory}, '[]'::jsonb)
+          || jsonb_build_array(${JSON.stringify({ disposition: "retryable_failure", reason: "native_session_lease_lost",
+            controllerBootId: controller.bootId, controllerGeneration: GENERATION })}::jsonb)`,
+      }).where(and(eq(nativeRunFinalizations.runId, run.runId), eq(nativeRunFinalizations.leaseOwner, run.leaseOwner),
+        eq(nativeRunFinalizations.attempt, 1), eq(nativeRunFinalizations.controllerGeneration, GENERATION),
+        eq(nativeRunFinalizations.controllerBootId, controller.bootId),
+        nativeExecutionLeaseHeldCondition({ remote: false, controllerGeneration: GENERATION })))
+        .returning({ runId: nativeRunFinalizations.runId });
+    });
+  }
+
+  it("lets failure settlement write a resumable failure under its own restart claim", async () => {
+    const run = await seedResumedSleptRun("resumed-settle");
+    expect(await settleFailureAsOwner(run)).toHaveLength(1);
+    const saved = await row(run.runId);
+    expect(saved).toMatchObject({ phase: "retryable_failure", leaseOwner: null, recoveryState: "resuming_session",
+      failureCode: "native_session_lease_lost" });
+    expect((saved.recoveryHistory as Array<{ disposition?: string }>).at(-1)?.disposition).toBe("retryable_failure");
+    await expect(renewNativeSessionExecutionLease(run.renewal)).rejects.toThrow("native_session_lease_lost");
+  });
+
+  const restartClaimChanged: Array<[string, () => Promise<SleptRun>]> = [
+    ["a graceful shutdown suspended the claim", () => seedResumedSleptRun("suspended", "awaiting_runner_reattach",
+      [restartClaimEntry({ disposition: "reattach_existing_runner" }),
+        { at: new Date().toISOString(), restartKind: "graceful", disposition: "restart_suspended", reason: "SIGTERM" }])],
+    ["the claim belongs to an earlier generation", () => seedResumedSleptRun("old-generation", "resuming_session",
+      [restartClaimEntry({ controllerGeneration: GENERATION - 1 })])],
+    ["the claim belongs to another boot", () => seedResumedSleptRun("other-boot", "resuming_session",
+      [restartClaimEntry({ controllerBootId: `boot-${randomUUID()}` })])],
+    ["the state does not match the claimed disposition", () => seedResumedSleptRun("mismatch", "awaiting_runner_reattach",
+      [restartClaimEntry()])],
+    ["recovery now waits for evidence", () => seedResumedSleptRun("evidence", "awaiting_evidence", [restartClaimEntry()])],
+    ["a later recovery request was recorded", () => seedResumedSleptRun("request", "resuming_session",
+      [restartClaimEntry()], randomUUID())],
+    ["failure settlement appended its outcome", () => seedResumedSleptRun("settled", "resuming_session",
+      [restartClaimEntry(), { at: new Date().toISOString(), disposition: "retryable_failure", reason: "native_session_lease_lost",
+        controllerBootId: controller.bootId, controllerGeneration: GENERATION }])],
+    ["no claim was ever recorded", () => seedResumedSleptRun("no-history", "resuming_session", [])],
+  ];
+
+  it.each(restartClaimChanged)("stays fenced out when %s", async (_name, seed) => {
+    const run = await seed();
+    const before = await row(run.runId);
+    await expect(renewNativeSessionExecutionLease(run.renewal)).rejects.toThrow("native_session_lease_lost");
+    expect(await releaseAsOwner(run)).toHaveLength(0);
+    expect(await settleFailureAsOwner(run)).toHaveLength(0);
+    expect(await row(run.runId)).toEqual(before);
+  });
+
   it("releases its own lapsed lease after the result was accepted, but never renews it", async () => {
     const accepted = await seedSleptRun("accepted");
     // Only the coordinator's accepted-result shape matters here; skip the

server/src/services/native-runtime/native-execution-lease.ts

@@ -39,13 +39,32 @@ const RENEWAL_STATEMENT_TIMEOUT_MS = 10_000;
  * Callers lock with `lockNativeExecutionLeaseForReacquire` first, so the run
  * row this reads is the latest committed version and stays so until commit.
  */
+const OWN_RESTART_CLAIM_STATES: ReadonlyArray<readonly [disposition: string, recoveryState: string]> = [
+  ["resume_dead_runner", "resuming_session"],
+  ["reattach_existing_runner", "awaiting_runner_reattach"],
+  ["bootstrap_incomplete", "bootstrap_incomplete"],
+];
+
+function nativeOwnRestartClaimUnchanged(): SQL {
+  const last = sql`(${nativeRunFinalizations.recoveryHistory} -> -1)`;
+  const pairs = sql.join(OWN_RESTART_CLAIM_STATES.map(([disposition, state]) =>
+    sql`(${last} ->> 'disposition' = ${disposition} and ${nativeRunFinalizations.recoveryState} = ${state})`), sql` or `);
+  return sql`(jsonb_typeof(${nativeRunFinalizations.recoveryHistory}) = 'array'
+    and (${pairs})
+    and ${last} ->> 'controllerGeneration' = ${nativeRunFinalizations.controllerGeneration}::text
+    and ${last} ->> 'controllerBootId' = ${nativeRunFinalizations.controllerBootId}
+    and ${nativeRunFinalizations.recoveryRequestId} is not distinct from (${last} ->> 'recoveryRequestId'))`;
+}
+
 function nativeExecutionLeaseUntouchedCondition(stage: "observed" | "accepted"): SQL {
   const coordinatorStage = stage === "observed"
     ? sql`${nativeRunFinalizations.phase} = 'observed' and ${nativeRunFinalizations.resultId} is null`
     : sql`${nativeRunFinalizations.phase} = 'workspace_finalizing' and ${nativeRunFinalizations.resultId} is not null`;
   return sql`(${coordinatorStage}
-    and ${nativeRunFinalizations.recoveryState} is null
+    and (
-    and ${nativeRunFinalizations.recoveryRequestId} is null
+      (${nativeRunFinalizations.recoveryState} is null and ${nativeRunFinalizations.recoveryRequestId} is null)
+      or ${nativeOwnRestartClaimUnchanged()}
+    )
     and ${nativeRunFinalizations.failureCode} is null
     and ${nativeRunFinalizations.nextAttemptAt} is null
     and ${nativeRunFinalizations.controllerHostHash} is distinct from ${OPERATOR_ATTESTED_CONTROLLER_HOST}