fix(native): restore retired backups after warm lease handoffs (#19299)

## Thinking Path

> - Paperclip manages AI agents and keeps their work across turns.
> - A native session can keep the same sandbox through several warm
lease handoffs.
> - Each handoff keeps the old lease row and records matching receipts
on both owners.
> - The retired-backup guard required exactly one row for a provider
allocation.
> - That rule rejected valid history after Paperclip destroyed the
sandbox.
> - This change uses the existing handoff proof to select the final
owner.
> - A verified backup can then restore without discarding ownership
checks.

## Linked Issues or Issue Description

Refs #19208.

**What happened?**

A follow-up received a new sandbox after its old sandbox was destroyed.
The old allocation had a normal warm lease handoff. Both lease rows
remained in PostgreSQL. The guard rejected them with
`runner_harness_state_mismatch: backup_without_reusable_lease`. Its
later same-run retry failed with
`native_remote_recovery_lease_mismatch`.

**Expected behavior**

Restore the verified backup when one final owner has a valid destroyed
receipt and every earlier owner has a matching handoff receipt. Keep
uncertain ownership blocked.

**Steps to reproduce**

1. Complete a native warm turn.
2. Hand the reusable sandbox to another turn.
3. Complete that turn and save its backup.
4. Close the workspace and destroy the sandbox.
5. Send a follow-up. The new allocation must restore the saved session.

**Paperclip version or commit**

Reproduced in the PostgreSQL regression on the parent of this change.
The affected deployed build also contains #19208.

## What Changed

- Select the single final lease owner. Validate all predecessor rows
with `retiredReusableLeasePredecessors`.
- Keep the destroyed receipt, exact backup stamp, and
newer-provider-work checks.
- Add database regressions for one and three handoffs. Reject missing or
changed receipts, an active predecessor, missing history, an unrelated
owner, a stopped source, another backup, and later provider work.
- Document the valid lease-history shape.

## Verification

- Red before the source change: the two valid-handoff cases returned
false. The other 16 database cases passed.
- Green after the change: all 18 database cases passed. They also passed
on current head `d16e098b8c2e6c0dda70984c1df21e3dc792ab34`.
- Command: `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-retired-backup-source.integration.test.ts`.
- The database, executor, retry-admission, and lease suites passed: 4
files, 1,308 tests.
- `pnpm -r typecheck` and `pnpm build` passed. The build stamp names
head `4ee733c3a979436196522f36537ff182e9df5398`. The full local suite
stopped in chat shard 3 after a 15-second timeout in `materializes
direct external-chat finals once and accepts the next turn without agent
bookkeeping writes`. Earlier shards passed 702 cases; that shard passed
303 other cases. The isolated test passes on this branch and unchanged
master `10a7d039c8efeac0fd655c24970df5ae3658414b`. All current-head CI
chat shards also passed. The earlier full-suite attempt remains failed;
its timeout cause is not established. No test or timeout was changed. CI
is waiting for its last Runner job.
- Current head `d16e098b8c2e6c0dda70984c1df21e3dc792ab34` removes four
comment lines from the previously tested head. Its 18 database cases
passed. Greptile reviewed this head at 5/5 with no actionable findings:
https://github.com/paperclipai/paperclip/pull/19299#issuecomment-6665446098.
CI remains pending.
- Live inspection found the matching predecessor/successor receipts and
the final owner's destroyed receipt. This was read-only evidence from
the affected build.
- A live provider run at the fixed revision has not been exercised.
Local Daytona credentials and an immutable test image are unavailable.
No deployment or original-task retry is claimed.
- Initial wider tests could not resolve the unbuilt plugin SDK. The SDK
build is now available. A sandboxed database launch could not bind
loopback; the same regression passed with the disposable database
outside that restriction.

## Risks

- A wrong owner could allow an old backup to rewind a session. The
shared validator must account for every lease row and match both sides
of each handoff.
- Live, unrelated, incomplete, or conflicting history stays blocked. The
existing backup integrity and provider-progress checks remain.
- No migration, permission, model, trust, timeout, or retry-policy
change is included.
- The original task still needs the fixed build and normal recovery.
Hosted acceptance remains unverified.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, code execution, and tool
use. The session does not expose the exact model ID or context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/execution-semantics.md

@@ -2074,7 +2074,10 @@ backup. Paperclip restores that backup into the new sandbox only when all of
 these are true: the backup's source lease is expired or failed with a provider
 `destroyed` termination receipt, the source lease's backup stamp names this
 exact backup manifest, and no provider event for the session was recorded after
-the backup completed. Any other state is ambiguous. The run then fails closed
+the backup completed. Normal warm handoffs can leave multiple lease records
+for that allocation. Restore requires one final owner and the existing matching
+predecessor/successor receipts for every earlier owner. Unrelated, live, or
+incompletely linked owners remain ambiguous. The run then fails closed
 with `runner_harness_state_mismatch: backup_without_reusable_lease`.
 
 A per-turn sandbox turn can finish through recovery without a Runner

server/src/services/native-runtime/native-retired-backup-source.integration.test.ts

@@ -1,7 +1,8 @@
 import { randomUUID } from "node:crypto";
 import { afterAll, beforeAll, describe, expect, it } from "vitest";
-import { agents, companies, createDb, environmentLeases, heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db";
+import { agents, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issues } from "@paperclipai/db";
 import { recoveryPostgresSupport, startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js";
+import { createReusableLeaseHandoff } from "../environment-lease-handoff.js";
 import { retiredSourceAuthorizesNativeBackupRestore } from "./native-retired-backup-source.js";
 
 const describeDb = recoveryPostgresSupport.supported ? describe : describe.skip;
@@ -19,7 +20,8 @@ describeDb("retired native backup source authority", () => {
   const completedAt = new Date("2026-11-05T05:02:50.000Z");
 
   async function fixture(options: { termination?: "destroyed" | "stopped" | null; status?: string; stampSha?: string;
-    providerEventAt?: Date; secondLease?: boolean } = {}) {
+    providerEventAt?: Date; secondLease?: boolean; handoffs?: number;
+    handoffFault?: "missing_receipt" | "changed_receipt" | "active_predecessor" | "missing_predecessor" } = {}) {
     const companyId = randomUUID(), agentId = randomUUID(), sessionId = randomUUID(), runnerInstanceId = randomUUID();
     const sourceRunId = randomUUID(), currentRunId = randomUUID(), leaseId = randomUUID(), providerLeaseId = randomUUID();
     await db.insert(companies).values({ id: companyId, name: "Retired source", issuePrefix: `R${companyId.slice(0, 5)}` });
@@ -37,7 +39,43 @@ describeDb("retired native backup source authority", () => {
           runId: sourceRunId, leaseId: id, provider: "daytona", providerLeaseId, state: options.termination ?? "destroyed",
           confirmedAt: new Date().toISOString() } }),
       } });
-    await db.insert(environmentLeases).values(lease(leaseId));
+    const sourceLease = lease(leaseId);
+    if (options.handoffs) {
+      const environmentId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID();
+      await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" });
+      await db.insert(issues).values({ id: issueId, companyId, title: "Retained session" });
+      await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId,
+        mode: "shared_workspace", strategyType: "project_primary", name: "Retained workspace" });
+      const history: Array<typeof environmentLeases.$inferInsert> = [];
+      for (let index = 0; index <= options.handoffs; index++) {
+        const ownerRunId = index === options.handoffs ? sourceRunId : randomUUID();
+        if (ownerRunId !== sourceRunId) await db.insert(heartbeatRuns).values({ id: ownerRunId, companyId, agentId,
+          status: "succeeded", runtimeMode: "native", nativeSessionId: sessionId,
+          invocationSource: "assignment", triggerDetail: "system" });
+        const acquiredAt = new Date(completedAt.getTime() - (options.handoffs - index + 1) * 60_000);
+        const row = { ...sourceLease, id: index === options.handoffs ? leaseId : randomUUID(),
+          heartbeatRunId: ownerRunId, environmentId, issueId, executionWorkspaceId: workspaceId, acquiredAt,
+          leasePolicy: "reuse_by_environment", metadata: { ...sourceLease.metadata, agentId } };
+        if (index < options.handoffs) {
+          const { nativeHarnessBackup: _backup, remoteExecutionTermination: _termination, ...metadata } = row.metadata;
+          history.push({ ...row, metadata });
+        } else history.push(row);
+      }
+      for (let index = 1; index < history.length; index++) {
+        const previous = history[index - 1]!, current = history[index]!;
+        const handoff = createReusableLeaseHandoff(previous as typeof environmentLeases.$inferSelect,
+          current as typeof environmentLeases.$inferSelect);
+        previous.releasedAt = current.acquiredAt;
+        previous.metadata = { ...previous.metadata, reusableLeaseSuccessor: handoff };
+        current.metadata = { ...current.metadata, reusableLeasePredecessor: handoff };
+      }
+      if (options.handoffFault === "missing_receipt") delete history[0]!.metadata!.reusableLeaseSuccessor;
+      if (options.handoffFault === "changed_receipt") history[0]!.metadata!.reusableLeaseSuccessor = {
+        ...(history[0]!.metadata!.reusableLeaseSuccessor as Record<string, unknown>), successorRunId: randomUUID() };
+      if (options.handoffFault === "active_predecessor") history[0]!.status = "active";
+      if (options.handoffFault === "missing_predecessor") history.shift();
+      await db.insert(environmentLeases).values(history);
+    } else await db.insert(environmentLeases).values(sourceLease);
     if (options.secondLease) await db.insert(environmentLeases).values(lease(randomUUID()));
     // The backup turn's own provider events precede the checkpoint.
     await db.insert(heartbeatRunEvents).values({ companyId, runId: sourceRunId, agentId, seq: 1, eventType: "turn.started",
@@ -54,6 +92,24 @@ describeDb("retired native backup source authority", () => {
     await expect(fixture()).resolves.toBe(true);
   });
 
+  it.each([1, 3])("authorizes a destroyed allocation after %i verified warm lease handoffs", async (handoffs) => {
+    await expect(fixture({ handoffs })).resolves.toBe(true);
+  });
+
+  it.each(["missing_receipt", "changed_receipt", "active_predecessor", "missing_predecessor"] as const)(
+    "fails closed for a warm lease chain with %s", async (handoffFault) => {
+      await expect(fixture({ handoffs: 2, handoffFault })).resolves.toBe(false);
+    });
+
+  it.each([
+    ["an unrelated owner", { secondLease: true }],
+    ["a stopped source", { termination: "stopped" as const }],
+    ["another checkpoint", { stampSha: `sha256:${"c".repeat(64)}` }],
+    ["later provider work", { providerEventAt: new Date(completedAt.getTime() + 60_000) }],
+  ])("keeps the backup restore held after warm handoffs with %s", async (_label, options) => {
+    await expect(fixture({ handoffs: 2, ...options })).resolves.toBe(false);
+  });
+
   it.each([
     ["the provider only stopped the source", { termination: "stopped" as const }],
     ["no termination receipt exists", { termination: null }],

server/src/services/native-runtime/native-retired-backup-source.ts

@@ -1,6 +1,7 @@
 import { createHash } from "node:crypto";
 import { and, eq, gt, inArray, ne } from "drizzle-orm";
 import { environmentLeases, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db";
+import { retiredReusableLeasePredecessors } from "../environment-lease-handoff.js";
 import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
 import { readNativeHarnessBackupStamp } from "./native-harness-backup-stamp.js";
 
@@ -42,9 +43,10 @@ export async function retiredSourceAuthorizesNativeBackupRestore(db: Db, input:
   ));
   const source = leases.filter(lease => lease.companyId === input.companyId
     && lease.providerLeaseId === input.manifest.sourceProviderLeaseId);
-  // One provider allocation maps to one host lease. Anything else is ambiguous.
-  if (source.length !== 1) return false;
-  const lease = source[0]!;
+  const tips = source.filter(lease => !Object.hasOwn(lease.metadata ?? {}, "reusableLeaseSuccessor"));
+  if (tips.length !== 1) return false;
+  const lease = tips[0]!;
+  if (!retiredReusableLeasePredecessors(lease, source.filter(row => row.id !== lease.id))) return false;
   const termination = lease.metadata?.remoteExecutionTermination as { state?: unknown } | undefined;
   // A released lease can still be selected for resume; only a retired one is final.
   if (!["expired", "failed"].includes(lease.status) || !hasRemoteTerminationReceipt(lease)