fix(recovery): continue from a retired sandbox's backup and hand spent export repairs to the board (#19208)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native tasks keep a warm sandbox and a verified per-turn backup of
the provider session.
> - Recovery holds stop a task when continuation is not safe. They must
have a next action and an owner.
> - In a staging soak, two kinds of held tasks could not move forward.
Users followed the product's own instruction, and nothing changed.
> - First, a reply to a task whose warm sandbox Paperclip had already
destroyed failed every time. The label then said "Send a message to
continue" again.
> - Second, an exhausted system workspace repair stayed active and
system-owned for hours. The board saw no owner, and a reply started
nothing.
> - This pull request lets a reply continue from the backup when the
source sandbox is provably gone, and hands a spent system repair to the
board with a named Retry.
> - The benefit is that productive work always has a next action that
can succeed, and nothing replays or runs twice.

## Linked Issues or Issue Description

**What happened?**
(a) A native warm task ran one successful turn. A later legacy follow-up
retired the warm sandbox lease. The next user reply got a new sandbox
(`created` lease). The executor found a verified backup but no reusable
lease, so it threw `runner_harness_state_mismatch:
backup_without_reusable_lease`. The run retried on the restart-recovery
path, failed `native_remote_recovery_lease_mismatch`, and ended as
`native_continuation_requires_reconciliation`. Workspace recovery then
said "Saved work recovered. Send a message to continue." Each new reply
repeated the same failure.
(b) A run failed `native_workspace_sync_out_retry_exhausted`. System
workspace recovery used its 6 bounded attempts and became `exhausted`.
The recovery action stayed `active` with `ownerType: system`. The UI
hides system-owned recovery actions, so the board saw no action. Nothing
ran for more than two hours.

**Expected behavior**
(a) When Paperclip destroyed the source sandbox and no provider work
followed the backup, the reply continues once from the backup. Ambiguous
states still fail closed.
(b) When the bounded budget is spent, the action names a board owner and
a Retry action that can progress it.

**Steps to reproduce**
(a) Finish a warm native turn. Retire its reusable lease through
obsolete-lease cleanup. Send a reply. The new run gets a `created` lease
and fails `backup_without_reusable_lease`.
(b) Let system workspace recovery reach its deadline or retry limit.
Read the recovery action. It is still `ownerType: system`.

Related merged work checked first: #19145, #19184, #19187, #19189,
#19190, #18943 (added the guard for migrated sessions). Open recovery
PRs (`gh pr list --search recovery --state open`) do not touch these
paths.

## What Changed

- `native-retired-backup-source.ts` (new):
`retiredSourceAuthorizesNativeBackupRestore` decides whether a created
lease may restore the backup. All must be true: the backup's source
lease is `expired` or `failed`; it has a provider termination receipt
with state `destroyed`; its `nativeHarnessBackup` stamp names this exact
manifest SHA-256, session and runner instance; exactly one host lease
maps to the provider allocation; and no provider durable event for the
session was recorded after the backup completed. The provider event type
set moved here and the executor imports it.
- `native-session-executor.ts`:
`shouldRestoreNativeHarnessBackupIntoSandbox` accepts
`retiredSourceContinuation`. A created lease with a configured reusable
lease evaluates it, beside the existing migrated-session path. The
restore uses the existing verified restore and stamps the new lease.
- `system-recovery.ts`: `updateSystemRecovery` moves an `exhausted`
action to `ownerType: board` and writes a next action and workspace
recovery message that name the attempt count, the last failure code, and
**Retry recovery**. Retry (`requestWorkspaceRecovery`) already starts a
new bounded generation and returns ownership to the system.
- `doc/execution-semantics.md`: documents both rules.

## Verification

- Red before the fix, green after:
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "restores
a warm backup into a created lease only when"`. Before: the
destroyed-source case rejected with `runner_harness_state_mismatch:
backup_without_reusable_lease`. After: it restores the runner state and
provider home into the new sandbox and stamps the new lease.
- `npx vitest run src/services/workspace-recovery.test.ts -t "hands an
exhausted system export repair"`. Before: `exhausted` with `ownerType:
"system"`. After: `ownerType: "board"`, the blocker says Retry recovery
and never says "send a message", no provider run or lease resume
happens, and Retry starts one new system generation.
- Safety cases (fail closed): a stopped (not destroyed) source, a stamp
for another checkpoint, provider work after the backup, a still-reusable
`released` lease, a missing receipt, and two host leases for one
allocation. These run in the executor test and in
`native-retired-backup-source.integration.test.ts` against embedded
Postgres.
- Related suites pass: `workspace-recovery.test.ts`,
`system-recovery-state.test.ts`, `native-session-executor.test.ts`,
`task-retry.test.ts`, `explicit-native-continuation.test.ts`,
`workspace-recovery-source.test.ts`,
`native-workspace-export-retry.test.ts`,
`native-remote-retry-admission.test.ts`,
`native-workspace-warm-reuse.test.ts`,
`native-workspace-sync-retention.test.ts`,
`heartbeat-observed-workspace-preservation.test.ts`,
`execution-control-work.test.ts` (2,224 tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Not done: a live run against staging. Staging access was read-only.
The staging evidence is the run event timeline described above.

## Risks

- (a) A backup restore after a lost sandbox could rewind a session if
newer provider state existed. The predicate requires a provider
`destroyed` receipt and no provider event after the checkpoint. A
stopped sandbox, a pending cleanup, or a still-reusable lease still
fails closed with the old error.
- (a) The event-time check can fail closed if a late event from the
backup turn is written after the checkpoint time. That keeps the old
behavior; it never authorizes a restore.
- (b) The action's next action text stays as the exhausted message while
a later Retry generation runs. The task blocker shows the live workspace
recovery message first, so users see "Recovering saved work…" during
that time.
- Not changed: when the predicate fails closed, the run still follows
the existing reconciliation path. That path can still end with the "Send
a message to continue" receipt. That residual is narrower now and needs
a separate change to the workspace recovery receipt.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with
tool use (shell, file edits, read-only browser inspection of staging).
Default context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/native-runtime/native-session-executor.test.ts

@@ -17299,6 +17299,113 @@ describe("runnerd provider runtime wiring", () => {
     }
   });
 
+  it.each([
+    ["the source sandbox was destroyed by Paperclip", "destroyed", "matching", false, true],
+    ["the source sandbox was only stopped", "stopped", "matching", false, false],
+    ["the source stamp names another backup", "destroyed", "foreign", false, false],
+    ["provider work followed the backup", "destroyed", "matching", true, false],
+  ] as const)("restores a warm backup into a created lease only when %s", async (_label, terminationState, stampKind, laterProviderWork, restores) => {
+    // Regression: a legacy follow-up retired the warm sandbox as an obsolete lease, so the
+    // next reply got a created lease. The backup was the newest checkpoint, yet
+    // every reply failed backup_without_reusable_lease and asked for another reply.
+    const remoteCwd = join(isolatedStateDirectory, "remote");
+    const current: NativeExecutionInputV1 = { ...execution,
+      binding: { ...execution.binding, companyId: randomUUID(), agentId: randomUUID(), issueId: randomUUID(),
+        runId: randomUUID(), executionWorkspaceId: randomUUID() },
+      session: { ...execution.session, normalizedSessionId: randomUUID(), lifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } },
+    };
+    const runnerInstanceId = "retired-warm-runner", completedAt = "2026-11-05T05:02:50.000Z";
+    const backupDirectory = { root: "" };
+    const sourceLease = (manifestSha256: string) => ({
+      id: "source-lease", companyId: current.binding.companyId, heartbeatRunId: "source-run", provider: "daytona",
+      providerLeaseId: "prior-sandbox", status: "expired", cleanupStatus: "success", releasedAt: new Date(),
+      metadata: {
+        nativeHarnessBackup: { schema: "paperclip.native-harness-backup-stamp.v2", normalizedSessionId: current.session.normalizedSessionId,
+          runnerInstanceId, sessionScopeSha256: "a".repeat(64), sourceProviderLeaseId: "prior-sandbox",
+          authorizedProviderLeaseId: "prior-sandbox", manifestSha256, completedAt },
+        remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: current.binding.companyId,
+          runId: "source-run", leaseId: "source-lease", provider: "daytona", providerLeaseId: "prior-sandbox",
+          state: terminationState, confirmedAt: new Date().toISOString() },
+      },
+    });
+    const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
+    const db = leaseDb(current, {}, {}, updates), ordinarySelect = db.select.bind(db);
+    db.select = ((fields?: Record<string, unknown>) => ({ from: (table: unknown) => {
+      if (table !== executionWorkspaces && table !== environmentLeases && table !== heartbeatRunEvents)
+        return ordinarySelect(fields as never).from(table as never);
+      const manifest = backupDirectory.root ? readFileSync(join(backupDirectory.root, "manifest.json")) : "";
+      const sha = `sha256:${createHash("sha256").update(stampKind === "matching" ? manifest : "other").digest("hex")}`;
+      const rows = table === executionWorkspaces ? []
+        : table === heartbeatRunEvents ? (laterProviderWork ? [{ runId: "later-run" }] : [])
+          : [{ metadata: {}, providerLeaseId: "sandbox-created", companyId: current.binding.companyId,
+            heartbeatRunId: current.binding.runId }, sourceLease(sha)];
+      const query = { then: Promise.resolve(rows).then.bind(Promise.resolve(rows)),
+        leftJoin: () => query, innerJoin: () => query, where: () => query, limit: () => query };
+      return query;
+    } })) as unknown as Db["select"];
+    const remoteExecute = vi.fn(async (command: { command: string; args?: string[] }) => {
+      if (command.args?.[0] === "--build-metadata") return { exitCode: 0, timedOut: false, stderr: "", stdout: JSON.stringify({
+        schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
+        packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
+        durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
+      }) };
+      if (command.args?.[0] === "--version") return { exitCode: 0, timedOut: false, stdout: "codex-cli 0.160.0", stderr: "" };
+      if (command.args?.[1]?.includes("base64")) {
+        try { return { exitCode: 0, timedOut: false, stdout: execFileSync("sh", command.args, { encoding: "utf8" }), stderr: "" }; }
+        catch { return { exitCode: 1, timedOut: false, stdout: "", stderr: "" }; }
+      }
+      return { exitCode: 0, timedOut: false, stdout: "", stderr: "" };
+    });
+    const syncIn = vi.fn(async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string }> }>) => {
+      for (const operation of operations) for (const file of operation.files) {
+        await mkdir(dirname(file.targetPath), { recursive: true });
+        await cp(file.sourcePath, file.targetPath, { recursive: true });
+      }
+    });
+    await createRunnerdBackend({ db, execution: current, runnerInstanceId, runnerIngressAuthorized: true,
+      runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd, environmentId: "environment",
+        leaseId: "lease-created", providerKey: "daytona", reusableLeaseConfigured: true,
+        effectiveCapabilities: { runnerWebSocketIngress: true },
+        sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "sandbox-created" },
+        runner: { execute: remoteExecute, syncIn },
+      } as never,
+    });
+    state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
+    const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise<void> };
+    const backupRoot = join(options.stateDirectory!, "failover-backups/current");
+    backupDirectory.root = backupRoot;
+    for (const directory of resolveNativeHarnessPersistenceProfile(current).directories)
+      await mkdir(join(backupRoot, directory.name), { recursive: true });
+    const runnerState = durableRunnerState({ runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId,
+      runnerInstanceId, environmentLeaseId: "lease-created" }, "suspended");
+    await writeFile(join(backupRoot, "runner/runner-state.json"), JSON.stringify(runnerState));
+    await writeFile(join(backupRoot, "runner/codex-provider-state.json"), JSON.stringify({
+      schema: "paperclip.runner.codex-provider-state.v1", lifecycle: "prepared", threadId: "retained-provider-thread",
+      providerSessionId: null, activeProviderTurnId: null, config: { provider: "codex", driver: "codex_app_server" },
+    }));
+    await writeFile(join(backupRoot, "codex-home/sentinel.txt"), "retained provider home");
+    await writeFile(join(backupRoot, "manifest.json"), JSON.stringify(buildNativeHarnessBackupManifest({
+      backupRoot, execution: current, runnerInstanceId, sourceProviderLeaseId: "prior-sandbox",
+      providerSessionIdentity: { providerSessionId: "retained-provider-thread", providerBackendSessionId: null, providerSessionIdentity: null },
+    })));
+    if (restores) {
+      await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
+      const remoteSession = join(remoteCwd, ".paperclip-runtime/paperclip-runner/sessions",
+        createHash("sha256").update(current.session.normalizedSessionId!).digest("hex"));
+      expect(JSON.parse(await readFile(join(remoteSession, "runner/runner-state.json"), "utf8"))).toEqual(runnerState);
+      expect(await readFile(join(remoteSession, "filesystem/codex-home/sentinel.txt"), "utf8")).toBe("retained provider home");
+      expect(updates.filter(update => update.table === environmentLeases)).toEqual([expect.objectContaining({
+        values: expect.objectContaining({ metadata: { nativeHarnessBackup: expect.objectContaining({
+          normalizedSessionId: current.session.normalizedSessionId, sourceProviderLeaseId: "prior-sandbox",
+          authorizedProviderLeaseId: "sandbox-created" }) } }),
+      })]);
+    } else {
+      await expect(options.prepareExternalRunnerState()).rejects.toThrow("backup_without_reusable_lease");
+      expect(syncIn).not.toHaveBeenCalled();
+      expect(updates.filter(update => update.table === environmentLeases)).toEqual([]);
+    }
+  });
+
   it("archives failover evidence with an explicitly replaced provider session", async () => {
     const remoteCwd = join(isolatedStateDirectory, "remote");
     const remoteExecute = vi.fn(async (command: { command: string; args?: string[] }) => {