fix(native): restore plugin checkpoints for same-run retries (#17814)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native sessions retain provider state across execution attempts.
> - A same-run retry can receive a replacement plugin task lease.
> - The executor loaded the old task checkpoint only for crash recovery.
> - An ordinary retry could fail its local authority check before
reading that checkpoint.
> - This change restores the checkpoint through the existing authorized
lease transfer.

## Linked Issues or Issue Description

**What happened?**

A native provider turn ended without its completion result. Paperclip
scheduled an exact-checkpoint retry on the same heartbeat run. The retry
acquired a new plugin task lease, then failed with
`runner_state_identity_mismatch: authority_indeterminate`.

The checkpoint restore branch required a `resume_dead_runner` restart
claim. Ordinary same-run retries have no such claim. A controlled
fixture with a retained remote checkpoint and no local lifecycle receipt
reproduces the error. The original controller file contents were
unavailable, so that part of the incident mechanism remains inferred.

**Expected behavior**

Load the retained plugin checkpoint after the existing policy proves the
lease transfer is safe. Keep the original run, session, and saved
control journal. Reject an unverified predecessor or a changed
checkpoint identity.

**Steps to reproduce**

1. Start a native session through a plugin task environment.
2. End its provider turn without a completion result and retain its
checkpoint.
3. Retry the same run on a replacement task lease without a restart
claim.
4. With no local lifecycle receipt, observe the authority error before
the checkpoint read.

**Paperclip version or commit**

Reproduced against source `507dc70be26c191e372f09e58dc0965e73bf8f57`.

**Deployment mode**

Native Runner with a plugin task environment. The observed provider used
ACPX Claude. The controlled executor fixture uses the shared native
boundary.

Related work: #17439 concerns recovery after controller replacement.
This fix uses the existing checkpoint contract and does not depend on
that PR.

## What Changed

- Restore prior plugin state for an ordinary same-run attempt when its
lease changes.
- Keep the existing company, agent, workspace, retained-state, and
termination checks.
- Extend executor regressions for ordinary retries, mismatched
checkpoints, and missing stop proof.
- Exercise the same-run checkpoint read through the database-backed task
API.
- Document the recovery rule and verification recipes.

## Verification

- Failing before: the ordinary retry reproduces
`authority_indeterminate`. Both new denial cases also fail because the
old branch never reaches the transfer policy.
- Passing after: all six focused checkpoint cases pass. The saved
control journal stays byte-for-byte unchanged. Rejected cases do not
construct a provider backend.
- `pnpm -r typecheck` passes.
- Map and module-boundary checks pass.
- All 858 executor, database lease-authority, plugin task, and ownership
tests pass. `pnpm build` passes.
- The broad local `pnpm test:run` attempt did not complete. It hit a
20-second timeout in the existing historical workspace-recovery HTTP
Retry case. That exact case then passed in isolation on the unchanged
head. The broad local attempt was stopped after the full CI matrix
passed; it is not counted as a local pass.
- Greptile reviewed current head
`5bc11fa978ab87d3d9c5d575fa9666185aaafc2c` at 5/5 with no actionable
findings. The full current-head CI matrix passed on attempt 2: 55 checks
passed, no pending or failed checks, and two expected Storybook jobs
skipped.
- The first broader local attempt could not bind embedded PostgreSQL
inside the sandbox (`EPERM`). It was stopped and restarted with loopback
access. This was a setup failure, separate from the observed red
regression.
- The first CI attempt failed two existing checks: the process-adapter
signoff happy path exhausted its run wait budget, and the workspace
export scheduler did not reach `complete` within 20 seconds. Both tests
pass in isolated local runs on the unchanged head. Both failed shards
passed on their single rerun without code changes. The initial failures
remain recorded. Their timing cause is not proven by the later passes.
- Live acceptance at the fixed revision has not run. The hosted
workspace still serves the earlier release. No deployment or retry of
the original task is claimed.

## Risks

The new checkpoint read runs only for same-run plugin recovery with a
replacement lease. Missing termination proof, changed scope, or a
changed Runner identity still stops execution. The lease resolver owns
these checks. The fix does not replay tools or change retry budgets. A
turn that omitted its completion result still needs the existing bounded
continuation to finish useful work.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The runtime does not expose a more specific backend model ID
or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/execution-semantics.md

@@ -1302,6 +1302,14 @@ recording the proof and successor lineage. Retained provider files are not edite
 
 Bootstrap retries, exact-checkpoint resumes, and fresh replacement sessions share three total provider attempts, including the original attempt. Linked run IDs, controller restarts, and duplicate wakes do not reset this budget. Automatic attempts retain the 30-second delay. Replacement scheduling and predecessor lineage commit together, with one successor per predecessor and admission through the normal task locks, authorization, pause, approval, and budget gates.
 
+When a same-run retry acquires a replacement plugin task lease, the executor
+reloads the prior task's durable Runner checkpoint before checking local
+lifecycle evidence. This includes retries after a provider turn omitted its
+completion result. The existing lease-transfer policy must verify the company,
+agent, workspace, retained session, and termination of prior execution. The
+checkpoint must match the exact Runner, lease, run, session, turn, and item.
+A missing restart claim does not remove these checks or authorize a fresh session.
+
 A prepared remote workspace interrupted before the executor's first durable
 claim may resume its untouched bootstrap on the same run. This requires attempt
 and controller generation zero, no prior owner, and no process, provider session,

maps/agent-access-map/execution-environments.md

@@ -176,6 +176,15 @@ The task API does not transfer files or reveal guest filesystem paths.
 This source update does not qualify live agent access, warm retention, or a provider.
 See the [execution contract](../../doc/plugins/PLUGIN_RUNNER_EXECUTION.md).
 
+Same-run checkpoint recovery also uses the saved lease path after a turn omits
+its completion result. It requires verified prior termination, retained session
+state, and matching company, agent, workspace, and complete Runner identity.
+The [executor regression](../../server/src/services/native-runtime/native-session-executor.test.ts)
+checks checkpoint retrieval and identity/termination denials. The
+[lease authority fixture](../../server/src/__tests__/environment-task-authority.integration.test.ts)
+checks the same-run read through the database-backed task API. These are
+controlled regressions; live provider acceptance remains separate.
+
 ## `images`
 
 Capability ID: `execution-environments/images`.

maps/feature-map/execution-environments.md

@@ -91,6 +91,12 @@ The [fake-provider integration](../../server/src/__tests__/plugin-runner-task.in
 checks acquisition, SDK dispatch, scope denial, stop proof and deletion recovery.
 Its WebSocket is mocked; it does not qualify a live provider or a real Runner handshake.
 
+For a same-run retry on a replacement plugin task, verify that the retained
+checkpoint loads before provider startup. The prior execution must have a stop
+receipt, and the checkpoint must match the original session identity. Include a
+turn that ends without a completion result; a successful initial turn alone does
+not exercise this recovery path. Reject missing stop proof and changed identities.
+
 Automated: [execution target](../../server/src/__tests__/environment-execution-target.test.ts) and [custom images](../../server/src/__tests__/environment-custom-images-service.test.ts) cover contracts; [SSH live test](../../server/src/__tests__/environment-live-ssh.test.ts) requires its own external setup.
 
 The [environment runtime regression](../../server/src/__tests__/environment-runtime.test.ts)

server/src/__tests__/environment-task-authority.integration.test.ts

@@ -111,6 +111,12 @@ databaseSuite("environment task lease authority", () => {
     const retryLeaseId = randomUUID();
     await db.insert(environmentLeases).values({ id: retryLeaseId, companyId, environmentId, heartbeatRunId: thirdRunId,
       status: "active", provider: prior.provider, providerLeaseId: "retry-task", metadata: pin });
+    const retryIdentity = { ...checkpointIdentity, runId: thirdRunId, turnId: "turn", itemId: "item" };
+    await expect(resolvePluginTaskResume(db, { companyId, leaseId: retryLeaseId,
+      priorIdentity: retryIdentity })).resolves.toBe("third-task");
+    await expect(runtime.task({ companyId, leaseId: retryLeaseId, operation: {
+      kind: "read_state", identity: retryIdentity, resumeFrom: "third-task",
+    } })).resolves.toMatchObject({ kind: "state", state: { ...retryIdentity, lifecycle: "suspended" } });
     await expect(runtime.task({ companyId, leaseId: retryLeaseId, operation: { ...submit, projectIds: [], resumeFrom: "third-task" } })).resolves.toMatchObject({ kind: "accepted" });
     const [retry] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, retryLeaseId));
     await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),

server/src/services/native-runtime/native-session-executor.test.ts

@@ -11862,7 +11862,7 @@ describe("runnerd provider runtime wiring", () => {
     }
   });
 
-  it.each(["verified", "wrong-state", "no-stop-proof"] as const)("admits only an authorized same-run plugin crash checkpoint: %s", async mode => {
+  it.each(["verified", "wrong-state", "no-stop-proof", "disposition-retry", "disposition-wrong-state", "disposition-no-stop-proof"] as const)("admits only an authorized same-run plugin checkpoint: %s", async mode => {
     const current = scopedNativeExecution(`plugin-crash-${mode}`);
     const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
       runnerInstanceId: "plugin-runner", environmentLeaseId: "old-task-lease", turnId: "old-turn", itemId: "old-item" };
@@ -11871,13 +11871,13 @@ describe("runnerd provider runtime wiring", () => {
     const journal = JSON.stringify(durableControlPlaneState(identity));
     await writeFile(join(root, "control-plane/control-plane-state.json"), journal);
     const resume = vi.spyOn(environmentTaskRuntime, "resolvePluginTaskResume").mockImplementation(async () => {
-      if (mode === "no-stop-proof") throw new Error("Plugin Runner retained session is not authorized");
+      if (mode.endsWith("no-stop-proof")) throw new Error("Plugin Runner retained session is not authorized");
       return "prior-task";
     });
     const task = vi.fn(async (operation: any) => {
       environmentTaskOperationSchema.parse(operation);
       if (operation.kind === "read_state") return { kind: "state", taskId: "next-task", state: {
-        ...durableRunnerState(identity, "ready"), ...(mode === "wrong-state" ? { itemId: "other" } : {}),
+        ...durableRunnerState(identity, "ready"), ...(mode.endsWith("wrong-state") ? { itemId: "other" } : {}),
       } };
       return { kind: "description", launchPreparationVersion: 1, projectAuthCheckVersion: 1, taskId: "next-task",
         runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } };
@@ -11886,18 +11886,18 @@ describe("runnerd provider runtime wiring", () => {
     try {
       const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
         runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: "new-task-lease", runnerTask: task },
-        restartRecovery: { kind: "resume_dead_runner", runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2,
-          providerAttempt: 2, restartKind: "hard", recoveryRequestId: null } });
-      if (mode === "verified") {
+        ...(mode.startsWith("disposition-") ? {} : { restartRecovery: { kind: "resume_dead_runner" as const, runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2,
+          providerAttempt: 2, restartKind: "hard" as const, recoveryRequestId: null } }) });
+      if (mode === "verified" || mode === "disposition-retry") {
         await expect(backend).resolves.toBeDefined();
         expect(state.createBackend).toHaveBeenCalledOnce();
         expect(JSON.parse(await readFile(join(root, "plugin-runner-state.json"), "utf8"))).toMatchObject({ ...identity, lifecycle: "ready" });
       } else {
-        await expect(backend).rejects.toThrow(mode === "wrong-state" ? "checkpoint identity mismatch" : "not authorized");
+        await expect(backend).rejects.toThrow(mode.endsWith("wrong-state") ? "checkpoint identity mismatch" : "not authorized");
         expect(state.createBackend).not.toHaveBeenCalled();
       }
       expect(await readFile(join(root, "control-plane/control-plane-state.json"), "utf8")).toBe(journal);
-      if (mode === "no-stop-proof") expect(task).not.toHaveBeenCalled();
+      if (mode.endsWith("no-stop-proof")) expect(task).not.toHaveBeenCalled();
       else expect(task).toHaveBeenCalledWith({ kind: "read_state", identity, resumeFrom: "prior-task" });
     } finally { resume.mockRestore(); }
   });

server/src/services/native-runtime/native-session-executor.ts

@@ -4951,13 +4951,14 @@ async function migrateRunnerdStateRootForExecution(input: {
       throw new Error("runner_state_identity_mismatch: session_scope_mismatch");
     }
     const taskTarget = input.runnerExecutionTarget;
-    if (input.restartRecovery?.kind === "resume_dead_runner" && durableIdentityMatchesExecution(identity, input.execution)
-      && taskTarget?.kind === "remote" && taskTarget.transport === "sandbox" && taskTarget.runnerTask && taskTarget.leaseId) {
+    if (durableIdentityMatchesExecution(identity, input.execution)
+      && taskTarget?.kind === "remote" && taskTarget.transport === "sandbox" && taskTarget.runnerTask && taskTarget.leaseId
+      && (input.restartRecovery?.kind === "resume_dead_runner"
+        || (!input.restartRecovery && identity.environmentLeaseId !== taskTarget.leaseId))) {
       const resumeFrom = taskTarget.sandboxLeaseAcquisition?.outcome === "resumed" ? undefined : await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
         leaseId: taskTarget.leaseId, priorIdentity: identity });
       const result = environmentTaskResultSchema.parse(await taskTarget.runnerTask({ kind: "read_state", identity, resumeFrom }));
       if (result.kind !== "state") throw new Error("Plugin Runner checkpoint unavailable");
-      // The new lease and source termination receipt authorize this exact same-run recovery.
       savePluginRunnerCheckpoint(scoped, result.state, identity);
     }
     if (input.restartRecovery?.kind === "bootstrap_incomplete") {