fix(recovery): preserve native workspaces after control deadlines (#17788)

## Thinking Path

> - Paperclip lets people manage agents and their saved work.
> - Native runs can prepare a remote workspace before provider
execution.
> - A finalization deadline can stop a run outside its normal exception
path.
> - That handler marked the run failed without recording its prepared
files for recovery.
> - Retry then waited for a lease whose preservation had no durable
owner.
> - This pull request records the exact source in the deadline
transaction through the existing recovery service.
> - Existing containment and file recovery can then settle the source
before another executor starts.

## Linked Issues or Issue Description

**What happened?**

A native run failed at its control deadline with a prepared workspace
and no accepted result. Its environment lease remained active. The run
had no workspace preservation record. A saved Retry waited for remote
cleanup without a preservation operation to finish that cleanup.

**Expected behavior**

Record unfinished files when the deadline marks a native run failed. Use
the existing stop-and-retain worker. Keep accepted results, exact source
binding, and current task ownership authoritative.

**Steps to reproduce**

1. Persist a native run with a prepared workspace, an active lease, and
an expired control deadline.
2. Reconcile the deadline twice.
3. Inspect the source lease and the run's workspace recovery record.

**Paperclip version or commit**

Reproduced against `a67a18c171816b097f46eae78fc58c7e55c9c565`.

**Deployment mode**

Hosted native Runner with a remote sandbox. The regression uses real
local PostgreSQL and no provider mocks.

Related: Refs #17741 for committed Retry delivery. Refs #17073 for
remote process-exit verification. This change records file preservation
after a deadline; it does not change either policy.

## What Changed

- Call the existing native workspace preservation service in the
transaction that fails an abandoned run.
- Preserve files after reassignment without changing the current task
owner or state.
- Add real-database cases for prepared, finalized, mismatched,
reassigned, and accepted-result states. Repeated reconciliation retains
one source obligation and creates no successor.
- Document the deadline behavior and its existing access boundaries.

## Verification

- Red: on unchanged production code, the prepared-workspace case failed
because its recovery record was null. Finalized and mismatched controls
passed.
- Green: all 10 deadline reconciliation tests pass. The existing native
workspace recovery suite passes all 48 cases.
- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- Map checks, module-boundary checks, and `git diff --check` passed.
- Current-head CI passes on `23bf2aa4b50d2c4b74a9ee2ee47034ae542d25c2`:
all GitHub Actions CI lanes, including grouped/serialized tests, Runner
verification, typecheck, build, E2E, and Canary Dry Run, succeeded. Two
Storybook-only checks were skipped. Greptile is 5/5 with no findings.
Superagent Security Scan also passed. Entire Gates reports passing
checks/findings and fails only because no reviewer has approved trail
#10.
- The first server shard 11/12 attempt passed all 1,750 assertions but
reported an unhandled early process-exit rejection in the unchanged
workspace-exposure fixture. Its focused local suite passed (28 passed, 3
skipped); one CI rerun passed.
- Full local `pnpm test:run` was started, then interrupted after the
complete CI lanes passed to avoid duplicate work. No full local suite
pass is claimed. Local targeted tests, typecheck, and build passed.
- On the affected staging instance, the existing deployed preservation
helper registered the exact failed source. Normal Retry then drove
actual sandbox stop, file recovery, and a persisted successful recovery
receipt. A successor reached a separate remote process-fingerprint
failure before provider execution. The automatic deadline change is not
deployed; task execution recovery remains incomplete.
- Initial local setup attempts did not reach product assertions: skipped
dependency lifecycle scripts left native PostgreSQL library links
absent, and the restricted shell blocked localhost binding. The
package's link hydration and an authorized isolated listener resolved
those setup prerequisites. One new negative-control assertion also
required null-safe access; the final red run retained only the intended
behavioral failure.

## Risks

- File preservation must not certify provider termination. The existing
stop receipt, source binding, copyback, and admission checks remain in
force.
- Preservation failure rolls back the deadline transaction, leaving its
durable deadline eligible for another bounded attempt.
- Reassignment preserves old files without changing the new owner's
task. Accepted results retain their current finalizer.
- No schema, credential, provider, or polling change. Historical failed
runs still require scoped recovery; this patch does not replay them.

## Model Used

OpenAI Codex based on GPT-6, with reasoning, repository inspection, code
execution, and browser tools. The exact serving model ID and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suites; full
local suite interrupted after CI passed)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (CI lanes pass; external Entire
reviewer-approval gate remains pending)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/execution-semantics.md

@@ -1336,6 +1336,13 @@ does not reset the provider-attempt budget.
 
 Provider execution and control-plane finalization have different clocks. A healthy provider can think or execute a long tool without output. Once execution settles, recovery and finalization control steps have a 60-second deadline, checked on startup and every 15 seconds. With a healthy database and scheduler, an abandoned transition must be repaired or surfaced within 90 seconds. Terminal persistence must not wait on provider cleanup or publication; a late finalizer cannot change a reassigned or closed task or release another run's locks. Historical ambiguous runs are never automatically replayed after an upgrade.
 
+When that deadline fails a native run with a prepared workspace, the same
+transaction records the exact source's file-preservation obligation. Existing
+stop-and-retain recovery contains the provider and saves the files before
+ordinary Retry can continue. Finalized workspaces, mismatched sources and
+accepted results keep their existing settlement paths. Reassignment does not
+discard the stopped run's files or grant authority over the new assignee.
+
 Every continuation carries the triggering request, ordered user direction, interaction outcomes, completed work, and explicit history coverage. A delivered message remains part of the task's request after its connection or approval resolves. The original title is background; a completed Notion read does not satisfy a later Gmail request. Author and source-trust boundaries survive rendering into both native and legacy prompts. Missing required history must be fetched before dispatch rather than described as complete.
 
 ### Interrupted conversation continuation

maps/agent-access-map/recovery.md

@@ -4,6 +4,10 @@ Product map: [Recovery](../feature-map/recovery.md).
 
 Feature ID: `recovery`.
 
+Native finalization deadline reconciliation records unfinished workspace
+preservation through the existing server-owned recovery path. It grants no
+additional agent API, provider-stop authority or permission to replay work.
+
 Ordinary board Retry can reauthorize saved GitHub input for the same App after
 reconnect. It pins current runtime authority and preserves sender, repository,
 conversation and task checks. Pending input takes precedence over the older

maps/feature-map/recovery.md

@@ -33,6 +33,8 @@ Agent access: [companion](../agent-access-map/recovery.md).
   workspace link when the destination is disabled. Local PostgreSQL and real
   filesystem regressions cover native and legacy recovery and one-time delivery;
   hosted provider recovery remains separate acceptance.
+  Native finalization deadline failures also retain the exact prepared source
+  for the same stop-and-save workflow before Retry admits another executor.
 - `unknown-outcome`: `recovery/unknown-outcome` — uncertain external side effects require inspection and
   an explicit decision, not automatic replay.
 - `user-continuation`: `recovery/user-continuation` — saved user messages after a stop are reconsidered once

server/src/services/execution-control-reconciliation.test.ts

@@ -2,7 +2,7 @@ import { enableRunFailureReporting } from "../__tests__/helpers/run-failure-repo
 import { randomUUID } from "node:crypto";
 import { eq } from "drizzle-orm";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
-import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, issueRecoveryActions, issueThreadInteractions, nativeRunResults, statusDecisions, workAssessments } from "@paperclipai/db";
+import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, issueRecoveryActions, issueThreadInteractions, nativeRunFinalizations, nativeRunResults, statusDecisions, workAssessments } from "@paperclipai/db";
 import {
   getEmbeddedPostgresTestSupport,
   startEmbeddedPostgresTestDatabase,
@@ -153,6 +153,65 @@ describeEmbeddedPostgres("reconcileAbandonedExecutionControl reports a genuine f
     expect(result.surfaced).toBe(1);
     expect(mockCaptureRunFailure.mock.calls.slice(captureCallsBefore)).toHaveLength(0);
   });
+
+  it.each(["prepared", "finalized", "mismatched", "reassigned", "accepted"] as const)("preserves only an exact unfinished native workspace after its deadline (%s)", async state => {
+    const f = await seedAbandonedRunFixture();
+    const [environment] = await db.insert(environments).values({ name: `Deadline ${f.runId}`, driver: "sandbox" }).returning();
+    const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
+      heartbeatRunId: f.runId, environmentId: environment.id, provider: "daytona", providerLeaseId: randomUUID(),
+      status: "active", leasePolicy: "reuse_by_environment", metadata: { remoteCwd: "/work" },
+    }).returning();
+    await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId, runnerProfileJson: {
+      nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: state === "finalized" ? state : "prepared",
+        descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
+        workspaceId: f.runId, leaseId: lease.id, providerLeaseId: state === "mismatched" ? randomUUID() : lease.providerLeaseId,
+        remoteCwd: "/work", resourceDisposition: "keep_running" },
+    } }).where(eq(heartbeatRuns.id, f.runId));
+    await db.insert(nativeRunFinalizations).values({ runId: f.runId, companyId: f.companyId, issueId: f.issueId,
+      phase: "observed", leaseOwner: "abandoned-controller", controlDeadlineAt: new Date(Date.now() - 60_000) });
+    if (state === "reassigned") {
+      const nextAgentId = randomUUID();
+      await db.insert(agents).values({ id: nextAgentId, companyId: f.companyId, name: "New owner", adapterType: "codex_local" });
+      await db.update(issues).set({ assigneeAgentId: nextAgentId, executionRunId: null, checkoutRunId: null })
+        .where(eq(issues.id, f.issueId));
+    }
+    if (state === "accepted") {
+      const contractId = randomUUID(), resultId = randomUUID();
+      await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId,
+        revision: 1, schemaVersion: "1", policyVersion: "test", risk: "low", completionAuthority: "agent_claim_policy",
+        incompleteCriteriaPolicy: "block", contractJson: {}, canonicalSha256: randomUUID(), createdByActorType: "agent", createdByActorId: f.agentId });
+      await db.update(heartbeatRuns).set({ completionContractId: contractId }).where(eq(heartbeatRuns.id, f.runId));
+      await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId: f.runId,
+        completionContractId: contractId, serverFingerprint: randomUUID(), schemaStatus: "valid", resultJson: {}, canonicalSha256: randomUUID() });
+      await db.update(nativeRunFinalizations).set({ resultId }).where(eq(nativeRunFinalizations.runId, f.runId));
+    }
+
+    await reconcileAbandonedExecutionControl(db);
+    await reconcileAbandonedExecutionControl(db);
+
+    const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
+    const [retained] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id));
+    expect(run).toMatchObject(state === "accepted" ? { status: "running", errorCode: null }
+      : { status: "failed", errorCode: "execution_finalization_deadline_exceeded" });
+    if (state === "prepared" || state === "reassigned") {
+      expect(run.resultJson).toMatchObject({ workspaceRestoreRecovery: { leaseIds: [lease.id] } });
+      expect(retained).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure",
+        metadata: { sandboxStopAndRetain: { runId: f.runId, providerLeaseId: lease.providerLeaseId } } });
+      const recovery = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
+      expect(recovery.filter(action => action.evidence?.workspaceRestoreRecovery)).toMatchObject([
+        { evidence: { runId: f.runId, workspaceRestoreRecovery: { leaseIds: [lease.id] } } },
+      ]);
+    } else {
+      expect(run.resultJson ?? {}).not.toHaveProperty("workspaceRestoreRecovery");
+      expect(retained).toMatchObject({ status: "active", leasePolicy: "reuse_by_environment" });
+    }
+    const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
+    if (state === "reassigned") {
+      expect(task.assigneeAgentId).not.toBe(f.agentId);
+      expect(task).toMatchObject({ status: "in_progress", executionRunId: null, checkoutRunId: null });
+    }
+    expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
+  });
 });
 
 describeEmbeddedPostgres("native review execution reconciliation admission", () => {

server/src/services/execution-control-reconciliation.ts

@@ -14,6 +14,7 @@ import { issueRecoveryActionService } from "./issue-recovery-actions.js";
 import { hasLiveLegacyController } from "./legacy-controller-lease.js";
 import { reportRunFailure } from "./run-failure-report.js";
 import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-runtime/native-review-participant.js";
+import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js";
 
 /** Only newly recorded control deadlines are eligible. Upgrades never replay ambiguous historical runs. */
 export async function reconcileAbandonedExecutionControl(
@@ -233,8 +234,10 @@ export async function reconcileAbandonedExecutionControl(
             ["done", "cancelled"].includes(task.status) ||
             (task.executionRunId && task.executionRunId !== run.id) ||
             (task.checkoutRunId && task.checkoutRunId !== run.id)
-          )
+          ) {
+            if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun);
             return true;
+          }
           await tx
             .update(issues)
             .set({ executionRunId: null, checkoutRunId: null, updatedAt: now })
@@ -261,6 +264,7 @@ export async function reconcileAbandonedExecutionControl(
             maxAttempts: 3,
             supersedeOnIdentityChange: true,
           });
+          if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun);
           return true;
         });
         if (repaired) surfaced += 1;