fix(recovery): preserve native workspaces after control deadlines (#17788)

## Thinking Path

> - Paperclip lets people manage agents and their saved work.
> - Native runs can prepare a remote workspace before provider
execution.
> - A finalization deadline can stop a run outside its normal exception
path.
> - That handler marked the run failed without recording its prepared
files for recovery.
> - Retry then waited for a lease whose preservation had no durable
owner.
> - This pull request records the exact source in the deadline
transaction through the existing recovery service.
> - Existing containment and file recovery can then settle the source
before another executor starts.

## Linked Issues or Issue Description

**What happened?**

A native run failed at its control deadline with a prepared workspace
and no accepted result. Its environment lease remained active. The run
had no workspace preservation record. A saved Retry waited for remote
cleanup without a preservation operation to finish that cleanup.

**Expected behavior**

Record unfinished files when the deadline marks a native run failed. Use
the existing stop-and-retain worker. Keep accepted results, exact source
binding, and current task ownership authoritative.

**Steps to reproduce**

1. Persist a native run with a prepared workspace, an active lease, and
an expired control deadline.
2. Reconcile the deadline twice.
3. Inspect the source lease and the run's workspace recovery record.

**Paperclip version or commit**

Reproduced against `a67a18c171816b097f46eae78fc58c7e55c9c565`.

**Deployment mode**

Hosted native Runner with a remote sandbox. The regression uses real
local PostgreSQL and no provider mocks.

Related: Refs #17741 for committed Retry delivery. Refs #17073 for
remote process-exit verification. This change records file preservation
after a deadline; it does not change either policy.

## What Changed

- Call the existing native workspace preservation service in the
transaction that fails an abandoned run.
- Preserve files after reassignment without changing the current task
owner or state.
- Add real-database cases for prepared, finalized, mismatched,
reassigned, and accepted-result states. Repeated reconciliation retains
one source obligation and creates no successor.
- Document the deadline behavior and its existing access boundaries.

## Verification

- Red: on unchanged production code, the prepared-workspace case failed
because its recovery record was null. Finalized and mismatched controls
passed.
- Green: all 10 deadline reconciliation tests pass. The existing native
workspace recovery suite passes all 48 cases.
- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- Map checks, module-boundary checks, and `git diff --check` passed.
- Current-head CI passes on `23bf2aa4b50d2c4b74a9ee2ee47034ae542d25c2`:
all GitHub Actions CI lanes, including grouped/serialized tests, Runner
verification, typecheck, build, E2E, and Canary Dry Run, succeeded. Two
Storybook-only checks were skipped. Greptile is 5/5 with no findings.
Superagent Security Scan also passed. Entire Gates reports passing
checks/findings and fails only because no reviewer has approved trail
#10.
- The first server shard 11/12 attempt passed all 1,750 assertions but
reported an unhandled early process-exit rejection in the unchanged
workspace-exposure fixture. Its focused local suite passed (28 passed, 3
skipped); one CI rerun passed.
- Full local `pnpm test:run` was started, then interrupted after the
complete CI lanes passed to avoid duplicate work. No full local suite
pass is claimed. Local targeted tests, typecheck, and build passed.
- On the affected staging instance, the existing deployed preservation
helper registered the exact failed source. Normal Retry then drove
actual sandbox stop, file recovery, and a persisted successful recovery
receipt. A successor reached a separate remote process-fingerprint
failure before provider execution. The automatic deadline change is not
deployed; task execution recovery remains incomplete.
- Initial local setup attempts did not reach product assertions: skipped
dependency lifecycle scripts left native PostgreSQL library links
absent, and the restricted shell blocked localhost binding. The
package's link hydration and an authorized isolated listener resolved
those setup prerequisites. One new negative-control assertion also
required null-safe access; the final red run retained only the intended
behavioral failure.

## Risks

- File preservation must not certify provider termination. The existing
stop receipt, source binding, copyback, and admission checks remain in
force.
- Preservation failure rolls back the deadline transaction, leaving its
durable deadline eligible for another bounded attempt.
- Reassignment preserves old files without changing the new owner's
task. Accepted results retain their current finalizer.
- No schema, credential, provider, or polling change. Historical failed
runs still require scoped recovery; this patch does not replay them.

## Model Used

OpenAI Codex based on GPT-6, with reasoning, repository inspection, code
execution, and browser tools. The exact serving model ID and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suites; full
local suite interrupted after CI passed)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (CI lanes pass; external Entire
reviewer-approval gate remains pending)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/execution-control-reconciliation.test.ts

@@ -2,7 +2,7 @@ import { enableRunFailureReporting } from "../__tests__/helpers/run-failure-repo
 import { randomUUID } from "node:crypto";
 import { eq } from "drizzle-orm";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
-import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, issueRecoveryActions, issueThreadInteractions, nativeRunResults, statusDecisions, workAssessments } from "@paperclipai/db";
+import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, issueRecoveryActions, issueThreadInteractions, nativeRunFinalizations, nativeRunResults, statusDecisions, workAssessments } from "@paperclipai/db";
 import {
   getEmbeddedPostgresTestSupport,
   startEmbeddedPostgresTestDatabase,
@@ -153,6 +153,65 @@ describeEmbeddedPostgres("reconcileAbandonedExecutionControl reports a genuine f
     expect(result.surfaced).toBe(1);
     expect(mockCaptureRunFailure.mock.calls.slice(captureCallsBefore)).toHaveLength(0);
   });
+
+  it.each(["prepared", "finalized", "mismatched", "reassigned", "accepted"] as const)("preserves only an exact unfinished native workspace after its deadline (%s)", async state => {
+    const f = await seedAbandonedRunFixture();
+    const [environment] = await db.insert(environments).values({ name: `Deadline ${f.runId}`, driver: "sandbox" }).returning();
+    const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
+      heartbeatRunId: f.runId, environmentId: environment.id, provider: "daytona", providerLeaseId: randomUUID(),
+      status: "active", leasePolicy: "reuse_by_environment", metadata: { remoteCwd: "/work" },
+    }).returning();
+    await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId, runnerProfileJson: {
+      nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: state === "finalized" ? state : "prepared",
+        descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
+        workspaceId: f.runId, leaseId: lease.id, providerLeaseId: state === "mismatched" ? randomUUID() : lease.providerLeaseId,
+        remoteCwd: "/work", resourceDisposition: "keep_running" },
+    } }).where(eq(heartbeatRuns.id, f.runId));
+    await db.insert(nativeRunFinalizations).values({ runId: f.runId, companyId: f.companyId, issueId: f.issueId,
+      phase: "observed", leaseOwner: "abandoned-controller", controlDeadlineAt: new Date(Date.now() - 60_000) });
+    if (state === "reassigned") {
+      const nextAgentId = randomUUID();
+      await db.insert(agents).values({ id: nextAgentId, companyId: f.companyId, name: "New owner", adapterType: "codex_local" });
+      await db.update(issues).set({ assigneeAgentId: nextAgentId, executionRunId: null, checkoutRunId: null })
+        .where(eq(issues.id, f.issueId));
+    }
+    if (state === "accepted") {
+      const contractId = randomUUID(), resultId = randomUUID();
+      await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId,
+        revision: 1, schemaVersion: "1", policyVersion: "test", risk: "low", completionAuthority: "agent_claim_policy",
+        incompleteCriteriaPolicy: "block", contractJson: {}, canonicalSha256: randomUUID(), createdByActorType: "agent", createdByActorId: f.agentId });
+      await db.update(heartbeatRuns).set({ completionContractId: contractId }).where(eq(heartbeatRuns.id, f.runId));
+      await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId: f.runId,
+        completionContractId: contractId, serverFingerprint: randomUUID(), schemaStatus: "valid", resultJson: {}, canonicalSha256: randomUUID() });
+      await db.update(nativeRunFinalizations).set({ resultId }).where(eq(nativeRunFinalizations.runId, f.runId));
+    }
+
+    await reconcileAbandonedExecutionControl(db);
+    await reconcileAbandonedExecutionControl(db);
+
+    const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
+    const [retained] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id));
+    expect(run).toMatchObject(state === "accepted" ? { status: "running", errorCode: null }
+      : { status: "failed", errorCode: "execution_finalization_deadline_exceeded" });
+    if (state === "prepared" || state === "reassigned") {
+      expect(run.resultJson).toMatchObject({ workspaceRestoreRecovery: { leaseIds: [lease.id] } });
+      expect(retained).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure",
+        metadata: { sandboxStopAndRetain: { runId: f.runId, providerLeaseId: lease.providerLeaseId } } });
+      const recovery = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
+      expect(recovery.filter(action => action.evidence?.workspaceRestoreRecovery)).toMatchObject([
+        { evidence: { runId: f.runId, workspaceRestoreRecovery: { leaseIds: [lease.id] } } },
+      ]);
+    } else {
+      expect(run.resultJson ?? {}).not.toHaveProperty("workspaceRestoreRecovery");
+      expect(retained).toMatchObject({ status: "active", leasePolicy: "reuse_by_environment" });
+    }
+    const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
+    if (state === "reassigned") {
+      expect(task.assigneeAgentId).not.toBe(f.agentId);
+      expect(task).toMatchObject({ status: "in_progress", executionRunId: null, checkoutRunId: null });
+    }
+    expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
+  });
 });
 
 describeEmbeddedPostgres("native review execution reconciliation admission", () => {

server/src/services/execution-control-reconciliation.ts

@@ -14,6 +14,7 @@ import { issueRecoveryActionService } from "./issue-recovery-actions.js";
 import { hasLiveLegacyController } from "./legacy-controller-lease.js";
 import { reportRunFailure } from "./run-failure-report.js";
 import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-runtime/native-review-participant.js";
+import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js";
 
 /** Only newly recorded control deadlines are eligible. Upgrades never replay ambiguous historical runs. */
 export async function reconcileAbandonedExecutionControl(
@@ -233,8 +234,10 @@ export async function reconcileAbandonedExecutionControl(
             ["done", "cancelled"].includes(task.status) ||
             (task.executionRunId && task.executionRunId !== run.id) ||
             (task.checkoutRunId && task.checkoutRunId !== run.id)
-          )
+          ) {
+            if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun);
             return true;
+          }
           await tx
             .update(issues)
             .set({ executionRunId: null, checkoutRunId: null, updatedAt: now })
@@ -261,6 +264,7 @@ export async function reconcileAbandonedExecutionControl(
             maxAttempts: 3,
             supersedeOnIdentityChange: true,
           });
+          if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun);
           return true;
         });
         if (repaired) surfaced += 1;