fix(recovery): preserve unfinished remote exports before Retry (#18678)

## Thinking Path

> - Paperclip manages agent work and preserves each task's saved files.
> - A completed native result can still need workspace export.
> - Remote workspaces now use a different saved descriptor from legacy
copyback.
> - The execution hold checked only the legacy descriptor.
> - Task Retry could admit another run before the original export
finished.
> - A later failure could then supersede the original recovery record.
> - This change keeps unfinished remote exports fenced and permits
explicit historical file recovery.
> - Recovery keeps human review and newer execution authority intact.

## Linked Issues or Issue Description

**What happened?**

A native run completed successfully, but its remote workspace export
failed. The recovery action used
`native_workspace_sync_out_retry_exhausted`. Task Retry created a second
task run while the original files still needed preservation. The newer
failure then superseded the original action. A board file-recovery
request could queue that cancelled record, but the worker excluded it
from discovery.

**Expected behavior**

Keep Retry behind the unfinished export. Save the exact source and
finalize the existing accepted result once. Preserve pending human
review. An explicit request for a superseded source must save its files
without applying an old result over newer work.

**Steps to reproduce**

1. Save an accepted successful native result with a prepared
`remoteWorkspace` descriptor.
2. Fail its final export and retain its exact sandbox lease.
3. Select Task Retry before preservation completes.
4. Observe another task run before the original workspace is saved.
5. Supersede the original action and request file recovery for its exact
source. Observe queued work that the worker does not select.

**Paperclip version**

Observed at `1f8f2e195f4897cd1c237bdc068de95780ab0c3c`. The regression
is present on current master before this patch.

**Deployment mode**

Authenticated hosted instance with native Paperclip Runner and remote
sandbox workspaces.

Related: #18403 introduced the remote workspace path. #18644 reduces
export scratch disk use. #17916 covers a separate legacy warm-copyback
policy case. This change repairs remote export admission and explicit
historical preservation.

## What Changed

- Include prepared remote descriptors in the existing native export
hold.
- Retain remote source identity in the blocker and continuation read
projections.
- Let the existing worker select a cancelled export-repair record only
after an explicit board file-recovery request. Keep the current action
and workspace binding unchanged.
- Add real PostgreSQL, HTTP, filesystem and Git regressions. Update
execution semantics and recovery maps.

## Verification

- Red before the admission fix: HTTP Retry creates two task runs instead
of one.
- Red before the historical-source fix: the explicit recovery stays
scheduled and saves no snapshot.
- All 588 focused tests pass across the workspace recovery, recovery
source, explicit continuation and native export retry suites. The three
new integration cases verify duplicate request identity, original result
finalization, restored bytes and Git HEAD, pending human review,
current-writer exclusion, and preservation without older-result
commitment over newer work.
- `pnpm -r typecheck`, `pnpm build`, map validation and `git diff
--check` pass.
- Full hosted CI passes on `c598718379fa939ec539e8e91a15a94d90321b59`,
including all server, workspace, serialized, Runner and browser lanes
([run](https://github.com/paperclipai/paperclip/actions/runs/41529739604)).
Greptile reviewed that commit at 5/5 with no findings; there are no
unresolved review threads. Paperclip Review also published a complete
5/5 assessment for that exact commit with zero findings. Its check
conclusion is advisory/neutral, titled `5/5`. The first attempt lacked a
structured assessment; one normal completion request recovered the saved
analysis and published the validated assessment.
- The local `pnpm test:run` passed all six chat partitions, then was
stopped during the general server group after equivalent full CI
completed. Its partial log is retained; it is not reported as a
completed local full-suite pass.
- The test provider boundary uses real temporary files and the
production archive path. It simulates provider resume/stop calls. The
deployed fix and original hosted task recovery remain unverified until
merge and rollout.
- The older local Node 24.11.0 runtime failed to load the installed tsx
worker. Node 26.4.0 passed the isolated worker check. Required
repository checks use an isolated Node 24.22.2 runtime. The earlier
failure logs remain recorded.

## Risks

- Recognizing another unfinished export can keep previously admitted
work waiting. This is intentional until persistence is verified.
- Explicit historical recovery can resume the exact stopped sandbox to
read files. It retains source, allocation, ownership, writer and
bounded-attempt checks. It starts no provider turn.
- A newer run still prevents commitment of an older accepted result.
This change does not infer that a cancelled run had no external effects.
- No schema, credential, deployment or fleet configuration changes.

## Model Used

OpenAI Codex, GPT-6 family. The runtime does not expose the exact model
ID or context-window size. Reasoning, repository inspection, tool use
and code execution assisted this change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/execution-blocker.ts

@@ -16,7 +16,7 @@ import { hasVerifiedWorkspaceRepair, workspaceRecoveryDisplay } from "./workspac
 
 const blockerContextKeys = ["issueId", "taskId", "source", "connectionIntentResolution",
   "interactionKind", "interactionStatus", "mutation", "interactionId"] as const;
-const blockerProfileKeys = ["adapterDispatch", "nativeExecutionInput", "nativeWorkspaceSync"] as const;
+const blockerProfileKeys = ["adapterDispatch", "nativeExecutionInput", "nativeWorkspaceSync", "remoteWorkspace"] as const;
 const blockerContextColumn = sql<Record<string, unknown> | null>`case
   when ${heartbeatRuns.contextSnapshot} is null then null else (
     select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb)

server/src/services/explicit-native-continuation.ts

@@ -52,7 +52,7 @@ const preflightProfile = sql<Run["runnerProfileJson"]>`case
       then jsonb_build_object('adapterType', field.value->'adapterType') else field.value end), '{}'::jsonb)
     from jsonb_each(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object'
       then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) as field(key, value)
-    where field.key in ('adapterDispatch', 'nativeWorkspaceSync')
+    where field.key in ('adapterDispatch', 'nativeWorkspaceSync', 'remoteWorkspace')
   ) end`;
 const continuationRunFields = {
   id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,

server/src/services/native-runtime/native-workspace-reconstruction.ts

@@ -36,7 +36,8 @@ export function unreconciledNativeExportPredicate() {
       and ${heartbeatRuns.nativeIssueId} = ${issueRecoveryActions.sourceIssueId}
       and ${heartbeatRuns.id}::text = ${issueRecoveryActions.evidence}->>'runId'
       and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed'
-      and ${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared')`,
+      and (${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared'
+        or ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'state' = 'prepared'))`,
   );
 }
 

server/src/services/workspace-recovery-source.ts

@@ -25,6 +25,10 @@ export class WorkspaceSourceInspectionLimit extends Error {}
 
 export function workspaceRecoverySourceEligible(includeCompleted = false) {
   const actionEligible = sql`(${issueRecoveryActions.status} in ('active', 'escalated') or
+    (${issueRecoveryActions.status} = 'cancelled'
+      and ${inArray(issueRecoveryActions.cause, workspaceExportCauses)}
+      and jsonb_typeof(${issueRecoveryActions.evidence}#>'{workspaceRecovery,requestIds}') = 'array'
+      and ${issueRecoveryActions.evidence}#>'{workspaceRecovery,requestIds}' <> '[]'::jsonb) or
     (${issueRecoveryActions.status} = 'resolved' and (
       (${issueRecoveryActions.evidence}#>>'{automaticRecovery,policy}' = 'preserve_without_replay_v1'
         and ${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked'

server/src/services/workspace-recovery.test.ts

@@ -49,6 +49,10 @@ import { and, eq, inArray, sql } from "drizzle-orm";
 import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
 import { agents, approvals, budgetPolicies, costEvents, completionContracts, issueApprovals, issueTreeHolds, issueTreeHoldMembers, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, workspaceOperations } from "@paperclipai/db";
 import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
+import { finalizeNativeRun } from "./native-runtime/native-run-finalizer.js";
+import { CONTROL_PLANE_CONFORMANCE_RESULT, CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../vendor/paperclip-runner/testing.js";
+import { issueThreadInteractions } from "@paperclipai/db";
+import { issueRecoveryActionService } from "./issue-recovery-actions.js";
 import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
 import type { EnvironmentLease } from "@paperclipai/shared";
 import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
@@ -190,6 +194,149 @@ describe("automatic retained workspace recovery", () => {
   const action = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!;
   const run = async (id: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]!;
 
+  function boardRecoveryApp() {
+    const app = express();
+    app.use(express.json());
+    app.use((req, _res, next) => { req.actor = { type: "board", source: "local_implicit", userId: "local-board" }; next(); });
+    app.use("/api", issueRoutes(db, {} as Parameters<typeof issueRoutes>[1]));
+    app.use(errorHandler);
+    return app;
+  }
+
+  async function acceptedRemoteExport() {
+    const f = await seed("native", false);
+    const source = await run(f.runId), workspaceId = randomUUID();
+    const remoteCwd = await realpath(f.remote);
+    const git = (args: string[], cwd = remoteCwd) => new Promise<string>((resolve, reject) =>
+      execFile("git", args, { cwd }, (error, stdout) => error ? reject(error) : resolve(stdout.trim())));
+    await git(["init"]);
+    await git(["add", "."]);
+    await git(["-c", "user.name=Recovery fixture", "-c", "user.email=recovery@example.invalid", "commit", "-m", "Completed work"]);
+    const commit = await git(["rev-parse", "HEAD"]);
+    targets.get(f.lease.id)!.remoteCwd = remoteCwd;
+    f.runtime.resumeRunLease.mockImplementation(async () => ({ providerLeaseId: f.providerLeaseId, metadata: { remoteCwd } }));
+    const authority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null,
+      source: { kind: "empty" }, writerRunId: f.runId, dirty: true,
+      location: { environmentId: f.lease.environmentId!, leaseId: f.lease.id, providerLeaseId: f.providerLeaseId, remoteCwd } };
+    await db.insert(executionWorkspaces).values({ id: workspaceId, companyId: f.companyId, sourceIssueId: f.issueId,
+      name: "Completed remote work", cwd: remoteCwd, providerType: "cloud_sandbox", mode: "isolated_workspace",
+      strategyType: "project_primary", status: "active", metadata: { remoteWorkspace: authority } });
+    await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, f.issueId));
+    const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
+    await db.update(environmentLeases).set({ executionWorkspaceId: workspaceId, leasePolicy: "reuse_by_environment",
+      metadata: { ...lease.metadata, remoteCwd, agentId: source.agentId } }).where(eq(environmentLeases.id, lease.id));
+    const contractId = randomUUID(), resultId = randomUUID();
+    await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId,
+      revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "low",
+      completionAuthority: "agent_claim_policy", incompleteCriteriaPolicy: "preserve_non_terminal",
+      contractJson: { revision: CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim.contractRevision,
+        objective: "Complete the work", criteria: [{ id: "objective", requirement: "Complete the task" }] },
+      canonicalSha256: "contract", createdByActorType: "system", createdByActorId: "test" });
+    await db.update(heartbeatRuns).set({ completionContractId: contractId, resultJson: {}, runnerProfileJson: {
+      adapterDispatch: { adapterType: "paperclip_runner" },
+      remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId: lease.id,
+        providerLeaseId: f.providerLeaseId, remoteCwd, epoch: 1, state: "prepared", snapshotId: null },
+    } }).where(eq(heartbeatRuns.id, f.runId));
+    await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId: f.runId,
+      completionContractId: contractId, schemaStatus: "accepted", serverFingerprint: "result", canonicalSha256: "result",
+      resultJson: { terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, result: CONTROL_PLANE_CONFORMANCE_RESULT } });
+    await db.update(nativeRunFinalizations).set({ resultId, failureCode: "native_workspace_sync_out_retry_exhausted" })
+      .where(eq(nativeRunFinalizations.runId, f.runId));
+    await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted", evidence: { runId: f.runId } })
+      .where(eq(issueRecoveryActions.id, f.actionId));
+    await adoptSystemRecovery(db, f);
+    return { ...f, source, workspaceId, resultId, git, commit };
+  }
+
+  it.each([false, true])("keeps Retry behind an accepted remote export and resumes only its saved result (review pending: %s)", async reviewPending => {
+    const f = await acceptedRemoteExport();
+    const { source, workspaceId, resultId, git, commit } = f;
+    const interactionId = randomUUID();
+    if (reviewPending) await db.insert(issueThreadInteractions).values({ id: interactionId, companyId: f.companyId,
+      issueId: f.issueId, sourceRunId: f.runId, createdByAgentId: source.agentId, kind: "request_confirmation",
+      title: "Review saved work", effectiveResolverPolicy: "human_only",
+      payload: { version: 1, prompt: "Review before publishing", rejectRequiresReason: true,
+        target: { type: "custom", key: "layout_review", label: "Review saved work" } } });
+    await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" });
+    const app = boardRecoveryApp();
+    const intent = { observedRunId: f.runId, idempotencyKey: randomUUID() };
+    const retry = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body;
+    expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body.requestId).toBe(retry.requestId);
+    await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!);
+    expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
+      sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text,
+        ${heartbeatRuns.contextSnapshot}->>'issueId') = ${f.issueId}`))).toHaveLength(1);
+    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: true });
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {
+      await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!);
+    }) ?? []);
+    expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
+    const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
+    const saved = readRemoteWorkspaceAuthority(workspace.metadata)!;
+    const restored = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId, snapshotId: saved.snapshotId! });
+    try {
+      expect(await readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("finished work\n");
+      expect(await git(["rev-parse", "HEAD"], restored.root)).toBe(commit);
+    }
+    finally { await restored.cleanup(); }
+    const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
+    expect(coordinator).toMatchObject({ phase: "workspace_finalizing", resultId });
+    expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!)))[0])
+      .toMatchObject({ status: "coalesced", runId: f.runId, payload: { taskRetry: { strategy: "finalize_result" } } });
+    expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(2);
+    await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
+    expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
+      .toMatchObject({ status: reviewPending ? "in_review" : "done", executionRunId: null, checkoutRunId: null });
+    expect((await run(f.runId)).status).toBe("succeeded");
+    if (reviewPending) expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0])
+      .toMatchObject({ status: "pending", result: null, resolvedAt: null });
+  }, 30_000);
+
+  it("preserves an explicitly requested superseded remote export without applying its result over newer work", async () => {
+    const f = await acceptedRemoteExport();
+    const [later] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId,
+      issueId: f.issueId, status: "cancelled", runtimeMode: "legacy", finishedAt: new Date(),
+      contextSnapshot: { issueId: f.issueId }, createdAt: new Date(f.source.createdAt.getTime() + 1_000) }).returning();
+    const [currentWorkspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId,
+      sourceIssueId: f.issueId, name: "Newer workspace", cwd: path.join(root, later.id), providerType: "cloud_sandbox",
+      mode: "isolated_workspace", strategyType: "project_primary", status: "active" }).returning();
+    await db.update(issues).set({ executionWorkspaceId: currentWorkspace.id }).where(eq(issues.id, f.issueId));
+    const laterAction = await issueRecoveryActionService(db).upsertSourceScoped({ companyId: f.companyId,
+      sourceIssueId: f.issueId, kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation",
+      fingerprint: later.id, evidence: { runId: later.id }, nextAction: "Inspect the stopped work",
+      ownerType: "board", supersedeOnIdentityChange: true });
+    expect((await action(f.actionId)).status).toBe("cancelled");
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []);
+    expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
+    const app = boardRecoveryApp();
+    const recoveryRequest = { actionId: f.actionId, runId: f.runId, idempotencyKey: randomUUID() };
+    await request(app).post(`/api/issues/${f.issueId}/recovery-actions/recover-workspace`).send(recoveryRequest).expect(202);
+    const admitted = await action(f.actionId);
+    await request(app).post(`/api/issues/${f.issueId}/recovery-actions/recover-workspace`).send(recoveryRequest).expect(202);
+    expect(await action(f.actionId)).toEqual(admitted);
+    await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, later.id));
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []);
+    expect(readSystemRecovery((await action(f.actionId)).evidence)?.failure?.code).toBe("workspace_writer_active");
+    expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
+    await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, later.id));
+    await makeRetryDue(f.actionId);
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []);
+    expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
+    const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId));
+    const saved = readRemoteWorkspaceAuthority(workspace.metadata)!;
+    const restored = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId: f.workspaceId, snapshotId: saved.snapshotId! });
+    try {
+      expect(await f.git(["rev-parse", "HEAD"], restored.root)).toBe(f.commit);
+      expect(await readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("finished work\n");
+    } finally { await restored.cleanup(); }
+    expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
+      .toMatchObject({ status: "blocked", executionWorkspaceId: currentWorkspace.id });
+    expect(await action(laterAction.id)).toEqual(expect.objectContaining({ status: "active", evidence: { runId: later.id } }));
+    expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
+      .toMatchObject({ phase: "terminal_failure", resultId: f.resultId });
+    expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]);
+  }, 30_000);
+
   it.each([
     { status: "failed", runtimeMode: "native", uploadFailure: "none" },
     { status: "cancelled", runtimeMode: "native", uploadFailure: "none" },