fix(recovery): continue saved input after verified workspace copy-back (#19422)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution preserves accepted results and task files after a
failure.
> - A later user message can request a new turn after the old execution
stops.
> - File recovery can finish by copying files back or by retaining a
verified archive.
> - The retained-result gate accepted the archive proof but missed a
verified copy-back receipt.
> - This change accepts that existing receipt for terminal failures and
preserves execution authority and prior results.

## Linked Issues or Issue Description

**What happened?**
A task keeps a user message queued with `finalizing_saved_result` after
its old native coordinator reached `terminal_failure`. The accepted
result and a successful, exact `workspace_recovery` copy-back receipt
are saved. The gate only checks the alternate
`workspace_recovery_snapshot` receipt. An older failed result can also
keep a newer failed turn's saved input blocked.

**Expected behavior**
A new user message or deliberate task Retry can continue from verified
recovered files after the old execution stops. Preserve the old result
and unknown action outcomes. Keep an active finalizer or missing stop
evidence blocked.

**Steps to reproduce**
1. Retain an accepted result when a native run reaches terminal failure.
2. Complete host copy-back with the exact run, allocation, descriptor
and final file digest.
3. Save a new user message or request task Retry.
4. Observe that admission still reports `finalizing_saved_result`.

**Paperclip version or commit**
Reproduced against `4d49a3128613b282fb344272ed7ca15f98dea2ac`.

**Deployment mode**
Authenticated native execution with retained workspace recovery. Local
regressions use disposable PostgreSQL and the real heartbeat queue. They
start no model provider or sandbox.

Related work: #19355 covers saved input after Stop. #18267 concerns
manual reconciliation of duplicate legacy holds; it excludes workspace
and system recovery and does not fix this receipt gate. I searched open
PRs and issues before starting this change.

## What Changed

- Reuse the existing host-verified repair receipt when an accepted
result belongs to a coordinator in `terminal_failure`.
- Retain the separate checks for active controllers, environment
termination, live executions, questions, approvals and current user
authority.
- Add message, saved-message and task Retry regressions, plus concurrent
queue delivery and invalid-receipt cases.
- Document this continuation path. Add no schema, API or background
worker.

## Verification

- All 483 tests pass across explicit continuation, task Retry,
quarantined Retry and finalized recovery migration suites.
- The final focused fixture has 17 cases. All pass with the change. With
the original production gate restored, all four positive cases fail and
all 13 safety cases pass.
- Negative cases cover missing, unfinished and mismatched receipts,
another task or source, unrecovered files, a live finalizer, an active
controller, missing remote stop and active execution.
- The real heartbeat path delivers one saved message under concurrent
recovery calls. Later recovery calls create no second turn. The old
accepted result and recovery budget stay unchanged; unknown outcomes
stay unknown.
- The first delivery fixture omitted the newer failed source needed by
the recovery scanner. I corrected the fixture to preserve that history
and repeated the unchanged-code comparison. Earlier failure logs remain
in the verification record.
- `pnpm -r typecheck`: passed.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`:
passed.
- `pnpm build`: passed.
- Full local `pnpm test:run`: running with a 20-minute bound; no
full-suite pass is claimed. All hosted CI checks passed. Greptile scored
the exact head `d04fb98892fd77853ad03e6b44cd612e805abc13` 5/5 with no
findings or review threads.
- Hosted delivery of the already saved input still requires a qualified
staging build containing this change.

## Risks

The new path requires the existing exact repair receipt and a terminal
failure. A coordinator still accepting or committing a result keeps the
previous hold. Stop, owner and live-run checks still run before
admission. The old result is neither applied nor replayed. Receipt
verification adds bounded indexed reads only on this retained-result
failure path. No migration or provider configuration changes are
required.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with xhigh
reasoning. The context-window size is not exposed. Used code editing,
tool calls and local test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, including the available runtime
details
- [x] I have checked ROADMAP.md and confirmed this is a scoped bug fix
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in this PR using the bug-report fields
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal task
id
- [x] I have run the affected tests locally and they pass
- [x] I have added regression tests and updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open findings
- [x] I will address all review findings before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/explicit-native-continuation.test.ts

@@ -18,7 +18,7 @@ import { beforeAll, afterAll, describe, it, expect, vi } from "vitest";
 import {
   applyPendingMigrations, approvals, issueApprovals, issueThreadInteractions, chatConversations, chatEndpoints, toolApplications, toolConnections,
   agentWakeupRequests, agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issueRecoveryActions,
-  issues, nativeRunFinalizations, nativeRunResults, completionContracts, environmentLeases, environments, issueRelations, issueTreeHolds, issueTreeHoldMembers, type Db,
+  issues, nativeRunFinalizations, nativeRunResults, completionContracts, environmentLeases, environments, issueRelations, issueTreeHolds, issueTreeHoldMembers, workspaceOperations, type Db,
 } from "@paperclipai/db";
 import { startEmbeddedPostgresTestDatabase, getEmbeddedPostgresTestSupport } from "../__tests__/helpers/embedded-postgres.js";
 import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
@@ -26,6 +26,7 @@ import { admitExplicitNativeContinuation } from "./explicit-native-continuation.
 import { adapterExecutionControls, createAdapterExecutionControl } from "./adapter-execution-control.js";
 import { CONVERSATION_CONTINUATION_POLICY } from "./conversation-continuation.js";
 import { buildExecutionContinuation } from "./execution-continuation.js";
+import { verifiedWorkspaceRepairReceipt } from "./workspace-recovery-receipt.js";
 import { heartbeatService, persistHeartbeatRunProcessMetadata, type HeartbeatEnvironmentRuntime } from "./heartbeat.js";
 import { getExecutionBlocker } from "./execution-blocker.js";
 import { createDurableChatWakeupRequest } from "./durable-chat-wakeup.js";
@@ -2260,6 +2261,141 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
       await settings.updateExperimental({ enableAgentChat: experimental.enableAgentChat });
     }
   });
+  async function recoveredCopybackFixture(origin: "message" | "queued_message" | "task_retry") {
+    const f = await seed(), resultId = await seedResultBeforeTerminalFailure(f);
+    const requestId = randomUUID(), operationId = randomUUID();
+    const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
+    const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: f.sourceRunId,
+      provider: "daytona", providerLeaseId: randomUUID() };
+    const reference = { schema: "paperclip.native-workspace-sync/v3", state: "finalized", workspaceId: randomUUID(),
+      leaseId: lease.id, providerLeaseId: lease.providerLeaseId, remoteCwd: "/workspaces/task",
+      descriptorSha256: "12".repeat(32), baselineSha256: "34".repeat(32), finalHostSha256: "56".repeat(32),
+      resourceDisposition: "destroy" };
+    const recovery = { ...createSystemRecovery(f.sourceRunId, origin === "task_retry" ? requestId : null),
+      state: "complete" as const, preservationOperationIds: [operationId] };
+    await db.update(issueRecoveryActions).set({ status: "cancelled", cause: "native_workspace_sync_out_retry_exhausted",
+      evidence: { runId: f.sourceRunId, workspaceRecovery: { schema: "paperclip.workspace-recovery.v1",
+        runId: f.sourceRunId, status: "recovered", stage: "saving_receipts", attempt: 1, requestIds: [] },
+        systemRecovery: recovery, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } },
+    }).where(eq(issueRecoveryActions.id, action.id));
+    await db.update(heartbeatRuns).set({ processPid: null, createdAt: new Date("2020-01-01T09:00:00Z"),
+      finishedAt: new Date("2020-01-01T10:00:00Z"), runnerProfileJson: { nativeWorkspaceSync: reference } })
+      .where(eq(heartbeatRuns.id, f.sourceRunId));
+    await db.update(issueComments).set({ createdAt: new Date("2020-01-01T13:00:00Z") }).where(eq(issueComments.id, f.commentId));
+    await db.insert(environmentLeases).values({ ...lease, status: "expired", leasePolicy: "ephemeral",
+      releasedAt: new Date(), cleanupStatus: "success", metadata: {
+        remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }),
+      },
+    });
+    await db.insert(workspaceOperations).values({ id: operationId, companyId: f.companyId, issueId: f.issueId,
+      heartbeatRunId: f.sourceRunId, phase: "workspace_recovery", status: "succeeded", metadata: {
+        recoveryActionId: action.id, leaseId: lease.id, providerLeaseId: lease.providerLeaseId,
+        descriptorSha256: reference.descriptorSha256, finalHostSha256: reference.finalHostSha256,
+      },
+    });
+    if (origin !== "message") await db.insert(agentWakeupRequests).values({ id: requestId, companyId: f.companyId, agentId: f.agentId,
+      source: "on_demand", triggerDetail: "manual", status: "deferred_issue_execution",
+      reason: origin === "task_retry" ? "task_retry" : "issue_commented", requestedByActorType: "user", requestedByActorId: f.actorId,
+      payload: { issueId: f.issueId, ...(origin === "task_retry" ? { taskRetry: { schema: "paperclip.task-retry.v1",
+        generationId: requestId, sourceRunId: f.sourceRunId, actorId: f.actorId, actorSource: "session", strategy: "evaluate" } }
+        : { commentId: f.commentId, executionWait: { reason: "finalizing_saved_result" },
+          _paperclipWakeContext: { source: "issue.comment", issueId: f.issueId, wakeReason: "issue_commented",
+            wakeCommentId: f.commentId, wakeCommentIds: [f.commentId] } }) },
+    });
+    return { ...f, resultId, action, lease, reference, operationId, requestId, recovery,
+      ...(origin === "task_retry" ? { taskRetryRequestId: requestId }
+        : origin === "queued_message" ? { queuedCommentRequestId: requestId } : {}) };
+  }
+  it.each(["message", "queued_message", "task_retry"] as const)("continues a terminal retained result after verified copy-back: %s", async origin => {
+    const f = await recoveredCopybackFixture(origin);
+    const [before] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId));
+    const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
+    expect(await verifiedWorkspaceRepairReceipt(db, source)).toBe(f.operationId);
+    const result = await admit(f);
+    expect(result).toMatchObject({ previousRunId: f.sourceRunId, commentId: f.commentId });
+    expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0]).toEqual(before);
+    const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id));
+    expect(action.evidence.automaticRecovery).toMatchObject({ actionOutcome: "unknown", replay: "explicit_user_continuation" });
+    expect(action.evidence.systemRecovery).toEqual(f.recovery);
+    expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0])
+      .toMatchObject({ phase: "terminal_failure", resultId: f.resultId });
+    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
+  });
+
+  it.each(["missing", "descriptor", "files", "lease", "provider", "source", "issue", "unfinished", "unrecovered",
+    "committing", "controller", "remote_stop", "active_run"] as const)("keeps retained-result continuation held without exact settled copy-back: %s", async fault => {
+    const f = await recoveredCopybackFixture("queued_message");
+    const [before] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId));
+    const [operation] = await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, f.operationId));
+    if (fault === "missing") await db.delete(workspaceOperations).where(eq(workspaceOperations.id, f.operationId));
+    const changed = fault === "descriptor" ? { descriptorSha256: "78".repeat(32) }
+      : fault === "files" ? { finalHostSha256: "78".repeat(32) }
+        : fault === "lease" ? { leaseId: randomUUID() }
+          : fault === "provider" ? { providerLeaseId: randomUUID() } : null;
+    if (changed) await db.update(workspaceOperations).set({ metadata: { ...operation.metadata, ...changed } })
+      .where(eq(workspaceOperations.id, f.operationId));
+    if (fault === "source") {
+      const [other] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed" }).returning();
+      await db.update(workspaceOperations).set({ heartbeatRunId: other.id }).where(eq(workspaceOperations.id, f.operationId));
+    }
+    if (fault === "issue") {
+      const [other] = await db.insert(issues).values({ companyId: f.companyId, title: "Other task" }).returning();
+      await db.update(workspaceOperations).set({ issueId: other.id }).where(eq(workspaceOperations.id, f.operationId));
+    }
+    if (fault === "unfinished") await db.update(workspaceOperations).set({ status: "running" }).where(eq(workspaceOperations.id, f.operationId));
+    if (fault === "unrecovered") await db.update(issueRecoveryActions).set({ evidence: { ...f.action.evidence,
+      runId: f.sourceRunId, systemRecovery: f.recovery, workspaceRecovery: { schema: "paperclip.workspace-recovery.v1",
+        runId: f.sourceRunId, status: "blocked", stage: "restoring_files", attempt: 1, requestIds: [] },
+    } }).where(eq(issueRecoveryActions.id, f.action.id));
+    if (fault === "committing") await db.update(nativeRunFinalizations).set({ phase: "result_accepted" })
+      .where(eq(nativeRunFinalizations.runId, f.sourceRunId));
+    if (fault === "controller") await db.update(nativeRunFinalizations).set({ leaseOwner: "live-controller" })
+      .where(eq(nativeRunFinalizations.runId, f.sourceRunId));
+    if (fault === "remote_stop") await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, f.lease.id));
+    if (fault === "active_run") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
+      nativeIssueId: f.issueId, runtimeMode: "native", status: "running" });
+    expect(await admit(f)).toBeNull();
+    expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0]).toEqual(before);
+    expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id)))[0].evidence)
+      .not.toHaveProperty("explicitUserContinuation");
+  });
+
+  it("delivers the saved message once after terminal result copy-back, without replaying its accepted result", async () => {
+    const f = await recoveredCopybackFixture("queued_message");
+    const failedRunId = randomUUID();
+    await db.insert(heartbeatRuns).values({ id: failedRunId, companyId: f.companyId, agentId: f.agentId,
+      nativeIssueId: f.issueId, runtimeMode: "native", status: "failed", errorCode: "native_provider_terminal_failed",
+      createdAt: new Date("2020-01-01T11:00:00Z"), finishedAt: new Date("2020-01-01T12:00:00Z"), contextSnapshot: { issueId: f.issueId } });
+    await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issueId, runId: failedRunId,
+      phase: "terminal_failure" });
+    await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
+      kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", fingerprint: failedRunId,
+      status: "resolved", outcome: "blocked", nextAction: "Continue from recorded work without replay.",
+      evidence: { runId: failedRunId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } });
+    const lease = { id: randomUUID(), companyId: f.companyId, heartbeatRunId: failedRunId,
+      provider: "daytona", providerLeaseId: randomUUID() };
+    await db.insert(environmentLeases).values({ ...lease, status: "expired", leasePolicy: "ephemeral",
+      releasedAt: new Date(), cleanupStatus: "success", metadata: {
+        remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }),
+      } });
+    await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
+    await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, f.requestId));
+    const heartbeat = heartbeatService(db);
+    await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]);
+    const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId));
+    expect(delivered).toMatchObject({ status: "coalesced", runId: expect.any(String) });
+    const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
+    expect(successors).toHaveLength(1);
+    expect(successors[0]).toMatchObject({ id: delivered.runId, contextSnapshot: {
+      previousRunId: failedRunId, forceFreshSession: true, wakeCommentIds: [f.commentId],
+    } });
+    await heartbeat.resumeExecutionWaitComments();
+    expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))))
+      .toHaveLength(1);
+    expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0])
+      .toMatchObject({ phase: "terminal_failure", resultId: f.resultId });
+  });
+
   it("defers an ordinary follow-up while a recovered export's successful result is still finalizing", async () => {
     const f = await seed(), resultId = await seedResultBeforeTerminalFailure(f);
     await db.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, failureDetail: null })

server/src/services/explicit-native-continuation.ts

@@ -370,7 +370,9 @@ export async function admitExplicitNativeContinuation(input: {
       const snapshotSettled = completedResult && freshIntent && action.id && recovery?.state === "complete"
         && await verifiedWorkspaceSnapshotReceipt(db, { companyId, issueId, runId: run.id,
           actionId: action.id, operationIds: recovery.preservationOperationIds });
-      if (completedResult && !(freshIntent && snapshotSettled))
+      const repairedTerminal = completedResult && coordinator.phase === "terminal_failure"
+        && await hasVerifiedWorkspaceRepair(db, run);
+      if (completedResult && !repairedTerminal && !(freshIntent && snapshotSettled))
         return blocked("finalizing_saved_result", "Paperclip is finishing the saved result.");
     }
     // A repaired export no longer blocks execution. Its machine receipt still