fix(server): retain stopped ephemeral workspaces after failed copy-back (#15790)

## Thinking Path

> - Paperclip manages work performed by agents.
> - A remote adapter must copy workspace changes back after execution.
> - Cancellation can stop and release an ephemeral sandbox before
copy-back fails.
> - The recovery recorder previously considered only active or already
held leases.
> - This change carries the original host lease into the recorder and
checks both stop receipts.
> - The exact stopped source receives a board repair obligation without
replaying work.

## Linked Issues or Issue Description

**What happened?**

A legacy remote run can report `workspaceRestoreFailure` after its
dedicated stop has released the source lease. The recorder misses that
source and does not create its file-repair action. Related prior work:
#15637 adds bounded transfer diagnostics; #15718 protects native
accepted finalization. This change addresses the separate legacy
recording order.

**Expected behavior**

Preserve the exact eligible source and record the existing board repair
obligation. Preserve cancellation, later task progress, and newer
execution ownership.

**Steps to reproduce**

1. Acquire an ephemeral remote lease for a legacy run.
2. Complete the dedicated stop-and-retain lifecycle and persist both
stop receipts.
3. Report a workspace copy-back failure from the original adapter
invocation.
4. Observe that the old recorder omits the released source. The
PostgreSQL regression reproduces this order.

**Paperclip version or commit**

de042e79ef.

**Deployment mode**

Server with PostgreSQL and a sandbox provider. The regression uses
disposable local PostgreSQL and synthetic provider receipts.

## What Changed

- Clone the realized host lease before adapter dispatch and carry it
into restore recording.
- Admit a released source only when its original and current policies
are ephemeral, its full ownership and workspace identity match, and its
dedicated retention receipt agrees with the independent
stopped-execution receipt.
- Reject reusable, expired, destroyed, rebound, competing, or
insufficiently proved sources. Preserve release state and receipts when
recording the existing no-replay repair action.
- Cover the actual stop-before-copy-back lifecycle, cleanup retries,
built-in dispatch, rollback, repeated recording, newer owners, and
ownership/receipt failures. Document the narrow scope.

## Verification

- `pnpm exec vitest run
server/src/__tests__/legacy-workspace-restore-recovery.test.ts
server/src/services/legacy-execution-recovery.test.ts`: 80 passed.
- Existing Grok confirmed-stop/copy-back boundary test: passed.
- The new PostgreSQL regression fails on the baseline because the
recovery marker is absent.
- Guard-removal controls fail when either the ephemeral-policy gate or
original-source identity checks are removed.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- Additional competing-owner/lifecycle checks: 6 passed, including older
stopped and destroyed allocation owners.
- `pnpm -r typecheck` and `pnpm build`: passed.
- Independent review approved the final source. The exact ephemeral
regression fails on the baseline; policy/identity guard-removal controls
fail their intended assertions.
- Compatibility with #15789 was checked in a disposable combined
checkout: both commit orders yield the same tree. The earlier broad run
passed 136 cases; the final guard/SDK delta passed 9 cases.
- Greptile: 5/5 on the current head, with no review threads.
- Exact-head CI: 53 successful checks and 2 expected skips; no failures
or pending checks. This includes all test shards and Canary Dry Run.
- The duplicate local `pnpm test:run` aggregate was stopped after full
CI passed. It is not reported as a completed local suite.
- No live provider calls, historical data repair, run replay, or
deployment was performed.

## Risks

No schema or API change. A qualifying source remains held for manual
file repair and eventual removal, which can retain provider storage
longer. This fix does not recover historical files or establish that the
provider still holds the bytes. Reusable sources remain excluded from
this new path because provider resume can start before the database
handoff. Existing active-source and previously recorded recovery
behavior is unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, and
independent agent review. The exact model revision and context-window
size are not exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/heartbeat.ts

@@ -4738,6 +4738,9 @@ export function heartbeatService(
       await bindIssueToPersistedExecutionWorkspace(persistedExecutionWorkspace);
       const workspaceRealization = realizationResult.workspaceRealization;
       const executionTarget = realizationResult.executionTarget;
+      // Preserve the host-owned source before adapter context can share lease
+      // metadata. A later copy-back failure must not adopt a rebound source.
+      const workspaceRestoreSource = structuredClone(realizationResult.lease);
       let instructionCopy: Awaited<ReturnType<typeof instructionCopies.prepare>> = null;
       let instructionSave: Record<string, unknown> | null = null;
       const instructionPreparationKey = createHash("sha256").update(JSON.stringify({
@@ -7157,7 +7160,7 @@ export function heartbeatService(
             };
             // Retention is the fallback even if recording this receipt fails.
             providerResourceDispositionForRun = "stop_and_retain";
-            await recordLegacyWorkspaceRestoreFailure(db, run, requiredWorkspaceRestoreEvidence);
+            await recordLegacyWorkspaceRestoreFailure(db, run, requiredWorkspaceRestoreEvidence, workspaceRestoreSource);
           }
           for (const stream of ["stdout", "stderr"] as const) {
             const tail = identityRedactor.finish(stream);

server/src/services/legacy-execution-recovery.ts

@@ -1,5 +1,5 @@
 import { hasRequiredWorkspaceRecovery, preserveWorkspaceRestoreRecoveryMetadata, LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
-import { hasUnrestoredRemoteWorkspace, preserveLegacyWorkspaceRestoreSources } from "./legacy-workspace-restore-recovery.js";
+import { hasUnrestoredRemoteWorkspace, preserveLegacyWorkspaceRestoreSources, type LegacyWorkspaceRestoreSource } from "./legacy-workspace-restore-recovery.js";
 import { isPreDispatchReviewWaitVerified } from "./pre-dispatch-review-wait.js";
 import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
 import { normalizeMaxTurnStopReason } from "./heartbeat-stop-metadata.js";
@@ -74,6 +74,7 @@ export async function terminalizeLegacyExecution(input: {
   writeConditions?: SQL[];
   /** Adapter settlement records a copy-back failure before host finalization. */
   recordRestoreFailureOnly?: boolean;
+  workspaceRestoreSource?: LegacyWorkspaceRestoreSource;
 }) {
   const { db, run, status } = input;
   let patch = input.patch;
@@ -122,7 +123,8 @@ export async function terminalizeLegacyExecution(input: {
       )
       .returning();
     if (!updated) return null;
-    const retainedLeaseIds = await preserveLegacyWorkspaceRestoreSources(tx as unknown as Db, input.recordRestoreFailureOnly ? { ...updated, status: "failed" } : updated);
+    const retainedLeaseIds = await preserveLegacyWorkspaceRestoreSources(tx as unknown as Db,
+      input.recordRestoreFailureOnly ? { ...updated, status: "failed" } : updated, input.workspaceRestoreSource);
     if (retainedLeaseIds.length) {
       [updated] = await tx.update(heartbeatRuns).set({ resultJson: {
         ...updated.resultJson,
@@ -310,9 +312,9 @@ export async function settleInterruptedNativeBootstrap(
 
 /** Capture the required-file obligation before unrelated host writes. Keeps
  * execution ownership while the adapter's host finalization is still running. */
-export async function recordLegacyWorkspaceRestoreFailure(db: Db, run: Run, evidence: Record<string, unknown>) {
+export async function recordLegacyWorkspaceRestoreFailure(db: Db, run: Run, evidence: Record<string, unknown>, source?: LegacyWorkspaceRestoreSource) {
   if (run.runtimeMode !== "legacy" || !hasWorkspaceRestoreFailure(evidence)) return;
   await terminalizeLegacyExecution({ db, run, status: run.status,
-    recordRestoreFailureOnly: true, patch: { resultJson: evidence },
+    recordRestoreFailureOnly: true, patch: { resultJson: evidence }, workspaceRestoreSource: source,
   });
 }

server/src/services/legacy-workspace-restore-recovery.ts

@@ -1,28 +1,57 @@
+import { isDeepStrictEqual } from "node:util";
 import { and, eq, inArray, isNotNull, ne, or, sql } from "drizzle-orm";
 import { environmentLeases, environments, heartbeatRuns, type Db } from "@paperclipai/db";
 import { hasWorkspaceRestoreFailure, WORKSPACE_RESTORE_FAILURE_CODES } from "@paperclipai/shared";
-import { prepareSandboxStopAndRetain } from "./sandbox-stop-and-retain.js";
+import { hasConfirmedSandboxStopAndRetain, prepareSandboxStopAndRetain } from "./sandbox-stop-and-retain.js";
 
 import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA } from "./workspace-restore-recovery-state.js";
 
 type Run = typeof heartbeatRuns.$inferSelect;
+/** The host's realized lease snapshot from before adapter execution. Never
+ * reconstruct this identity from a terminal status or a search of old leases. */
+export type LegacyWorkspaceRestoreSource = Pick<typeof environmentLeases.$inferSelect,
+  "id" | "companyId" | "heartbeatRunId" | "environmentId" | "issueId" | "executionWorkspaceId"
+  | "provider" | "providerLeaseId" | "acquiredAt" | "leasePolicy" | "metadata">;
 
 function needsRemoteRestore(run: Run): boolean {
   return run.runtimeMode === "legacy" && ["failed", "timed_out", "interrupted", "cancelled"].includes(run.status)
     && hasWorkspaceRestoreFailure(run.resultJson);
 }
 
-function leaseScope(run: Run) {
+function leaseScope(run: Run, source?: LegacyWorkspaceRestoreSource) {
   return and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
     or(eq(environmentLeases.status, "active"), and(
       sql`${environmentLeases.metadata}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
-      sql`${environmentLeases.metadata}->'workspaceRestoreRecovery'->>'runId' = ${run.id}`)),
+      sql`${environmentLeases.metadata}->'workspaceRestoreRecovery'->>'runId' = ${run.id}`),
+      source ? and(eq(environmentLeases.id, source.id), eq(environmentLeases.status, "released")) : undefined),
     isNotNull(environmentLeases.providerLeaseId),
     isNotNull(environmentLeases.provider), ne(environmentLeases.provider, "local"),
     or(eq(environments.driver, "sandbox"), sql`${environmentLeases.metadata}->>'driver' = 'sandbox'`,
       sql`${environmentLeases.metadata}->>'sandboxProviderPlugin' = 'true'`));
 }
 
+function matchesOriginalSource(lease: typeof environmentLeases.$inferSelect, run: Run, source: LegacyWorkspaceRestoreSource) {
+  const issueId = run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId ?? null;
+  const workspaceId = run.contextSnapshot?.executionWorkspaceId;
+  const remoteCwd = source.metadata?.remoteCwd;
+  const realization = source.metadata?.workspaceRealization;
+  if (typeof remoteCwd !== "string" || !remoteCwd.trim()
+    || (typeof workspaceId === "string" && source.executionWorkspaceId !== workspaceId)) return false;
+  if (realization !== undefined && realization !== null) {
+    if (typeof realization !== "object" || Array.isArray(realization)) return false;
+    const root = (realization as Record<string, unknown>).authoritativeRoot;
+    if (root !== undefined && (typeof root !== "string" || !root.trim())) return false;
+  }
+  return source.leasePolicy === "ephemeral"
+    && lease.leasePolicy === source.leasePolicy
+    && source.companyId === run.companyId && source.heartbeatRunId === run.id && source.issueId === issueId
+    && (["id", "companyId", "heartbeatRunId", "environmentId", "issueId", "executionWorkspaceId", "provider", "providerLeaseId"] as const)
+      .every(key => lease[key] === source[key])
+    && lease.acquiredAt.getTime() === source.acquiredAt.getTime()
+    && (["pluginId", "sandboxProviderPlugin", "remoteCwd", "workspaceRealization"] as const)
+      .every(key => isDeepStrictEqual(lease.metadata?.[key], source.metadata?.[key]));
+}
+
 /** A local lock timeout has no remote source to retain and keeps its old policy. */
 export async function hasUnrestoredRemoteWorkspace(db: Db, run: Run): Promise<boolean> {
   if (!needsRemoteRestore(run)) return false;
@@ -35,14 +64,40 @@ export async function hasUnrestoredRemoteWorkspace(db: Db, run: Run): Promise<bo
 /** Call in the terminal-status transaction. No cleanup worker may observe the
  * failed run without its exact stop-only intent. The original allocation must
  * not enter warm reuse; it belongs to the board's workspace repair action. */
-export async function preserveLegacyWorkspaceRestoreSources(db: Db, run: Run): Promise<string[]> {
+export async function preserveLegacyWorkspaceRestoreSources(db: Db, run: Run, source?: LegacyWorkspaceRestoreSource): Promise<string[]> {
   if (!needsRemoteRestore(run)) return [];
   const rows = await db.select({ lease: environmentLeases }).from(environmentLeases)
     .leftJoin(environments, eq(environments.id, environmentLeases.environmentId))
-    .where(leaseScope(run)).for("update", { of: environmentLeases });
+    .where(leaseScope(run, source)).for("update", { of: environmentLeases });
   const retained: string[] = [];
   for (const { lease } of rows) {
-    if (lease.status !== "active") { retained.push(lease.id); continue; }
+    if (lease.status !== "active") {
+      const recovery = lease.metadata?.workspaceRestoreRecovery as Record<string, unknown> | undefined;
+      if (recovery?.schema === LEGACY_WORKSPACE_RECOVERY_SCHEMA && recovery.runId === run.id) {
+        retained.push(lease.id); continue;
+      }
+      // Cancellation can finish the dedicated stop before copy-back reports its
+      // failure. Only the host's original source and two matching receipts can
+      // turn that released row into a file-repair hold.
+      if (!source || !matchesOriginalSource(lease, run, source) || !hasConfirmedSandboxStopAndRetain(lease)) continue;
+      const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
+        ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!),
+        eq(environmentLeases.providerLeaseId, lease.providerLeaseId!),
+      )).limit(1);
+      // An ephemeral source has no legitimate reuse history. Any other physical
+      // owner makes retention ambiguous, including an older owner that later
+      // destroyed the allocation. Do not expose cross-company owner identity.
+      if (other) continue;
+      await db.update(environmentLeases).set({ leasePolicy: "retain_on_failure",
+        metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ workspaceRestoreRecovery: {
+          schema: LEGACY_WORKSPACE_RECOVERY_SCHEMA, runId: run.id,
+        } })}::jsonb`,
+      }).where(eq(environmentLeases.id, lease.id));
+      // Reusable sources are excluded: provider resume precedes the database
+      // handoff, so a row lock cannot fence an in-flight resume. Keep the release
+      // status and receipts; this repair hold grants no execute right.
+      retained.push(lease.id); continue;
+    }
     // The stop-only worker rechecks exact allocation ownership, including
     // competing leases, before dispatch. Persist the intent even when cleanup
     // must wait, so no terminal-run sweep can fall back to destruction.

server/src/services/sandbox-stop-and-retain.ts

@@ -2,13 +2,46 @@ import { randomUUID } from "node:crypto";
 import { and, eq, sql, type SQL } from "drizzle-orm";
 import { environmentLeases, type Db } from "@paperclipai/db";
 import { isBuiltinSandboxProvider } from "./sandbox-provider-runtime.js";
-import { remoteTerminationReceipt } from "./remote-execution-termination.js";
+import { hasRemoteTerminationReceipt, remoteTerminationReceipt } from "./remote-execution-termination.js";
 import { hasNativeWorkspaceExportResume, readNativeWorkspaceExportResume, settleNativeWorkspaceExportResume } from "./native-runtime/native-workspace-export-resume.js";
 
 type Lease = Pick<typeof environmentLeases.$inferSelect,
   "id" | "companyId" | "heartbeatRunId" | "provider" | "providerLeaseId" | "metadata">;
 export const SANDBOX_STOP_AND_RETAIN_KEY = "sandboxStopAndRetain";
 
+/** A successful stop is not by itself proof that the allocation was retained.
+ * Require the dedicated stop-only dispatch receipt and its independent stop
+ * confirmation, both bound to the original lease. No provider work is allowed. */
+export function hasConfirmedSandboxStopAndRetain(lease: typeof environmentLeases.$inferSelect): boolean {
+  const acquiredAt = lease.acquiredAt.getTime();
+  const releasedAt = lease.releasedAt?.getTime();
+  const expiresAt = lease.expiresAt?.getTime();
+  if (!Number.isFinite(acquiredAt) || releasedAt === undefined || !Number.isFinite(releasedAt)
+    || (expiresAt !== undefined && (!Number.isFinite(expiresAt) || expiresAt <= Date.now()))) return false;
+  const value = lease.metadata?.sandboxStopAndRetainReceipt;
+  if (!value || typeof value !== "object" || Array.isArray(value)) return false;
+  const receipt = value as Record<string, unknown>;
+  const termination = lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
+  if (lease.status !== "released" || !hasRemoteTerminationReceipt(lease)
+    || termination?.state !== "stopped" || hasStopOnlyCleanup(lease)
+    || lease.metadata?.pendingCleanupInFlight === true
+    || receipt.schema !== "paperclip.sandbox-stop-and-retain-receipt.v1"
+    || receipt.companyId !== lease.companyId || receipt.runId !== lease.heartbeatRunId
+    || receipt.leaseId !== lease.id || receipt.provider !== lease.provider
+    || receipt.providerLeaseId !== lease.providerLeaseId
+    || typeof receipt.requestId !== "string" || !receipt.requestId.trim()
+    || typeof receipt.confirmedAt !== "string" || receipt.confirmedAt !== termination.confirmedAt) return false;
+  const confirmedAt = Date.parse(receipt.confirmedAt);
+  if (!Number.isFinite(confirmedAt) || confirmedAt < acquiredAt || confirmedAt > releasedAt) return false;
+  if (receipt.builtinProvider !== undefined) {
+    return receipt.builtinProvider === lease.provider && isBuiltinSandboxProvider(lease.provider!)
+      && receipt.method === "builtin.stopLease" && receipt.pluginId === undefined
+      && lease.metadata?.pluginId == null && !lease.metadata?.sandboxProviderPlugin;
+  }
+  return receipt.method === "environmentStopLease" && typeof receipt.pluginId === "string"
+    && !!receipt.pluginId.trim() && receipt.pluginId === lease.metadata?.pluginId;
+}
+
 export function hasStopOnlyCleanup(lease: Pick<Lease, "metadata">): boolean {
   return hasNativeWorkspaceExportResume(lease)
     || Object.prototype.hasOwnProperty.call(lease.metadata ?? {}, SANDBOX_STOP_AND_RETAIN_KEY);