fix(recovery): let preservation owners reopen retained sources (#17053)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Recovery must save retained workspace files before ordinary
execution can resume.
> - The environment runtime blocks access to retained sources to prevent
unsafe reuse or deletion.
> - The recovery worker also used that ordinary resume path.
> - The guard returned no lease, so recovery reported an invalid source
binding before it could save files.
> - This pull request gives the physical preservation owner a separate,
source-bound resume path.
> - Ordinary execution and destructive cleanup keep their existing
guards.

## Linked Issues or Issue Description

Refs #16965.

**What happened?**

A retained workspace with a workspaceRestoreRecovery marker could not
recover through the real environment runtime. The runtime returned null
before calling the provider. Recovery classified this as
source_binding_invalid. Earlier recovery fixtures replaced the runtime
facade, so they did not exercise this guard.

**Expected behavior**

The verified preservation owner can reopen its exact stopped allocation
to save files. Other callers remain blocked. Recovery preserves newer
local files and does not start another agent or repeat a completed
external action.

**Steps to reproduce**

1. Seed a terminal source run with a retained sandbox, a recovery
action, and the workspaceRestoreRecovery lease marker.
2. Give the source a verified stop receipt.
3. Call recoverRetainedWorkspace through environmentRuntimeService.
4. Before this change, the recovery action returns to blocked with
source_binding_invalid. No files are copied.

**Paperclip version or commit**

Reproduced on master at 65f59e339a. The
same failure was observed in a served build containing #16965.

## What Changed

- Add an internal preservation resume method. It validates the exact
company, task, run, action, lease, environment, current physical owner,
and stop-only cleanup claim before provider access.
- Keep ordinary resume and destructive cleanup blocked for protected
sources.
- Run all retained-workspace fixtures through the assembled environment
runtime. Add a failing-before/passing-after protected-source regression
and four rejection cases.
- Use the same runtime path and protection marker in the opt-in Daytona
interruption probe. Allow a verified immutable snapshot UUID as its
input.
- Document the authority boundary and verification recipes.

## Verification

- Red: the protected-source regression failed with expected recovered,
received blocked. It recorded source_binding_invalid before provider
resume.
- Green: 235 tests passed across workspace recovery, durable task Retry,
and environment runtime. These include saved files, newer local files,
bounded retries, competing owners, and no extra run or wake.
- pnpm -r typecheck and pnpm build passed. The initial sandboxed
typecheck could not open a tsx IPC pipe; the same command passed with
local process access.
- node scripts/check-maps.mjs and git diff --check passed.
- Live Daytona qualification passed using the same immutable snapshot ID
as the affected deployment. Export was interrupted after a real
download; a second controller restored the files and verified one
original agent start, one simulated external action, one saved snapshot,
and a stopped retained source. The disposable sandbox was deleted after
verification.
- Greptile reviewed 804dfe24ef at 5/5 with
no actionable findings or unresolved review threads.
- Full current-commit CI passed:
https://github.com/paperclipai/paperclip/actions/runs/39540771747.
Required ci / verify and ci / e2e passed. Superagent, Socket, and Snyk
checks passed.
- Full local pnpm test:run did not pass: 22,992 tests passed, one
managed-run identity test timed out at 60 seconds, and 100 tests skipped
across the first general-server phase. The command stopped before its
later phases. The unchanged agent-identity.test.ts suite then passed all
11 tests in isolation in 15.53 seconds. The original timeout remains
recorded; it is not reclassified as a full local pass. The same commit's
complete CI test matrix passed.
- The optional Paperclip Review completed with a neutral advisory 5/5 on
the same head and no actionable findings. It inspected all eight changed
files and connected code. It did not independently run tests or the
provider probe.
- Hosted ordinary Retry recorded one durable intent and no successor run
while file preservation remained blocked. The existing published review
stayed unchanged. Hosted recovery remains unverified until this
follow-up is deployed. A Retry acknowledgement alone does not prove
recovery.

## Risks

- Recovery can reopen a retained sandbox. The dedicated path requires
the current physical ownership claim and the exact source's stop-only
intent. It does not clear the ordinary protection marker.
- Provider failures retain the source and use the existing bounded
recovery budget. The provider cannot substitute another allocation.
- The live probe uses a disposable sandbox and simulated
agent/external-effect counters. It does not prove a live LLM or GitHub
review journey.
- No schema, public API, permission, budget, pause, or approval changes.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution,
browser verification, and test tools. The exact served model variant and
context window are not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; full suite
still running)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/environment-runtime.ts

@@ -4,6 +4,8 @@ import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourcePro
 import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";
 import { beginIdleTrackedWork } from "./task-admission.js";
 import { historicalProviderCredentials } from "./historical-provider-credentials.js";
+import { ownedWorkspaceRecoveryLease, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
+import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
 import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";
 import { JsonRpcCallError, readEnvironmentAcquisitionDiagnostic, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk";
 import { preserveEnvironmentSyncOutErrorDiagnostic } from "./environment-sync-out-error.js";
@@ -4569,6 +4571,19 @@ export function environmentRuntimeService(
       return await driver.resumeRunLease(input);
     },
 
+    async resumeWorkspaceRecoveryLease(input: {
+      source: WorkspaceRecoverySourceIdentity; leaseId: string; environmentId: string;
+      ownership: NativeWorkspaceFinalizationOwnership;
+    }): Promise<PluginEnvironmentLease | EnvironmentLease | null> {
+      const { lease, environment } = await ownedWorkspaceRecoveryLease(db, { ...input.source,
+        leaseId: input.leaseId, environmentId: input.environmentId, ownership: input.ownership });
+      const driver = requireDriverKey(getLeaseDriverKey(lease, environment as Environment));
+      if (!driver.resumeRunLease) throw new Error(`Environment driver "${driver.driver}" does not support lease resume.`);
+      // Maintenance reopens the protected source only to preserve its files.
+      // Ordinary resume and destructive cleanup keep their existing fences.
+      return await driver.resumeRunLease({ environment: environment as Environment, lease: lease as EnvironmentLease });
+    },
+
     async destroyRunLease(input: EnvironmentDriverLeaseInput): Promise<EnvironmentLease | null> {
       if (await nativeWorkspaceExportPendingForLease(db, input.lease)) return null;
       if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;

server/src/services/workspace-recovery-source.ts

@@ -3,11 +3,31 @@ import { environmentLeases, environments, heartbeatRuns, issueRecoveryActions, n
 import { adapterExecutionControls } from "./adapter-execution-control.js";
 import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
 import { SystemRecoveryFailure } from "./system-recovery-state.js";
+import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
+import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
 
 export const workspaceExportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
 export const terminalWorkspaceRunStates = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
 export type WorkspaceRecoverySourceIdentity = { companyId: string; issueId: string; runId: string; actionId: string };
 
+/** The preservation owner may reopen only its exact stop-only allocation.
+ * This does not release the source's ordinary execution or destruction fence. */
+export async function ownedWorkspaceRecoveryLease(db: Db, input: WorkspaceRecoverySourceIdentity & {
+  leaseId: string; environmentId: string; ownership: NativeWorkspaceFinalizationOwnership;
+}) {
+  await input.ownership.assertHeld();
+  const inspected = await inspectWorkspaceRecoverySource(db, input);
+  const source = inspected.sources.find(item => item.lease.id === input.leaseId);
+  if (!source || source.environment.id !== input.environmentId || source.lease.status !== "pending_cleanup"
+    || source.lease.metadata?.pendingCleanupInFlight !== true
+    || source.lease.metadata?.pendingCleanupAttemptId !== input.ownership.token
+    || !readStopOnlyCleanup(source.lease)) {
+    throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
+  }
+  await input.ownership.assertHeld();
+  return source;
+}
+
 /** Resolve only the immutable source recorded by the failed run. Never discover
  * a candidate by recency, mutable adapter configuration, or a provider search. */
 export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRecoverySourceIdentity) {

server/src/services/workspace-recovery.test.ts

@@ -1,6 +1,7 @@
 import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js";
 import { createHeartbeatWorkspaceResolver, resolveExecutionWorkspaceReuseRequestForIssue } from "./heartbeat/workspaces.js";
 import { environmentService } from "./environments.js";
+import { classifyEnvironmentCapabilities, environmentRuntimeService } from "./environment-runtime.js";
 import { environmentRunOrchestrator } from "./environment-run-orchestrator.js";
 import { executionWorkspaceService } from "./execution-workspaces.js";
 import { workspaceOperationService } from "./workspace-operations.js";
@@ -111,7 +112,11 @@ describe("automatic retained workspace recovery", () => {
       fingerprint: runId, nextAction: "Manual repair required", evidence: { runId, workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery } });
     const runtime = { resumeRunLease: vi.fn(async (_input: { lease: EnvironmentLease }) => ({ providerLeaseId, metadata: { remoteCwd: remote } })),
       retryPendingSandboxTeardown: vi.fn(async () => ({ providerLeaseId, state: "stopped" })) };
-    const input = { db, companyId, issueId, runId, actionId, environmentRuntime: runtime as never };
+    const environmentRuntime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox",
+      acquireRunLease: vi.fn(), releaseRunLease: vi.fn(), ...runtime,
+      resolveCapabilities: async () => classifyEnvironmentCapabilities({ verifiedMethods: [], declared: null, narrowing: null }),
+    }] });
+    const input = { db, companyId, issueId, runId, actionId, environmentRuntime };
     return { ...input, input, runtime, host, remote, lease, providerLeaseId };
   }
   const action = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!;
@@ -283,6 +288,52 @@ describe("automatic retained workspace recovery", () => {
     expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, f.companyId), eq(workspaceOperations.phase, "workspace_recovery")))).toHaveLength(1);
   }, 30_000);
 
+  it("recovers a protected retained source through the assembled environment runtime", async () => {
+    const f = await seed();
+    await db.update(environmentLeases).set({ metadata: { ...f.lease.metadata,
+      workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", runId: f.runId },
+      remoteExecutionTermination: remoteTerminationReceipt(f.lease, { providerLeaseId: f.providerLeaseId, state: "stopped" }),
+    } }).where(eq(environmentLeases.id, f.lease.id));
+    const runtime = f.input.environmentRuntime;
+    const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
+    const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId!));
+    expect(await runtime.resumeRunLease({ environment: environment as never, lease: lease as EnvironmentLease })).toBeNull();
+    expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
+    await recoverRetainedWorkspace({ ...f.input, environmentRuntime: runtime });
+    expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered");
+    expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
+    expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
+    expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
+    await recoverRetainedWorkspace({ ...f.input, environmentRuntime: runtime });
+    expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
+    expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
+    expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
+  }, 30_000);
+
+  it.each(["changed_claim", "missing_stop_intent", "wrong_lease", "wrong_environment"])(
+    "does not reopen a retained source without exact recovery authority: %s", async kind => {
+      const f = await seed();
+      const runtime = f.input.environmentRuntime;
+      const resume = runtime.resumeWorkspaceRecoveryLease.bind(runtime);
+      vi.spyOn(runtime, "resumeWorkspaceRecoveryLease").mockImplementationOnce(async input => {
+        if (kind === "changed_claim") await db.update(environmentLeases).set({ metadata: sql`
+          ${environmentLeases.metadata} || '{"pendingCleanupAttemptId":"other-owner"}'::jsonb`
+        }).where(eq(environmentLeases.id, f.lease.id));
+        if (kind === "missing_stop_intent") await db.update(environmentLeases).set({ metadata: sql`
+          ${environmentLeases.metadata} - 'sandboxStopAndRetain'`
+        }).where(eq(environmentLeases.id, f.lease.id));
+        return resume({ ...input,
+          ...(kind === "wrong_lease" ? { leaseId: randomUUID() } : {}),
+          ...(kind === "wrong_environment" ? { environmentId: randomUUID() } : {}),
+        });
+      });
+      await recoverRetainedWorkspace(f.input);
+      expect(readSystemRecovery((await action(f.actionId)).evidence)?.failure?.code).toBe("source_binding_invalid");
+      expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
+      expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
+      expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
+    }, 30_000);
+
   it("keeps verified copyback files in the preserved workspace on ordinary Retry", async () => {
     const f = await seed();
     await recoverRetainedWorkspace(f.input);

server/src/services/workspace-recovery.ts

@@ -169,8 +169,8 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
         }).catch(() => undefined); }, 30_000);
         let stopSettled = false;
         try {
-          const opened = await input.environmentRuntime.resumeRunLease({ environment: environment as unknown as Environment,
-            lease: cleanupLease as EnvironmentLease });
+          const opened = await input.environmentRuntime.resumeWorkspaceRecoveryLease({ source: input,
+            leaseId: cleanupLease.id, environmentId: environment.id, ownership });
           if (opened?.providerLeaseId !== lease.providerLeaseId || opened.metadata?.remoteCwd !== remoteCwd)
             throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
           await inspectWorkspaceRecoverySource(db, input);