fix(recovery): let preservation owners reopen retained sources (#17053)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Recovery must save retained workspace files before ordinary
execution can resume.
> - The environment runtime blocks access to retained sources to prevent
unsafe reuse or deletion.
> - The recovery worker also used that ordinary resume path.
> - The guard returned no lease, so recovery reported an invalid source
binding before it could save files.
> - This pull request gives the physical preservation owner a separate,
source-bound resume path.
> - Ordinary execution and destructive cleanup keep their existing
guards.

## Linked Issues or Issue Description

Refs #16965.

**What happened?**

A retained workspace with a workspaceRestoreRecovery marker could not
recover through the real environment runtime. The runtime returned null
before calling the provider. Recovery classified this as
source_binding_invalid. Earlier recovery fixtures replaced the runtime
facade, so they did not exercise this guard.

**Expected behavior**

The verified preservation owner can reopen its exact stopped allocation
to save files. Other callers remain blocked. Recovery preserves newer
local files and does not start another agent or repeat a completed
external action.

**Steps to reproduce**

1. Seed a terminal source run with a retained sandbox, a recovery
action, and the workspaceRestoreRecovery lease marker.
2. Give the source a verified stop receipt.
3. Call recoverRetainedWorkspace through environmentRuntimeService.
4. Before this change, the recovery action returns to blocked with
source_binding_invalid. No files are copied.

**Paperclip version or commit**

Reproduced on master at 65f59e339a. The
same failure was observed in a served build containing #16965.

## What Changed

- Add an internal preservation resume method. It validates the exact
company, task, run, action, lease, environment, current physical owner,
and stop-only cleanup claim before provider access.
- Keep ordinary resume and destructive cleanup blocked for protected
sources.
- Run all retained-workspace fixtures through the assembled environment
runtime. Add a failing-before/passing-after protected-source regression
and four rejection cases.
- Use the same runtime path and protection marker in the opt-in Daytona
interruption probe. Allow a verified immutable snapshot UUID as its
input.
- Document the authority boundary and verification recipes.

## Verification

- Red: the protected-source regression failed with expected recovered,
received blocked. It recorded source_binding_invalid before provider
resume.
- Green: 235 tests passed across workspace recovery, durable task Retry,
and environment runtime. These include saved files, newer local files,
bounded retries, competing owners, and no extra run or wake.
- pnpm -r typecheck and pnpm build passed. The initial sandboxed
typecheck could not open a tsx IPC pipe; the same command passed with
local process access.
- node scripts/check-maps.mjs and git diff --check passed.
- Live Daytona qualification passed using the same immutable snapshot ID
as the affected deployment. Export was interrupted after a real
download; a second controller restored the files and verified one
original agent start, one simulated external action, one saved snapshot,
and a stopped retained source. The disposable sandbox was deleted after
verification.
- Greptile reviewed 804dfe24ef at 5/5 with
no actionable findings or unresolved review threads.
- Full current-commit CI passed:
https://github.com/paperclipai/paperclip/actions/runs/39540771747.
Required ci / verify and ci / e2e passed. Superagent, Socket, and Snyk
checks passed.
- Full local pnpm test:run did not pass: 22,992 tests passed, one
managed-run identity test timed out at 60 seconds, and 100 tests skipped
across the first general-server phase. The command stopped before its
later phases. The unchanged agent-identity.test.ts suite then passed all
11 tests in isolation in 15.53 seconds. The original timeout remains
recorded; it is not reclassified as a full local pass. The same commit's
complete CI test matrix passed.
- The optional Paperclip Review completed with a neutral advisory 5/5 on
the same head and no actionable findings. It inspected all eight changed
files and connected code. It did not independently run tests or the
provider probe.
- Hosted ordinary Retry recorded one durable intent and no successor run
while file preservation remained blocked. The existing published review
stayed unchanged. Hosted recovery remains unverified until this
follow-up is deployed. A Retry acknowledgement alone does not prove
recovery.

## Risks

- Recovery can reopen a retained sandbox. The dedicated path requires
the current physical ownership claim and the exact source's stop-only
intent. It does not clear the ordinary protection marker.
- Provider failures retain the source and use the existing bounded
recovery budget. The provider cannot substitute another allocation.
- The live probe uses a disposable sandbox and simulated
agent/external-effect counters. It does not prove a live LLM or GitHub
review journey.
- No schema, public API, permission, budget, pause, or approval changes.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution,
browser verification, and test tools. The exact served model variant and
context window are not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; full suite
still running)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-21 07:27:45 -05:00
1 parent 30b753f614
commit eaacadeba3
8 files changed
+131 -9

No files matched your search

+8
View File
@@ -33,6 +33,12 @@ The source recovery action holds a validated `systemRecovery` record. Its
revision and generation fence late bookkeeping writes. Physical exports still
use the existing workspace ownership lock and controller stop receipts.
The export owner can reopen its exact stopped sandbox through the environment
runtime's recovery path. This path checks the current source action, run,
environment, allocation, stop-only intent, and physical ownership claim before
calling the provider. The retained-source fence still blocks ordinary resume
and destruction. Recovery does not clear it to gain access to the saved files.
Execution containment and file preservation are independent facts:
- Another agent needs authoritative containment of the old executor.
@@ -113,6 +119,8 @@ Run the focused PostgreSQL/filesystem suites from `server/`:
`server/scripts/workspace-recovery-provider.test.mjs` is an opt-in controlled
Daytona probe. It requires `PAPERCLIP_RECOVERY_LIVE=1`, `DAYTONA_API_KEY`,
`PAPERCLIP_DAYTONA_SDK_PATH` and an immutable `PAPERCLIP_RECOVERY_IMAGE`.
An immutable provider snapshot UUID in `PAPERCLIP_RECOVERY_SNAPSHOT_ID` can
replace the image digest. The probe checks its ID and active state first.
It interrupts export,
joins the first controller, starts a second controller and verifies files,
preservation receipts, agent-start count and external-operation count before
+6
View File
@@ -212,6 +212,12 @@ Gaps: An acknowledged retry can remain pending or fail; report actual useful res
Capability ID: `recovery/workspace`.
The internal preservation resume path grants no board or agent execution API.
It requires the exact source action, company, run, lease and environment, a
held physical workspace owner, and its current stop-only cleanup claim. The
ordinary resume and destruction fences still apply to protected sources. The
assembled-runtime regression checks this separation before any provider call.
Review: `operation-reviewed`.
Each workspace repair/export/reconstruction/physical-owner operation has its own proof and authority.
+8
View File
@@ -222,6 +222,14 @@ cover quiet live recovery, read-only access, stale or missing stop proof,
source-only repair, and preserved task status. These tests use fixtures; separately
verify original files and their attached artifact before recording a real repair.
For a source with a retained-file protection marker, exercise recovery through
the assembled environment runtime. Its exact preservation owner may reopen the
stopped allocation to read saved files; ordinary resume stays blocked. Reject a
changed claim, missing stop intent, wrong lease or wrong environment before a
provider call. Verify saved and newer local files, one preservation receipt, and
no additional agent turn. The PostgreSQL regression covers this boundary; the
opt-in provider probe also uses this runtime path with the protection marker.
Repeat source-only repair with a `succeeded` legacy run that still has a retained
file obligation. Its result must remain successful, its current task owner and
status must stay unchanged, and later saved input must become eligible for normal
@@ -15,6 +15,7 @@ import { NOOP_PLUGIN_TRACER } from "@paperclipai/plugin-sdk";
import { startEmbeddedPostgresTestDatabase } from "../src/__tests__/helpers/embedded-postgres.js";
import { readSystemRecovery } from "../src/services/system-recovery-state.js";
import { recoverRetainedWorkspace } from "../src/services/workspace-recovery.js";
import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../src/services/environment-runtime.js";
import { attachPreservedWorkspaceSnapshots } from "../src/services/workspace-recovery-snapshot.js";
import { performSyncOut } from "../../packages/plugins/sandbox-providers/daytona/src/file-sync.js";
@@ -68,7 +69,11 @@ it.skipIf(!live || !worker)("provider controller", async () => {
},
syncOut: operations => performSyncOut({ sandbox, operations, remoteDir: f.remoteCwd, timeoutSeconds: 120 }),
} };
const environmentRuntime = {
const driver = {
driver: "sandbox",
acquireRunLease: async () => { throw new Error("Recovery must not acquire another sandbox"); },
releaseRunLease: async () => { throw new Error("Recovery must not destroy its retained source"); },
resolveCapabilities: async () => classifyEnvironmentCapabilities({ verifiedMethods: [] }),
resumeRunLease: async () => {
await sandbox.start(120); await sandbox.refreshData();
if (sandbox.state !== "started") throw new Error("Provider did not confirm start");
@@ -80,6 +85,7 @@ it.skipIf(!live || !worker)("provider controller", async () => {
return { providerLeaseId: sandbox.id, state: "stopped" };
},
};
const environmentRuntime = environmentRuntimeService(db, { drivers: [driver] });
await recoverRetainedWorkspace({ db, ...f, environmentRuntime });
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId));
const state = readSystemRecovery(action.evidence);
@@ -90,7 +96,9 @@ it.skipIf(!live || !worker)("provider controller", async () => {
it.skipIf(!live || Boolean(worker))("preserves provider files across an interrupted export and controller restart without restarting the old agent", async () => {
const image = process.env.PAPERCLIP_RECOVERY_IMAGE;
if (!image?.includes("@sha256:")) throw new Error("An immutable qualification image is required");
const snapshotId = process.env.PAPERCLIP_RECOVERY_SNAPSHOT_ID;
if (!image?.includes("@sha256:") && !/^[0-9a-f-]{36}$/i.test(snapshotId ?? ""))
throw new Error("An immutable qualification image or snapshot ID is required");
const directory = await mkdtemp(path.join(os.tmpdir(), "paperclip-provider-recovery-"));
const priorHome = process.env.PAPERCLIP_HOME;
process.env.PAPERCLIP_HOME = path.join(directory, "home");
@@ -100,11 +108,16 @@ it.skipIf(!live || Boolean(worker))("preserves provider files across an interrup
const db = createDb(database.connectionString);
const daytona = await client();
let sandbox;
const evidence = { image, interrupted: false, restored: false, oldAgentStarts: null, externalOperations: null,
const evidence = { image, snapshotId, interrupted: false, restored: false, oldAgentStarts: null, externalOperations: null,
prematureCleanup: false, controllerProcesses: [], snapshotCount: 0 };
try {
sandbox = await daytona.create({ image, autoStopInterval: 10, autoArchiveInterval: 60,
resources: { cpu: 2, memory: 4, disk: 10 }, labels: { purpose: "paperclip-recovery-qualification" } }, { timeout: 300 });
if (snapshotId) {
const snapshot = await daytona.snapshot.get(snapshotId);
expect(snapshot.id).toBe(snapshotId);
expect(snapshot.state).toBe("active");
}
sandbox = await daytona.create({ ...(snapshotId ? { snapshot: snapshotId } : { image, resources: { cpu: 2, memory: 4, disk: 10 } }),
autoStopInterval: 10, autoArchiveInterval: 60, labels: { purpose: "paperclip-recovery-qualification" } }, { timeout: 300 });
const remoteCwd = "/tmp/paperclip-recovery-fixture";
await sandbox.process.executeCommand(`mkdir -p ${remoteCwd}`, undefined, undefined, 30);
await sandbox.process.createSession("old-agent");
@@ -124,7 +137,8 @@ it.skipIf(!live || Boolean(worker))("preserves provider files across an interrup
const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId: runId,
environmentId: environment.id, provider: "daytona", providerLeaseId: sandbox.id,
leasePolicy: "retain_on_failure", status: "released", releasedAt: new Date(),
metadata: { remoteCwd, driver: "sandbox", pluginId: randomUUID(), sandboxProviderPlugin: true } }).returning();
metadata: { remoteCwd, driver: "sandbox", pluginId: randomUUID(), sandboxProviderPlugin: true,
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", runId } } }).returning();
const recovery = { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [lease.id] };
await db.update(heartbeatRuns).set({ resultJson: { workspaceRestoreRecovery: recovery, workspaceRestoreFailure: "restore_failed" } })
.where(eq(heartbeatRuns.id, runId));
@@ -4,6 +4,8 @@ import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourcePro
import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";
import { beginIdleTrackedWork } from "./task-admission.js";
import { historicalProviderCredentials } from "./historical-provider-credentials.js";
import { ownedWorkspaceRecoveryLease, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js";
import { JsonRpcCallError, readEnvironmentAcquisitionDiagnostic, readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk";
import { preserveEnvironmentSyncOutErrorDiagnostic } from "./environment-sync-out-error.js";
@@ -4569,6 +4571,19 @@ export function environmentRuntimeService(
return await driver.resumeRunLease(input);
},
async resumeWorkspaceRecoveryLease(input: {
source: WorkspaceRecoverySourceIdentity; leaseId: string; environmentId: string;
ownership: NativeWorkspaceFinalizationOwnership;
}): Promise<PluginEnvironmentLease | EnvironmentLease | null> {
const { lease, environment } = await ownedWorkspaceRecoveryLease(db, { ...input.source,
leaseId: input.leaseId, environmentId: input.environmentId, ownership: input.ownership });
const driver = requireDriverKey(getLeaseDriverKey(lease, environment as Environment));
if (!driver.resumeRunLease) throw new Error(`Environment driver "${driver.driver}" does not support lease resume.`);
// Maintenance reopens the protected source only to preserve its files.
// Ordinary resume and destructive cleanup keep their existing fences.
return await driver.resumeRunLease({ environment: environment as Environment, lease: lease as EnvironmentLease });
},
async destroyRunLease(input: EnvironmentDriverLeaseInput): Promise<EnvironmentLease | null> {
if (await nativeWorkspaceExportPendingForLease(db, input.lease)) return null;
if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;
@@ -3,11 +3,31 @@ import { environmentLeases, environments, heartbeatRuns, issueRecoveryActions, n
import { adapterExecutionControls } from "./adapter-execution-control.js";
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
import { SystemRecoveryFailure } from "./system-recovery-state.js";
import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
export const workspaceExportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
export const terminalWorkspaceRunStates = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
export type WorkspaceRecoverySourceIdentity = { companyId: string; issueId: string; runId: string; actionId: string };
/** The preservation owner may reopen only its exact stop-only allocation.
* This does not release the source's ordinary execution or destruction fence. */
export async function ownedWorkspaceRecoveryLease(db: Db, input: WorkspaceRecoverySourceIdentity & {
leaseId: string; environmentId: string; ownership: NativeWorkspaceFinalizationOwnership;
}) {
await input.ownership.assertHeld();
const inspected = await inspectWorkspaceRecoverySource(db, input);
const source = inspected.sources.find(item => item.lease.id === input.leaseId);
if (!source || source.environment.id !== input.environmentId || source.lease.status !== "pending_cleanup"
|| source.lease.metadata?.pendingCleanupInFlight !== true
|| source.lease.metadata?.pendingCleanupAttemptId !== input.ownership.token
|| !readStopOnlyCleanup(source.lease)) {
throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
}
await input.ownership.assertHeld();
return source;
}
/** Resolve only the immutable source recorded by the failed run. Never discover
* a candidate by recency, mutable adapter configuration, or a provider search. */
export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRecoverySourceIdentity) {
+52 -1
View File
@@ -1,6 +1,7 @@
import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js";
import { createHeartbeatWorkspaceResolver, resolveExecutionWorkspaceReuseRequestForIssue } from "./heartbeat/workspaces.js";
import { environmentService } from "./environments.js";
import { classifyEnvironmentCapabilities, environmentRuntimeService } from "./environment-runtime.js";
import { environmentRunOrchestrator } from "./environment-run-orchestrator.js";
import { executionWorkspaceService } from "./execution-workspaces.js";
import { workspaceOperationService } from "./workspace-operations.js";
@@ -111,7 +112,11 @@ describe("automatic retained workspace recovery", () => {
fingerprint: runId, nextAction: "Manual repair required", evidence: { runId, workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery } });
const runtime = { resumeRunLease: vi.fn(async (_input: { lease: EnvironmentLease }) => ({ providerLeaseId, metadata: { remoteCwd: remote } })),
retryPendingSandboxTeardown: vi.fn(async () => ({ providerLeaseId, state: "stopped" })) };
const input = { db, companyId, issueId, runId, actionId, environmentRuntime: runtime as never };
const environmentRuntime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox",
acquireRunLease: vi.fn(), releaseRunLease: vi.fn(), ...runtime,
resolveCapabilities: async () => classifyEnvironmentCapabilities({ verifiedMethods: [], declared: null, narrowing: null }),
}] });
const input = { db, companyId, issueId, runId, actionId, environmentRuntime };
return { ...input, input, runtime, host, remote, lease, providerLeaseId };
}
const action = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!;
@@ -283,6 +288,52 @@ describe("automatic retained workspace recovery", () => {
expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, f.companyId), eq(workspaceOperations.phase, "workspace_recovery")))).toHaveLength(1);
}, 30_000);
it("recovers a protected retained source through the assembled environment runtime", async () => {
const f = await seed();
await db.update(environmentLeases).set({ metadata: { ...f.lease.metadata,
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", runId: f.runId },
remoteExecutionTermination: remoteTerminationReceipt(f.lease, { providerLeaseId: f.providerLeaseId, state: "stopped" }),
} }).where(eq(environmentLeases.id, f.lease.id));
const runtime = f.input.environmentRuntime;
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId!));
expect(await runtime.resumeRunLease({ environment: environment as never, lease: lease as EnvironmentLease })).toBeNull();
expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
await recoverRetainedWorkspace({ ...f.input, environmentRuntime: runtime });
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered");
expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
await recoverRetainedWorkspace({ ...f.input, environmentRuntime: runtime });
expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
}, 30_000);
it.each(["changed_claim", "missing_stop_intent", "wrong_lease", "wrong_environment"])(
"does not reopen a retained source without exact recovery authority: %s", async kind => {
const f = await seed();
const runtime = f.input.environmentRuntime;
const resume = runtime.resumeWorkspaceRecoveryLease.bind(runtime);
vi.spyOn(runtime, "resumeWorkspaceRecoveryLease").mockImplementationOnce(async input => {
if (kind === "changed_claim") await db.update(environmentLeases).set({ metadata: sql`
${environmentLeases.metadata} || '{"pendingCleanupAttemptId":"other-owner"}'::jsonb`
}).where(eq(environmentLeases.id, f.lease.id));
if (kind === "missing_stop_intent") await db.update(environmentLeases).set({ metadata: sql`
${environmentLeases.metadata} - 'sandboxStopAndRetain'`
}).where(eq(environmentLeases.id, f.lease.id));
return resume({ ...input,
...(kind === "wrong_lease" ? { leaseId: randomUUID() } : {}),
...(kind === "wrong_environment" ? { environmentId: randomUUID() } : {}),
});
});
await recoverRetainedWorkspace(f.input);
expect(readSystemRecovery((await action(f.actionId)).evidence)?.failure?.code).toBe("source_binding_invalid");
expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
}, 30_000);
it("keeps verified copyback files in the preserved workspace on ordinary Retry", async () => {
const f = await seed();
await recoverRetainedWorkspace(f.input);
+2 -2
View File
@@ -169,8 +169,8 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
}).catch(() => undefined); }, 30_000);
let stopSettled = false;
try {
const opened = await input.environmentRuntime.resumeRunLease({ environment: environment as unknown as Environment,
lease: cleanupLease as EnvironmentLease });
const opened = await input.environmentRuntime.resumeWorkspaceRecoveryLease({ source: input,
leaseId: cleanupLease.id, environmentId: environment.id, ownership });
if (opened?.providerLeaseId !== lease.providerLeaseId || opened.metadata?.remoteCwd !== remoteCwd)
throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
await inspectWorkspaceRecoverySource(db, input);