fix(workspaces): attach a recovered saved version once, not once per failed run (#22484)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a run starts in a new sandbox, Paperclip prepares the task
workspace. It also attaches the saved files of earlier failed runs under
`.paperclip-recovered/<operation>` so the agent can continue their work
> - A run that restores a saved workspace and then fails can save the
same version again. Workspace recovery keeps a receipt for that run. One
saved version can then have a receipt from each of several failed runs
> - Preparation attached the files once for each receipt. It removed
duplicates only by receipt operation ID
> - On staging, six receipts named the same 1 GB saved version. Each new
sandbox received six copies (about 6 GB and 171k entries) before its
first save. The first save then failed with
`remote_workspace_sandbox_disk_full`. The read-only fallback also
refused with `budget_exceeded`, because its inventory limit is 100,000
entries
> - The next sandbox received the same six copies. No run could succeed,
so nothing retired the copies, and the task failed in the same way about
every 30 minutes
> - This pull request attaches each saved version once
> - The benefit is that a task with several recovered failed runs gets
one copy of their saved work, and its sandbox has space for its first
save

## Linked Issues or Issue Description

Refs #22466 (names this failure `sandbox_disk_full`). Refs #22383,
#22239, #22068 (other full-sandbox-disk handling). Those changes name
the failure and stop repeats. This change removes the cause of this
case.

**What happened?**
On staging, three tasks failed during setup again and again with
`remote_workspace_sandbox_disk_full: the sandbox disk is full; free
space in the sandbox before saving the workspace`. Each run used a new
sandbox. One workspace failed its initial save 18 times in about 8
hours. Its metadata recorded six recovered attachments. All six came
from one earlier workspace, one snapshot and one sha256 (1,040,502,418
bytes, 28,576 entries). Only the source run was different. Each of those
six source runs had restored that snapshot, saved it again unchanged
(`reusedSnapshotId` equals the parent), and failed for another reason.

**Expected behavior**
The new workspace gets one copy of the saved version. Its first save has
space.

**Steps to reproduce**
1. Save a workspace (version A).
2. Start a writer that restores A, saves A again unchanged, and fails.
Record its recovery receipt.
3. Do step 2 again with a second writer.
4. Prepare a new workspace with both failed writers as recovery
predecessors.
5. See two full copies of A under `.paperclip-recovered/`.

## What Changed

- `remote-workspace-recovery-attachments.ts`:
`attachPreservedRemoteWorkspaceSnapshots` first collects the verified
receipts. It then attaches each saved version (source workspace,
snapshot ID, sha256) once, under the receipt of the earliest source run.
If any receipt of a version is retired, the version is retired. Receipt
verification, access checks, and the provenance of an attached copy do
not change.
- `remote-workspace-snapshot-reuse.test.ts`: a new database-backed test
uses the existing reused-save recovery fixture with two failed writers.
The `same` case expects one copy. The `changed` case (the second writer
saved different files) expects two copies. Both cases check that a
retired receipt retires its whole version. The recovery stamping moved
into a helper (`recoverFailedWriter`) for both writers.

## Verification

- Red: on unmodified `master`, `pnpm vitest run
src/services/remote-workspace-snapshot-reuse.test.ts -t "attaches each
saved version once"` fails for the `same` case: `expected [ {…}, {…} ]
to deeply equal [ ObjectContaining{…} ]`, which is two copies of one
version. The `changed` case passes.
- Green: with the change, `pnpm vitest run
src/services/remote-workspace-snapshot-reuse.test.ts
src/services/workspace-recovery-preparation.test.ts
src/services/remote-workspace-recovery-receipt-fallback.test.ts` passes:
3 files, 125 tests. These include the existing same-run duplicate check,
retired copies, unverifiable receipts, and storage backend selection.
- `node scripts/typecheck-layers.mjs` in `server/`: no errors in
`server/src`. My sandbox installed only the server dependencies, so
`cli/`, `ui/` and `packages/kv-demo-mcp-server` reported missing-module
errors there. CI runs the full typecheck.
- I did not run a live sandbox preparation. The staging workspace
records above give the sizes and the six duplicate receipts.

## Risks

- Low. Only duplicate copies of one saved version are skipped. Different
versions are all attached, as before.
- The agent sees one recovered folder for that version instead of
several identical folders. The recovered-files list is context for the
agent only. No settlement step needs one copy per source run.
- Existing workspaces keep their old records. In a new sandbox, the
records of copies that are not attached again are dropped, as for any
copy that is absent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-21 03:07:06 -08:00
1 parent 1b1861c29d
commit 4631d3b41a
2 files changed
+79 -13

No files matched your search

@@ -6,7 +6,7 @@ import { removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js";
import { LEGACY_WORKSPACE_RECOVERY_MIGRATION } from "./legacy-workspace-recovery-migration.js";
import { and, eq, inArray, sql } from "drizzle-orm";
import { executionWorkspaceSnapshots, executionWorkspaces, heartbeatRuns, workspaceOperations, type Db } from "@paperclipai/db";
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type RemoteWorkspaceRunReference } from "@paperclipai/shared";
import { readSnapshot, verifiedWorkspaceSnapshotReceipt } from "./workspace-recovery-snapshot.js";
import { recoveryRunAncestry } from "./recovery-run-ancestry.js";
import { immutableRemoteWorkspaceRecoveryAttachment } from "./workspace-recovery-receipt.js";
@@ -54,7 +54,8 @@ export async function attachPreservedRemoteWorkspaceSnapshots(input: {
eq(workspaceOperations.phase, "workspace_recovery"), sql`${workspaceOperations.metadata}->>'storage' = 'remote_workspace'`,
)) : [];
const canonicalSources = new Set(canonicalReceipts.map(receipt => receipt.runId));
const result: Array<{ operationId: string; path: string; sourceRunId: string }> = [];
const candidates: Array<{ source: typeof sources[number]; remote: RemoteWorkspaceRunReference;
proof: NonNullable<Awaited<ReturnType<typeof immutableRemoteWorkspaceRecoveryAttachment>>> }> = [];
for (const source of sources) {
if (source.runnerProfileJson?.remoteWorkspace == null) {
if (canonicalSources.has(source.id)) skipUnverifiedRecoverySource(input, source.id, "receipt_without_reference");
@@ -79,14 +80,30 @@ export async function attachPreservedRemoteWorkspaceSnapshots(input: {
skipUnverifiedRecoverySource(input, source.id, "receipt_unverified");
continue;
}
candidates.push({ source, remote, proof });
}
// A run that restores a save and then fails can save the same version again,
// so one saved version can carry a receipt from each of several failed runs.
// Its files are one piece of work: attach them once, under the earliest
// receipt. Each extra copy filled the sandbox disk before the first save.
// Retiring any receipt of a version retires the version.
const versionOf = (candidate: typeof candidates[number]) =>
JSON.stringify([candidate.proof.workspaceId, candidate.proof.snapshotId, candidate.proof.sha256]);
const retiredVersions = new Set(candidates
.filter(candidate => input.retiredOperationIds?.has(candidate.proof.operationId)).map(versionOf));
const attachedVersions = new Set<string>();
candidates.sort((a, b) => a.source.createdAt.getTime() - b.source.createdAt.getTime() || a.source.id.localeCompare(b.source.id));
const result: Array<{ operationId: string; path: string; sourceRunId: string }> = [];
for (const candidate of candidates) {
const { source, remote, proof } = candidate, version = versionOf(candidate);
if (retiredVersions.has(version) || attachedVersions.has(version)) continue;
const operationId = proof.operationId;
if (input.retiredOperationIds?.has(operationId)) continue;
await assertExecutionWorkspaceRecordAccess(input.db, { type: "agent", source: "agent_jwt",
companyId: input.companyId, agentId: writer.agentId, runId: writer.id,
onBehalfOfUserId: writer.responsibleUserId === "local-board" ? null : writer.responsibleUserId },
input.companyId, remote.workspaceId);
const saved = await getWorkspaceSnapshot(input.db, { companyId: input.companyId,
workspaceId: remote.workspaceId, snapshotId: remote.snapshotId });
workspaceId: remote.workspaceId, snapshotId: proof.snapshotId });
if (!saved || saved.sha256 !== proof.sha256 || saved.generation !== proof.generation) {
skipUnverifiedRecoverySource(input, source.id, "receipt_snapshot_missing");
continue;
@@ -117,6 +134,7 @@ export async function attachPreservedRemoteWorkspaceSnapshots(input: {
await rename(staging, destination);
} finally { await removeTaskDirectoryTree(staging); }
}
attachedVersions.add(version);
result.push({ operationId, path: destination, sourceRunId: source.id });
}
return result;
@@ -2,7 +2,7 @@ import { execFile } from "node:child_process";
import { randomUUID } from "node:crypto";
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import { cp, mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";
import { cp, mkdir, mkdtemp, readdir, readFile, realpath, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
@@ -24,6 +24,7 @@ import { captureRemoteWorkspaceDirectory } from "./remote-workspace-snapshot-arc
import { immutableRemoteWorkspaceRecoveryAttachment, verifiedWorkspaceRepairReceipt } from "./workspace-recovery-receipt.js";
import { remoteWorkspaceReplacementEvidence } from "./native-runtime/native-safe-replacement.js";
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
import { attachPreservedRemoteWorkspaceSnapshots } from "./remote-workspace-recovery-attachments.js";
import {
createSavedWorkspaceLoader, prepareRemoteWorkspace, readRemoteWorkspaceAuthority, reserveRemoteWorkspace, saveRemoteWorkspace,
} from "./remote-workspaces.js";
@@ -113,28 +114,39 @@ describe.skipIf(!support.supported)("captured unchanged workspace snapshots", ()
captures: () => captures - capturesBefore };
}
async function recoveredReuseFixture() {
const f = await fixture(), saved = await f.save();
const run = async () => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.b.runId)))[0]!;
/** Fails a writer after its final save and records the recovery receipt that
* workspace recovery keeps for that saved version. */
async function recoverFailedWriter(f: Awaited<ReturnType<typeof fixture>>, writer: { runId: string; lease: { id: string } },
saved: { snapshotId: string; sha256: string; generation: number }) {
const run = async () => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, writer.runId)))[0]!;
const current = await run(), reference = remoteWorkspaceRunReferenceSchema.parse(current.runnerProfileJson?.remoteWorkspace);
const [originalLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.b.lease.id));
const [originalLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, writer.lease.id));
const [stoppedLease] = await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
metadata: { ...originalLease.metadata, remoteExecutionTermination: remoteTerminationReceipt(originalLease,
{ providerLeaseId: originalLease.providerLeaseId!, state: "stopped" }) } }).where(eq(environmentLeases.id, originalLease.id)).returning();
await db.update(issues).set({ executionWorkspaceId: f.a.workspaceId }).where(eq(issues.id, f.issueId));
await db.update(heartbeatRuns).set({ status: "failed", runtimeMode: "native", nativeIssueId: f.issueId, finishedAt: new Date(),
runnerProfileJson: { ...current.runnerProfileJson, nativeExecutionInput: { workspace: { cwd: reference.remoteCwd } } } })
.where(eq(heartbeatRuns.id, f.b.runId));
.where(eq(heartbeatRuns.id, writer.runId));
// An earlier failed writer's recovery has settled before this one fails.
await db.update(issueRecoveryActions).set({ status: "resolved", resolvedAt: new Date() }).where(and(
eq(issueRecoveryActions.companyId, f.companyId), eq(issueRecoveryActions.sourceIssueId, f.issueId),
eq(issueRecoveryActions.status, "active")));
const [action] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
kind: "stranded_assigned_issue", cause: "fixture", fingerprint: randomUUID(), ownerType: "board",
nextAction: "Continue recovered work", evidence: { workspaceRecovery: { schema: "paperclip.workspace-recovery.v1",
runId: f.b.runId, status: "recovered", stage: "saving_receipts", attempt: 1, requestIds: [] } } }).returning();
runId: writer.runId, status: "recovered", stage: "saving_receipts", attempt: 1, requestIds: [] } } }).returning();
const [receipt] = await db.insert(workspaceOperations).values({ companyId: f.companyId, issueId: f.issueId,
executionWorkspaceId: f.a.workspaceId, heartbeatRunId: f.b.runId, phase: "workspace_recovery", status: "succeeded",
executionWorkspaceId: f.a.workspaceId, heartbeatRunId: writer.runId, phase: "workspace_recovery", status: "succeeded",
metadata: { storage: "remote_workspace", recoveryActionId: action.id, leaseId: reference.leaseId,
providerLeaseId: reference.providerLeaseId, epoch: reference.epoch, snapshotId: saved.snapshotId,
sha256: saved.sha256, generation: saved.generation } }).returning();
return { ...f, saved, run, stoppedLease, receipt };
return { run, stoppedLease, receipt };
}
async function recoveredReuseFixture() {
const f = await fixture(), saved = await f.save();
return { ...f, saved, ...await recoverFailedWriter(f, f.b, saved) };
}
it("accepts a verified reused archive for recovery and stopped native replacement", async () => {
@@ -161,6 +173,42 @@ describe.skipIf(!support.supported)("captured unchanged workspace snapshots", ()
.toEqual({ workspacePreserved: false, providerStopped: true });
});
it.each(["same", "changed"] as const)("attaches each saved version once when two recovered runs saved the %s version", async version => {
// Each failed run that restored a save and saved it again keeps its own
// recovery receipt for that one version. A new workspace receives every
// such run as a predecessor. Staging attached six copies of one 1 GB save
// this way, so each new sandbox filled its disk before its first save.
const f = await recoveredReuseFixture();
const c = await f.writer(f.a.workspaceId);
if (version === "changed") await writeFile(path.join(c.remoteRoot, "saved.txt"), "later recovered result\n");
const savedC = await saveRemoteWorkspace({ db, storage: f.storage, workspace: await f.workspace(), runId: c.runId,
lease: c.lease, target: f.target(c.lease.id), assertOwned: async () => undefined });
expect(savedC.snapshotId === f.saved.snapshotId).toBe(version === "same");
const recoveredC = await recoverFailedWriter(f, c, savedC);
expect(await immutableRemoteWorkspaceRecoveryAttachment(db, await recoveredC.run()))
.toMatchObject({ operationId: recoveredC.receipt.id, snapshotId: savedC.snapshotId });
const [successor] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: f.issueId,
runtimeMode: "legacy", status: "running", responsibleUserId: "local-board", contextSnapshot: { issueId: f.issueId } }).returning();
const scope = { db, storage: f.storage, companyId: f.companyId, issueId: f.issueId, runId: successor.id,
workspaceId: randomUUID(), sourceRunIds: [f.b.runId, c.runId] };
const destination = await realpath(await mkdtemp(path.join(root, "recovered-attachments-")));
const attached = await attachPreservedRemoteWorkspaceSnapshots({ ...scope, cwd: destination });
const copies = await readdir(path.join(destination, ".paperclip-recovered"));
if (version === "same") {
expect(attached).toEqual([expect.objectContaining({ operationId: f.receipt.id, sourceRunId: f.b.runId })]);
expect(copies).toEqual([f.receipt.id]);
} else {
expect(attached.map(item => item.operationId).sort()).toEqual([f.receipt.id, recoveredC.receipt.id].sort());
expect(copies.sort()).toEqual([f.receipt.id, recoveredC.receipt.id].sort());
}
expect(await readFile(path.join(destination, ".paperclip-recovered", f.receipt.id, "files", "saved.txt"), "utf8")).toBe("saved result\n");
// Retiring the copy that was shown retires every receipt of its version.
const later = await realpath(await mkdtemp(path.join(root, "retired-attachments-")));
const remaining = await attachPreservedRemoteWorkspaceSnapshots({ ...scope, cwd: later,
retiredOperationIds: new Set([f.receipt.id]) });
expect(remaining.map(item => item.operationId)).toEqual(version === "same" ? [] : [recoveredC.receipt.id]);
});
it.each(["missing_marker", "pending_generation", "journaled_upload", "wrong_epoch", "wrong_lease",
"unfinished_work", "unknown_work", "wrong_generation", "wrong_boundary", "foreign_run"])(
"refuses an unverified reuse receipt for all recovery readers: %s", async fault => {