fix(workspaces): ignore killed-transfer scratch in sparse checkpoints (#19730)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run in a remote sandbox copies its workspace back to the
host after the turn. A sparse checkpoint copies only changed entries,
and the host validates each entry against the sandbox's own manifest
> - Every Daytona transfer stages a `.paperclip-upload-<uuid>` scratch
file at the sandbox workspace root and removes it only in the same
process
> - A control-plane kill during any transfer leaves that scratch behind.
The sparse checkpoint then recorded it as a workspace file, but
providers never export it. Every later copyback and every saved-work
recovery attempt failed with "Workspace checkpoint payload mismatch"
> - The failed recovery then left an accepted Retry showing "Paperclip
is finishing the saved result" with no run
> - This pull request stops recording transfer scratch, makes the
checkpoint export an exact copy, names the failing entry, and gives a
Retry behind a spent recovery an honest reason and a working action
> - The benefit is that a restart during a transfer no longer makes a
task's saved work permanently unrecoverable

## Linked Issues or Issue Description

Refs #19721 (RC-50: the killed copyback owner; this PR covers the
separate RC-50b cause)

**What happened?**

Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a
SIGKILL of the control plane, the native result was accepted, and then
workspace copyback failed with "Workspace checkpoint payload mismatch".
In s14 all 6 saved-work recovery attempts failed the same way. A Retry
accepted during recovery kept `retryWait.reason =
finalizing_saved_result` after the recovery budget was spent. No run was
created.

**Steps to reproduce**

1. Start a native Claude run in a Daytona sandbox.
2. SIGKILL the control plane while any provider transfer is in flight.
3. Let the run finish its turn and copy its workspace back.

**Expected behavior**

Copyback succeeds. Provider scratch is not workspace content. A Retry
behind a spent recovery names the failure and can start a new recovery.

**Actual behavior**

The s14 sandbox (6f4a0b6a) still held
`/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`.
Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona
provider stages transfer scratch at `remoteDir`, which is the lease
`remoteCwd` (the workspace root). The runtime workspace wipe
deliberately preserves `.paperclip-upload-*`. The checkpoint helper
(`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar.
The ordinary export excludes `.paperclip-upload*`, so the host received
nothing for that entry. `validateCheckpointPayload` then threw "payload
mismatch" on every attempt. The file-mode hypothesis was disproved: the
sandbox umask is 022 and the scratch is mode 644.

## What Changed

- `packages/adapter-utils/src/exclude-patterns.ts`: new
`isSandboxTransferScratchPath`, which matches a root-level
`.paperclip-upload-*` entry. This agrees with the wipe's preserve rule
in `buildWorkspaceTarExtractCommand`.
- `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox
checkpoint helper skips that scratch. A mismatch error now names the
entry and its recorded, baseline and received forms (kind, octal mode,
short hash; never contents).
- `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the
checkpoint export requests `preserveSourceEntries: true`. The checkpoint
output is an exact copy that the host validates entry by entry. An
ordinary export applies the host umask and drops `._*` and
`.paperclip-upload*` names at any depth. That gives the same permanent
mismatch for a legitimate user file such as `._summary`,
`notes/.paperclip-upload-plan.md`, or a `0775` script.
- `server/src/services/explicit-native-continuation.ts` and
`task-interruption-presentation.ts`: when the run's saved-work recovery
is exhausted, a waiting Retry reports `saved_work_recovery_failed` with
Retry available, instead of `finalizing_saved_result`. The existing task
Retry path starts one new recovery generation.
- `doc/execution-semantics.md`: a short additive note.

## Verification

- `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses
the real checkpoint helper and an export that behaves like Daytona's
ordinary export:
- Before the fix: `Workspace checkpoint payload mismatch at
".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …;
baseline absent; received absent`. That is the wave error.
  - After the fix: the scratch is not recorded, and validation passes.
- A second case shows that reserved-looking user names and `0775` modes
need the exact export.
- `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an
end-to-end `prepareSandboxManagedRuntime` restore through a provider
model. In that model, ordinary exports drop reserved names and apply the
umask, and preserve exports copy exactly. Two red checks:
- Without `preserveSourceEntries`, it fails with the mismatch on
`._summary`.
- Without the scratch skip, the stale tar leaks into the host workspace.
- Green: the agent file, the `._summary` file and the `0775` mode
arrive, and no scratch arrives.
- `server/src/services/workspace-recovery-preparation-gates.test.ts`,
"names a spent saved-work recovery behind an accepted Retry, and the
next Retry finishes it" (s14 shape):
  - Red: `finalizing_saved_result` after 6 failed attempts.
- Green: `saved_work_recovery_failed` with `canRetry: true`. The next
Retry starts a new generation, recovery completes, and the Retry
continues in exactly one new run.
- Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime,
seed-reuse and managed-seed (168 passed). Server preparation-gates (47),
explicit-native-continuation and task-interruption-presentation (455),
workspace-recovery admission and retry-copyback, and
native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils
and server.
- Live evidence: I started the stopped s14 sandbox read-only, listed its
workspace, and stopped it again. No Daytona replay was run. The runner
content ID of this checkout matches no built image.

## Risks

- A user file named `.paperclip-upload-*` directly at the workspace root
is no longer captured by a sparse checkpoint. Providers already drop
that name from every ordinary export, and the runtime already reserves
it at the root.
- The checkpoint export now uses the provider's exact-copy mode. A
workspace file with setuid, setgid or sticky bits is refused by that
mode, as it is for the existing exact-copy transfers. The ordinary
export would have stripped those bits and failed validation anyway.
- The stale scratch tar stays in the sandbox. It is ignored, not
deleted, because a concurrent transfer can own a file with the same
prefix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs, embedded Postgres, a
read-only Daytona sandbox listing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-06 22:46:16 -06:00
1 parent b21884bdcf
commit c45c7cb854
9 files changed
+212 -11

No files matched your search

+6
View File
@@ -3660,6 +3660,12 @@ Runner command and receipt size limits remain unchanged.
Sparse remote workspace checkpoints capture at most eight regular files at once.
Each file retains the content hash and mutation checks. A failed capture settles
its active readers before closing the manifest and publishes no success receipt.
A capture never records provider transfer scratch (`.paperclip-upload-*` at the
workspace root). A control-plane kill during a transfer can leave that scratch
behind, and providers never export it. The checkpoint export requests an exact
copy (`preserveSourceEntries`), so file modes and names such as `._*` arrive as
recorded. A mismatch error names the entry and its recorded, baseline and
received forms; it never includes file contents.
### Warm sandbox continuity
@@ -23,6 +23,18 @@ export function excludePatternMatches(relative: string, pattern: string): boolea
return isRelativePathOrDescendant(relative, pattern);
}
/**
* Sandbox providers stage transfer scratch at the workspace root as
* `.paperclip-upload-<uuid>…` (see `buildWorkspaceTarExtractCommand` and the
* Daytona/Kubernetes `SCRATCH_PREFIX`). A control-plane kill during a transfer
* leaves it behind, and providers never export it. It is not workspace content.
* The literal stays inline: this function is serialized into the sandbox
* checkpoint helper.
*/
export function isSandboxTransferScratchPath(relative: string): boolean {
return !relative.includes("/") && relative.startsWith(".paperclip-upload-");
}
export function shouldExcludePath(relative: string, exclude: readonly string[]): boolean {
return exclude.some((entry) => excludePatternMatches(relative, entry));
}
@@ -2227,8 +2227,11 @@ export async function prepareSandboxManagedRuntime(input: {
}
if (captured) {
const output = path.join(tempDir, "checkpoint");
// The checkpoint output is an exact copy that the host validates entry
// by entry. Ordinary export applies the host umask and drops `._*` names,
// which turns a legitimate workspace file into a permanent mismatch.
const outbound: SandboxSyncOperation[] = [{ operationId: nextSyncOperationId(), files: [
{ sourcePath: remoteOutput, targetPath: output, kind: "directory" },
{ sourcePath: remoteOutput, targetPath: output, kind: "directory", preserveSourceEntries: true },
] }];
assertSyncOperationsConfined(outbound, { sourceRoots: [runtimeRootDir], targetRoots: [tempDir] });
await withWorkspaceRestoreStep("workspace_transfer", () => input.client.syncOut!(outbound));
@@ -171,3 +171,61 @@ it("drains active readers before closing either database after a walk failure",
expect(stats.databaseCloseActive).toEqual([0, 0]);
await expect(fs.stat(path.join(f.output, "receipt.json"))).rejects.toMatchObject({ code: "ENOENT" });
});
// Daytona's ordinary directory export (no preserveSourceEntries): GNU tar
// `--exclude '._*' --exclude '.paperclip-upload*'` at any depth, extracted on
// the host without `-p`, so the host umask (022) applies.
async function ordinaryProviderExport(source: string, target: string) {
await fs.cp(source, target, { recursive: true, verbatimSymlinks: true,
filter: entry => !/^(\._|\.paperclip-upload)/.test(path.basename(entry)) });
const strip = async (directory: string): Promise<void> => {
for (const child of await fs.readdir(directory, { withFileTypes: true })) {
const full = path.join(directory, child.name);
if (child.isDirectory()) await strip(full);
else if (child.isFile()) await fs.chmod(full, (await fs.stat(full)).mode & 0o7777 & ~0o022);
}
};
await strip(target);
}
// RC-50b (waves s12/s14): a control-plane kill during a provider transfer left
// `.paperclip-upload-<uuid>.tar` at the sandbox workspace root (found in the s14
// sandbox, mtime equal to the SIGKILL). Providers never export that scratch, so
// recording it made every later checkpoint validation fail.
it("never records provider transfer scratch left at the workspace root by a killed transfer", async () => {
const f = await fixture("none");
await fs.writeFile(path.join(f.root, ".paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar"), Buffer.alloc(4096, 7));
await fs.writeFile(path.join(f.root, "file-00"), "changed by the agent");
await f.run();
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
snapshots.push(snapshot);
const received = path.join(path.dirname(f.output), "received");
await ordinaryProviderExport(path.join(f.output, "payload"), received);
// Before the fix: "payload mismatch at ".paperclip-upload-….tar": … received absent".
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: received }))
.toMatchObject({ changedFiles: 1 });
expect([...snapshot.entries].map(([name]) => name).filter(name => name.startsWith(".paperclip-upload"))).toEqual([]);
await expect(fs.readdir(received)).resolves.toEqual(["file-00"]);
});
it("needs an exact export for reserved-looking user names and group-writable modes", async () => {
const f = await fixture("none");
await fs.mkdir(path.join(f.root, "notes"));
await fs.writeFile(path.join(f.root, "notes", ".paperclip-upload-plan.md"), "user notes");
await fs.writeFile(path.join(f.root, "._summary"), "user summary");
await fs.writeFile(path.join(f.root, "build.sh"), "#!/bin/sh\n");
await fs.chmod(path.join(f.root, "build.sh"), 0o775);
await f.run();
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
snapshots.push(snapshot);
expect(snapshot.entries.get("notes/.paperclip-upload-plan.md")).toMatchObject({ kind: "file" });
const ordinary = path.join(path.dirname(f.output), "ordinary");
await ordinaryProviderExport(path.join(f.output, "payload"), ordinary);
await expect(validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: ordinary }))
.rejects.toThrow(/payload mismatch at ".*": sandbox recorded file mode=\d+ sha256=\w+; baseline absent; received (absent|file mode=755)/);
// The runtime requests preserveSourceEntries for this export: an exact copy.
const exact = path.join(path.dirname(f.output), "exact");
await fs.cp(path.join(f.output, "payload"), exact, { recursive: true, verbatimSymlinks: true });
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: exact }))
.toMatchObject({ changedFiles: 3 });
});
@@ -453,3 +453,54 @@ it.each(["full_root", "full_special"] as const)("rejects unsafe %s data before h
expect(await fs.readFile(f.outside, "utf8")).toBe("private");
await f.prepared.cleanupWorkspaceSnapshot();
}, 30000);
// RC-50b: models the Daytona provider. Every transfer stages its own
// `.paperclip-upload-*` scratch at the workspace root, and a SIGKILL during a
// transfer leaves it there. An ordinary export drops `._*` and
// `.paperclip-upload*` names at any depth and extracts under the host umask;
// a preserveSourceEntries export copies the source exactly.
it("restores a sparse checkpoint after a killed transfer left provider scratch, keeping exact entries", async () => {
const { prepareSandboxManagedRuntime } = await import("./sandbox-managed-runtime.js");
const f = await fixture(); await disposeDirectorySnapshot(f.baseline);
const remote = path.join(f.temp, "remote");
const reserved = /^(\._|\.paperclip-upload)/;
const run = async (command: string) => { await execute("sh", ["-c", command]); };
type Operations = import("./sandbox-managed-runtime.js").SandboxSyncOperation[];
const stripUmask = async (directory: string): Promise<void> => {
for (const child of await fs.readdir(directory, { withFileTypes: true })) {
const full = path.join(directory, child.name);
if (child.isDirectory()) await stripUmask(full);
else if (child.isFile()) await fs.chmod(full, (await fs.stat(full)).mode & 0o7777 & ~0o022);
}
};
const transfer = (outbound: boolean) => async (operations: Operations) => {
for (const operation of operations) {
for (const file of operation.files) {
const ordinaryExport = outbound && file.preserveSourceEntries !== true;
await fs.mkdir(path.dirname(file.targetPath), { recursive: true });
await fs.cp(file.sourcePath, file.targetPath, { recursive: true, verbatimSymlinks: true,
filter: (source) => !file.exclude?.some((exclude) => path.relative(file.sourcePath, source) === exclude)
&& !(ordinaryExport && reserved.test(path.basename(source))) });
if (ordinaryExport && (await fs.lstat(file.targetPath)).isDirectory()) await stripUmask(file.targetPath);
}
for (const command of operation.postUploadCommands ?? []) await run(command.command);
}
return { operations: operations.map((op) => ({ operationId: op.operationId, filesTransferred: 0, bytesTransferred: 0 })) };
};
const client = { makeDir: (dir: string) => fs.mkdir(dir, { recursive: true }).then(() => {}),
writeFile: (file: string, data: ArrayBuffer) => fs.writeFile(file, Buffer.from(data)), readFile: (file: string) => fs.readFile(file),
listFiles: (dir: string) => fs.readdir(dir), remove: (file: string) => fs.rm(file, { force: true, recursive: true }), run,
syncIn: transfer(false), syncOut: transfer(true) };
const prepared = await prepareSandboxManagedRuntime({ spec: { provider: "test", sandboxId: "test", remoteCwd: remote, timeoutMs: 30000, apiKey: null, transport: "sandbox" },
adapterKey: "test", client, workspaceLocalDir: f.root, workspaceCheckpoint: true });
// The s14 sandbox kept this exact file; its mtime equals the SIGKILL.
await fs.writeFile(path.join(remote, ".paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar"), Buffer.alloc(2048, 1));
await fs.writeFile(path.join(remote, "new"), "agent work");
await fs.writeFile(path.join(remote, "._summary"), "user summary");
await fs.writeFile(path.join(remote, "build.sh"), "#!/bin/sh\n"); await fs.chmod(path.join(remote, "build.sh"), 0o775);
await prepared.restoreWorkspace();
expect(await fs.readFile(path.join(f.root, "new"), "utf8")).toBe("agent work");
expect(await fs.readFile(path.join(f.root, "._summary"), "utf8")).toBe("user summary");
expect((await fs.stat(path.join(f.root, "build.sh"))).mode & 0o777).toBe(0o775);
expect((await fs.readdir(f.root)).filter(name => name.startsWith(".paperclip-upload"))).toEqual([]);
}, 30000);
@@ -2,7 +2,7 @@ import fs from "node:fs/promises";
import path from "node:path";
import { DatabaseSync } from "node:sqlite";
import { WorkspaceManifestWriter, WorkspaceManifestMap, workspacePaths, workspacePathMatcher } from "./workspace-manifest.js";
import { shouldExcludePath, excludePatternMatches, isRelativePathOrDescendant } from "./exclude-patterns.js";
import { shouldExcludePath, excludePatternMatches, isRelativePathOrDescendant, isSandboxTransferScratchPath } from "./exclude-patterns.js";
import { openDirectorySnapshot, type DirectorySnapshot } from "./workspace-restore-merge.js";
export async function writeCheckpointBaseline(snapshot: DirectorySnapshot, filePath: string): Promise<void> {
@@ -20,6 +20,7 @@ async function checkpointMain(
createHash: typeof import("node:crypto").createHash,
createReadStream: typeof import("node:fs").createReadStream,
shouldExcludePath: (relative: string, exclude: readonly string[]) => boolean,
isSandboxTransferScratchPath: (relative: string) => boolean,
) {
const [rootArg, baselinePath, outputDir, exclusions] = process.argv.slice(2);
const root = await fs.realpath(rootArg!);
@@ -89,7 +90,9 @@ async function checkpointMain(
for (let p = name; p; p = p.includes("/") ? p.slice(0, p.lastIndexOf("/")) : "") {
if (ignored.get(p)) { ignoredPath = true; break; }
}
if (ignoredPath || shouldExcludePath(name, exclude)) continue;
// Provider transfer scratch is never exported, so capturing it makes
// every later validation fail (RC-50b).
if (ignoredPath || shouldExcludePath(name, exclude) || isSandboxTransferScratchPath(name)) continue;
const before = await fs.lstat(path.join(root, name));
scanned++;
if (before.isFile()) {
@@ -114,7 +117,7 @@ async function checkpointMain(
}
export function workspaceCheckpointScript(): string {
return `import fs from 'node:fs/promises';\nimport path from 'node:path';\nimport { DatabaseSync } from 'node:sqlite';\nimport { createHash } from 'node:crypto';\nimport { createReadStream } from 'node:fs';\nconst __name = (fn) => fn;\n${isRelativePathOrDescendant.toString()}\nfunction pathContainsSegmentOrDescendant(relative, segment) { return relative === segment || relative.startsWith(segment + '/') || relative.endsWith('/' + segment) || relative.includes('/' + segment + '/'); }\n${excludePatternMatches.toString()}\n${shouldExcludePath.toString()}\nawait (${checkpointMain.toString()})(fs, path, DatabaseSync, createHash, createReadStream, shouldExcludePath);\n`;
return `import fs from 'node:fs/promises';\nimport path from 'node:path';\nimport { DatabaseSync } from 'node:sqlite';\nimport { createHash } from 'node:crypto';\nimport { createReadStream } from 'node:fs';\nconst __name = (fn) => fn;\n${isRelativePathOrDescendant.toString()}\nfunction pathContainsSegmentOrDescendant(relative, segment) { return relative === segment || relative.startsWith(segment + '/') || relative.endsWith('/' + segment) || relative.includes('/' + segment + '/'); }\n${excludePatternMatches.toString()}\n${shouldExcludePath.toString()}\n${isSandboxTransferScratchPath.toString()}\nawait (${checkpointMain.toString()})(fs, path, DatabaseSync, createHash, createReadStream, shouldExcludePath, isSandboxTransferScratchPath);\n`;
}
export function readCheckpointSnapshot(filePath: string, baseline: DirectorySnapshot): DirectorySnapshot {
@@ -131,6 +134,21 @@ export function readCheckpointSnapshot(filePath: string, baseline: DirectorySnap
return snapshot;
}
type CheckpointEntryLike = { kind: string; mode?: number; hash?: string; target?: string } | undefined;
/** Names which entry disagreed and how, so a deterministic mismatch is diagnosable.
* Modes are octal permission bits; hashes are shortened. File contents never appear. */
function checkpointPayloadMismatch(relative: string, recorded: CheckpointEntryLike, baseline: CheckpointEntryLike, received: CheckpointEntryLike) {
const describe = (entry: CheckpointEntryLike) => !entry ? "absent"
: entry.kind === "file" ? `file mode=${((entry.mode ?? 0) & 0o7777).toString(8)} sha256=${String(entry.hash).slice(0, 12)}`
: entry.kind === "symlink" ? "symlink" : entry.kind;
const shownPath = relative.length > 240 ? `${relative.slice(0, 240)}…` : relative;
return Object.assign(new Error(`Workspace checkpoint payload mismatch at ${JSON.stringify(shownPath)}: `
+ `sandbox recorded ${describe(recorded)}; baseline ${describe(baseline)}; received ${describe(received)}`), {
checkpointMismatch: { path: relative, recorded, baseline, received },
});
}
export async function validateCheckpointPayload(input: { baseline: DirectorySnapshot; snapshot: DirectorySnapshot; payload: string }): Promise<WorkspaceCheckpointMetrics> {
const { captureDirectorySnapshot, disposeDirectorySnapshot } = await import("./workspace-restore-merge.js");
const ignored = workspacePathMatcher(input.baseline.ignoredPaths);
@@ -156,7 +174,9 @@ export async function validateCheckpointPayload(input: { baseline: DirectorySnap
throw Object.assign(new Error("Unsafe workspace checkpoint symlink"), { code: "WORKSPACE_RESTORE_UNSAFE_ARCHIVE" });
}
}
if (!equal(entry, input.baseline.entries.get(relative)) && !equal(entry, actual.entries.get(relative))) throw new Error("Workspace checkpoint payload mismatch");
if (!equal(entry, input.baseline.entries.get(relative)) && !equal(entry, actual.entries.get(relative))) {
throw checkpointPayloadMismatch(relative, entry, input.baseline.entries.get(relative), actual.entries.get(relative));
}
}
for (const [relative, entry] of actual.entries) {
assertIncluded(relative);
@@ -405,15 +405,23 @@ export async function admitExplicitNativeContinuation(input: {
// Nothing finishes the saved result while a stopped copyback owner is
// unverified. The run's live retained recovery records that hold; name
// its board action instead of an endless finalizing wait (RC-50).
const [ownerHold] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
const live = await db.select({
ownerHold: sql<boolean>`${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' = 'operator_required'
and ${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`,
recoveryState: sql<string | null>`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}'`,
}).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
inArray(issueRecoveryActions.status, ["active", "escalated"]),
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
sql`${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' = 'operator_required'`,
sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`)).limit(1);
return ownerHold
? blocked("workspace_owner_stop_required", "A board member must confirm that the previous server stopped copying files. Your Retry is saved.")
: blocked("finalizing_saved_result", "Paperclip is finishing the saved result.");
sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`));
if (live.some(row => row.ownerHold))
return blocked("workspace_owner_stop_required", "A board member must confirm that the previous server stopped copying files. Your Retry is saved.");
// A spent recovery budget finishes nothing by itself. Another Retry
// starts one new recovery generation (task-retry), so name that action
// instead of an endless finalizing wait (RC-50b, s14).
if (live.length && live.every(row => row.recoveryState === "exhausted"))
return blocked("saved_work_recovery_failed", "Saving the agent's files failed after automatic retries. Retry to try again.");
return blocked("finalizing_saved_result", "Paperclip is finishing the saved result.");
}
}
// A repaired export no longer blocks execution. Its machine receipt still
@@ -83,6 +83,8 @@ const retryWaitMessages: Record<string, string> = {
"budget.blocked": "Retry is waiting for the budget to allow new work. Check Costs for unrecorded or unpriced usage.",
finalizing_saved_result: "Paperclip is finishing the saved result before Retry can continue.",
// Another click cannot release a copyback owner that stopped without reporting.
// Retry starts a new recovery generation; the agent does not repeat its work.
saved_work_recovery_failed: "Saving the agent's files failed after automatic retries. Retry to try again; the agent will not repeat its work.",
workspace_owner_stop_required: "Retry is waiting for a board member to confirm that the previous server stopped copying files. The agent's result and your Retry are saved.",
cleanup_quarantined: "Retry cannot continue this attempt. Send a new message after the previous execution has stopped.",
cancelled_by_operator: "This attempt was stopped. Send a new message to continue from saved work.",
@@ -362,6 +362,47 @@ describe("automatic retained workspace recovery", () => {
}
}, 60_000);
// RC-50b (wave s14/t05): Retry was accepted while saved-work recovery ran.
// Every export attempt failed, the budget was spent, and the Retry still said
// "finishing the saved result" with no run. It must name the failure and a
// second Retry must start one new recovery that finishes the saved result.
it("names a spent saved-work recovery behind an accepted Retry, and the next Retry finishes it", async () => {
const f = await acceptedRemoteExport();
const issue = (await db.select().from(issues).where(eq(issues.id, f.issueId)))[0];
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId, status: "running" });
const app = boardRecoveryApp();
const first = (await request(app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body;
f.runtime.resumeRunLease.mockRejectedValue(new Error("Workspace checkpoint payload mismatch"));
for (let attempt = 0; attempt < 6; attempt++) {
await recoverRetainedWorkspace(f.input);
await heartbeatService(db).resumeTaskRetry(f.companyId, first.requestId!);
await makeRetryDue(f.actionId);
}
expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "exhausted", attempts: 6 });
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, first.requestId!)))[0])
.toMatchObject({ status: "deferred_issue_execution", payload: { executionWait: { reason: "saved_work_recovery_failed" } } });
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true,
retryWait: { reason: "saved_work_recovery_failed" } });
// The working action: one more Retry starts one new recovery generation.
f.runtime.resumeRunLease.mockReset();
f.runtime.resumeRunLease.mockImplementation(async () => ({ providerLeaseId: f.providerLeaseId, metadata: { remoteCwd: await realpath(f.remote) } }));
const second = (await request(app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body;
expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "scheduled", attempts: 0 });
await recoverRetainedWorkspace(f.input);
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
await resumePreservedResultFinalization(db, f.companyId, f.runId);
await heartbeatService(db).resumeTaskRetry(f.companyId, second.requestId!);
// After a board handoff the Retry continues from the recovered files in
// exactly one new run (as in the s12 case), never another wait.
const [settled] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, second.requestId!));
expect(settled).toMatchObject({ status: "coalesced", runId: expect.any(String), payload: { taskRetry: { strategy: "continue" } } });
expect(await db.select({ id: heartbeatRuns.id, previousRunId: sql<string>`${heartbeatRuns.contextSnapshot}->>'previousRunId'` })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))))
.toEqual([{ id: settled.runId, previousRunId: f.runId }]);
}, 60_000);
it("preserves an explicitly requested superseded remote export without applying its result over newer work", async () => {
const f = await acceptedRemoteExport();
const [later] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId,