From c45c7cb854faf67fbee079a3f2e4a25108b98f65 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 22:46:16 -0600 Subject: [PATCH] fix(workspaces): ignore killed-transfer scratch in sparse checkpoints (#19730) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run in a remote sandbox copies its workspace back to the host after the turn. A sparse checkpoint copies only changed entries, and the host validates each entry against the sandbox's own manifest > - Every Daytona transfer stages a `.paperclip-upload-` scratch file at the sandbox workspace root and removes it only in the same process > - A control-plane kill during any transfer leaves that scratch behind. The sparse checkpoint then recorded it as a workspace file, but providers never export it. Every later copyback and every saved-work recovery attempt failed with "Workspace checkpoint payload mismatch" > - The failed recovery then left an accepted Retry showing "Paperclip is finishing the saved result" with no run > - This pull request stops recording transfer scratch, makes the checkpoint export an exact copy, names the failing entry, and gives a Retry behind a spent recovery an honest reason and a working action > - The benefit is that a restart during a transfer no longer makes a task's saved work permanently unrecoverable ## Linked Issues or Issue Description Refs #19721 (RC-50: the killed copyback owner; this PR covers the separate RC-50b cause) **What happened?** Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a SIGKILL of the control plane, the native result was accepted, and then workspace copyback failed with "Workspace checkpoint payload mismatch". In s14 all 6 saved-work recovery attempts failed the same way. A Retry accepted during recovery kept `retryWait.reason = finalizing_saved_result` after the recovery budget was spent. No run was created. **Steps to reproduce** 1. Start a native Claude run in a Daytona sandbox. 2. SIGKILL the control plane while any provider transfer is in flight. 3. Let the run finish its turn and copy its workspace back. **Expected behavior** Copyback succeeds. Provider scratch is not workspace content. A Retry behind a spent recovery names the failure and can start a new recovery. **Actual behavior** The s14 sandbox (6f4a0b6a) still held `/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`. Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona provider stages transfer scratch at `remoteDir`, which is the lease `remoteCwd` (the workspace root). The runtime workspace wipe deliberately preserves `.paperclip-upload-*`. The checkpoint helper (`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar. The ordinary export excludes `.paperclip-upload*`, so the host received nothing for that entry. `validateCheckpointPayload` then threw "payload mismatch" on every attempt. The file-mode hypothesis was disproved: the sandbox umask is 022 and the scratch is mode 644. ## What Changed - `packages/adapter-utils/src/exclude-patterns.ts`: new `isSandboxTransferScratchPath`, which matches a root-level `.paperclip-upload-*` entry. This agrees with the wipe's preserve rule in `buildWorkspaceTarExtractCommand`. - `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox checkpoint helper skips that scratch. A mismatch error now names the entry and its recorded, baseline and received forms (kind, octal mode, short hash; never contents). - `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the checkpoint export requests `preserveSourceEntries: true`. The checkpoint output is an exact copy that the host validates entry by entry. An ordinary export applies the host umask and drops `._*` and `.paperclip-upload*` names at any depth. That gives the same permanent mismatch for a legitimate user file such as `._summary`, `notes/.paperclip-upload-plan.md`, or a `0775` script. - `server/src/services/explicit-native-continuation.ts` and `task-interruption-presentation.ts`: when the run's saved-work recovery is exhausted, a waiting Retry reports `saved_work_recovery_failed` with Retry available, instead of `finalizing_saved_result`. The existing task Retry path starts one new recovery generation. - `doc/execution-semantics.md`: a short additive note. ## Verification - `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses the real checkpoint helper and an export that behaves like Daytona's ordinary export: - Before the fix: `Workspace checkpoint payload mismatch at ".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …; baseline absent; received absent`. That is the wave error. - After the fix: the scratch is not recorded, and validation passes. - A second case shows that reserved-looking user names and `0775` modes need the exact export. - `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an end-to-end `prepareSandboxManagedRuntime` restore through a provider model. In that model, ordinary exports drop reserved names and apply the umask, and preserve exports copy exactly. Two red checks: - Without `preserveSourceEntries`, it fails with the mismatch on `._summary`. - Without the scratch skip, the stale tar leaks into the host workspace. - Green: the agent file, the `._summary` file and the `0775` mode arrive, and no scratch arrives. - `server/src/services/workspace-recovery-preparation-gates.test.ts`, "names a spent saved-work recovery behind an accepted Retry, and the next Retry finishes it" (s14 shape): - Red: `finalizing_saved_result` after 6 failed attempts. - Green: `saved_work_recovery_failed` with `canRetry: true`. The next Retry starts a new generation, recovery completes, and the Retry continues in exactly one new run. - Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime, seed-reuse and managed-seed (168 passed). Server preparation-gates (47), explicit-native-continuation and task-interruption-presentation (455), workspace-recovery admission and retry-copyback, and native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils and server. - Live evidence: I started the stopped s14 sandbox read-only, listed its workspace, and stopped it again. No Daytona replay was run. The runner content ID of this checkout matches no built image. ## Risks - A user file named `.paperclip-upload-*` directly at the workspace root is no longer captured by a sparse checkpoint. Providers already drop that name from every ordinary export, and the runtime already reserves it at the root. - The checkpoint export now uses the provider's exact-copy mode. A workspace file with setuid, setgid or sticky bits is refused by that mode, as it is for the existing exact-copy transfers. The ordinary export would have stripped those bits and failed validation anyway. - The stale scratch tar stays in the sandbox. It is ignored, not deleted, because a concurrent transfer can own a file with the same prefix. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs, embedded Postgres, a read-only Daytona sandbox listing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 6 ++ .../adapter-utils/src/exclude-patterns.ts | 12 ++++ .../src/sandbox-managed-runtime.ts | 5 +- .../src/workspace-checkpoint-capture.test.ts | 58 +++++++++++++++++++ .../src/workspace-checkpoint.test.ts | 51 ++++++++++++++++ .../adapter-utils/src/workspace-checkpoint.ts | 28 +++++++-- .../services/explicit-native-continuation.ts | 20 +++++-- .../task-interruption-presentation.ts | 2 + ...rkspace-recovery-preparation-gates.test.ts | 41 +++++++++++++ 9 files changed, 212 insertions(+), 11 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 5ecc0a6309b..0affca3c77c 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -3660,6 +3660,12 @@ Runner command and receipt size limits remain unchanged. Sparse remote workspace checkpoints capture at most eight regular files at once. Each file retains the content hash and mutation checks. A failed capture settles its active readers before closing the manifest and publishes no success receipt. +A capture never records provider transfer scratch (`.paperclip-upload-*` at the +workspace root). A control-plane kill during a transfer can leave that scratch +behind, and providers never export it. The checkpoint export requests an exact +copy (`preserveSourceEntries`), so file modes and names such as `._*` arrive as +recorded. A mismatch error names the entry and its recorded, baseline and +received forms; it never includes file contents. ### Warm sandbox continuity diff --git a/packages/adapter-utils/src/exclude-patterns.ts b/packages/adapter-utils/src/exclude-patterns.ts index 5cc9eeb1a85..8850f655a14 100644 --- a/packages/adapter-utils/src/exclude-patterns.ts +++ b/packages/adapter-utils/src/exclude-patterns.ts @@ -23,6 +23,18 @@ export function excludePatternMatches(relative: string, pattern: string): boolea return isRelativePathOrDescendant(relative, pattern); } +/** + * Sandbox providers stage transfer scratch at the workspace root as + * `.paperclip-upload-…` (see `buildWorkspaceTarExtractCommand` and the + * Daytona/Kubernetes `SCRATCH_PREFIX`). A control-plane kill during a transfer + * leaves it behind, and providers never export it. It is not workspace content. + * The literal stays inline: this function is serialized into the sandbox + * checkpoint helper. + */ +export function isSandboxTransferScratchPath(relative: string): boolean { + return !relative.includes("/") && relative.startsWith(".paperclip-upload-"); +} + export function shouldExcludePath(relative: string, exclude: readonly string[]): boolean { return exclude.some((entry) => excludePatternMatches(relative, entry)); } diff --git a/packages/adapter-utils/src/sandbox-managed-runtime.ts b/packages/adapter-utils/src/sandbox-managed-runtime.ts index 026b992786c..91cc2d7d87b 100644 --- a/packages/adapter-utils/src/sandbox-managed-runtime.ts +++ b/packages/adapter-utils/src/sandbox-managed-runtime.ts @@ -2227,8 +2227,11 @@ export async function prepareSandboxManagedRuntime(input: { } if (captured) { const output = path.join(tempDir, "checkpoint"); + // The checkpoint output is an exact copy that the host validates entry + // by entry. Ordinary export applies the host umask and drops `._*` names, + // which turns a legitimate workspace file into a permanent mismatch. const outbound: SandboxSyncOperation[] = [{ operationId: nextSyncOperationId(), files: [ - { sourcePath: remoteOutput, targetPath: output, kind: "directory" }, + { sourcePath: remoteOutput, targetPath: output, kind: "directory", preserveSourceEntries: true }, ] }]; assertSyncOperationsConfined(outbound, { sourceRoots: [runtimeRootDir], targetRoots: [tempDir] }); await withWorkspaceRestoreStep("workspace_transfer", () => input.client.syncOut!(outbound)); diff --git a/packages/adapter-utils/src/workspace-checkpoint-capture.test.ts b/packages/adapter-utils/src/workspace-checkpoint-capture.test.ts index 8da42a827eb..a9c1fe4e507 100644 --- a/packages/adapter-utils/src/workspace-checkpoint-capture.test.ts +++ b/packages/adapter-utils/src/workspace-checkpoint-capture.test.ts @@ -171,3 +171,61 @@ it("drains active readers before closing either database after a walk failure", expect(stats.databaseCloseActive).toEqual([0, 0]); await expect(fs.stat(path.join(f.output, "receipt.json"))).rejects.toMatchObject({ code: "ENOENT" }); }); + +// Daytona's ordinary directory export (no preserveSourceEntries): GNU tar +// `--exclude '._*' --exclude '.paperclip-upload*'` at any depth, extracted on +// the host without `-p`, so the host umask (022) applies. +async function ordinaryProviderExport(source: string, target: string) { + await fs.cp(source, target, { recursive: true, verbatimSymlinks: true, + filter: entry => !/^(\._|\.paperclip-upload)/.test(path.basename(entry)) }); + const strip = async (directory: string): Promise => { + for (const child of await fs.readdir(directory, { withFileTypes: true })) { + const full = path.join(directory, child.name); + if (child.isDirectory()) await strip(full); + else if (child.isFile()) await fs.chmod(full, (await fs.stat(full)).mode & 0o7777 & ~0o022); + } + }; + await strip(target); +} + +// RC-50b (waves s12/s14): a control-plane kill during a provider transfer left +// `.paperclip-upload-.tar` at the sandbox workspace root (found in the s14 +// sandbox, mtime equal to the SIGKILL). Providers never export that scratch, so +// recording it made every later checkpoint validation fail. +it("never records provider transfer scratch left at the workspace root by a killed transfer", async () => { + const f = await fixture("none"); + await fs.writeFile(path.join(f.root, ".paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar"), Buffer.alloc(4096, 7)); + await fs.writeFile(path.join(f.root, "file-00"), "changed by the agent"); + await f.run(); + const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline); + snapshots.push(snapshot); + const received = path.join(path.dirname(f.output), "received"); + await ordinaryProviderExport(path.join(f.output, "payload"), received); + // Before the fix: "payload mismatch at ".paperclip-upload-….tar": … received absent". + expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: received })) + .toMatchObject({ changedFiles: 1 }); + expect([...snapshot.entries].map(([name]) => name).filter(name => name.startsWith(".paperclip-upload"))).toEqual([]); + await expect(fs.readdir(received)).resolves.toEqual(["file-00"]); +}); + +it("needs an exact export for reserved-looking user names and group-writable modes", async () => { + const f = await fixture("none"); + await fs.mkdir(path.join(f.root, "notes")); + await fs.writeFile(path.join(f.root, "notes", ".paperclip-upload-plan.md"), "user notes"); + await fs.writeFile(path.join(f.root, "._summary"), "user summary"); + await fs.writeFile(path.join(f.root, "build.sh"), "#!/bin/sh\n"); + await fs.chmod(path.join(f.root, "build.sh"), 0o775); + await f.run(); + const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline); + snapshots.push(snapshot); + expect(snapshot.entries.get("notes/.paperclip-upload-plan.md")).toMatchObject({ kind: "file" }); + const ordinary = path.join(path.dirname(f.output), "ordinary"); + await ordinaryProviderExport(path.join(f.output, "payload"), ordinary); + await expect(validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: ordinary })) + .rejects.toThrow(/payload mismatch at ".*": sandbox recorded file mode=\d+ sha256=\w+; baseline absent; received (absent|file mode=755)/); + // The runtime requests preserveSourceEntries for this export: an exact copy. + const exact = path.join(path.dirname(f.output), "exact"); + await fs.cp(path.join(f.output, "payload"), exact, { recursive: true, verbatimSymlinks: true }); + expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: exact })) + .toMatchObject({ changedFiles: 3 }); +}); diff --git a/packages/adapter-utils/src/workspace-checkpoint.test.ts b/packages/adapter-utils/src/workspace-checkpoint.test.ts index 064cfa6eeae..2be8c2882d2 100644 --- a/packages/adapter-utils/src/workspace-checkpoint.test.ts +++ b/packages/adapter-utils/src/workspace-checkpoint.test.ts @@ -453,3 +453,54 @@ it.each(["full_root", "full_special"] as const)("rejects unsafe %s data before h expect(await fs.readFile(f.outside, "utf8")).toBe("private"); await f.prepared.cleanupWorkspaceSnapshot(); }, 30000); + +// RC-50b: models the Daytona provider. Every transfer stages its own +// `.paperclip-upload-*` scratch at the workspace root, and a SIGKILL during a +// transfer leaves it there. An ordinary export drops `._*` and +// `.paperclip-upload*` names at any depth and extracts under the host umask; +// a preserveSourceEntries export copies the source exactly. +it("restores a sparse checkpoint after a killed transfer left provider scratch, keeping exact entries", async () => { + const { prepareSandboxManagedRuntime } = await import("./sandbox-managed-runtime.js"); + const f = await fixture(); await disposeDirectorySnapshot(f.baseline); + const remote = path.join(f.temp, "remote"); + const reserved = /^(\._|\.paperclip-upload)/; + const run = async (command: string) => { await execute("sh", ["-c", command]); }; + type Operations = import("./sandbox-managed-runtime.js").SandboxSyncOperation[]; + const stripUmask = async (directory: string): Promise => { + for (const child of await fs.readdir(directory, { withFileTypes: true })) { + const full = path.join(directory, child.name); + if (child.isDirectory()) await stripUmask(full); + else if (child.isFile()) await fs.chmod(full, (await fs.stat(full)).mode & 0o7777 & ~0o022); + } + }; + const transfer = (outbound: boolean) => async (operations: Operations) => { + for (const operation of operations) { + for (const file of operation.files) { + const ordinaryExport = outbound && file.preserveSourceEntries !== true; + await fs.mkdir(path.dirname(file.targetPath), { recursive: true }); + await fs.cp(file.sourcePath, file.targetPath, { recursive: true, verbatimSymlinks: true, + filter: (source) => !file.exclude?.some((exclude) => path.relative(file.sourcePath, source) === exclude) + && !(ordinaryExport && reserved.test(path.basename(source))) }); + if (ordinaryExport && (await fs.lstat(file.targetPath)).isDirectory()) await stripUmask(file.targetPath); + } + for (const command of operation.postUploadCommands ?? []) await run(command.command); + } + return { operations: operations.map((op) => ({ operationId: op.operationId, filesTransferred: 0, bytesTransferred: 0 })) }; + }; + const client = { makeDir: (dir: string) => fs.mkdir(dir, { recursive: true }).then(() => {}), + writeFile: (file: string, data: ArrayBuffer) => fs.writeFile(file, Buffer.from(data)), readFile: (file: string) => fs.readFile(file), + listFiles: (dir: string) => fs.readdir(dir), remove: (file: string) => fs.rm(file, { force: true, recursive: true }), run, + syncIn: transfer(false), syncOut: transfer(true) }; + const prepared = await prepareSandboxManagedRuntime({ spec: { provider: "test", sandboxId: "test", remoteCwd: remote, timeoutMs: 30000, apiKey: null, transport: "sandbox" }, + adapterKey: "test", client, workspaceLocalDir: f.root, workspaceCheckpoint: true }); + // The s14 sandbox kept this exact file; its mtime equals the SIGKILL. + await fs.writeFile(path.join(remote, ".paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar"), Buffer.alloc(2048, 1)); + await fs.writeFile(path.join(remote, "new"), "agent work"); + await fs.writeFile(path.join(remote, "._summary"), "user summary"); + await fs.writeFile(path.join(remote, "build.sh"), "#!/bin/sh\n"); await fs.chmod(path.join(remote, "build.sh"), 0o775); + await prepared.restoreWorkspace(); + expect(await fs.readFile(path.join(f.root, "new"), "utf8")).toBe("agent work"); + expect(await fs.readFile(path.join(f.root, "._summary"), "utf8")).toBe("user summary"); + expect((await fs.stat(path.join(f.root, "build.sh"))).mode & 0o777).toBe(0o775); + expect((await fs.readdir(f.root)).filter(name => name.startsWith(".paperclip-upload"))).toEqual([]); +}, 30000); diff --git a/packages/adapter-utils/src/workspace-checkpoint.ts b/packages/adapter-utils/src/workspace-checkpoint.ts index c1711523495..b6a30c0d255 100644 --- a/packages/adapter-utils/src/workspace-checkpoint.ts +++ b/packages/adapter-utils/src/workspace-checkpoint.ts @@ -2,7 +2,7 @@ import fs from "node:fs/promises"; import path from "node:path"; import { DatabaseSync } from "node:sqlite"; import { WorkspaceManifestWriter, WorkspaceManifestMap, workspacePaths, workspacePathMatcher } from "./workspace-manifest.js"; -import { shouldExcludePath, excludePatternMatches, isRelativePathOrDescendant } from "./exclude-patterns.js"; +import { shouldExcludePath, excludePatternMatches, isRelativePathOrDescendant, isSandboxTransferScratchPath } from "./exclude-patterns.js"; import { openDirectorySnapshot, type DirectorySnapshot } from "./workspace-restore-merge.js"; export async function writeCheckpointBaseline(snapshot: DirectorySnapshot, filePath: string): Promise { @@ -20,6 +20,7 @@ async function checkpointMain( createHash: typeof import("node:crypto").createHash, createReadStream: typeof import("node:fs").createReadStream, shouldExcludePath: (relative: string, exclude: readonly string[]) => boolean, + isSandboxTransferScratchPath: (relative: string) => boolean, ) { const [rootArg, baselinePath, outputDir, exclusions] = process.argv.slice(2); const root = await fs.realpath(rootArg!); @@ -89,7 +90,9 @@ async function checkpointMain( for (let p = name; p; p = p.includes("/") ? p.slice(0, p.lastIndexOf("/")) : "") { if (ignored.get(p)) { ignoredPath = true; break; } } - if (ignoredPath || shouldExcludePath(name, exclude)) continue; + // Provider transfer scratch is never exported, so capturing it makes + // every later validation fail (RC-50b). + if (ignoredPath || shouldExcludePath(name, exclude) || isSandboxTransferScratchPath(name)) continue; const before = await fs.lstat(path.join(root, name)); scanned++; if (before.isFile()) { @@ -114,7 +117,7 @@ async function checkpointMain( } export function workspaceCheckpointScript(): string { - return `import fs from 'node:fs/promises';\nimport path from 'node:path';\nimport { DatabaseSync } from 'node:sqlite';\nimport { createHash } from 'node:crypto';\nimport { createReadStream } from 'node:fs';\nconst __name = (fn) => fn;\n${isRelativePathOrDescendant.toString()}\nfunction pathContainsSegmentOrDescendant(relative, segment) { return relative === segment || relative.startsWith(segment + '/') || relative.endsWith('/' + segment) || relative.includes('/' + segment + '/'); }\n${excludePatternMatches.toString()}\n${shouldExcludePath.toString()}\nawait (${checkpointMain.toString()})(fs, path, DatabaseSync, createHash, createReadStream, shouldExcludePath);\n`; + return `import fs from 'node:fs/promises';\nimport path from 'node:path';\nimport { DatabaseSync } from 'node:sqlite';\nimport { createHash } from 'node:crypto';\nimport { createReadStream } from 'node:fs';\nconst __name = (fn) => fn;\n${isRelativePathOrDescendant.toString()}\nfunction pathContainsSegmentOrDescendant(relative, segment) { return relative === segment || relative.startsWith(segment + '/') || relative.endsWith('/' + segment) || relative.includes('/' + segment + '/'); }\n${excludePatternMatches.toString()}\n${shouldExcludePath.toString()}\n${isSandboxTransferScratchPath.toString()}\nawait (${checkpointMain.toString()})(fs, path, DatabaseSync, createHash, createReadStream, shouldExcludePath, isSandboxTransferScratchPath);\n`; } export function readCheckpointSnapshot(filePath: string, baseline: DirectorySnapshot): DirectorySnapshot { @@ -131,6 +134,21 @@ export function readCheckpointSnapshot(filePath: string, baseline: DirectorySnap return snapshot; } +type CheckpointEntryLike = { kind: string; mode?: number; hash?: string; target?: string } | undefined; + +/** Names which entry disagreed and how, so a deterministic mismatch is diagnosable. + * Modes are octal permission bits; hashes are shortened. File contents never appear. */ +function checkpointPayloadMismatch(relative: string, recorded: CheckpointEntryLike, baseline: CheckpointEntryLike, received: CheckpointEntryLike) { + const describe = (entry: CheckpointEntryLike) => !entry ? "absent" + : entry.kind === "file" ? `file mode=${((entry.mode ?? 0) & 0o7777).toString(8)} sha256=${String(entry.hash).slice(0, 12)}` + : entry.kind === "symlink" ? "symlink" : entry.kind; + const shownPath = relative.length > 240 ? `${relative.slice(0, 240)}…` : relative; + return Object.assign(new Error(`Workspace checkpoint payload mismatch at ${JSON.stringify(shownPath)}: ` + + `sandbox recorded ${describe(recorded)}; baseline ${describe(baseline)}; received ${describe(received)}`), { + checkpointMismatch: { path: relative, recorded, baseline, received }, + }); +} + export async function validateCheckpointPayload(input: { baseline: DirectorySnapshot; snapshot: DirectorySnapshot; payload: string }): Promise { const { captureDirectorySnapshot, disposeDirectorySnapshot } = await import("./workspace-restore-merge.js"); const ignored = workspacePathMatcher(input.baseline.ignoredPaths); @@ -156,7 +174,9 @@ export async function validateCheckpointPayload(input: { baseline: DirectorySnap throw Object.assign(new Error("Unsafe workspace checkpoint symlink"), { code: "WORKSPACE_RESTORE_UNSAFE_ARCHIVE" }); } } - if (!equal(entry, input.baseline.entries.get(relative)) && !equal(entry, actual.entries.get(relative))) throw new Error("Workspace checkpoint payload mismatch"); + if (!equal(entry, input.baseline.entries.get(relative)) && !equal(entry, actual.entries.get(relative))) { + throw checkpointPayloadMismatch(relative, entry, input.baseline.entries.get(relative), actual.entries.get(relative)); + } } for (const [relative, entry] of actual.entries) { assertIncluded(relative); diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts index afd1f00b2d6..d96797f44e9 100644 --- a/server/src/services/explicit-native-continuation.ts +++ b/server/src/services/explicit-native-continuation.ts @@ -405,15 +405,23 @@ export async function admitExplicitNativeContinuation(input: { // Nothing finishes the saved result while a stopped copyback owner is // unverified. The run's live retained recovery records that hold; name // its board action instead of an endless finalizing wait (RC-50). - const [ownerHold] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( + const live = await db.select({ + ownerHold: sql`${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' = 'operator_required' + and ${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`, + recoveryState: sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}'`, + }).from(issueRecoveryActions).where(and( eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId), inArray(issueRecoveryActions.status, ["active", "escalated"]), sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, - sql`${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' = 'operator_required'`, - sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`)).limit(1); - return ownerHold - ? blocked("workspace_owner_stop_required", "A board member must confirm that the previous server stopped copying files. Your Retry is saved.") - : blocked("finalizing_saved_result", "Paperclip is finishing the saved result."); + sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`)); + if (live.some(row => row.ownerHold)) + return blocked("workspace_owner_stop_required", "A board member must confirm that the previous server stopped copying files. Your Retry is saved."); + // A spent recovery budget finishes nothing by itself. Another Retry + // starts one new recovery generation (task-retry), so name that action + // instead of an endless finalizing wait (RC-50b, s14). + if (live.length && live.every(row => row.recoveryState === "exhausted")) + return blocked("saved_work_recovery_failed", "Saving the agent's files failed after automatic retries. Retry to try again."); + return blocked("finalizing_saved_result", "Paperclip is finishing the saved result."); } } // A repaired export no longer blocks execution. Its machine receipt still diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index 632b3a96843..ccd6a546bb8 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -83,6 +83,8 @@ const retryWaitMessages: Record = { "budget.blocked": "Retry is waiting for the budget to allow new work. Check Costs for unrecorded or unpriced usage.", finalizing_saved_result: "Paperclip is finishing the saved result before Retry can continue.", // Another click cannot release a copyback owner that stopped without reporting. + // Retry starts a new recovery generation; the agent does not repeat its work. + saved_work_recovery_failed: "Saving the agent's files failed after automatic retries. Retry to try again; the agent will not repeat its work.", workspace_owner_stop_required: "Retry is waiting for a board member to confirm that the previous server stopped copying files. The agent's result and your Retry are saved.", cleanup_quarantined: "Retry cannot continue this attempt. Send a new message after the previous execution has stopped.", cancelled_by_operator: "This attempt was stopped. Send a new message to continue from saved work.", diff --git a/server/src/services/workspace-recovery-preparation-gates.test.ts b/server/src/services/workspace-recovery-preparation-gates.test.ts index 9e1f34187ac..2797aef13b5 100644 --- a/server/src/services/workspace-recovery-preparation-gates.test.ts +++ b/server/src/services/workspace-recovery-preparation-gates.test.ts @@ -362,6 +362,47 @@ describe("automatic retained workspace recovery", () => { } }, 60_000); + // RC-50b (wave s14/t05): Retry was accepted while saved-work recovery ran. + // Every export attempt failed, the budget was spent, and the Retry still said + // "finishing the saved result" with no run. It must name the failure and a + // second Retry must start one new recovery that finishes the saved result. + it("names a spent saved-work recovery behind an accepted Retry, and the next Retry finishes it", async () => { + const f = await acceptedRemoteExport(); + const issue = (await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]; + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId, status: "running" }); + const app = boardRecoveryApp(); + const first = (await request(app).post(`/api/issues/${f.issueId}/retry`) + .send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body; + f.runtime.resumeRunLease.mockRejectedValue(new Error("Workspace checkpoint payload mismatch")); + for (let attempt = 0; attempt < 6; attempt++) { + await recoverRetainedWorkspace(f.input); + await heartbeatService(db).resumeTaskRetry(f.companyId, first.requestId!); + await makeRetryDue(f.actionId); + } + expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "exhausted", attempts: 6 }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, first.requestId!)))[0]) + .toMatchObject({ status: "deferred_issue_execution", payload: { executionWait: { reason: "saved_work_recovery_failed" } } }); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true, + retryWait: { reason: "saved_work_recovery_failed" } }); + // The working action: one more Retry starts one new recovery generation. + f.runtime.resumeRunLease.mockReset(); + f.runtime.resumeRunLease.mockImplementation(async () => ({ providerLeaseId: f.providerLeaseId, metadata: { remoteCwd: await realpath(f.remote) } })); + const second = (await request(app).post(`/api/issues/${f.issueId}/retry`) + .send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body; + expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "scheduled", attempts: 0 }); + await recoverRetainedWorkspace(f.input); + expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete"); + await resumePreservedResultFinalization(db, f.companyId, f.runId); + await heartbeatService(db).resumeTaskRetry(f.companyId, second.requestId!); + // After a board handoff the Retry continues from the recovered files in + // exactly one new run (as in the s12 case), never another wait. + const [settled] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, second.requestId!)); + expect(settled).toMatchObject({ status: "coalesced", runId: expect.any(String), payload: { taskRetry: { strategy: "continue" } } }); + expect(await db.select({ id: heartbeatRuns.id, previousRunId: sql`${heartbeatRuns.contextSnapshot}->>'previousRunId'` }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")))) + .toEqual([{ id: settled.runId, previousRunId: f.runId }]); + }, 60_000); + it("preserves an explicitly requested superseded remote export without applying its result over newer work", async () => { const f = await acceptedRemoteExport(); const [later] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId,