Files
PaperClipAI/packages/adapter-utils/src/workspace-checkpoint-capture.test.ts
T
DottaandClaude Opus 5.5 c45c7cb854 fix(workspaces): ignore killed-transfer scratch in sparse checkpoints (#19730)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run in a remote sandbox copies its workspace back to the
host after the turn. A sparse checkpoint copies only changed entries,
and the host validates each entry against the sandbox's own manifest
> - Every Daytona transfer stages a `.paperclip-upload-<uuid>` scratch
file at the sandbox workspace root and removes it only in the same
process
> - A control-plane kill during any transfer leaves that scratch behind.
The sparse checkpoint then recorded it as a workspace file, but
providers never export it. Every later copyback and every saved-work
recovery attempt failed with "Workspace checkpoint payload mismatch"
> - The failed recovery then left an accepted Retry showing "Paperclip
is finishing the saved result" with no run
> - This pull request stops recording transfer scratch, makes the
checkpoint export an exact copy, names the failing entry, and gives a
Retry behind a spent recovery an honest reason and a working action
> - The benefit is that a restart during a transfer no longer makes a
task's saved work permanently unrecoverable

## Linked Issues or Issue Description

Refs #19721 (RC-50: the killed copyback owner; this PR covers the
separate RC-50b cause)

**What happened?**

Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a
SIGKILL of the control plane, the native result was accepted, and then
workspace copyback failed with "Workspace checkpoint payload mismatch".
In s14 all 6 saved-work recovery attempts failed the same way. A Retry
accepted during recovery kept `retryWait.reason =
finalizing_saved_result` after the recovery budget was spent. No run was
created.

**Steps to reproduce**

1. Start a native Claude run in a Daytona sandbox.
2. SIGKILL the control plane while any provider transfer is in flight.
3. Let the run finish its turn and copy its workspace back.

**Expected behavior**

Copyback succeeds. Provider scratch is not workspace content. A Retry
behind a spent recovery names the failure and can start a new recovery.

**Actual behavior**

The s14 sandbox (6f4a0b6a) still held
`/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`.
Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona
provider stages transfer scratch at `remoteDir`, which is the lease
`remoteCwd` (the workspace root). The runtime workspace wipe
deliberately preserves `.paperclip-upload-*`. The checkpoint helper
(`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar.
The ordinary export excludes `.paperclip-upload*`, so the host received
nothing for that entry. `validateCheckpointPayload` then threw "payload
mismatch" on every attempt. The file-mode hypothesis was disproved: the
sandbox umask is 022 and the scratch is mode 644.

## What Changed

- `packages/adapter-utils/src/exclude-patterns.ts`: new
`isSandboxTransferScratchPath`, which matches a root-level
`.paperclip-upload-*` entry. This agrees with the wipe's preserve rule
in `buildWorkspaceTarExtractCommand`.
- `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox
checkpoint helper skips that scratch. A mismatch error now names the
entry and its recorded, baseline and received forms (kind, octal mode,
short hash; never contents).
- `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the
checkpoint export requests `preserveSourceEntries: true`. The checkpoint
output is an exact copy that the host validates entry by entry. An
ordinary export applies the host umask and drops `._*` and
`.paperclip-upload*` names at any depth. That gives the same permanent
mismatch for a legitimate user file such as `._summary`,
`notes/.paperclip-upload-plan.md`, or a `0775` script.
- `server/src/services/explicit-native-continuation.ts` and
`task-interruption-presentation.ts`: when the run's saved-work recovery
is exhausted, a waiting Retry reports `saved_work_recovery_failed` with
Retry available, instead of `finalizing_saved_result`. The existing task
Retry path starts one new recovery generation.
- `doc/execution-semantics.md`: a short additive note.

## Verification

- `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses
the real checkpoint helper and an export that behaves like Daytona's
ordinary export:
- Before the fix: `Workspace checkpoint payload mismatch at
".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …;
baseline absent; received absent`. That is the wave error.
  - After the fix: the scratch is not recorded, and validation passes.
- A second case shows that reserved-looking user names and `0775` modes
need the exact export.
- `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an
end-to-end `prepareSandboxManagedRuntime` restore through a provider
model. In that model, ordinary exports drop reserved names and apply the
umask, and preserve exports copy exactly. Two red checks:
- Without `preserveSourceEntries`, it fails with the mismatch on
`._summary`.
- Without the scratch skip, the stale tar leaks into the host workspace.
- Green: the agent file, the `._summary` file and the `0775` mode
arrive, and no scratch arrives.
- `server/src/services/workspace-recovery-preparation-gates.test.ts`,
"names a spent saved-work recovery behind an accepted Retry, and the
next Retry finishes it" (s14 shape):
  - Red: `finalizing_saved_result` after 6 failed attempts.
- Green: `saved_work_recovery_failed` with `canRetry: true`. The next
Retry starts a new generation, recovery completes, and the Retry
continues in exactly one new run.
- Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime,
seed-reuse and managed-seed (168 passed). Server preparation-gates (47),
explicit-native-continuation and task-interruption-presentation (455),
workspace-recovery admission and retry-copyback, and
native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils
and server.
- Live evidence: I started the stopped s14 sandbox read-only, listed its
workspace, and stopped it again. No Daytona replay was run. The runner
content ID of this checkout matches no built image.

## Risks

- A user file named `.paperclip-upload-*` directly at the workspace root
is no longer captured by a sparse checkpoint. Providers already drop
that name from every ordinary export, and the runtime already reserves
it at the root.
- The checkpoint export now uses the provider's exact-copy mode. A
workspace file with setuid, setgid or sticky bits is refused by that
mode, as it is for the existing exact-copy transfers. The ordinary
export would have stripped those bits and failed validation anyway.
- The stale scratch tar stays in the sandbox. It is ignored, not
deleted, because a concurrent transfer can own a file with the same
prefix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs, embedded Postgres, a
read-only Daytona sandbox listing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:46:16 -06:00

232 lines
13 KiB
TypeScript

import fs from "node:fs/promises";
import path from "node:path";
import os from "node:os";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { DatabaseSync } from "node:sqlite";
import { afterEach, expect, it, vi } from "vitest";
import { captureDirectorySnapshot, disposeDirectorySnapshot, type DirectorySnapshot } from "./workspace-restore-merge.js";
import { writeCheckpointBaseline, workspaceCheckpointScript, readCheckpointSnapshot, validateCheckpointPayload } from "./workspace-checkpoint.js";
const execute = promisify(execFile);
const roots: string[] = [];
const snapshots: DirectorySnapshot[] = [];
afterEach(async () => {
vi.unstubAllEnvs();
await Promise.all(snapshots.splice(0).map(disposeDirectorySnapshot));
await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true })));
});
async function fixture(failure: "none" | "read" | "mutation" | "full_db" | "walk") {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), "checkpoint-capture-"));
roots.push(temp);
vi.stubEnv("PAPERCLIP_HOME", path.join(temp, "home"));
const root = path.join(temp, "root");
await fs.mkdir(root);
for (let index = 0; index < 32; index++) {
await fs.writeFile(path.join(root, `${failure === "full_db" ? "x".repeat(200) : ""}file-${String(index).padStart(2, "0")}`), Buffer.alloc(64 * 1024, index));
}
const baseline = await captureDirectorySnapshot(root, { diskBacked: true, exclude: [] });
snapshots.push(baseline);
const manifest = path.join(temp, "baseline.sqlite");
await writeCheckpointBaseline(baseline, manifest);
const script = path.join(temp, "checkpoint.mjs");
await fs.writeFile(script, workspaceCheckpointScript());
const tracker = path.join(temp, "tracker.mjs");
await fs.writeFile(tracker, `
import fs from 'node:fs';
import path from 'node:path';
import { Readable } from 'node:stream';
import { syncBuiltinESMExports } from 'node:module';
import { DatabaseSync } from 'node:sqlite';
const original = fs.createReadStream;
const originalClose = DatabaseSync.prototype.close;
const originalExec = DatabaseSync.prototype.exec;
const originalPrepare = DatabaseSync.prototype.prepare;
const originalLstat = fs.promises.lstat;
let heldPath;
const stats = { opened: 0, closed: 0, active: 0, peak: 0, sqliteFull: 0, databaseCloseActive: [] };
DatabaseSync.prototype.close = function() { stats.databaseCloseActive.push(stats.active); return originalClose.call(this); };
DatabaseSync.prototype.exec = function(sql) {
return originalExec.call(this, process.env.CHECKPOINT_FIXTURE_FAILURE === 'full_db' && sql.startsWith('PRAGMA max_page_count=') ? 'PRAGMA max_page_count=2' : sql);
};
DatabaseSync.prototype.prepare = function(sql) {
const statement = originalPrepare.call(this, sql);
if (sql.startsWith('INSERT INTO records')) {
const originalRun = statement.run;
statement.run = function(...args) {
try { return originalRun.apply(this, args); }
catch (error) { if (error.message.includes('database or disk is full')) stats.sqliteFull++; throw error; }
};
}
return statement;
};
fs.promises.lstat = async function(file, ...args) {
if (process.env.CHECKPOINT_FIXTURE_FAILURE === 'walk' && stats.opened > 0 && String(file) !== heldPath) throw new Error('fixture-checkpoint-walk-failure');
return originalLstat.call(this, file, ...args);
};
fs.createReadStream = function(file, options) {
let waiting = false;
const victim = path.basename(String(file)) === 'file-00';
let stream = process.env.CHECKPOINT_FIXTURE_FAILURE === 'read' && victim
? new Readable({ read() { if (!waiting) { waiting = true; setTimeout(() => this.destroy(new Error('fixture-checkpoint-read-failure')), 20); } } })
: original.call(this, file, options);
const holdForWalk = process.env.CHECKPOINT_FIXTURE_FAILURE === 'walk' && stats.opened === 0;
if (holdForWalk) heldPath = String(file);
if (holdForWalk || (process.env.CHECKPOINT_FIXTURE_FAILURE === 'full_db' && String(file).endsWith('file-15'))) {
const source = stream;
stream = Readable.from((async function*() { await new Promise(resolve => setTimeout(resolve, 50)); yield* source; })());
}
if (process.env.CHECKPOINT_FIXTURE_FAILURE === 'mutation' && victim) stream.once('data', () => fs.appendFileSync(file, '!'));
stats.opened++; stats.active++; stats.peak = Math.max(stats.peak, stats.active);
stream.once('close', () => { stats.closed++; stats.active--; });
return stream;
};
syncBuiltinESMExports();
process.once('exit', () => fs.writeFileSync(process.env.CHECKPOINT_FIXTURE_STATS, JSON.stringify(stats)));
`);
const output = path.join(temp, "output");
const statsPath = path.join(temp, "stats.json");
const run = () => execute(process.execPath, ["--import", tracker, script, root, manifest, output, "[]"], {
timeout: 10_000,
env: { ...process.env, CHECKPOINT_FIXTURE_FAILURE: failure, CHECKPOINT_FIXTURE_STATS: statsPath },
});
return { root, baseline, run, output, stats: async () => JSON.parse(await fs.readFile(statsPath, "utf8")) as { opened: number; closed: number; active: number; peak: number; sqliteFull: number; databaseCloseActive: number[] } };
}
it("bounds concurrent file readers and retains an unchanged checkpoint", async () => {
const f = await fixture("none");
await f.run();
const stats = await f.stats();
expect(stats.peak).toBeGreaterThan(1);
expect(stats.peak).toBeLessThanOrEqual(8);
expect(stats.active).toBe(0);
expect(stats.closed).toBe(stats.opened);
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
snapshots.push(snapshot);
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: path.join(f.output, "payload") }))
.toMatchObject({ changedFiles: 0, payloadBytes: 0, scannedEntries: 32 });
});
it("captures every changed file across bounded batches with valid directory parents and modes", async () => {
const f = await fixture("none");
for (let index = 0; index < 32; index++) {
await fs.writeFile(path.join(f.root, `file-${String(index).padStart(2, "0")}`), Buffer.alloc(64 * 1024, index + 1));
}
await fs.chmod(path.join(f.root, "file-00"), 0o755);
await fs.mkdir(path.join(f.root, "nested", "inner"), { recursive: true });
await fs.writeFile(path.join(f.root, "nested", "inner", "new"), "new content");
await fs.symlink("nested/inner/new", path.join(f.root, "link"));
await f.run();
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
snapshots.push(snapshot);
const payload = path.join(f.output, "payload");
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload }))
.toMatchObject({ changedFiles: 33, payloadBytes: 32 * 64 * 1024 + 11, scannedEntries: 36 });
for (let index = 0; index < 32; index++) {
expect((await fs.readFile(path.join(payload, `file-${String(index).padStart(2, "0")}`)))
.equals(Buffer.alloc(64 * 1024, index + 1))).toBe(true);
}
expect((await fs.stat(path.join(payload, "file-00"))).mode & 0o777).toBe(0o755);
expect(await fs.readlink(path.join(payload, "link"))).toBe("nested/inner/new");
const stats = await f.stats();
expect(stats.peak).toBeLessThanOrEqual(8);
expect(stats.active).toBe(0);
expect(stats.closed).toBe(stats.opened);
});
it.each(["read", "mutation"] as const)("settles readers and publishes no receipt after a %s failure", async failure => {
const f = await fixture(failure);
await expect(f.run()).rejects.toThrow(failure === "read" ? "fixture-checkpoint-read-failure" : "Workspace changed during checkpoint capture");
const stats = await f.stats();
expect(stats.opened).toBeGreaterThan(1);
expect(stats.active).toBe(0);
expect(stats.closed).toBe(stats.opened);
await expect(fs.stat(path.join(f.output, "receipt.json"))).rejects.toMatchObject({ code: "ENOENT" });
const db = new DatabaseSync(path.join(f.output, "manifest.sqlite"), { readOnly: true });
try { expect(db.prepare("SELECT count(*) AS count FROM records").get()!.count).toBe(0); }
finally { db.close(); }
});
it("does not commit later captures after SQLite automatically rolls back a full manifest", async () => {
const f = await fixture("full_db");
await expect(f.run()).rejects.toThrow(/database or disk is full|transaction ended during capture/);
await expect(fs.stat(path.join(f.output, "receipt.json"))).rejects.toMatchObject({ code: "ENOENT" });
const db = new DatabaseSync(path.join(f.output, "manifest.sqlite"), { readOnly: true });
try { expect(db.prepare("SELECT count(*) AS count FROM records").get()!.count).toBe(0); }
finally { db.close(); }
const stats = await f.stats();
expect(stats.sqliteFull).toBeGreaterThan(0);
expect(stats.active).toBe(0);
expect(stats.databaseCloseActive).toEqual([0, 0]);
});
it("drains active readers before closing either database after a walk failure", async () => {
const f = await fixture("walk");
await expect(f.run()).rejects.toThrow("fixture-checkpoint-walk-failure");
const stats = await f.stats();
expect(stats.opened).toBeGreaterThan(0);
expect(stats.active).toBe(0);
expect(stats.closed).toBe(stats.opened);
expect(stats.databaseCloseActive).toEqual([0, 0]);
await expect(fs.stat(path.join(f.output, "receipt.json"))).rejects.toMatchObject({ code: "ENOENT" });
});
// Daytona's ordinary directory export (no preserveSourceEntries): GNU tar
// `--exclude '._*' --exclude '.paperclip-upload*'` at any depth, extracted on
// the host without `-p`, so the host umask (022) applies.
async function ordinaryProviderExport(source: string, target: string) {
await fs.cp(source, target, { recursive: true, verbatimSymlinks: true,
filter: entry => !/^(\._|\.paperclip-upload)/.test(path.basename(entry)) });
const strip = async (directory: string): Promise<void> => {
for (const child of await fs.readdir(directory, { withFileTypes: true })) {
const full = path.join(directory, child.name);
if (child.isDirectory()) await strip(full);
else if (child.isFile()) await fs.chmod(full, (await fs.stat(full)).mode & 0o7777 & ~0o022);
}
};
await strip(target);
}
// RC-50b (waves s12/s14): a control-plane kill during a provider transfer left
// `.paperclip-upload-<uuid>.tar` at the sandbox workspace root (found in the s14
// sandbox, mtime equal to the SIGKILL). Providers never export that scratch, so
// recording it made every later checkpoint validation fail.
it("never records provider transfer scratch left at the workspace root by a killed transfer", async () => {
const f = await fixture("none");
await fs.writeFile(path.join(f.root, ".paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar"), Buffer.alloc(4096, 7));
await fs.writeFile(path.join(f.root, "file-00"), "changed by the agent");
await f.run();
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
snapshots.push(snapshot);
const received = path.join(path.dirname(f.output), "received");
await ordinaryProviderExport(path.join(f.output, "payload"), received);
// Before the fix: "payload mismatch at ".paperclip-upload-….tar": … received absent".
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: received }))
.toMatchObject({ changedFiles: 1 });
expect([...snapshot.entries].map(([name]) => name).filter(name => name.startsWith(".paperclip-upload"))).toEqual([]);
await expect(fs.readdir(received)).resolves.toEqual(["file-00"]);
});
it("needs an exact export for reserved-looking user names and group-writable modes", async () => {
const f = await fixture("none");
await fs.mkdir(path.join(f.root, "notes"));
await fs.writeFile(path.join(f.root, "notes", ".paperclip-upload-plan.md"), "user notes");
await fs.writeFile(path.join(f.root, "._summary"), "user summary");
await fs.writeFile(path.join(f.root, "build.sh"), "#!/bin/sh\n");
await fs.chmod(path.join(f.root, "build.sh"), 0o775);
await f.run();
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
snapshots.push(snapshot);
expect(snapshot.entries.get("notes/.paperclip-upload-plan.md")).toMatchObject({ kind: "file" });
const ordinary = path.join(path.dirname(f.output), "ordinary");
await ordinaryProviderExport(path.join(f.output, "payload"), ordinary);
await expect(validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: ordinary }))
.rejects.toThrow(/payload mismatch at ".*": sandbox recorded file mode=\d+ sha256=\w+; baseline absent; received (absent|file mode=755)/);
// The runtime requests preserveSourceEntries for this export: an exact copy.
const exact = path.join(path.dirname(f.output), "exact");
await fs.cp(path.join(f.output, "payload"), exact, { recursive: true, verbatimSymlinks: true });
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: exact }))
.toMatchObject({ changedFiles: 3 });
});