mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 06:01:49 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run in a remote sandbox copies its workspace back to the host after the turn. A sparse checkpoint copies only changed entries, and the host validates each entry against the sandbox's own manifest > - Every Daytona transfer stages a `.paperclip-upload-<uuid>` scratch file at the sandbox workspace root and removes it only in the same process > - A control-plane kill during any transfer leaves that scratch behind. The sparse checkpoint then recorded it as a workspace file, but providers never export it. Every later copyback and every saved-work recovery attempt failed with "Workspace checkpoint payload mismatch" > - The failed recovery then left an accepted Retry showing "Paperclip is finishing the saved result" with no run > - This pull request stops recording transfer scratch, makes the checkpoint export an exact copy, names the failing entry, and gives a Retry behind a spent recovery an honest reason and a working action > - The benefit is that a restart during a transfer no longer makes a task's saved work permanently unrecoverable ## Linked Issues or Issue Description Refs #19721 (RC-50: the killed copyback owner; this PR covers the separate RC-50b cause) **What happened?** Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a SIGKILL of the control plane, the native result was accepted, and then workspace copyback failed with "Workspace checkpoint payload mismatch". In s14 all 6 saved-work recovery attempts failed the same way. A Retry accepted during recovery kept `retryWait.reason = finalizing_saved_result` after the recovery budget was spent. No run was created. **Steps to reproduce** 1. Start a native Claude run in a Daytona sandbox. 2. SIGKILL the control plane while any provider transfer is in flight. 3. Let the run finish its turn and copy its workspace back. **Expected behavior** Copyback succeeds. Provider scratch is not workspace content. A Retry behind a spent recovery names the failure and can start a new recovery. **Actual behavior** The s14 sandbox (6f4a0b6a) still held `/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`. Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona provider stages transfer scratch at `remoteDir`, which is the lease `remoteCwd` (the workspace root). The runtime workspace wipe deliberately preserves `.paperclip-upload-*`. The checkpoint helper (`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar. The ordinary export excludes `.paperclip-upload*`, so the host received nothing for that entry. `validateCheckpointPayload` then threw "payload mismatch" on every attempt. The file-mode hypothesis was disproved: the sandbox umask is 022 and the scratch is mode 644. ## What Changed - `packages/adapter-utils/src/exclude-patterns.ts`: new `isSandboxTransferScratchPath`, which matches a root-level `.paperclip-upload-*` entry. This agrees with the wipe's preserve rule in `buildWorkspaceTarExtractCommand`. - `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox checkpoint helper skips that scratch. A mismatch error now names the entry and its recorded, baseline and received forms (kind, octal mode, short hash; never contents). - `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the checkpoint export requests `preserveSourceEntries: true`. The checkpoint output is an exact copy that the host validates entry by entry. An ordinary export applies the host umask and drops `._*` and `.paperclip-upload*` names at any depth. That gives the same permanent mismatch for a legitimate user file such as `._summary`, `notes/.paperclip-upload-plan.md`, or a `0775` script. - `server/src/services/explicit-native-continuation.ts` and `task-interruption-presentation.ts`: when the run's saved-work recovery is exhausted, a waiting Retry reports `saved_work_recovery_failed` with Retry available, instead of `finalizing_saved_result`. The existing task Retry path starts one new recovery generation. - `doc/execution-semantics.md`: a short additive note. ## Verification - `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses the real checkpoint helper and an export that behaves like Daytona's ordinary export: - Before the fix: `Workspace checkpoint payload mismatch at ".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …; baseline absent; received absent`. That is the wave error. - After the fix: the scratch is not recorded, and validation passes. - A second case shows that reserved-looking user names and `0775` modes need the exact export. - `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an end-to-end `prepareSandboxManagedRuntime` restore through a provider model. In that model, ordinary exports drop reserved names and apply the umask, and preserve exports copy exactly. Two red checks: - Without `preserveSourceEntries`, it fails with the mismatch on `._summary`. - Without the scratch skip, the stale tar leaks into the host workspace. - Green: the agent file, the `._summary` file and the `0775` mode arrive, and no scratch arrives. - `server/src/services/workspace-recovery-preparation-gates.test.ts`, "names a spent saved-work recovery behind an accepted Retry, and the next Retry finishes it" (s14 shape): - Red: `finalizing_saved_result` after 6 failed attempts. - Green: `saved_work_recovery_failed` with `canRetry: true`. The next Retry starts a new generation, recovery completes, and the Retry continues in exactly one new run. - Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime, seed-reuse and managed-seed (168 passed). Server preparation-gates (47), explicit-native-continuation and task-interruption-presentation (455), workspace-recovery admission and retry-copyback, and native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils and server. - Live evidence: I started the stopped s14 sandbox read-only, listed its workspace, and stopped it again. No Daytona replay was run. The runner content ID of this checkout matches no built image. ## Risks - A user file named `.paperclip-upload-*` directly at the workspace root is no longer captured by a sparse checkpoint. Providers already drop that name from every ordinary export, and the runtime already reserves it at the root. - The checkpoint export now uses the provider's exact-copy mode. A workspace file with setuid, setgid or sticky bits is refused by that mode, as it is for the existing exact-copy transfers. The ordinary export would have stripped those bits and failed validation anyway. - The stale scratch tar stays in the sandbox. It is ignored, not deleted, because a concurrent transfer can own a file with the same prefix. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs, embedded Postgres, a read-only Daytona sandbox listing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
232 lines
13 KiB
TypeScript
232 lines
13 KiB
TypeScript
import fs from "node:fs/promises";
|
|
import path from "node:path";
|
|
import os from "node:os";
|
|
import { execFile } from "node:child_process";
|
|
import { promisify } from "node:util";
|
|
import { DatabaseSync } from "node:sqlite";
|
|
import { afterEach, expect, it, vi } from "vitest";
|
|
import { captureDirectorySnapshot, disposeDirectorySnapshot, type DirectorySnapshot } from "./workspace-restore-merge.js";
|
|
import { writeCheckpointBaseline, workspaceCheckpointScript, readCheckpointSnapshot, validateCheckpointPayload } from "./workspace-checkpoint.js";
|
|
|
|
const execute = promisify(execFile);
|
|
const roots: string[] = [];
|
|
const snapshots: DirectorySnapshot[] = [];
|
|
afterEach(async () => {
|
|
vi.unstubAllEnvs();
|
|
await Promise.all(snapshots.splice(0).map(disposeDirectorySnapshot));
|
|
await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true })));
|
|
});
|
|
|
|
async function fixture(failure: "none" | "read" | "mutation" | "full_db" | "walk") {
|
|
const temp = await fs.mkdtemp(path.join(os.tmpdir(), "checkpoint-capture-"));
|
|
roots.push(temp);
|
|
vi.stubEnv("PAPERCLIP_HOME", path.join(temp, "home"));
|
|
const root = path.join(temp, "root");
|
|
await fs.mkdir(root);
|
|
for (let index = 0; index < 32; index++) {
|
|
await fs.writeFile(path.join(root, `${failure === "full_db" ? "x".repeat(200) : ""}file-${String(index).padStart(2, "0")}`), Buffer.alloc(64 * 1024, index));
|
|
}
|
|
const baseline = await captureDirectorySnapshot(root, { diskBacked: true, exclude: [] });
|
|
snapshots.push(baseline);
|
|
const manifest = path.join(temp, "baseline.sqlite");
|
|
await writeCheckpointBaseline(baseline, manifest);
|
|
const script = path.join(temp, "checkpoint.mjs");
|
|
await fs.writeFile(script, workspaceCheckpointScript());
|
|
const tracker = path.join(temp, "tracker.mjs");
|
|
await fs.writeFile(tracker, `
|
|
import fs from 'node:fs';
|
|
import path from 'node:path';
|
|
import { Readable } from 'node:stream';
|
|
import { syncBuiltinESMExports } from 'node:module';
|
|
import { DatabaseSync } from 'node:sqlite';
|
|
const original = fs.createReadStream;
|
|
const originalClose = DatabaseSync.prototype.close;
|
|
const originalExec = DatabaseSync.prototype.exec;
|
|
const originalPrepare = DatabaseSync.prototype.prepare;
|
|
const originalLstat = fs.promises.lstat;
|
|
let heldPath;
|
|
const stats = { opened: 0, closed: 0, active: 0, peak: 0, sqliteFull: 0, databaseCloseActive: [] };
|
|
DatabaseSync.prototype.close = function() { stats.databaseCloseActive.push(stats.active); return originalClose.call(this); };
|
|
DatabaseSync.prototype.exec = function(sql) {
|
|
return originalExec.call(this, process.env.CHECKPOINT_FIXTURE_FAILURE === 'full_db' && sql.startsWith('PRAGMA max_page_count=') ? 'PRAGMA max_page_count=2' : sql);
|
|
};
|
|
DatabaseSync.prototype.prepare = function(sql) {
|
|
const statement = originalPrepare.call(this, sql);
|
|
if (sql.startsWith('INSERT INTO records')) {
|
|
const originalRun = statement.run;
|
|
statement.run = function(...args) {
|
|
try { return originalRun.apply(this, args); }
|
|
catch (error) { if (error.message.includes('database or disk is full')) stats.sqliteFull++; throw error; }
|
|
};
|
|
}
|
|
return statement;
|
|
};
|
|
fs.promises.lstat = async function(file, ...args) {
|
|
if (process.env.CHECKPOINT_FIXTURE_FAILURE === 'walk' && stats.opened > 0 && String(file) !== heldPath) throw new Error('fixture-checkpoint-walk-failure');
|
|
return originalLstat.call(this, file, ...args);
|
|
};
|
|
fs.createReadStream = function(file, options) {
|
|
let waiting = false;
|
|
const victim = path.basename(String(file)) === 'file-00';
|
|
let stream = process.env.CHECKPOINT_FIXTURE_FAILURE === 'read' && victim
|
|
? new Readable({ read() { if (!waiting) { waiting = true; setTimeout(() => this.destroy(new Error('fixture-checkpoint-read-failure')), 20); } } })
|
|
: original.call(this, file, options);
|
|
const holdForWalk = process.env.CHECKPOINT_FIXTURE_FAILURE === 'walk' && stats.opened === 0;
|
|
if (holdForWalk) heldPath = String(file);
|
|
if (holdForWalk || (process.env.CHECKPOINT_FIXTURE_FAILURE === 'full_db' && String(file).endsWith('file-15'))) {
|
|
const source = stream;
|
|
stream = Readable.from((async function*() { await new Promise(resolve => setTimeout(resolve, 50)); yield* source; })());
|
|
}
|
|
if (process.env.CHECKPOINT_FIXTURE_FAILURE === 'mutation' && victim) stream.once('data', () => fs.appendFileSync(file, '!'));
|
|
stats.opened++; stats.active++; stats.peak = Math.max(stats.peak, stats.active);
|
|
stream.once('close', () => { stats.closed++; stats.active--; });
|
|
return stream;
|
|
};
|
|
syncBuiltinESMExports();
|
|
process.once('exit', () => fs.writeFileSync(process.env.CHECKPOINT_FIXTURE_STATS, JSON.stringify(stats)));
|
|
`);
|
|
const output = path.join(temp, "output");
|
|
const statsPath = path.join(temp, "stats.json");
|
|
const run = () => execute(process.execPath, ["--import", tracker, script, root, manifest, output, "[]"], {
|
|
timeout: 10_000,
|
|
env: { ...process.env, CHECKPOINT_FIXTURE_FAILURE: failure, CHECKPOINT_FIXTURE_STATS: statsPath },
|
|
});
|
|
return { root, baseline, run, output, stats: async () => JSON.parse(await fs.readFile(statsPath, "utf8")) as { opened: number; closed: number; active: number; peak: number; sqliteFull: number; databaseCloseActive: number[] } };
|
|
}
|
|
|
|
it("bounds concurrent file readers and retains an unchanged checkpoint", async () => {
|
|
const f = await fixture("none");
|
|
await f.run();
|
|
const stats = await f.stats();
|
|
expect(stats.peak).toBeGreaterThan(1);
|
|
expect(stats.peak).toBeLessThanOrEqual(8);
|
|
expect(stats.active).toBe(0);
|
|
expect(stats.closed).toBe(stats.opened);
|
|
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
|
|
snapshots.push(snapshot);
|
|
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: path.join(f.output, "payload") }))
|
|
.toMatchObject({ changedFiles: 0, payloadBytes: 0, scannedEntries: 32 });
|
|
});
|
|
|
|
it("captures every changed file across bounded batches with valid directory parents and modes", async () => {
|
|
const f = await fixture("none");
|
|
for (let index = 0; index < 32; index++) {
|
|
await fs.writeFile(path.join(f.root, `file-${String(index).padStart(2, "0")}`), Buffer.alloc(64 * 1024, index + 1));
|
|
}
|
|
await fs.chmod(path.join(f.root, "file-00"), 0o755);
|
|
await fs.mkdir(path.join(f.root, "nested", "inner"), { recursive: true });
|
|
await fs.writeFile(path.join(f.root, "nested", "inner", "new"), "new content");
|
|
await fs.symlink("nested/inner/new", path.join(f.root, "link"));
|
|
await f.run();
|
|
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
|
|
snapshots.push(snapshot);
|
|
const payload = path.join(f.output, "payload");
|
|
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload }))
|
|
.toMatchObject({ changedFiles: 33, payloadBytes: 32 * 64 * 1024 + 11, scannedEntries: 36 });
|
|
for (let index = 0; index < 32; index++) {
|
|
expect((await fs.readFile(path.join(payload, `file-${String(index).padStart(2, "0")}`)))
|
|
.equals(Buffer.alloc(64 * 1024, index + 1))).toBe(true);
|
|
}
|
|
expect((await fs.stat(path.join(payload, "file-00"))).mode & 0o777).toBe(0o755);
|
|
expect(await fs.readlink(path.join(payload, "link"))).toBe("nested/inner/new");
|
|
const stats = await f.stats();
|
|
expect(stats.peak).toBeLessThanOrEqual(8);
|
|
expect(stats.active).toBe(0);
|
|
expect(stats.closed).toBe(stats.opened);
|
|
});
|
|
|
|
it.each(["read", "mutation"] as const)("settles readers and publishes no receipt after a %s failure", async failure => {
|
|
const f = await fixture(failure);
|
|
await expect(f.run()).rejects.toThrow(failure === "read" ? "fixture-checkpoint-read-failure" : "Workspace changed during checkpoint capture");
|
|
const stats = await f.stats();
|
|
expect(stats.opened).toBeGreaterThan(1);
|
|
expect(stats.active).toBe(0);
|
|
expect(stats.closed).toBe(stats.opened);
|
|
await expect(fs.stat(path.join(f.output, "receipt.json"))).rejects.toMatchObject({ code: "ENOENT" });
|
|
const db = new DatabaseSync(path.join(f.output, "manifest.sqlite"), { readOnly: true });
|
|
try { expect(db.prepare("SELECT count(*) AS count FROM records").get()!.count).toBe(0); }
|
|
finally { db.close(); }
|
|
});
|
|
|
|
it("does not commit later captures after SQLite automatically rolls back a full manifest", async () => {
|
|
const f = await fixture("full_db");
|
|
await expect(f.run()).rejects.toThrow(/database or disk is full|transaction ended during capture/);
|
|
await expect(fs.stat(path.join(f.output, "receipt.json"))).rejects.toMatchObject({ code: "ENOENT" });
|
|
const db = new DatabaseSync(path.join(f.output, "manifest.sqlite"), { readOnly: true });
|
|
try { expect(db.prepare("SELECT count(*) AS count FROM records").get()!.count).toBe(0); }
|
|
finally { db.close(); }
|
|
const stats = await f.stats();
|
|
expect(stats.sqliteFull).toBeGreaterThan(0);
|
|
expect(stats.active).toBe(0);
|
|
expect(stats.databaseCloseActive).toEqual([0, 0]);
|
|
});
|
|
|
|
it("drains active readers before closing either database after a walk failure", async () => {
|
|
const f = await fixture("walk");
|
|
await expect(f.run()).rejects.toThrow("fixture-checkpoint-walk-failure");
|
|
const stats = await f.stats();
|
|
expect(stats.opened).toBeGreaterThan(0);
|
|
expect(stats.active).toBe(0);
|
|
expect(stats.closed).toBe(stats.opened);
|
|
expect(stats.databaseCloseActive).toEqual([0, 0]);
|
|
await expect(fs.stat(path.join(f.output, "receipt.json"))).rejects.toMatchObject({ code: "ENOENT" });
|
|
});
|
|
|
|
// Daytona's ordinary directory export (no preserveSourceEntries): GNU tar
|
|
// `--exclude '._*' --exclude '.paperclip-upload*'` at any depth, extracted on
|
|
// the host without `-p`, so the host umask (022) applies.
|
|
async function ordinaryProviderExport(source: string, target: string) {
|
|
await fs.cp(source, target, { recursive: true, verbatimSymlinks: true,
|
|
filter: entry => !/^(\._|\.paperclip-upload)/.test(path.basename(entry)) });
|
|
const strip = async (directory: string): Promise<void> => {
|
|
for (const child of await fs.readdir(directory, { withFileTypes: true })) {
|
|
const full = path.join(directory, child.name);
|
|
if (child.isDirectory()) await strip(full);
|
|
else if (child.isFile()) await fs.chmod(full, (await fs.stat(full)).mode & 0o7777 & ~0o022);
|
|
}
|
|
};
|
|
await strip(target);
|
|
}
|
|
|
|
// RC-50b (waves s12/s14): a control-plane kill during a provider transfer left
|
|
// `.paperclip-upload-<uuid>.tar` at the sandbox workspace root (found in the s14
|
|
// sandbox, mtime equal to the SIGKILL). Providers never export that scratch, so
|
|
// recording it made every later checkpoint validation fail.
|
|
it("never records provider transfer scratch left at the workspace root by a killed transfer", async () => {
|
|
const f = await fixture("none");
|
|
await fs.writeFile(path.join(f.root, ".paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar"), Buffer.alloc(4096, 7));
|
|
await fs.writeFile(path.join(f.root, "file-00"), "changed by the agent");
|
|
await f.run();
|
|
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
|
|
snapshots.push(snapshot);
|
|
const received = path.join(path.dirname(f.output), "received");
|
|
await ordinaryProviderExport(path.join(f.output, "payload"), received);
|
|
// Before the fix: "payload mismatch at ".paperclip-upload-….tar": … received absent".
|
|
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: received }))
|
|
.toMatchObject({ changedFiles: 1 });
|
|
expect([...snapshot.entries].map(([name]) => name).filter(name => name.startsWith(".paperclip-upload"))).toEqual([]);
|
|
await expect(fs.readdir(received)).resolves.toEqual(["file-00"]);
|
|
});
|
|
|
|
it("needs an exact export for reserved-looking user names and group-writable modes", async () => {
|
|
const f = await fixture("none");
|
|
await fs.mkdir(path.join(f.root, "notes"));
|
|
await fs.writeFile(path.join(f.root, "notes", ".paperclip-upload-plan.md"), "user notes");
|
|
await fs.writeFile(path.join(f.root, "._summary"), "user summary");
|
|
await fs.writeFile(path.join(f.root, "build.sh"), "#!/bin/sh\n");
|
|
await fs.chmod(path.join(f.root, "build.sh"), 0o775);
|
|
await f.run();
|
|
const snapshot = readCheckpointSnapshot(path.join(f.output, "manifest.sqlite"), f.baseline);
|
|
snapshots.push(snapshot);
|
|
expect(snapshot.entries.get("notes/.paperclip-upload-plan.md")).toMatchObject({ kind: "file" });
|
|
const ordinary = path.join(path.dirname(f.output), "ordinary");
|
|
await ordinaryProviderExport(path.join(f.output, "payload"), ordinary);
|
|
await expect(validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: ordinary }))
|
|
.rejects.toThrow(/payload mismatch at ".*": sandbox recorded file mode=\d+ sha256=\w+; baseline absent; received (absent|file mode=755)/);
|
|
// The runtime requests preserveSourceEntries for this export: an exact copy.
|
|
const exact = path.join(path.dirname(f.output), "exact");
|
|
await fs.cp(path.join(f.output, "payload"), exact, { recursive: true, verbatimSymlinks: true });
|
|
expect(await validateCheckpointPayload({ baseline: f.baseline, snapshot, payload: exact }))
|
|
.toMatchObject({ changedFiles: 3 });
|
|
});
|