mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix: preserve warm Codex turns with incremental managed file checkpoints (#14735)
## Thinking Path > - Paperclip manages AI agents and keeps their instructions and files durable. > - Native Codex runners can keep a process alive between compatible turns. > - Managed file collection stopped that process after each turn, which defeated warm reuse. > - Agent folders can contain large images and other files, so full copies on every turn are expensive. > - This change keeps one managed directory for the live session and saves only file changes after each turn. > - Ownership, authorization, instruction changes, and process retirement still control when reuse is safe. ## Linked Issues or Issue Description Related: #13710 introduced native warm session reuse. This fixes managed file collection that still forced those sessions to stop. No duplicate open PR or issue was found. **What happened?** With managed instructions and warm native Codex enabled, consecutive turns reused a Daytona sandbox but started a new runner process each time. The managed directory collector required process termination before saving files. **Expected behavior** Compatible turns keep the same process and managed `AGENT_HOME`. Each completed turn saves added, changed, and deleted files before the next turn starts. Unchanged large files do not transfer again. **Steps to reproduce** 1. Use a native Codex agent with managed instructions and a reusable Daytona environment. 2. Enable warm session reuse and run three turns on the same task. 3. Write a large binary on the first turn, edit a small note on each turn, and delete a file on the second turn. 4. Compare process identity across turns and read the canonical files through the public agent-files API. **Paperclip version or commit** Reproduced on `d30b03bd8c17604cdab1533eeeeb087aba30e8b1`. **Deployment mode** Local server with remote Daytona execution; cloud native runner uses the same path. ## What Changed - Retain the managed directory only for the verified owner of a live native Codex session. - Checkpoint each completed turn before releasing the session for reuse. Retry unstable captures, then stop and collect when a warm checkpoint cannot be validated. - Compare metadata and cached hashes, stream only changed file payloads, record deletions, and validate path, content, quota, and authorization before saving. - Rotate sessions when canonical files, loaded instructions, credentials, or launch policy change. Fence stale collection and cleanup callbacks from later owners. - Keep cleanup and recovery aware of the current session owner. Recheck canonical files under the writer lock at handoff, attach the successor collector before fallible bookkeeping, and emit one final save receipt on checkpoint fallback. Preserve storage warnings across unchanged checkpoints. - Add regression coverage and a three-turn Daytona test with independent public API file checks, an unchanged 8 MiB binary, deletion checks, and strict process identity checks. - Document checkpoint consistency, lifecycle behavior, and local run-log counters. - Replace a timing assumption in the Daytona teardown test with explicit transfer-arrival gates after CI exposed an unset release callback. ## Verification - Full local `pnpm -r typecheck` and `pnpm build` passed. Server checks were repeated after the final storage-warning fix. - Runner E2E typecheck and 749 runner E2E unit tests passed. - Focused file checkpoint, directory ownership, instruction collection, native session, and merge tests passed. After review fixes, the managed-directory and native-session suites passed 550 tests, including intervening canonical edits, same-run fresh restore, failed handoff collection, and one-call fallback collection. Server typecheck passed again. The Daytona plugin suite passed 218 tests. The quota-warning regression failed before the fix and passed afterward. - Three real Daytona campaigns passed before the final handoff review fixes. The latest kept PID 547 across all three turns. The first checkpoint copied 8,388,635 bytes; the next two copied 36 and 54 bytes. Public API reads verified the binary, note contents, and deletion after every turn. Test cleanup deleted the sandbox. - The final head was also deployed to an isolated cloud staging instance and passed three UI-triggered native Codex turns with managed instructions. All three retained the same process ID/start time, native session, provider session, runner instance, and Daytona sandbox. Checkpoints copied 8,388,643 bytes on turn 1, then only 52 and 78 bytes on turns 2 and 3; those warm captures also hashed only 52 and 78 bytes. Independent canonical API reads verified every byte of the unchanged 8 MiB binary and the exact note contents after every turn; the deleted file returned 404 after turns 2 and 3. After restoring the original lifecycle and agent-auth configuration, removing the temporary secret, pausing the test agent, and deleting both test sandboxes, independent canonical API reads still verified the entire binary, the final 78-byte three-line note, and the deletion. The native runner flag remained enabled and the final serving revision remained the PR head. - Two earlier staging attempts are preserved as failures and are excluded from the acceptance result: a saved ChatGPT login failed with a provider routing 401, and its subsequent stopped-sandbox retry failed before provider startup with a closed-lease admission error. The successful campaign used a fresh sandbox and a temporary encrypted API-key binding. The stopped-lease retry remains unexplained; this campaign does not establish recovery of that failed sandbox. - All [Paperclip CI gates](https://github.com/paperclipai/paperclip/actions/runs/36750397355) pass on `26ef2ef56a389259246809805c0b34a4747eb86b`, including full test partitions, build, typecheck, runner verification, E2E shards, and the Canary clean public-npm install. Greptile reviewed that exact head at 5/5 with no unresolved review threads or outstanding findings. - Full local repository coverage used the existing CI partitions, but the 40,000-file Git streaming stress test timed out and its local retry was interrupted by macOS thermal emergency sleep; this is not a green full local suite claim. The exact stress test passed on the final head in [CI server shard 2/12](https://github.com/paperclipai/paperclip/actions/runs/36750397355/job/110008294290), in 111.9 seconds. - Repeat the live test with configured credentials and a Linux runner artifact: `pnpm test:e2e:runner -- --id daytona-warm-continuity.runner-codex.daytona.warm-three-turn`. ## Risks - This is a file-level checkpoint, not an atomic snapshot of the whole folder. Background writes after a capture are saved by the next checkpoint or final stopped collection. - Metadata scans still visit all paths. Modified files transfer in full; unchanged files do not rehash or transfer. - Incorrect ownership or reuse could collect the wrong directory. Run ownership fences, current authorization, stable capture validation, and stopped collection fallbacks are covered by tests. - Warm reuse remains opt-in. No database migration or fleet default changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use, and test execution. The exact serving model ID and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
d432dc7fa3
commit
b54b2dc35c
23 files changed
+1195
-92
No files matched your search
@@ -18,6 +18,7 @@ import { resolveManagedInstructionsRoot } from "../services/agent-instructions.j
|
||||
import { buildNativeRuntimeContext } from "../services/native-runtime/runtime-context.js";
|
||||
import type { EnvironmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
|
||||
import { AgentDirectoryReuseInvalidatedError, agentDirectoryWorkingCopyService } from "../services/agent-directory-working-copies.js";
|
||||
|
||||
describe("persistent agent directories", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
@@ -31,10 +32,10 @@ describe("persistent agent directories", () => {
|
||||
const initial = "\uFEFF# Original\r\n☃\n";
|
||||
const target = () => ({ companyId, agentId });
|
||||
const board = () => ({ type: "board" as const, userId, source: "session" as const });
|
||||
async function run() {
|
||||
async function run(options: { warm?: boolean; reuseRunId?: string } = {}) {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
return (await copies.prepare({ ...target(), runId, cwd: home }))!;
|
||||
return (await copies.prepare({ ...target(), runId, cwd: home, ...options }))!;
|
||||
}
|
||||
beforeAll(async () => {
|
||||
home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "instruction-working-copies-")));
|
||||
@@ -165,16 +166,18 @@ describe("persistent agent directories", () => {
|
||||
expect(await fs.readFile(path.join(root, "next-task.txt"), "utf8")).toBe("still working");
|
||||
});
|
||||
|
||||
it("warns on each run at the file limit and clears the warning after ordinary agent cleanup", async () => {
|
||||
it.each([false, true])("warns on each run at the file limit and clears the warning after ordinary agent cleanup (warm=%s)", async (warm) => {
|
||||
await sparseFile(path.join(root, "full.bin"), MAX_AGENT_FILE_BYTES);
|
||||
const first = await run();
|
||||
const first = await run({ warm });
|
||||
expect(first.receipt?.storageWarning).toContain("256 MiB");
|
||||
expect(instructionWorkingCopyGuidance(first)).toContain("Runs can continue");
|
||||
const unchanged = await copies.collectStopped({ companyId, runId: first.runId });
|
||||
expect(unchanged).toMatchObject({ state: "unchanged", errorCode: null });
|
||||
const unchanged = await copies[warm ? "checkpointWarm" : "collectStopped"]({ companyId, runId: first.runId });
|
||||
expect(unchanged).toMatchObject({ state: warm ? "warm_saved" : "unchanged", errorCode: null });
|
||||
if (warm) expect(unchanged?.receipt?.checkpointState).toBe("unchanged");
|
||||
expect(unchanged?.receipt?.storageWarning).toContain("Agent storage is full");
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
const second = await run();
|
||||
const second = await run({ warm, ...(warm ? { reuseRunId: first.runId } : {}) });
|
||||
if (warm) expect(second.localRoot).toBe(first.localRoot);
|
||||
expect(second.receipt?.storageWarning).toContain("Agent storage is full");
|
||||
await fs.unlink(path.join(second.localRoot, "full.bin"));
|
||||
await fs.writeFile(path.join(second.localRoot, "task-output.txt"), "work continues");
|
||||
@@ -443,6 +446,150 @@ describe("persistent agent directories", () => {
|
||||
expect(after.context.aggregateDigest).toBe(before.context.aggregateDigest);
|
||||
expect(after.context.instructions.bundle.fileCount).toBe(1);
|
||||
});
|
||||
it("checkpoints three managed turns into one live directory, fences old cleanup, and persists after retirement", async () => {
|
||||
let copy = await run({ warm: true });
|
||||
const original = copy;
|
||||
for (let turn = 1; turn <= 3; turn++) {
|
||||
await fs.appendFile(path.join(copy.localRoot, "memory.txt"), `turn ${turn}\n`);
|
||||
const saved = (await copies.checkpointWarm({ companyId, runId: copy.runId }))!;
|
||||
expect(saved.state).toBe("warm_saved");
|
||||
expect(saved.processStoppedAt).toBeNull();
|
||||
expect(await agentFileStore(db).read(companyId, agentId, "memory.txt", board())).toEqual(Buffer.from(Array.from({ length: turn }, (_, i) => `turn ${i + 1}\n`).join("")));
|
||||
await copies.release(companyId, copy.runId);
|
||||
expect(await fs.stat(copy.localRoot)).toBeTruthy();
|
||||
if (turn < 3) {
|
||||
expect(await copies.canReuseWarm(companyId, agentId, copy.runId)).toBe(true);
|
||||
copy = await run({ warm: true, reuseRunId: copy.runId });
|
||||
expect(copy.executionRoot).toBe(original.executionRoot);
|
||||
await copies.collectStopped({ companyId, runId: original.runId });
|
||||
await copies.release(companyId, original.runId);
|
||||
expect(await fs.stat(copy.localRoot)).toBeTruthy();
|
||||
}
|
||||
}
|
||||
// A child may write after the last turn's checkpoint; session retirement
|
||||
// must collect this delta using the current owner, not the first run.
|
||||
await fs.writeFile(path.join(copy.localRoot, "late.txt"), "after turn");
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId }))?.state).toBe("saved");
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
const fresh = await run();
|
||||
expect(await fs.readFile(path.join(fresh.localRoot, "memory.txt"), "utf8")).toBe("turn 1\nturn 2\nturn 3\n");
|
||||
expect(await fs.readFile(path.join(fresh.localRoot, "late.txt"), "utf8")).toBe("after turn");
|
||||
});
|
||||
|
||||
it("rechecks canonical edits at handoff and releases the writer lock before retirement and fresh restore", async () => {
|
||||
const first = await run({ warm: true });
|
||||
await copies.checkpointWarm({ companyId, runId: first.runId });
|
||||
expect(await copies.canReuseWarm(companyId, agentId, first.runId)).toBe(true);
|
||||
await agentFileStore(db).write({ ...target(), path: "editor.txt", bytes: Buffer.from("new canonical content"), baseHash: null }, board());
|
||||
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
await expect(copies.prepare({ ...target(), runId, cwd: home, warm: true, reuseRunId: first.runId })).rejects.toBeInstanceOf(AgentDirectoryReuseInvalidatedError);
|
||||
expect(JSON.parse(await fs.readFile(path.join(path.dirname(first.localRoot), "owner.json"), "utf8")).runId).toBe(first.runId);
|
||||
// Both paths need the canonical writer lock. A handoff failure must release
|
||||
// it before orchestration stops the old session and restores a fresh copy.
|
||||
expect((await copies.collectStopped({ companyId, runId: first.runId }))?.state).toBe("unchanged");
|
||||
const next = (await copies.prepare({ ...target(), runId, cwd: home, warm: true }))!;
|
||||
expect(next.localRoot).not.toBe(first.localRoot);
|
||||
expect(await fs.readFile(path.join(next.localRoot, "editor.txt"), "utf8")).toBe("new canonical content");
|
||||
}, 10_000);
|
||||
|
||||
it("attaches the successor collector before fallible post-handoff bookkeeping", async () => {
|
||||
const first = await run({ warm: true });
|
||||
await copies.checkpointWarm({ companyId, runId: first.runId });
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
let collectorRunId = first.runId;
|
||||
const handoff = vi.fn((copy: typeof first) => { collectorRunId = copy.runId; });
|
||||
const directories = agentDirectoryWorkingCopyService(db, copies.get, async () => { throw new Error("injected post-handoff database failure"); });
|
||||
|
||||
await expect(directories.prepare({ ...target(), runId, cwd: home, warm: true, reuseRunId: first.runId, onWarmHandoff: handoff }))
|
||||
.rejects.toThrow("injected post-handoff database failure");
|
||||
expect(handoff).toHaveBeenCalledOnce();
|
||||
expect(collectorRunId).toBe(runId);
|
||||
await fs.writeFile(path.join(first.localRoot, "last-write.txt"), "saved after failed handoff");
|
||||
// The stale collector cannot touch the successor; the attached collector
|
||||
// still saves and cleans the actual owner after process retirement.
|
||||
await copies.collectStopped({ companyId, runId: first.runId });
|
||||
expect((await fs.stat(first.localRoot)).isDirectory()).toBe(true);
|
||||
expect((await copies.collectStopped({ companyId, runId: collectorRunId }))?.state).toBe("saved");
|
||||
expect(await fs.readFile(path.join(root, "last-write.txt"), "utf8")).toBe("saved after failed handoff");
|
||||
await expect(fs.stat(first.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
}, 10_000);
|
||||
it("retires loaded instruction edits while allowing ordinary personal-file edits to stay warm", async () => {
|
||||
const copy = await run({ warm: true });
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "# New loaded policy\n");
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.state).toBe("warm_saved");
|
||||
expect(await copies.canReuseWarm(companyId, agentId, copy.runId)).toBe(false);
|
||||
await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
const fresh = await run({ warm: true });
|
||||
expect(fresh.executionRoot).not.toBe(copy.executionRoot);
|
||||
expect(await fs.readFile(path.join(fresh.localRoot, entryFile), "utf8")).toBe("# New loaded policy\n");
|
||||
});
|
||||
it("refreshes after editor changes and preserves concurrent unrelated files", async () => {
|
||||
const copy = await run({ warm: true });
|
||||
await fs.writeFile(path.join(copy.localRoot, "mine.txt"), "from run");
|
||||
await agentFileStore(db).write({ ...target(), path: "board.txt", bytes: Buffer.from("from board"), baseHash: null }, board());
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.state).toBe("warm_saved");
|
||||
expect(await copies.canReuseWarm(companyId, agentId, copy.runId)).toBe(false);
|
||||
expect(await fs.readFile(path.join(root, "board.txt"), "utf8")).toBe("from board");
|
||||
expect(await fs.readFile(path.join(root, "mine.txt"), "utf8")).toBe("from run");
|
||||
await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
});
|
||||
it("keeps an invalid warm checkpoint unsaved and requests stopped collection", async () => {
|
||||
const copy = await run({ warm: true });
|
||||
await fs.symlink(root, path.join(copy.localRoot, "escape"));
|
||||
const failed = await copies.checkpointWarm({ companyId, runId: copy.runId });
|
||||
expect(failed?.state).toBe("prepared");
|
||||
expect(failed?.errorCode).toBe("AGENT_FILES_CHECKPOINT_UNSTABLE");
|
||||
expect(failed?.processStoppedAt).toBeNull();
|
||||
expect(await copies.canReuseWarm(companyId, agentId, copy.runId)).toBe(false);
|
||||
await fs.rm(path.join(copy.localRoot, "escape"));
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId }))?.state).toBe("unchanged");
|
||||
});
|
||||
it("rejects over-quota warm changes without replacing saved bytes, then permits a clean future run", async () => {
|
||||
const copy = await run({ warm: true });
|
||||
const handle = await fs.open(path.join(copy.localRoot, "too-large.bin"), "w");
|
||||
await handle.truncate(MAX_AGENT_FILE_BYTES + 1); await handle.close();
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.errorCode).toBe("AGENT_FILES_CHECKPOINT_UNSTABLE");
|
||||
const stopped = (await copies.collectStopped({ companyId, runId: copy.runId }))!;
|
||||
expect(stopped).toMatchObject({ state: "unavailable", errorCode: "AGENT_FILES_LIMIT_EXCEEDED" });
|
||||
expect(stopped.receipt?.storageWarning).toContain("Agent storage is full");
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await run({ warm: true })).toBeTruthy();
|
||||
});
|
||||
it("rechecks authorization even for an unchanged warm checkpoint", async () => {
|
||||
const copy = await run({ warm: true });
|
||||
await copies.checkpointWarm({ companyId, runId: copy.runId });
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.companyId, companyId));
|
||||
await db.update(companyMemberships).set({ membershipRole: "member" }).where(eq(companyMemberships.companyId, companyId));
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.errorCode).toBe("AGENT_FILES_CHECKPOINT_UNSTABLE");
|
||||
expect(await copies.canReuseWarm(companyId, agentId, copy.runId)).toBe(false);
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId }))?.state).toBe("unavailable");
|
||||
});
|
||||
it("reclaims a warm remote owner after verified sandbox destruction without claiming unsaved tail bytes", async () => {
|
||||
const copy = await run({ warm: true });
|
||||
await fs.writeFile(path.join(copy.localRoot, "saved.txt"), "checkpoint");
|
||||
await copies.checkpointWarm({ companyId, runId: copy.runId });
|
||||
const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/task";
|
||||
const lease = { id: leaseId, companyId, environmentId, heartbeatRunId: copy.runId, provider: "daytona", providerLeaseId: "destroyed-warm" };
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" });
|
||||
await db.insert(environmentLeases).values({ ...lease, status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
||||
metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } });
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", runtimeMode: "native" }).where(eq(heartbeatRuns.id, copy.runId));
|
||||
const saved = (await copies.get(companyId, copy.runId))!;
|
||||
await db.update(agentInstructionWorkingCopies).set({ location: `remote:${environmentId}`,
|
||||
executionRoot: path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, copy.runId),
|
||||
receipt: { ...saved.receipt, cleanup: { leaseId, remoteCwd } } }).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
await copies.recoverStopped();
|
||||
const recovered = (await copies.get(companyId, copy.runId))!;
|
||||
expect(recovered).toMatchObject({ state: "unavailable", errorCode: "AGENT_FILES_FINAL_COLLECTION_UNAVAILABLE" });
|
||||
expect(recovered.receipt?.cleanupPending).toBe(false);
|
||||
expect(await fs.readFile(path.join(root, "saved.txt"), "utf8")).toBe("checkpoint");
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
it("uses the workspace transport to restore after destruction of the remote filesystem", async () => {
|
||||
const remoteCwd = path.join(home, "remote-task");
|
||||
await fs.mkdir(remoteCwd, { recursive: true });
|
||||
@@ -496,6 +643,52 @@ describe("persistent agent directories", () => {
|
||||
await copies.release(companyId, second.runId);
|
||||
});
|
||||
|
||||
it("checkpoints and reuses the remote directory through the real transport without copying unchanged bytes", async () => {
|
||||
const remoteCwd = path.join(home, "warm-remote-task");
|
||||
await fs.mkdir(remoteCwd, { recursive: true });
|
||||
const runner: import("@paperclipai/adapter-utils/command-managed-runtime").CommandManagedRuntimeRunner = {
|
||||
execute: async input => {
|
||||
const startedAt = new Date().toISOString();
|
||||
const env = { ...process.env, ...input.env };
|
||||
const args = [...(input.args ?? [])];
|
||||
if (input.stdin != null && (args[0] === "-c" || args[0] === "-lc")) {
|
||||
env.PAPERCLIP_TEST_STDIN = input.stdin;
|
||||
args[1] = `printf '%s' "$PAPERCLIP_TEST_STDIN" | (${args[1]})`;
|
||||
}
|
||||
try {
|
||||
const result = await execFile(input.command, args, { cwd: input.cwd, env, timeout: input.timeoutMs, maxBuffer: 32 * 1024 * 1024 });
|
||||
return { exitCode: 0, signal: null, timedOut: false, stdout: result.stdout, stderr: result.stderr, pid: null, startedAt };
|
||||
} catch (error) {
|
||||
const e = error as { code?: number; signal?: NodeJS.Signals; stdout?: string; stderr?: string };
|
||||
return { exitCode: typeof e.code === "number" ? e.code : 1, signal: e.signal ?? null, timedOut: false, stdout: e.stdout ?? "", stderr: e.stderr ?? "", pid: null, startedAt };
|
||||
}
|
||||
},
|
||||
};
|
||||
const executionTarget = { kind: "remote" as const, transport: "sandbox" as const, environmentId: randomUUID(), remoteCwd, runner };
|
||||
const prepare = async (reuseRunId?: string) => {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
return (await copies.prepare({ ...target(), runId, cwd: home, target: executionTarget, warm: true, reuseRunId }))!;
|
||||
};
|
||||
let copy = await prepare();
|
||||
const original = copy;
|
||||
await fs.writeFile(path.join(copy.executionRoot, "image.bin"), Buffer.alloc(32768, 93));
|
||||
for (let turn = 1; turn <= 3; turn++) {
|
||||
await fs.appendFile(path.join(copy.executionRoot, "memory.txt"), `turn ${turn}\n`);
|
||||
const saved = (await copies.checkpointWarm({ companyId, runId: copy.runId, target: executionTarget }))!;
|
||||
expect(saved, JSON.stringify({ code: saved.errorCode, message: saved.errorMessage })).toMatchObject({ state: "warm_saved", errorCode: null });
|
||||
if (turn > 1) expect(saved.receipt?.checkpointStats).toMatchObject({ copiedFiles: 1, copiedBytes: turn * 7, hashedBytes: turn * 7 });
|
||||
expect(await fs.readFile(path.join(root, "memory.txt"), "utf8")).toBe(Array.from({ length: turn }, (_, i) => `turn ${i + 1}\n`).join(""));
|
||||
expect((await fs.readdir(path.dirname(copy.localRoot))).filter(name => name.startsWith("checkpoint-"))).toEqual([]);
|
||||
expect((await fs.readdir(path.join(copy.executionRoot, ".paperclip-runtime"))).filter(name => name.startsWith("checkpoint-"))).toEqual([]);
|
||||
if (turn < 3) { copy = await prepare(copy.runId); expect(copy.executionRoot).toBe(original.executionRoot); }
|
||||
}
|
||||
await copies.collectStopped({ companyId, runId: original.runId, target: executionTarget });
|
||||
expect(await fs.stat(copy.executionRoot)).toBeTruthy();
|
||||
await copies.collectStopped({ companyId, runId: copy.runId, target: executionTarget });
|
||||
await expect(fs.stat(copy.executionRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
it("stages SSH agent files at the registered root without a nested task workspace", async () => {
|
||||
const remoteCwd = path.join(home, "ssh-task");
|
||||
const exclude = vi.spyOn(executionTargetTools, "runAdapterExecutionTargetShellCommand").mockResolvedValue({ exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" });
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { captureAgentFiles } from "../services/scripts/agent-file-checkpoint.mjs";
|
||||
import { captureAgentFileCheckpoint, validateAgentFileCheckpoint } from "../services/agent-file-checkpoints.js";
|
||||
|
||||
describe("incremental managed file checkpoints", () => {
|
||||
let root: string;
|
||||
beforeEach(async () => { root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "agent-checkpoint-"))); await fs.mkdir(path.join(root, "live")); });
|
||||
afterEach(async () => { vi.restoreAllMocks(); await fs.rm(root, { recursive: true, force: true }); });
|
||||
const live = () => path.join(root, "live");
|
||||
it("does not read or copy a large unchanged binary when only a note changes", async () => {
|
||||
const image = Buffer.alloc(64 * 1024 * 1024, 0x93);
|
||||
await fs.writeFile(path.join(live(), "image.bin"), image);
|
||||
await fs.writeFile(path.join(live(), "note.txt"), "one");
|
||||
const first = await captureAgentFiles(live());
|
||||
await fs.writeFile(path.join(live(), "note.txt"), "two");
|
||||
const delta = await captureAgentFileCheckpoint({ runId: "fixture", localRoot: live(), executionRoot: live(), previous: first.manifest });
|
||||
try {
|
||||
expect(delta.stats.hashedBytes).toBe(3);
|
||||
expect(delta.stats.copiedBytes).toBe(3);
|
||||
expect(delta.stats.copiedFiles).toBe(1);
|
||||
expect(await fs.readFile(path.join(delta.directory, "files", "note.txt"), "utf8")).toBe("two");
|
||||
await expect(fs.stat(path.join(delta.directory, "files", "image.bin"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(delta.manifest.entries.find(([p]) => p === "image.bin")?.[1].hash).toBe(first.manifest.entries.find(([p]) => p === "image.bin")?.[1].hash);
|
||||
} finally { await delta.cleanup(); }
|
||||
});
|
||||
it("records deletions and empty directories without resending unchanged data", async () => {
|
||||
await fs.writeFile(path.join(live(), "deleted"), "gone");
|
||||
await fs.writeFile(path.join(live(), "kept"), "keep");
|
||||
const first = await captureAgentFiles(live());
|
||||
await fs.rm(path.join(live(), "deleted"));
|
||||
await fs.mkdir(path.join(live(), "empty"));
|
||||
const next = await captureAgentFiles(live(), first.manifest, path.join(root, "delta"));
|
||||
expect(next.stats.hashedBytes).toBe(0);
|
||||
expect(next.stats.copiedBytes).toBe(0);
|
||||
expect(next.manifest.entries.map(([p]) => p)).toEqual(["empty", "kept"]);
|
||||
});
|
||||
it("detects same-size writes with a restored mtime", async () => {
|
||||
const file = path.join(live(), "note");
|
||||
await fs.writeFile(file, "aaaa");
|
||||
const before = await fs.stat(file);
|
||||
const first = await captureAgentFiles(live());
|
||||
await fs.writeFile(file, "bbbb");
|
||||
await fs.utimes(file, before.atime, before.mtime);
|
||||
const next = await captureAgentFiles(live(), first.manifest, path.join(root, "delta"));
|
||||
expect(next.stats.copiedBytes).toBe(4);
|
||||
expect(next.manifest.entries[0]![1].hash).not.toBe(first.manifest.entries[0]![1].hash);
|
||||
});
|
||||
it.each(["symlink", "hardlink"])("refuses a %s without publishing a manifest", async kind => {
|
||||
const outside = path.join(root, "outside"); await fs.writeFile(outside, "private");
|
||||
if (kind === "symlink") await fs.symlink(outside, path.join(live(), "link"));
|
||||
else await fs.link(outside, path.join(live(), "link"));
|
||||
await expect(captureAgentFiles(live(), undefined, path.join(root, "delta"))).rejects.toMatchObject({ code: "AGENT_FILES_UNSAFE_PATH" });
|
||||
await expect(fs.stat(path.join(root, "delta", "checkpoint.json"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
it("rejects a sparse oversized file before reading its bytes", async () => {
|
||||
const f = await fs.open(path.join(live(), "large"), "w"); await f.truncate(256 * 1024 * 1024 + 1); await f.close();
|
||||
await expect(captureAgentFiles(live())).rejects.toMatchObject({ code: "AGENT_FILES_LIMIT_EXCEEDED" });
|
||||
});
|
||||
it("validates received contents independently of the remote manifest", async () => {
|
||||
await fs.writeFile(path.join(live(), "note"), "expected");
|
||||
const output = path.join(root, "delta");
|
||||
await captureAgentFiles(live(), undefined, output);
|
||||
await fs.writeFile(path.join(output, "files", "note"), "tampered");
|
||||
await expect(validateAgentFileCheckpoint(output, { entries: [] })).rejects.toThrow("payload mismatch");
|
||||
});
|
||||
it.each(["../escape", "/absolute", "nested/.paperclip-runtime/secret"])("rejects a forged checkpoint path %s", async name => {
|
||||
const output = path.join(root, "delta"); await fs.mkdir(output);
|
||||
await fs.writeFile(path.join(output, "checkpoint.json"), JSON.stringify({ version: 1, entries: [[name, { kind: "dir" }]] }));
|
||||
await expect(validateAgentFileCheckpoint(output, { entries: [] })).rejects.toThrow("AGENT_FILES_UNSAFE_PATH");
|
||||
});
|
||||
it.skipIf(process.platform !== "darwin")("accepts root-owned macOS temp aliases while rejecting user symlink roots", async () => {
|
||||
await fs.writeFile(path.join(live(), "note"), "one");
|
||||
const alias = live().replace(/^\/private(?=\/(var|tmp)\/)/, "");
|
||||
expect((await captureAgentFiles(alias)).manifest.entries).toHaveLength(1);
|
||||
await fs.symlink(live(), path.join(root, "user-link"));
|
||||
await expect(captureAgentFiles(path.join(root, "user-link"))).rejects.toThrow("AGENT_FILES_UNSAFE_PATH");
|
||||
});
|
||||
it("rejects a file growing while its bytes are streamed", async () => {
|
||||
const filename = path.join(live(), "writer");
|
||||
await fs.writeFile(filename, Buffer.alloc(1024 * 1024, 1));
|
||||
const original = fs.open.bind(fs);
|
||||
let inject = true;
|
||||
vi.spyOn(fs, "open").mockImplementation(async (...args) => {
|
||||
const handle = await original(...args);
|
||||
if (String(args[0]) === filename && inject) {
|
||||
inject = false;
|
||||
const stream = handle.createReadStream.bind(handle);
|
||||
handle.createReadStream = ((options: Parameters<typeof stream>[0]) => (async function* () {
|
||||
let first = true;
|
||||
for await (const chunk of stream(options)) {
|
||||
if (first) { first = false; await fs.appendFile(filename, Buffer.alloc(1024 * 1024, 2)); }
|
||||
yield chunk;
|
||||
}
|
||||
})()) as typeof handle.createReadStream;
|
||||
}
|
||||
return handle;
|
||||
});
|
||||
await expect(captureAgentFiles(live(), undefined, path.join(root, "delta"))).rejects.toMatchObject({ code: "AGENT_FILES_CHANGED_DURING_CHECKPOINT" });
|
||||
await expect(fs.stat(path.join(root, "delta", "checkpoint.json"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
it("detects mode changes and renames without retransferring unrelated files", async () => {
|
||||
await fs.writeFile(path.join(live(), "script"), "execute");
|
||||
await fs.writeFile(path.join(live(), "rename"), "move");
|
||||
const first = await captureAgentFiles(live());
|
||||
await fs.chmod(path.join(live(), "script"), 0o755);
|
||||
await fs.rename(path.join(live(), "rename"), path.join(live(), "renamed"));
|
||||
const next = await captureAgentFiles(live(), first.manifest, path.join(root, "delta"));
|
||||
expect(next.stats.copiedFiles).toBe(2);
|
||||
expect(next.manifest.entries.map(([name]) => name)).toEqual(["renamed", "script"]);
|
||||
expect((await fs.stat(path.join(root, "delta", "files", "script"))).mode & 0o777).toBe(0o755);
|
||||
});
|
||||
it("excludes only the remote transport's reserved root, never user file paths", async () => {
|
||||
await fs.mkdir(path.join(live(), ".paperclip-runtime"));
|
||||
await fs.symlink("/missing", path.join(live(), ".paperclip-runtime", "transport-only"));
|
||||
await fs.writeFile(path.join(live(), "note"), "saved");
|
||||
await expect(captureAgentFiles(live())).rejects.toThrow("AGENT_FILES_UNSAFE_PATH");
|
||||
const captured = await captureAgentFiles(live(), undefined, undefined, true, true);
|
||||
expect(captured.manifest.entries.map(([name]) => name)).toEqual(["note"]);
|
||||
});
|
||||
it("does not acknowledge a write made after the captured generation", async () => {
|
||||
await fs.writeFile(path.join(live(), "note"), "one");
|
||||
const first = await captureAgentFiles(live(), undefined, path.join(root, "first"));
|
||||
await fs.writeFile(path.join(live(), "note"), "two");
|
||||
const second = await captureAgentFiles(live(), first.manifest, path.join(root, "second"));
|
||||
expect(second.stats.copiedFiles).toBe(1);
|
||||
expect(await fs.readFile(path.join(root, "first", "files", "note"), "utf8")).toBe("one");
|
||||
expect(await fs.readFile(path.join(root, "second", "files", "note"), "utf8")).toBe("two");
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user