From 17e8bd773543d62ab3f444c55255acfd82575efd Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Tue, 3 Nov 2026 05:09:52 -0800 Subject: [PATCH] fix(runner): save native checkpoints when the sandbox disk is full (#18705) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox save two things after each turn: the provider checkpoint and the workspace export > - An agent can fill the sandbox disk during its own work, for example with a package install > - The turn still finishes, but both saves need free sandbox disk, so they fail and the task becomes blocked with no clear reason > - This pull request makes the checkpoint work on a full disk and makes a full disk visible in the export failure > - The benefit is fewer stranded checkpoints, and an operator sees at once why an export failed ## Linked Issues or Issue Description Related: #18715 added the shared-principal response-recovery control synchronized below. #18644 reduced the scratch disk that the workspace capture uses. #18172 added the disk-free streaming fallback to the Daytona directory sync-out. Both were deployed when this failure occurred. Neither changes the checkpoint copy, and neither can help when the disk has no free space. **What happened?** A native agent ran a package install for a large monorepo in a 10 GB Daytona sandbox. `df` in the run showed `10G 10G 4.6M 100%`. The agent committed its work and reported `done`. Then: 1. The provider checkpoint made a filtered copy of the provider directory inside the sandbox (`tar -cf archive; tar -xf archive -C snapshot`) to apply its exclusions. On the full disk, tar failed. The run recorded `runner_remote_checkpoint_snapshot_failed` without the reason, and the partial copy stayed on the disk. 2. Workspace finalization failed three times, and file preservation failed six times. The capture builds its archive inside the sandbox, so each attempt failed. Every failure was recorded as `remote_workspace_operation_failed` / `export_interrupted`, so nothing named the full disk. 3. The task went to `blocked`. The only copy of the commit was in the retained sandbox. **Expected behavior** The checkpoint does not need free sandbox disk when the provider can stream. A workspace export that cannot fit says that the sandbox disk is full. **Steps to reproduce** 1. Run a native agent with a provider whose checkpoint has exclusions (for example ACPX) in a small remote sandbox. 2. In the turn, fill the sandbox disk (for example `pnpm install` of a large repository, or `fallocate` of the free space) and finish the task. 3. The checkpoint fails with `runner_remote_checkpoint_snapshot_failed`, and every workspace export attempt fails with a generic code. **Paperclip version or commit** Observed on `410b0e27015`. This change is based on `5955dd4eb4c`. ## What Changed - Synchronize the shared-principal response-recovery regression with its saved comments and empty final deadline. Cover an in-flight discovery invalidation. Keep every authority lock, exact publication, error, and no-retry assertion. This changes test synchronization only. - `syncRemoteRunnerDirectoryOut`: when the in-sandbox checkpoint copy fails with a full-disk tar diagnostic, remove the partial copy and pass the exclusions to the provider sync-out (`exclude` on the directory mapping). The Daytona sync-out streams the directory without sandbox disk when it cannot write its archive. Other copy failures and timeouts still throw `runner_remote_checkpoint_snapshot_failed`. They now also remove the partial copy. - `remote-workspace-transport.ts`: the helper install and the capture scripts report `remote_workspace_sandbox_disk_full: ...` when Node reports `ENOSPC` or `EDQUOT`, or a Git or tar child reports "No space left on device". `command()` passes only that exact message through, like the existing unfinished-Git-operation message. - `native-workspace-finalizer.ts`: a remote save that fails with that message adds it to the failed operation's stderr. The failure code, the retry budget, and the recovery flow do not change. ## Verification - Current-head infrastructure interruption: the [Build job](https://github.com/paperclipai/paperclip/actions/runs/41637668555/job/125923826158) completed the actual build and production bundle boot, then the runner received a shutdown signal during the clean public Grok npm-install check. GitHub marked that step cancelled. The exact host-stop cause was not retained. One bounded rerun of only the failed Build and dependent `ci / verify` jobs passed on the unchanged head. [Build attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41637668555/job/125931400745) completed successfully, including the clean Grok npm-install check. No product code, assertion or workflow changed for this interruption. - CI failure retained: commit `6c5937fcae` failed [chat shard 1/3](https://github.com/paperclipai/paperclip/actions/runs/41634638570/job/125913957217) because the shared-principal fixture saw zero saved comments after `worker.ready` and zero active work. The other chat child passed. The exact case passed once locally and then 30 times on the same Node 24.22.2 version as CI; those passes alone do not explain the CI interleaving. - Controlled RED/GREEN: invalidate the real database discovery once while its read is in flight. The worker correctly retains a future reconstruction deadline, but the old initial-pass/active-count barrier reproduces the same zero-versus-one comment assertion. Waiting for both saved comments and final `active=0`/`nextWakeAt=null` passes. Six real-PostgreSQL controls pass on final commit `a60e5629e8`, including endpoint independence, source revocation, and external lock failure/recovery. No sleep, timeout increase, weakened assertion, or product scheduling/authorization change was added. - A local server typecheck first lacked the checkout's built Runner declarations. After the declared Runner TypeScript build, the server typecheck passes on `a60e5629e8`. Final-head remote verification is complete: 54 successful checks, 3 expected skips, no failures or pending checks, and successful `ci / verify` and `ci / e2e` aggregates. Greptile is 5/5 on this exact commit with no unresolved review threads. - Merge preparation: rebased onto master `b22ef22d56` and kept both test-import additions. Added the checkpoint fallback and capacity-failure contract to `doc/execution-semantics.md`. On rebased commit `6c5937fcae`, 47 focused checkpoint, transport, finalization and retained warm-projection tests passed across three files. The name filter skipped 1,040 unrelated cases. Map and whitespace checks passed. The final-head CI and Greptile results are recorded above. - `npx vitest run server/src/services/native-runtime/native-session-executor.test.ts server/src/services/native-runtime/native-workspace-finalizer.test.ts server/src/services/remote-workspace-snapshot-archive.test.ts` passes (1,152 tests). - `npx vitest run server/src/services/remote-workspace-transport.test.ts server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts` passes (30 tests). - New tests: - Checkpoint, with real files: a full-disk snapshot leaves a partial copy. The provider then gets the exclusions, the session files are kept, the package caches are left out, and the partial copy is removed. - Checkpoint: a permission failure and a timeout still fail, remove the partial copy, and do not call the sync-out. - Transport: the real capture scripts run with an `ENOSPC` preload, for the helper install and for the archive write. Both report the full-disk message and leave no partial attempt. A message with extra text still collapses to `remote_workspace_operation_failed`. - Finalizer: the operation stderr names the full disk and keeps `workspace_sync_out_failed`. Other failures keep the bare code. - With the source change removed, the 3 checkpoint cases and the 2 transport ENOSPC cases fail. - `npx tsc --noEmit -p server/tsconfig.json` is clean. - Suites that refer to the changed error codes (`explicit-native-continuation`, `native-recovery-notice-work`, `native-run-finalizer-telemetry`, `native-workspace-export-retry`, `remote-workspaces.integration`) pass. In `chat-channels.integration`, 2 cases fail locally on unmodified `master` too (a bundled skill path outside the checkout, and a chat run timing assertion). They are not related to this change. - Manual recovery of the observed task: the retained sandbox was resumed with free space, file preservation was requested again, and the saved workspace contained the agent's commit. ## Risks - The fallback runs only after the snapshot fails with a full-disk diagnostic. The normal path does not change. - Provider exclusions are not anchored to the directory root. In the fallback, a nested entry that has an excluded name (for example a nested `tmp` or `config.toml` in a Codex home) is also left out. The fallback never copies more than the snapshot, so credentials stay out of the checkpoint. - The fallback depends on the provider honoring `exclude`, which the sync protocol requires. The Daytona and Kubernetes providers honor it. - A workspace export on a full disk still fails. This change only names the cause. A full fix needs one of these: free-space reserve for the export, removal of re-creatable caches before the export, or a capture that streams its archive. Each of these changes behavior, so this pull request does not include one. - No migration. ## Model Used - OpenAI Codex, GPT-6 family, assisted with merge preparation, conflict resolution, documentation, test synchronization and verification. The harness does not expose the exact serving model ID or context-window size. - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). It did the root-cause analysis from run events, tool output, and sandbox inspection, wrote the code and tests, and ran local verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Dotta --- doc/execution-semantics.md | 9 ++ .../chat-channels.integration.test.ts | 28 ++++++- .../native-session-executor.test.ts | 82 ++++++++++++++++++- .../native-runtime/native-session-executor.ts | 55 ++++++++++--- ...native-workspace-finalizer-failure.test.ts | 36 ++++++-- .../native-workspace-finalizer.ts | 9 +- .../remote-workspace-transport.test.ts | 36 ++++++++ .../services/remote-workspace-transport.ts | 23 ++++-- 8 files changed, 252 insertions(+), 26 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index dceb36391cd..1731fc5df0e 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2377,6 +2377,15 @@ runnerd recovery, and durable control-plane readers. These paths synchronously read and parse the full JSON file, so memory use and parse time grow with file size. Remote checkpoint archive and expanded-size limits remain 64 MiB. +Remote provider checkpoints normally apply exclusions through a filtered copy +inside the sandbox. If that copy fails with a full-disk diagnostic, the controller +removes the partial copy and passes the exclusions to the provider's sync-out. +That transfer can stream without another sandbox archive. Provider exclusions can +also omit a nested entry with the same name; credentials remain excluded. Other +copy failures and timeouts still fail. Workspace capture itself still needs free +sandbox space. Its failed save operation names `remote_workspace_sandbox_disk_full` +when that capacity is exhausted, without changing retry or ownership rules. + Native assigned app tools save governed responses above 500,000 encoded bytes before returning a bounded reference through PRP. `read_tool_result` retrieves the saved JSON in UTF-8 pages under the same task, agent, and responsible-user access checks. diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index a139347bed6..109b25df894 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -74864,7 +74864,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { } }); - it("settles responses on different endpoints for one principal without retrying its own identity lock", async () => { + it.each(["initial", "rescoped"] as const)("settles responses on different endpoints for one principal without retrying its own identity lock: %s", async (mode) => { const fixture = await seedCompany(); const first = await configuredSlackEndpoint(fixture); const second = await configuredSlackEndpoint(fixture); @@ -74882,10 +74882,36 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { const functionName = `hold_identity_${randomUUID().replaceAll("-", "")}`; await db.execute(sql.raw(`create function ${functionName}() returns trigger language plpgsql as $$ begin perform pg_sleep(0.1); return NEW; end $$`)); await db.execute(sql.raw(`create trigger ${functionName} before insert on issue_comments for each row execute function ${functionName}()`)); + const scopedWork = await import("../services/scoped-deadline-work.js"); + const register = scopedWork.registerScopedDeadlineWork; + let invalidated = false; + let invalidateSource: (() => void) | undefined; + const registration = vi.spyOn(scopedWork, "registerScopedDeadlineWork").mockImplementation((owner, options, canRun) => + register(owner, { ...options, load: async (scope, signal) => { + const entries = await options.load(scope, signal); + // An authority registration can invalidate discovery while its read is + // in flight. The real worker retains another pass under its deadline. + if (mode === "rescoped" && !invalidated) { + invalidated = true; + invalidateSource!(); + } + return entries; + } }, canRun)); const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + invalidateSource = () => worker.invalidate(); + registration.mockRestore(); try { await worker.ready; await expect.poll(() => idleWorkSnapshot().active, { timeout: 10_000 }).toBe(0); + if (mode === "rescoped") { + expect(invalidated).toBe(true); + expect(coordinator.nextWakeAt()).not.toBeNull(); + } + await expect.poll(async () => Promise.all(contexts.map(async context => + (await db.select({ id: issueComments.id }).from(issueComments) + .where(eq(issueComments.createdByRunId, context.runId))).length)), { timeout: 10_000 }).toEqual([1, 1]); + await expect.poll(() => ({ active: idleWorkSnapshot().active, nextWakeAt: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, nextWakeAt: null }); expect(errors).toEqual([]); for (const context of contexts) { const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId)); diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 57b549bffd1..1ee386fce17 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -37,7 +37,7 @@ import { import { execFile, execFileSync, spawn, spawnSync } from "node:child_process"; import { tmpdir } from "node:os"; import { inspect } from "node:util"; -import { basename, dirname, join } from "node:path"; +import { basename, dirname, join, relative } from "node:path"; import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; import { activityLog, @@ -3421,6 +3421,86 @@ describe("remote provider checkpoint snapshots", () => { expect(syncOut).toHaveBeenCalledOnce(); }); + it("lets the provider apply the exclusions when the sandbox disk cannot hold the snapshot", async () => { + const root = await mkdtemp(join(tmpdir(), "provider-cache-checkpoint-full-")); + try { + const live = join(root, "live", "acpx"); + const backup = join(root, "backup", "acpx"); + const sessionPath = `acpx/${acpxRuntimeSessionDirectoryName("session")}`; + const files: Record = { + "pi-home/sessions/session.jsonl": "history", + "data/pnpm/config.json": "{}", + "home/.npm/_cacache/index": "npm-cache", + "data/pnpm/store/v10/files/00/blob": "pnpm-store", + }; + for (const [path, contents] of Object.entries(files)) { + await mkdir(join(live, sessionPath, path, ".."), { recursive: true }); + await writeFile(join(live, sessionPath, path), contents); + } + const excludeEntries = resolveNativeHarnessPersistenceProfile({ + provider: { kind: "acpx", agent: "pi" }, + session: { normalizedSessionId: "session", driverKind: "acpx_runtime" }, + } as unknown as NativeExecutionInputV1).directories.find(entry => entry.name === "acpx")!.excludeEntries; + const commands: string[] = []; + const execute = vi.fn(async ({ command, args }: { command: string; args: string[] }) => { + const script = String(args[1]); + commands.push(script); + if (script.includes(" -cf ")) { + // Leave a partial copy behind, as tar does when the disk fills. + const snapshot = /mkdir -p '([^']+)'/.exec(script)![1]!; + await mkdir(join(snapshot, sessionPath), { recursive: true }); + return { exitCode: 2, timedOut: false, stdout: "", + stderr: "tar: ./acpx/session/pi-home/sessions/session.jsonl: Wrote only 4096 of 10240 bytes\n" }; + } + return { exitCode: 0, timedOut: false, stdout: execFileSync(command, args, { encoding: "utf8" }), stderr: "" }; + }); + // Provider exclusions match after any `/`, like GNU tar --exclude. + const syncOut = vi.fn(async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string; exclude?: string[] }> }>) => { + for (const operation of operations) for (const file of operation.files) { + await cp(file.sourcePath, file.targetPath, { recursive: true, filter: (path) => { + const entry = relative(file.sourcePath, path); + return !(file.exclude ?? []).some(pattern => entry === pattern || entry.endsWith(`/${pattern}`)); + } }); + } + }); + + await syncRemoteRunnerDirectoryOut({ runner: { execute, syncOut } as never, + sourcePath: live, targetPath: backup, mode: 0o700, excludeEntries }); + + expect(syncOut).toHaveBeenCalledOnce(); + expect(syncOut.mock.calls[0]![0][0]!.files[0]).toMatchObject({ sourcePath: live, exclude: [...excludeEntries] }); + expect(await readFile(join(backup, sessionPath, "pi-home/sessions/session.jsonl"), "utf8")).toBe("history"); + expect(await readFile(join(backup, sessionPath, "data/pnpm/config.json"), "utf8")).toBe("{}"); + for (const path of ["home/.npm", "data/pnpm/store"]) + await expect(access(join(backup, sessionPath, path))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await readdir(join(root, "live"))).toEqual(["acpx"]); + expect(commands.filter(script => script.startsWith("rm -rf -- "))).toHaveLength(1); + } finally { await rm(root, { recursive: true, force: true }); } + }); + + it.each([ + { name: "a tar failure that is not a full disk", stderr: "tar: ./auth.json: Cannot open: Permission denied\n", timedOut: false }, + { name: "a timeout", stderr: "No space left on device\n", timedOut: true }, + ])("removes the partial snapshot and does not widen the copy after $name", async ({ stderr, timedOut }) => { + const execute = vi.fn() + .mockResolvedValueOnce({ exitCode: 0, timedOut: false, stdout: "", stderr: "" }) + .mockResolvedValueOnce({ exitCode: 2, timedOut, stdout: "", stderr }) + .mockResolvedValue({ exitCode: 0, timedOut: false, stdout: "", stderr: "" }); + const syncOut = vi.fn(async () => undefined); + await expect(syncRemoteRunnerDirectoryOut({ + runner: { execute, syncOut } as never, + sourcePath: "/remote/session/filesystem/codex-home", + targetPath: "/tmp/paperclip-checkpoint-test-codex-home", + mode: 0o700, + excludeEntries: ["tmp", "auth.json"], + })).rejects.toThrow("runner_remote_checkpoint_snapshot_failed"); + expect(syncOut).not.toHaveBeenCalled(); + expect(execute).toHaveBeenCalledTimes(3); + expect(String(execute.mock.calls[2]?.[0]?.args?.[1])).toMatch( + /^rm -rf -- '\/remote\/session\/filesystem\/\.paperclip-checkpoint-/, + ); + }); + it("rejects unsafe relative checkpoint exclusions", async () => { const execute = vi.fn().mockResolvedValue({ exitCode: 0, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index f6df58888bd..da9b2b858cf 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -11205,6 +11205,25 @@ function archiveExcludeArgs(entries: readonly string[]): string[] { return entries.map((entry) => `--exclude=./${entry}`); } +// The provider sync-out recognizes the same tar diagnostics for a full disk. +const SANDBOX_DISK_FULL_PATTERN = + /No space left on device|Disk quota exceeded|File\s?size limit exceeded|Wrote only \d+ of \d+ bytes/i; + +async function removeRemoteCheckpointSnapshot( + runner: CommandManagedRuntimeRunner, + snapshotPath: string, +): Promise { + const escapedSnapshot = snapshotPath.replaceAll("'", "'\\''"); + await runner + .execute({ + command: "sh", + args: ["-c", `rm -rf -- '${escapedSnapshot}'`], + bypassSession: true, + timeoutMs: 30_000, + }) + .catch(() => undefined); +} + export async function stageRemoteRunnerDirectory(input: { target: Extract; runner: CommandManagedRuntimeRunner; @@ -11540,6 +11559,14 @@ function assertSafeExtractedCheckpoint(root: string): void { * provider supplies a native syncOut implementation, create a sibling snapshot * first so the live runtime remains untouched and the provider's archive safety * checks still apply to every persisted entry. + * + * The snapshot needs free sandbox disk. When the disk is full, the provider + * applies the exclusions itself: its transfer can stream the directory without + * writing an archive into the sandbox. Provider exclusions are not anchored to + * the directory root, so this fallback also leaves out a nested entry that has + * an excluded name (for example a nested `tmp`). It never copies more than the + * snapshot would, and a checkpoint without those entries is more useful than + * no checkpoint. */ export async function syncRemoteRunnerDirectoryOut(input: { runner: CommandManagedRuntimeRunner; @@ -11564,6 +11591,7 @@ export async function syncRemoteRunnerDirectoryOut(input: { if (input.runner.syncOut) { let syncSourcePath = input.sourcePath; let snapshotPath: string | null = null; + let providerExclude: string[] | null = null; if (excluded.length > 0) { const checkpointId = randomUUID(); snapshotPath = posix.join( @@ -11587,9 +11615,21 @@ export async function syncRemoteRunnerDirectoryOut(input: { timeoutMs: 120_000, }); if (snapshotResult.exitCode !== 0 || snapshotResult.timedOut) { - throw new Error("runner_remote_checkpoint_snapshot_failed"); + // A partial copy would hold sandbox disk that the next save needs. + await removeRemoteCheckpointSnapshot(input.runner, snapshotPath); + snapshotPath = null; + if ( + snapshotResult.timedOut || + !SANDBOX_DISK_FULL_PATTERN.test( + `${snapshotResult.stderr}\n${snapshotResult.stdout}`, + ) + ) { + throw new Error("runner_remote_checkpoint_snapshot_failed"); + } + providerExclude = [...excluded]; + } else { + syncSourcePath = snapshotPath; } - syncSourcePath = snapshotPath; } try { await input.runner.syncOut([ @@ -11601,21 +11641,14 @@ export async function syncRemoteRunnerDirectoryOut(input: { targetPath: input.targetPath, kind: "directory", mode: input.mode, + ...(providerExclude ? { exclude: providerExclude } : {}), }, ], }, ]); } finally { if (snapshotPath) { - const escapedSnapshot = snapshotPath.replaceAll("'", "'\\''"); - await input.runner - .execute({ - command: "sh", - args: ["-c", `rm -rf -- '${escapedSnapshot}'`], - bypassSession: true, - timeoutMs: 30_000, - }) - .catch(() => undefined); + await removeRemoteCheckpointSnapshot(input.runner, snapshotPath); } } return; diff --git a/server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts b/server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts index 46d6c7cdde4..d5f9cb6e829 100644 --- a/server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts +++ b/server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts @@ -2,7 +2,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import { GitShallowMetadataError } from "@paperclipai/adapter-utils/git-workspace-sync"; import type { Db } from "@paperclipai/db"; -const sync = vi.hoisted(() => ({ resume: vi.fn() })); +const sync = vi.hoisted(() => ({ resume: vi.fn(), save: vi.fn() })); vi.mock("./native-workspace-finalization-ownership.js", () => ({ withNativeWorkspaceFinalizationOwnership: async (_input: unknown, action: (owner: unknown) => unknown) => ({ acquired: true, value: await action({ token: "fixture-owner", assertHeld: async () => {} }) }), @@ -11,8 +11,13 @@ vi.mock("./native-workspace-sync.js", () => ({ readNativeWorkspaceSyncReference: () => ({ leaseId: "lease", providerLeaseId: "sandbox", workspaceId: "workspace" }), resumeNativeWorkspaceSync: sync.resume, })); +vi.mock("../remote-workspaces.js", () => ({ + readRemoteWorkspaceAuthority: (metadata: { remoteWorkspace?: unknown } | null) => metadata?.remoteWorkspace ?? null, + saveRemoteWorkspace: sync.save, +})); vi.mock("../environments.js", () => ({ environmentService: () => ({ - getLeaseById: async () => ({ id: "lease", companyId: "company", environmentId: "environment", providerLeaseId: "sandbox", status: "active" }), + getLeaseById: async () => ({ id: "lease", companyId: "company", environmentId: "environment", providerLeaseId: "sandbox", status: "active", + heartbeatRunId: "run", executionWorkspaceId: "workspace" }), getById: async () => ({ id: "environment", companyId: "company" }), }) })); vi.mock("../environment-execution-target.js", () => ({ resolveEnvironmentExecutionTarget: async () => ({ kind: "remote" }) })); @@ -20,12 +25,13 @@ vi.mock("../workspace-operations.js", () => ({ workspaceOperationService: () => createRecorder: () => ({ recordOperation: async (input: { run: () => Promise }) => input.run() }), }) })); import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer.js"; +import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL } from "../remote-workspace-transport.js"; -function fixtureDb(): Db { +function fixtureDb(workspace?: Record): Db { const responses = [[{ - companyId: "company", runtimeMode: "native", issueId: "issue", resultId: "result", + companyId: "company", runtimeMode: "native", issueId: "issue", resultId: "result", issueWorkspaceId: workspace?.id, runnerProfileJson: { nativeWorkspaceSync: {}, nativeExecutionInput: { binding: {} } }, - }], [{ phase: "result_accepted", nextAttemptAt: null, resultId: "result" }], []]; + }], [{ phase: "result_accepted", nextAttemptAt: null, resultId: "result" }], [], [], ...(workspace ? [[workspace]] : [])]; return { select: () => { const query = { from: () => query, innerJoin: () => query, where: () => query, orderBy: () => query, @@ -70,3 +76,23 @@ describe("native workspace finalization without a manual export repair", () => { expect(result).toMatchObject({ status: "failed", exitCode: 1, stderr: "workspace_sync_out_unrecoverable\n" }); }); }); + +describe("native remote workspace save failures", () => { + const workspace = { id: "workspace", companyId: "company", cwd: "/workspace", + metadata: { remoteWorkspace: { location: { leaseId: "lease", providerLeaseId: "sandbox", environmentId: "environment" } } } }; + beforeEach(() => sync.save.mockReset()); + + it("names a full sandbox disk on the failed operation without changing the retry code", async () => { + sync.save.mockRejectedValueOnce(new Error(REMOTE_WORKSPACE_SANDBOX_DISK_FULL)); + const result = await resumeNativeWorkspaceFinalization({ db: fixtureDb(workspace), runId: "run", environmentRuntime: {} as never }); + expect(sync.save).toHaveBeenCalledOnce(); + expect(result).toMatchObject({ status: "failed", exitCode: 1, + stderr: `workspace_sync_out_failed\n${REMOTE_WORKSPACE_SANDBOX_DISK_FULL}\n`, metadata: { workspaceSync: { code: "workspace_sync_out_failed" } } }); + }); + + it("keeps other save failures to the bare code", async () => { + sync.save.mockRejectedValueOnce(new Error("remote_workspace_operation_failed")); + const result = await resumeNativeWorkspaceFinalization({ db: fixtureDb(workspace), runId: "run", environmentRuntime: {} as never }); + expect(result).toMatchObject({ status: "failed", exitCode: 1, stderr: "workspace_sync_out_failed\n" }); + }); +}); diff --git a/server/src/services/native-runtime/native-workspace-finalizer.ts b/server/src/services/native-runtime/native-workspace-finalizer.ts index 1094de249b4..f7528ce82a2 100644 --- a/server/src/services/native-runtime/native-workspace-finalizer.ts +++ b/server/src/services/native-runtime/native-workspace-finalizer.ts @@ -1,5 +1,6 @@ import { classifyNativeWorkspaceFailure } from "./native-workspace-failure.js"; import { readRemoteWorkspaceAuthority, saveRemoteWorkspace } from "../remote-workspaces.js"; +import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL } from "../remote-workspace-transport.js"; import { recordNativeFinalizationFailure } from "./native-run-finalizer.js"; import { inspectPluginWorkspaceForFinalization } from "./native-plugin-workspace-finalization.js"; import type { WorkspaceWorkers } from "../plugin-workspaces.js"; @@ -38,11 +39,12 @@ function readString(value: unknown) { function workspaceSyncFailure( code: "workspace_sync_out_failed" | "workspace_sync_out_unrecoverable", + cause?: string, ) { return { status: "failed" as const, exitCode: 1, - stderr: `${code}\n`, + stderr: cause ? `${code}\n${cause}\n` : `${code}\n`, metadata: { workspaceSync: { code } }, }; } @@ -190,8 +192,11 @@ export async function attemptNativeWorkspaceFinalization(input: { metadata: { remoteWorkspace: saved } }; } catch (error) { await ownership.assertHeld(); + // The code keeps the retry policy. A full sandbox disk is named on the + // operation, because no retry can save the workspace until space is freed. + const diskFull = error instanceof Error && error.message === REMOTE_WORKSPACE_SANDBOX_DISK_FULL; return workspaceSyncFailure(classifyNativeWorkspaceFailure(error).code === "workspace_sync_out_unrecoverable" - ? "workspace_sync_out_unrecoverable" : "workspace_sync_out_failed"); + ? "workspace_sync_out_unrecoverable" : "workspace_sync_out_failed", diskFull ? REMOTE_WORKSPACE_SANDBOX_DISK_FULL : undefined); } } if (nativeWorkspaceSync) { diff --git a/server/src/services/remote-workspace-transport.test.ts b/server/src/services/remote-workspace-transport.test.ts index be74e5fefbd..a78612d2832 100644 --- a/server/src/services/remote-workspace-transport.test.ts +++ b/server/src/services/remote-workspace-transport.test.ts @@ -311,6 +311,42 @@ describe("remote workspace transport safety with real sandbox commands", () => { .rejects.toThrow(/^remote_workspace_operation_failed$/); }); + it.each([ + { step: "helper install", failing: String.raw`/archive\.cjs\.[^/]+\.tmp$/` }, + { step: "archive write", failing: String.raw`/workspace\.pcws$/` }, + ])("names a full sandbox disk when the $step runs out of space", async ({ failing }) => { + await fs.writeFile(path.join(workspace, "result.txt"), "synthetic saved result\n"); + // Every sandbox Node process inherits this preload. A matching write keeps + // one byte and then fails with ENOSPC, as a real full disk does. + const preload = path.join(scratch, "enospc.cjs"); + await fs.writeFile(preload, String.raw`const fs=require('node:fs/promises'),failing=${failing}; + const enospc=file=>Object.assign(new Error("ENOSPC: no space left on device, write '"+file+"'"),{code:'ENOSPC',errno:-28,syscall:'write'}); + const writeFile=fs.writeFile,open=fs.open; + fs.writeFile=async(file,data,...rest)=>{if(!failing.test(String(file)))return writeFile(file,data,...rest); + await writeFile(file,Buffer.from(data).subarray(0,1),...rest);throw enospc(file);}; + fs.open=async(file,...rest)=>{const handle=await open(file,...rest);if(failing.test(String(file))){const write=handle.writeFile.bind(handle); + handle.writeFile=async data=>{await write(Buffer.from(data).subarray(0,1));throw enospc(file);};}return handle;}; + require('node:module').syncBuiltinESMExports();`); + const { remote, execute } = target(); + const run = execute.getMockImplementation()!; + execute.mockImplementation(input => run({ ...input, env: { ...input.env, NODE_OPTIONS: `--require ${preload}` } })); + const operationId = randomUUID(); + await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId })) + .rejects.toThrow(/^remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace$/); + const directory = path.join(workspace, ".paperclip-runtime", "workspace-snapshots", operationId); + const leftovers = await fs.readdir(directory).catch(() => []); + expect(leftovers.filter(name => name !== "archive.cjs")).toEqual([]); + }); + + it("keeps the full-disk diagnostic exact", async () => { + const { remote, execute } = target(); + execute.mockResolvedValueOnce({ exitCode: 1, stdout: "", timedOut: false, signal: null, pid: process.pid, + startedAt: new Date().toISOString(), + stderr: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace\nSYNTHETIC_PRIVATE_PATH" }); + await expect(captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() })) + .rejects.toThrow(/^remote_workspace_operation_failed$/); + }); + it("never follows a preexisting snapshot receipt temporary symlink", async () => { await fs.writeFile(path.join(workspace, "result.txt"), "synthetic saved result\n"); const operationId = randomUUID(); diff --git a/server/src/services/remote-workspace-transport.ts b/server/src/services/remote-workspace-transport.ts index 45a4a2417d5..ab4b17dbbd0 100644 --- a/server/src/services/remote-workspace-transport.ts +++ b/server/src/services/remote-workspace-transport.ts @@ -16,12 +16,19 @@ export function requireRemoteWorkspaceTarget(target: AdapterExecutionTarget | nu return target; } +/** The workspace archive is built inside the sandbox, so a full sandbox disk stops the save. */ +export const REMOTE_WORKSPACE_SANDBOX_DISK_FULL = + "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the workspace"; +// Node reports ENOSPC or EDQUOT. Git and tar child processes report the text. +const DISK_FULL_CHECK = String.raw`const diskFull=e=>['ENOSPC','EDQUOT'].includes(e?.code)||/No space left on device|Disk quota exceeded/i.test(String(e?.stderr??'')+String(e?.message??''));`; + async function command(target: RemoteWorkspaceTarget, args: string[], env?: Record, timeoutMs = 300_000) { const result = await target.runner!.execute({ command: "node", args, cwd: target.remoteCwd, env: { NODE_OPTIONS: "", NODE_PATH: "", ...env }, timeoutMs, bypassSession: true }); if (result.exitCode !== 0 || result.timedOut) { const gitOperation = "remote_workspace_archive_git_operation_in_progress: finish or abort the Git operation before saving the workspace"; - throw new Error(!result.timedOut && result.stderr.trim() === gitOperation ? gitOperation : "remote_workspace_operation_failed"); + const reported = result.timedOut ? "" : result.stderr.trim(); + throw new Error([gitOperation, REMOTE_WORKSPACE_SANDBOX_DISK_FULL].includes(reported) ? reported : "remote_workspace_operation_failed"); } return result.stdout; } @@ -153,13 +160,16 @@ async function installArchiveHelper(target: RemoteWorkspaceTarget, remoteCwd: st if (!/^[a-f0-9-]{36}$/.test(operationId)) throw new Error("remote_workspace_operation_identity_invalid"); const directory = path.posix.join(remoteCwd, ".paperclip-runtime", "workspace-snapshots", operationId); const helper = path.posix.join(directory, "archive.cjs"); - const install = String.raw`const fs=require('node:fs/promises'),path=require('node:path');(async()=>{ + const install = String.raw`const fs=require('node:fs/promises'),path=require('node:path');${DISK_FULL_CHECK}(async()=>{ const [root,dir,file,body]=process.argv.slice(1);let p='/'; for(const s of root.split('/').filter(Boolean)){p=path.join(p,s);await fs.mkdir(p,{mode:0o700}).catch(e=>{if(e.code!=='EEXIST')throw e});const st=await fs.lstat(p);if(st.isSymbolicLink()||!st.isDirectory())throw Error('unsafe');} for(const s of path.relative(root,dir).split('/')){p=path.join(p,s);await fs.mkdir(p,{mode:0o700}).catch(e=>{if(e.code!=='EEXIST')throw e});const st=await fs.lstat(p);if(st.isSymbolicLink()||!st.isDirectory())throw Error('unsafe');} const existing=await fs.lstat(file).catch(e=>{if(e.code==='ENOENT')return null;throw e});if(existing&&(!existing.isFile()||existing.isSymbolicLink()))throw Error('unsafe'); - const temp=file+'.'+require('node:crypto').randomUUID()+'.tmp';await fs.writeFile(temp,Buffer.from(body,'base64'),{mode:0o600,flag:'wx'});await fs.rename(temp,file); - })().catch(()=>{process.exitCode=1});`; + const temp=file+'.'+require('node:crypto').randomUUID()+'.tmp'; + // A failed write can leave partial bytes. Remove only the temporary file this run created. + try{await fs.writeFile(temp,Buffer.from(body,'base64'),{mode:0o600,flag:'wx'});await fs.rename(temp,file);} + catch(e){if(e?.code!=='EEXIST')await fs.rm(temp,{force:true}).catch(()=>{});throw e;} + })().catch(error=>{if(diskFull(error))process.stderr.write(${JSON.stringify(REMOTE_WORKSPACE_SANDBOX_DISK_FULL)});process.exitCode=1});`; await command(target, ["-e", install, remoteCwd, directory, helper, Buffer.from(remoteWorkspaceArchiveScript()).toString("base64")]); return { directory, helper, archive: path.posix.join(directory, "workspace.pcws"), receipt: path.posix.join(directory, "receipt.json") }; } @@ -186,7 +196,7 @@ export async function captureRemoteWorkspaceArchive(input: { const staged = await installArchiveHelper(input.target, input.remoteCwd, input.operationId); // Publish the archive and receipt as one immutable directory. Concurrent // invocations own separate attempts; no invocation unlinks another's archive. - const capture = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process');const {promisify}=require('node:util');(async()=>{ + const capture = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process');const {promisify}=require('node:util');${DISK_FULL_CHECK}(async()=>{ const [helper,root,directory]=process.argv.slice(1),canonical=path.join(directory,'captured'); const readPair=async(dir,layout,required)=>{ const st=await fs.lstat(dir).catch(e=>{if(e.code==='ENOENT')return null;throw e});if(!st)return null; @@ -211,7 +221,8 @@ export async function captureRemoteWorkspaceArchive(input: { }process.stdout.write(JSON.stringify(result));}finally{if(attempt)await fs.rm(attempt,{recursive:true,force:true});} })().catch(error=>{ const gitOperation='remote_workspace_archive_git_operation_in_progress: finish or abort the Git operation before saving the workspace'; - process.stderr.write(typeof error?.stderr==='string'&&error.stderr.trim()===gitOperation?gitOperation:'remote_workspace_capture_failed');process.exitCode=1; + process.stderr.write(typeof error?.stderr==='string'&&error.stderr.trim()===gitOperation?gitOperation + :diskFull(error)?${JSON.stringify(REMOTE_WORKSPACE_SANDBOX_DISK_FULL)}:'remote_workspace_capture_failed');process.exitCode=1; });`; const { layout, ...receipt } = JSON.parse(await command(input.target, ["-e", capture, staged.helper, input.remoteCwd, staged.directory])) as { manifestDigest: string; sha256: string; byteSize: number; entries: number; layout: "legacy" | "captured";