mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a Paperclip-managed sandbox run under the Paperclip runner (`runnerd`), which keeps durable state on the sandbox disk > - An agent can fill that disk with its own work. In one case the agent installed a Rust toolchain and a second checkout with dependencies on a 10 GiB sandbox > - `runnerd` then could not write its state and exited. Each relaunch tried to upload its launch files to the full disk and failed, 83 times in 7 minutes > - The run ended as a generic provider failure that asked a person to inspect it, and Retry repeated the same failure > - This pull request keeps a small disk reserve that belongs to Paperclip. Recovery releases it to relaunch the runner, and stops at once with a named code when nothing is left to free > - The benefit is that the agent's turn can continue and see the full disk in its own tool output, instead of the run dying with it ## Linked Issues or Issue Description **What happened?** An agent in a managed Daytona sandbox filled the 10 GiB disk. `runnerd` exited with `failed to persist bounded diagnostic: No space left on device (os error 28)`. Runner process recovery then retried for the full 420000 ms reconnect grace. Every attempt failed with `.paperclip-upload-<id>.tar.gz: create: ... no space left on device`. The run failed as `native_provider_terminal_failed`, and the recovery asked a board user to verify that the provider stopped. **Steps to reproduce** 1. Run an agent in a managed sandbox and have it fill the disk (for example a Rust toolchain plus a full dependency install). 2. `runnerd` exits when it next writes its state. 3. Watch recovery retry the relaunch upload until the grace expires. **Expected behavior** Paperclip relaunches its runner on a disk the agent filled, so the turn continues and the agent sees the full disk in its tools. When the runner truly cannot start, recovery says so at once with a named cause instead of retrying for 7 minutes. **Environment** Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB disk), `paperclip_runner` native runtime. ## What Changed - New `server/src/services/native-runtime/remote-runner-disk-reserve.ts` owns the reserve: hold/release scripts, the full-disk classifier, and `createRemoteRunnerDiskReserve`. A release waits for an in-flight hold, and no hold starts after a release. - Both remote runner launchers (agent runs and Dot) get a reserve path in managed sandboxes only: `<workspace>/.paperclip-runtime/paperclip-runner/disk-reserve`. Workspace saves and Git already exclude `.paperclip-runtime`. SSH hosts get no reserve. - After the first launch's identity is confirmed, the launcher holds 256 MiB in the background, only while 2 GiB more is free. It is off the startup path. - A relaunch releases the reserve before it stages. A launch that hits a full disk while staging releases the reserve and stages once more. - Release tries twice and reports `released`, `not_held` or `failed`. Only `not_held` ends the launch with `sandbox_disk_full`. A release that cannot run rethrows the original error, so recovery still retries. - Command-based staging (no `syncIn`) keeps a full disk recognizable with an `ENOSPC` code instead of a bare `runner_remote_*_staging_failed`. - `SANDBOX_DISK_FULL_FAILURE_CODE` and `isSandboxDiskFullFailure` live in the runner package. Runner process recovery checks the error code and stops at once instead of retrying for the whole grace. Other staging failures still retry. ## Verification - Red before the fix, green after: - `server/src/services/native-runtime/remote-runner-disk-reserve.test.ts`: release-and-restage, release before relaunch, `sandbox_disk_full` when release frees too little, no retry for a failure that is not a full disk, hold/release scripts in a real shell, no hold below the free-space floor. Before the fix, the launch rejected with the exact upload error from the incident. - `packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts` "fails at once when a replacement cannot be staged because the sandbox disk is full": before the fix it was still retrying after 4 s; after, it fails with `sandbox_disk_full` after one replacement. - Kept: "retries a replacement whose staging failed before any process launched" still passes, so a raw transient staging failure still retries. - Updated: "confirms no replacement process exists when staging fails before the launch RPC" now expects `sandbox_disk_full` with the upload error as cause. Exit is still confirmed and no launch RPC runs. - Real sandbox from the same fleet snapshot (Daytona, 10 GiB): hold created a 256 MiB reserve. The disk was filled to 100%. A 5 MB write then failed. Release returned `true`, and a 50 MB write then succeeded. A second release returned `false`. Hold on the now 98% full disk did not create a reserve. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. `pnpm verify` result is in a comment below. - Review round: tests for a lost release RPC, hold/release ordering, command-based staging on a full disk, the hold after a dispatched first launch (not on relaunch), and no reserve on SSH. Each launcher-level test failed with its fix removed. - Focused suites on the review revision: transport 335/335, launcher 130/130 (filtered), launch assets 35/35, reserve 14/14. Server and runner typecheck pass. - Not run: a full product run in which an agent fills the disk and the relaunched `runnerd` resumes the turn end to end. ## Risks - Each managed sandbox holds 256 MiB more disk while it has more than 2 GiB free. That is 2.5% of a 10 GiB sandbox. - A relaunch for any reason releases the reserve for the rest of that run. The next run's launch takes it again. - If the agent keeps writing after the relaunch, the disk can fill again. The second time, nothing is left to release, and the run fails at once with `sandbox_disk_full` instead of after 7 minutes. - The run's error code is still `native_provider_terminal_failed`. The message now names `sandbox_disk_full`. A dedicated failure code and recovery text are a follow-up. - `runnerd` loses its own error message when it cannot persist its diagnostic. This PR does not change the Rust binary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
122 lines
5.2 KiB
TypeScript
122 lines
5.2 KiB
TypeScript
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
|
import { SANDBOX_DISK_FULL_FAILURE_CODE } from "../../vendor/paperclip-runner/index.js";
|
|
|
|
/**
|
|
* The agent can fill a managed sandbox's disk. runnerd then cannot write its
|
|
* state and exits, and its relaunch cannot upload its launch files. Paperclip
|
|
* holds this much disk in each managed sandbox and releases it to relaunch the
|
|
* runner, so the agent's turn continues and its tools report the full disk.
|
|
*/
|
|
export const REMOTE_RUNNER_DISK_RESERVE_BYTES = 256 * 1024 * 1024;
|
|
// The reserve is taken only while this much more is free, so it never takes
|
|
// space from an agent that is already close to the limit.
|
|
const MINIMUM_FREE_BYTES = 2 * 1024 * 1024 * 1024;
|
|
|
|
type Runner = Pick<CommandManagedRuntimeRunner, "execute">;
|
|
|
|
const HOLD_SCRIPT = [
|
|
'set -u; reserve=$1; bytes=$2; minimum_free_kib=$3; [ -f "$reserve" ] && exit 0',
|
|
'directory=$(dirname -- "$reserve"); mkdir -p -- "$directory" || exit 1',
|
|
`free_kib=$(df -Pk -- "$directory" | awk 'NR == 2 { print $4 }')`,
|
|
'[ -n "$free_kib" ] && [ "$free_kib" -ge "$minimum_free_kib" ] || exit 0',
|
|
'partial="$reserve.partial"; rm -f -- "$partial"',
|
|
'fallocate -l "$bytes" "$partial" 2>/dev/null || dd if=/dev/zero of="$partial" bs=65536 count=$((bytes / 65536)) 2>/dev/null || { rm -f -- "$partial"; exit 1; }',
|
|
'mv -f -- "$partial" "$reserve"',
|
|
].join("\n");
|
|
|
|
// rm needs no free space. The last line names the outcome.
|
|
const RELEASE_SCRIPT =
|
|
'rm -f -- "$1.partial" || exit 1; if [ -f "$1" ]; then rm -f -- "$1" && echo released; else echo not_held; fi';
|
|
|
|
export async function holdRemoteRunnerDiskReserve(
|
|
runner: Runner,
|
|
reservePath: string,
|
|
options: { bytes?: number; minimumFreeBytes?: number } = {},
|
|
): Promise<void> {
|
|
const bytes = options.bytes ?? REMOTE_RUNNER_DISK_RESERVE_BYTES;
|
|
const minimumFreeBytes = options.minimumFreeBytes ?? MINIMUM_FREE_BYTES;
|
|
await runner.execute({
|
|
command: "sh",
|
|
args: ["-c", HOLD_SCRIPT, "paperclip-runner-disk-reserve-hold", reservePath,
|
|
String(bytes), String(Math.ceil((minimumFreeBytes + bytes) / 1024))],
|
|
bypassSession: true,
|
|
timeoutMs: 30_000,
|
|
});
|
|
}
|
|
|
|
export type DiskReserveRelease = "released" | "not_held" | "failed";
|
|
|
|
/** Tries twice: one lost sandbox RPC must not read as "nothing left to free". */
|
|
export async function releaseRemoteRunnerDiskReserve(runner: Runner, reservePath: string): Promise<DiskReserveRelease> {
|
|
for (let attempt = 0; attempt < 2; attempt += 1) {
|
|
const result = await runner.execute({
|
|
command: "sh",
|
|
args: ["-c", RELEASE_SCRIPT, "paperclip-runner-disk-reserve-release", reservePath],
|
|
bypassSession: true,
|
|
timeoutMs: 10_000,
|
|
}).catch(() => null);
|
|
if (result?.exitCode !== 0 || result.timedOut) continue;
|
|
const outcome = result.stdout.trim().split("\n").at(-1);
|
|
if (outcome === "released" || outcome === "not_held") return outcome;
|
|
}
|
|
return "failed";
|
|
}
|
|
|
|
/**
|
|
* The reserve of one runner launcher. A release waits for an in-flight hold,
|
|
* and no hold starts after a release, so a relaunch never stages while the
|
|
* space it freed is taken back.
|
|
*/
|
|
export function createRemoteRunnerDiskReserve(runner: Runner, reservePath: string) {
|
|
let pendingHold: Promise<void> | null = null;
|
|
let releaseRequested = false;
|
|
return {
|
|
hold(): void {
|
|
if (releaseRequested || pendingHold) return;
|
|
pendingHold = holdRemoteRunnerDiskReserve(runner, reservePath).catch(() => undefined);
|
|
},
|
|
async release(): Promise<DiskReserveRelease> {
|
|
releaseRequested = true;
|
|
await pendingHold;
|
|
return releaseRemoteRunnerDiskReserve(runner, reservePath);
|
|
},
|
|
};
|
|
}
|
|
|
|
export type RemoteRunnerDiskReserve = ReturnType<typeof createRemoteRunnerDiskReserve>;
|
|
|
|
export function isSandboxDiskFullError(error: unknown): boolean {
|
|
if (!error || typeof error !== "object") return false;
|
|
const { code, message } = error as { code?: unknown; message?: unknown };
|
|
if (code === "ENOSPC" || code === "EDQUOT") return true;
|
|
return typeof message === "string" && isSandboxDiskFullText(message);
|
|
}
|
|
|
|
export function isSandboxDiskFullText(text: string): boolean {
|
|
return /no space left on device|disk quota exceeded|\bENOSPC\b|\bEDQUOT\b/i.test(text);
|
|
}
|
|
|
|
/** A staging command's failure, keeping a full disk recognizable without its output. */
|
|
export function remoteStagingFailure(
|
|
code: string,
|
|
result: { stdout?: string; stderr?: string },
|
|
): Error {
|
|
return isSandboxDiskFullText(`${result.stderr ?? ""}\n${result.stdout ?? ""}`)
|
|
? Object.assign(new Error(`${code}: no space left on device`), { code: "ENOSPC" })
|
|
: new Error(code);
|
|
}
|
|
|
|
export function sandboxDiskFullFailure(cause: unknown): Error {
|
|
return Object.assign(
|
|
new Error(`${SANDBOX_DISK_FULL_FAILURE_CODE}: the sandbox disk is full, so the runner cannot be started`, { cause }),
|
|
{ code: SANDBOX_DISK_FULL_FAILURE_CODE },
|
|
);
|
|
}
|
|
|
|
/** Managed sandboxes only; never on a user's own host. `.paperclip-runtime` is outside every saved directory. */
|
|
export function remoteRunnerDiskReservePath(target: { transport: string; remoteCwd: string }): string | undefined {
|
|
return target.transport === "sandbox"
|
|
? `${target.remoteCwd.replace(/\/+$/, "")}/.paperclip-runtime/paperclip-runner/disk-reserve`
|
|
: undefined;
|
|
}
|