Files
PaperClipAI/server/src/services/native-runtime/remote-runner-disk-reserve.ts
T
DottaandClaude Opus 5.5 73dc33c620 fix(runner): keep a disk reserve so a full sandbox disk does not strand the runner (#22068)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a Paperclip-managed sandbox run under the Paperclip runner
(`runnerd`), which keeps durable state on the sandbox disk
> - An agent can fill that disk with its own work. In one case the agent
installed a Rust toolchain and a second checkout with dependencies on a
10 GiB sandbox
> - `runnerd` then could not write its state and exited. Each relaunch
tried to upload its launch files to the full disk and failed, 83 times
in 7 minutes
> - The run ended as a generic provider failure that asked a person to
inspect it, and Retry repeated the same failure
> - This pull request keeps a small disk reserve that belongs to
Paperclip. Recovery releases it to relaunch the runner, and stops at
once with a named code when nothing is left to free
> - The benefit is that the agent's turn can continue and see the full
disk in its own tool output, instead of the run dying with it

## Linked Issues or Issue Description

**What happened?**
An agent in a managed Daytona sandbox filled the 10 GiB disk. `runnerd`
exited with `failed to persist bounded diagnostic: No space left on
device (os error 28)`. Runner process recovery then retried for the full
420000 ms reconnect grace. Every attempt failed with
`.paperclip-upload-<id>.tar.gz: create: ... no space left on device`.
The run failed as `native_provider_terminal_failed`, and the recovery
asked a board user to verify that the provider stopped.

**Steps to reproduce**
1. Run an agent in a managed sandbox and have it fill the disk (for
example a Rust toolchain plus a full dependency install).
2. `runnerd` exits when it next writes its state.
3. Watch recovery retry the relaunch upload until the grace expires.

**Expected behavior**
Paperclip relaunches its runner on a disk the agent filled, so the turn
continues and the agent sees the full disk in its tools. When the runner
truly cannot start, recovery says so at once with a named cause instead
of retrying for 7 minutes.

**Environment**
Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB
disk), `paperclip_runner` native runtime.

## What Changed

- New `server/src/services/native-runtime/remote-runner-disk-reserve.ts`
owns the reserve: hold/release scripts, the full-disk classifier, and
`createRemoteRunnerDiskReserve`. A release waits for an in-flight hold,
and no hold starts after a release.
- Both remote runner launchers (agent runs and Dot) get a reserve path
in managed sandboxes only:
`<workspace>/.paperclip-runtime/paperclip-runner/disk-reserve`.
Workspace saves and Git already exclude `.paperclip-runtime`. SSH hosts
get no reserve.
- After the first launch's identity is confirmed, the launcher holds 256
MiB in the background, only while 2 GiB more is free. It is off the
startup path.
- A relaunch releases the reserve before it stages. A launch that hits a
full disk while staging releases the reserve and stages once more.
- Release tries twice and reports `released`, `not_held` or `failed`.
Only `not_held` ends the launch with `sandbox_disk_full`. A release that
cannot run rethrows the original error, so recovery still retries.
- Command-based staging (no `syncIn`) keeps a full disk recognizable
with an `ENOSPC` code instead of a bare
`runner_remote_*_staging_failed`.
- `SANDBOX_DISK_FULL_FAILURE_CODE` and `isSandboxDiskFullFailure` live
in the runner package. Runner process recovery checks the error code and
stops at once instead of retrying for the whole grace. Other staging
failures still retry.

## Verification

- Red before the fix, green after:
-
`server/src/services/native-runtime/remote-runner-disk-reserve.test.ts`:
release-and-restage, release before relaunch, `sandbox_disk_full` when
release frees too little, no retry for a failure that is not a full
disk, hold/release scripts in a real shell, no hold below the free-space
floor. Before the fix, the launch rejected with the exact upload error
from the incident.
- `packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`
"fails at once when a replacement cannot be staged because the sandbox
disk is full": before the fix it was still retrying after 4 s; after, it
fails with `sandbox_disk_full` after one replacement.
- Kept: "retries a replacement whose staging failed before any process
launched" still passes, so a raw transient staging failure still
retries.
- Updated: "confirms no replacement process exists when staging fails
before the launch RPC" now expects `sandbox_disk_full` with the upload
error as cause. Exit is still confirmed and no launch RPC runs.
- Real sandbox from the same fleet snapshot (Daytona, 10 GiB): hold
created a 256 MiB reserve. The disk was filled to 100%. A 5 MB write
then failed. Release returned `true`, and a 50 MB write then succeeded.
A second release returned `false`. Hold on the now 98% full disk did not
create a reserve.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. `pnpm
verify` result is in a comment below.
- Review round: tests for a lost release RPC, hold/release ordering,
command-based staging on a full disk, the hold after a dispatched first
launch (not on relaunch), and no reserve on SSH. Each launcher-level
test failed with its fix removed.
- Focused suites on the review revision: transport 335/335, launcher
130/130 (filtered), launch assets 35/35, reserve 14/14. Server and
runner typecheck pass.
- Not run: a full product run in which an agent fills the disk and the
relaunched `runnerd` resumes the turn end to end.

## Risks

- Each managed sandbox holds 256 MiB more disk while it has more than 2
GiB free. That is 2.5% of a 10 GiB sandbox.
- A relaunch for any reason releases the reserve for the rest of that
run. The next run's launch takes it again.
- If the agent keeps writing after the relaunch, the disk can fill
again. The second time, nothing is left to release, and the run fails at
once with `sandbox_disk_full` instead of after 7 minutes.
- The run's error code is still `native_provider_terminal_failed`. The
message now names `sandbox_disk_full`. A dedicated failure code and
recovery text are a follow-up.
- `runnerd` loses its own error message when it cannot persist its
diagnostic. This PR does not change the Rust binary.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:45:20 -06:00

122 lines
5.2 KiB
TypeScript

import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
import { SANDBOX_DISK_FULL_FAILURE_CODE } from "../../vendor/paperclip-runner/index.js";
/**
* The agent can fill a managed sandbox's disk. runnerd then cannot write its
* state and exits, and its relaunch cannot upload its launch files. Paperclip
* holds this much disk in each managed sandbox and releases it to relaunch the
* runner, so the agent's turn continues and its tools report the full disk.
*/
export const REMOTE_RUNNER_DISK_RESERVE_BYTES = 256 * 1024 * 1024;
// The reserve is taken only while this much more is free, so it never takes
// space from an agent that is already close to the limit.
const MINIMUM_FREE_BYTES = 2 * 1024 * 1024 * 1024;
type Runner = Pick<CommandManagedRuntimeRunner, "execute">;
const HOLD_SCRIPT = [
'set -u; reserve=$1; bytes=$2; minimum_free_kib=$3; [ -f "$reserve" ] && exit 0',
'directory=$(dirname -- "$reserve"); mkdir -p -- "$directory" || exit 1',
`free_kib=$(df -Pk -- "$directory" | awk 'NR == 2 { print $4 }')`,
'[ -n "$free_kib" ] && [ "$free_kib" -ge "$minimum_free_kib" ] || exit 0',
'partial="$reserve.partial"; rm -f -- "$partial"',
'fallocate -l "$bytes" "$partial" 2>/dev/null || dd if=/dev/zero of="$partial" bs=65536 count=$((bytes / 65536)) 2>/dev/null || { rm -f -- "$partial"; exit 1; }',
'mv -f -- "$partial" "$reserve"',
].join("\n");
// rm needs no free space. The last line names the outcome.
const RELEASE_SCRIPT =
'rm -f -- "$1.partial" || exit 1; if [ -f "$1" ]; then rm -f -- "$1" && echo released; else echo not_held; fi';
export async function holdRemoteRunnerDiskReserve(
runner: Runner,
reservePath: string,
options: { bytes?: number; minimumFreeBytes?: number } = {},
): Promise<void> {
const bytes = options.bytes ?? REMOTE_RUNNER_DISK_RESERVE_BYTES;
const minimumFreeBytes = options.minimumFreeBytes ?? MINIMUM_FREE_BYTES;
await runner.execute({
command: "sh",
args: ["-c", HOLD_SCRIPT, "paperclip-runner-disk-reserve-hold", reservePath,
String(bytes), String(Math.ceil((minimumFreeBytes + bytes) / 1024))],
bypassSession: true,
timeoutMs: 30_000,
});
}
export type DiskReserveRelease = "released" | "not_held" | "failed";
/** Tries twice: one lost sandbox RPC must not read as "nothing left to free". */
export async function releaseRemoteRunnerDiskReserve(runner: Runner, reservePath: string): Promise<DiskReserveRelease> {
for (let attempt = 0; attempt < 2; attempt += 1) {
const result = await runner.execute({
command: "sh",
args: ["-c", RELEASE_SCRIPT, "paperclip-runner-disk-reserve-release", reservePath],
bypassSession: true,
timeoutMs: 10_000,
}).catch(() => null);
if (result?.exitCode !== 0 || result.timedOut) continue;
const outcome = result.stdout.trim().split("\n").at(-1);
if (outcome === "released" || outcome === "not_held") return outcome;
}
return "failed";
}
/**
* The reserve of one runner launcher. A release waits for an in-flight hold,
* and no hold starts after a release, so a relaunch never stages while the
* space it freed is taken back.
*/
export function createRemoteRunnerDiskReserve(runner: Runner, reservePath: string) {
let pendingHold: Promise<void> | null = null;
let releaseRequested = false;
return {
hold(): void {
if (releaseRequested || pendingHold) return;
pendingHold = holdRemoteRunnerDiskReserve(runner, reservePath).catch(() => undefined);
},
async release(): Promise<DiskReserveRelease> {
releaseRequested = true;
await pendingHold;
return releaseRemoteRunnerDiskReserve(runner, reservePath);
},
};
}
export type RemoteRunnerDiskReserve = ReturnType<typeof createRemoteRunnerDiskReserve>;
export function isSandboxDiskFullError(error: unknown): boolean {
if (!error || typeof error !== "object") return false;
const { code, message } = error as { code?: unknown; message?: unknown };
if (code === "ENOSPC" || code === "EDQUOT") return true;
return typeof message === "string" && isSandboxDiskFullText(message);
}
export function isSandboxDiskFullText(text: string): boolean {
return /no space left on device|disk quota exceeded|\bENOSPC\b|\bEDQUOT\b/i.test(text);
}
/** A staging command's failure, keeping a full disk recognizable without its output. */
export function remoteStagingFailure(
code: string,
result: { stdout?: string; stderr?: string },
): Error {
return isSandboxDiskFullText(`${result.stderr ?? ""}\n${result.stdout ?? ""}`)
? Object.assign(new Error(`${code}: no space left on device`), { code: "ENOSPC" })
: new Error(code);
}
export function sandboxDiskFullFailure(cause: unknown): Error {
return Object.assign(
new Error(`${SANDBOX_DISK_FULL_FAILURE_CODE}: the sandbox disk is full, so the runner cannot be started`, { cause }),
{ code: SANDBOX_DISK_FULL_FAILURE_CODE },
);
}
/** Managed sandboxes only; never on a user's own host. `.paperclip-runtime` is outside every saved directory. */
export function remoteRunnerDiskReservePath(target: { transport: string; remoteCwd: string }): string | undefined {
return target.transport === "sandbox"
? `${target.remoteCwd.replace(/\/+$/, "")}/.paperclip-runtime/paperclip-runner/disk-reserve`
: undefined;
}