mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 23:08:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a Paperclip-managed sandbox run under the Paperclip runner (`runnerd`), which keeps durable state on the sandbox disk > - An agent can fill that disk with its own work. In one case the agent installed a Rust toolchain and a second checkout with dependencies on a 10 GiB sandbox > - `runnerd` then could not write its state and exited. Each relaunch tried to upload its launch files to the full disk and failed, 83 times in 7 minutes > - The run ended as a generic provider failure that asked a person to inspect it, and Retry repeated the same failure > - This pull request keeps a small disk reserve that belongs to Paperclip. Recovery releases it to relaunch the runner, and stops at once with a named code when nothing is left to free > - The benefit is that the agent's turn can continue and see the full disk in its own tool output, instead of the run dying with it ## Linked Issues or Issue Description **What happened?** An agent in a managed Daytona sandbox filled the 10 GiB disk. `runnerd` exited with `failed to persist bounded diagnostic: No space left on device (os error 28)`. Runner process recovery then retried for the full 420000 ms reconnect grace. Every attempt failed with `.paperclip-upload-<id>.tar.gz: create: ... no space left on device`. The run failed as `native_provider_terminal_failed`, and the recovery asked a board user to verify that the provider stopped. **Steps to reproduce** 1. Run an agent in a managed sandbox and have it fill the disk (for example a Rust toolchain plus a full dependency install). 2. `runnerd` exits when it next writes its state. 3. Watch recovery retry the relaunch upload until the grace expires. **Expected behavior** Paperclip relaunches its runner on a disk the agent filled, so the turn continues and the agent sees the full disk in its tools. When the runner truly cannot start, recovery says so at once with a named cause instead of retrying for 7 minutes. **Environment** Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB disk), `paperclip_runner` native runtime. ## What Changed - New `server/src/services/native-runtime/remote-runner-disk-reserve.ts` owns the reserve: hold/release scripts, the full-disk classifier, and `createRemoteRunnerDiskReserve`. A release waits for an in-flight hold, and no hold starts after a release. - Both remote runner launchers (agent runs and Dot) get a reserve path in managed sandboxes only: `<workspace>/.paperclip-runtime/paperclip-runner/disk-reserve`. Workspace saves and Git already exclude `.paperclip-runtime`. SSH hosts get no reserve. - After the first launch's identity is confirmed, the launcher holds 256 MiB in the background, only while 2 GiB more is free. It is off the startup path. - A relaunch releases the reserve before it stages. A launch that hits a full disk while staging releases the reserve and stages once more. - Release tries twice and reports `released`, `not_held` or `failed`. Only `not_held` ends the launch with `sandbox_disk_full`. A release that cannot run rethrows the original error, so recovery still retries. - Command-based staging (no `syncIn`) keeps a full disk recognizable with an `ENOSPC` code instead of a bare `runner_remote_*_staging_failed`. - `SANDBOX_DISK_FULL_FAILURE_CODE` and `isSandboxDiskFullFailure` live in the runner package. Runner process recovery checks the error code and stops at once instead of retrying for the whole grace. Other staging failures still retry. ## Verification - Red before the fix, green after: - `server/src/services/native-runtime/remote-runner-disk-reserve.test.ts`: release-and-restage, release before relaunch, `sandbox_disk_full` when release frees too little, no retry for a failure that is not a full disk, hold/release scripts in a real shell, no hold below the free-space floor. Before the fix, the launch rejected with the exact upload error from the incident. - `packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts` "fails at once when a replacement cannot be staged because the sandbox disk is full": before the fix it was still retrying after 4 s; after, it fails with `sandbox_disk_full` after one replacement. - Kept: "retries a replacement whose staging failed before any process launched" still passes, so a raw transient staging failure still retries. - Updated: "confirms no replacement process exists when staging fails before the launch RPC" now expects `sandbox_disk_full` with the upload error as cause. Exit is still confirmed and no launch RPC runs. - Real sandbox from the same fleet snapshot (Daytona, 10 GiB): hold created a 256 MiB reserve. The disk was filled to 100%. A 5 MB write then failed. Release returned `true`, and a 50 MB write then succeeded. A second release returned `false`. Hold on the now 98% full disk did not create a reserve. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. `pnpm verify` result is in a comment below. - Review round: tests for a lost release RPC, hold/release ordering, command-based staging on a full disk, the hold after a dispatched first launch (not on relaunch), and no reserve on SSH. Each launcher-level test failed with its fix removed. - Focused suites on the review revision: transport 335/335, launcher 130/130 (filtered), launch assets 35/35, reserve 14/14. Server and runner typecheck pass. - Not run: a full product run in which an agent fills the disk and the relaunched `runnerd` resumes the turn end to end. ## Risks - Each managed sandbox holds 256 MiB more disk while it has more than 2 GiB free. That is 2.5% of a 10 GiB sandbox. - A relaunch for any reason releases the reserve for the rest of that run. The next run's launch takes it again. - If the agent keeps writing after the relaunch, the disk can fill again. The second time, nothing is left to release, and the run fails at once with `sandbox_disk_full` instead of after 7 minutes. - The run's error code is still `native_provider_terminal_failed`. The message now names `sandbox_disk_full`. A dedicated failure code and recovery text are a follow-up. - `runnerd` loses its own error message when it cannot persist its diagnostic. This PR does not change the Rust binary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>