Files
PaperClipAI/server/src/services/native-runtime
DottaandClaude Opus 5.5 d1ca0cc218 fix(native): stop retrying a run that filled its sandbox disk (#22239)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs execute in a managed sandbox with a fixed disk
size
> - An agent can fill that disk. The runner then exits with "No space
left on device". The disk reserve restarts the runner, but the turn in
progress can still be lost, and the run fails
> - The run failure did not carry the full-disk fact. The run was
classified as a generic provider or runner failure. Automatic recovery
then started the same work on a fresh sandbox of the same size, which
filled the disk again. After three attempts the task stopped in
"Recovery needed" with no useful explanation
> - This pull request names the cause `sandbox_disk_full` when the
runner reported a full disk during the run, and stops automatic retries
for that cause
> - The benefit is that Paperclip no longer spends sandboxes and time on
retries that cannot succeed, and the task tells the user what to do in
plain language

## Linked Issues or Issue Description

**What happened?**
A native run in a managed sandbox filled the 10 GiB disk. The runner
exited with "No space left on device (os error 28)". The runner was
restored, but the in-flight turn was lost and the run failed as
`native_provider_terminal_failed`. Automatic recovery repeated the same
work on a fresh sandbox of the same size three times. Each attempt
filled the disk again. The task ended in "Recovery needed" with an
unrelated cause.

**Expected behavior**
When a run ends because its sandbox disk is full, the run failure names
that cause. Automatic recovery does not repeat the same work on a
sandbox of the same size. The task shows a plain-language explanation
and a working Retry.

**Steps to reproduce**
1. Run a native agent task in a managed sandbox whose work installs more
than the free disk space.
2. Wait until the runner exits with "No space left on device".
3. See the run fail as a generic provider failure, and see automatic
retries start fresh sandboxes that fail the same way.

**Relevant logs or output**
`paperclip-runner` exited with "No space left on device (os error 28)".
The run then failed with `provider_turn_lost_on_restore`, classified as
`native_provider_terminal_failed`.

Refs #22068 (disk reserve that restarts the runner). Refs #22235
(continues a turn that a runner restore lost, once, in the same
sandbox). This PR covers the case where the run still ends.

## What Changed

- Server: each run's diagnostic lease now records when a runner
diagnostic reports a full disk (`ENOSPC`, `EDQUOT`, "No space left on
device", "Disk quota exceeded"). The runner's exit detail already
reaches the server as a diagnostic, so no runner change is needed. The
fact belongs to that run's diagnostic epoch only. A later run on the
same warm session, or a diagnostic after the epoch retires, does not see
it.
- Server: native failure classification returns `sandbox_disk_full` when
either (a) the run's lease saw a full-disk report, or (b) the error
chain carries the runner's existing `sandbox_disk_full` launch code (a
relaunch that cannot stage on the full disk). A fixed list of failures
keeps its own meaning: permission, approval, usage limit, model,
runtime, integrity, cleanup quarantine, ownership, saved-state and
admission failures.
- Server: `sandbox_disk_full` is a permanent native failure. The run
does not take another same-run attempt. Native safe replacement does not
pick it up, because it is not one of its failure codes. The task becomes
blocked with a board-owned recovery action and a plain-language next
action.
- Server: the bounded retry scheduler (`scheduleBoundedRetryForRun`)
refuses `sandbox_disk_full`. It does not schedule a `transient_failure`
retry on a fresh sandbox of the same size.
- UI: the task chat shows "Disk full" and a plain explanation instead of
the raw failure code.
- Retry stays available. `sandbox_disk_full` is not a reconciliation
cause, so the board's Retry starts a new run when the user chooses.

## Verification

- Red before the fix (captured on the unfixed code):
- `heartbeat-retry-scheduling.test.ts` "does not repeat a run that
filled its sandbox disk on a fresh sandbox of the same size": expected
`not_scheduled`, received `scheduled`.
- `native-sandbox-disk-full.test.ts` "names the full disk and stops
instead of retrying the same work": expected `sandbox_disk_full`,
received `native_provider_terminal_failed`.
- `native-sandbox-disk-full.test.ts` "names the full disk when the
runner could not be relaunched on it": expected `sandbox_disk_full`,
received `native_session_interrupted`.
- Green after the fix:
- `cd server && npx vitest run
src/services/native-runtime/native-sandbox-disk-full.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/services/native-runtime/remote-runner-disk-reserve.test.ts` (90
passed).
- `cd ui && npx vitest run src/components/TaskChatThread.test.tsx` (426
passed).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-diagnostics.test.ts`.
- Safety cases in the tests:
- A runner that exited for another reason still gets its bounded retry.
- A turn lost on restore without a full-disk report keeps
`native_provider_terminal_failed`.
- Integrity and approval failures keep their codes when a full disk was
reported.
- A full-disk report belongs only to its own run's diagnostic epoch; run
B on the same session does not see run A's report.
- After a full-disk report, other runner and provider loss codes
(session.open timeout, initialize timeout, protocol error, process
status failure) also become `sandbox_disk_full`.
- Worst-case automatic run starts per hour across a company: before,
each disk-full task started up to 3 automatic runs, each on a fresh
sandbox (3 × the number of affected tasks). After, a disk-full failure
starts 0 automatic runs. Only a person's Retry starts a new run. Real
transient failures keep their existing bounded lanes, so their worst
case is unchanged.
- Not run locally: the full server suite, the full typecheck and build.
Required CI covers them.
- Not verified live on staging yet. A live check needs a run that fills
its sandbox disk after this change deploys.

## Risks

- A run whose runner reported a full disk, and which later fails with a
runner or provider loss for an unrelated reason, is now labeled
`sandbox_disk_full` and is not retried automatically. The board's Retry
still works. This is the safer error: a full disk is the most likely
cause, and an automatic retry would repeat the same disk use.
- The full-disk report lives on the run's in-memory diagnostic lease. A
server restart between the report and the failure loses it, and the run
falls back to the previous classification. That is the previous
behavior, not a new failure.
- Low risk otherwise. No schema or API change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:19:12 +00:00
..
…