Files
PaperClipAI/server
33ffaa5b9a fix(heartbeat): name a full sandbox disk that stops a workspace save during setup (#22466)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a run starts, Paperclip prepares the task's remote workspace
and saves it. The save builds an archive inside the sandbox
> - When the sandbox disk is full, that save fails with
`remote_workspace_sandbox_disk_full: the sandbox disk is full; ...`
> - #22383 taught the setup failure path to name a full sandbox disk
`sandbox_disk_full`. That code tells the user what to do, and automatic
retry and generic continuation recovery do not repeat it
> - The setup matcher accepts the runner's form, `sandbox_disk_full:
...`, at the start of the text or after `: `. In the save failure,
`sandbox_disk_full:` follows `remote_workspace_`, so it does not match
> - These runs therefore fail as the generic `setup_failed`. They share
one Sentry group with unrelated uncoded setup failures, and they are
started again on the same full disk
> - This pull request recognizes the exact save failure in the setup
failure path
> - The benefit is that these runs fail as `sandbox_disk_full`, get
their own Sentry group, show the existing plain-language action and are
not repeated automatically

## Linked Issues or Issue Description

Refs #22383 (names a full sandbox disk that fails a run during setup).
This PR covers the workspace save during setup, which that change does
not match.

**What happened?**
On staging, several runs failed while preparing their workspace with
`remote_workspace_sandbox_disk_full: the sandbox disk is full; free
space in the sandbox before saving the workspace`. Each run stayed in
preparation for about 19 minutes. The run had `errorCode: setup_failed`
and `resultJson.failureCause: "Error"`. The Sentry fingerprint
(`setup_failed`, adapter, `setup`, `exception:Error`) put these runs in
the same group as other uncoded setup failures. Each task was then
started again on the same sandbox and failed the same way.

**Expected behavior**
The run fails as `sandbox_disk_full`, like the other full-sandbox-disk
setup failures. The task shows the existing action (free disk space,
then press Retry). Bounded retry and generic continuation recovery do
not repeat it.

**Steps to reproduce**
1. Prepare a remote workspace in a sandbox whose disk is full.
2. Let the preparation save fail with
`REMOTE_WORKSPACE_SANDBOX_DISK_FULL`, either directly or after the
read-only fallback also fails (the fallback rethrows the same message
with the fallback failure as its cause).
3. See the run fail as `setup_failed`.

## What Changed

- `remote-workspace-transport.ts`: add
`isRemoteWorkspaceSandboxDiskFullError`. It is true when the error, or
one of its first causes, has the exact server-built
`REMOTE_WORKSPACE_SANDBOX_DISK_FULL` message. It does not match other
text.
- `run-completion.ts`: the setup failure ladder uses `sandbox_disk_full`
for that error too. The rung's position does not change, so named setup
codes (restore failure, timeout, storage and others) keep their
precedence.
- Tests in `run-completion.test.ts` with a real database: the direct
save failure, the save failure that keeps the read-only fallback failure
as its cause, and another failed save
(`remote_workspace_archive_changed`) that stays `setup_failed`. The
existing host-disk test still passes.

The message, the saved workspace preservation path and the
workspace-recovery handling are unchanged. Retry and continuation
behavior for `sandbox_disk_full` already exists
(`scheduleBoundedRetryForRun` returns `not_scheduled`; recovery lists
the code as non-retryable).

## Verification

- Red: on unmodified master, both new disk-full cases fail with
`errorCode: "setup_failed"`, the same code as the staging runs.
- Green: with the change, `pnpm vitest run
src/services/heartbeat/run-completion.test.ts
src/services/native-runtime/native-sandbox-disk-full.test.ts
src/services/remote-workspace-transport.test.ts`: 352 of 355 pass. The 3
failures are in `remote-workspace-transport.test.ts` (restore lock
`directory_shared`, ambient checkout filters, trusted SSH rewrites).
They fail the same way on unmodified master in my sandbox, so they come
from its Git setup.
- Retry and recovery behavior for the code: `pnpm vitest run
src/__tests__/failed-continuation-continues-again.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/services/workspace-recovery-disk-full.test.ts
src/services/recovery/provider-failure-classification.test.ts`: 4 files,
141 tests pass.
- The full server typecheck did not run locally because the sandbox has
too little memory and disk. CI runs it.

## Risks

- Low. The change only renames a setup failure that already cannot be
repaired by starting again on the same sandbox. It reuses an existing
code with existing retry, recovery and UI handling.
- Runs that hit this failure are no longer started again automatically.
A person frees disk space and presses Retry, as for the other
`sandbox_disk_full` failures.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 20:32:44 -08:00
..