Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 d1ca0cc218 fix(native): stop retrying a run that filled its sandbox disk (#22239)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs execute in a managed sandbox with a fixed disk
size
> - An agent can fill that disk. The runner then exits with "No space
left on device". The disk reserve restarts the runner, but the turn in
progress can still be lost, and the run fails
> - The run failure did not carry the full-disk fact. The run was
classified as a generic provider or runner failure. Automatic recovery
then started the same work on a fresh sandbox of the same size, which
filled the disk again. After three attempts the task stopped in
"Recovery needed" with no useful explanation
> - This pull request names the cause `sandbox_disk_full` when the
runner reported a full disk during the run, and stops automatic retries
for that cause
> - The benefit is that Paperclip no longer spends sandboxes and time on
retries that cannot succeed, and the task tells the user what to do in
plain language

## Linked Issues or Issue Description

**What happened?**
A native run in a managed sandbox filled the 10 GiB disk. The runner
exited with "No space left on device (os error 28)". The runner was
restored, but the in-flight turn was lost and the run failed as
`native_provider_terminal_failed`. Automatic recovery repeated the same
work on a fresh sandbox of the same size three times. Each attempt
filled the disk again. The task ended in "Recovery needed" with an
unrelated cause.

**Expected behavior**
When a run ends because its sandbox disk is full, the run failure names
that cause. Automatic recovery does not repeat the same work on a
sandbox of the same size. The task shows a plain-language explanation
and a working Retry.

**Steps to reproduce**
1. Run a native agent task in a managed sandbox whose work installs more
than the free disk space.
2. Wait until the runner exits with "No space left on device".
3. See the run fail as a generic provider failure, and see automatic
retries start fresh sandboxes that fail the same way.

**Relevant logs or output**
`paperclip-runner` exited with "No space left on device (os error 28)".
The run then failed with `provider_turn_lost_on_restore`, classified as
`native_provider_terminal_failed`.

Refs #22068 (disk reserve that restarts the runner). Refs #22235
(continues a turn that a runner restore lost, once, in the same
sandbox). This PR covers the case where the run still ends.

## What Changed

- Server: each run's diagnostic lease now records when a runner
diagnostic reports a full disk (`ENOSPC`, `EDQUOT`, "No space left on
device", "Disk quota exceeded"). The runner's exit detail already
reaches the server as a diagnostic, so no runner change is needed. The
fact belongs to that run's diagnostic epoch only. A later run on the
same warm session, or a diagnostic after the epoch retires, does not see
it.
- Server: native failure classification returns `sandbox_disk_full` when
either (a) the run's lease saw a full-disk report, or (b) the error
chain carries the runner's existing `sandbox_disk_full` launch code (a
relaunch that cannot stage on the full disk). A fixed list of failures
keeps its own meaning: permission, approval, usage limit, model,
runtime, integrity, cleanup quarantine, ownership, saved-state and
admission failures.
- Server: `sandbox_disk_full` is a permanent native failure. The run
does not take another same-run attempt. Native safe replacement does not
pick it up, because it is not one of its failure codes. The task becomes
blocked with a board-owned recovery action and a plain-language next
action.
- Server: the bounded retry scheduler (`scheduleBoundedRetryForRun`)
refuses `sandbox_disk_full`. It does not schedule a `transient_failure`
retry on a fresh sandbox of the same size.
- UI: the task chat shows "Disk full" and a plain explanation instead of
the raw failure code.
- Retry stays available. `sandbox_disk_full` is not a reconciliation
cause, so the board's Retry starts a new run when the user chooses.

## Verification

- Red before the fix (captured on the unfixed code):
- `heartbeat-retry-scheduling.test.ts` "does not repeat a run that
filled its sandbox disk on a fresh sandbox of the same size": expected
`not_scheduled`, received `scheduled`.
- `native-sandbox-disk-full.test.ts` "names the full disk and stops
instead of retrying the same work": expected `sandbox_disk_full`,
received `native_provider_terminal_failed`.
- `native-sandbox-disk-full.test.ts` "names the full disk when the
runner could not be relaunched on it": expected `sandbox_disk_full`,
received `native_session_interrupted`.
- Green after the fix:
- `cd server && npx vitest run
src/services/native-runtime/native-sandbox-disk-full.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/services/native-runtime/remote-runner-disk-reserve.test.ts` (90
passed).
- `cd ui && npx vitest run src/components/TaskChatThread.test.tsx` (426
passed).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-diagnostics.test.ts`.
- Safety cases in the tests:
- A runner that exited for another reason still gets its bounded retry.
- A turn lost on restore without a full-disk report keeps
`native_provider_terminal_failed`.
- Integrity and approval failures keep their codes when a full disk was
reported.
- A full-disk report belongs only to its own run's diagnostic epoch; run
B on the same session does not see run A's report.
- After a full-disk report, other runner and provider loss codes
(session.open timeout, initialize timeout, protocol error, process
status failure) also become `sandbox_disk_full`.
- Worst-case automatic run starts per hour across a company: before,
each disk-full task started up to 3 automatic runs, each on a fresh
sandbox (3 × the number of affected tasks). After, a disk-full failure
starts 0 automatic runs. Only a person's Retry starts a new run. Real
transient failures keep their existing bounded lanes, so their worst
case is unchanged.
- Not run locally: the full server suite, the full typecheck and build.
Required CI covers them.
- Not verified live on staging yet. A live check needs a run that fills
its sandbox disk after this change deploys.

## Risks

- A run whose runner reported a full disk, and which later fails with a
runner or provider loss for an unrelated reason, is now labeled
`sandbox_disk_full` and is not retried automatically. The board's Retry
still works. This is the safer error: a full disk is the most likely
cause, and an automatic retry would repeat the same disk use.
- The full-disk report lives on the run's in-memory diagnostic lease. A
server restart between the report and the failure loses it, and the run
falls back to the previous classification. That is the previous
behavior, not a new failure.
- Low risk otherwise. No schema or API change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:19:12 +00:00
DottaandPaperclip 396b7fc62a fix(runner): scope session diagnostics to the active run recorder (#18944)
## Thinking Path

> - Paperclip manages agents and their work.
> - The native executor can retain a transport between runs.
> - Each run recorder has a shorter lifetime than that transport.
> - The transport kept its first recorder after warm reuse and
completion.
> - A later close could write to the wrong run or a sealed object log.
> - This change binds diagnostics to the active run epoch and retires
that binding at execution exit.
> - The existing recorder still owns accepted-byte draining and honest
incomplete finalization.

## Linked Issues or Issue Description

**What happened?**

A warm native transport retained the first run's diagnostic callback.
Reusing the session for another run did not change that callback. A
later idle close could append to a retired writer and emit a persistence
warning.

**Expected behavior**

Active diagnostics belong to the current company, session and run. A
retired transport callback must not use a completed recorder. Diagnostic
metadata must not delay essential execution cleanup.

**Steps to reproduce**

Run the native warm-session regression with two sequential run owners.
Close the retained transport after both runs finish. The original code
calls the first recorder. The real completion regression also observes
the resulting closed-writer warning after normal lease cleanup.

This is a local source defect with behavioral red/green proof. Repeated
hosted generic persistence warnings motivated the investigation, but
their original exceptions were not retained. This PR does not claim to
identify every such warning or repair an export failure.

Related #18379 queues sanitized performance spans in heartbeat. It does
not own transport diagnostic callback lifetime.

## What Changed

- Bind transport diagnostics to an exact active company/session/run
epoch, including warm reuse.
- Activate a retained channel in the existing session-admission hook
after identity validation and before attachment starts, including an
attachment failure before `onSession`.
- Forward active diagnostics synchronously. Detach their recorder at
execution exit. Keep the original asynchronous metadata semantics.
- Keep idle transport evidence inspectable. Log only a fixed category
and trusted company/run identifiers to the process logger.
- Preserve recorder-owned byte draining, redaction, source coverage and
genuine persistence-failure warnings.
- Cover actual native executor, completion and lease-close boundaries,
including stalled local appends and exact object-log source/hash
preservation.
- Place the retained-workspace test fixture under the existing
published-path contract after merging newer recovery tests from master.
Production validation and native/catalog assertions remain unchanged.

## Verification

- Current head: `c500e4933a85a4c453f66c872fa335108c661b15`. Normal
master merges preserve upstream behavior. The one textual conflict was
in the executor imports.
- The first GitHub run tested merge tree
`a4cb59448a3b5cc2047ac41c691720e6acec6e8d`, which included newer
recovery tests from master. Five cases failed with
`remote_workspace_published_path_invalid`. The same five failures
reproduced on an identical local tree. The fixture now puts retained
files under `.paperclip-workspaces/<workspaceId>` and keeps the target,
lease and authority bound to that path. Production validation and the
native-success/catalog-denial assertions are unchanged.
- On unchanged `c500e493`, the full recovery file passed **244/244**
tests. Existing cold-restore and occupied-root controls passed **4/4**.
Server typecheck, server production build, module boundaries and
whitespace checks passed. An intermediate missing-import setup failure
was corrected and is separate from the five-case behavioral red.
- Before that fixture correction, local `f2fdaddb` passed **1,221
tests** across the five diagnostic, executor, output, completion and
object-store files. Recursive typecheck and full repository build also
passed. Those results are tied to that earlier revision; [fresh
current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41842043248)
must qualify the final merge.
- Behavioral regressions cover the original closed-writer warning, warm
A-to-B ownership, and an actual Runner attachment exception before
`onSession`. The admission hook gives the attachment diagnostic to B and
leaves A untouched. The attachment fixture uses synthetic event
persistence and outer failure projection; it does not certify a real
database terminal projection.
- The tests preserve synchronous diagnostic admission and asynchronous
metadata recording. A stalled append reaches the existing bounded
finalization result with unknown size/hash. The normal object-log
control preserves canonical bytes, SHA256 and complete source coverage
after lease cleanup.
- An initial local run had **37 PostgreSQL startup failures** because
the Darwin package lacked aliases to its included versioned libraries.
Only isolated `node_modules` aliases were restored. No source or
lockfile changed for this setup issue. A passing intermediate run
crossed a source merge and is excluded from immutable-head proof.
- An earlier combined test run passed 1,216 assertions but exited with a
completion `afterAll` timeout. Its cause remains unproved. Later
isolated 60/60 passes used the original timeout and do not erase that
failure. An earlier repository-wide `pnpm test:run` was interrupted
after its source became obsolete; it is not full-suite or current-head
proof.

Greptile reviewed the exact `c500e493` head at **5/5** with no open
findings. Fresh CI must pass before merge. The native review is queued
behind its separate saved-file recovery; no current-head native
assessment is claimed.

The provider execution loop is a fixture. These tests do not start a
model or prove hosted recovery. Missing accounting receipts remain
pending, not zero cost. Checkout-release invocation is observed; this is
not a separate task-arbiter journey. The original native review
published a valid finding before a separate workspace-export failure.
The finding is fixed; this PR does not claim to fix that export failure
or every hosted persistence warning.

## Risks

Idle process logs intentionally omit raw provider text. The existing
bounded transport evidence retains its own diagnostics. Accepted fs
appends that do not settle retain the existing unknown-size/hash warning
and cannot be certified complete. No new timeout, storage policy, schema
or provider execution is added.

## Model Used

OpenAI `gpt-6.1-sol` through Codex assisted with diagnosis,
implementation, tests and review. The exact context-window size is not
exposed. Tool use and code execution were enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run local tests and recorded passing checks, pending checks
and unresolved failures above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will require all Paperclip CI gates to be green before merge
- [x] I will require Greptile 5/5 with no unresolved findings before
merge
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 08:04:52 -06:00