mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 20:29:18 +01:00
28ed2f1db0eec8750be6ecb20c74694cb584d8e0
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d1ca0cc218 |
fix(native): stop retrying a run that filled its sandbox disk (#22239)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs execute in a managed sandbox with a fixed disk size > - An agent can fill that disk. The runner then exits with "No space left on device". The disk reserve restarts the runner, but the turn in progress can still be lost, and the run fails > - The run failure did not carry the full-disk fact. The run was classified as a generic provider or runner failure. Automatic recovery then started the same work on a fresh sandbox of the same size, which filled the disk again. After three attempts the task stopped in "Recovery needed" with no useful explanation > - This pull request names the cause `sandbox_disk_full` when the runner reported a full disk during the run, and stops automatic retries for that cause > - The benefit is that Paperclip no longer spends sandboxes and time on retries that cannot succeed, and the task tells the user what to do in plain language ## Linked Issues or Issue Description **What happened?** A native run in a managed sandbox filled the 10 GiB disk. The runner exited with "No space left on device (os error 28)". The runner was restored, but the in-flight turn was lost and the run failed as `native_provider_terminal_failed`. Automatic recovery repeated the same work on a fresh sandbox of the same size three times. Each attempt filled the disk again. The task ended in "Recovery needed" with an unrelated cause. **Expected behavior** When a run ends because its sandbox disk is full, the run failure names that cause. Automatic recovery does not repeat the same work on a sandbox of the same size. The task shows a plain-language explanation and a working Retry. **Steps to reproduce** 1. Run a native agent task in a managed sandbox whose work installs more than the free disk space. 2. Wait until the runner exits with "No space left on device". 3. See the run fail as a generic provider failure, and see automatic retries start fresh sandboxes that fail the same way. **Relevant logs or output** `paperclip-runner` exited with "No space left on device (os error 28)". The run then failed with `provider_turn_lost_on_restore`, classified as `native_provider_terminal_failed`. Refs #22068 (disk reserve that restarts the runner). Refs #22235 (continues a turn that a runner restore lost, once, in the same sandbox). This PR covers the case where the run still ends. ## What Changed - Server: each run's diagnostic lease now records when a runner diagnostic reports a full disk (`ENOSPC`, `EDQUOT`, "No space left on device", "Disk quota exceeded"). The runner's exit detail already reaches the server as a diagnostic, so no runner change is needed. The fact belongs to that run's diagnostic epoch only. A later run on the same warm session, or a diagnostic after the epoch retires, does not see it. - Server: native failure classification returns `sandbox_disk_full` when either (a) the run's lease saw a full-disk report, or (b) the error chain carries the runner's existing `sandbox_disk_full` launch code (a relaunch that cannot stage on the full disk). A fixed list of failures keeps its own meaning: permission, approval, usage limit, model, runtime, integrity, cleanup quarantine, ownership, saved-state and admission failures. - Server: `sandbox_disk_full` is a permanent native failure. The run does not take another same-run attempt. Native safe replacement does not pick it up, because it is not one of its failure codes. The task becomes blocked with a board-owned recovery action and a plain-language next action. - Server: the bounded retry scheduler (`scheduleBoundedRetryForRun`) refuses `sandbox_disk_full`. It does not schedule a `transient_failure` retry on a fresh sandbox of the same size. - UI: the task chat shows "Disk full" and a plain explanation instead of the raw failure code. - Retry stays available. `sandbox_disk_full` is not a reconciliation cause, so the board's Retry starts a new run when the user chooses. ## Verification - Red before the fix (captured on the unfixed code): - `heartbeat-retry-scheduling.test.ts` "does not repeat a run that filled its sandbox disk on a fresh sandbox of the same size": expected `not_scheduled`, received `scheduled`. - `native-sandbox-disk-full.test.ts` "names the full disk and stops instead of retrying the same work": expected `sandbox_disk_full`, received `native_provider_terminal_failed`. - `native-sandbox-disk-full.test.ts` "names the full disk when the runner could not be relaunched on it": expected `sandbox_disk_full`, received `native_session_interrupted`. - Green after the fix: - `cd server && npx vitest run src/services/native-runtime/native-sandbox-disk-full.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/native-runtime/remote-runner-disk-reserve.test.ts` (90 passed). - `cd ui && npx vitest run src/components/TaskChatThread.test.tsx` (426 passed). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-session-diagnostics.test.ts`. - Safety cases in the tests: - A runner that exited for another reason still gets its bounded retry. - A turn lost on restore without a full-disk report keeps `native_provider_terminal_failed`. - Integrity and approval failures keep their codes when a full disk was reported. - A full-disk report belongs only to its own run's diagnostic epoch; run B on the same session does not see run A's report. - After a full-disk report, other runner and provider loss codes (session.open timeout, initialize timeout, protocol error, process status failure) also become `sandbox_disk_full`. - Worst-case automatic run starts per hour across a company: before, each disk-full task started up to 3 automatic runs, each on a fresh sandbox (3 × the number of affected tasks). After, a disk-full failure starts 0 automatic runs. Only a person's Retry starts a new run. Real transient failures keep their existing bounded lanes, so their worst case is unchanged. - Not run locally: the full server suite, the full typecheck and build. Required CI covers them. - Not verified live on staging yet. A live check needs a run that fills its sandbox disk after this change deploys. ## Risks - A run whose runner reported a full disk, and which later fails with a runner or provider loss for an unrelated reason, is now labeled `sandbox_disk_full` and is not retried automatically. The board's Retry still works. This is the safer error: a full disk is the most likely cause, and an automatic retry would repeat the same disk use. - The full-disk report lives on the run's in-memory diagnostic lease. A server restart between the report and the failure loses it, and the run falls back to the previous classification. That is the previous behavior, not a new failure. - Low risk otherwise. No schema or API change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
396b7fc62a |
fix(runner): scope session diagnostics to the active run recorder (#18944)
## Thinking Path > - Paperclip manages agents and their work. > - The native executor can retain a transport between runs. > - Each run recorder has a shorter lifetime than that transport. > - The transport kept its first recorder after warm reuse and completion. > - A later close could write to the wrong run or a sealed object log. > - This change binds diagnostics to the active run epoch and retires that binding at execution exit. > - The existing recorder still owns accepted-byte draining and honest incomplete finalization. ## Linked Issues or Issue Description **What happened?** A warm native transport retained the first run's diagnostic callback. Reusing the session for another run did not change that callback. A later idle close could append to a retired writer and emit a persistence warning. **Expected behavior** Active diagnostics belong to the current company, session and run. A retired transport callback must not use a completed recorder. Diagnostic metadata must not delay essential execution cleanup. **Steps to reproduce** Run the native warm-session regression with two sequential run owners. Close the retained transport after both runs finish. The original code calls the first recorder. The real completion regression also observes the resulting closed-writer warning after normal lease cleanup. This is a local source defect with behavioral red/green proof. Repeated hosted generic persistence warnings motivated the investigation, but their original exceptions were not retained. This PR does not claim to identify every such warning or repair an export failure. Related #18379 queues sanitized performance spans in heartbeat. It does not own transport diagnostic callback lifetime. ## What Changed - Bind transport diagnostics to an exact active company/session/run epoch, including warm reuse. - Activate a retained channel in the existing session-admission hook after identity validation and before attachment starts, including an attachment failure before `onSession`. - Forward active diagnostics synchronously. Detach their recorder at execution exit. Keep the original asynchronous metadata semantics. - Keep idle transport evidence inspectable. Log only a fixed category and trusted company/run identifiers to the process logger. - Preserve recorder-owned byte draining, redaction, source coverage and genuine persistence-failure warnings. - Cover actual native executor, completion and lease-close boundaries, including stalled local appends and exact object-log source/hash preservation. - Place the retained-workspace test fixture under the existing published-path contract after merging newer recovery tests from master. Production validation and native/catalog assertions remain unchanged. ## Verification - Current head: `c500e4933a85a4c453f66c872fa335108c661b15`. Normal master merges preserve upstream behavior. The one textual conflict was in the executor imports. - The first GitHub run tested merge tree `a4cb59448a3b5cc2047ac41c691720e6acec6e8d`, which included newer recovery tests from master. Five cases failed with `remote_workspace_published_path_invalid`. The same five failures reproduced on an identical local tree. The fixture now puts retained files under `.paperclip-workspaces/<workspaceId>` and keeps the target, lease and authority bound to that path. Production validation and the native-success/catalog-denial assertions are unchanged. - On unchanged `c500e493`, the full recovery file passed **244/244** tests. Existing cold-restore and occupied-root controls passed **4/4**. Server typecheck, server production build, module boundaries and whitespace checks passed. An intermediate missing-import setup failure was corrected and is separate from the five-case behavioral red. - Before that fixture correction, local `f2fdaddb` passed **1,221 tests** across the five diagnostic, executor, output, completion and object-store files. Recursive typecheck and full repository build also passed. Those results are tied to that earlier revision; [fresh current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41842043248) must qualify the final merge. - Behavioral regressions cover the original closed-writer warning, warm A-to-B ownership, and an actual Runner attachment exception before `onSession`. The admission hook gives the attachment diagnostic to B and leaves A untouched. The attachment fixture uses synthetic event persistence and outer failure projection; it does not certify a real database terminal projection. - The tests preserve synchronous diagnostic admission and asynchronous metadata recording. A stalled append reaches the existing bounded finalization result with unknown size/hash. The normal object-log control preserves canonical bytes, SHA256 and complete source coverage after lease cleanup. - An initial local run had **37 PostgreSQL startup failures** because the Darwin package lacked aliases to its included versioned libraries. Only isolated `node_modules` aliases were restored. No source or lockfile changed for this setup issue. A passing intermediate run crossed a source merge and is excluded from immutable-head proof. - An earlier combined test run passed 1,216 assertions but exited with a completion `afterAll` timeout. Its cause remains unproved. Later isolated 60/60 passes used the original timeout and do not erase that failure. An earlier repository-wide `pnpm test:run` was interrupted after its source became obsolete; it is not full-suite or current-head proof. Greptile reviewed the exact `c500e493` head at **5/5** with no open findings. Fresh CI must pass before merge. The native review is queued behind its separate saved-file recovery; no current-head native assessment is claimed. The provider execution loop is a fixture. These tests do not start a model or prove hosted recovery. Missing accounting receipts remain pending, not zero cost. Checkout-release invocation is observed; this is not a separate task-arbiter journey. The original native review published a valid finding before a separate workspace-export failure. The finding is fixed; this PR does not claim to fix that export failure or every hosted persistence warning. ## Risks Idle process logs intentionally omit raw provider text. The existing bounded transport evidence retains its own diagnostics. Accepted fs appends that do not settle retain the existing unknown-size/hash warning and cannot be certified complete. No new timeout, storage policy, schema or provider execution is added. ## Model Used OpenAI `gpt-6.1-sol` through Codex assisted with diagnosis, implementation, tests and review. The exact context-window size is not exposed. Tool use and code execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run local tests and recorded passing checks, pending checks and unresolved failures above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will require all Paperclip CI gates to be green before merge - [x] I will require Greptile 5/5 with no unresolved findings before merge - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |