From d1ca0cc2189e5c237138703cf1bdb2f400e7edd4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 18 Nov 2026 17:19:12 -0600 Subject: [PATCH] fix(native): stop retrying a run that filled its sandbox disk (#22239) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs execute in a managed sandbox with a fixed disk size > - An agent can fill that disk. The runner then exits with "No space left on device". The disk reserve restarts the runner, but the turn in progress can still be lost, and the run fails > - The run failure did not carry the full-disk fact. The run was classified as a generic provider or runner failure. Automatic recovery then started the same work on a fresh sandbox of the same size, which filled the disk again. After three attempts the task stopped in "Recovery needed" with no useful explanation > - This pull request names the cause `sandbox_disk_full` when the runner reported a full disk during the run, and stops automatic retries for that cause > - The benefit is that Paperclip no longer spends sandboxes and time on retries that cannot succeed, and the task tells the user what to do in plain language ## Linked Issues or Issue Description **What happened?** A native run in a managed sandbox filled the 10 GiB disk. The runner exited with "No space left on device (os error 28)". The runner was restored, but the in-flight turn was lost and the run failed as `native_provider_terminal_failed`. Automatic recovery repeated the same work on a fresh sandbox of the same size three times. Each attempt filled the disk again. The task ended in "Recovery needed" with an unrelated cause. **Expected behavior** When a run ends because its sandbox disk is full, the run failure names that cause. Automatic recovery does not repeat the same work on a sandbox of the same size. The task shows a plain-language explanation and a working Retry. **Steps to reproduce** 1. Run a native agent task in a managed sandbox whose work installs more than the free disk space. 2. Wait until the runner exits with "No space left on device". 3. See the run fail as a generic provider failure, and see automatic retries start fresh sandboxes that fail the same way. **Relevant logs or output** `paperclip-runner` exited with "No space left on device (os error 28)". The run then failed with `provider_turn_lost_on_restore`, classified as `native_provider_terminal_failed`. Refs #22068 (disk reserve that restarts the runner). Refs #22235 (continues a turn that a runner restore lost, once, in the same sandbox). This PR covers the case where the run still ends. ## What Changed - Server: each run's diagnostic lease now records when a runner diagnostic reports a full disk (`ENOSPC`, `EDQUOT`, "No space left on device", "Disk quota exceeded"). The runner's exit detail already reaches the server as a diagnostic, so no runner change is needed. The fact belongs to that run's diagnostic epoch only. A later run on the same warm session, or a diagnostic after the epoch retires, does not see it. - Server: native failure classification returns `sandbox_disk_full` when either (a) the run's lease saw a full-disk report, or (b) the error chain carries the runner's existing `sandbox_disk_full` launch code (a relaunch that cannot stage on the full disk). A fixed list of failures keeps its own meaning: permission, approval, usage limit, model, runtime, integrity, cleanup quarantine, ownership, saved-state and admission failures. - Server: `sandbox_disk_full` is a permanent native failure. The run does not take another same-run attempt. Native safe replacement does not pick it up, because it is not one of its failure codes. The task becomes blocked with a board-owned recovery action and a plain-language next action. - Server: the bounded retry scheduler (`scheduleBoundedRetryForRun`) refuses `sandbox_disk_full`. It does not schedule a `transient_failure` retry on a fresh sandbox of the same size. - UI: the task chat shows "Disk full" and a plain explanation instead of the raw failure code. - Retry stays available. `sandbox_disk_full` is not a reconciliation cause, so the board's Retry starts a new run when the user chooses. ## Verification - Red before the fix (captured on the unfixed code): - `heartbeat-retry-scheduling.test.ts` "does not repeat a run that filled its sandbox disk on a fresh sandbox of the same size": expected `not_scheduled`, received `scheduled`. - `native-sandbox-disk-full.test.ts` "names the full disk and stops instead of retrying the same work": expected `sandbox_disk_full`, received `native_provider_terminal_failed`. - `native-sandbox-disk-full.test.ts` "names the full disk when the runner could not be relaunched on it": expected `sandbox_disk_full`, received `native_session_interrupted`. - Green after the fix: - `cd server && npx vitest run src/services/native-runtime/native-sandbox-disk-full.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/native-runtime/remote-runner-disk-reserve.test.ts` (90 passed). - `cd ui && npx vitest run src/components/TaskChatThread.test.tsx` (426 passed). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-session-diagnostics.test.ts`. - Safety cases in the tests: - A runner that exited for another reason still gets its bounded retry. - A turn lost on restore without a full-disk report keeps `native_provider_terminal_failed`. - Integrity and approval failures keep their codes when a full disk was reported. - A full-disk report belongs only to its own run's diagnostic epoch; run B on the same session does not see run A's report. - After a full-disk report, other runner and provider loss codes (session.open timeout, initialize timeout, protocol error, process status failure) also become `sandbox_disk_full`. - Worst-case automatic run starts per hour across a company: before, each disk-full task started up to 3 automatic runs, each on a fresh sandbox (3 × the number of affected tasks). After, a disk-full failure starts 0 automatic runs. Only a person's Retry starts a new run. Real transient failures keep their existing bounded lanes, so their worst case is unchanged. - Not run locally: the full server suite, the full typecheck and build. Required CI covers them. - Not verified live on staging yet. A live check needs a run that fills its sandbox disk after this change deploys. ## Risks - A run whose runner reported a full disk, and which later fails with a runner or provider loss for an unrelated reason, is now labeled `sandbox_disk_full` and is not retried automatically. The board's Retry still works. This is the safer error: a full disk is the most likely cause, and an automatic retry would repeat the same disk use. - The full-disk report lives on the run's in-memory diagnostic lease. A server restart between the report and the failure loses it, and the run falls back to the previous classification. That is the previous behavior, not a new failure. - Low risk otherwise. No schema or API change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../heartbeat-retry-scheduling.test.ts | 16 +++++ server/src/services/heartbeat/retries.ts | 5 ++ .../native-sandbox-disk-full.test.ts | 71 +++++++++++++++++++ .../native-sandbox-disk-full.ts | 56 +++++++++++++++ .../native-session-diagnostics.ts | 7 +- .../native-runtime/native-session-executor.ts | 20 +++++- ui/src/components/TaskChatThread.test.tsx | 36 ++++++++++ ui/src/components/TaskChatThread.tsx | 9 ++- 8 files changed, 214 insertions(+), 6 deletions(-) create mode 100644 server/src/services/native-runtime/native-sandbox-disk-full.test.ts create mode 100644 server/src/services/native-runtime/native-sandbox-disk-full.ts diff --git a/server/src/__tests__/heartbeat-retry-scheduling.test.ts b/server/src/__tests__/heartbeat-retry-scheduling.test.ts index 55e75d888ec..e29d44ef727 100644 --- a/server/src/__tests__/heartbeat-retry-scheduling.test.ts +++ b/server/src/__tests__/heartbeat-retry-scheduling.test.ts @@ -265,6 +265,22 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => { expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0); }); + it("does not repeat a run that filled its sandbox disk on a fresh sandbox of the same size", async () => { + const runId = randomUUID(), companyId = randomUUID(), agentId = randomUUID(); + const now = new Date("2026-04-20T12:00:00.000Z"); + await seedRetryFixture({ runId, companyId, agentId, now, errorCode: "sandbox_disk_full" }); + expect(await heartbeat.scheduleBoundedRetry(runId, { now, random: () => 0 })) + .toMatchObject({ outcome: "not_scheduled", reason: expect.stringContaining("disk") }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0); + }); + + it("still retries a runner that exited for another reason", async () => { + const runId = randomUUID(), companyId = randomUUID(), agentId = randomUUID(); + const now = new Date("2026-04-20T12:00:00.000Z"); + await seedRetryFixture({ runId, companyId, agentId, now, errorCode: "native_runner_process_exited" }); + expect(await heartbeat.scheduleBoundedRetry(runId, { now, random: () => 0 })).toMatchObject({ outcome: "scheduled" }); + }); + it.each(["restore_unsafe_archive", "restore_lock_timeout"])("keeps the existing retry budget for %s", async (classification) => { const runId = randomUUID(), companyId = randomUUID(), agentId = randomUUID(); const now = new Date("2026-04-20T12:00:00.000Z"); diff --git a/server/src/services/heartbeat/retries.ts b/server/src/services/heartbeat/retries.ts index 6e5ee8e85dc..fd7fdeefb83 100644 --- a/server/src/services/heartbeat/retries.ts +++ b/server/src/services/heartbeat/retries.ts @@ -91,6 +91,7 @@ import { import type { AppendHeartbeatRunEventInput } from "../heartbeat-run-events.js"; import type { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js"; import { currentRunFailureText } from "../native-runtime/native-exact-resume-failure.js"; +import { NATIVE_SANDBOX_DISK_FULL, NATIVE_SANDBOX_DISK_FULL_ACTION } from "../native-runtime/native-sandbox-disk-full.js"; import type { createHeartbeatRunState } from "./run-state.js"; import type { createHeartbeatRunPreparation } from "./run-preparation.js"; import type { createRunDispatch, PostCommitEffect } from "../../modules/run-dispatch/index.js"; @@ -537,6 +538,10 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen || ["openai_dot", "muse"].includes(String(parseObject(parseObject(parseObject(run.runnerProfileJson).nativeExecutionInput).provider).kind))) { return { outcome: "not_scheduled" as const, reason: "Dot external execution must be reconciled before a new assignment; Paperclip cannot confirm its external stop.", issueId: readNonEmptyString(run.contextSnapshot?.issueId) }; } + if (run.errorCode === NATIVE_SANDBOX_DISK_FULL) { + return { outcome: "not_scheduled" as const, reason: NATIVE_SANDBOX_DISK_FULL_ACTION, + issueId: readNonEmptyString(run.contextSnapshot?.issueId) }; + } if (run.errorCode === "provider_tool_definition_invalid") { return { outcome: "not_scheduled" as const, reason: "Repair the invalid tool definitions before starting a new attempt.", diff --git a/server/src/services/native-runtime/native-sandbox-disk-full.test.ts b/server/src/services/native-runtime/native-sandbox-disk-full.test.ts new file mode 100644 index 00000000000..14e5acdcbe1 --- /dev/null +++ b/server/src/services/native-runtime/native-sandbox-disk-full.test.ts @@ -0,0 +1,71 @@ +import { describe, expect, it } from "vitest"; +import { NativeProviderTerminalFailure, NativeSessionProtocolIntegrityError, SANDBOX_DISK_FULL_FAILURE_CODE } from "../../vendor/paperclip-runner/index.js"; +import { nativeSessionFailureDisposition, nativeSessionFailureSourceCode } from "./native-session-executor.js"; +import { NATIVE_SANDBOX_DISK_FULL } from "./native-sandbox-disk-full.js"; +import { NativeSessionDiagnostics } from "./native-session-diagnostics.js"; + +const turnLostOnRestore = () => new NativeProviderTerminalFailure("provider_turn_lost_on_restore", true); + +describe("a native run that ends because its sandbox disk filled", () => { + it("names the full disk and stops instead of retrying the same work", () => { + const code = nativeSessionFailureSourceCode(turnLostOnRestore(), true); + expect(code).toBe("sandbox_disk_full"); + expect(nativeSessionFailureDisposition(1, new Date(), code)).toEqual({ + phase: "terminal_failure", failureCode: "sandbox_disk_full", nextAttemptAt: null }); + }); + + it("names the full disk when the runner could not be relaunched on it", () => { + const launch = Object.assign(new Error("sandbox_disk_full: the sandbox disk is full, so the runner cannot be started"), + { code: SANDBOX_DISK_FULL_FAILURE_CODE }); + const transport = new Error("Notification transport failed closed: sandbox_disk_full: the sandbox disk is full", { cause: launch }); + expect(nativeSessionFailureSourceCode(transport)).toBe("sandbox_disk_full"); + }); + + it("names the full disk when a failed relaunch reaches the run as a failed event", () => { + const failed = new NativeProviderTerminalFailure("notification_transport_failed", false, + "Notification transport failed closed: sandbox_disk_full: the sandbox disk is full, so the runner cannot be started"); + expect(nativeSessionFailureSourceCode(failed)).toBe("sandbox_disk_full"); + const owner = { companyId: "company", sessionScope: "session", runId: "run-a" }; + const channel = new NativeSessionDiagnostics(owner, owner, () => undefined, () => undefined); + const lease = channel.activate(owner, async () => undefined); + channel.record("[paperclip-runner] runnerd diagnostic: sandbox_disk_full: the sandbox disk is full, so the runner cannot be started\n"); + expect(lease.sandboxDiskFullObserved()).toBe(true); + }); + + it("keeps other failures as they are", () => { + expect(nativeSessionFailureSourceCode(turnLostOnRestore())).toBe("native_provider_terminal_failed"); + expect(nativeSessionFailureSourceCode(new NativeSessionProtocolIntegrityError("semantic_input_digest_mismatch"), true)) + .toBe("native_event_replay_conflict"); + expect(nativeSessionFailureSourceCode(new NativeProviderTerminalFailure("approval_required", false, "Approval required"), true)) + .toBe("native_provider_approval_required"); + expect(nativeSessionFailureDisposition(1, new Date(), nativeSessionFailureSourceCode(new Error("runnerd exited")))) + .toMatchObject({ phase: "retryable_failure" }); + }); + + it.each(["runner_ingress_unavailable: PRP command session.open timed out", "provider_initialize_timeout", "provider_request_protocol_error", + "provider_process_status_failed"])("names the full disk for a later %s on the same run", message => { + expect(nativeSessionFailureSourceCode(new Error(message), true)).toBe("sandbox_disk_full"); + expect(nativeSessionFailureDisposition(1, new Date(), nativeSessionFailureSourceCode(new Error(message), true))) + .toMatchObject({ phase: "terminal_failure", failureCode: "sandbox_disk_full" }); + }); + + it("keeps a full-disk report on the run that saw it", () => { + const owner = { companyId: "company", sessionScope: "session", runId: "run-a" }; + const channel = new NativeSessionDiagnostics(owner, owner, () => undefined, () => undefined); + const sink = async () => undefined; + const first = channel.activate(owner, sink); + channel.record("[paperclip-runner] runnerd diagnostic: runner process disconnected; recovery is allowed for 420000ms\n"); + expect(first.sandboxDiskFullObserved()).toBe(false); + channel.record("[paperclip-runner] runnerd diagnostic: failed to write state: No space left on device (os error 28)\n"); + expect(first.sandboxDiskFullObserved()).toBe(true); + first.retire(); + channel.record("[paperclip-runner] runnerd diagnostic: late exit: No space left on device (os error 28)\n"); + const second = channel.activate({ ...owner, runId: "run-b" }, sink); + channel.record("[paperclip-runner] runnerd diagnostic: runner process restored its durable PRP session\n"); + expect(second.sandboxDiskFullObserved()).toBe(false); + }); + + it("uses the runner's own code", () => { + expect(NATIVE_SANDBOX_DISK_FULL).toBe(SANDBOX_DISK_FULL_FAILURE_CODE); + }); +}); diff --git a/server/src/services/native-runtime/native-sandbox-disk-full.ts b/server/src/services/native-runtime/native-sandbox-disk-full.ts new file mode 100644 index 00000000000..2ca342e055e --- /dev/null +++ b/server/src/services/native-runtime/native-sandbox-disk-full.ts @@ -0,0 +1,56 @@ +import { isSandboxDiskFullFailure } from "../../vendor/paperclip-runner/index.js"; +import { isSandboxDiskFullText } from "./remote-runner-disk-reserve.js"; + +export const NATIVE_SANDBOX_DISK_FULL = "sandbox_disk_full" as const; + +export const NATIVE_SANDBOX_DISK_FULL_ACTION = + "The agent filled the disk of its computer, so its runner stopped. Paperclip does not repeat this automatically: " + + "a new computer has the same disk size, and the same work would fill it again. Ask the agent to use less disk space, " + + "for example by deleting build output, caches or dependencies it no longer needs, then press Retry."; + +const CODES_THAT_KEEP_THEIR_MEANING: ReadonlySet = new Set([ + "native_permission_declined", + "native_provider_approval_required", + "native_provider_usage_limit", + "native_provider_model_rejected", + "native_provider_runtime_missing", + "native_provider_state_incompatible", + "native_provider_side_effect_pending", + "native_provider_startup_rejected", + "native_event_replay_conflict", + "native_session_cleanup_quarantined", + "native_governed_wait_settlement_pending", + "native_predecessor_retirement_unproven", + "native_adopted_runner_authentication_timeout", + "native_controller_journal_behind_runner", + "native_current_wake_comments_unread", + "native_current_wake_comments_changed_after_read", + "runner_task_admission_rejected", + "runner_ingress_access_rejected", + "runner_remote_provider_artifact_incompatible", + "planning_mode_unsupported", +]); + +const NAMED_DISK_FULL_FAILURE = /(?:^|: )sandbox_disk_full: /; + +export function isNativeSandboxDiskFullReport(text: string): boolean { + return isSandboxDiskFullText(text) || NAMED_DISK_FULL_FAILURE.test(text); +} + +function diskFullCause(error: unknown, depth = 0): boolean { + if (!error || typeof error !== "object" || depth > 5) return false; + if (isSandboxDiskFullFailure(error)) return true; + const value = error as { cause?: unknown; errors?: unknown; message?: unknown }; + if (typeof value.message === "string" && NAMED_DISK_FULL_FAILURE.test(value.message)) return true; + return diskFullCause(value.cause, depth + 1) + || (Array.isArray(value.errors) && value.errors.slice(0, 3).some(item => diskFullCause(item, depth + 1))); +} + +export function nativeSandboxDiskFullSourceCode( + code: T, + error: unknown, + runnerSandboxDiskFull: boolean, +): T | typeof NATIVE_SANDBOX_DISK_FULL { + if (CODES_THAT_KEEP_THEIR_MEANING.has(code)) return code; + return runnerSandboxDiskFull || diskFullCause(error) ? NATIVE_SANDBOX_DISK_FULL : code; +} diff --git a/server/src/services/native-runtime/native-session-diagnostics.ts b/server/src/services/native-runtime/native-session-diagnostics.ts index d824304eb48..d86b19669cb 100644 --- a/server/src/services/native-runtime/native-session-diagnostics.ts +++ b/server/src/services/native-runtime/native-session-diagnostics.ts @@ -1,3 +1,5 @@ +import { isNativeSandboxDiskFullReport } from "./native-sandbox-disk-full.js"; + export interface NativeDiagnosticOwner { companyId: string; sessionScope: string; @@ -6,12 +8,13 @@ export interface NativeDiagnosticOwner { export interface NativeDiagnosticLease { retire(): void; + sandboxDiskFullObserved(): boolean; } export type NativeIdleDiagnosticKind = "transport_close_requested" | "session_diagnostic"; type Sink = (stream: "stdout" | "stderr", chunk: string) => Promise; -type Epoch = { owner: NativeDiagnosticOwner; sink?: Sink }; +type Epoch = { owner: NativeDiagnosticOwner; sink?: Sink; sandboxDiskFull?: boolean }; const sessions = new WeakMap(); export class NativeSessionDiagnostics { @@ -39,6 +42,7 @@ export class NativeSessionDiagnostics { this.lastOwner = epoch.owner; return { retire: () => { if (this.active === epoch) this.active = undefined; }, + sandboxDiskFullObserved: () => epoch.sandboxDiskFull === true, }; } @@ -55,6 +59,7 @@ export class NativeSessionDiagnostics { record(chunk: string): void { const epoch = this.active; + if (epoch && isNativeSandboxDiskFullReport(chunk)) epoch.sandboxDiskFull = true; if (!epoch?.sink) { const kind = chunk.startsWith("[paperclip-runner] runnerd diagnostic: runner transport close requested:") ? "transport_close_requested" : "session_diagnostic"; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 86eb62b791d..d676b2f2d8b 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -6,6 +6,7 @@ import { retryTransientLockContention } from "../transient-lock-contention.js"; import { nativeLeaseHolder, notifyNativeTokenLeaseClaimed } from "./native-token-lease-settlement.js"; import { captureRemoteNativeProviderArtifacts, shareNativeProviderArtifactCapture } from "./native-provider-artifact-capture.js"; import { NativeSessionDiagnostics, nativeSessionDiagnostics, type NativeDiagnosticLease } from "./native-session-diagnostics.js"; +import { NATIVE_SANDBOX_DISK_FULL, NATIVE_SANDBOX_DISK_FULL_ACTION, nativeSandboxDiskFullSourceCode } from "./native-sandbox-disk-full.js"; import { recordAgentProviderRateLimits } from "../agent-provider-rate-limits.js"; import { nativePostResultFailureCategory, type NativePostResultFailureCategory } from "./native-post-result-failure.js"; import { logger } from "../../middleware/logger.js"; @@ -9064,7 +9065,8 @@ export function nativeSessionFailureDisposition( sourceFailureCode === "native_provider_usage_limit" || sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED || sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED || - sourceFailureCode === NATIVE_CONTROLLER_JOURNAL_BEHIND_RUNNER; + sourceFailureCode === NATIVE_CONTROLLER_JOURNAL_BEHIND_RUNNER || + sourceFailureCode === NATIVE_SANDBOX_DISK_FULL; const permissionDeclined = sourceFailureCode === "native_permission_declined"; const exhausted = permanentFailure || permissionDeclined || attempt >= 3; return { @@ -9242,7 +9244,11 @@ function runnerClosedTurnStartSession(error: unknown, depth = 0): boolean { return runnerClosedTurnStartSession(value.cause, depth + 1); } -export function nativeSessionFailureSourceCode( +export function nativeSessionFailureSourceCode(error: unknown, runnerSandboxDiskFull = false) { + return nativeSandboxDiskFullSourceCode(classifyNativeSessionFailure(error), error, runnerSandboxDiskFull); +} + +function classifyNativeSessionFailure( error: unknown, ): | "native_permission_declined" @@ -10997,6 +11003,9 @@ async function executePaperclipNativeSessionAttempt( cleanupOnly = false, ownDiagnosticLease?: (lease: NativeDiagnosticLease) => void, ): Promise { + const attemptDiagnosticLeases: NativeDiagnosticLease[] = []; + const ownRunDiagnosticLease = ownDiagnosticLease; + ownDiagnosticLease = lease => { attemptDiagnosticLeases.push(lease); ownRunDiagnosticLease?.(lease); }; if (input.execution.provider.kind === "acpx" && input.execution.provider.agent === "antigravity") { const candidate = resolveAntigravityCandidateAdmission({ companyId: input.execution.binding.companyId, model: input.execution.provider.model, @@ -13277,7 +13286,8 @@ async function executePaperclipNativeSessionAttempt( throw error; } const now = new Date(); - const startupClassifiedFailureCode = nativeSessionFailureSourceCode(error); + const startupClassifiedFailureCode = nativeSessionFailureSourceCode(error, + attemptDiagnosticLeases.some(lease => lease.sandboxDiskFullObserved())); // A timed-out session.open may still have opened a provider session // before its runner stopped. The same-run retry must not reuse it. const classifiedFailureCode = startupClassifiedFailureCode === "provider_session_open_timeout" && @@ -13497,6 +13507,8 @@ async function executePaperclipNativeSessionAttempt( ? RUNNER_TASK_ADMISSION_REJECTED_ACTION : sourceFailureCode === "runner_ingress_access_rejected" ? "The computer provider rejected access. Restore the environment's credentials or permissions before retrying. Automatic retries are stopped." + : sourceFailureCode === NATIVE_SANDBOX_DISK_FULL + ? NATIVE_SANDBOX_DISK_FULL_ACTION : sourceFailureCode === "native_provider_terminal_failed" ? "The provider session is permanently unusable. Verify stopped execution, completed actions, and task context before starting a linked continuation." : sourceFailureCode === "native_governed_wait_settlement_pending" @@ -13776,6 +13788,8 @@ async function executePaperclipNativeSessionAttempt( ? RUNNER_TASK_ADMISSION_REJECTED_ACTION : sourceFailureCode === "runner_ingress_access_rejected" ? "The computer provider rejected access. Restore the environment's credentials or permissions before retrying. Automatic retries are stopped." + : sourceFailureCode === NATIVE_SANDBOX_DISK_FULL + ? NATIVE_SANDBOX_DISK_FULL_ACTION : sourceFailureCode === "native_provider_terminal_failed" ? "Verify that the failed provider stopped and reconcile its action outcomes. A linked continuation can proceed only after these checks succeed." : sourceFailureCode === "native_provider_model_rejected" && !recoveryEvidence.providerSessionEstablished diff --git a/ui/src/components/TaskChatThread.test.tsx b/ui/src/components/TaskChatThread.test.tsx index fbb312553dd..ef337fa382e 100644 --- a/ui/src/components/TaskChatThread.test.tsx +++ b/ui/src/components/TaskChatThread.test.tsx @@ -2434,6 +2434,42 @@ describe("TaskChatThread runtime transcript selection", () => { expect(container.querySelector('[data-testid="task-chat-run-failed-try-again"]')).toBeNull(); }); + it("explains a full sandbox disk without exposing its error code", async () => { + renderThread({ + comments: [], onAdd: async () => {}, issueStatus: "blocked", onRetryFailedRun: vi.fn(), + linkedRuns: [{ + runId: "native-disk-full", runtimeMode: "native", status: "failed", errorCode: "sandbox_disk_full", + agentId: "agent-1", agentName: "Runner", adapterType: "paperclip_runner", + createdAt: "2026-08-25T18:00:00.000Z", startedAt: "2026-08-25T18:00:00.000Z", finishedAt: "2026-08-25T18:00:02.000Z", + }], + }); + const marker = container.querySelector('[data-testid="task-chat-collapsible-marker"]'); + expect(marker?.textContent).toContain("Disk full"); + flushSync(() => marker!.querySelector('button[aria-expanded="false"]')!.click()); + const details = container.querySelector('[data-testid="task-chat-collapsible-marker-details"]'); + expect(details?.textContent).toContain("Ask the agent to use less disk space"); + expect(details?.textContent).not.toContain("sandbox_disk_full"); + }); + + it("explains a full sandbox disk beside the task's interruption without a second Retry", async () => { + renderThread({ + comments: [], onAdd: async () => {}, issueStatus: "blocked", onRetryFailedRun: vi.fn(), + interruption: { intentId: null, responsibility: "system", state: "failed", nextRetryAt: null, canRetry: true, + userAction: null, sourceRunIds: ["native-disk-full"], observedRunId: "native-disk-full" }, + linkedRuns: [{ + runId: "native-disk-full", runtimeMode: "native", status: "failed", errorCode: "sandbox_disk_full", + agentId: "agent-1", agentName: "Runner", adapterType: "paperclip_runner", + createdAt: "2026-08-25T18:00:00.000Z", startedAt: "2026-08-25T18:00:00.000Z", finishedAt: "2026-08-25T18:00:02.000Z", + }], + }); + const marker = container.querySelector('[data-testid="task-chat-collapsible-marker"]'); + expect(marker?.textContent).toContain("Disk full"); + flushSync(() => marker!.querySelector('button[aria-expanded="false"]')!.click()); + expect(container.querySelector('[data-testid="task-chat-collapsible-marker-details"]')?.textContent) + .toContain("Ask the agent to use less disk space"); + expect(container.querySelector('[data-testid="task-chat-run-failed-try-again"]')).toBeNull(); + }); + it("explains a native provider usage limit without exposing its error code", async () => { const onRetryFailedRun = vi.fn(); renderThread({ diff --git a/ui/src/components/TaskChatThread.tsx b/ui/src/components/TaskChatThread.tsx index cf7ec215898..24736265263 100644 --- a/ui/src/components/TaskChatThread.tsx +++ b/ui/src/components/TaskChatThread.tsx @@ -1865,6 +1865,8 @@ export function TaskChatThread(props: TaskChatThreadProps) { ? "Approval required" : code === "native_provider_usage_limit" && source.status === "failed" ? "Usage limit reached" + : code === "sandbox_disk_full" && source.status === "failed" + ? "Disk full" : source.status === "cancelled" ? "Run cancelled" : source.status === "interrupted" @@ -1891,6 +1893,8 @@ export function TaskChatThread(props: TaskChatThreadProps) { : code === "native_provider_usage_limit" && source.status === "failed" ? "The model provider has reached its current usage limit. Try again after the limit resets." + : code === "sandbox_disk_full" && source.status === "failed" + ? "The agent filled the disk of its computer, so the run stopped. Ask the agent to use less disk space, then retry." : code === "provider_frame_too_large" ? "Provider output exceeded the safe limit." : source.status === "timed_out" @@ -1902,7 +1906,8 @@ export function TaskChatThread(props: TaskChatThreadProps) { : undefined; const finishedAt = meta?.finishedAt ?? meta?.startedAt ?? meta?.createdAt; - if (!historical && (!props.interruption || credentialAccess || + const diskFullBesideInterruption = Boolean(props.interruption) && code === "sandbox_disk_full" && source.status === "failed"; + if (!historical && (!props.interruption || credentialAccess || diskFullBesideInterruption || (props.interruption.responsibility === "user" && ["native_provider_model_rejected", "native_provider_approval_required"].includes(code)))) entriesWithFailures.push({ ms: toMs(finishedAt), @@ -1915,7 +1920,7 @@ export function TaskChatThread(props: TaskChatThreadProps) { tone: source.status === "cancelled" ? "neutral" : "error", label, detail, - retryable: code !== "native_session_cleanup_quarantined", + retryable: code !== "native_session_cleanup_quarantined" && !diskFullBesideInterruption, ...(credentialAccess ? { credentialAccess, retryable: false } : {}), collapsible: true, runId: source.id,