test(chaos): add DF1, a sandbox disk-full fault on the final workspace save (#20810)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs real agent
tasks and breaks the control plane on purpose, so failures show up
before staging users see them
> - A census of 48 hours of staging failures found runs whose result
save-back failed because the sandbox disk was full (class F). The
harness never filled a sandbox disk, so it missed that class
> - #20758 made a full-disk save end recovery at once with cause
`sandbox_disk_full` and a Retry. Nothing in chaos exercised that path
end to end
> - A real full disk cannot be made on demand. The owner chose a
test-only hook that makes the save report a full disk, not a small disk
> - This pull request adds the hook at the save's sandbox boundary and a
DF1 fault that arms it, clears it, and grades the result
> - The benefit is that every wave can now check the disk-full rest, the
Retry offer, and that Retry works after space is freed

## Linked Issues or Issue Description

Refs #20758 (product behaviour under test), #20678 (finalization hands a
full-disk copy-back to recovery)

**What is the problem?**
The chaos harness had no fault for a full sandbox disk during result
save-back. Staging saw this failure (census class F in
`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos
could not reproduce it.

**What is the expected behaviour?**
A wave can schedule DF1. While DF1 is armed, the task's final workspace
saves fail as a full disk. When DF1 is cleared, saves succeed again. The
task must rest with `sandbox_disk_full` and a Retry, and that Retry must
finish the task.

**Why does it matter?**
Without a fault, a regression in the disk-full path (no Retry, a wrong
cause, or a Retry that does not work) reaches staging users before chaos
sees it.

## What Changed

- `server/src/services/sandbox-disk-full-fault.ts` (new, test-only):
`sandboxDiskFullFaultTarget` wraps the target of a final save only when
the harness armed that save. The capture commands then exit 1 with the
exact message the in-sandbox script writes on ENOSPC, and the provider
download fails with `No space left on device`. The gate follows
`@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL`
is read once at import and ignored under `NODE_ENV=production`.
Disarmed, the hook is one comparison and returns the save's own target.
- `server/src/services/remote-workspaces.ts`: the capture call in
`saveRemoteWorkspace` passes its target through the hook. No other
product change. `workspace-recovery.ts` is not touched.
- `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the
control-file seam (atomic writes), the rest evidence
(`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`),
the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry
that must finish the task).
- Harness wiring: `--faults DF1` (refused without `--env daytona`, with
a storm, or with PV faults), no scheduled restart, arm at launch, clear
at the first rest just before the Retry click, receipts in
`<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: {
faultedSaves, restedDiskFull, retryOffered, retryWorked }`.
- Docs: a "Sandbox disk full (DF1)" section in
`tests/restart-chaos/README.md`, and the census plan marks DF1 as built.

## Verification

- `cd server && pnpm exec vitest run
src/services/sandbox-disk-full-fault.test.ts`: the gate is off by
default and under `NODE_ENV=production`, the save's own target comes
back when disarmed, only a listed final save is faulted, the product
capture turns it into the named disk-full failure, and clearing the file
disarms the next save.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/disk-full-faults.test.ts
tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/provider-faults.test.ts`: planning, the control
file, and grading.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the
server `tsc --noEmit`: clean.
- Live Daytona waves (Codex, `--storage s3`, image content `96c4339f…`,
run through the two-slot queue):
- `w-20261111T131314-s9411` (B1): the hook was armed in the server but
no save reached it. B1 uses a local worktree synced into the sandbox,
not a remote workspace. Graded `not_applied` as designed. The harness
now requires `--storage s3` and the docs point at R1.
- `w-20261111T131941-s9412` (R1): 3 faulted saves. t01 rested `blocked`
with `sandbox_disk_full`, `user_action_required`, board owner, API
`canRetry: true`. t02 was cleared too early, while recovery still
checked its source; that recovery then saved and the task finished.
Fixed in 626f552e (wait for the disk-full verdict).
- `w-20261111T134316-s9413` (R1, after the fix): 2/2 tasks rested with
`sandbox_disk_full` and API `canRetry: true`. In both, the board showed
**"Recover saved files"** and no Retry button, so the UI Retry click
failed (`ui_retry_button_missing`). Retry-worked is not shown. The wave
was interrupted by SIGTERM during the post-rest file reads, before final
grading.
- Local wave: not run. A local run has no sandbox workspace save, so DF1
cannot fire there; the harness refuses `--env local`.

## Risks

- Low risk for production. The hook reads its variable once at startup
and is ignored under `NODE_ENV=production`. Without the variable it does
one comparison and returns the same target object.
- The hook fails every sandbox command of an armed capture, including
the read-only fallback's inventory. A real full disk can let that
inventory succeed and fail later at the download. The outcome is the
same (`sandbox_disk_full`), but the fallback's inventory step is not
exercised under DF1.
- DF1 runs only with `--env daytona --storage s3` and a remote workspace
(R1). A local run has no sandbox workspace save, so the harness refuses
DF1 there. DF1 is refused in `--matrix` waves, and T1's restart check is
skipped for DF1.

## Model Used

- Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), in Claude
Code with tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

server/src/services/remote-workspaces.ts

@@ -9,6 +9,7 @@ import os from "node:os";
 import path from "node:path";
 import { and, eq, inArray, isNull, or, sql } from "drizzle-orm";
 import { crashPoint } from "@paperclipai/shared/crash-points";
+import { sandboxDiskFullFaultTarget } from "./sandbox-disk-full-fault.js";
 import {
   companies, environmentLeases, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRuns, issues, workspaceOperations, type Db,
 } from "@paperclipai/db";
@@ -912,7 +913,9 @@ export async function saveRemoteWorkspace(input: {
     const existingUpload = snapshot.status === "uploaded" ? await getWorkspaceSnapshot(input.db, { companyId: input.workspace.companyId,
       workspaceId: input.workspace.id, snapshotId: snapshot.id }) : null;
     if (existingUpload) timing.reusedUpload = true;
-    captured = existingUpload ? null : await captureRemoteWorkspaceArchive({ target, remoteCwd: expected.location.remoteCwd,
+    captured = existingUpload ? null : await captureRemoteWorkspaceArchive({ remoteCwd: expected.location.remoteCwd,
+      // Test-only DF1 hook (sandbox-disk-full-fault.ts): the target itself unless a harness armed this save.
+      target: sandboxDiskFullFaultTarget(target, { boundary, issueId: operation.issueId, runId: input.runId }),
       operationId: operation.id, timing });
     await input.assertOwned();
     const uploadStarted = performance.now();

server/src/services/sandbox-disk-full-fault.test.ts

@@ -0,0 +1,78 @@
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { afterEach, describe, expect, it } from "vitest";
+import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL, captureRemoteWorkspaceArchive, type RemoteWorkspaceTarget } from "./remote-workspace-transport.js";
+import {
+  DISK_FULL_FAULT_CONTROL_ENV, DISK_FULL_FAULT_RECEIPT_ENV, DISK_FULL_FAULT_SYNC_OUT_MESSAGE,
+  readDiskFullFaultGate, sandboxDiskFullFaultTarget, setDiskFullFaultGateForTest,
+} from "./sandbox-disk-full-fault.js";
+
+const ISSUE = "11111111-1111-4111-8111-111111111111";
+const RUN = "22222222-2222-4222-8222-222222222222";
+const OPERATION = "33333333-3333-4333-8333-333333333333";
+
+function sandboxTarget(calls: string[]): RemoteWorkspaceTarget {
+  return {
+    kind: "remote", transport: "sandbox", remoteCwd: "/workspace", providerKey: "test",
+    runner: {
+      execute: async () => { calls.push("execute"); return { exitCode: 0, signal: null, timedOut: false, stdout: "{}", stderr: "", pid: 1, startedAt: null }; },
+      syncIn: async () => { calls.push("syncIn"); return {} as never; },
+      syncOut: async () => { calls.push("syncOut"); return {} as never; },
+    },
+  } as unknown as RemoteWorkspaceTarget;
+}
+
+describe("sandbox disk-full fault gate", () => {
+  let restore: (() => void) | null = null;
+  const dirs: string[] = [];
+  afterEach(() => {
+    restore?.(); restore = null;
+    for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true });
+  });
+  const scratch = () => { const dir = mkdtempSync(path.join(os.tmpdir(), "disk-full-fault-")); dirs.push(dir); return dir; };
+
+  it("is off unless the harness switch is set, and never on in production", () => {
+    expect(readDiskFullFaultGate({})).toBeNull();
+    expect(readDiskFullFaultGate({ [DISK_FULL_FAULT_CONTROL_ENV]: "  " })).toBeNull();
+    expect(readDiskFullFaultGate({ NODE_ENV: "production", [DISK_FULL_FAULT_CONTROL_ENV]: "/tmp/control" })).toBeNull();
+    expect(readDiskFullFaultGate({ NODE_ENV: "test", [DISK_FULL_FAULT_CONTROL_ENV]: "/tmp/control", [DISK_FULL_FAULT_RECEIPT_ENV]: "/tmp/receipt" }))
+      .toEqual({ controlPath: "/tmp/control", receiptPath: "/tmp/receipt" });
+  });
+
+  it("returns the save's own target when the process was started without the switch", () => {
+    const target = sandboxTarget([]);
+    expect(sandboxDiskFullFaultTarget(target, { boundary: "final", issueId: ISSUE, runId: RUN })).toBe(target);
+  });
+
+  it("faults only a final save whose issue or run is listed, and stops when cleared", async () => {
+    const dir = scratch();
+    const control = path.join(dir, "control"), receipt = path.join(dir, "receipts.jsonl");
+    restore = setDiskFullFaultGateForTest({ controlPath: control, receiptPath: receipt });
+    const calls: string[] = [];
+    const target = sandboxTarget(calls);
+    // No control file yet: disarmed.
+    expect(sandboxDiskFullFaultTarget(target, { boundary: "final", issueId: ISSUE, runId: RUN })).toBe(target);
+    writeFileSync(control, `other-issue\n${ISSUE}\n`);
+    expect(sandboxDiskFullFaultTarget(target, { boundary: "initial", issueId: ISSUE, runId: RUN })).toBe(target);
+    expect(sandboxDiskFullFaultTarget(target, { boundary: "final", issueId: "other", runId: "other-run" })).toBe(target);
+
+    const faulted = sandboxDiskFullFaultTarget(target, { boundary: "final", issueId: ISSUE, runId: RUN });
+    expect(faulted).not.toBe(target);
+    expect(await faulted.runner!.execute({ command: "node" })).toMatchObject({ exitCode: 1, stderr: REMOTE_WORKSPACE_SANDBOX_DISK_FULL });
+    await expect(faulted.runner!.syncOut!([])).rejects.toThrow(DISK_FULL_FAULT_SYNC_OUT_MESSAGE);
+    await faulted.runner!.syncIn!([]);
+    expect(calls).toEqual(["syncIn"]);
+    expect(JSON.parse(readFileSync(receipt, "utf8").trim())).toMatchObject({ issueId: ISSUE, runId: RUN, boundary: "final" });
+
+    // The save's own capture path turns the sandbox's answer into the named disk-full failure.
+    await expect(captureRemoteWorkspaceArchive({ target: faulted, remoteCwd: "/workspace", operationId: OPERATION }))
+      .rejects.toThrow(REMOTE_WORKSPACE_SANDBOX_DISK_FULL);
+
+    // A run id arms it too; clearing the file disarms the next save.
+    writeFileSync(control, `${RUN}\n`);
+    expect(sandboxDiskFullFaultTarget(target, { boundary: "final", issueId: null, runId: RUN })).not.toBe(target);
+    writeFileSync(control, "");
+    expect(sandboxDiskFullFaultTarget(target, { boundary: "final", issueId: ISSUE, runId: RUN })).toBe(target);
+  });
+});

server/src/services/sandbox-disk-full-fault.ts

@@ -0,0 +1,82 @@
+/**
+ * Test-only sandbox disk-full fault (restart-chaos DF1, tests/restart-chaos/disk-full-faults.ts).
+ *
+ * A real full sandbox disk cannot be made on demand, locally or on Daytona. This hook makes a
+ * final workspace save see what a full sandbox returns, and leaves every decision after that to
+ * the product: the save's own failure handling, finalization, saved-work recovery and Retry.
+ *
+ * Arming (test mode only), in the same style as `@paperclipai/shared/crash-points`:
+ *
+ *   PAPERCLIP_DISK_FULL_FAULT_CONTROL=<file>  each final save reads this file. A line that equals
+ *                                             the save's issue id or run id arms that save. No file,
+ *                                             or no matching line, means disarmed.
+ *   PAPERCLIP_DISK_FULL_FAULT_RECEIPT=<file>  optional; one JSON line is appended per faulted save.
+ *
+ * While a save is armed, every sandbox command its capture runs exits 1 with the exact message the
+ * in-sandbox capture script writes on ENOSPC, and the provider's file download fails with tar's
+ * "No space left on device". Restore and other operations are untouched.
+ *
+ * The variable is read once, when this module is first imported, and is ignored when
+ * `NODE_ENV=production`. Disarmed, the hook is one comparison against a module constant and
+ * returns the save's own target. Only the chaos harness sets the variable, on the server it starts.
+ *
+ * This is test instrumentation, not Telemetry or Observability: nothing is recorded or sent
+ * anywhere unless a test arms a save.
+ */
+import { appendFileSync, readFileSync } from "node:fs";
+import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL, type RemoteWorkspaceTarget } from "./remote-workspace-transport.js";
+
+export const DISK_FULL_FAULT_CONTROL_ENV = "PAPERCLIP_DISK_FULL_FAULT_CONTROL";
+export const DISK_FULL_FAULT_RECEIPT_ENV = "PAPERCLIP_DISK_FULL_FAULT_RECEIPT";
+/** The provider download's failure on a full disk: tar cannot write its archive. */
+export const DISK_FULL_FAULT_SYNC_OUT_MESSAGE = "tar: workspace archive: Cannot write: No space left on device";
+
+export type DiskFullFaultReceipt = { issueId: string | null; runId: string; boundary: string; pid: number; at: string };
+
+type Gate = { controlPath: string; receiptPath: string | null };
+
+export function readDiskFullFaultGate(env: NodeJS.ProcessEnv | undefined): Gate | null {
+  if (!env || env.NODE_ENV === "production") return null;
+  const controlPath = env[DISK_FULL_FAULT_CONTROL_ENV]?.trim();
+  if (!controlPath) return null;
+  return { controlPath, receiptPath: env[DISK_FULL_FAULT_RECEIPT_ENV]?.trim() || null };
+}
+
+let gate: Gate | null = readDiskFullFaultGate(typeof process === "undefined" ? undefined : process.env);
+if (gate) process.stderr.write(`[disk-full-fault] armed from ${gate.controlPath} pid=${process.pid}\n`);
+
+function armed(current: Gate, ids: readonly string[]): boolean {
+  let text: string;
+  try { text = readFileSync(current.controlPath, "utf8"); } catch { return false; }
+  const lines = new Set(text.split("\n").map(line => line.trim()).filter(Boolean));
+  return ids.some(id => lines.has(id));
+}
+
+/**
+ * The target a workspace save captures through. Disarmed, or for any boundary other than the
+ * final copy-back, it is `target` itself.
+ */
+export function sandboxDiskFullFaultTarget<T extends RemoteWorkspaceTarget>(target: T,
+  save: { boundary: string; issueId: string | null; runId: string }): T {
+  if (gate === null) return target;
+  if (save.boundary !== "final" || !armed(gate, [save.runId, ...(save.issueId ? [save.issueId] : [])])) return target;
+  const receipt: DiskFullFaultReceipt = { issueId: save.issueId, runId: save.runId, boundary: save.boundary,
+    pid: process.pid, at: new Date().toISOString() };
+  if (gate.receiptPath) {
+    try { appendFileSync(gate.receiptPath, `${JSON.stringify(receipt)}\n`); } catch { /* the log line still records it */ }
+  }
+  process.stderr.write(`[disk-full-fault] save ${JSON.stringify(receipt)}\n`);
+  const runner = target.runner!;
+  const full = Object.create(runner) as typeof runner;
+  full.execute = async () => ({ exitCode: 1, signal: null, timedOut: false, stdout: "",
+    stderr: REMOTE_WORKSPACE_SANDBOX_DISK_FULL, pid: null, startedAt: null });
+  if (runner.syncOut) full.syncOut = async () => { throw new Error(DISK_FULL_FAULT_SYNC_OUT_MESSAGE); };
+  return { ...target, runner: full };
+}
+
+/** Replaces the gate inside a test process; returns a function that restores the previous one. */
+export function setDiskFullFaultGateForTest(next: Gate | null): () => void {
+  const previous = gate;
+  gate = next;
+  return () => { gate = previous; };
+}