test(chaos): run waves against S3-compatible storage with a fault proxy, and click Retry in the real board UI (#20288)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud deployments keep run logs, traces, attachments, artifacts and
saved workspace files in S3-compatible object storage
> - The restart chaos harness ran only with local disk storage, so
storage failures and slowdowns were never tested
> - It also clicked Retry through the API, so a missing or broken Retry
button in the board UI went unnoticed (staging PAP-1683: "the
saved-files recovery Retry does nothing")
> - This pull request runs waves against a loopback MinIO behind a fault
proxy, adds store-outage, latency and throttling faults, and clicks
every Retry in the real board UI
> - The benefit is that storage outages and broken Retry paths now
surface in chaos waves. The first ST1 wave found that a run starting
during a store outage fails immediately and exhausts recovery

## Linked Issues or Issue Description

**What happened?**
Staging saves to object storage and has failures there that local chaos
waves cannot reproduce, because the harness used `local_disk`. The
harness also clicked Retry through the API, which skips the board's
actual button.

**Expected behavior**
Chaos waves can run the server in cloud storage mode, inject storage
faults, and press Retry the way a user does.

**Actual behavior**
They could not.

## What Changed

- `--storage s3`:
- starts a loopback MinIO (`RELEASE.2025-04-22T22-12-26Z`, from
`~/.cache/paperclip/tools/minio/minio` or `RESTART_CHAOS_MINIO_BIN`)
with its data under the instance temp root and random root credentials;
  - creates the bucket;
  - puts a fault proxy in front;
- points the server's `PAPERCLIP_STORAGE_*` settings at the proxy, with
path-style addressing and a per-wave prefix. The supervisor declares
these as explicit workspace storage for the isolation check.
- `storage-proxy.ts`: a transparent, non-retrying S3 fault proxy that
keeps the signed Host header. Rules match by method, path and remaining
count. Faults are outage (503 `ServiceUnavailable`), latency with
jitter, a 5xx or `SlowDown` error rate, connection reset, and 403
`ExpiredToken`. It counts requests by method and status, written to
`storage-proxy.json`.
- `--storage-faults ST1,ST2,ST3` (`storage-chaos.ts`): seeded 60–180 s
windows of outage, 1–10 s latency, or 30% 503 / `SlowDown`, applied
while the wave runs. They're logged in the timeline and in
`storage-faults.json`.
- `board-ui.ts`: every task and subtask Retry the grader needs is now
clicked in the board UI. The harness uses headless Chromium through
Playwright, opens `/issues/<identifier>`, waits for the visible Retry
button and captures the `POST /issues/:id/retry` the click sends.
- R3 fails a missing button as `ui_retry_button_missing (user saw: …)`
and a click that sends nothing as `ui_retry_no_request`.
  - Crash-point waves, which serve no UI, keep the API path.
- README: a cloud storage section.

## Verification

- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 244
passed. New tests:
- the proxy against a fake upstream: passthrough, bounded outage,
`SlowDown`, `ExpiredToken`, latency and reset;
- live MinIO through the proxy: `If-None-Match` returns 412, and an
injected 503 reaches the client;
- Playwright against local pages: a working button, a missing button,
and a dead button;
- the observer clicking through the board, and R3 signatures for UI
failures;
  - window planning, apply and clear;
  - `--storage` and `--storage-faults` parsing.

  The observer and grader tests failed before the change.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Live waves on master 610b54f551 with this branch:
- Baseline `--storage s3`, A1/C1, K1/K2 (w-20261109T144738-s9901): 5/8.
There were 2,319 S3 requests through the proxy and 18 conditional-write
412 fences. All 3 Retries were clicked in the board UI, returned 202
Accepted, and the new run succeeded. The 3 failures are a Claude session
issue (#20285), not storage.
- ST1 (w-20261109T145628-s9902): 1/8. A run that starts during a store
outage fails within about 1 s as `adapter_failed`, automatic recovery
exhausts within the outage, and UI Retries are accepted but do not
recover. This has been reported for a product fix.

## Risks

- `--storage s3` needs a locally built MinIO. MinIO no longer publishes
community binaries. Without the binary the wave fails at start with the
path to provide, and the live store test is skipped.
- The UI Retry needs the dev UI to load. A slow first Vite compile can
delay the click, up to the 60 s wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tests/restart-chaos/README.md

@@ -171,6 +171,44 @@ receipt names both commits (`upgrade` in `faults.json`, `upgrade_switched` in th
 timeline). K14 is local-only, because a Daytona runner must match its image, and
 it is never part of a storm's default faults.
 
+## Cloud storage (`--storage s3`, `--storage-faults`)
+
+```sh
+node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \
+  --adapters codex,claude --workloads A1,C1 --faults K1,K2 --concurrency 4 --tasks 8 --seed 24 \
+  --storage s3 --storage-faults ST1,ST2,ST3
+```
+
+`--storage s3` runs the server the way cloud mode does: `PAPERCLIP_STORAGE_PROVIDER=s3`, so run logs
+and provider traces use the `s3_segments` backend and attachments, artifacts and other stored files go
+to object storage. The store is a loopback MinIO (`RELEASE.2025-04-22T22-12-26Z`, built from source at
+`~/.cache/paperclip/tools/minio/minio` or `RESTART_CHAOS_MINIO_BIN`) with its data under the instance
+temp root, behind a fault proxy that every server S3 call goes through. The proxy's request counts by
+method and status are written to `storage-proxy.json`.
+
+`--storage-faults` adds seeded windows of 60–180 s, alongside the restart faults:
+
+| Fault | Store behaviour during the window |
+| --- | --- |
+| ST1 | outage: every request answers 503 `ServiceUnavailable` |
+| ST2 | slow store: 1–10 s added latency with jitter |
+| ST3 | 30% of requests answer 503, half of them `SlowDown` |
+
+`--storage-fault-seconds 15-45` sets the window length range (5–900 s), for example to find how long
+an outage a run survives. The windows are logged in the timeline and written to `storage-faults.json`.
+
+With `--env daytona --storage s3`, each task's workspace is a remote workspace: the controller saves
+and restores it through the store, so the faults also hit workspace save, restore and copyback. R1
+(`--workspace-repo`) runs on Daytona only with `--storage s3`: the project points at the checkout's
+GitHub `origin` over https at its `origin/master` commit, so the sandbox clones the repository and its
+snapshots are Paperclip-sized. Its oracle reads the task's files through the board API and runs its
+commands on a local copy.
+
+Every Retry the grader needs is clicked in the real board UI: the harness opens the issue in a headless
+Chromium, waits for the visible Retry button and clicks it. A task whose Retry button is missing fails
+R3 as `ui_retry_button_missing (user saw: …)`; a click that sends no retry request fails as
+`ui_retry_no_request`. Crash-point waves serve no UI and click Retry through the API.
+
 ## Daytona (`--env daytona`)
 
 The control plane stays local; agent runs execute in Daytona sandboxes from an

tests/restart-chaos/api.ts

@@ -20,16 +20,16 @@ export function rateLimitDelayMs(response: { status: number; headers: { get(name
 export class ChaosApi {
   constructor(readonly baseUrl: string, private readonly timeoutMs = 10_000, private readonly rateLimitAttempts = 5) {}
 
-  private async request<T>(method: string, route: string, body?: unknown, timeoutMs = this.timeoutMs): Promise<T> {
+  private async request<T>(method: string, route: string, body?: unknown, timeoutMs = this.timeoutMs, text = false): Promise<T> {
     for (let attempt = 1; ; attempt += 1) {
-      const outcome = await this.attempt<T>(method, route, body, timeoutMs);
+      const outcome = await this.attempt<T>(method, route, body, timeoutMs, text);
       if (!("retryAfterMs" in outcome)) return outcome.value;
       if (method !== "GET" || attempt >= this.rateLimitAttempts) throw outcome.error;
       await new Promise(resolve => setTimeout(resolve, Math.min(10_000, outcome.retryAfterMs * attempt)));
     }
   }
 
-  private async attempt<T>(method: string, route: string, body: unknown, timeoutMs: number):
+  private async attempt<T>(method: string, route: string, body: unknown, timeoutMs: number, asText = false):
     Promise<{ value: T } | { retryAfterMs: number; error: ChaosApiError }> {
     let response: Response;
     try {
@@ -50,10 +50,11 @@ export class ChaosApi {
       if (retryAfterMs !== null) return { retryAfterMs, error };
       throw error;
     }
-    return { value: (text ? JSON.parse(text) : null) as T };
+    return { value: (asText ? text : text ? JSON.parse(text) : null) as T };
   }
 
   get<T>(route: string, timeoutMs?: number) { return this.request<T>("GET", route, undefined, timeoutMs); }
+  getText(route: string, timeoutMs?: number) { return this.request<string>("GET", route, undefined, timeoutMs, true); }
   post<T>(route: string, body?: unknown, timeoutMs?: number) { return this.request<T>("POST", route, body ?? {}, timeoutMs); }
   put<T>(route: string, body?: unknown) { return this.request<T>("PUT", route, body ?? {}); }
   patch<T>(route: string, body?: unknown) { return this.request<T>("PATCH", route, body ?? {}); }

tests/restart-chaos/board-ui.test.ts

@@ -0,0 +1,86 @@
+import { createServer, type Server } from "node:http";
+import { once } from "node:events";
+import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import { BoardUi, UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST, isRetryRequestFor } from "./board-ui.js";
+
+const retry = (issue: string) => `fetch('/api/issues/${issue}/retry',{method:'POST'})`;
+const pages: Record<string, string> = {
+  "/issues/CHA-1": `<div role="status">Couldn’t finish this attempt. <button onclick="${retry("i1")}">Retry</button></div>`,
+  "/issues/CHA-2": `<div role="status">Saving files from the previous attempt</div>`,
+  "/issues/CHA-3": `<div role="status">Automatic recovery stopped. <button>Retry</button></div>`,
+  "/issues/CHA-4": `<section><p>Couldn’t load the subtasks.</p><button onclick="window.reloaded=true">Retry</button></section>
+    <div role="status">Couldn’t finish this attempt. <button onclick="${retry("i4")}">Retry</button></div>`,
+  "/issues/CHA-5": `<div role="status">Couldn’t finish this attempt. <button onclick="${retry("other")}">Retry</button></div>`,
+};
+let server: Server;
+let baseUrl: string;
+const posts: string[] = [];
+
+beforeAll(async () => {
+  server = createServer((incoming, outgoing) => {
+    if (incoming.method === "POST") {
+      posts.push(incoming.url ?? "");
+      outgoing.writeHead(202, { "content-type": "application/json" }).end(JSON.stringify({ accepted: true }));
+      return;
+    }
+    const body = pages[incoming.url ?? ""];
+    outgoing.writeHead(body ? 200 : 404, { "content-type": "text/html; charset=utf-8" }).end(`<!doctype html><html><body>${body ?? ""}</body></html>`);
+  });
+  server.listen(0, "127.0.0.1"); await once(server, "listening");
+  baseUrl = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
+});
+afterAll(async () => { server.closeAllConnections(); server.close(); });
+
+async function withBoard<T>(run: (board: BoardUi) => Promise<T>): Promise<T> {
+  const board = new BoardUi(baseUrl);
+  try { return await run(board); } finally { await board.close(); }
+}
+
+describe("BoardUi.clickRetry", () => {
+  it("clicks the visible Retry button and reports the retry request it sent", async () => {
+    await withBoard(async board => {
+      expect(await board.clickRetry("CHA-1", "i1", 10_000)).toMatchObject({ clicked: true, httpStatus: 202, accepted: true, error: null,
+        bannerText: expect.stringContaining("Couldn’t finish this attempt") });
+    });
+    expect(posts).toContain("/api/issues/i1/retry");
+  }, 60_000);
+
+  it("reports a missing Retry button with the status text the user sees", async () => {
+    await withBoard(async board => {
+      expect(await board.clickRetry("CHA-2", "i2", 1_500)).toMatchObject({ clicked: false, error: UI_RETRY_BUTTON_MISSING,
+        bannerText: "Saving files from the previous attempt" });
+    });
+  }, 60_000);
+
+  it("reports a Retry button whose click sends no retry request", async () => {
+    await withBoard(async board => {
+      expect(await board.clickRetry("CHA-3", "i3", 1_500, 500)).toMatchObject({ clicked: true, httpStatus: null, error: UI_RETRY_NO_REQUEST });
+    });
+  }, 60_000);
+
+  it("prefers the recovery banner's Retry over another Retry earlier on the page", async () => {
+    await withBoard(async board => {
+      expect(await board.clickRetry("CHA-4", "i4", 10_000, 500)).toMatchObject({ clicked: true, httpStatus: 202, accepted: true });
+    });
+    expect(posts).toContain("/api/issues/i4/retry");
+  }, 60_000);
+
+  it("does not count a retry request for a different issue", async () => {
+    await withBoard(async board => {
+      expect(await board.clickRetry("CHA-5", "i5", 1_500, 500)).toMatchObject({ clicked: true, error: UI_RETRY_NO_REQUEST });
+    });
+  }, 60_000);
+
+  it("reports a browser that cannot launch and still closes cleanly", async () => {
+    const board = new BoardUi(baseUrl, () => Promise.reject(new Error("browser missing")));
+    expect(await board.clickRetry("CHA-1", "i1", 1_000)).toMatchObject({ clicked: false, error: "browser missing" });
+    await expect(board.close()).resolves.toBeUndefined();
+  });
+
+  it("matches retry requests by the issue's identifier or id only", () => {
+    expect(isRetryRequestFor(["CHA-1", "i1"], "POST", "http://x/api/issues/i1/retry")).toBe(true);
+    expect(isRetryRequestFor(["CHA-1", "i1"], "POST", "http://x/api/issues/CHA-1/retry")).toBe(true);
+    expect(isRetryRequestFor(["CHA-1", "i1"], "POST", "http://x/api/issues/i2/retry")).toBe(false);
+    expect(isRetryRequestFor(["CHA-1", "i1"], "GET", "http://x/api/issues/i1/retry")).toBe(false);
+  });
+});

tests/restart-chaos/board-ui.ts

@@ -0,0 +1,87 @@
+import { chromium, type Browser, type ElementHandle, type Page } from "@playwright/test";
+import { UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST } from "./types.js";
+
+export { UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST };
+
+export interface UiRetryClick {
+  clicked: boolean;
+  httpStatus: number | null;
+  accepted: boolean | null;
+  error: string | null;
+  bannerText: string | null;
+}
+
+const PER_BUTTON_REQUEST_MS = 5_000;
+
+export function isRetryRequestFor(issueRefs: readonly string[], method: string, url: string): boolean {
+  if (method !== "POST") return false;
+  const match = /\/api\/issues\/([^/]+)\/retry$/.exec(new URL(url).pathname);
+  return Boolean(match && issueRefs.includes(decodeURIComponent(match[1]!)));
+}
+
+async function retryButtonsBannerFirst(page: Page): Promise<ElementHandle[]> {
+  const candidates: Array<{ button: ElementHandle; inBanner: boolean }> = [];
+  for (const button of await page.getByRole("button", { name: "Retry", exact: true }).elementHandles()) {
+    if (!await button.isVisible().catch(() => false)) continue;
+    const inBanner = await button.evaluate(node => Boolean((node as Element).closest("[role=status], [role=alert]"))).catch(() => false);
+    candidates.push({ button, inBanner });
+  }
+  return [...candidates.filter(candidate => candidate.inBanner), ...candidates.filter(candidate => !candidate.inBanner)].map(candidate => candidate.button);
+}
+
+export async function clickRetryOnPage(page: Page, url: string, issueRefs: readonly string[], timeoutMs: number,
+  perButtonMs = PER_BUTTON_REQUEST_MS): Promise<UiRetryClick> {
+  await page.goto(url, { waitUntil: "domcontentloaded", timeout: timeoutMs });
+  const banner = async () => {
+    const texts = await page.locator("[role=status], [role=alert]").allInnerTexts().catch(() => []);
+    return texts.map(text => text.trim()).filter(Boolean).join(" | ").slice(0, 500) || null;
+  };
+  try {
+    await page.getByRole("button", { name: "Retry", exact: true }).first().waitFor({ state: "visible", timeout: timeoutMs });
+  } catch {
+    return { clicked: false, httpStatus: null, accepted: null, error: UI_RETRY_BUTTON_MISSING, bannerText: await banner() };
+  }
+  const bannerText = await banner();
+  for (const button of await retryButtonsBannerFirst(page)) {
+    const response = page.waitForResponse(candidate => isRetryRequestFor(issueRefs, candidate.request().method(), candidate.url()),
+      { timeout: perButtonMs }).catch(() => null);
+    await button.click().catch(() => undefined);
+    const answered = await response;
+    if (!answered) continue;
+    const body = await answered.json().catch(() => null) as { accepted?: unknown } | null;
+    return { clicked: true, httpStatus: answered.status(), accepted: answered.ok() ? body?.accepted === true : false,
+      error: answered.ok() ? null : `retry request failed with ${answered.status()}`, bannerText };
+  }
+  return { clicked: true, httpStatus: null, accepted: null, error: UI_RETRY_NO_REQUEST, bannerText };
+}
+
+export class BoardUi {
+  private browser: Promise<Browser> | null = null;
+
+  constructor(private readonly baseUrl: string, private readonly launch: () => Promise<Browser> = () => chromium.launch({ headless: true })) {}
+
+  async clickRetry(identifier: string, issueId: string = identifier, timeoutMs = 60_000, perButtonMs = PER_BUTTON_REQUEST_MS): Promise<UiRetryClick> {
+    let context: Awaited<ReturnType<Browser["newContext"]>> | null = null;
+    try {
+      this.browser ??= this.launch();
+      context = await (await this.browser).newContext({ viewport: { width: 1440, height: 1000 } });
+      return await clickRetryOnPage(await context.newPage(), `${this.baseUrl}/issues/${encodeURIComponent(identifier)}`,
+        [identifier, issueId], timeoutMs, perButtonMs);
+    } catch (error) {
+      return { clicked: false, httpStatus: null, accepted: null, error: error instanceof Error ? error.message : String(error), bannerText: null };
+    } finally {
+      await context?.close().catch(() => undefined);
+    }
+  }
+
+  async close(): Promise<void> {
+    const browser = this.browser;
+    this.browser = null;
+    if (!browser) return;
+    try {
+      await (await browser).close();
+    } catch {
+      return;
+    }
+  }
+}

tests/restart-chaos/config.ts

@@ -1,6 +1,7 @@
 import { execFileSync } from "node:child_process";
 import path from "node:path";
 import { parseArgs } from "node:util";
+import { STORAGE_FAULT_KINDS } from "./storage-chaos.js";
 import {
   ADAPTER_IDS, ENVIRONMENT_KINDS, FAULT_KINDS, TRIGGER_IDS, WORKLOAD_IDS,
   type AdapterId, type EnvironmentKind, type FaultKind, type TriggerId, type WaveOptions, type WorkloadId,
@@ -98,13 +99,16 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
       "crash-point-hold-ms": { type: "string" },
       "upgrade-from": { type: "string" },
       "workspace-repo": { type: "string" },
+      storage: { type: "string" },
+      "storage-faults": { type: "string" },
+      "storage-fault-seconds": { type: "string" },
     },
   });
   const keep = values.keep === true;
   const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000);
   const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000);
   if (values.replay !== undefined) {
-    for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo"] as const) {
+    for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds"] as const) {
       if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`);
     }
     return { mode: "replay", ref: parseReplayRef(values.replay), keep,
@@ -174,15 +178,22 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
     if (crashPoint) throw new Error("--upgrade-from does not combine with --crash-point");
     options.upgradeFrom = path.resolve(upgradeFrom);
   }
+  const storage = values.storage ?? "local";
   const workspaceRepo = values["workspace-repo"];
   if (options.workloads.includes("R1") !== (workspaceRepo !== undefined)) {
     throw new Error("R1 (code change in a cloned repository) and --workspace-repo <git repository> go together: pass both or neither");
   }
   if (workspaceRepo !== undefined) {
-    if (env !== "local") throw new Error("--workspace-repo runs only with --env local");
+    if (env !== "local" && storage !== "s3") throw new Error("--workspace-repo on Daytona needs --storage s3, so the clone reaches the sandbox as a remote workspace");
     if (options.workloads.some(id => id === "B1" || id === "B5")) throw new Error("--workspace-repo replaces the chaoscli fixture that B1 and B5 need");
     options.workspaceRepo = path.resolve(workspaceRepo);
-    options.workspaceCommit = execFileSync("git", ["-C", options.workspaceRepo, "rev-parse", "HEAD"], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
+    const git = (...args: string[]) => execFileSync("git", ["-C", options.workspaceRepo!, ...args], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
+    if (env === "daytona") {
+      options.workspaceGitUrl = publicGitHubUrl(git("remote", "get-url", "origin"));
+      options.workspaceCommit = git("rev-parse", "origin/master");
+    } else {
+      options.workspaceCommit = git("rev-parse", "HEAD");
+    }
   }
   // H1 pauses the whole company when its budget hard stop is crossed: no other task can share its
   // wave, its tasks run one at a time, and the hard stop must be on.
@@ -204,9 +215,28 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
         + "for chaos tasks. Lower one of them (RESTART_CHAOS_DAYTONA_VCPU_BUDGET sets the budget).");
     }
   }
+  if (storage !== "local" && storage !== "s3") throw new Error("--storage must be local or s3");
+  if (storage === "s3") options.storage = "s3";
+  if (values["storage-faults"] !== undefined) {
+    if (storage !== "s3") throw new Error("--storage-faults needs --storage s3");
+    options.storageFaults = list<"ST1" | "ST2" | "ST3">(values["storage-faults"], STORAGE_FAULT_KINDS, "storage-faults", []);
+  }
+  if (values["storage-fault-seconds"] !== undefined) {
+    if (!options.storageFaults?.length) throw new Error("--storage-fault-seconds needs --storage-faults");
+    const match = /^(\d+)-(\d+)$/.exec(values["storage-fault-seconds"]);
+    const [min, max] = match ? [Number(match[1]), Number(match[2])] : [0, -1];
+    if (!match || min < 5 || max < min || max > 900) throw new Error("--storage-fault-seconds takes <min>-<max> seconds, 5 to 900, min not above max");
+    options.storageFaultMs = [min * 1000, max * 1000];
+  }
   return { mode: "wave", options, keep };
 }
 
+export function publicGitHubUrl(remote: string): string {
+  const match = /^(?:https:\/\/github\.com\/|git@github\.com:|ssh:\/\/git@github\.com\/)([^/\s]+\/[^/\s]+?)(?:\.git)?\/?$/.exec(remote.trim());
+  if (!match) throw new Error(`--workspace-repo on Daytona needs a GitHub origin the sandbox can clone over https; origin is ${remote}`);
+  return `https://github.com/${match[1]}.git`;
+}
+
 export function waveIdFor(seed: number, at = new Date()): string {
   const stamp = at.toISOString().replace(/[-:]/g, "").replace(/\.\d{3}Z$/, "");
   return `w-${stamp}-s${seed}`;

tests/restart-chaos/grader.test.ts

@@ -1,5 +1,5 @@
 import { describe, expect, it } from "vitest";
-import { crossingFaults, gradeR3, gradeR4, gradeR5, gradeTask, STALL_LIMIT_MS } from "./grader.js";
+import { crossingFaults, gradeR2, gradeR3, gradeR4, gradeR5, gradeTask, STALL_LIMIT_MS } from "./grader.js";
 import { isResting } from "./observer.js";
 import type { FaultReceipt, OracleArtifacts, RunSample, TaskRecord, TaskSample } from "./types.js";
 import { BENCH_EXPECTED, parseBenchRecord, r1TestChecksSlug, stallWakeChecks, workloadDefinitions } from "./workloads.js";
@@ -238,6 +238,21 @@ describe("T-oracles", () => {
     expect(stallWakeChecks({ ...stalled, childStalls: [] } as never)).toEqual([]);
   });
 
+  it("grades a Retry the user could not click in the board UI under its own R3 signature", () => {
+    const record = { plan: { workload: "A1" }, launchError: null, finalStatus: "blocked", timedOut: false, createdAt: 0, restAt: 1,
+      samples: [{ at: 1, reachable: true, serverReady: true, issueStatus: "blocked", interruption: { state: "failed", canRetry: true } }],
+      retry: { at: 2, observedRunId: "r1", httpStatus: null, accepted: null, error: "ui_retry_button_missing", via: "ui", bannerText: "Saving files" } } as unknown as TaskRecord;
+    expect(gradeR3(record)).toEqual({ verdict: "fail", detail: "ui_retry_button_missing (user saw: Saving files)" });
+    expect(gradeR3({ ...record, retry: { ...record.retry!, clicked: true, error: "ui_retry_no_request" } } as TaskRecord).detail).toMatch(/^ui_retry_no_request/);
+  });
+
+  it("reports R2 as not checked for a sandbox-only workspace instead of passing it", () => {
+    const record = { plan: { workload: "R1" }, launchError: null, finalStatus: "done", timedOut: false, createdAt: 0, restAt: 1,
+      remoteWorkspace: true, preFaultFiles: null, samples: [{ at: 1, reachable: true, serverReady: true, issueStatus: "done", interruption: null }] } as unknown as TaskRecord;
+    expect(gradeR2(record, { verdict: "pass", checks: [] })).toEqual({ verdict: "n/a",
+      detail: "remote workspace: files before and after the fault are not compared from this host" });
+  });
+
   it("grades S1: stopped run cancelled, nothing ran before the continue comment, brief has both markers", () => {
     const base: OracleArtifacts = { issueStatus: "done", documents: { brief: "CHAOS-abcd1234\nCONTINUE-abcd1234\n- one\n- two\n- three\n" },
       agentComments: [], files: {}, commands: {}, workspacePath: null,

tests/restart-chaos/grader.ts

@@ -1,4 +1,4 @@
-import type { FaultReceipt, OracleResult, RuleId, RuleResult, TaskGrade, TaskRecord, TaskSample } from "./types.js";
+import { UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST, type FaultReceipt, type OracleResult, type RuleId, type RuleResult, type TaskGrade, type TaskRecord, type TaskSample } from "./types.js";
 import { g1Decision } from "./workloads.js";
 
 /** Pure R1–R6 grading over observer samples (plan §2). No I/O. */
@@ -89,6 +89,9 @@ export function gradeR2(record: TaskRecord, oracle: OracleResult): RuleResult {
   if (fileChecks.length && record.finalStatus === "done") {
     return { verdict: "fail", detail: `workspace_files_missing_after_done: ${fileChecks.map(check => check.name).join(",")}` };
   }
+  if (record.remoteWorkspace && !record.preFaultFiles) {
+    return { verdict: "n/a", detail: "remote workspace: files before and after the fault are not compared from this host" };
+  }
   const transient = record.samples.some(sample => sample.interruption?.workspaceFailure);
   return { verdict: "pass", detail: transient ? "workspace failure appeared during recovery and cleared" : "no workspace failure at rest" };
 }
@@ -108,6 +111,9 @@ function gradeOwnRetry(record: TaskRecord): RuleResult {
     }
     return { verdict: "n/a", detail: "no Retry needed" };
   }
+  if (record.retry.via === "ui" && (record.retry.error === UI_RETRY_BUTTON_MISSING || record.retry.error === UI_RETRY_NO_REQUEST)) {
+    return { verdict: "fail", detail: `${record.retry.error}${record.retry.bannerText ? ` (user saw: ${record.retry.bannerText})` : ""}` };
+  }
   if (record.retry.error || record.retry.accepted !== true) {
     return { verdict: "fail", detail: `retry_rejected: http=${record.retry.httpStatus ?? "none"} accepted=${String(record.retry.accepted)} ${record.retry.error ?? ""}`.trim() };
   }

tests/restart-chaos/main.ts

@@ -13,6 +13,10 @@ import os from "node:os";
 import path from "node:path";
 import { setTimeout as delay } from "node:timers/promises";
 import { ChaosApi, isSocketTypeOfServiceRace } from "./api.js";
+import { BoardUi } from "./board-ui.js";
+import { DEFAULT_MINIO_BINARY, MINIO_RELEASE, minioBinary, objectStoreServerEnvironment, startMinio, type ObjectStore } from "./object-store.js";
+import { StorageChaos, planStorageFaults, replayStorageWindows, type StorageFaultReceipt } from "./storage-chaos.js";
+import { startStorageFaultProxy } from "./storage-proxy.js";
 import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from "./config.js";
 import { assertControllerMatchesImage, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js";
 import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js";
@@ -55,6 +59,11 @@ async function loadPlan(): Promise<{ plan: WavePlan; keep: boolean }> {
     const observed = crossed ? observedFaultSchedule(crossed.createdAt, crossed.faultIds, faults, command.ref.taskId) : [];
     if (!observed.length) console.warn("Replay: no recorded fault sequence for this task; using its planned fault only.");
     let plan = replayWave(source, command.ref, Date.now(), command.overrides, observed);
+    if (source.options.storage === "s3" && crossed?.createdAt) {
+      const receipts = await readFile(path.join(sourceDir, "storage-faults.json"), "utf8").then(text => JSON.parse(text) as StorageFaultReceipt[]).catch(() => []);
+      const ended = Math.max(crossed.createdAt, ...receipts.map(receipt => receipt.endedAt ?? 0));
+      plan = { ...plan, options: { ...plan.options, storageWindows: replayStorageWindows(receipts, crossed.createdAt, ended) } };
+    }
     // K14: a task that launched after the upgrade replays on the current build.
     const launchBuild = graded?.record.serverBuildAtLaunch ?? logged?.[command.ref.taskId]?.serverBuildAtLaunch;
     if (plan.options.upgradeFrom && launchBuild === "current") plan = replayOnCurrentBuild(plan);
@@ -211,6 +220,10 @@ async function main(): Promise<number> {
   let anyFailure = false;
   let stopFailed = false;
   let remoteCleanupPending = false;
+  let board: BoardUi | null = null;
+  let objectStore: ObjectStore | null = null;
+  let storageChaos: StorageChaos | null = null;
+  let storageProxy: Awaited<ReturnType<typeof startStorageFaultProxy>> | null = null;
   let speedSection: string | null = null;
   let harnessError: unknown;
   process.on("uncaughtException", error => {
@@ -237,6 +250,14 @@ async function main(): Promise<number> {
   let faultFailures = 0;
   let faultsQueued = 0;
   try {
+    if (plan.options.storage === "s3") {
+      const binary = minioBinary();
+      if (!binary) throw new Error(`--storage s3 needs MinIO ${MINIO_RELEASE} at ${DEFAULT_MINIO_BINARY} or RESTART_CHAOS_MINIO_BIN`);
+      objectStore = await startMinio({ binary, root: supervisor.paths.temporaryRoot });
+      storageProxy = await startStorageFaultProxy(objectStore.endpoint);
+      Object.assign(serverEnvironment, objectStoreServerEnvironment(objectStore, storageProxy.endpoint, `chaos/${plan.waveId}/`));
+      timeline.add("object_store_started", { release: MINIO_RELEASE, endpoint: objectStore.endpoint, proxy: storageProxy.endpoint, bucket: objectStore.bucket });
+    }
     await supervisor.start(serverEnvironment);
     const api = new ChaosApi(supervisor.baseUrl);
     const lab = await seedChaosLab({ api, env: plan.options.env, waitForReadyAgents: true, cancelled: () => interrupted, daytonaImage: plan.options.daytonaImage, daytonaCpu: plan.options.daytonaCpu,
@@ -245,7 +266,7 @@ async function main(): Promise<number> {
         ...(plan.tasks.some(task => task.workload === "M1") ? plan.options.adapters : [])])],
       concurrency: plan.options.concurrency, budgetCents: plan.options.budgetCents, masks: plan.options.masks,
       fixtureRoot: path.join(supervisor.paths.temporaryRoot, "fixtures"), workspaceRepo: plan.options.workspaceRepo,
-      workspaceCommit: plan.options.workspaceCommit, waveId: plan.waveId, timeline,
+      workspaceCommit: plan.options.workspaceCommit, workspaceGitUrl: plan.options.workspaceGitUrl, waveId: plan.waveId, timeline,
       adapterConfigOverrides: chaosLifecycleOverrides(plan.options.lifecycle),
       onEnvironment: owned => {
         sweeper = new DaytonaWaveSweeper({ apiKey: process.env.DAYTONA_API_KEY!, companyId: owned.companyId, environmentId: owned.environmentId, api });
@@ -253,7 +274,8 @@ async function main(): Promise<number> {
     spendTarget = { api, companyId: lab.companyId };
     // The instance root holds the secrets master key and the database: oracle code may not read it.
     const protectedRoots = [os.homedir(), supervisor.paths.temporaryRoot, resultsRoot];
-    const observers = plan.tasks.map(task => new TaskObserver(task, api, lab, timeline, supervisor, protectedRoots));
+    board = serverEnvironment.SERVE_UI === "false" ? null : new BoardUi(supervisor.baseUrl);
+    const observers = plan.tasks.map(task => new TaskObserver(task, api, lab, timeline, supervisor, protectedRoots, board));
     const pending = [...observers];
     const launched: TaskObserver[] = [];
     const active: TaskObserver[] = [];
@@ -261,6 +283,13 @@ async function main(): Promise<number> {
     let lastDoneAt = Date.now();
     // K13: storm faults fire on their own clock while tasks keep launching; then the storm drains.
     const storm = plan.storm ? new StormRun(plan.storm, Date.now()) : null;
+    if (storageProxy && (plan.options.storageWindows?.length || plan.options.storageFaults?.length)) {
+      const windows = plan.options.storageWindows ?? planStorageFaults({ kinds: plan.options.storageFaults!, seed: plan.seed,
+        spanMs: plan.storm?.durationMs ?? plan.tasks.length * 4 * 60_000, ...(plan.options.storageFaultMs ? { durationMs: plan.options.storageFaultMs } : {}) });
+      storageChaos = new StorageChaos(windows, Date.now(), storageProxy, (kind, data) => timeline.add(kind, data));
+      if (!plan.options.storageWindows) storageChaos.start();
+      timeline.add("storage_faults_planned", { windows: windows.map(window => `${window.kind}@${window.startMs}ms+${window.durationMs}ms`) });
+    }
     if (storm) timeline.add("storm_started", { durationMs: storm.plan.durationMs, plannedFaults: storm.plan.faults.length });
     // Faults are serialized; each one hits every task in flight when it starts.
     const fireFault = (kind: FaultKind, holdMs: number, requestedBy: string, exposed: readonly TaskObserver[],
@@ -373,6 +402,7 @@ async function main(): Promise<number> {
         const next = pending.shift()!;
         launched.push(next);
         await next.launch();
+        if (plan.options.storageWindows && storageChaos && launched.length === 1) storageChaos.start();
         if (!next.done) active.push(next);
       }
       await Promise.all(active.filter(observer => now - (lastPoll.get(observer) ?? 0) >= plan.options.pollMs).map(async observer => {
@@ -518,6 +548,7 @@ async function main(): Promise<number> {
   } finally {
     // Best effort for an interrupted or failed wave; the server may already be down.
     if (supervisor.serverReady) await recordWaveSpend().catch(() => undefined);
+    await board?.close().catch(() => undefined);
     // A failure here (for example writing the report) must not skip stopping the controller and Postgres.
     try { if (sweeper) try {
       const owned = sweeper as DaytonaWaveSweeper;
@@ -556,6 +587,13 @@ async function main(): Promise<number> {
         timeline.add("supervisor_stop_failed", { error: String(error) });
       }
     }
+    storageChaos?.stop();
+    if (storageProxy) {
+      if (storageChaos) await writeFile(path.join(waveDir, "storage-faults.json"), `${JSON.stringify(storageChaos.receipts, null, 2)}\n`, { mode: 0o600 }).catch(() => undefined);
+      await writeFile(path.join(waveDir, "storage-proxy.json"), `${JSON.stringify(storageProxy.counts(), null, 2)}\n`, { mode: 0o600 }).catch(() => undefined);
+      await storageProxy.close().catch(() => undefined);
+    }
+    await objectStore?.stop().catch(error => timeline.add("object_store_stop_failed", { error: String(error) }));
     const retention = await releaseInstance(supervisor.paths.temporaryRoot, { keep, stopFailed, remoteCleanupPending, owner: plan.waveId })
       .catch(error => { timeline.add("instance_cleanup_failed", { error: String(error) }); return null; });
     if (retention?.kept) timeline.add("instance_kept", { temporaryRoot: supervisor.paths.temporaryRoot, reason: retention.reason });

tests/restart-chaos/object-store.test.ts

@@ -0,0 +1,49 @@
+import { mkdtemp, rm } from "node:fs/promises";
+import { createRequire } from "node:module";
+import os from "node:os";
+import path from "node:path";
+import { describe, expect, it } from "vitest";
+import { minioBinary, objectStoreServerEnvironment, startMinio } from "./object-store.js";
+import { startStorageFaultProxy } from "./storage-proxy.js";
+
+const binary = minioBinary();
+
+describe("object store", () => {
+  it("points the server's storage settings at the given endpoint, path-style", () => {
+    expect(objectStoreServerEnvironment({ bucket: "b", region: "us-east-1", accessKeyId: "k", secretAccessKey: "s" }, "http://127.0.0.1:9", "chaos/w1/"))
+      .toEqual({ PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: "b", PAPERCLIP_STORAGE_S3_REGION: "us-east-1",
+        PAPERCLIP_STORAGE_S3_ENDPOINT: "http://127.0.0.1:9", PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE: "true", PAPERCLIP_STORAGE_S3_PREFIX: "chaos/w1/",
+        AWS_ACCESS_KEY_ID: "k", AWS_SECRET_ACCESS_KEY: "s", AWS_SESSION_TOKEN: "", AWS_CREDENTIAL_EXPIRATION: "", AWS_EC2_METADATA_DISABLED: "true" });
+  });
+
+  it("fails fast with the binary's path when MinIO cannot start", async () => {
+    const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-minio-bad-"));
+    try {
+      const started = Date.now();
+      await expect(startMinio({ binary: path.join(root, "not-a-binary"), root, readyTimeoutMs: 20_000 })).rejects.toThrow(/could not start from .*not-a-binary/);
+      expect(Date.now() - started).toBeLessThan(5_000);
+    } finally { await rm(root, { recursive: true, force: true }); }
+  });
+
+  it.skipIf(!binary)("starts a loopback MinIO that honours conditional writes through the fault proxy", async () => {
+    const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-minio-"));
+    const store = await startMinio({ binary: binary!, root });
+    const proxy = await startStorageFaultProxy(store.endpoint);
+    const { S3Client, PutObjectCommand } = createRequire(new URL("../../server/package.json", import.meta.url))("@aws-sdk/client-s3");
+    const client = new S3Client({ endpoint: proxy.endpoint, region: store.region, forcePathStyle: true,
+      credentials: { accessKeyId: store.accessKeyId, secretAccessKey: store.secretAccessKey } });
+    try {
+      await client.send(new PutObjectCommand({ Bucket: store.bucket, Key: "a", Body: "1", IfNoneMatch: "*" }));
+      await expect(client.send(new PutObjectCommand({ Bucket: store.bucket, Key: "a", Body: "2", IfNoneMatch: "*" })))
+        .rejects.toMatchObject({ $metadata: { httpStatusCode: 412 } });
+      proxy.setRules([{ fault: { kind: "outage" }, methods: ["PUT"], remaining: 10 }]);
+      await expect(client.send(new PutObjectCommand({ Bucket: store.bucket, Key: "b", Body: "3" }))).rejects.toMatchObject({ $metadata: { httpStatusCode: 503 } });
+      expect(proxy.counts().injected).toBeGreaterThanOrEqual(1);
+    } finally {
+      client.destroy();
+      await proxy.close();
+      await store.stop();
+      await rm(root, { recursive: true, force: true });
+    }
+  }, 60_000);
+});

tests/restart-chaos/object-store.ts

@@ -0,0 +1,98 @@
+import { spawn, type ChildProcess } from "node:child_process";
+import { randomBytes } from "node:crypto";
+import { existsSync } from "node:fs";
+import { mkdir } from "node:fs/promises";
+import { createRequire } from "node:module";
+import { createServer } from "node:net";
+import os from "node:os";
+import path from "node:path";
+import { setTimeout as delay } from "node:timers/promises";
+
+export const MINIO_RELEASE = "RELEASE.2025-04-22T22-12-26Z";
+export const DEFAULT_MINIO_BINARY = path.join(os.homedir(), ".cache", "paperclip", "tools", "minio", "minio");
+
+export function minioBinary(environment: NodeJS.ProcessEnv = process.env): string | null {
+  const candidate = environment.RESTART_CHAOS_MINIO_BIN || DEFAULT_MINIO_BINARY;
+  return existsSync(candidate) ? candidate : null;
+}
+
+async function freeLoopbackPort(): Promise<number> {
+  const server = createServer();
+  await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
+  const port = (server.address() as { port: number }).port;
+  await new Promise<void>(resolve => server.close(() => resolve()));
+  return port;
+}
+
+export interface ObjectStore {
+  endpoint: string;
+  bucket: string;
+  region: string;
+  accessKeyId: string;
+  secretAccessKey: string;
+  dataDir: string;
+  stop(): Promise<void>;
+}
+
+export function objectStoreServerEnvironment(store: Pick<ObjectStore, "bucket" | "region" | "accessKeyId" | "secretAccessKey">,
+  endpoint: string, prefix: string): Record<string, string> {
+  return {
+    PAPERCLIP_STORAGE_PROVIDER: "s3",
+    PAPERCLIP_STORAGE_S3_BUCKET: store.bucket,
+    PAPERCLIP_STORAGE_S3_REGION: store.region,
+    PAPERCLIP_STORAGE_S3_ENDPOINT: endpoint,
+    PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE: "true",
+    PAPERCLIP_STORAGE_S3_PREFIX: prefix,
+    AWS_ACCESS_KEY_ID: store.accessKeyId,
+    AWS_SECRET_ACCESS_KEY: store.secretAccessKey,
+    AWS_SESSION_TOKEN: "",
+    AWS_CREDENTIAL_EXPIRATION: "",
+    AWS_EC2_METADATA_DISABLED: "true",
+  };
+}
+
+export async function startMinio(input: { binary: string; root: string; bucket?: string; readyTimeoutMs?: number }): Promise<ObjectStore> {
+  const dataDir = path.join(input.root, "object-store");
+  await mkdir(dataDir, { recursive: true, mode: 0o700 });
+  const [port, consolePort] = [await freeLoopbackPort(), await freeLoopbackPort()];
+  const accessKeyId = `chaos${randomBytes(6).toString("hex")}`;
+  const secretAccessKey = randomBytes(24).toString("hex");
+  const child: ChildProcess = spawn(input.binary, ["server", dataDir, "--address", `127.0.0.1:${port}`, "--console-address", `127.0.0.1:${consolePort}`, "--quiet"], {
+    env: { PATH: process.env.PATH ?? "", HOME: input.root, MINIO_ROOT_USER: accessKeyId, MINIO_ROOT_PASSWORD: secretAccessKey,
+      MINIO_BROWSER: "off", MINIO_UPDATE: "off" },
+    stdio: ["ignore", "ignore", "pipe"],
+  });
+  let stderr = "";
+  let spawnError: Error | null = null;
+  child.on("error", error => { spawnError = error; });
+  child.stderr?.on("data", chunk => { stderr = `${stderr}${chunk}`.slice(-4_000); });
+  const endpoint = `http://127.0.0.1:${port}`;
+  const stop = async () => {
+    if (spawnError || child.pid === undefined || child.exitCode !== null || child.signalCode !== null) return;
+    const exited = new Promise<void>(resolve => child.once("exit", () => resolve()));
+    child.kill("SIGTERM");
+    await Promise.race([exited, delay(10_000).then(() => { child.kill("SIGKILL"); return exited; })]);
+  };
+  const deadline = Date.now() + (input.readyTimeoutMs ?? 30_000);
+  for (;;) {
+    if (spawnError) throw new Error(`MinIO could not start from ${input.binary}: ${(spawnError as Error).message}`);
+    if (child.exitCode !== null) throw new Error(`MinIO exited before it was ready (${child.exitCode}): ${stderr}`);
+    const live = await fetch(`${endpoint}/minio/health/live`).then(response => response.ok).catch(() => false);
+    if (live) break;
+    if (Date.now() > deadline) { await stop(); throw new Error(`MinIO did not become ready: ${stderr}`); }
+    await delay(200);
+  }
+  const bucket = input.bucket ?? `chaos-${randomBytes(4).toString("hex")}`;
+  const region = "us-east-1";
+  const { S3Client, CreateBucketCommand } = createRequire(new URL("../../server/package.json", import.meta.url))("@aws-sdk/client-s3");
+  const client = new S3Client({ endpoint, region, forcePathStyle: true, credentials: { accessKeyId, secretAccessKey } });
+  try {
+    await client.send(new CreateBucketCommand({ Bucket: bucket }));
+  } catch (error) {
+    await stop();
+    throw error;
+  } finally {
+    client.destroy();
+  }
+  return { endpoint, bucket, region, accessKeyId, secretAccessKey, dataDir, stop };
+}

tests/restart-chaos/observer.ts

@@ -1,6 +1,6 @@
 import { spawnSync } from "node:child_process";
 import { randomUUID } from "node:crypto";
-import { readFile, readdir } from "node:fs/promises";
+import { access, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
 import os from "node:os";
 import path from "node:path";
 import { setTimeout as delay } from "node:timers/promises";
@@ -13,6 +13,7 @@ import type {
   TaskSample,
 } from "./types.js";
 import { restartProcessStartMatches } from "../runner-e2e/process-tree-owner.js";
+import type { UiRetryClick } from "./board-ui.js";
 import { runOracleCommand } from "./oracle-sandbox.js";
 import { taskFaultSchedule } from "./scheduler.js";
 import { g1ChildTitle, g1Decision, workloadDefinitions, type WorkloadAgents } from "./workloads.js";
@@ -192,7 +193,8 @@ export class TaskObserver {
     private readonly timeline: Timeline,
     private readonly supervisor: { serverReady: boolean; faultInProgress: boolean; serverBuild?: "upgrade_from" | "current" },
     /** Paths oracle commands may not read: the operator's home, the instance root (secrets key, database) and results. */
-    private readonly protectedRoots: readonly string[] = [os.homedir()]) {
+    private readonly protectedRoots: readonly string[] = [os.homedir()],
+    private readonly board: { clickRetry(identifier: string, issueId: string): Promise<UiRetryClick> } | null = null) {
     this.record = { plan, issueId: null, identifier: null, createdAt: null, launchError: null, samples: [], retry: null,
       restAt: null, finalStatus: null, deadlineAt: null, timedOut: false, followUp: null, faultIds: [], sweep: null,
       preFaultFiles: null, lostFiles: null };
@@ -394,7 +396,11 @@ export class TaskObserver {
     // One task's snapshot failure must not end the wave; R2 fails with the reason.
     try {
       const workspace = await this.workspacePath();
-      if (workspace) { this.record.preFaultFiles = await listWorkspaceFiles(workspace); this.record.preFaultWorkspacePath = workspace; }
+      if (workspace && this.lab.environmentKind !== "local" && !await access(workspace).then(() => true, () => false)) {
+        this.record.remoteWorkspace = true;
+      } else if (workspace) {
+        this.record.preFaultFiles = await listWorkspaceFiles(workspace); this.record.preFaultWorkspacePath = workspace;
+      }
     } catch (error) {
       this.record.preFaultSnapshotError = error instanceof Error ? error.message : String(error);
       this.timeline.add("pre_fault_snapshot_failed", { taskId: this.plan.taskId, error: this.record.preFaultSnapshotError });
@@ -422,6 +428,7 @@ export class TaskObserver {
     if (this.done || !this.record.issueId) return;
     const sample = await this.sample(now);
     this.record.samples.push(sample);
+    await this.captureRemoteFilesAtDone(sample);
     this.readySince = sample.reachable && sample.serverReady ? (this.readySince ?? now) : null;
     await this.observeBenchProgress();
     await this.governanceStep(sample, now);
@@ -637,16 +644,8 @@ export class TaskObserver {
       const interruption = child?.interruption as Json | null | undefined;
       if (interruption?.canRetry !== true) continue;
       const observedRunId = typeof interruption.observedRunId === "string" ? interruption.observedRunId : null;
-      const attempt: RetryAttempt & { childIssueId: string } = { childIssueId, at: now, observedRunId, httpStatus: null, accepted: null, error: null };
-      try {
-        const result = await this.api.post<{ accepted?: boolean }>(`/api/issues/${childIssueId}/retry`, { idempotencyKey: randomUUID(), observedRunId });
-        attempt.httpStatus = 200;
-        attempt.accepted = result?.accepted === true;
-        attempt.acceptedAt = Math.max(now, Date.now());
-      } catch (error) {
-        attempt.httpStatus = error instanceof ChaosApiError ? error.status : null;
-        attempt.error = error instanceof Error ? error.message : String(error);
-      }
+      const childRef = typeof child?.identifier === "string" ? child.identifier : childIssueId;
+      const attempt: RetryAttempt & { childIssueId: string } = { childIssueId, ...await this.clickRetry(childRef, childIssueId, observedRunId, now) };
       (this.record.childRetries ??= []).push(attempt);
       this.timeline.add("task_child_retry_clicked", { taskId: this.plan.taskId, ...attempt });
       clicked = clicked || attempt.accepted === true;
@@ -665,10 +664,22 @@ export class TaskObserver {
 
   private async retry(sample: TaskSample, now: number): Promise<RetryAttempt> {
     const observedRunId = sample.interruption?.observedRunId ?? (this.issue?.executionBlocker?.runId as string | undefined) ?? null;
+    const attempt = await this.clickRetry(this.record.identifier ?? this.record.issueId!, this.record.issueId!, observedRunId, now);
+    this.timeline.add("task_retry_clicked", { taskId: this.plan.taskId, ...attempt });
+    return attempt;
+  }
+
+  private async clickRetry(issueRef: string, issueId: string, observedRunId: string | null, now: number): Promise<RetryAttempt> {
     const attempt: RetryAttempt = { at: now, observedRunId, httpStatus: null, accepted: null, error: null };
+    if (this.board) {
+      const click = await this.board.clickRetry(issueRef, issueId);
+      Object.assign(attempt, { via: "ui", httpStatus: click.httpStatus, accepted: click.accepted, error: click.error, bannerText: click.bannerText });
+      if (click.accepted) attempt.acceptedAt = Math.max(now, Date.now());
+      return attempt;
+    }
+    attempt.via = "api";
     try {
-      const result = await this.api.post<{ accepted?: boolean }>(`/api/issues/${this.record.issueId}/retry`,
-        { idempotencyKey: randomUUID(), observedRunId });
+      const result = await this.api.post<{ accepted?: boolean }>(`/api/issues/${issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId });
       attempt.httpStatus = 200;
       attempt.accepted = result?.accepted === true;
       attempt.acceptedAt = Math.max(now, Date.now());
@@ -676,10 +687,27 @@ export class TaskObserver {
       attempt.httpStatus = error instanceof ChaosApiError ? error.status : null;
       attempt.error = error instanceof Error ? error.message : String(error);
     }
-    this.timeline.add("task_retry_clicked", { taskId: this.plan.taskId, ...attempt });
     return attempt;
   }
 
+  private async remoteWorkspaceFile(issueId: string, relative: string): Promise<string | null> {
+    const captured = this.record.remoteFiles?.[relative];
+    if (typeof captured === "string") return captured;
+    return await this.api.getText(`/api/issues/${issueId}/file-resources/content?path=${encodeURIComponent(relative)}&download=true`, 15_000)
+      .catch(() => null);
+  }
+
+  private async captureRemoteFilesAtDone(sample: TaskSample): Promise<void> {
+    const wanted = this.definition.collect.files(this.plan.nonce);
+    if (!wanted.length || this.lab.environmentKind === "local" || sample.issueStatus !== "done" || !this.record.issueId) return;
+    const captured = this.record.remoteFiles ??= {};
+    for (const relative of wanted) {
+      if (typeof captured[relative] === "string") continue;
+      captured[relative] = await this.api.getText(`/api/issues/${this.record.issueId}/file-resources/content?path=${encodeURIComponent(relative)}&download=true`, 15_000)
+        .catch(() => null);
+    }
+  }
+
   async workspacePath(): Promise<string | null> {
     if (!this.definition.usesRepository) return null;
     const current = this.issue?.currentExecutionWorkspace as Json | null | undefined;
@@ -876,16 +904,30 @@ export class TaskObserver {
     }
     const files: Record<string, string | null> = {};
     const commands: OracleArtifacts["commands"] = {};
-    if (workspacePath) {
-      for (const relative of definition.collect.files(this.plan.nonce)) {
-        files[relative] = await readFile(path.join(workspacePath, relative), "utf8").catch(() => null);
-      }
-      // Agent-editable code: run it sandboxed, away from the operator's home directory.
+    const runCommands = async (cwd: string) => {
       for (const command of definition.collect.commands) {
         const argv = typeof command.argv === "function" ? command.argv(this.plan.nonce) : command.argv;
-        commands[command.name] = await runOracleCommand(argv, workspacePath, { protectedRoots: this.protectedRoots,
+        commands[command.name] = await runOracleCommand(argv, cwd, { protectedRoots: this.protectedRoots,
           readableFiles: argv[0] === process.execPath ? [process.execPath] : [] });
       }
+    };
+    const wanted = definition.collect.files(this.plan.nonce);
+    if (workspacePath && await access(workspacePath).then(() => true, () => false)) {
+      for (const relative of wanted) files[relative] = await readFile(path.join(workspacePath, relative), "utf8").catch(() => null);
+      await runCommands(workspacePath);
+    } else if (workspacePath && wanted.length) {
+      const staged = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-oracle-"));
+      try {
+        for (const relative of wanted) {
+          files[relative] = await this.remoteWorkspaceFile(issueId, relative);
+          if (files[relative] === null) continue;
+          await mkdir(path.dirname(path.join(staged, relative)), { recursive: true });
+          await writeFile(path.join(staged, relative), files[relative]!);
+        }
+        await runCommands(staged);
+      } finally {
+        await rm(staged, { recursive: true, force: true });
+      }
     }
     return {
       issueStatus: issue?.status ?? this.record.finalStatus,

tests/restart-chaos/remote-oracle.test.ts

@@ -0,0 +1,109 @@
+import { createServer, type Server } from "node:http";
+import { once } from "node:events";
+import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import { ChaosApi } from "./api.js";
+import { TaskObserver } from "./observer.js";
+import type { ChaosLab } from "./seed.js";
+import { Timeline } from "./timeline.js";
+import type { TaskPlan } from "./types.js";
+
+const nonce = "abcd1234";
+const modulePath = `packages/shared/src/chaos/slug-${nonce}.ts`;
+const testPath = `packages/shared/src/chaos/slug-${nonce}.test.ts`;
+const remoteFiles: Record<string, string> = {
+  [modulePath]: `export function chaosSlug(input: string): string {
+  return input.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "");
+}
+`,
+  [testPath]: `import { test } from "node:test";
+import assert from "node:assert/strict";
+import { chaosSlug } from "./slug-${nonce}.ts";
+test("slug", () => {
+  assert.equal(chaosSlug("Hello Chaos World"), "hello-chaos-world");
+  assert.equal(chaosSlug("  a__b  "), "a-b");
+  assert.equal(chaosSlug("x"), "x");
+});
+`,
+};
+const lab = { companyId: "c1", projectId: "p1", environmentId: "e1", environmentKind: "daytona", fixtureRepositoryPath: "/x",
+  agents: { codex: { id: "a1", name: "Chaos codex" }, claude: { id: "a2", name: "Chaos claude" } } } as unknown as ChaosLab;
+const plan: TaskPlan = { taskId: "t01", index: 0, workload: "R1", adapter: "codex", fault: "K1", fireAfterMs: 999_999,
+  holdMs: 0, followUpAfterMs: null, nonce };
+let server: Server;
+let sandboxReleased = false;
+let baseUrl: string;
+
+beforeAll(async () => {
+  server = createServer((incoming, outgoing) => {
+    const url = new URL(incoming.url ?? "/", "http://x");
+    if (url.pathname === "/api/issues/i1/file-resources/content" && url.searchParams.get("download") === "true") {
+      if (incoming.headers.origin !== baseUrl) { outgoing.writeHead(403).end(); return; }
+      if (sandboxReleased) { outgoing.writeHead(422, { "content-type": "application/json" }).end(JSON.stringify({ code: "workspace_unavailable" })); return; }
+      const body = remoteFiles[url.searchParams.get("path") ?? ""];
+      outgoing.writeHead(body ? 200 : 404, { "content-type": "text/plain" }).end(body ?? "");
+      return;
+    }
+    outgoing.writeHead(404).end();
+  });
+  server.listen(0, "127.0.0.1"); await once(server, "listening");
+  baseUrl = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
+});
+afterAll(async () => { server.closeAllConnections(); server.close(); });
+
+describe("oracle for a remote workspace", () => {
+  it("reads the task's files through the board API and runs the oracle commands on a local copy", async () => {
+    const api = {
+      baseUrl,
+      async get(route: string) {
+        if (route.endsWith("/comments")) return [];
+        return { id: "i1", status: "done", currentExecutionWorkspace: { cwd: "/home/daytona/workspaces/not-on-this-host" } };
+      },
+      async post() { return { id: "i1", identifier: "CHA-1" }; },
+      getText: (route: string) => new ChaosApi(baseUrl).getText(route),
+    } as unknown as ChaosApi;
+    const observer = new TaskObserver(plan, api, lab, new Timeline(null), { serverReady: true, faultInProgress: false });
+    await observer.launch();
+    sandboxReleased = false;
+    const artifacts = await observer.collectArtifacts();
+    expect(artifacts.files).toEqual(remoteFiles);
+    expect(artifacts.commands.node_test).toMatchObject({ exitCode: 0 });
+    expect(artifacts.commands.slug?.stdout).toContain("hello-chaos-world");
+  }, 60_000);
+
+  it("skips the local pre-fault listing for a workspace that lives in the sandbox, instead of failing R2", async () => {
+    const api = {
+      baseUrl,
+      async get(route: string) { return route.endsWith("/runs") ? [] : { id: "i1", status: "in_progress", currentExecutionWorkspace: { cwd: "/home/daytona/workspaces/not-on-this-host" } }; },
+      async post() { return { id: "i1", identifier: "CHA-1" }; },
+    } as unknown as ChaosApi;
+    const observer = new TaskObserver(plan, api, lab, new Timeline(null), { serverReady: true, faultInProgress: false });
+    await observer.launch();
+    await observer.poll(observer.record.createdAt! + 1_000);
+    await observer.snapshotBeforeFault();
+    expect(observer.record).toMatchObject({ preFaultFiles: null, remoteWorkspace: true });
+    expect(observer.record.preFaultSnapshotError).toBeUndefined();
+  });
+
+  it("captures the task's files when it first sees the task done, before the sandbox is released", async () => {
+    const api = {
+      baseUrl,
+      async get(route: string) {
+        if (route.endsWith("/runs") || route.endsWith("/comments")) return [];
+        return { id: "i1", status: "done", currentExecutionWorkspace: { cwd: "/home/daytona/workspaces/not-on-this-host" } };
+      },
+      async post() { return { id: "i1", identifier: "CHA-1" }; },
+      getText: (route: string) => new ChaosApi(baseUrl).getText(route),
+    } as unknown as ChaosApi;
+    const observer = new TaskObserver(plan, api, lab, new Timeline(null), { serverReady: true, faultInProgress: false });
+    await observer.launch();
+    sandboxReleased = false;
+    await observer.poll(observer.record.createdAt! + 1_000);
+    sandboxReleased = true;
+    try {
+      const artifacts = await observer.collectArtifacts();
+      expect(artifacts.files).toEqual(remoteFiles);
+      expect(artifacts.commands.node_test).toMatchObject({ exitCode: 0 });
+    } finally { sandboxReleased = false; }
+  }, 60_000);
+});
+

tests/restart-chaos/scheduler.test.ts

@@ -1,5 +1,9 @@
+import { mkdtempSync, rmSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { execFileSync } from "node:child_process";
 import { describe, expect, it } from "vitest";
-import { parseCommand, parseReplayRef, waveIdFor } from "./config.js";
+import { parseCommand, parseReplayRef, waveIdFor, publicGitHubUrl } from "./config.js";
 import { chaosLifecycleOverrides } from "./seed.js";
 import { K4_HOLD_RANGE_MS, STORM_GAP_RANGE_MS, createRng, observedFaultSchedule, planStorm, planTasks, planWave, replayWave,
   StormRun, stormActive, stormFaultDue, taskFaultSchedule, type StormState } from "./scheduler.js";
@@ -47,6 +51,37 @@ describe("scheduler determinism", () => {
     expect(single.options.workloads).not.toContain("M1");
     expect(both.options.workloads).toContain("M1");
     expect(() => parseCommand(["--adapters", "codex", "--workloads", "M1"])).toThrow(/two agents/);
+    const stored = parseCommand(["--storage", "s3", "--seed", "5"]);
+    if (stored.mode !== "wave") throw new Error("expected a wave");
+    expect(stored.options.storage).toBe("s3");
+    expect(both.options.storage).toBeUndefined();
+    expect(() => parseCommand(["--storage", "gcs"])).toThrow(/local or s3/);
+    const faulted = parseCommand(["--storage", "s3", "--storage-faults", "ST1,ST3", "--seed", "5"]);
+    if (faulted.mode !== "wave") throw new Error("expected a wave");
+    expect(faulted.options.storageFaults).toEqual(["ST1", "ST3"]);
+    expect(() => parseCommand(["--storage-faults", "ST1"])).toThrow(/needs --storage s3/);
+    const short = parseCommand(["--storage", "s3", "--storage-faults", "ST1", "--storage-fault-seconds", "15-45", "--seed", "5"]);
+    if (short.mode !== "wave") throw new Error("expected a wave");
+    expect(short.options.storageFaultMs).toEqual([15_000, 45_000]);
+    expect(() => parseCommand(["--storage", "s3", "--storage-faults", "ST1", "--storage-fault-seconds", "45-15"])).toThrow(/storage-fault-seconds/);
+    expect(() => parseCommand(["--storage", "s3", "--storage-fault-seconds", "15-45"])).toThrow(/needs --storage-faults/);
+    const repo = mkdtempSync(path.join(os.tmpdir(), "restart-chaos-origin-"));
+    const git = (...args: string[]) => execFileSync("git", args, { cwd: repo, stdio: "pipe", encoding: "utf8" }).trim();
+    git("init", "-q"); git("-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "--allow-empty", "-m", "base");
+    git("remote", "add", "origin", "git@github.com:acme/widget.git");
+    git("update-ref", "refs/remotes/origin/master", "HEAD");
+    const masterCommit = git("rev-parse", "HEAD");
+    expect(() => parseCommand(["--env", "daytona", "--daytona-image", "ghcr.io/x/y@sha256:" + "a".repeat(64), "--workloads", "R1", "--workspace-repo", repo]))
+      .toThrow(/needs --storage s3/);
+    const daytonaR1 = parseCommand(["--env", "daytona", "--daytona-image", "ghcr.io/x/y@sha256:" + "a".repeat(64), "--workloads", "R1",
+      "--workspace-repo", repo, "--storage", "s3", "--seed", "5"]);
+    if (daytonaR1.mode !== "wave") throw new Error("expected a wave");
+    expect(daytonaR1.options).toMatchObject({ env: "daytona", storage: "s3", workloads: ["R1"] });
+    expect(daytonaR1.options).toMatchObject({ workspaceGitUrl: "https://github.com/acme/widget.git", workspaceCommit: masterCommit });
+    rmSync(repo, { recursive: true, force: true });
+    expect(publicGitHubUrl("git@github.com:paperclipai/paperclip.git")).toBe("https://github.com/paperclipai/paperclip.git");
+    expect(publicGitHubUrl("https://github.com/paperclipai/paperclip")).toBe("https://github.com/paperclipai/paperclip.git");
+    expect(() => publicGitHubUrl("/Users/me/repo")).toThrow(/GitHub origin/);
     expect(() => parseCommand(["--k4-hold-max-seconds", "601"])).toThrow(/between 120 and 600/);
     expect(() => parseCommand(["--replay", "w-20261106T101112-s42/t03", "--k4-hold-max-seconds", "400"])).toThrow(/do not combine/);
   });

tests/restart-chaos/seed.test.ts

@@ -60,6 +60,15 @@ describe("seedChaosLab workspace isolation", () => {
     expect(skipping.calls.some(call => call.method === "GET" && call.route.startsWith("/api/agents/"))).toBe(false);
   });
 
+  it("registers a remote git source, with no host folder, for a Daytona workspace repository", async () => {
+    const { api, calls } = fakeApi(body => ({ ...body }));
+    await seedChaosLab({ api, env: "local", adapters: ["codex"], concurrency: 2, budgetCents: 3_000, fixtureRoot: path.join(fixtureRoot, "remote"),
+      waveId: "w-test", timeline: new Timeline(null), workspaceGitUrl: "https://github.com/paperclipai/paperclip.git", workspaceCommit: "a".repeat(40) });
+    const project = calls.find(call => call.method === "POST" && call.route.endsWith("/projects"));
+    expect((project!.body as { workspace: unknown }).workspace).toEqual({ name: "Primary", sourceType: "git_repo",
+      repoUrl: "https://github.com/paperclipai/paperclip.git", repoRef: "a".repeat(40), isPrimary: true });
+  });
+
   // RC-21: the project's git_worktree policy is gated behind the instance's
   // isolated-workspaces setting. Without it every repo task ran in the shared
   // primary checkout, so concurrent tasks wrote into one tree.

tests/restart-chaos/seed.ts

@@ -177,6 +177,7 @@ export async function seedChaosLab(input: {
   agentReadyTimeoutMs?: number;
   workspaceRepo?: string;
   workspaceCommit?: string;
+  workspaceGitUrl?: string;
   waveId: string;
   timeline: Timeline;
   /** Merged into each agent's adapterConfig (for example a runner lifecycle mode). */
@@ -284,7 +285,9 @@ export async function seedChaosLab(input: {
       environmentId,
       workspaceStrategy: { type: "git_worktree" },
     },
-    workspace: { name: "Primary", sourceType: "local_path", cwd: fixtureRepositoryPath, isPrimary: true },
+    workspace: input.workspaceGitUrl
+      ? { name: "Primary", sourceType: "git_repo", repoUrl: input.workspaceGitUrl, repoRef: input.workspaceCommit, isPrimary: true }
+      : { name: "Primary", sourceType: "local_path", cwd: fixtureRepositoryPath, isPrimary: true },
   });
   timeline.add("seed_project", { projectId: project.id, fixtureRepositoryPath });
   return { companyId: company.id, environmentId, environmentKind: input.env, projectId: project.id, fixtureRepositoryPath, agents };

tests/restart-chaos/storage-chaos.test.ts

@@ -0,0 +1,85 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { StorageChaos, activeStorageWindow, planStorageFaults, replayStorageWindows, type StorageFaultWindow } from "./storage-chaos.js";
+import type { StorageFaultRule } from "./storage-proxy.js";
+
+describe("storage fault planning", () => {
+  it("plans seeded, non-overlapping windows of the requested kinds across the wave", () => {
+    const windows = planStorageFaults({ kinds: ["ST1", "ST2", "ST3"], seed: 7, spanMs: 30 * 60_000 });
+    expect(windows.length).toBeGreaterThan(5);
+    expect(planStorageFaults({ kinds: ["ST1", "ST2", "ST3"], seed: 7, spanMs: 30 * 60_000 })).toEqual(windows);
+    expect(new Set(windows.map(window => window.kind))).toEqual(new Set(["ST1", "ST2", "ST3"]));
+    for (const [index, window] of windows.entries()) {
+      expect(window.durationMs).toBeGreaterThanOrEqual(60_000);
+      expect(window.durationMs).toBeLessThanOrEqual(180_000);
+      if (index) expect(window.startMs).toBeGreaterThanOrEqual(windows[index - 1]!.startMs + windows[index - 1]!.durationMs + 60_000);
+      if (window.kind === "ST1") expect(window.rules).toEqual([{ fault: { kind: "outage", status: 503 } }]);
+      if (window.kind === "ST2") expect(window.rules[0]!.fault).toMatchObject({ kind: "latency" });
+      if (window.kind === "ST3") expect(window.rules[0]!.fault).toMatchObject({ kind: "errors", rate: 0.3, status: 503 });
+    }
+    expect(planStorageFaults({ kinds: [], seed: 7, spanMs: 60_000 })).toEqual([]);
+  });
+
+  it("applies each window to the proxy while it is active and clears it afterwards", () => {
+    const windows: StorageFaultWindow[] = [{ kind: "ST1", startMs: 1_000, durationMs: 2_000, rules: [{ fault: { kind: "outage" } }] }];
+    expect(activeStorageWindow(windows, 999)).toBeNull();
+    expect(activeStorageWindow(windows, 1_000)).toBe(windows[0]);
+    const applied: Array<readonly StorageFaultRule[] | "clear"> = [];
+    const events: string[] = [];
+    const chaos = new StorageChaos(windows, 0, { setRules: rules => applied.push(rules), clear: () => applied.push("clear") }, kind => events.push(kind));
+    chaos.tick(500); chaos.tick(1_500); chaos.tick(2_000); chaos.tick(3_500);
+    expect(applied).toEqual([windows[0]!.rules, "clear"]);
+    expect(events).toEqual(["storage_fault_started", "storage_fault_ended"]);
+    expect(chaos.receipts).toEqual([{ kind: "ST1", startedAt: 1_500, endedAt: 3_500, rules: windows[0]!.rules }]);
+    chaos.tick(1_200);
+    chaos.stop(1_300);
+    expect(applied.at(-1)).toBe("clear");
+  });
+});
+
+describe("storage fault clock and replay", () => {
+  afterEach(() => { vi.useRealTimers(); });
+
+  it("opens and closes windows on its own clock, not only when the wave loop gets to it", () => {
+    vi.useFakeTimers({ now: 0 });
+    const windows: StorageFaultWindow[] = [{ kind: "ST1", startMs: 2_000, durationMs: 3_000, rules: [{ fault: { kind: "outage" } }] }];
+    const applied: Array<readonly StorageFaultRule[] | "clear"> = [];
+    const chaos = new StorageChaos(windows, 0, { setRules: rules => applied.push(rules), clear: () => applied.push("clear") }, () => undefined);
+    chaos.start(500);
+    vi.advanceTimersByTime(2_500);
+    expect(applied).toEqual([windows[0]!.rules]);
+    vi.advanceTimersByTime(3_000);
+    expect(applied).toEqual([windows[0]!.rules, "clear"]);
+    chaos.stop();
+    vi.advanceTimersByTime(10_000);
+    expect(applied).toHaveLength(2);
+  });
+
+  it("measures window offsets from when it starts, so a replay can start the clock at its task's launch", () => {
+    vi.useFakeTimers({ now: 0 });
+    const windows: StorageFaultWindow[] = [{ kind: "ST1", startMs: 2_000, durationMs: 3_000, rules: [{ fault: { kind: "outage" } }] }];
+    const applied: Array<readonly StorageFaultRule[] | "clear"> = [];
+    const chaos = new StorageChaos(windows, 0, { setRules: rules => applied.push(rules), clear: () => applied.push("clear") }, () => undefined);
+    vi.advanceTimersByTime(60_000);
+    chaos.start(500);
+    vi.advanceTimersByTime(1_500);
+    expect(applied).toEqual([]);
+    vi.advanceTimersByTime(1_000);
+    expect(applied).toEqual([windows[0]!.rules]);
+    chaos.stop();
+  });
+
+  it("replays the storage windows that crossed a task, at offsets from its launch", () => {
+    const rules: StorageFaultRule[] = [{ fault: { kind: "outage", status: 503 } }];
+    const receipts = [
+      { kind: "ST1" as const, startedAt: 1_000, endedAt: 5_000, rules },
+      { kind: "ST2" as const, startedAt: 20_000, endedAt: 30_000, rules: [{ fault: { kind: "latency" as const, ms: 9_000 } }] },
+      { kind: "ST1" as const, startedAt: 40_000, endedAt: null, rules },
+    ];
+    expect(replayStorageWindows(receipts, 10_000, 45_000)).toEqual([
+      { kind: "ST2", startMs: 10_000, durationMs: 10_000, rules: receipts[1]!.rules },
+      { kind: "ST1", startMs: 30_000, durationMs: 5_000, rules },
+    ]);
+    expect(replayStorageWindows(receipts, 3_000, 45_000)[0]).toEqual({ kind: "ST1", startMs: 0, durationMs: 2_000, rules });
+  });
+});
+

tests/restart-chaos/storage-chaos.ts

@@ -0,0 +1,103 @@
+import type { StorageFaultRule } from "./storage-proxy.js";
+
+export const STORAGE_FAULT_KINDS = ["ST1", "ST2", "ST3"] as const;
+export type StorageFaultKind = typeof STORAGE_FAULT_KINDS[number];
+
+export interface StorageFaultWindow { kind: StorageFaultKind; startMs: number; durationMs: number; rules: StorageFaultRule[] }
+
+function rng(seed: number) {
+  let state = seed >>> 0 || 1;
+  return () => {
+    state ^= state << 13; state >>>= 0;
+    state ^= state >>> 17;
+    state ^= state << 5; state >>>= 0;
+    return state / 0x1_0000_0000;
+  };
+}
+
+export function storageFaultRules(kind: StorageFaultKind, random: () => number): StorageFaultRule[] {
+  switch (kind) {
+    case "ST1": return [{ fault: { kind: "outage", status: 503 } }];
+    case "ST2": {
+      const ms = 1_000 + Math.round(random() * 9_000);
+      return [{ fault: { kind: "latency", ms, jitterMs: Math.round(ms / 4) } }];
+    }
+    case "ST3": return [{ fault: { kind: "errors", rate: 0.3, status: 503, slowDown: random() < 0.5 } }];
+  }
+}
+
+export function planStorageFaults(input: { kinds: readonly StorageFaultKind[]; seed: number; spanMs: number;
+  firstAtMs?: number; gapMs?: readonly [number, number]; durationMs?: readonly [number, number] }): StorageFaultWindow[] {
+  if (!input.kinds.length) return [];
+  const random = rng(input.seed ^ 0x5f3759df);
+  const [gapMin, gapMax] = input.gapMs ?? [60_000, 120_000];
+  const [durationMin, durationMax] = input.durationMs ?? [60_000, 180_000];
+  const windows: StorageFaultWindow[] = [];
+  let at = input.firstAtMs ?? 45_000;
+  while (at < input.spanMs) {
+    const kind = input.kinds[Math.floor(random() * input.kinds.length)]!;
+    const durationMs = durationMin + Math.round(random() * (durationMax - durationMin));
+    windows.push({ kind, startMs: at, durationMs, rules: storageFaultRules(kind, random) });
+    at += durationMs + gapMin + Math.round(random() * (gapMax - gapMin));
+  }
+  return windows;
+}
+
+export function activeStorageWindow(windows: readonly StorageFaultWindow[], elapsedMs: number): StorageFaultWindow | null {
+  return windows.find(window => elapsedMs >= window.startMs && elapsedMs < window.startMs + window.durationMs) ?? null;
+}
+
+export interface StorageFaultReceipt { kind: StorageFaultKind; startedAt: number; endedAt: number | null; rules: StorageFaultRule[] }
+
+export function replayStorageWindows(receipts: readonly StorageFaultReceipt[], taskCreatedAt: number, endedAt: number): StorageFaultWindow[] {
+  return receipts.flatMap(receipt => {
+    const start = Math.max(receipt.startedAt, taskCreatedAt);
+    const end = receipt.endedAt ?? endedAt;
+    return end > start ? [{ kind: receipt.kind, startMs: start - taskCreatedAt, durationMs: end - start, rules: receipt.rules }] : [];
+  });
+}
+
+export class StorageChaos {
+  private applied: StorageFaultWindow | null = null;
+  private timer: ReturnType<typeof setInterval> | null = null;
+  readonly receipts: StorageFaultReceipt[] = [];
+
+  constructor(private readonly windows: readonly StorageFaultWindow[], private startedAt: number,
+    private readonly proxy: { setRules(rules: readonly StorageFaultRule[]): void; clear(): void },
+    private readonly record: (kind: string, data: Record<string, unknown>) => void) {}
+
+  tick(now = Date.now()): void {
+    const active = activeStorageWindow(this.windows, now - this.startedAt);
+    if (active === this.applied) return;
+    if (this.applied) {
+      this.proxy.clear();
+      const receipt = this.receipts.at(-1);
+      if (receipt) receipt.endedAt = now;
+      this.record("storage_fault_ended", { kind: this.applied.kind });
+    }
+    this.applied = active;
+    if (active) {
+      this.proxy.setRules(active.rules);
+      this.receipts.push({ kind: active.kind, startedAt: now, endedAt: null, rules: active.rules });
+      this.record("storage_fault_started", { kind: active.kind, durationMs: active.durationMs, rules: active.rules });
+    }
+  }
+
+  start(intervalMs = 1_000, at = Date.now()): void {
+    if (this.timer) return;
+    this.startedAt = at;
+    this.timer = setInterval(() => this.tick(Date.now()), intervalMs);
+    this.timer.unref?.();
+    this.tick(Date.now());
+  }
+
+  stop(now = Date.now()): void {
+    if (this.timer) { clearInterval(this.timer); this.timer = null; }
+    if (!this.applied) return;
+    this.proxy.clear();
+    const receipt = this.receipts.at(-1);
+    if (receipt) receipt.endedAt = now;
+    this.record("storage_fault_ended", { kind: this.applied.kind, atStop: true });
+    this.applied = null;
+  }
+}

tests/restart-chaos/storage-proxy.test.ts

@@ -0,0 +1,92 @@
+import { createServer, type Server } from "node:http";
+import { once } from "node:events";
+import { afterEach, describe, expect, it } from "vitest";
+import { startStorageFaultProxy } from "./storage-proxy.js";
+
+const closers: Array<() => Promise<unknown>> = [];
+afterEach(async () => { for (const close of closers.splice(0).reverse()) await close(); });
+
+async function upstream() {
+  const seen: string[] = [];
+  const server: Server = createServer((incoming, outgoing) => {
+    let body = "";
+    incoming.on("data", chunk => { body += chunk; });
+    incoming.on("end", () => { seen.push(`${incoming.method} ${incoming.url} ${body}`); outgoing.writeHead(200, { etag: "\"e1\"" }).end("stored"); });
+  });
+  server.listen(0, "127.0.0.1"); await once(server, "listening");
+  closers.push(async () => { server.closeAllConnections(); server.close(); });
+  const address = server.address() as { port: number };
+  return { endpoint: `http://127.0.0.1:${address.port}`, seen };
+}
+
+async function proxyFor(endpoint: string, random?: () => number) {
+  const proxy = await startStorageFaultProxy(endpoint, random);
+  closers.push(() => proxy.close());
+  return proxy;
+}
+
+describe("storage fault proxy", () => {
+  it("passes requests through untouched when no rule matches", async () => {
+    const store = await upstream();
+    const proxy = await proxyFor(store.endpoint);
+    const response = await fetch(`${proxy.endpoint}/bucket/key`, { method: "PUT", body: "bytes" });
+    expect([response.status, await response.text(), response.headers.get("etag")]).toEqual([200, "stored", "\"e1\""]);
+    expect(store.seen).toEqual(["PUT /bucket/key bytes"]);
+    expect(proxy.counts()).toMatchObject({ requests: 1, injected: 0, byMethod: { PUT: 1 } });
+  });
+
+  it("injects a store outage for matching requests only, for a bounded number of hits", async () => {
+    const store = await upstream();
+    const proxy = await proxyFor(store.endpoint);
+    proxy.setRules([{ fault: { kind: "outage" }, methods: ["PUT"], pathIncludes: "workspace-snapshots", remaining: 2 }]);
+    const put = () => fetch(`${proxy.endpoint}/bucket/c/workspace-snapshots/w/s.pcws`, { method: "PUT", body: "x" });
+    const first = await put();
+    expect([first.status, await first.text()]).toEqual([503, expect.stringContaining("<Code>ServiceUnavailable</Code>")]);
+    expect((await put()).status).toBe(503);
+    expect((await put()).status).toBe(200);
+    expect((await fetch(`${proxy.endpoint}/bucket/c/run-logs/r`)).status).toBe(200);
+    expect(store.seen).toHaveLength(2);
+    expect(proxy.counts()).toMatchObject({ injected: 2, byStatus: { 503: 2, 200: 2 } });
+  });
+
+  it("answers SlowDown throttling and expired credentials the way S3 does", async () => {
+    const store = await upstream();
+    const proxy = await proxyFor(store.endpoint, () => 0);
+    proxy.setRules([{ fault: { kind: "errors", rate: 0.5, status: 503, slowDown: true } }]);
+    const slow = await fetch(`${proxy.endpoint}/bucket/a`);
+    expect([slow.status, await slow.text()]).toEqual([503, expect.stringContaining("<Code>SlowDown</Code>")]);
+    proxy.setRules([{ fault: { kind: "expired_credentials" } }]);
+    const expired = await fetch(`${proxy.endpoint}/bucket/a`);
+    expect([expired.status, await expired.text()]).toEqual([403, expect.stringContaining("<Code>ExpiredToken</Code>")]);
+    proxy.clear();
+    expect((await fetch(`${proxy.endpoint}/bucket/a`)).status).toBe(200);
+  });
+
+  it("delays requests by the configured latency and drops connections on reset", async () => {
+    const store = await upstream();
+    const proxy = await proxyFor(store.endpoint, () => 0);
+    proxy.setRules([{ fault: { kind: "latency", ms: 150 } }]);
+    const started = Date.now();
+    expect((await fetch(`${proxy.endpoint}/bucket/a`)).status).toBe(200);
+    expect(Date.now() - started).toBeGreaterThanOrEqual(140);
+    proxy.setRules([{ fault: { kind: "reset" } }]);
+    await expect(fetch(`${proxy.endpoint}/bucket/a`)).rejects.toThrow();
+  });
+
+  it("cancels a delayed request when the client goes away and counts store connection failures", async () => {
+    const store = await upstream();
+    const proxy = await proxyFor(store.endpoint, () => 0);
+    proxy.setRules([{ fault: { kind: "latency", ms: 400 } }]);
+    const controller = new AbortController();
+    const pending = fetch(`${proxy.endpoint}/bucket/late`, { signal: controller.signal }).catch(() => null);
+    await new Promise(resolve => setTimeout(resolve, 50));
+    controller.abort();
+    await pending;
+    await new Promise(resolve => setTimeout(resolve, 600));
+    expect(store.seen).toEqual([]);
+    const dead = await proxyFor("http://127.0.0.1:1");
+    expect((await fetch(`${dead.endpoint}/bucket/a`)).status).toBe(502);
+    expect(dead.counts()).toMatchObject({ requests: 1, byStatus: { 502: 1 } });
+  });
+});
+

tests/restart-chaos/storage-proxy.ts

@@ -0,0 +1,114 @@
+import { Agent, createServer, request, type IncomingMessage, type ServerResponse } from "node:http";
+import { once } from "node:events";
+import type { Socket } from "node:net";
+
+export type StorageFault =
+  | { kind: "pass" }
+  | { kind: "outage"; status?: number }
+  | { kind: "latency"; ms: number; jitterMs?: number }
+  | { kind: "errors"; rate: number; status?: number; slowDown?: boolean }
+  | { kind: "reset" }
+  | { kind: "expired_credentials" };
+
+export interface StorageFaultRule { fault: StorageFault; methods?: readonly string[]; pathIncludes?: string; remaining?: number }
+
+export interface StorageProxyCounts { requests: number; injected: number; byMethod: Record<string, number>; byStatus: Record<string, number> }
+
+const XML = "application/xml";
+function s3Error(code: string, message: string) {
+  return `<?xml version="1.0" encoding="UTF-8"?><Error><Code>${code}</Code><Message>${message}</Message></Error>`;
+}
+
+export function faultResponse(fault: StorageFault): { status: number; body: string } | "reset" | null {
+  switch (fault.kind) {
+    case "outage": return { status: fault.status ?? 503, body: s3Error("ServiceUnavailable", "Injected storage outage") };
+    case "expired_credentials": return { status: 403, body: s3Error("ExpiredToken", "The provided token has expired.") };
+    case "reset": return "reset";
+    default: return null;
+  }
+}
+
+export function matches(rule: StorageFaultRule, method: string, url: string): boolean {
+  if (rule.remaining !== undefined && rule.remaining <= 0) return false;
+  if (rule.methods && !rule.methods.includes(method)) return false;
+  return !rule.pathIncludes || url.includes(rule.pathIncludes);
+}
+
+export async function startStorageFaultProxy(endpoint: string, random: () => number = Math.random) {
+  const target = new URL(endpoint);
+  if (target.protocol !== "http:" || !["127.0.0.1", "localhost", "[::1]"].includes(target.hostname)) {
+    throw new Error("Storage fault injection requires a loopback store");
+  }
+  let rules: StorageFaultRule[] = [];
+  const counts: StorageProxyCounts = { requests: 0, injected: 0, byMethod: {}, byStatus: {} };
+  const sockets = new Set<Socket>();
+  const delayed = new Set<ReturnType<typeof setTimeout>>();
+  const agent = new Agent({ keepAlive: true });
+  const note = (method: string, status: number | "reset", injected: boolean) => {
+    counts.requests += 1;
+    counts.byMethod[method] = (counts.byMethod[method] ?? 0) + 1;
+    counts.byStatus[String(status)] = (counts.byStatus[String(status)] ?? 0) + 1;
+    if (injected) counts.injected += 1;
+  };
+  const forward = (incoming: IncomingMessage, outgoing: ServerResponse) => {
+    const upstream = request(target, { method: incoming.method, path: incoming.url, headers: incoming.headers, agent }, response => {
+      response.on("error", () => outgoing.destroy());
+      note(incoming.method ?? "GET", response.statusCode ?? 502, false);
+      outgoing.writeHead(response.statusCode ?? 502, response.headers);
+      response.pipe(outgoing);
+    });
+    upstream.on("error", () => {
+      if (outgoing.headersSent) { outgoing.destroy(); return; }
+      note(incoming.method ?? "GET", 502, false);
+      outgoing.writeHead(502).end();
+    });
+    incoming.on("aborted", () => upstream.destroy());
+    outgoing.on("close", () => { if (!outgoing.writableFinished) upstream.destroy(); });
+    incoming.pipe(upstream);
+  };
+  const server = createServer((incoming, outgoing) => {
+    const method = incoming.method ?? "GET";
+    const rule = rules.find(candidate => matches(candidate, method, incoming.url ?? "/"));
+    if (rule?.remaining !== undefined) rule.remaining -= 1;
+    const fault = rule?.fault ?? { kind: "pass" as const };
+    if (fault.kind === "latency") {
+      const delay = fault.ms + Math.round((fault.jitterMs ?? 0) * random());
+      incoming.pause();
+      const timer = setTimeout(() => {
+        delayed.delete(timer);
+        if (outgoing.destroyed || incoming.destroyed) return;
+        incoming.resume();
+        forward(incoming, outgoing);
+      }, delay);
+      delayed.add(timer);
+      outgoing.on("close", () => { clearTimeout(timer); delayed.delete(timer); });
+      return;
+    }
+    if (fault.kind === "errors" && random() >= fault.rate) { forward(incoming, outgoing); return; }
+    const injected = fault.kind === "errors"
+      ? { status: fault.status ?? 503, body: fault.slowDown ? s3Error("SlowDown", "Please reduce your request rate.") : s3Error("InternalError", "Injected storage error") }
+      : faultResponse(fault);
+    if (!injected) { forward(incoming, outgoing); return; }
+    incoming.resume();
+    if (injected === "reset") { note(method, "reset", true); incoming.socket.destroy(); return; }
+    note(method, injected.status, true);
+    outgoing.writeHead(injected.status, { "content-type": XML, "content-length": Buffer.byteLength(injected.body) }).end(injected.body);
+  });
+  server.on("connection", socket => { sockets.add(socket); socket.on("close", () => sockets.delete(socket)); });
+  server.listen(0, "127.0.0.1"); await once(server, "listening");
+  const address = server.address();
+  if (!address || typeof address === "string") throw new Error("Storage proxy did not bind");
+  return {
+    endpoint: `http://127.0.0.1:${address.port}`,
+    counts: () => structuredClone(counts),
+    setRules(next: readonly StorageFaultRule[]) { rules = next.map(rule => ({ ...rule })); },
+    clear() { rules = []; },
+    async close() {
+      for (const timer of delayed) clearTimeout(timer);
+      delayed.clear();
+      const closed = once(server, "close"); server.close(); agent.destroy();
+      for (const socket of sockets) socket.destroy();
+      await closed;
+    },
+  };
+}

tests/restart-chaos/supervisor.ts

@@ -214,7 +214,9 @@ export class ChaosSupervisor {
       ...extraEnvironment,
     };
     const environment = buildPaperclipServerEnvironment(source, overrides);
-    assertIsolatedServerEnvironment(environment, { temporaryRoot, paperclipHome: home, configPath });
+    const workspaceStorage = extraEnvironment.PAPERCLIP_STORAGE_PROVIDER === "s3"
+      ? Object.fromEntries(Object.entries(extraEnvironment).filter(([key]) => key.startsWith("PAPERCLIP_STORAGE_"))) : undefined;
+    assertIsolatedServerEnvironment(environment, { temporaryRoot, paperclipHome: home, configPath, ...(workspaceStorage ? { workspaceStorage } : {}) });
     this.environment = Object.fromEntries(Object.entries(environment).filter((entry): entry is [string, string] => entry[1] !== undefined));
     await prepareSourceServerDependencies(root, this.environment, false);
   }

tests/restart-chaos/triggers.test.ts

@@ -146,6 +146,28 @@ describe("trigger firing", () => {
     expect(observer.record.triggerMissed).toEqual(["comment"]);
   });
 
+  it("clicks Retry through the board UI when the wave drives a browser, and records what the user saw", async () => {
+    const state = { runs: [{ runId: "r1", status: "failed", finishedAt: new Date().toISOString() }] as Json[],
+      issue: { id: "i1", status: "blocked", interruption: { state: "failed", canRetry: true, observedRunId: "r1" } } };
+    const clicks: string[] = [];
+    const board = { clickRetry: async (issue: string) => { clicks.push(issue);
+      return { clicked: true, httpStatus: 202, accepted: true, error: null, bannerText: "Couldn’t finish this attempt." }; } };
+    const observer = new TaskObserver(base, scriptedApi(state), lab, new Timeline(null), ready, undefined, board);
+    await observer.launch();
+    const start = observer.record.createdAt!;
+    await observer.poll(start + 1_000);
+    await observer.poll(start + 1_000 + 12_001);
+    expect(clicks).toEqual(["CHA-1"]);
+    expect(observer.record.retry).toMatchObject({ via: "ui", accepted: true, httpStatus: 202, bannerText: "Couldn’t finish this attempt." });
+
+    const missing = { clickRetry: async () => ({ clicked: false, httpStatus: null, accepted: null, error: "ui_retry_button_missing", bannerText: "Saving files from the previous attempt" }) };
+    const blind = new TaskObserver(base, scriptedApi(state), lab, new Timeline(null), ready, undefined, missing);
+    await blind.launch();
+    await blind.poll(blind.record.createdAt! + 1_000);
+    await blind.poll(blind.record.createdAt! + 1_000 + 12_001);
+    expect(blind.record.retry).toMatchObject({ via: "ui", accepted: null, error: "ui_retry_button_missing" });
+  });
+
   it("retry: fires the extra K1 right after an accepted Retry click, and closes unfired triggers at sweep", async () => {
     const state = { runs: [{ runId: "r1", status: "failed", finishedAt: new Date().toISOString() }] as Json[],
       issue: { id: "i1", status: "blocked", interruption: { state: "needs_retry", canRetry: true, observedRunId: "r1" } } };

tests/restart-chaos/types.ts

@@ -73,7 +73,12 @@ export interface WaveOptions {
    */
   upgradeFrom?: string;
   workspaceRepo?: string;
+  storage?: "s3";
+  storageFaults?: Array<"ST1" | "ST2" | "ST3">;
+  storageFaultMs?: [number, number];
+  storageWindows?: import("./storage-chaos.js").StorageFaultWindow[];
   workspaceCommit?: string;
+  workspaceGitUrl?: string;
 }
 
 export interface StormFault {
@@ -183,8 +188,13 @@ export interface RetryAttempt {
   httpStatus: number | null;
   accepted: boolean | null;
   error: string | null;
+  via?: "ui" | "api";
+  bannerText?: string | null;
 }
 
+export const UI_RETRY_BUTTON_MISSING = "ui_retry_button_missing";
+export const UI_RETRY_NO_REQUEST = "ui_retry_no_request";
+
 export interface FaultReceipt {
   faultId: string;
   kind: FaultKind;
@@ -297,6 +307,8 @@ export interface TaskRecord {
   workspaceMovedTo?: string;
   /** Set when the pre-fault snapshot could not read the workspace; R2 then fails visibly. */
   preFaultSnapshotError?: string | null;
+  remoteWorkspace?: boolean;
+  remoteFiles?: Record<string, string | null>;
   /** Pre-fault files missing at rest; null when not checked. */
   lostFiles: string[] | null;
   /** B5: distinct bench progress values the observer read during polling. */