From 908d2aae7ae348c83c3e8385428241ebf4633960 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 9 Nov 2026 13:15:41 -0600 Subject: [PATCH] test(chaos): run waves against S3-compatible storage with a fault proxy, and click Retry in the real board UI (#20288) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud deployments keep run logs, traces, attachments, artifacts and saved workspace files in S3-compatible object storage > - The restart chaos harness ran only with local disk storage, so storage failures and slowdowns were never tested > - It also clicked Retry through the API, so a missing or broken Retry button in the board UI went unnoticed (staging PAP-1683: "the saved-files recovery Retry does nothing") > - This pull request runs waves against a loopback MinIO behind a fault proxy, adds store-outage, latency and throttling faults, and clicks every Retry in the real board UI > - The benefit is that storage outages and broken Retry paths now surface in chaos waves. The first ST1 wave found that a run starting during a store outage fails immediately and exhausts recovery ## Linked Issues or Issue Description **What happened?** Staging saves to object storage and has failures there that local chaos waves cannot reproduce, because the harness used `local_disk`. The harness also clicked Retry through the API, which skips the board's actual button. **Expected behavior** Chaos waves can run the server in cloud storage mode, inject storage faults, and press Retry the way a user does. **Actual behavior** They could not. ## What Changed - `--storage s3`: - starts a loopback MinIO (`RELEASE.2025-04-22T22-12-26Z`, from `~/.cache/paperclip/tools/minio/minio` or `RESTART_CHAOS_MINIO_BIN`) with its data under the instance temp root and random root credentials; - creates the bucket; - puts a fault proxy in front; - points the server's `PAPERCLIP_STORAGE_*` settings at the proxy, with path-style addressing and a per-wave prefix. The supervisor declares these as explicit workspace storage for the isolation check. - `storage-proxy.ts`: a transparent, non-retrying S3 fault proxy that keeps the signed Host header. Rules match by method, path and remaining count. Faults are outage (503 `ServiceUnavailable`), latency with jitter, a 5xx or `SlowDown` error rate, connection reset, and 403 `ExpiredToken`. It counts requests by method and status, written to `storage-proxy.json`. - `--storage-faults ST1,ST2,ST3` (`storage-chaos.ts`): seeded 60–180 s windows of outage, 1–10 s latency, or 30% 503 / `SlowDown`, applied while the wave runs. They're logged in the timeline and in `storage-faults.json`. - `board-ui.ts`: every task and subtask Retry the grader needs is now clicked in the board UI. The harness uses headless Chromium through Playwright, opens `/issues/`, waits for the visible Retry button and captures the `POST /issues/:id/retry` the click sends. - R3 fails a missing button as `ui_retry_button_missing (user saw: …)` and a click that sends nothing as `ui_retry_no_request`. - Crash-point waves, which serve no UI, keep the API path. - README: a cloud storage section. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 244 passed. New tests: - the proxy against a fake upstream: passthrough, bounded outage, `SlowDown`, `ExpiredToken`, latency and reset; - live MinIO through the proxy: `If-None-Match` returns 412, and an injected 503 reaches the client; - Playwright against local pages: a working button, a missing button, and a dead button; - the observer clicking through the board, and R3 signatures for UI failures; - window planning, apply and clear; - `--storage` and `--storage-faults` parsing. The observer and grader tests failed before the change. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - Live waves on master 610b54f551 with this branch: - Baseline `--storage s3`, A1/C1, K1/K2 (w-20261109T144738-s9901): 5/8. There were 2,319 S3 requests through the proxy and 18 conditional-write 412 fences. All 3 Retries were clicked in the board UI, returned 202 Accepted, and the new run succeeded. The 3 failures are a Claude session issue (#20285), not storage. - ST1 (w-20261109T145628-s9902): 1/8. A run that starts during a store outage fails within about 1 s as `adapter_failed`, automatic recovery exhausts within the outage, and UI Retries are accepted but do not recover. This has been reported for a product fix. ## Risks - `--storage s3` needs a locally built MinIO. MinIO no longer publishes community binaries. Without the binary the wave fails at start with the path to provide, and the live store test is skipped. - The UI Retry needs the dev UI to load. A slow first Vite compile can delay the click, up to the 60 s wait. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- tests/restart-chaos/README.md | 38 ++++++++ tests/restart-chaos/api.ts | 9 +- tests/restart-chaos/board-ui.test.ts | 86 ++++++++++++++++ tests/restart-chaos/board-ui.ts | 87 +++++++++++++++++ tests/restart-chaos/config.ts | 36 ++++++- tests/restart-chaos/grader.test.ts | 17 +++- tests/restart-chaos/grader.ts | 8 +- tests/restart-chaos/main.ts | 42 +++++++- tests/restart-chaos/object-store.test.ts | 49 ++++++++++ tests/restart-chaos/object-store.ts | 98 +++++++++++++++++++ tests/restart-chaos/observer.ts | 86 +++++++++++----- tests/restart-chaos/remote-oracle.test.ts | 109 +++++++++++++++++++++ tests/restart-chaos/scheduler.test.ts | 37 ++++++- tests/restart-chaos/seed.test.ts | 9 ++ tests/restart-chaos/seed.ts | 5 +- tests/restart-chaos/storage-chaos.test.ts | 85 ++++++++++++++++ tests/restart-chaos/storage-chaos.ts | 103 +++++++++++++++++++ tests/restart-chaos/storage-proxy.test.ts | 92 +++++++++++++++++ tests/restart-chaos/storage-proxy.ts | 114 ++++++++++++++++++++++ tests/restart-chaos/supervisor.ts | 4 +- tests/restart-chaos/triggers.test.ts | 22 +++++ tests/restart-chaos/types.ts | 12 +++ 22 files changed, 1112 insertions(+), 36 deletions(-) create mode 100644 tests/restart-chaos/board-ui.test.ts create mode 100644 tests/restart-chaos/board-ui.ts create mode 100644 tests/restart-chaos/object-store.test.ts create mode 100644 tests/restart-chaos/object-store.ts create mode 100644 tests/restart-chaos/remote-oracle.test.ts create mode 100644 tests/restart-chaos/storage-chaos.test.ts create mode 100644 tests/restart-chaos/storage-chaos.ts create mode 100644 tests/restart-chaos/storage-proxy.test.ts create mode 100644 tests/restart-chaos/storage-proxy.ts diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 25d1f3ae704..10484cdc381 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -171,6 +171,44 @@ receipt names both commits (`upgrade` in `faults.json`, `upgrade_switched` in th timeline). K14 is local-only, because a Daytona runner must match its image, and it is never part of a storm's default faults. +## Cloud storage (`--storage s3`, `--storage-faults`) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ + --adapters codex,claude --workloads A1,C1 --faults K1,K2 --concurrency 4 --tasks 8 --seed 24 \ + --storage s3 --storage-faults ST1,ST2,ST3 +``` + +`--storage s3` runs the server the way cloud mode does: `PAPERCLIP_STORAGE_PROVIDER=s3`, so run logs +and provider traces use the `s3_segments` backend and attachments, artifacts and other stored files go +to object storage. The store is a loopback MinIO (`RELEASE.2025-04-22T22-12-26Z`, built from source at +`~/.cache/paperclip/tools/minio/minio` or `RESTART_CHAOS_MINIO_BIN`) with its data under the instance +temp root, behind a fault proxy that every server S3 call goes through. The proxy's request counts by +method and status are written to `storage-proxy.json`. + +`--storage-faults` adds seeded windows of 60–180 s, alongside the restart faults: + +| Fault | Store behaviour during the window | +| --- | --- | +| ST1 | outage: every request answers 503 `ServiceUnavailable` | +| ST2 | slow store: 1–10 s added latency with jitter | +| ST3 | 30% of requests answer 503, half of them `SlowDown` | + +`--storage-fault-seconds 15-45` sets the window length range (5–900 s), for example to find how long +an outage a run survives. The windows are logged in the timeline and written to `storage-faults.json`. + +With `--env daytona --storage s3`, each task's workspace is a remote workspace: the controller saves +and restores it through the store, so the faults also hit workspace save, restore and copyback. R1 +(`--workspace-repo`) runs on Daytona only with `--storage s3`: the project points at the checkout's +GitHub `origin` over https at its `origin/master` commit, so the sandbox clones the repository and its +snapshots are Paperclip-sized. Its oracle reads the task's files through the board API and runs its +commands on a local copy. + +Every Retry the grader needs is clicked in the real board UI: the harness opens the issue in a headless +Chromium, waits for the visible Retry button and clicks it. A task whose Retry button is missing fails +R3 as `ui_retry_button_missing (user saw: …)`; a click that sends no retry request fails as +`ui_retry_no_request`. Crash-point waves serve no UI and click Retry through the API. + ## Daytona (`--env daytona`) The control plane stays local; agent runs execute in Daytona sandboxes from an diff --git a/tests/restart-chaos/api.ts b/tests/restart-chaos/api.ts index b5e2fb8619d..eeebfb674e9 100644 --- a/tests/restart-chaos/api.ts +++ b/tests/restart-chaos/api.ts @@ -20,16 +20,16 @@ export function rateLimitDelayMs(response: { status: number; headers: { get(name export class ChaosApi { constructor(readonly baseUrl: string, private readonly timeoutMs = 10_000, private readonly rateLimitAttempts = 5) {} - private async request(method: string, route: string, body?: unknown, timeoutMs = this.timeoutMs): Promise { + private async request(method: string, route: string, body?: unknown, timeoutMs = this.timeoutMs, text = false): Promise { for (let attempt = 1; ; attempt += 1) { - const outcome = await this.attempt(method, route, body, timeoutMs); + const outcome = await this.attempt(method, route, body, timeoutMs, text); if (!("retryAfterMs" in outcome)) return outcome.value; if (method !== "GET" || attempt >= this.rateLimitAttempts) throw outcome.error; await new Promise(resolve => setTimeout(resolve, Math.min(10_000, outcome.retryAfterMs * attempt))); } } - private async attempt(method: string, route: string, body: unknown, timeoutMs: number): + private async attempt(method: string, route: string, body: unknown, timeoutMs: number, asText = false): Promise<{ value: T } | { retryAfterMs: number; error: ChaosApiError }> { let response: Response; try { @@ -50,10 +50,11 @@ export class ChaosApi { if (retryAfterMs !== null) return { retryAfterMs, error }; throw error; } - return { value: (text ? JSON.parse(text) : null) as T }; + return { value: (asText ? text : text ? JSON.parse(text) : null) as T }; } get(route: string, timeoutMs?: number) { return this.request("GET", route, undefined, timeoutMs); } + getText(route: string, timeoutMs?: number) { return this.request("GET", route, undefined, timeoutMs, true); } post(route: string, body?: unknown, timeoutMs?: number) { return this.request("POST", route, body ?? {}, timeoutMs); } put(route: string, body?: unknown) { return this.request("PUT", route, body ?? {}); } patch(route: string, body?: unknown) { return this.request("PATCH", route, body ?? {}); } diff --git a/tests/restart-chaos/board-ui.test.ts b/tests/restart-chaos/board-ui.test.ts new file mode 100644 index 00000000000..6cfd43267e1 --- /dev/null +++ b/tests/restart-chaos/board-ui.test.ts @@ -0,0 +1,86 @@ +import { createServer, type Server } from "node:http"; +import { once } from "node:events"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { BoardUi, UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST, isRetryRequestFor } from "./board-ui.js"; + +const retry = (issue: string) => `fetch('/api/issues/${issue}/retry',{method:'POST'})`; +const pages: Record = { + "/issues/CHA-1": `
Couldn’t finish this attempt.
`, + "/issues/CHA-2": `
Saving files from the previous attempt
`, + "/issues/CHA-3": `
Automatic recovery stopped.
`, + "/issues/CHA-4": `

Couldn’t load the subtasks.

+
Couldn’t finish this attempt.
`, + "/issues/CHA-5": `
Couldn’t finish this attempt.
`, +}; +let server: Server; +let baseUrl: string; +const posts: string[] = []; + +beforeAll(async () => { + server = createServer((incoming, outgoing) => { + if (incoming.method === "POST") { + posts.push(incoming.url ?? ""); + outgoing.writeHead(202, { "content-type": "application/json" }).end(JSON.stringify({ accepted: true })); + return; + } + const body = pages[incoming.url ?? ""]; + outgoing.writeHead(body ? 200 : 404, { "content-type": "text/html; charset=utf-8" }).end(`${body ?? ""}`); + }); + server.listen(0, "127.0.0.1"); await once(server, "listening"); + baseUrl = `http://127.0.0.1:${(server.address() as { port: number }).port}`; +}); +afterAll(async () => { server.closeAllConnections(); server.close(); }); + +async function withBoard(run: (board: BoardUi) => Promise): Promise { + const board = new BoardUi(baseUrl); + try { return await run(board); } finally { await board.close(); } +} + +describe("BoardUi.clickRetry", () => { + it("clicks the visible Retry button and reports the retry request it sent", async () => { + await withBoard(async board => { + expect(await board.clickRetry("CHA-1", "i1", 10_000)).toMatchObject({ clicked: true, httpStatus: 202, accepted: true, error: null, + bannerText: expect.stringContaining("Couldn’t finish this attempt") }); + }); + expect(posts).toContain("/api/issues/i1/retry"); + }, 60_000); + + it("reports a missing Retry button with the status text the user sees", async () => { + await withBoard(async board => { + expect(await board.clickRetry("CHA-2", "i2", 1_500)).toMatchObject({ clicked: false, error: UI_RETRY_BUTTON_MISSING, + bannerText: "Saving files from the previous attempt" }); + }); + }, 60_000); + + it("reports a Retry button whose click sends no retry request", async () => { + await withBoard(async board => { + expect(await board.clickRetry("CHA-3", "i3", 1_500, 500)).toMatchObject({ clicked: true, httpStatus: null, error: UI_RETRY_NO_REQUEST }); + }); + }, 60_000); + + it("prefers the recovery banner's Retry over another Retry earlier on the page", async () => { + await withBoard(async board => { + expect(await board.clickRetry("CHA-4", "i4", 10_000, 500)).toMatchObject({ clicked: true, httpStatus: 202, accepted: true }); + }); + expect(posts).toContain("/api/issues/i4/retry"); + }, 60_000); + + it("does not count a retry request for a different issue", async () => { + await withBoard(async board => { + expect(await board.clickRetry("CHA-5", "i5", 1_500, 500)).toMatchObject({ clicked: true, error: UI_RETRY_NO_REQUEST }); + }); + }, 60_000); + + it("reports a browser that cannot launch and still closes cleanly", async () => { + const board = new BoardUi(baseUrl, () => Promise.reject(new Error("browser missing"))); + expect(await board.clickRetry("CHA-1", "i1", 1_000)).toMatchObject({ clicked: false, error: "browser missing" }); + await expect(board.close()).resolves.toBeUndefined(); + }); + + it("matches retry requests by the issue's identifier or id only", () => { + expect(isRetryRequestFor(["CHA-1", "i1"], "POST", "http://x/api/issues/i1/retry")).toBe(true); + expect(isRetryRequestFor(["CHA-1", "i1"], "POST", "http://x/api/issues/CHA-1/retry")).toBe(true); + expect(isRetryRequestFor(["CHA-1", "i1"], "POST", "http://x/api/issues/i2/retry")).toBe(false); + expect(isRetryRequestFor(["CHA-1", "i1"], "GET", "http://x/api/issues/i1/retry")).toBe(false); + }); +}); diff --git a/tests/restart-chaos/board-ui.ts b/tests/restart-chaos/board-ui.ts new file mode 100644 index 00000000000..029f2cba161 --- /dev/null +++ b/tests/restart-chaos/board-ui.ts @@ -0,0 +1,87 @@ +import { chromium, type Browser, type ElementHandle, type Page } from "@playwright/test"; +import { UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST } from "./types.js"; + +export { UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST }; + +export interface UiRetryClick { + clicked: boolean; + httpStatus: number | null; + accepted: boolean | null; + error: string | null; + bannerText: string | null; +} + +const PER_BUTTON_REQUEST_MS = 5_000; + +export function isRetryRequestFor(issueRefs: readonly string[], method: string, url: string): boolean { + if (method !== "POST") return false; + const match = /\/api\/issues\/([^/]+)\/retry$/.exec(new URL(url).pathname); + return Boolean(match && issueRefs.includes(decodeURIComponent(match[1]!))); +} + +async function retryButtonsBannerFirst(page: Page): Promise { + const candidates: Array<{ button: ElementHandle; inBanner: boolean }> = []; + for (const button of await page.getByRole("button", { name: "Retry", exact: true }).elementHandles()) { + if (!await button.isVisible().catch(() => false)) continue; + const inBanner = await button.evaluate(node => Boolean((node as Element).closest("[role=status], [role=alert]"))).catch(() => false); + candidates.push({ button, inBanner }); + } + return [...candidates.filter(candidate => candidate.inBanner), ...candidates.filter(candidate => !candidate.inBanner)].map(candidate => candidate.button); +} + +export async function clickRetryOnPage(page: Page, url: string, issueRefs: readonly string[], timeoutMs: number, + perButtonMs = PER_BUTTON_REQUEST_MS): Promise { + await page.goto(url, { waitUntil: "domcontentloaded", timeout: timeoutMs }); + const banner = async () => { + const texts = await page.locator("[role=status], [role=alert]").allInnerTexts().catch(() => []); + return texts.map(text => text.trim()).filter(Boolean).join(" | ").slice(0, 500) || null; + }; + try { + await page.getByRole("button", { name: "Retry", exact: true }).first().waitFor({ state: "visible", timeout: timeoutMs }); + } catch { + return { clicked: false, httpStatus: null, accepted: null, error: UI_RETRY_BUTTON_MISSING, bannerText: await banner() }; + } + const bannerText = await banner(); + for (const button of await retryButtonsBannerFirst(page)) { + const response = page.waitForResponse(candidate => isRetryRequestFor(issueRefs, candidate.request().method(), candidate.url()), + { timeout: perButtonMs }).catch(() => null); + await button.click().catch(() => undefined); + const answered = await response; + if (!answered) continue; + const body = await answered.json().catch(() => null) as { accepted?: unknown } | null; + return { clicked: true, httpStatus: answered.status(), accepted: answered.ok() ? body?.accepted === true : false, + error: answered.ok() ? null : `retry request failed with ${answered.status()}`, bannerText }; + } + return { clicked: true, httpStatus: null, accepted: null, error: UI_RETRY_NO_REQUEST, bannerText }; +} + +export class BoardUi { + private browser: Promise | null = null; + + constructor(private readonly baseUrl: string, private readonly launch: () => Promise = () => chromium.launch({ headless: true })) {} + + async clickRetry(identifier: string, issueId: string = identifier, timeoutMs = 60_000, perButtonMs = PER_BUTTON_REQUEST_MS): Promise { + let context: Awaited> | null = null; + try { + this.browser ??= this.launch(); + context = await (await this.browser).newContext({ viewport: { width: 1440, height: 1000 } }); + return await clickRetryOnPage(await context.newPage(), `${this.baseUrl}/issues/${encodeURIComponent(identifier)}`, + [identifier, issueId], timeoutMs, perButtonMs); + } catch (error) { + return { clicked: false, httpStatus: null, accepted: null, error: error instanceof Error ? error.message : String(error), bannerText: null }; + } finally { + await context?.close().catch(() => undefined); + } + } + + async close(): Promise { + const browser = this.browser; + this.browser = null; + if (!browser) return; + try { + await (await browser).close(); + } catch { + return; + } + } +} diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index 246fb37f786..d0cf0e1e2c6 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -1,6 +1,7 @@ import { execFileSync } from "node:child_process"; import path from "node:path"; import { parseArgs } from "node:util"; +import { STORAGE_FAULT_KINDS } from "./storage-chaos.js"; import { ADAPTER_IDS, ENVIRONMENT_KINDS, FAULT_KINDS, TRIGGER_IDS, WORKLOAD_IDS, type AdapterId, type EnvironmentKind, type FaultKind, type TriggerId, type WaveOptions, type WorkloadId, @@ -98,13 +99,16 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC "crash-point-hold-ms": { type: "string" }, "upgrade-from": { type: "string" }, "workspace-repo": { type: "string" }, + storage: { type: "string" }, + "storage-faults": { type: "string" }, + "storage-fault-seconds": { type: "string" }, }, }); const keep = values.keep === true; const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000); const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000); if (values.replay !== undefined) { - for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo"] as const) { + for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds"] as const) { if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`); } return { mode: "replay", ref: parseReplayRef(values.replay), keep, @@ -174,15 +178,22 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC if (crashPoint) throw new Error("--upgrade-from does not combine with --crash-point"); options.upgradeFrom = path.resolve(upgradeFrom); } + const storage = values.storage ?? "local"; const workspaceRepo = values["workspace-repo"]; if (options.workloads.includes("R1") !== (workspaceRepo !== undefined)) { throw new Error("R1 (code change in a cloned repository) and --workspace-repo go together: pass both or neither"); } if (workspaceRepo !== undefined) { - if (env !== "local") throw new Error("--workspace-repo runs only with --env local"); + if (env !== "local" && storage !== "s3") throw new Error("--workspace-repo on Daytona needs --storage s3, so the clone reaches the sandbox as a remote workspace"); if (options.workloads.some(id => id === "B1" || id === "B5")) throw new Error("--workspace-repo replaces the chaoscli fixture that B1 and B5 need"); options.workspaceRepo = path.resolve(workspaceRepo); - options.workspaceCommit = execFileSync("git", ["-C", options.workspaceRepo, "rev-parse", "HEAD"], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim(); + const git = (...args: string[]) => execFileSync("git", ["-C", options.workspaceRepo!, ...args], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim(); + if (env === "daytona") { + options.workspaceGitUrl = publicGitHubUrl(git("remote", "get-url", "origin")); + options.workspaceCommit = git("rev-parse", "origin/master"); + } else { + options.workspaceCommit = git("rev-parse", "HEAD"); + } } // H1 pauses the whole company when its budget hard stop is crossed: no other task can share its // wave, its tasks run one at a time, and the hard stop must be on. @@ -204,9 +215,28 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC + "for chaos tasks. Lower one of them (RESTART_CHAOS_DAYTONA_VCPU_BUDGET sets the budget)."); } } + if (storage !== "local" && storage !== "s3") throw new Error("--storage must be local or s3"); + if (storage === "s3") options.storage = "s3"; + if (values["storage-faults"] !== undefined) { + if (storage !== "s3") throw new Error("--storage-faults needs --storage s3"); + options.storageFaults = list<"ST1" | "ST2" | "ST3">(values["storage-faults"], STORAGE_FAULT_KINDS, "storage-faults", []); + } + if (values["storage-fault-seconds"] !== undefined) { + if (!options.storageFaults?.length) throw new Error("--storage-fault-seconds needs --storage-faults"); + const match = /^(\d+)-(\d+)$/.exec(values["storage-fault-seconds"]); + const [min, max] = match ? [Number(match[1]), Number(match[2])] : [0, -1]; + if (!match || min < 5 || max < min || max > 900) throw new Error("--storage-fault-seconds takes - seconds, 5 to 900, min not above max"); + options.storageFaultMs = [min * 1000, max * 1000]; + } return { mode: "wave", options, keep }; } +export function publicGitHubUrl(remote: string): string { + const match = /^(?:https:\/\/github\.com\/|git@github\.com:|ssh:\/\/git@github\.com\/)([^/\s]+\/[^/\s]+?)(?:\.git)?\/?$/.exec(remote.trim()); + if (!match) throw new Error(`--workspace-repo on Daytona needs a GitHub origin the sandbox can clone over https; origin is ${remote}`); + return `https://github.com/${match[1]}.git`; +} + export function waveIdFor(seed: number, at = new Date()): string { const stamp = at.toISOString().replace(/[-:]/g, "").replace(/\.\d{3}Z$/, ""); return `w-${stamp}-s${seed}`; diff --git a/tests/restart-chaos/grader.test.ts b/tests/restart-chaos/grader.test.ts index 424399704b5..cf6aed66e04 100644 --- a/tests/restart-chaos/grader.test.ts +++ b/tests/restart-chaos/grader.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { crossingFaults, gradeR3, gradeR4, gradeR5, gradeTask, STALL_LIMIT_MS } from "./grader.js"; +import { crossingFaults, gradeR2, gradeR3, gradeR4, gradeR5, gradeTask, STALL_LIMIT_MS } from "./grader.js"; import { isResting } from "./observer.js"; import type { FaultReceipt, OracleArtifacts, RunSample, TaskRecord, TaskSample } from "./types.js"; import { BENCH_EXPECTED, parseBenchRecord, r1TestChecksSlug, stallWakeChecks, workloadDefinitions } from "./workloads.js"; @@ -238,6 +238,21 @@ describe("T-oracles", () => { expect(stallWakeChecks({ ...stalled, childStalls: [] } as never)).toEqual([]); }); + it("grades a Retry the user could not click in the board UI under its own R3 signature", () => { + const record = { plan: { workload: "A1" }, launchError: null, finalStatus: "blocked", timedOut: false, createdAt: 0, restAt: 1, + samples: [{ at: 1, reachable: true, serverReady: true, issueStatus: "blocked", interruption: { state: "failed", canRetry: true } }], + retry: { at: 2, observedRunId: "r1", httpStatus: null, accepted: null, error: "ui_retry_button_missing", via: "ui", bannerText: "Saving files" } } as unknown as TaskRecord; + expect(gradeR3(record)).toEqual({ verdict: "fail", detail: "ui_retry_button_missing (user saw: Saving files)" }); + expect(gradeR3({ ...record, retry: { ...record.retry!, clicked: true, error: "ui_retry_no_request" } } as TaskRecord).detail).toMatch(/^ui_retry_no_request/); + }); + + it("reports R2 as not checked for a sandbox-only workspace instead of passing it", () => { + const record = { plan: { workload: "R1" }, launchError: null, finalStatus: "done", timedOut: false, createdAt: 0, restAt: 1, + remoteWorkspace: true, preFaultFiles: null, samples: [{ at: 1, reachable: true, serverReady: true, issueStatus: "done", interruption: null }] } as unknown as TaskRecord; + expect(gradeR2(record, { verdict: "pass", checks: [] })).toEqual({ verdict: "n/a", + detail: "remote workspace: files before and after the fault are not compared from this host" }); + }); + it("grades S1: stopped run cancelled, nothing ran before the continue comment, brief has both markers", () => { const base: OracleArtifacts = { issueStatus: "done", documents: { brief: "CHAOS-abcd1234\nCONTINUE-abcd1234\n- one\n- two\n- three\n" }, agentComments: [], files: {}, commands: {}, workspacePath: null, diff --git a/tests/restart-chaos/grader.ts b/tests/restart-chaos/grader.ts index 282473f4ecc..4fc16669af6 100644 --- a/tests/restart-chaos/grader.ts +++ b/tests/restart-chaos/grader.ts @@ -1,4 +1,4 @@ -import type { FaultReceipt, OracleResult, RuleId, RuleResult, TaskGrade, TaskRecord, TaskSample } from "./types.js"; +import { UI_RETRY_BUTTON_MISSING, UI_RETRY_NO_REQUEST, type FaultReceipt, type OracleResult, type RuleId, type RuleResult, type TaskGrade, type TaskRecord, type TaskSample } from "./types.js"; import { g1Decision } from "./workloads.js"; /** Pure R1–R6 grading over observer samples (plan §2). No I/O. */ @@ -89,6 +89,9 @@ export function gradeR2(record: TaskRecord, oracle: OracleResult): RuleResult { if (fileChecks.length && record.finalStatus === "done") { return { verdict: "fail", detail: `workspace_files_missing_after_done: ${fileChecks.map(check => check.name).join(",")}` }; } + if (record.remoteWorkspace && !record.preFaultFiles) { + return { verdict: "n/a", detail: "remote workspace: files before and after the fault are not compared from this host" }; + } const transient = record.samples.some(sample => sample.interruption?.workspaceFailure); return { verdict: "pass", detail: transient ? "workspace failure appeared during recovery and cleared" : "no workspace failure at rest" }; } @@ -108,6 +111,9 @@ function gradeOwnRetry(record: TaskRecord): RuleResult { } return { verdict: "n/a", detail: "no Retry needed" }; } + if (record.retry.via === "ui" && (record.retry.error === UI_RETRY_BUTTON_MISSING || record.retry.error === UI_RETRY_NO_REQUEST)) { + return { verdict: "fail", detail: `${record.retry.error}${record.retry.bannerText ? ` (user saw: ${record.retry.bannerText})` : ""}` }; + } if (record.retry.error || record.retry.accepted !== true) { return { verdict: "fail", detail: `retry_rejected: http=${record.retry.httpStatus ?? "none"} accepted=${String(record.retry.accepted)} ${record.retry.error ?? ""}`.trim() }; } diff --git a/tests/restart-chaos/main.ts b/tests/restart-chaos/main.ts index 9cd7061d9fb..10d1f767afb 100644 --- a/tests/restart-chaos/main.ts +++ b/tests/restart-chaos/main.ts @@ -13,6 +13,10 @@ import os from "node:os"; import path from "node:path"; import { setTimeout as delay } from "node:timers/promises"; import { ChaosApi, isSocketTypeOfServiceRace } from "./api.js"; +import { BoardUi } from "./board-ui.js"; +import { DEFAULT_MINIO_BINARY, MINIO_RELEASE, minioBinary, objectStoreServerEnvironment, startMinio, type ObjectStore } from "./object-store.js"; +import { StorageChaos, planStorageFaults, replayStorageWindows, type StorageFaultReceipt } from "./storage-chaos.js"; +import { startStorageFaultProxy } from "./storage-proxy.js"; import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from "./config.js"; import { assertControllerMatchesImage, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js"; import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js"; @@ -55,6 +59,11 @@ async function loadPlan(): Promise<{ plan: WavePlan; keep: boolean }> { const observed = crossed ? observedFaultSchedule(crossed.createdAt, crossed.faultIds, faults, command.ref.taskId) : []; if (!observed.length) console.warn("Replay: no recorded fault sequence for this task; using its planned fault only."); let plan = replayWave(source, command.ref, Date.now(), command.overrides, observed); + if (source.options.storage === "s3" && crossed?.createdAt) { + const receipts = await readFile(path.join(sourceDir, "storage-faults.json"), "utf8").then(text => JSON.parse(text) as StorageFaultReceipt[]).catch(() => []); + const ended = Math.max(crossed.createdAt, ...receipts.map(receipt => receipt.endedAt ?? 0)); + plan = { ...plan, options: { ...plan.options, storageWindows: replayStorageWindows(receipts, crossed.createdAt, ended) } }; + } // K14: a task that launched after the upgrade replays on the current build. const launchBuild = graded?.record.serverBuildAtLaunch ?? logged?.[command.ref.taskId]?.serverBuildAtLaunch; if (plan.options.upgradeFrom && launchBuild === "current") plan = replayOnCurrentBuild(plan); @@ -211,6 +220,10 @@ async function main(): Promise { let anyFailure = false; let stopFailed = false; let remoteCleanupPending = false; + let board: BoardUi | null = null; + let objectStore: ObjectStore | null = null; + let storageChaos: StorageChaos | null = null; + let storageProxy: Awaited> | null = null; let speedSection: string | null = null; let harnessError: unknown; process.on("uncaughtException", error => { @@ -237,6 +250,14 @@ async function main(): Promise { let faultFailures = 0; let faultsQueued = 0; try { + if (plan.options.storage === "s3") { + const binary = minioBinary(); + if (!binary) throw new Error(`--storage s3 needs MinIO ${MINIO_RELEASE} at ${DEFAULT_MINIO_BINARY} or RESTART_CHAOS_MINIO_BIN`); + objectStore = await startMinio({ binary, root: supervisor.paths.temporaryRoot }); + storageProxy = await startStorageFaultProxy(objectStore.endpoint); + Object.assign(serverEnvironment, objectStoreServerEnvironment(objectStore, storageProxy.endpoint, `chaos/${plan.waveId}/`)); + timeline.add("object_store_started", { release: MINIO_RELEASE, endpoint: objectStore.endpoint, proxy: storageProxy.endpoint, bucket: objectStore.bucket }); + } await supervisor.start(serverEnvironment); const api = new ChaosApi(supervisor.baseUrl); const lab = await seedChaosLab({ api, env: plan.options.env, waitForReadyAgents: true, cancelled: () => interrupted, daytonaImage: plan.options.daytonaImage, daytonaCpu: plan.options.daytonaCpu, @@ -245,7 +266,7 @@ async function main(): Promise { ...(plan.tasks.some(task => task.workload === "M1") ? plan.options.adapters : [])])], concurrency: plan.options.concurrency, budgetCents: plan.options.budgetCents, masks: plan.options.masks, fixtureRoot: path.join(supervisor.paths.temporaryRoot, "fixtures"), workspaceRepo: plan.options.workspaceRepo, - workspaceCommit: plan.options.workspaceCommit, waveId: plan.waveId, timeline, + workspaceCommit: plan.options.workspaceCommit, workspaceGitUrl: plan.options.workspaceGitUrl, waveId: plan.waveId, timeline, adapterConfigOverrides: chaosLifecycleOverrides(plan.options.lifecycle), onEnvironment: owned => { sweeper = new DaytonaWaveSweeper({ apiKey: process.env.DAYTONA_API_KEY!, companyId: owned.companyId, environmentId: owned.environmentId, api }); @@ -253,7 +274,8 @@ async function main(): Promise { spendTarget = { api, companyId: lab.companyId }; // The instance root holds the secrets master key and the database: oracle code may not read it. const protectedRoots = [os.homedir(), supervisor.paths.temporaryRoot, resultsRoot]; - const observers = plan.tasks.map(task => new TaskObserver(task, api, lab, timeline, supervisor, protectedRoots)); + board = serverEnvironment.SERVE_UI === "false" ? null : new BoardUi(supervisor.baseUrl); + const observers = plan.tasks.map(task => new TaskObserver(task, api, lab, timeline, supervisor, protectedRoots, board)); const pending = [...observers]; const launched: TaskObserver[] = []; const active: TaskObserver[] = []; @@ -261,6 +283,13 @@ async function main(): Promise { let lastDoneAt = Date.now(); // K13: storm faults fire on their own clock while tasks keep launching; then the storm drains. const storm = plan.storm ? new StormRun(plan.storm, Date.now()) : null; + if (storageProxy && (plan.options.storageWindows?.length || plan.options.storageFaults?.length)) { + const windows = plan.options.storageWindows ?? planStorageFaults({ kinds: plan.options.storageFaults!, seed: plan.seed, + spanMs: plan.storm?.durationMs ?? plan.tasks.length * 4 * 60_000, ...(plan.options.storageFaultMs ? { durationMs: plan.options.storageFaultMs } : {}) }); + storageChaos = new StorageChaos(windows, Date.now(), storageProxy, (kind, data) => timeline.add(kind, data)); + if (!plan.options.storageWindows) storageChaos.start(); + timeline.add("storage_faults_planned", { windows: windows.map(window => `${window.kind}@${window.startMs}ms+${window.durationMs}ms`) }); + } if (storm) timeline.add("storm_started", { durationMs: storm.plan.durationMs, plannedFaults: storm.plan.faults.length }); // Faults are serialized; each one hits every task in flight when it starts. const fireFault = (kind: FaultKind, holdMs: number, requestedBy: string, exposed: readonly TaskObserver[], @@ -373,6 +402,7 @@ async function main(): Promise { const next = pending.shift()!; launched.push(next); await next.launch(); + if (plan.options.storageWindows && storageChaos && launched.length === 1) storageChaos.start(); if (!next.done) active.push(next); } await Promise.all(active.filter(observer => now - (lastPoll.get(observer) ?? 0) >= plan.options.pollMs).map(async observer => { @@ -518,6 +548,7 @@ async function main(): Promise { } finally { // Best effort for an interrupted or failed wave; the server may already be down. if (supervisor.serverReady) await recordWaveSpend().catch(() => undefined); + await board?.close().catch(() => undefined); // A failure here (for example writing the report) must not skip stopping the controller and Postgres. try { if (sweeper) try { const owned = sweeper as DaytonaWaveSweeper; @@ -556,6 +587,13 @@ async function main(): Promise { timeline.add("supervisor_stop_failed", { error: String(error) }); } } + storageChaos?.stop(); + if (storageProxy) { + if (storageChaos) await writeFile(path.join(waveDir, "storage-faults.json"), `${JSON.stringify(storageChaos.receipts, null, 2)}\n`, { mode: 0o600 }).catch(() => undefined); + await writeFile(path.join(waveDir, "storage-proxy.json"), `${JSON.stringify(storageProxy.counts(), null, 2)}\n`, { mode: 0o600 }).catch(() => undefined); + await storageProxy.close().catch(() => undefined); + } + await objectStore?.stop().catch(error => timeline.add("object_store_stop_failed", { error: String(error) })); const retention = await releaseInstance(supervisor.paths.temporaryRoot, { keep, stopFailed, remoteCleanupPending, owner: plan.waveId }) .catch(error => { timeline.add("instance_cleanup_failed", { error: String(error) }); return null; }); if (retention?.kept) timeline.add("instance_kept", { temporaryRoot: supervisor.paths.temporaryRoot, reason: retention.reason }); diff --git a/tests/restart-chaos/object-store.test.ts b/tests/restart-chaos/object-store.test.ts new file mode 100644 index 00000000000..f7abc70543a --- /dev/null +++ b/tests/restart-chaos/object-store.test.ts @@ -0,0 +1,49 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import { createRequire } from "node:module"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { minioBinary, objectStoreServerEnvironment, startMinio } from "./object-store.js"; +import { startStorageFaultProxy } from "./storage-proxy.js"; + +const binary = minioBinary(); + +describe("object store", () => { + it("points the server's storage settings at the given endpoint, path-style", () => { + expect(objectStoreServerEnvironment({ bucket: "b", region: "us-east-1", accessKeyId: "k", secretAccessKey: "s" }, "http://127.0.0.1:9", "chaos/w1/")) + .toEqual({ PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: "b", PAPERCLIP_STORAGE_S3_REGION: "us-east-1", + PAPERCLIP_STORAGE_S3_ENDPOINT: "http://127.0.0.1:9", PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE: "true", PAPERCLIP_STORAGE_S3_PREFIX: "chaos/w1/", + AWS_ACCESS_KEY_ID: "k", AWS_SECRET_ACCESS_KEY: "s", AWS_SESSION_TOKEN: "", AWS_CREDENTIAL_EXPIRATION: "", AWS_EC2_METADATA_DISABLED: "true" }); + }); + + it("fails fast with the binary's path when MinIO cannot start", async () => { + const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-minio-bad-")); + try { + const started = Date.now(); + await expect(startMinio({ binary: path.join(root, "not-a-binary"), root, readyTimeoutMs: 20_000 })).rejects.toThrow(/could not start from .*not-a-binary/); + expect(Date.now() - started).toBeLessThan(5_000); + } finally { await rm(root, { recursive: true, force: true }); } + }); + + it.skipIf(!binary)("starts a loopback MinIO that honours conditional writes through the fault proxy", async () => { + const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-minio-")); + const store = await startMinio({ binary: binary!, root }); + const proxy = await startStorageFaultProxy(store.endpoint); + const { S3Client, PutObjectCommand } = createRequire(new URL("../../server/package.json", import.meta.url))("@aws-sdk/client-s3"); + const client = new S3Client({ endpoint: proxy.endpoint, region: store.region, forcePathStyle: true, + credentials: { accessKeyId: store.accessKeyId, secretAccessKey: store.secretAccessKey } }); + try { + await client.send(new PutObjectCommand({ Bucket: store.bucket, Key: "a", Body: "1", IfNoneMatch: "*" })); + await expect(client.send(new PutObjectCommand({ Bucket: store.bucket, Key: "a", Body: "2", IfNoneMatch: "*" }))) + .rejects.toMatchObject({ $metadata: { httpStatusCode: 412 } }); + proxy.setRules([{ fault: { kind: "outage" }, methods: ["PUT"], remaining: 10 }]); + await expect(client.send(new PutObjectCommand({ Bucket: store.bucket, Key: "b", Body: "3" }))).rejects.toMatchObject({ $metadata: { httpStatusCode: 503 } }); + expect(proxy.counts().injected).toBeGreaterThanOrEqual(1); + } finally { + client.destroy(); + await proxy.close(); + await store.stop(); + await rm(root, { recursive: true, force: true }); + } + }, 60_000); +}); diff --git a/tests/restart-chaos/object-store.ts b/tests/restart-chaos/object-store.ts new file mode 100644 index 00000000000..f7f03555a6a --- /dev/null +++ b/tests/restart-chaos/object-store.ts @@ -0,0 +1,98 @@ +import { spawn, type ChildProcess } from "node:child_process"; +import { randomBytes } from "node:crypto"; +import { existsSync } from "node:fs"; +import { mkdir } from "node:fs/promises"; +import { createRequire } from "node:module"; +import { createServer } from "node:net"; +import os from "node:os"; +import path from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; + +export const MINIO_RELEASE = "RELEASE.2025-04-22T22-12-26Z"; +export const DEFAULT_MINIO_BINARY = path.join(os.homedir(), ".cache", "paperclip", "tools", "minio", "minio"); + +export function minioBinary(environment: NodeJS.ProcessEnv = process.env): string | null { + const candidate = environment.RESTART_CHAOS_MINIO_BIN || DEFAULT_MINIO_BINARY; + return existsSync(candidate) ? candidate : null; +} + +async function freeLoopbackPort(): Promise { + const server = createServer(); + await new Promise((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); }); + const port = (server.address() as { port: number }).port; + await new Promise(resolve => server.close(() => resolve())); + return port; +} + +export interface ObjectStore { + endpoint: string; + bucket: string; + region: string; + accessKeyId: string; + secretAccessKey: string; + dataDir: string; + stop(): Promise; +} + +export function objectStoreServerEnvironment(store: Pick, + endpoint: string, prefix: string): Record { + return { + PAPERCLIP_STORAGE_PROVIDER: "s3", + PAPERCLIP_STORAGE_S3_BUCKET: store.bucket, + PAPERCLIP_STORAGE_S3_REGION: store.region, + PAPERCLIP_STORAGE_S3_ENDPOINT: endpoint, + PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE: "true", + PAPERCLIP_STORAGE_S3_PREFIX: prefix, + AWS_ACCESS_KEY_ID: store.accessKeyId, + AWS_SECRET_ACCESS_KEY: store.secretAccessKey, + AWS_SESSION_TOKEN: "", + AWS_CREDENTIAL_EXPIRATION: "", + AWS_EC2_METADATA_DISABLED: "true", + }; +} + +export async function startMinio(input: { binary: string; root: string; bucket?: string; readyTimeoutMs?: number }): Promise { + const dataDir = path.join(input.root, "object-store"); + await mkdir(dataDir, { recursive: true, mode: 0o700 }); + const [port, consolePort] = [await freeLoopbackPort(), await freeLoopbackPort()]; + const accessKeyId = `chaos${randomBytes(6).toString("hex")}`; + const secretAccessKey = randomBytes(24).toString("hex"); + const child: ChildProcess = spawn(input.binary, ["server", dataDir, "--address", `127.0.0.1:${port}`, "--console-address", `127.0.0.1:${consolePort}`, "--quiet"], { + env: { PATH: process.env.PATH ?? "", HOME: input.root, MINIO_ROOT_USER: accessKeyId, MINIO_ROOT_PASSWORD: secretAccessKey, + MINIO_BROWSER: "off", MINIO_UPDATE: "off" }, + stdio: ["ignore", "ignore", "pipe"], + }); + let stderr = ""; + let spawnError: Error | null = null; + child.on("error", error => { spawnError = error; }); + child.stderr?.on("data", chunk => { stderr = `${stderr}${chunk}`.slice(-4_000); }); + const endpoint = `http://127.0.0.1:${port}`; + const stop = async () => { + if (spawnError || child.pid === undefined || child.exitCode !== null || child.signalCode !== null) return; + const exited = new Promise(resolve => child.once("exit", () => resolve())); + child.kill("SIGTERM"); + await Promise.race([exited, delay(10_000).then(() => { child.kill("SIGKILL"); return exited; })]); + }; + const deadline = Date.now() + (input.readyTimeoutMs ?? 30_000); + for (;;) { + if (spawnError) throw new Error(`MinIO could not start from ${input.binary}: ${(spawnError as Error).message}`); + if (child.exitCode !== null) throw new Error(`MinIO exited before it was ready (${child.exitCode}): ${stderr}`); + const live = await fetch(`${endpoint}/minio/health/live`).then(response => response.ok).catch(() => false); + if (live) break; + if (Date.now() > deadline) { await stop(); throw new Error(`MinIO did not become ready: ${stderr}`); } + await delay(200); + } + const bucket = input.bucket ?? `chaos-${randomBytes(4).toString("hex")}`; + const region = "us-east-1"; + const { S3Client, CreateBucketCommand } = createRequire(new URL("../../server/package.json", import.meta.url))("@aws-sdk/client-s3"); + const client = new S3Client({ endpoint, region, forcePathStyle: true, credentials: { accessKeyId, secretAccessKey } }); + try { + await client.send(new CreateBucketCommand({ Bucket: bucket })); + } catch (error) { + await stop(); + throw error; + } finally { + client.destroy(); + } + return { endpoint, bucket, region, accessKeyId, secretAccessKey, dataDir, stop }; +} diff --git a/tests/restart-chaos/observer.ts b/tests/restart-chaos/observer.ts index 226b53d5b32..94397449e30 100644 --- a/tests/restart-chaos/observer.ts +++ b/tests/restart-chaos/observer.ts @@ -1,6 +1,6 @@ import { spawnSync } from "node:child_process"; import { randomUUID } from "node:crypto"; -import { readFile, readdir } from "node:fs/promises"; +import { access, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { setTimeout as delay } from "node:timers/promises"; @@ -13,6 +13,7 @@ import type { TaskSample, } from "./types.js"; import { restartProcessStartMatches } from "../runner-e2e/process-tree-owner.js"; +import type { UiRetryClick } from "./board-ui.js"; import { runOracleCommand } from "./oracle-sandbox.js"; import { taskFaultSchedule } from "./scheduler.js"; import { g1ChildTitle, g1Decision, workloadDefinitions, type WorkloadAgents } from "./workloads.js"; @@ -192,7 +193,8 @@ export class TaskObserver { private readonly timeline: Timeline, private readonly supervisor: { serverReady: boolean; faultInProgress: boolean; serverBuild?: "upgrade_from" | "current" }, /** Paths oracle commands may not read: the operator's home, the instance root (secrets key, database) and results. */ - private readonly protectedRoots: readonly string[] = [os.homedir()]) { + private readonly protectedRoots: readonly string[] = [os.homedir()], + private readonly board: { clickRetry(identifier: string, issueId: string): Promise } | null = null) { this.record = { plan, issueId: null, identifier: null, createdAt: null, launchError: null, samples: [], retry: null, restAt: null, finalStatus: null, deadlineAt: null, timedOut: false, followUp: null, faultIds: [], sweep: null, preFaultFiles: null, lostFiles: null }; @@ -394,7 +396,11 @@ export class TaskObserver { // One task's snapshot failure must not end the wave; R2 fails with the reason. try { const workspace = await this.workspacePath(); - if (workspace) { this.record.preFaultFiles = await listWorkspaceFiles(workspace); this.record.preFaultWorkspacePath = workspace; } + if (workspace && this.lab.environmentKind !== "local" && !await access(workspace).then(() => true, () => false)) { + this.record.remoteWorkspace = true; + } else if (workspace) { + this.record.preFaultFiles = await listWorkspaceFiles(workspace); this.record.preFaultWorkspacePath = workspace; + } } catch (error) { this.record.preFaultSnapshotError = error instanceof Error ? error.message : String(error); this.timeline.add("pre_fault_snapshot_failed", { taskId: this.plan.taskId, error: this.record.preFaultSnapshotError }); @@ -422,6 +428,7 @@ export class TaskObserver { if (this.done || !this.record.issueId) return; const sample = await this.sample(now); this.record.samples.push(sample); + await this.captureRemoteFilesAtDone(sample); this.readySince = sample.reachable && sample.serverReady ? (this.readySince ?? now) : null; await this.observeBenchProgress(); await this.governanceStep(sample, now); @@ -637,16 +644,8 @@ export class TaskObserver { const interruption = child?.interruption as Json | null | undefined; if (interruption?.canRetry !== true) continue; const observedRunId = typeof interruption.observedRunId === "string" ? interruption.observedRunId : null; - const attempt: RetryAttempt & { childIssueId: string } = { childIssueId, at: now, observedRunId, httpStatus: null, accepted: null, error: null }; - try { - const result = await this.api.post<{ accepted?: boolean }>(`/api/issues/${childIssueId}/retry`, { idempotencyKey: randomUUID(), observedRunId }); - attempt.httpStatus = 200; - attempt.accepted = result?.accepted === true; - attempt.acceptedAt = Math.max(now, Date.now()); - } catch (error) { - attempt.httpStatus = error instanceof ChaosApiError ? error.status : null; - attempt.error = error instanceof Error ? error.message : String(error); - } + const childRef = typeof child?.identifier === "string" ? child.identifier : childIssueId; + const attempt: RetryAttempt & { childIssueId: string } = { childIssueId, ...await this.clickRetry(childRef, childIssueId, observedRunId, now) }; (this.record.childRetries ??= []).push(attempt); this.timeline.add("task_child_retry_clicked", { taskId: this.plan.taskId, ...attempt }); clicked = clicked || attempt.accepted === true; @@ -665,10 +664,22 @@ export class TaskObserver { private async retry(sample: TaskSample, now: number): Promise { const observedRunId = sample.interruption?.observedRunId ?? (this.issue?.executionBlocker?.runId as string | undefined) ?? null; + const attempt = await this.clickRetry(this.record.identifier ?? this.record.issueId!, this.record.issueId!, observedRunId, now); + this.timeline.add("task_retry_clicked", { taskId: this.plan.taskId, ...attempt }); + return attempt; + } + + private async clickRetry(issueRef: string, issueId: string, observedRunId: string | null, now: number): Promise { const attempt: RetryAttempt = { at: now, observedRunId, httpStatus: null, accepted: null, error: null }; + if (this.board) { + const click = await this.board.clickRetry(issueRef, issueId); + Object.assign(attempt, { via: "ui", httpStatus: click.httpStatus, accepted: click.accepted, error: click.error, bannerText: click.bannerText }); + if (click.accepted) attempt.acceptedAt = Math.max(now, Date.now()); + return attempt; + } + attempt.via = "api"; try { - const result = await this.api.post<{ accepted?: boolean }>(`/api/issues/${this.record.issueId}/retry`, - { idempotencyKey: randomUUID(), observedRunId }); + const result = await this.api.post<{ accepted?: boolean }>(`/api/issues/${issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId }); attempt.httpStatus = 200; attempt.accepted = result?.accepted === true; attempt.acceptedAt = Math.max(now, Date.now()); @@ -676,10 +687,27 @@ export class TaskObserver { attempt.httpStatus = error instanceof ChaosApiError ? error.status : null; attempt.error = error instanceof Error ? error.message : String(error); } - this.timeline.add("task_retry_clicked", { taskId: this.plan.taskId, ...attempt }); return attempt; } + private async remoteWorkspaceFile(issueId: string, relative: string): Promise { + const captured = this.record.remoteFiles?.[relative]; + if (typeof captured === "string") return captured; + return await this.api.getText(`/api/issues/${issueId}/file-resources/content?path=${encodeURIComponent(relative)}&download=true`, 15_000) + .catch(() => null); + } + + private async captureRemoteFilesAtDone(sample: TaskSample): Promise { + const wanted = this.definition.collect.files(this.plan.nonce); + if (!wanted.length || this.lab.environmentKind === "local" || sample.issueStatus !== "done" || !this.record.issueId) return; + const captured = this.record.remoteFiles ??= {}; + for (const relative of wanted) { + if (typeof captured[relative] === "string") continue; + captured[relative] = await this.api.getText(`/api/issues/${this.record.issueId}/file-resources/content?path=${encodeURIComponent(relative)}&download=true`, 15_000) + .catch(() => null); + } + } + async workspacePath(): Promise { if (!this.definition.usesRepository) return null; const current = this.issue?.currentExecutionWorkspace as Json | null | undefined; @@ -876,16 +904,30 @@ export class TaskObserver { } const files: Record = {}; const commands: OracleArtifacts["commands"] = {}; - if (workspacePath) { - for (const relative of definition.collect.files(this.plan.nonce)) { - files[relative] = await readFile(path.join(workspacePath, relative), "utf8").catch(() => null); - } - // Agent-editable code: run it sandboxed, away from the operator's home directory. + const runCommands = async (cwd: string) => { for (const command of definition.collect.commands) { const argv = typeof command.argv === "function" ? command.argv(this.plan.nonce) : command.argv; - commands[command.name] = await runOracleCommand(argv, workspacePath, { protectedRoots: this.protectedRoots, + commands[command.name] = await runOracleCommand(argv, cwd, { protectedRoots: this.protectedRoots, readableFiles: argv[0] === process.execPath ? [process.execPath] : [] }); } + }; + const wanted = definition.collect.files(this.plan.nonce); + if (workspacePath && await access(workspacePath).then(() => true, () => false)) { + for (const relative of wanted) files[relative] = await readFile(path.join(workspacePath, relative), "utf8").catch(() => null); + await runCommands(workspacePath); + } else if (workspacePath && wanted.length) { + const staged = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-oracle-")); + try { + for (const relative of wanted) { + files[relative] = await this.remoteWorkspaceFile(issueId, relative); + if (files[relative] === null) continue; + await mkdir(path.dirname(path.join(staged, relative)), { recursive: true }); + await writeFile(path.join(staged, relative), files[relative]!); + } + await runCommands(staged); + } finally { + await rm(staged, { recursive: true, force: true }); + } } return { issueStatus: issue?.status ?? this.record.finalStatus, diff --git a/tests/restart-chaos/remote-oracle.test.ts b/tests/restart-chaos/remote-oracle.test.ts new file mode 100644 index 00000000000..2e7c81fc180 --- /dev/null +++ b/tests/restart-chaos/remote-oracle.test.ts @@ -0,0 +1,109 @@ +import { createServer, type Server } from "node:http"; +import { once } from "node:events"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { ChaosApi } from "./api.js"; +import { TaskObserver } from "./observer.js"; +import type { ChaosLab } from "./seed.js"; +import { Timeline } from "./timeline.js"; +import type { TaskPlan } from "./types.js"; + +const nonce = "abcd1234"; +const modulePath = `packages/shared/src/chaos/slug-${nonce}.ts`; +const testPath = `packages/shared/src/chaos/slug-${nonce}.test.ts`; +const remoteFiles: Record = { + [modulePath]: `export function chaosSlug(input: string): string { + return input.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, ""); +} +`, + [testPath]: `import { test } from "node:test"; +import assert from "node:assert/strict"; +import { chaosSlug } from "./slug-${nonce}.ts"; +test("slug", () => { + assert.equal(chaosSlug("Hello Chaos World"), "hello-chaos-world"); + assert.equal(chaosSlug(" a__b "), "a-b"); + assert.equal(chaosSlug("x"), "x"); +}); +`, +}; +const lab = { companyId: "c1", projectId: "p1", environmentId: "e1", environmentKind: "daytona", fixtureRepositoryPath: "/x", + agents: { codex: { id: "a1", name: "Chaos codex" }, claude: { id: "a2", name: "Chaos claude" } } } as unknown as ChaosLab; +const plan: TaskPlan = { taskId: "t01", index: 0, workload: "R1", adapter: "codex", fault: "K1", fireAfterMs: 999_999, + holdMs: 0, followUpAfterMs: null, nonce }; +let server: Server; +let sandboxReleased = false; +let baseUrl: string; + +beforeAll(async () => { + server = createServer((incoming, outgoing) => { + const url = new URL(incoming.url ?? "/", "http://x"); + if (url.pathname === "/api/issues/i1/file-resources/content" && url.searchParams.get("download") === "true") { + if (incoming.headers.origin !== baseUrl) { outgoing.writeHead(403).end(); return; } + if (sandboxReleased) { outgoing.writeHead(422, { "content-type": "application/json" }).end(JSON.stringify({ code: "workspace_unavailable" })); return; } + const body = remoteFiles[url.searchParams.get("path") ?? ""]; + outgoing.writeHead(body ? 200 : 404, { "content-type": "text/plain" }).end(body ?? ""); + return; + } + outgoing.writeHead(404).end(); + }); + server.listen(0, "127.0.0.1"); await once(server, "listening"); + baseUrl = `http://127.0.0.1:${(server.address() as { port: number }).port}`; +}); +afterAll(async () => { server.closeAllConnections(); server.close(); }); + +describe("oracle for a remote workspace", () => { + it("reads the task's files through the board API and runs the oracle commands on a local copy", async () => { + const api = { + baseUrl, + async get(route: string) { + if (route.endsWith("/comments")) return []; + return { id: "i1", status: "done", currentExecutionWorkspace: { cwd: "/home/daytona/workspaces/not-on-this-host" } }; + }, + async post() { return { id: "i1", identifier: "CHA-1" }; }, + getText: (route: string) => new ChaosApi(baseUrl).getText(route), + } as unknown as ChaosApi; + const observer = new TaskObserver(plan, api, lab, new Timeline(null), { serverReady: true, faultInProgress: false }); + await observer.launch(); + sandboxReleased = false; + const artifacts = await observer.collectArtifacts(); + expect(artifacts.files).toEqual(remoteFiles); + expect(artifacts.commands.node_test).toMatchObject({ exitCode: 0 }); + expect(artifacts.commands.slug?.stdout).toContain("hello-chaos-world"); + }, 60_000); + + it("skips the local pre-fault listing for a workspace that lives in the sandbox, instead of failing R2", async () => { + const api = { + baseUrl, + async get(route: string) { return route.endsWith("/runs") ? [] : { id: "i1", status: "in_progress", currentExecutionWorkspace: { cwd: "/home/daytona/workspaces/not-on-this-host" } }; }, + async post() { return { id: "i1", identifier: "CHA-1" }; }, + } as unknown as ChaosApi; + const observer = new TaskObserver(plan, api, lab, new Timeline(null), { serverReady: true, faultInProgress: false }); + await observer.launch(); + await observer.poll(observer.record.createdAt! + 1_000); + await observer.snapshotBeforeFault(); + expect(observer.record).toMatchObject({ preFaultFiles: null, remoteWorkspace: true }); + expect(observer.record.preFaultSnapshotError).toBeUndefined(); + }); + + it("captures the task's files when it first sees the task done, before the sandbox is released", async () => { + const api = { + baseUrl, + async get(route: string) { + if (route.endsWith("/runs") || route.endsWith("/comments")) return []; + return { id: "i1", status: "done", currentExecutionWorkspace: { cwd: "/home/daytona/workspaces/not-on-this-host" } }; + }, + async post() { return { id: "i1", identifier: "CHA-1" }; }, + getText: (route: string) => new ChaosApi(baseUrl).getText(route), + } as unknown as ChaosApi; + const observer = new TaskObserver(plan, api, lab, new Timeline(null), { serverReady: true, faultInProgress: false }); + await observer.launch(); + sandboxReleased = false; + await observer.poll(observer.record.createdAt! + 1_000); + sandboxReleased = true; + try { + const artifacts = await observer.collectArtifacts(); + expect(artifacts.files).toEqual(remoteFiles); + expect(artifacts.commands.node_test).toMatchObject({ exitCode: 0 }); + } finally { sandboxReleased = false; } + }, 60_000); +}); + diff --git a/tests/restart-chaos/scheduler.test.ts b/tests/restart-chaos/scheduler.test.ts index 455c884bbc5..9d5128d3488 100644 --- a/tests/restart-chaos/scheduler.test.ts +++ b/tests/restart-chaos/scheduler.test.ts @@ -1,5 +1,9 @@ +import { mkdtempSync, rmSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { execFileSync } from "node:child_process"; import { describe, expect, it } from "vitest"; -import { parseCommand, parseReplayRef, waveIdFor } from "./config.js"; +import { parseCommand, parseReplayRef, waveIdFor, publicGitHubUrl } from "./config.js"; import { chaosLifecycleOverrides } from "./seed.js"; import { K4_HOLD_RANGE_MS, STORM_GAP_RANGE_MS, createRng, observedFaultSchedule, planStorm, planTasks, planWave, replayWave, StormRun, stormActive, stormFaultDue, taskFaultSchedule, type StormState } from "./scheduler.js"; @@ -47,6 +51,37 @@ describe("scheduler determinism", () => { expect(single.options.workloads).not.toContain("M1"); expect(both.options.workloads).toContain("M1"); expect(() => parseCommand(["--adapters", "codex", "--workloads", "M1"])).toThrow(/two agents/); + const stored = parseCommand(["--storage", "s3", "--seed", "5"]); + if (stored.mode !== "wave") throw new Error("expected a wave"); + expect(stored.options.storage).toBe("s3"); + expect(both.options.storage).toBeUndefined(); + expect(() => parseCommand(["--storage", "gcs"])).toThrow(/local or s3/); + const faulted = parseCommand(["--storage", "s3", "--storage-faults", "ST1,ST3", "--seed", "5"]); + if (faulted.mode !== "wave") throw new Error("expected a wave"); + expect(faulted.options.storageFaults).toEqual(["ST1", "ST3"]); + expect(() => parseCommand(["--storage-faults", "ST1"])).toThrow(/needs --storage s3/); + const short = parseCommand(["--storage", "s3", "--storage-faults", "ST1", "--storage-fault-seconds", "15-45", "--seed", "5"]); + if (short.mode !== "wave") throw new Error("expected a wave"); + expect(short.options.storageFaultMs).toEqual([15_000, 45_000]); + expect(() => parseCommand(["--storage", "s3", "--storage-faults", "ST1", "--storage-fault-seconds", "45-15"])).toThrow(/storage-fault-seconds/); + expect(() => parseCommand(["--storage", "s3", "--storage-fault-seconds", "15-45"])).toThrow(/needs --storage-faults/); + const repo = mkdtempSync(path.join(os.tmpdir(), "restart-chaos-origin-")); + const git = (...args: string[]) => execFileSync("git", args, { cwd: repo, stdio: "pipe", encoding: "utf8" }).trim(); + git("init", "-q"); git("-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "--allow-empty", "-m", "base"); + git("remote", "add", "origin", "git@github.com:acme/widget.git"); + git("update-ref", "refs/remotes/origin/master", "HEAD"); + const masterCommit = git("rev-parse", "HEAD"); + expect(() => parseCommand(["--env", "daytona", "--daytona-image", "ghcr.io/x/y@sha256:" + "a".repeat(64), "--workloads", "R1", "--workspace-repo", repo])) + .toThrow(/needs --storage s3/); + const daytonaR1 = parseCommand(["--env", "daytona", "--daytona-image", "ghcr.io/x/y@sha256:" + "a".repeat(64), "--workloads", "R1", + "--workspace-repo", repo, "--storage", "s3", "--seed", "5"]); + if (daytonaR1.mode !== "wave") throw new Error("expected a wave"); + expect(daytonaR1.options).toMatchObject({ env: "daytona", storage: "s3", workloads: ["R1"] }); + expect(daytonaR1.options).toMatchObject({ workspaceGitUrl: "https://github.com/acme/widget.git", workspaceCommit: masterCommit }); + rmSync(repo, { recursive: true, force: true }); + expect(publicGitHubUrl("git@github.com:paperclipai/paperclip.git")).toBe("https://github.com/paperclipai/paperclip.git"); + expect(publicGitHubUrl("https://github.com/paperclipai/paperclip")).toBe("https://github.com/paperclipai/paperclip.git"); + expect(() => publicGitHubUrl("/Users/me/repo")).toThrow(/GitHub origin/); expect(() => parseCommand(["--k4-hold-max-seconds", "601"])).toThrow(/between 120 and 600/); expect(() => parseCommand(["--replay", "w-20261106T101112-s42/t03", "--k4-hold-max-seconds", "400"])).toThrow(/do not combine/); }); diff --git a/tests/restart-chaos/seed.test.ts b/tests/restart-chaos/seed.test.ts index 380fd27f808..17d70c30086 100644 --- a/tests/restart-chaos/seed.test.ts +++ b/tests/restart-chaos/seed.test.ts @@ -60,6 +60,15 @@ describe("seedChaosLab workspace isolation", () => { expect(skipping.calls.some(call => call.method === "GET" && call.route.startsWith("/api/agents/"))).toBe(false); }); + it("registers a remote git source, with no host folder, for a Daytona workspace repository", async () => { + const { api, calls } = fakeApi(body => ({ ...body })); + await seedChaosLab({ api, env: "local", adapters: ["codex"], concurrency: 2, budgetCents: 3_000, fixtureRoot: path.join(fixtureRoot, "remote"), + waveId: "w-test", timeline: new Timeline(null), workspaceGitUrl: "https://github.com/paperclipai/paperclip.git", workspaceCommit: "a".repeat(40) }); + const project = calls.find(call => call.method === "POST" && call.route.endsWith("/projects")); + expect((project!.body as { workspace: unknown }).workspace).toEqual({ name: "Primary", sourceType: "git_repo", + repoUrl: "https://github.com/paperclipai/paperclip.git", repoRef: "a".repeat(40), isPrimary: true }); + }); + // RC-21: the project's git_worktree policy is gated behind the instance's // isolated-workspaces setting. Without it every repo task ran in the shared // primary checkout, so concurrent tasks wrote into one tree. diff --git a/tests/restart-chaos/seed.ts b/tests/restart-chaos/seed.ts index bd3fd179197..f44a83be5ce 100644 --- a/tests/restart-chaos/seed.ts +++ b/tests/restart-chaos/seed.ts @@ -177,6 +177,7 @@ export async function seedChaosLab(input: { agentReadyTimeoutMs?: number; workspaceRepo?: string; workspaceCommit?: string; + workspaceGitUrl?: string; waveId: string; timeline: Timeline; /** Merged into each agent's adapterConfig (for example a runner lifecycle mode). */ @@ -284,7 +285,9 @@ export async function seedChaosLab(input: { environmentId, workspaceStrategy: { type: "git_worktree" }, }, - workspace: { name: "Primary", sourceType: "local_path", cwd: fixtureRepositoryPath, isPrimary: true }, + workspace: input.workspaceGitUrl + ? { name: "Primary", sourceType: "git_repo", repoUrl: input.workspaceGitUrl, repoRef: input.workspaceCommit, isPrimary: true } + : { name: "Primary", sourceType: "local_path", cwd: fixtureRepositoryPath, isPrimary: true }, }); timeline.add("seed_project", { projectId: project.id, fixtureRepositoryPath }); return { companyId: company.id, environmentId, environmentKind: input.env, projectId: project.id, fixtureRepositoryPath, agents }; diff --git a/tests/restart-chaos/storage-chaos.test.ts b/tests/restart-chaos/storage-chaos.test.ts new file mode 100644 index 00000000000..af19b8eb24b --- /dev/null +++ b/tests/restart-chaos/storage-chaos.test.ts @@ -0,0 +1,85 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { StorageChaos, activeStorageWindow, planStorageFaults, replayStorageWindows, type StorageFaultWindow } from "./storage-chaos.js"; +import type { StorageFaultRule } from "./storage-proxy.js"; + +describe("storage fault planning", () => { + it("plans seeded, non-overlapping windows of the requested kinds across the wave", () => { + const windows = planStorageFaults({ kinds: ["ST1", "ST2", "ST3"], seed: 7, spanMs: 30 * 60_000 }); + expect(windows.length).toBeGreaterThan(5); + expect(planStorageFaults({ kinds: ["ST1", "ST2", "ST3"], seed: 7, spanMs: 30 * 60_000 })).toEqual(windows); + expect(new Set(windows.map(window => window.kind))).toEqual(new Set(["ST1", "ST2", "ST3"])); + for (const [index, window] of windows.entries()) { + expect(window.durationMs).toBeGreaterThanOrEqual(60_000); + expect(window.durationMs).toBeLessThanOrEqual(180_000); + if (index) expect(window.startMs).toBeGreaterThanOrEqual(windows[index - 1]!.startMs + windows[index - 1]!.durationMs + 60_000); + if (window.kind === "ST1") expect(window.rules).toEqual([{ fault: { kind: "outage", status: 503 } }]); + if (window.kind === "ST2") expect(window.rules[0]!.fault).toMatchObject({ kind: "latency" }); + if (window.kind === "ST3") expect(window.rules[0]!.fault).toMatchObject({ kind: "errors", rate: 0.3, status: 503 }); + } + expect(planStorageFaults({ kinds: [], seed: 7, spanMs: 60_000 })).toEqual([]); + }); + + it("applies each window to the proxy while it is active and clears it afterwards", () => { + const windows: StorageFaultWindow[] = [{ kind: "ST1", startMs: 1_000, durationMs: 2_000, rules: [{ fault: { kind: "outage" } }] }]; + expect(activeStorageWindow(windows, 999)).toBeNull(); + expect(activeStorageWindow(windows, 1_000)).toBe(windows[0]); + const applied: Array = []; + const events: string[] = []; + const chaos = new StorageChaos(windows, 0, { setRules: rules => applied.push(rules), clear: () => applied.push("clear") }, kind => events.push(kind)); + chaos.tick(500); chaos.tick(1_500); chaos.tick(2_000); chaos.tick(3_500); + expect(applied).toEqual([windows[0]!.rules, "clear"]); + expect(events).toEqual(["storage_fault_started", "storage_fault_ended"]); + expect(chaos.receipts).toEqual([{ kind: "ST1", startedAt: 1_500, endedAt: 3_500, rules: windows[0]!.rules }]); + chaos.tick(1_200); + chaos.stop(1_300); + expect(applied.at(-1)).toBe("clear"); + }); +}); + +describe("storage fault clock and replay", () => { + afterEach(() => { vi.useRealTimers(); }); + + it("opens and closes windows on its own clock, not only when the wave loop gets to it", () => { + vi.useFakeTimers({ now: 0 }); + const windows: StorageFaultWindow[] = [{ kind: "ST1", startMs: 2_000, durationMs: 3_000, rules: [{ fault: { kind: "outage" } }] }]; + const applied: Array = []; + const chaos = new StorageChaos(windows, 0, { setRules: rules => applied.push(rules), clear: () => applied.push("clear") }, () => undefined); + chaos.start(500); + vi.advanceTimersByTime(2_500); + expect(applied).toEqual([windows[0]!.rules]); + vi.advanceTimersByTime(3_000); + expect(applied).toEqual([windows[0]!.rules, "clear"]); + chaos.stop(); + vi.advanceTimersByTime(10_000); + expect(applied).toHaveLength(2); + }); + + it("measures window offsets from when it starts, so a replay can start the clock at its task's launch", () => { + vi.useFakeTimers({ now: 0 }); + const windows: StorageFaultWindow[] = [{ kind: "ST1", startMs: 2_000, durationMs: 3_000, rules: [{ fault: { kind: "outage" } }] }]; + const applied: Array = []; + const chaos = new StorageChaos(windows, 0, { setRules: rules => applied.push(rules), clear: () => applied.push("clear") }, () => undefined); + vi.advanceTimersByTime(60_000); + chaos.start(500); + vi.advanceTimersByTime(1_500); + expect(applied).toEqual([]); + vi.advanceTimersByTime(1_000); + expect(applied).toEqual([windows[0]!.rules]); + chaos.stop(); + }); + + it("replays the storage windows that crossed a task, at offsets from its launch", () => { + const rules: StorageFaultRule[] = [{ fault: { kind: "outage", status: 503 } }]; + const receipts = [ + { kind: "ST1" as const, startedAt: 1_000, endedAt: 5_000, rules }, + { kind: "ST2" as const, startedAt: 20_000, endedAt: 30_000, rules: [{ fault: { kind: "latency" as const, ms: 9_000 } }] }, + { kind: "ST1" as const, startedAt: 40_000, endedAt: null, rules }, + ]; + expect(replayStorageWindows(receipts, 10_000, 45_000)).toEqual([ + { kind: "ST2", startMs: 10_000, durationMs: 10_000, rules: receipts[1]!.rules }, + { kind: "ST1", startMs: 30_000, durationMs: 5_000, rules }, + ]); + expect(replayStorageWindows(receipts, 3_000, 45_000)[0]).toEqual({ kind: "ST1", startMs: 0, durationMs: 2_000, rules }); + }); +}); + diff --git a/tests/restart-chaos/storage-chaos.ts b/tests/restart-chaos/storage-chaos.ts new file mode 100644 index 00000000000..cb2c50c76b0 --- /dev/null +++ b/tests/restart-chaos/storage-chaos.ts @@ -0,0 +1,103 @@ +import type { StorageFaultRule } from "./storage-proxy.js"; + +export const STORAGE_FAULT_KINDS = ["ST1", "ST2", "ST3"] as const; +export type StorageFaultKind = typeof STORAGE_FAULT_KINDS[number]; + +export interface StorageFaultWindow { kind: StorageFaultKind; startMs: number; durationMs: number; rules: StorageFaultRule[] } + +function rng(seed: number) { + let state = seed >>> 0 || 1; + return () => { + state ^= state << 13; state >>>= 0; + state ^= state >>> 17; + state ^= state << 5; state >>>= 0; + return state / 0x1_0000_0000; + }; +} + +export function storageFaultRules(kind: StorageFaultKind, random: () => number): StorageFaultRule[] { + switch (kind) { + case "ST1": return [{ fault: { kind: "outage", status: 503 } }]; + case "ST2": { + const ms = 1_000 + Math.round(random() * 9_000); + return [{ fault: { kind: "latency", ms, jitterMs: Math.round(ms / 4) } }]; + } + case "ST3": return [{ fault: { kind: "errors", rate: 0.3, status: 503, slowDown: random() < 0.5 } }]; + } +} + +export function planStorageFaults(input: { kinds: readonly StorageFaultKind[]; seed: number; spanMs: number; + firstAtMs?: number; gapMs?: readonly [number, number]; durationMs?: readonly [number, number] }): StorageFaultWindow[] { + if (!input.kinds.length) return []; + const random = rng(input.seed ^ 0x5f3759df); + const [gapMin, gapMax] = input.gapMs ?? [60_000, 120_000]; + const [durationMin, durationMax] = input.durationMs ?? [60_000, 180_000]; + const windows: StorageFaultWindow[] = []; + let at = input.firstAtMs ?? 45_000; + while (at < input.spanMs) { + const kind = input.kinds[Math.floor(random() * input.kinds.length)]!; + const durationMs = durationMin + Math.round(random() * (durationMax - durationMin)); + windows.push({ kind, startMs: at, durationMs, rules: storageFaultRules(kind, random) }); + at += durationMs + gapMin + Math.round(random() * (gapMax - gapMin)); + } + return windows; +} + +export function activeStorageWindow(windows: readonly StorageFaultWindow[], elapsedMs: number): StorageFaultWindow | null { + return windows.find(window => elapsedMs >= window.startMs && elapsedMs < window.startMs + window.durationMs) ?? null; +} + +export interface StorageFaultReceipt { kind: StorageFaultKind; startedAt: number; endedAt: number | null; rules: StorageFaultRule[] } + +export function replayStorageWindows(receipts: readonly StorageFaultReceipt[], taskCreatedAt: number, endedAt: number): StorageFaultWindow[] { + return receipts.flatMap(receipt => { + const start = Math.max(receipt.startedAt, taskCreatedAt); + const end = receipt.endedAt ?? endedAt; + return end > start ? [{ kind: receipt.kind, startMs: start - taskCreatedAt, durationMs: end - start, rules: receipt.rules }] : []; + }); +} + +export class StorageChaos { + private applied: StorageFaultWindow | null = null; + private timer: ReturnType | null = null; + readonly receipts: StorageFaultReceipt[] = []; + + constructor(private readonly windows: readonly StorageFaultWindow[], private startedAt: number, + private readonly proxy: { setRules(rules: readonly StorageFaultRule[]): void; clear(): void }, + private readonly record: (kind: string, data: Record) => void) {} + + tick(now = Date.now()): void { + const active = activeStorageWindow(this.windows, now - this.startedAt); + if (active === this.applied) return; + if (this.applied) { + this.proxy.clear(); + const receipt = this.receipts.at(-1); + if (receipt) receipt.endedAt = now; + this.record("storage_fault_ended", { kind: this.applied.kind }); + } + this.applied = active; + if (active) { + this.proxy.setRules(active.rules); + this.receipts.push({ kind: active.kind, startedAt: now, endedAt: null, rules: active.rules }); + this.record("storage_fault_started", { kind: active.kind, durationMs: active.durationMs, rules: active.rules }); + } + } + + start(intervalMs = 1_000, at = Date.now()): void { + if (this.timer) return; + this.startedAt = at; + this.timer = setInterval(() => this.tick(Date.now()), intervalMs); + this.timer.unref?.(); + this.tick(Date.now()); + } + + stop(now = Date.now()): void { + if (this.timer) { clearInterval(this.timer); this.timer = null; } + if (!this.applied) return; + this.proxy.clear(); + const receipt = this.receipts.at(-1); + if (receipt) receipt.endedAt = now; + this.record("storage_fault_ended", { kind: this.applied.kind, atStop: true }); + this.applied = null; + } +} diff --git a/tests/restart-chaos/storage-proxy.test.ts b/tests/restart-chaos/storage-proxy.test.ts new file mode 100644 index 00000000000..6a5b90f461c --- /dev/null +++ b/tests/restart-chaos/storage-proxy.test.ts @@ -0,0 +1,92 @@ +import { createServer, type Server } from "node:http"; +import { once } from "node:events"; +import { afterEach, describe, expect, it } from "vitest"; +import { startStorageFaultProxy } from "./storage-proxy.js"; + +const closers: Array<() => Promise> = []; +afterEach(async () => { for (const close of closers.splice(0).reverse()) await close(); }); + +async function upstream() { + const seen: string[] = []; + const server: Server = createServer((incoming, outgoing) => { + let body = ""; + incoming.on("data", chunk => { body += chunk; }); + incoming.on("end", () => { seen.push(`${incoming.method} ${incoming.url} ${body}`); outgoing.writeHead(200, { etag: "\"e1\"" }).end("stored"); }); + }); + server.listen(0, "127.0.0.1"); await once(server, "listening"); + closers.push(async () => { server.closeAllConnections(); server.close(); }); + const address = server.address() as { port: number }; + return { endpoint: `http://127.0.0.1:${address.port}`, seen }; +} + +async function proxyFor(endpoint: string, random?: () => number) { + const proxy = await startStorageFaultProxy(endpoint, random); + closers.push(() => proxy.close()); + return proxy; +} + +describe("storage fault proxy", () => { + it("passes requests through untouched when no rule matches", async () => { + const store = await upstream(); + const proxy = await proxyFor(store.endpoint); + const response = await fetch(`${proxy.endpoint}/bucket/key`, { method: "PUT", body: "bytes" }); + expect([response.status, await response.text(), response.headers.get("etag")]).toEqual([200, "stored", "\"e1\""]); + expect(store.seen).toEqual(["PUT /bucket/key bytes"]); + expect(proxy.counts()).toMatchObject({ requests: 1, injected: 0, byMethod: { PUT: 1 } }); + }); + + it("injects a store outage for matching requests only, for a bounded number of hits", async () => { + const store = await upstream(); + const proxy = await proxyFor(store.endpoint); + proxy.setRules([{ fault: { kind: "outage" }, methods: ["PUT"], pathIncludes: "workspace-snapshots", remaining: 2 }]); + const put = () => fetch(`${proxy.endpoint}/bucket/c/workspace-snapshots/w/s.pcws`, { method: "PUT", body: "x" }); + const first = await put(); + expect([first.status, await first.text()]).toEqual([503, expect.stringContaining("ServiceUnavailable")]); + expect((await put()).status).toBe(503); + expect((await put()).status).toBe(200); + expect((await fetch(`${proxy.endpoint}/bucket/c/run-logs/r`)).status).toBe(200); + expect(store.seen).toHaveLength(2); + expect(proxy.counts()).toMatchObject({ injected: 2, byStatus: { 503: 2, 200: 2 } }); + }); + + it("answers SlowDown throttling and expired credentials the way S3 does", async () => { + const store = await upstream(); + const proxy = await proxyFor(store.endpoint, () => 0); + proxy.setRules([{ fault: { kind: "errors", rate: 0.5, status: 503, slowDown: true } }]); + const slow = await fetch(`${proxy.endpoint}/bucket/a`); + expect([slow.status, await slow.text()]).toEqual([503, expect.stringContaining("SlowDown")]); + proxy.setRules([{ fault: { kind: "expired_credentials" } }]); + const expired = await fetch(`${proxy.endpoint}/bucket/a`); + expect([expired.status, await expired.text()]).toEqual([403, expect.stringContaining("ExpiredToken")]); + proxy.clear(); + expect((await fetch(`${proxy.endpoint}/bucket/a`)).status).toBe(200); + }); + + it("delays requests by the configured latency and drops connections on reset", async () => { + const store = await upstream(); + const proxy = await proxyFor(store.endpoint, () => 0); + proxy.setRules([{ fault: { kind: "latency", ms: 150 } }]); + const started = Date.now(); + expect((await fetch(`${proxy.endpoint}/bucket/a`)).status).toBe(200); + expect(Date.now() - started).toBeGreaterThanOrEqual(140); + proxy.setRules([{ fault: { kind: "reset" } }]); + await expect(fetch(`${proxy.endpoint}/bucket/a`)).rejects.toThrow(); + }); + + it("cancels a delayed request when the client goes away and counts store connection failures", async () => { + const store = await upstream(); + const proxy = await proxyFor(store.endpoint, () => 0); + proxy.setRules([{ fault: { kind: "latency", ms: 400 } }]); + const controller = new AbortController(); + const pending = fetch(`${proxy.endpoint}/bucket/late`, { signal: controller.signal }).catch(() => null); + await new Promise(resolve => setTimeout(resolve, 50)); + controller.abort(); + await pending; + await new Promise(resolve => setTimeout(resolve, 600)); + expect(store.seen).toEqual([]); + const dead = await proxyFor("http://127.0.0.1:1"); + expect((await fetch(`${dead.endpoint}/bucket/a`)).status).toBe(502); + expect(dead.counts()).toMatchObject({ requests: 1, byStatus: { 502: 1 } }); + }); +}); + diff --git a/tests/restart-chaos/storage-proxy.ts b/tests/restart-chaos/storage-proxy.ts new file mode 100644 index 00000000000..3940f9e9640 --- /dev/null +++ b/tests/restart-chaos/storage-proxy.ts @@ -0,0 +1,114 @@ +import { Agent, createServer, request, type IncomingMessage, type ServerResponse } from "node:http"; +import { once } from "node:events"; +import type { Socket } from "node:net"; + +export type StorageFault = + | { kind: "pass" } + | { kind: "outage"; status?: number } + | { kind: "latency"; ms: number; jitterMs?: number } + | { kind: "errors"; rate: number; status?: number; slowDown?: boolean } + | { kind: "reset" } + | { kind: "expired_credentials" }; + +export interface StorageFaultRule { fault: StorageFault; methods?: readonly string[]; pathIncludes?: string; remaining?: number } + +export interface StorageProxyCounts { requests: number; injected: number; byMethod: Record; byStatus: Record } + +const XML = "application/xml"; +function s3Error(code: string, message: string) { + return `${code}${message}`; +} + +export function faultResponse(fault: StorageFault): { status: number; body: string } | "reset" | null { + switch (fault.kind) { + case "outage": return { status: fault.status ?? 503, body: s3Error("ServiceUnavailable", "Injected storage outage") }; + case "expired_credentials": return { status: 403, body: s3Error("ExpiredToken", "The provided token has expired.") }; + case "reset": return "reset"; + default: return null; + } +} + +export function matches(rule: StorageFaultRule, method: string, url: string): boolean { + if (rule.remaining !== undefined && rule.remaining <= 0) return false; + if (rule.methods && !rule.methods.includes(method)) return false; + return !rule.pathIncludes || url.includes(rule.pathIncludes); +} + +export async function startStorageFaultProxy(endpoint: string, random: () => number = Math.random) { + const target = new URL(endpoint); + if (target.protocol !== "http:" || !["127.0.0.1", "localhost", "[::1]"].includes(target.hostname)) { + throw new Error("Storage fault injection requires a loopback store"); + } + let rules: StorageFaultRule[] = []; + const counts: StorageProxyCounts = { requests: 0, injected: 0, byMethod: {}, byStatus: {} }; + const sockets = new Set(); + const delayed = new Set>(); + const agent = new Agent({ keepAlive: true }); + const note = (method: string, status: number | "reset", injected: boolean) => { + counts.requests += 1; + counts.byMethod[method] = (counts.byMethod[method] ?? 0) + 1; + counts.byStatus[String(status)] = (counts.byStatus[String(status)] ?? 0) + 1; + if (injected) counts.injected += 1; + }; + const forward = (incoming: IncomingMessage, outgoing: ServerResponse) => { + const upstream = request(target, { method: incoming.method, path: incoming.url, headers: incoming.headers, agent }, response => { + response.on("error", () => outgoing.destroy()); + note(incoming.method ?? "GET", response.statusCode ?? 502, false); + outgoing.writeHead(response.statusCode ?? 502, response.headers); + response.pipe(outgoing); + }); + upstream.on("error", () => { + if (outgoing.headersSent) { outgoing.destroy(); return; } + note(incoming.method ?? "GET", 502, false); + outgoing.writeHead(502).end(); + }); + incoming.on("aborted", () => upstream.destroy()); + outgoing.on("close", () => { if (!outgoing.writableFinished) upstream.destroy(); }); + incoming.pipe(upstream); + }; + const server = createServer((incoming, outgoing) => { + const method = incoming.method ?? "GET"; + const rule = rules.find(candidate => matches(candidate, method, incoming.url ?? "/")); + if (rule?.remaining !== undefined) rule.remaining -= 1; + const fault = rule?.fault ?? { kind: "pass" as const }; + if (fault.kind === "latency") { + const delay = fault.ms + Math.round((fault.jitterMs ?? 0) * random()); + incoming.pause(); + const timer = setTimeout(() => { + delayed.delete(timer); + if (outgoing.destroyed || incoming.destroyed) return; + incoming.resume(); + forward(incoming, outgoing); + }, delay); + delayed.add(timer); + outgoing.on("close", () => { clearTimeout(timer); delayed.delete(timer); }); + return; + } + if (fault.kind === "errors" && random() >= fault.rate) { forward(incoming, outgoing); return; } + const injected = fault.kind === "errors" + ? { status: fault.status ?? 503, body: fault.slowDown ? s3Error("SlowDown", "Please reduce your request rate.") : s3Error("InternalError", "Injected storage error") } + : faultResponse(fault); + if (!injected) { forward(incoming, outgoing); return; } + incoming.resume(); + if (injected === "reset") { note(method, "reset", true); incoming.socket.destroy(); return; } + note(method, injected.status, true); + outgoing.writeHead(injected.status, { "content-type": XML, "content-length": Buffer.byteLength(injected.body) }).end(injected.body); + }); + server.on("connection", socket => { sockets.add(socket); socket.on("close", () => sockets.delete(socket)); }); + server.listen(0, "127.0.0.1"); await once(server, "listening"); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Storage proxy did not bind"); + return { + endpoint: `http://127.0.0.1:${address.port}`, + counts: () => structuredClone(counts), + setRules(next: readonly StorageFaultRule[]) { rules = next.map(rule => ({ ...rule })); }, + clear() { rules = []; }, + async close() { + for (const timer of delayed) clearTimeout(timer); + delayed.clear(); + const closed = once(server, "close"); server.close(); agent.destroy(); + for (const socket of sockets) socket.destroy(); + await closed; + }, + }; +} diff --git a/tests/restart-chaos/supervisor.ts b/tests/restart-chaos/supervisor.ts index 53d5b8a26f5..8623c25706e 100644 --- a/tests/restart-chaos/supervisor.ts +++ b/tests/restart-chaos/supervisor.ts @@ -214,7 +214,9 @@ export class ChaosSupervisor { ...extraEnvironment, }; const environment = buildPaperclipServerEnvironment(source, overrides); - assertIsolatedServerEnvironment(environment, { temporaryRoot, paperclipHome: home, configPath }); + const workspaceStorage = extraEnvironment.PAPERCLIP_STORAGE_PROVIDER === "s3" + ? Object.fromEntries(Object.entries(extraEnvironment).filter(([key]) => key.startsWith("PAPERCLIP_STORAGE_"))) : undefined; + assertIsolatedServerEnvironment(environment, { temporaryRoot, paperclipHome: home, configPath, ...(workspaceStorage ? { workspaceStorage } : {}) }); this.environment = Object.fromEntries(Object.entries(environment).filter((entry): entry is [string, string] => entry[1] !== undefined)); await prepareSourceServerDependencies(root, this.environment, false); } diff --git a/tests/restart-chaos/triggers.test.ts b/tests/restart-chaos/triggers.test.ts index a238bd28b0b..25908cf61eb 100644 --- a/tests/restart-chaos/triggers.test.ts +++ b/tests/restart-chaos/triggers.test.ts @@ -146,6 +146,28 @@ describe("trigger firing", () => { expect(observer.record.triggerMissed).toEqual(["comment"]); }); + it("clicks Retry through the board UI when the wave drives a browser, and records what the user saw", async () => { + const state = { runs: [{ runId: "r1", status: "failed", finishedAt: new Date().toISOString() }] as Json[], + issue: { id: "i1", status: "blocked", interruption: { state: "failed", canRetry: true, observedRunId: "r1" } } }; + const clicks: string[] = []; + const board = { clickRetry: async (issue: string) => { clicks.push(issue); + return { clicked: true, httpStatus: 202, accepted: true, error: null, bannerText: "Couldn’t finish this attempt." }; } }; + const observer = new TaskObserver(base, scriptedApi(state), lab, new Timeline(null), ready, undefined, board); + await observer.launch(); + const start = observer.record.createdAt!; + await observer.poll(start + 1_000); + await observer.poll(start + 1_000 + 12_001); + expect(clicks).toEqual(["CHA-1"]); + expect(observer.record.retry).toMatchObject({ via: "ui", accepted: true, httpStatus: 202, bannerText: "Couldn’t finish this attempt." }); + + const missing = { clickRetry: async () => ({ clicked: false, httpStatus: null, accepted: null, error: "ui_retry_button_missing", bannerText: "Saving files from the previous attempt" }) }; + const blind = new TaskObserver(base, scriptedApi(state), lab, new Timeline(null), ready, undefined, missing); + await blind.launch(); + await blind.poll(blind.record.createdAt! + 1_000); + await blind.poll(blind.record.createdAt! + 1_000 + 12_001); + expect(blind.record.retry).toMatchObject({ via: "ui", accepted: null, error: "ui_retry_button_missing" }); + }); + it("retry: fires the extra K1 right after an accepted Retry click, and closes unfired triggers at sweep", async () => { const state = { runs: [{ runId: "r1", status: "failed", finishedAt: new Date().toISOString() }] as Json[], issue: { id: "i1", status: "blocked", interruption: { state: "needs_retry", canRetry: true, observedRunId: "r1" } } }; diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index 0e430ba82f7..8e0abad18bc 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -73,7 +73,12 @@ export interface WaveOptions { */ upgradeFrom?: string; workspaceRepo?: string; + storage?: "s3"; + storageFaults?: Array<"ST1" | "ST2" | "ST3">; + storageFaultMs?: [number, number]; + storageWindows?: import("./storage-chaos.js").StorageFaultWindow[]; workspaceCommit?: string; + workspaceGitUrl?: string; } export interface StormFault { @@ -183,8 +188,13 @@ export interface RetryAttempt { httpStatus: number | null; accepted: boolean | null; error: string | null; + via?: "ui" | "api"; + bannerText?: string | null; } +export const UI_RETRY_BUTTON_MISSING = "ui_retry_button_missing"; +export const UI_RETRY_NO_REQUEST = "ui_retry_no_request"; + export interface FaultReceipt { faultId: string; kind: FaultKind; @@ -297,6 +307,8 @@ export interface TaskRecord { workspaceMovedTo?: string; /** Set when the pre-fault snapshot could not read the workspace; R2 then fails visibly. */ preFaultSnapshotError?: string | null; + remoteWorkspace?: boolean; + remoteFiles?: Record; /** Pre-fault files missing at rest; null when not checked. */ lostFiles: string[] | null; /** B5: distinct bench progress values the observer read during polling. */