Files
PaperClipAI/tests/restart-chaos/api.ts
T
DottaandClaude Opus 5.5 908d2aae7a test(chaos): run waves against S3-compatible storage with a fault proxy, and click Retry in the real board UI (#20288)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud deployments keep run logs, traces, attachments, artifacts and
saved workspace files in S3-compatible object storage
> - The restart chaos harness ran only with local disk storage, so
storage failures and slowdowns were never tested
> - It also clicked Retry through the API, so a missing or broken Retry
button in the board UI went unnoticed (staging PAP-1683: "the
saved-files recovery Retry does nothing")
> - This pull request runs waves against a loopback MinIO behind a fault
proxy, adds store-outage, latency and throttling faults, and clicks
every Retry in the real board UI
> - The benefit is that storage outages and broken Retry paths now
surface in chaos waves. The first ST1 wave found that a run starting
during a store outage fails immediately and exhausts recovery

## Linked Issues or Issue Description

**What happened?**
Staging saves to object storage and has failures there that local chaos
waves cannot reproduce, because the harness used `local_disk`. The
harness also clicked Retry through the API, which skips the board's
actual button.

**Expected behavior**
Chaos waves can run the server in cloud storage mode, inject storage
faults, and press Retry the way a user does.

**Actual behavior**
They could not.

## What Changed

- `--storage s3`:
- starts a loopback MinIO (`RELEASE.2025-04-22T22-12-26Z`, from
`~/.cache/paperclip/tools/minio/minio` or `RESTART_CHAOS_MINIO_BIN`)
with its data under the instance temp root and random root credentials;
  - creates the bucket;
  - puts a fault proxy in front;
- points the server's `PAPERCLIP_STORAGE_*` settings at the proxy, with
path-style addressing and a per-wave prefix. The supervisor declares
these as explicit workspace storage for the isolation check.
- `storage-proxy.ts`: a transparent, non-retrying S3 fault proxy that
keeps the signed Host header. Rules match by method, path and remaining
count. Faults are outage (503 `ServiceUnavailable`), latency with
jitter, a 5xx or `SlowDown` error rate, connection reset, and 403
`ExpiredToken`. It counts requests by method and status, written to
`storage-proxy.json`.
- `--storage-faults ST1,ST2,ST3` (`storage-chaos.ts`): seeded 60–180 s
windows of outage, 1–10 s latency, or 30% 503 / `SlowDown`, applied
while the wave runs. They're logged in the timeline and in
`storage-faults.json`.
- `board-ui.ts`: every task and subtask Retry the grader needs is now
clicked in the board UI. The harness uses headless Chromium through
Playwright, opens `/issues/<identifier>`, waits for the visible Retry
button and captures the `POST /issues/:id/retry` the click sends.
- R3 fails a missing button as `ui_retry_button_missing (user saw: …)`
and a click that sends nothing as `ui_retry_no_request`.
  - Crash-point waves, which serve no UI, keep the API path.
- README: a cloud storage section.

## Verification

- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 244
passed. New tests:
- the proxy against a fake upstream: passthrough, bounded outage,
`SlowDown`, `ExpiredToken`, latency and reset;
- live MinIO through the proxy: `If-None-Match` returns 412, and an
injected 503 reaches the client;
- Playwright against local pages: a working button, a missing button,
and a dead button;
- the observer clicking through the board, and R3 signatures for UI
failures;
  - window planning, apply and clear;
  - `--storage` and `--storage-faults` parsing.

  The observer and grader tests failed before the change.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Live waves on master 610b54f551 with this branch:
- Baseline `--storage s3`, A1/C1, K1/K2 (w-20261109T144738-s9901): 5/8.
There were 2,319 S3 requests through the proxy and 18 conditional-write
412 fences. All 3 Retries were clicked in the board UI, returned 202
Accepted, and the new run succeeded. The 3 failures are a Claude session
issue (#20285), not storage.
- ST1 (w-20261109T145628-s9902): 1/8. A run that starts during a store
outage fails within about 1 s as `adapter_failed`, automatic recovery
exhausts within the outage, and UI Retries are accepted but do not
recover. This has been reported for a product fix.

## Risks

- `--storage s3` needs a locally built MinIO. MinIO no longer publishes
community binaries. Without the binary the wave fails at start with the
path to provide, and the live store test is skipped.
- The UI Retry needs the dev UI to load. A slow first Vite compile can
delay the click, up to the 60 s wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:15:41 -06:00

66 lines
3.6 KiB
TypeScript

/** Minimal board API client for an isolated local_trusted instance. */
export function isSocketTypeOfServiceRace(error: unknown): boolean {
const candidate = error as { code?: unknown; syscall?: unknown } | null;
return candidate?.code === "EINVAL" && candidate.syscall === "setTypeOfService";
}
export class ChaosApiError extends Error {
constructor(readonly status: number | null, message: string) { super(message); this.name = "ChaosApiError"; }
}
export function rateLimitDelayMs(response: { status: number; headers: { get(name: string): string | null } }, text: string): number | null {
if (response.status !== 429) return null;
let seconds = Number(response.headers.get("retry-after"));
if (!Number.isFinite(seconds) || seconds <= 0) {
try { seconds = Number((JSON.parse(text) as { retryAfterSeconds?: unknown }).retryAfterSeconds); } catch { seconds = NaN; }
}
return Number.isFinite(seconds) && seconds > 0 ? Math.min(10_000, seconds * 1_000) : 1_000;
}
export class ChaosApi {
constructor(readonly baseUrl: string, private readonly timeoutMs = 10_000, private readonly rateLimitAttempts = 5) {}
private async request<T>(method: string, route: string, body?: unknown, timeoutMs = this.timeoutMs, text = false): Promise<T> {
for (let attempt = 1; ; attempt += 1) {
const outcome = await this.attempt<T>(method, route, body, timeoutMs, text);
if (!("retryAfterMs" in outcome)) return outcome.value;
if (method !== "GET" || attempt >= this.rateLimitAttempts) throw outcome.error;
await new Promise(resolve => setTimeout(resolve, Math.min(10_000, outcome.retryAfterMs * attempt)));
}
}
private async attempt<T>(method: string, route: string, body: unknown, timeoutMs: number, asText = false):
Promise<{ value: T } | { retryAfterMs: number; error: ChaosApiError }> {
let response: Response;
try {
response = await fetch(`${this.baseUrl}${route}`, {
method,
headers: { origin: this.baseUrl, ...(body === undefined ? {} : { "content-type": "application/json" }) },
body: body === undefined ? undefined : JSON.stringify(body),
signal: AbortSignal.timeout(timeoutMs),
cache: "no-store",
});
} catch (error) {
throw new ChaosApiError(null, `${method} ${route} unreachable: ${error instanceof Error ? error.message : String(error)}`);
}
const text = await response.text();
if (!response.ok) {
const error = new ChaosApiError(response.status, `${method} ${route} returned ${response.status}: ${text.slice(0, 500)}`);
const retryAfterMs = rateLimitDelayMs(response, text);
if (retryAfterMs !== null) return { retryAfterMs, error };
throw error;
}
return { value: (asText ? text : text ? JSON.parse(text) : null) as T };
}
get<T>(route: string, timeoutMs?: number) { return this.request<T>("GET", route, undefined, timeoutMs); }
getText(route: string, timeoutMs?: number) { return this.request<string>("GET", route, undefined, timeoutMs, true); }
post<T>(route: string, body?: unknown, timeoutMs?: number) { return this.request<T>("POST", route, body ?? {}, timeoutMs); }
put<T>(route: string, body?: unknown) { return this.request<T>("PUT", route, body ?? {}); }
patch<T>(route: string, body?: unknown) { return this.request<T>("PATCH", route, body ?? {}); }
async delete(route: string, options?: { allowNotFound?: boolean; timeout?: number }): Promise<void> {
try { await this.request<unknown>("DELETE", route, undefined, options?.timeout); }
catch (error) { if (!(options?.allowNotFound && error instanceof ChaosApiError && error.status === 404)) throw error; }
}
}