From 1a89fca1ce2289f1eff6c53637364be7f832f622 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 10 Nov 2026 03:56:25 -0600 Subject: [PATCH] test(restart-chaos): add --rr-stop-on recovering so RR follow-up stops land inside recovery (#20511) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - #20480 added repeated restarts (RR1–RR3): a K1, then more restarts while the replacement recovers. > - In the first four live waves, no follow-up stop landed inside recovery. Every stop saw `/api/ready` report `ready`. > - The saved receipts show why. `/api/ready` reports `recovering` only 15–80 ms before `/api/health` answers, and `ready` 0.25–0.5 s after it. A gap measured from health, even 1 s, always lands after recovery. > - This pull request adds `--rr-stop-on recovering`. Each follow-up stop fires on the first `recovering` status, polled from the replacement's spawn. > - The benefit is that RR waves now test a restart that interrupts recovery itself. ## Linked Issues or Issue Description Refs #20480 **What is the problem?** RR follow-up stops wait a gap from the replacement's health. On a local host recovery ends about 0.3 s after health, so no stop in four waves (14 tasks, 18 follow-up stops) landed inside recovery. **What is the solution?** Add `--rr-stop-on recovering`. It replaces the gap: the harness polls `/api/ready` every 50 ms from spawn and stops the replacement on its first `recovering` status. This is the phase poll that K6 already uses. **Why is it needed?** The deploy loops of 2026-11-08 and 2026-11-09 restarted the server while it recovered interrupted runs. The harness must reach that window to test it. ## What Changed - `repeated-restarts.ts`: `--rr-stop-on recovering` is parsed by `parseRepeatedRestartOptions`. It needs an RR fault and does not combine with `--rr-gap-seconds` or `--replay`. `runRepeatedRestarts` takes `stopOn`: with it, no gap is slept, each stop waits on the new `waitForStopPhase` hook, and only the last replacement is waited on to health. - `supervisor.ts`: `waitForStopPhase` reuses the K6 phase poll (`waitForCrashLoopPhase`, 50 ms). The record carries `stopOn`. - `types.ts`, `config.ts`, `main.ts`: the `rrStopOn` option, its flag, and one line to pass it to the supervisor. - The evidence shows each stop's `/api/ready` status (`phaseAtStop`). With `stopOn`, `gapMs` is null, and the earlier entries have no `healthyAt`. - README: the option and the measured timings. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 438 tests pass. New tests cover option parsing, stops on the first `recovering` with no sleep, health waits only on the last replacement, a `ready` stop when recovery was too short to see, an exit before a status, and a missing watcher. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - A live wave from this branch (`--workloads A1,B5,C1 --faults RR1,RR3 --rr-stop-on recovering`, codex, 4 tasks) follows; its results go to a comment. ## Risks - Harness-only change; no product code changes. RR waves without the flag are unchanged. - A stop on `recovering` lands before the replacement answers health. This is the K6 behavior, and the server log shows it. A replacement that never reports a status fails the fault after the 5-minute ready timeout, as K6 does. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude --- tests/restart-chaos/README.md | 12 ++- tests/restart-chaos/config.ts | 5 +- tests/restart-chaos/main.ts | 1 + tests/restart-chaos/repeated-restarts.test.ts | 96 ++++++++++++++++++- tests/restart-chaos/repeated-restarts.ts | 67 +++++++++++-- tests/restart-chaos/supervisor.ts | 27 +++++- tests/restart-chaos/types.ts | 2 + 7 files changed, 191 insertions(+), 19 deletions(-) diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 423d8ee0af1..93db4f57445 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -826,8 +826,16 @@ moment the replacement reports healthy, it stops that replacement and starts ano The gaps are seeded per task (`restartSteps` in `plan.json`), and drawn only for RR tasks, so the plans of existing seeds do not change. `--rr-gap-seconds ` or `-` (1–300) replaces every RR -gap: for example `--rr-gap-seconds 5` for a stop that always lands in recovery, or `30-60` for a slow -deploy loop. RR faults do not combine with `--storm-minutes`, whose faults carry no planned steps. +gap: for example `--rr-gap-seconds 5` for a short deploy-loop gap, or `30-60` for a slow +deploy loop. To stop inside recovery, use `--rr-stop-on recovering` (below). RR faults do not combine with `--storm-minutes`, whose faults carry no planned steps. + +`--rr-stop-on recovering` replaces the gap: each follow-up stop fires on the first `/api/ready` poll +(every 50 ms, from the replacement's spawn) that reports `recovering`, the K6 phase poll. Use it to +land the stops inside recovery. In the first live waves `/api/ready` reported `recovering` only +15–80 ms before `/api/health` answered, and `ready` 0.25–0.5 s after it, so every gap from health +(even 1 s) landed after recovery. A replacement that goes straight to `ready` is stopped there and +records `ready`. The receipt records `stopOn`, and each follow-up entry has `gapMs: null`; only the last +replacement is waited on to health, so the earlier entries have no `healthyAt`. The first replacement (of the K1 kill) is also watched from its spawn, not from its health. **Evidence.** Each stop and replacement (the first kill included) is recorded with its signal, planned gap, stopped PID, the `/api/ready` status just before the stop (`recovering` means it landed inside diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index b5d61a24a6d..d68740f5951 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -120,13 +120,14 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC "provider-fault-restart": { type: "string" }, "provider-fault-restart-after-seconds": { type: "string" }, "rr-gap-seconds": { type: "string" }, + "rr-stop-on": { type: "string" }, }, }); const keep = values.keep === true; const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000); const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000); if (values.replay !== undefined) { - for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds"] as const) { + for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on"] as const) { if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`); } return { mode: "replay", ref: parseReplayRef(values.replay), keep, @@ -224,7 +225,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults"); } Object.assign(options, parseTimeFaultOptions(options, values["time-age-hours"])); - Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"])); + Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"], values["rr-stop-on"])); const upgradeFrom = values["upgrade-from"]; if (options.faults.includes("K14") !== (upgradeFrom !== undefined)) { throw new Error("K14 (upgrade mid-run) and --upgrade-from go together: pass both or neither"); diff --git a/tests/restart-chaos/main.ts b/tests/restart-chaos/main.ts index 861a3a9a686..ba95ece9264 100644 --- a/tests/restart-chaos/main.ts +++ b/tests/restart-chaos/main.ts @@ -321,6 +321,7 @@ async function main(): Promise { rules: plan.tasks.flatMap(task => task.providerFault ? [`${task.taskId}:${describeProviderFault(task.providerFault)}`] : []) }); } if (plan.options.timeAgeMs) supervisor.timeAgeMs = plan.options.timeAgeMs; + if (plan.options.rrStopOn) supervisor.rrStopOn = plan.options.rrStopOn; await supervisor.start(serverEnvironment); // TM2/TM3 age the database; the observer and grader read run times back at their real times. const api: ChaosApi = new TimeShiftedApi(supervisor.baseUrl, supervisor.agingShifts); diff --git a/tests/restart-chaos/repeated-restarts.test.ts b/tests/restart-chaos/repeated-restarts.test.ts index cc1f0eec243..ab649592080 100644 --- a/tests/restart-chaos/repeated-restarts.test.ts +++ b/tests/restart-chaos/repeated-restarts.test.ts @@ -4,7 +4,7 @@ import { crossingFaults } from "./grader.js"; import { planMatrixWave } from "./journey-matrix.js"; import { createRng, observedFaultSchedule, planTasks, taskFaultSchedule } from "./scheduler.js"; import { - RR3_KILL_RANGE, RR_GAP_RANGE_MS, newRepeatedRestartRecord, planRestartSteps, recordFirstRestart, runRepeatedRestarts, type RepeatedRestartRecord, type RestartStep, + RR3_KILL_RANGE, RR_GAP_RANGE_MS, newRepeatedRestartRecord, planRestartSteps, recordFirstRestart, runRepeatedRestarts, stopPhaseWatcher, type RepeatedRestartRecord, type RestartStep, } from "./repeated-restarts.js"; import type { FaultReceipt, TaskRecord } from "./types.js"; @@ -27,6 +27,17 @@ describe("repeated restart options", () => { expect(() => parseCommand(["--faults", "RR1", "--rr-gap-seconds", "5s", "--workloads", "A1"])).toThrow(/ or -/); }); + it("takes --rr-stop-on recovering instead of a gap, only with an RR fault", () => { + const parsed = parseCommand(["--faults", "RR1,RR3", "--rr-stop-on", "recovering", "--workloads", "A1", "--seed", "1"]); + expect(parsed.mode === "wave" && parsed.options.rrStopOn).toBe("recovering"); + expect(parsed.mode === "wave" && parsed.options.rrGapMs).toBeUndefined(); + expect(() => parseCommand(["--faults", "K1", "--rr-stop-on", "recovering", "--workloads", "A1"])).toThrow(/add RR1, RR2 or RR3/); + expect(() => parseCommand(["--faults", "RR1", "--rr-stop-on", "ready", "--workloads", "A1"])).toThrow(/must be recovering/); + expect(() => parseCommand(["--faults", "RR1", "--rr-stop-on", "recovering", "--rr-gap-seconds", "1", "--workloads", "A1"])) + .toThrow(/do not combine it with --rr-gap-seconds/); + expect(() => parseCommand(["--replay", "w-20261110T000000-s1/t01", "--rr-stop-on", "recovering"])).toThrow(/--rr-stop-on/); + }); + it("refuses a storm, whose faults carry no planned restarts", () => { expect(() => parseCommand(["--faults", "RR1", "--storm-minutes", "10"])).toThrow(/storm/); }); @@ -151,6 +162,19 @@ describe("the record built before the first kill", () => { expect(build(undefined).planned).toEqual(planRestartSteps("RR3", createRng(9).int)); }); + it("never gives the first replacement the last one's health time", () => { + const watched = { ...build(planned), stopOn: "recovering" as const }; + recordFirstRestart(watched, { stoppedAt: 5_400, restartedAt: 5_400, healthyAt: null }, 77); + watched.restarts.push({ ...watched.restarts[0]!, index: 1, healthyAt: 40_000 }); + // The evidence write after the sequence sees the receipt's health: the last replacement's. + recordFirstRestart(watched, { stoppedAt: 5_400, restartedAt: 5_400, healthyAt: 40_000 }, null); + expect(watched.restarts[0]!.healthyAt).toBeNull(); + const gapped = build(planned); + gapped.restarts.push({ ...gapped.restarts[0]!, index: 1, healthyAt: 40_000 }); + recordFirstRestart(gapped, { stoppedAt: 5_400, restartedAt: 5_400, healthyAt: 40_000 }, null); + expect(gapped.restarts[0]!.healthyAt).toBeNull(); + }); + it("does not overwrite first-kill times already recorded", () => { const rr = build(planned); recordFirstRestart(rr, { stoppedAt: 5_400, restartedAt: 5_400, healthyAt: 13_000 }, 77); @@ -217,6 +241,76 @@ describe("runRepeatedRestarts", () => { }); }); +describe("runRepeatedRestarts with --rr-stop-on recovering", () => { + it("stops each replacement on its first recovering status, with no gap, and waits for health only on the last", async () => { + const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 12_000 }, { signal: "SIGKILL", gapMs: 25_000 }]; + const fake = fakeHooks(); + const watched: number[] = []; + let healthChecks = 0; + const hooks = { ...fake.hooks, + waitForStopPhase: async (server: { pid: number }, target: "recovering") => { watched.push(server.pid); await fake.hooks.sleep(150); return target; }, + waitHealthy: async () => { healthChecks += 1; return fake.hooks.waitHealthy(); } }; + const rr = record("RR3", steps); + const result = await runRepeatedRestarts({ record: rr, steps, server: { pid: 100 }, healthyAt: fake.now(), hooks, stopOn: "recovering" }); + expect(watched).toEqual([100, 200]); + expect(fake.log).toEqual(["sleep 150", "stop 100 SIGKILL", "start 200", "sleep 150", "stop 200 SIGKILL", "start 201"]); + expect(healthChecks).toBe(1); + expect(rr.restarts.map(entry => [entry.phaseAtStop, entry.gapMs, entry.replacementPid])).toEqual([["recovering", null, 200], ["recovering", null, 201]]); + expect(rr.restarts[0]!.healthyAt).toBeNull(); + expect(rr.restarts[1]!.healthyAt).toBe(result.healthyAt); + }); + + it("uses the status the first replacement fired on from spawn, without waiting for its health", async () => { + const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 12_000 }, { signal: "SIGKILL", gapMs: 25_000 }]; + const fake = fakeHooks(); + const first = { pid: 100 }; + const watched: number[] = []; + const watch = async (server: { pid: number }) => { watched.push(server.pid); return "recovering"; }; + const rr = record("RR3", steps); + // The first kill's restart watched pid 100 from spawn and saw `recovering`; health was never awaited. + await runRepeatedRestarts({ record: rr, steps, server: first, healthyAt: fake.now(), stopOn: "recovering", + hooks: { ...fake.hooks, waitForStopPhase: stopPhaseWatcher(first, "recovering", watch) } }); + expect(watched).toEqual([200]); + expect(rr.restarts.map(entry => entry.phaseAtStop)).toEqual(["recovering", "recovering"]); + expect(fake.log).toEqual(["stop 100 SIGKILL", "start 200", "stop 200 SIGKILL", "start 201"]); + }); + + it("uses the first status once, and watches the first server when none was recorded", async () => { + const first = { pid: 100 }; + const watch = async () => "ready"; + const cached = stopPhaseWatcher(first, "recovering", watch); + expect(await cached(first, "recovering")).toBe("recovering"); + expect(await cached(first, "recovering")).toBe("ready"); + expect(await stopPhaseWatcher(first, undefined, watch)(first, "recovering")).toBe("ready"); + }); + + it("records `ready` when recovery was too short to see", async () => { + const steps: RestartStep[] = [{ signal: "SIGTERM", gapMs: 5_000 }]; + const fake = fakeHooks(); + const rr = record("RR2", steps); + await runRepeatedRestarts({ record: rr, steps, server: { pid: 100 }, healthyAt: fake.now(), stopOn: "recovering", + hooks: { ...fake.hooks, waitForStopPhase: async () => "ready" } }); + expect(rr.restarts[0]!.phaseAtStop).toBe("ready"); + expect(fake.log).toEqual(["stop 100 SIGTERM", "start 200"]); + }); + + it("fails when the replacement exits before it reports a status", async () => { + const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 1_000 }]; + const fake = fakeHooks(); + await expect(runRepeatedRestarts({ record: record("RR1", steps), steps, server: { pid: 100 }, healthyAt: fake.now(), stopOn: "recovering", + hooks: { ...fake.hooks, waitForStopPhase: async () => { throw new Error("Paperclip server exited during the crash loop"); } } })) + .rejects.toThrow(/exited/); + expect(fake.log).toEqual([]); + }); + + it("refuses to run without a phase watcher", async () => { + const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 1_000 }]; + const fake = fakeHooks(); + await expect(runRepeatedRestarts({ record: record("RR1", steps), steps, server: { pid: 100 }, healthyAt: fake.now(), stopOn: "recovering", + hooks: fake.hooks })).rejects.toThrow(/phase watcher/); + }); +}); + describe("grading a repeated restart as one fault", () => { it("crosses a task with one fault window from the first kill to the last replacement's ready", () => { const receipt = { diff --git a/tests/restart-chaos/repeated-restarts.ts b/tests/restart-chaos/repeated-restarts.ts index fe29bbf00ac..b102a88568a 100644 --- a/tests/restart-chaos/repeated-restarts.ts +++ b/tests/restart-chaos/repeated-restarts.ts @@ -49,14 +49,29 @@ export function planRestartSteps(kind: RepeatedRestartKind, int: (min: number, m const MIN_GAP_SECONDS = 1; const MAX_GAP_SECONDS = 300; -/** `--rr-gap-seconds ` or `-` (1–300 s): the gap from a replacement's health to the next stop. */ +/** `--rr-stop-on recovering`: stop each replacement on its first `recovering` status instead of after a gap. */ +export const RR_STOP_ON = ["recovering"] as const; +export type RepeatedRestartStopOn = (typeof RR_STOP_ON)[number]; + +/** + * `--rr-gap-seconds ` or `-` (1–300 s): the gap from a replacement's health to the next stop. + * `--rr-stop-on recovering`: no gap; each follow-up stop fires on the first `/api/ready` poll (every 50 ms + * from spawn) that reports `recovering`. On a local host `ready` follows health by about 0.3 s, so any + * gap from health lands after recovery. + */ export function parseRepeatedRestartOptions(options: Pick, - rawGapSeconds: string | undefined): Pick { + rawGapSeconds: string | undefined, rawStopOn?: string): Pick { const repeated = options.faults.some(isRepeatedRestartFault); if (rawGapSeconds !== undefined && !repeated) throw new Error("--rr-gap-seconds sets the RR restart gap; add RR1, RR2 or RR3 to --faults"); + if (rawStopOn !== undefined && !repeated) throw new Error("--rr-stop-on sets when RR restarts fire; add RR1, RR2 or RR3 to --faults"); if (!repeated) return {}; // A storm fault carries no planned restart steps, and a storm already restarts again and again. if (options.stormMinutes) throw new Error("Repeated restarts (RR*) are planned per task; do not combine them with --storm-minutes"); + if (rawStopOn !== undefined) { + if (!(RR_STOP_ON as readonly string[]).includes(rawStopOn)) throw new Error(`--rr-stop-on must be ${RR_STOP_ON.join(" or ")}`); + if (rawGapSeconds !== undefined) throw new Error("--rr-stop-on replaces the RR gap; do not combine it with --rr-gap-seconds"); + return { rrStopOn: rawStopOn as RepeatedRestartStopOn }; + } if (rawGapSeconds === undefined) return {}; const match = /^([0-9]+)(?:-([0-9]+))?$/.exec(rawGapSeconds.trim()); if (!match) throw new Error("--rr-gap-seconds takes or - seconds"); @@ -86,6 +101,10 @@ export interface RepeatedRestartEntry { export interface RepeatedRestartRecord { kind: RepeatedRestartKind; + /** `--rr-stop-on`: the follow-up stops fired on this status instead of after `planned[].gapMs`. */ + stopOn?: RepeatedRestartStopOn; + /** `stopOn`: the status the first replacement reported when the watch from its spawn fired. */ + firstStopPhase?: string; planned: RestartStep[]; restarts: RepeatedRestartEntry[]; /** `/repeated-restarts/.json`. */ @@ -111,10 +130,27 @@ export function recordFirstRestart(record: RepeatedRestartRecord, receipt: { sto const first = record.restarts[0]!; first.stoppedAt ??= receipt.stoppedAt; first.restartedAt ??= receipt.restartedAt; - first.healthyAt ??= receipt.healthyAt; + // The receipt's health is the first replacement's only before any follow-up stop; later it is the last + // replacement's. With `stopOn` the first replacement is never waited on to health at all. + if (!record.stopOn && record.restarts.length === 1) first.healthyAt ??= receipt.healthyAt; first.replacementPid ??= replacementPid; } +/** + * `stopOn`: the stop-status watcher for `runRepeatedRestarts`. The first replacement was already watched + * from its spawn (inside the first kill's restart) and fired with `firstPhase`; it is used once, for that + * server, so the first follow-up stop does not wait for health and miss a short recovery. Every later + * replacement is watched with `watch`. + */ +export function stopPhaseWatcher

(first: P, firstPhase: string | null | undefined, + watch: (server: P, target: RepeatedRestartStopOn) => Promise): (server: P, target: RepeatedRestartStopOn) => Promise { + let cached = firstPhase ?? null; + return async (server, target) => { + if (server === first && cached !== null) { const phase = cached; cached = null; return phase; } + return watch(server, target); + }; +} + export interface RepeatedRestartHooks

{ now(): number; sleep(ms: number): Promise; @@ -128,6 +164,11 @@ export interface RepeatedRestartHooks

{ start(): Promise

; /** Resolves with the time the replacement reported healthy. */ waitHealthy(server: P): Promise; + /** + * `stopOn`: poll `/api/ready` from the replacement's spawn until it reports `recovering` (or already `ready`, + * when recovery was too short to see), and resolve with that status. Rejects if the server exits first. + */ + waitForStopPhase?(server: P, target: RepeatedRestartStopOn): Promise; onEntry?(entry: RepeatedRestartEntry, stage: "stopping" | "healthy"): void; } @@ -142,19 +183,24 @@ export async function runRepeatedRestarts

(input: { server: P; healthyAt: number; hooks: RepeatedRestartHooks

; + /** `--rr-stop-on`: fire each follow-up stop on this status, polled from spawn, instead of after the gap. */ + stopOn?: RepeatedRestartStopOn; }): Promise<{ server: P; healthyAt: number }> { - const { record, steps, hooks } = input; + const { record, steps, hooks, stopOn } = input; + if (stopOn && !hooks.waitForStopPhase) throw new Error("--rr-stop-on needs a phase watcher"); let server = input.server; let healthyAt = input.healthyAt; - for (const step of steps) { + for (const [stepIndex, step] of steps.entries()) { const index = record.restarts.length; - const wait = healthyAt + step.gapMs - hooks.now(); - if (wait > 0) await hooks.sleep(wait); + if (!stopOn) { + const wait = healthyAt + step.gapMs - hooks.now(); + if (wait > 0) await hooks.sleep(wait); + } if (hooks.cancelled()) throw new Error(`Harness shutdown interrupted repeated restart ${index}`); const exited = hooks.exitStatus(server); if (exited) throw new Error(`Paperclip server exited on its own before repeated restart ${index} (code=${exited.code} signal=${exited.signal})`); - const phaseAtStop = await hooks.readPhase(); - const entry: RepeatedRestartEntry = { index, signal: step.signal, gapMs: step.gapMs, pid: hooks.pidOf(server), phaseAtStop, + const phaseAtStop = stopOn ? await hooks.waitForStopPhase!(server, stopOn) : await hooks.readPhase(); + const entry: RepeatedRestartEntry = { index, signal: step.signal, gapMs: stopOn ? null : step.gapMs, pid: hooks.pidOf(server), phaseAtStop, stopAt: hooks.now(), stoppedAt: null, restartedAt: null, replacementPid: null, healthyAt: null }; record.restarts.push(entry); hooks.onEntry?.(entry, "stopping"); @@ -164,6 +210,9 @@ export async function runRepeatedRestarts

(input: { entry.restartedAt = hooks.now(); server = await hooks.start(); entry.replacementPid = hooks.pidOf(server); + // `stopOn`: a replacement that is stopped again is watched from spawn, before it is healthy + // (`recovering` shows just before health). Only the last one is waited on to health. + if (stopOn && stepIndex < steps.length - 1) continue; healthyAt = await hooks.waitHealthy(server); entry.healthyAt = healthyAt; hooks.onEntry?.(entry, "healthy"); diff --git a/tests/restart-chaos/supervisor.ts b/tests/restart-chaos/supervisor.ts index ff7d1f85a9f..fe0a4b45ab5 100644 --- a/tests/restart-chaos/supervisor.ts +++ b/tests/restart-chaos/supervisor.ts @@ -35,7 +35,7 @@ import { runnerE2ETypeScriptProcessArgs } from "../runner-e2e/web-server-command import type { MatrixExecution } from "../runner-e2e/types.js"; import { DEFAULT_TIME_AGE_MS, ProcessFreezer, ageWaveDatabase, databaseUrlFromConfig, runFreezeFault, type AgingShift, type TimeFaultRecord } from "./time-faults.js"; import type { Timeline } from "./timeline.js"; -import { isRepeatedRestartFault, newRepeatedRestartRecord, recordFirstRestart, runRepeatedRestarts, type RepeatedRestartKind, +import { isRepeatedRestartFault, newRepeatedRestartRecord, recordFirstRestart, runRepeatedRestarts, stopPhaseWatcher, type RepeatedRestartKind, type RepeatedRestartStopOn, type RepeatedRestartRecord, type RestartStep } from "./repeated-restarts.js"; import type { FaultKind, FaultReceipt, TriggerId } from "./types.js"; @@ -138,6 +138,8 @@ export class ChaosSupervisor { crashPointHoldMs = 0; /** TM2/TM3: how far each aging fault shifts the wave's database back. */ timeAgeMs = DEFAULT_TIME_AGE_MS; + /** RR1–RR3 (`--rr-stop-on`): stop each replacement on this `/api/ready` status instead of after its gap. */ + rrStopOn: RepeatedRestartStopOn | null = null; /** Every aging shift so far, in order; the grader's API reads times through them (time-faults.ts). */ readonly agingShifts: AgingShift[] = []; private readonly createdAt = Date.now(); @@ -460,7 +462,15 @@ export class ChaosSupervisor { this.timeline.add("fault_restarting", { faultId: receipt.faultId }); return this.spawnServer(); }, - waitForReady: async replacement => { receipt.healthyAt = await this.waitForHealth(replacement, receipt.phases); }, + waitForReady: async replacement => { + // RR with `--rr-stop-on`: watch the first replacement from spawn; waiting for health first would let a + // short recovery pass between two 250 ms health polls. Only the last replacement is waited on to health. + if (receipt.repeatedRestart?.stopOn) { + receipt.repeatedRestart.firstStopPhase = await this.waitForCrashLoopPhase(replacement, receipt.repeatedRestart.stopOn, receipt.phases); + return; + } + receipt.healthyAt = await this.waitForHealth(replacement, receipt.phases); + }, cancelled: () => this.shuttingDown, record: async progress => { outage = progress; }, }); @@ -542,9 +552,10 @@ export class ChaosSupervisor { /** RR1–RR3: the planned steps and the first kill (as K1), recorded before that kill. */ private repeatedRestartRecord(kind: RepeatedRestartKind, receipt: FaultReceipt, planned: readonly RestartStep[] | undefined, firstPid: number | null): RepeatedRestartRecord { - return newRepeatedRestartRecord({ kind, planned, firstPid, startedAt: receipt.startedAt, + const record = newRepeatedRestartRecord({ kind, planned, firstPid, startedAt: receipt.startedAt, evidencePath: path.join(path.dirname(this.paths.serverLogPath), "repeated-restarts", `${receipt.faultId}.json`), int: (min, max) => min + Math.floor(Math.random() * (max - min + 1)) }); + return this.rrStopOn ? { ...record, stopOn: this.rrStopOn } : record; } /** @@ -554,8 +565,12 @@ export class ChaosSupervisor { private async repeatedRestarts(record: RepeatedRestartRecord, receipt: FaultReceipt): Promise { const first = this.child; recordFirstRestart(record, receipt, first?.pid ?? null); - if (!first || receipt.healthyAt === null) throw new Error(`${record.kind} has no healthy replacement after its first kill`); - const result = await runRepeatedRestarts({ record, steps: record.planned, server: first, healthyAt: receipt.healthyAt, hooks: { + if (!first || (receipt.healthyAt === null && !record.stopOn)) throw new Error(`${record.kind} has no healthy replacement after its first kill`); + // `--rr-stop-on`: the first replacement was already watched from spawn to its stop status (`firstStopPhase`). + const waitForStopPhase = stopPhaseWatcher(first, record.firstStopPhase, + (server: ChildProcess, target: RepeatedRestartStopOn) => this.waitForCrashLoopPhase(server, target, receipt.phases)); + const result = await runRepeatedRestarts({ record, steps: record.planned, server: first, healthyAt: receipt.healthyAt ?? Date.now(), + ...(record.stopOn ? { stopOn: record.stopOn } : {}), hooks: { now: () => Date.now(), sleep: ms => delay(ms), cancelled: () => this.shuttingDown, @@ -570,6 +585,8 @@ export class ChaosSupervisor { }, start: () => this.spawnServer(), waitHealthy: server => this.waitForHealth(server, receipt.phases), + // The K6 phase poll (50 ms): fires on `recovering`, or on `ready` when recovery was too short to see. + waitForStopPhase, onEntry: (entry, stage) => { if (stage === "stopping") { this.timeline.add("repeated_restart_stop", { faultId: receipt.faultId, index: entry.index, signal: entry.signal, gapMs: entry.gapMs, diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index 7f680dd70d0..b1d07b2563a 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -117,6 +117,8 @@ export interface WaveOptions { providerFaultRestartAfterMs?: number; /** RR1–RR3: the gap from a replacement's health to the next stop (`--rr-gap-seconds`); absent keeps each kind's default. */ rrGapMs?: [number, number]; + /** RR1–RR3 (`--rr-stop-on recovering`): stop each replacement on its first `recovering` status instead of after a gap. */ + rrStopOn?: "recovering"; } export interface StormFault {