diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 394f83f292c..7e9cb211a11 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -759,6 +759,48 @@ when the board decides after a day, and the approved subtask must exist exactly - Times kept outside Postgres are not aged: in-memory timers and the runner's own state files. Neither are times inside `text` columns or HMACs computed over a time. +## Repeated restarts (RR1–RR3, `--rr-gap-seconds`) + +```sh +# Each kind across running tasks. +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ + --adapters codex --workloads A1,B5,C1 --faults RR1,RR2,RR3 --concurrency 3 --tasks 6 --seed 401 +# A second kill while an approval is pending. +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ + --adapters codex --workloads G1 --faults RR1 --triggers approval --concurrency 2 --tasks 2 --seed 402 +``` + +Every other fault restarts the server once. Real deploy loops restart it again 5–60 s after it comes +back, while it is still recovering the runs the first restart interrupted. An RR fault starts as a K1 +(`SIGKILL` the server tree, start a replacement at once). After each planned gap, measured from the +moment the replacement reports healthy, it stops that replacement and starts another one. + +| Fault | Restarts | Default gap after health | +| --- | --- | --- | +| RR1 double crash | K1, then one more `SIGKILL` | 3–20 s | +| RR2 crash, then graceful | K1, then one `SIGTERM` (as K2) | 3–20 s | +| RR3 crash loop | 3–4 `SIGKILL`s in all; the last replacement stays up | 10–30 s | + +The gaps are seeded per task (`restartSteps` in `plan.json`), and drawn only for RR tasks, so the plans +of existing seeds do not change. `--rr-gap-seconds ` or `-` (1–300) replaces every RR +gap: for example `--rr-gap-seconds 5` for a stop that always lands in recovery, or `30-60` for a slow +deploy loop. RR faults do not combine with `--storm-minutes`, whose faults carry no planned steps. + +**Evidence.** Each stop and replacement (the first kill included) is recorded with its signal, planned +gap, stopped PID, the `/api/ready` status just before the stop (`recovering` means it landed inside +recovery), and the stopped, restarted and healthy times: +- in `/repeated-restarts/.json`, also written when the sequence fails part way (the planned steps are recorded before the first kill, so a replay of a failed sequence keeps its gaps); +- in the fault receipt (`repeatedRestart` in `faults.json`, and in each failure's `startup-phases.json`); +- in the timeline (`repeated_restart_stop`, `repeated_restart_healthy`). + +**Grading.** The whole sequence is one fault receipt, so it is one fault window: from the first kill to +the last replacement's ready. Crossing, R1's outage allowance, R5's ready clock, the T-oracle, task +deadlines and the restart-recovery speed budget treat it as one fault. No check is relaxed: the task +must reach its correct terminal state with no Retry (R3) and no duplicated work (R4). The observer does +not click Retry and no other fault fires until the last replacement is ready. A `--replay` of an RR +task restarts again after the same gaps. A replacement that exits on its own during a gap fails the +fault as a harness error, with the stops so far recorded. + ## Fault timing (`--triggers`) Each task samples one trigger with its cell. `time` (default) fires at the @@ -787,7 +829,7 @@ re-arms the task's own triggered faults instead of using their offsets. | File | Role | | --- | --- | -| `supervisor.ts` | Owns the server. Reuses `tests/runner-e2e` process code (config, ports, env isolation, wrapper-owned Postgres, process-tree stopper, `replaceController`). Faults: K1 SIGKILL + immediate restart, K2 SIGTERM + restart, K4 SIGKILL + 2–5 min hold, K6 crash loop. The controller runtime cache is kept (discarding it is K11). | +| `supervisor.ts` | Owns the server. Reuses `tests/runner-e2e` process code (config, ports, env isolation, wrapper-owned Postgres, process-tree stopper, `replaceController`). Faults: K1 SIGKILL + immediate restart, K2 SIGTERM + restart, K4 SIGKILL + 2–5 min hold, K6 crash loop, RR1–RR3 repeated restarts. The controller runtime cache is kept (discarding it is K11). | | `seed.ts` | Chaos Lab company, encrypted provider secrets, native Codex / ACPX Claude / OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project (git repo: python CLI, one failing test, `bench.sh` ≈ 4 min) with isolated git-worktree workspaces. It first turns on the instance `enableIsolatedWorkspaces` setting; without it the server ignores the project policy and every repo task shares the primary checkout. | | `scheduler.ts` | Seeded plan: workload × adapter × fault cells and a random fire time per task. Saved to `results//plan.json`. `--replay` reruns one task alone with the same choices and fires the recorded sequence of faults that crossed it (other tasks' faults included) at the same offsets from launch. | | `observer.ts` | Polls each issue and its runs. When a task rests needing Retry it calls `POST /api/issues/:id/retry` once (once more only if a later fault started after that click), with the body `useTaskRetry.ts` sends. Collects T-oracle artifacts and the R6 sweep. | @@ -803,6 +845,7 @@ re-arms the task's own triggered faults instead of using their offsets. | `acpx-provider-faults.ts`, `provider-shim/acpx-*` | PV1–PV3 for Claude (ACPX): the stand-in runner package, the server hook, the sidecar/provider tap, and `--provider-fault-restart`. | | `question-handoff.ts` | Q1 saved question: the board's answer step and its oracle. | | `time-faults.ts`, `time-faults-config.ts` | TM1–TM3: process freezing, database aging with recorded rows, and the time-corrected API the grader reads through. | +| `repeated-restarts.ts` | RR1–RR3: seeded restart steps, `--rr-gap-seconds`, and the stop-and-replace loop the supervisor runs. | | `daytona-sweep.ts` | Daytona R6: read-only reconciliation of this wave's labelled sandboxes with leases; cleanup only through Paperclip. | ## Output (gitignored) diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index 2a77eb86271..1463d90c1cc 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -5,6 +5,7 @@ import { applyJourneyOptions } from "./journey-matrix.js"; import { takesJourneyTriggers } from "./journeys.js"; import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js"; import { parseTimeFaultOptions } from "./time-faults-config.js"; +import { parseRepeatedRestartOptions } from "./repeated-restarts.js"; import { PROVIDER_FAULT_KINDS, PROVIDER_FAULT_WORKLOADS, isProviderFault, parseProviderDelays, parseProviderMessage, type ProviderFaultKind } from "./provider-faults.js"; import { PROVIDER_FAULT_ADAPTERS, parseProviderFaultRestart } from "./acpx-provider-faults.js"; import { @@ -116,13 +117,14 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC "time-age-hours": { type: "string" }, "provider-fault-restart": { type: "string" }, "provider-fault-restart-after-seconds": { type: "string" }, + "rr-gap-seconds": { type: "string" }, }, }); const keep = values.keep === true; const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000); const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000); if (values.replay !== undefined) { - for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds"] as const) { + for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds"] as const) { if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`); } return { mode: "replay", ref: parseReplayRef(values.replay), keep, @@ -220,6 +222,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults"); } Object.assign(options, parseTimeFaultOptions(options, values["time-age-hours"])); + Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"])); const upgradeFrom = values["upgrade-from"]; if (options.faults.includes("K14") !== (upgradeFrom !== undefined)) { throw new Error("K14 (upgrade mid-run) and --upgrade-from go together: pass both or neither"); diff --git a/tests/restart-chaos/evidence.ts b/tests/restart-chaos/evidence.ts index 67721b1cc95..e60053caa54 100644 --- a/tests/restart-chaos/evidence.ts +++ b/tests/restart-chaos/evidence.ts @@ -83,7 +83,8 @@ export async function writeEvidenceBundle(input: { await write("task-record.json", record); await write("faults.json", relevant.length ? relevant : input.faults); await write("startup-phases.json", (relevant.length ? relevant : input.faults).map(fault => ({ faultId: fault.faultId, kind: fault.kind, - startedAt: fault.startedAt, stoppedAt: fault.stoppedAt, restartedAt: fault.restartedAt, healthyAt: fault.healthyAt, readyAt: fault.readyAt, phases: fault.phases }))); + startedAt: fault.startedAt, stoppedAt: fault.stoppedAt, restartedAt: fault.restartedAt, healthyAt: fault.healthyAt, readyAt: fault.readyAt, phases: fault.phases, + ...(fault.repeatedRestart ? { repeatedRestarts: fault.repeatedRestart.restarts } : {}) }))); await write("timeline.json", input.timeline); if (input.artifacts) await write("oracle-artifacts.json", input.artifacts); await attempt("server-log", async () => write("server-log-window.txt", await logWindow(input.serverLogPath, relevant.length ? relevant : input.faults))); diff --git a/tests/restart-chaos/journey-matrix.ts b/tests/restart-chaos/journey-matrix.ts index 88b26271794..b88571c51b0 100644 --- a/tests/restart-chaos/journey-matrix.ts +++ b/tests/restart-chaos/journey-matrix.ts @@ -14,6 +14,7 @@ import { WORKLOAD_TRIGGERS } from "./journeys.js"; import { withProviderFaultRules } from "./provider-faults.js"; import { createRng, k4HoldRange, TRIGGER_DELAY_RANGE_MS } from "./scheduler.js"; import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js"; +import { isRepeatedRestartFault, planRestartSteps } from "./repeated-restarts.js"; import { workloadDefinitions } from "./workloads.js"; import type { AdapterId, TaskFaultKind, TaskPlan, TriggerId, WaveOptions, WavePlan, WorkloadId } from "./types.js"; @@ -41,7 +42,7 @@ export function matrixMayLaunch(next: Pick, inFlight: Readonl return inFlight.every(task => (task.fault === "K0") === baseline); } -export function planMatrixTasks(options: Pick): TaskPlan[] { +export function planMatrixTasks(options: Pick): TaskPlan[] { const rng = createRng((options.seed ^ 0x6a6d7478) >>> 0); return matrixCells(options).map((cell, index) => { const definition = workloadDefinitions[cell.workload]; @@ -57,6 +58,7 @@ export function planMatrixTasks(options: Pick { if (storm) timeline.add("storm_started", { durationMs: storm.plan.durationMs, plannedFaults: storm.plan.faults.length }); // Faults are serialized; each one hits every task in flight when it starts. const fireFault = (kind: FaultKind, holdMs: number, requestedBy: string, exposed: readonly TaskObserver[], - trigger?: { trigger: TriggerId; delayMs: number }, onSettled?: () => void) => { + trigger?: { trigger: TriggerId; delayMs: number }, onSettled?: () => void, restartSteps?: RestartStep[]) => { faultsQueued += 1; - faultWork.push(supervisor.fault(kind, holdMs, requestedBy, trigger).finally(() => { faultsQueued -= 1; onSettled?.(); }).then(receipt => { + faultWork.push(supervisor.fault(kind, holdMs, requestedBy, trigger, restartSteps).finally(() => { faultsQueued -= 1; onSettled?.(); }).then(receipt => { faults.push(receipt); checkWorktree(receipt.faultId); for (const candidate of exposed) { @@ -498,7 +499,7 @@ async function main(): Promise { const trigger = due.trigger && due.trigger !== "time" ? { trigger: due.trigger, delayMs: due.delayMs ?? 0 } : undefined; if (trigger) timeline.add("trigger_fired", { taskId: observer.plan.taskId, fault: due.kind, ...trigger }); // Inside a storm a triggered fault is a restart too: the next storm gap runs from its settle. - fireFault(due.kind, due.holdMs, observer.plan.taskId, exposed, trigger, storm ? () => storm.settled(Date.now()) : undefined); + fireFault(due.kind, due.holdMs, observer.plan.taskId, exposed, trigger, storm ? () => storm.settled(Date.now()) : undefined, due.restartSteps); } if (storm && !faultsQueued && !supervisor.faultInProgress && supervisor.serverReady) { const due = storm.due(Date.now()); diff --git a/tests/restart-chaos/repeated-restarts.test.ts b/tests/restart-chaos/repeated-restarts.test.ts new file mode 100644 index 00000000000..cc1f0eec243 --- /dev/null +++ b/tests/restart-chaos/repeated-restarts.test.ts @@ -0,0 +1,233 @@ +import { describe, expect, it } from "vitest"; +import { parseCommand } from "./config.js"; +import { crossingFaults } from "./grader.js"; +import { planMatrixWave } from "./journey-matrix.js"; +import { createRng, observedFaultSchedule, planTasks, taskFaultSchedule } from "./scheduler.js"; +import { + RR3_KILL_RANGE, RR_GAP_RANGE_MS, newRepeatedRestartRecord, planRestartSteps, recordFirstRestart, runRepeatedRestarts, type RepeatedRestartRecord, type RestartStep, +} from "./repeated-restarts.js"; +import type { FaultReceipt, TaskRecord } from "./types.js"; + +describe("repeated restart options", () => { + it("accepts RR faults with each kind's default gaps", () => { + const parsed = parseCommand(["--faults", "RR1,RR2,RR3", "--adapters", "codex", "--workloads", "A1", "--seed", "1"]); + if (parsed.mode !== "wave") throw new Error("expected a wave"); + expect(parsed.options.faults).toEqual(["RR1", "RR2", "RR3"]); + expect(parsed.options.rrGapMs).toBeUndefined(); + }); + + it("takes --rr-gap-seconds as one value or a range, only with an RR fault", () => { + const one = parseCommand(["--faults", "RR1", "--rr-gap-seconds", "5", "--workloads", "A1", "--seed", "1"]); + expect(one.mode === "wave" && one.options.rrGapMs).toEqual([5_000, 5_000]); + const range = parseCommand(["--faults", "RR3", "--rr-gap-seconds", "10-40", "--workloads", "A1", "--seed", "1"]); + expect(range.mode === "wave" && range.options.rrGapMs).toEqual([10_000, 40_000]); + expect(() => parseCommand(["--faults", "K1", "--rr-gap-seconds", "5", "--workloads", "A1"])).toThrow(/add RR1, RR2 or RR3/); + expect(() => parseCommand(["--faults", "RR1", "--rr-gap-seconds", "0", "--workloads", "A1"])).toThrow(/between 1 and 300/); + expect(() => parseCommand(["--faults", "RR1", "--rr-gap-seconds", "20-10", "--workloads", "A1"])).toThrow(/min not above max/); + expect(() => parseCommand(["--faults", "RR1", "--rr-gap-seconds", "5s", "--workloads", "A1"])).toThrow(/ or -/); + }); + + it("refuses a storm, whose faults carry no planned restarts", () => { + expect(() => parseCommand(["--faults", "RR1", "--storm-minutes", "10"])).toThrow(/storm/); + }); + + it("refuses --rr-gap-seconds with --replay", () => { + expect(() => parseCommand(["--replay", "w-20261110T000000-s1/t01", "--rr-gap-seconds", "5"])).toThrow(/--rr-gap-seconds/); + }); +}); + +describe("repeated restart planning", () => { + const within = (value: number, [min, max]: readonly [number, number]) => value >= min && value <= max; + + it("plans one SIGKILL (RR1), one SIGTERM (RR2) or 2-3 more SIGKILLs (RR3) after the first kill", () => { + for (let seed = 0; seed < 50; seed += 1) { + const rng = createRng(seed); + const rr1 = planRestartSteps("RR1", rng.int); + expect(rr1).toHaveLength(1); + expect(rr1[0]!.signal).toBe("SIGKILL"); + expect(within(rr1[0]!.gapMs, RR_GAP_RANGE_MS.RR1)).toBe(true); + const rr2 = planRestartSteps("RR2", rng.int); + expect(rr2.map(step => step.signal)).toEqual(["SIGTERM"]); + expect(within(rr2[0]!.gapMs, RR_GAP_RANGE_MS.RR2)).toBe(true); + const rr3 = planRestartSteps("RR3", rng.int); + expect(within(rr3.length + 1, RR3_KILL_RANGE)).toBe(true); + expect(rr3.every(step => step.signal === "SIGKILL" && within(step.gapMs, RR_GAP_RANGE_MS.RR3))).toBe(true); + } + }); + + it("draws both RR3 lengths over seeds", () => { + const lengths = new Set(Array.from({ length: 40 }, (_, seed) => planRestartSteps("RR3", createRng(seed).int).length)); + expect([...lengths].sort()).toEqual([2, 3]); + }); + + it("uses the --rr-gap-seconds range for every kind", () => { + const steps = planRestartSteps("RR3", createRng(3).int, [7_000, 7_000]); + expect(steps.every(step => step.gapMs === 7_000)).toBe(true); + }); + + it("gives RR tasks seeded restart steps and leaves the plans of other faults unchanged", () => { + const options = { seed: 11, tasks: 9, workloads: ["A1" as const], adapters: ["codex" as const] }; + const plain = planTasks({ ...options, faults: ["K1", "K2"] }); + expect(plain.every(task => task.restartSteps === undefined)).toBe(true); + expect(planTasks({ ...options, faults: ["K1", "K2"], rrGapMs: [5_000, 5_000] })).toEqual(plain); + const repeated = planTasks({ ...options, faults: ["RR1", "RR2", "RR3"] }); + expect(repeated.every(task => (task.restartSteps?.length ?? 0) >= 1 && task.holdMs === 0)).toBe(true); + expect(planTasks({ ...options, faults: ["RR1", "RR2", "RR3"] })).toEqual(repeated); + }); + + it("carries the steps into the task's fault schedule and the matrix plan", () => { + const [task] = planTasks({ seed: 5, tasks: 1, workloads: ["A1"], adapters: ["codex"], faults: ["RR3"] }); + expect(taskFaultSchedule(task!)).toEqual([{ kind: "RR3", holdMs: 0, atMs: task!.fireAfterMs, restartSteps: task!.restartSteps }]); + const parsed = parseCommand(["--matrix", "--workloads", "Q2", "--adapters", "codex", "--faults", "RR1", "--seed", "4"]); + if (parsed.mode !== "wave") throw new Error("expected a wave"); + const matrix = planMatrixWave("w-matrix", parsed.options); + const faulted = matrix.tasks.filter(cell => cell.fault === "RR1"); + expect(faulted.length).toBeGreaterThan(0); + expect(faulted.every(cell => cell.restartSteps?.length === 1)).toBe(true); + }); + + it("replays the recorded steps of a fault that crossed the task", () => { + const planned: RestartStep[] = [{ signal: "SIGKILL", gapMs: 12_000 }, { signal: "SIGKILL", gapMs: 25_000 }]; + const schedule = observedFaultSchedule(1_000, ["f01"], [{ faultId: "f01", kind: "RR3", holdMs: 0, startedAt: 31_000, + repeatedRestart: { planned } }]); + expect(schedule).toEqual([{ kind: "RR3", holdMs: 0, atMs: 30_000, restartSteps: planned }]); + }); +}); + +/** A fake server tree on a fake clock: each start takes `bootMs` to be healthy. */ +function fakeHooks(options: { bootMs?: number; stopMs?: number; exitDuringGap?: number; cancelAfterStops?: number } = {}) { + let clock = 100_000; + let nextPid = 200; + const log: string[] = []; + const exited = new Set(); + let stops = 0; + const hooks = { + now: () => clock, + sleep: async (ms: number) => { log.push(`sleep ${ms}`); clock += ms; }, + cancelled: () => options.cancelAfterStops !== undefined && stops >= options.cancelAfterStops, + pidOf: (server: { pid: number }) => server.pid, + exitStatus: (server: { pid: number }) => exited.has(server.pid) ? { code: 1, signal: null } : null, + readPhase: async () => "recovering", + stop: async (server: { pid: number }, signal: "SIGKILL" | "SIGTERM") => { + stops += 1; + log.push(`stop ${server.pid} ${signal}`); + clock += options.stopMs ?? 500; + exited.add(server.pid); + }, + start: async () => { + const server = { pid: nextPid++ }; + log.push(`start ${server.pid}`); + if (options.exitDuringGap === server.pid) exited.add(server.pid); + return server; + }, + waitHealthy: async () => { clock += options.bootMs ?? 8_000; return clock; }, + }; + return { hooks, log, now: () => clock }; +} + +function record(kind: "RR1" | "RR2" | "RR3", planned: RestartStep[]): RepeatedRestartRecord { + return { kind, planned, restarts: [] }; +} + +describe("the record built before the first kill", () => { + const planned: RestartStep[] = [{ signal: "SIGKILL", gapMs: 12_000 }, { signal: "SIGKILL", gapMs: 25_000 }]; + const build = (steps: RestartStep[] | undefined) => newRepeatedRestartRecord({ kind: "RR3", planned: steps, firstPid: 41, startedAt: 5_000, + evidencePath: "/wave/repeated-restarts/f01.json", int: createRng(9).int }); + + it("keeps the planned steps when the first replacement never becomes healthy", () => { + const rr = build(planned); + // The first restart failed: the server stopped and a replacement was spawned, but it never reported healthy. + recordFirstRestart(rr, { stoppedAt: 5_400, restartedAt: 5_400, healthyAt: null }, null); + expect(rr.planned).toEqual(planned); + expect(rr.planned).not.toBe(planned); + expect(rr.restarts).toEqual([{ index: 0, signal: "SIGKILL", gapMs: null, pid: 41, phaseAtStop: "ready", stopAt: 5_000, + stoppedAt: 5_400, restartedAt: 5_400, replacementPid: null, healthyAt: null }]); + // A replay of the failed fault fires the same gaps. + expect(observedFaultSchedule(1_000, ["f01"], [{ faultId: "f01", kind: "RR3", holdMs: 0, startedAt: 5_000, repeatedRestart: rr }])[0]!.restartSteps) + .toEqual(planned); + }); + + it("draws a sequence of its kind when no steps were planned", () => { + expect(build(undefined).planned).toEqual(planRestartSteps("RR3", createRng(9).int)); + }); + + it("does not overwrite first-kill times already recorded", () => { + const rr = build(planned); + recordFirstRestart(rr, { stoppedAt: 5_400, restartedAt: 5_400, healthyAt: 13_000 }, 77); + recordFirstRestart(rr, { stoppedAt: 9_999, restartedAt: 9_999, healthyAt: 99_999 }, null); + expect(rr.restarts[0]).toMatchObject({ stoppedAt: 5_400, restartedAt: 5_400, healthyAt: 13_000, replacementPid: 77 }); + }); +}); + +describe("runRepeatedRestarts", () => { + it("stops each replacement its gap after it reported healthy, then starts the next", async () => { + const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 10_000 }, { signal: "SIGKILL", gapMs: 20_000 }, { signal: "SIGKILL", gapMs: 15_000 }]; + const fake = fakeHooks({ bootMs: 6_000, stopMs: 400 }); + const rr = record("RR3", steps); + const result = await runRepeatedRestarts({ record: rr, steps, server: { pid: 100 }, healthyAt: fake.now(), hooks: fake.hooks }); + expect(fake.log).toEqual([ + "sleep 10000", "stop 100 SIGKILL", "start 200", + "sleep 20000", "stop 200 SIGKILL", "start 201", + "sleep 15000", "stop 201 SIGKILL", "start 202", + ]); + expect(result.server).toEqual({ pid: 202 }); + expect(rr.restarts.map(entry => [entry.index, entry.pid, entry.replacementPid, entry.phaseAtStop])).toEqual([ + [0, 100, 200, "recovering"], [1, 200, 201, "recovering"], [2, 201, 202, "recovering"]]); + for (const [index, entry] of rr.restarts.entries()) { + const previousHealthy = index === 0 ? 100_000 : rr.restarts[index - 1]!.healthyAt!; + expect(entry.stopAt - previousHealthy).toBe(steps[index]!.gapMs); + expect(entry.stoppedAt! - entry.stopAt).toBe(400); + expect(entry.healthyAt! - entry.restartedAt!).toBe(6_000); + } + expect(result.healthyAt).toBe(rr.restarts[2]!.healthyAt); + }); + + it("sends SIGTERM for a graceful step (RR2)", async () => { + const steps: RestartStep[] = [{ signal: "SIGTERM", gapMs: 3_000 }]; + const fake = fakeHooks(); + const rr = record("RR2", steps); + await runRepeatedRestarts({ record: rr, steps, server: { pid: 100 }, healthyAt: fake.now(), hooks: fake.hooks }); + expect(fake.log).toEqual(["sleep 3000", "stop 100 SIGTERM", "start 200"]); + expect(rr.restarts[0]!.signal).toBe("SIGTERM"); + }); + + it("does not wait when the gap already passed (the health check returned late)", async () => { + const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 3_000 }]; + const fake = fakeHooks(); + await runRepeatedRestarts({ record: record("RR1", steps), steps, server: { pid: 100 }, healthyAt: fake.now() - 5_000, hooks: fake.hooks }); + expect(fake.log).toEqual(["stop 100 SIGKILL", "start 200"]); + }); + + it("fails, with what fired recorded, when a replacement exits on its own during a gap", async () => { + const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 10_000 }, { signal: "SIGKILL", gapMs: 10_000 }]; + const fake = fakeHooks({ exitDuringGap: 200 }); + const rr = record("RR3", steps); + await expect(runRepeatedRestarts({ record: rr, steps, server: { pid: 100 }, healthyAt: fake.now(), hooks: fake.hooks })) + .rejects.toThrow(/exited on its own before repeated restart 1/); + expect(rr.restarts).toHaveLength(1); + expect(fake.log).not.toContain("stop 200 SIGKILL"); + }); + + it("stops at harness shutdown without starting another replacement", async () => { + const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 1_000 }, { signal: "SIGKILL", gapMs: 1_000 }]; + const fake = fakeHooks({ cancelAfterStops: 1 }); + await expect(runRepeatedRestarts({ record: record("RR3", steps), steps, server: { pid: 100 }, healthyAt: fake.now(), hooks: fake.hooks })) + .rejects.toThrow(/shutdown/); + expect(fake.log).toEqual(["sleep 1000", "stop 100 SIGKILL"]); + }); +}); + +describe("grading a repeated restart as one fault", () => { + it("crosses a task with one fault window from the first kill to the last replacement's ready", () => { + const receipt = { + faultId: "f01", kind: "RR3", requestedBy: "t01", signal: "SIGKILL", holdMs: 0, requestedAt: 10_000, startedAt: 10_000, + stoppedAt: 10_500, restartedAt: 10_500, healthyAt: 90_000, readyAt: 95_000, phases: [], serverLogOffsetAtStart: 0, + serverLogOffsetAtReady: 0, error: null, + } satisfies FaultReceipt; + // A run that started between the first and the last kill overlaps the one window. + const task = { plan: { taskId: "t02" }, createdAt: 50_000, restAt: 200_000, samples: [{ at: 60_000, reachable: true, serverReady: false, + runs: [{ id: "r1", status: "succeeded", createdAt: new Date(50_000).toISOString(), startedAt: null, + finishedAt: new Date(70_000).toISOString(), errorCode: null, retryOfRunId: null, wakeCommentIds: [], livenessState: null }] }] } as unknown as TaskRecord; + expect(crossingFaults(task, [receipt])).toEqual(["f01"]); + }); +}); diff --git a/tests/restart-chaos/repeated-restarts.ts b/tests/restart-chaos/repeated-restarts.ts new file mode 100644 index 00000000000..fe29bbf00ac --- /dev/null +++ b/tests/restart-chaos/repeated-restarts.ts @@ -0,0 +1,172 @@ +/** + * Repeated restarts (RR1–RR3): a restart that lands while the server is still recovering from the + * previous one, as in a deploy loop. Each RR fault starts as a K1 (SIGKILL the server tree, start a + * replacement at once). Then, after each planned gap from the moment the replacement reports + * healthy, it stops the replacement again and starts another one: + * + * - RR1 double crash: one more SIGKILL. + * - RR2 crash, then graceful: one SIGTERM (as K2). + * - RR3 crash loop: 3–4 SIGKILLs in all, then the last replacement stays up. + * + * The whole sequence is one fault receipt, so the grader sees one fault window from the first kill + * to the last replacement's ready. Planning, option parsing and the step loop live here with no + * process code, so they are unit-tested; the supervisor supplies the process hooks. + */ +import type { WaveOptions } from "./types.js"; + +export const REPEATED_RESTART_KINDS = ["RR1", "RR2", "RR3"] as const; +export type RepeatedRestartKind = (typeof REPEATED_RESTART_KINDS)[number]; + +export function isRepeatedRestartFault(kind: string): kind is RepeatedRestartKind { + return (REPEATED_RESTART_KINDS as readonly string[]).includes(kind); +} + +/** One restart after the first kill: wait `gapMs` from the previous replacement's health, then stop it with `signal`. */ +export interface RestartStep { signal: "SIGKILL" | "SIGTERM"; gapMs: number } + +/** Default gaps, from health to the next stop: inside the recovery window (RR1, RR2), or a loop of deploys (RR3). */ +export const RR_GAP_RANGE_MS: Record = { + RR1: [3_000, 20_000], + RR2: [3_000, 20_000], + RR3: [10_000, 30_000], +}; + +/** RR3: how many kills in all, the first one included. */ +export const RR3_KILL_RANGE: readonly [number, number] = [3, 4]; + +/** The seeded restarts that follow the first kill. `gapRangeMs` (`--rr-gap-seconds`) replaces the default gaps. */ +export function planRestartSteps(kind: RepeatedRestartKind, int: (min: number, max: number) => number, + gapRangeMs?: readonly [number, number]): RestartStep[] { + const range = gapRangeMs ?? RR_GAP_RANGE_MS[kind]; + if (kind === "RR1") return [{ signal: "SIGKILL", gapMs: int(...range) }]; + if (kind === "RR2") return [{ signal: "SIGTERM", gapMs: int(...range) }]; + const kills = int(...RR3_KILL_RANGE); + const steps: RestartStep[] = []; + for (let index = 1; index < kills; index += 1) steps.push({ signal: "SIGKILL", gapMs: int(...range) }); + return steps; +} + +const MIN_GAP_SECONDS = 1; +const MAX_GAP_SECONDS = 300; + +/** `--rr-gap-seconds ` or `-` (1–300 s): the gap from a replacement's health to the next stop. */ +export function parseRepeatedRestartOptions(options: Pick, + rawGapSeconds: string | undefined): Pick { + const repeated = options.faults.some(isRepeatedRestartFault); + if (rawGapSeconds !== undefined && !repeated) throw new Error("--rr-gap-seconds sets the RR restart gap; add RR1, RR2 or RR3 to --faults"); + if (!repeated) return {}; + // A storm fault carries no planned restart steps, and a storm already restarts again and again. + if (options.stormMinutes) throw new Error("Repeated restarts (RR*) are planned per task; do not combine them with --storm-minutes"); + if (rawGapSeconds === undefined) return {}; + const match = /^([0-9]+)(?:-([0-9]+))?$/.exec(rawGapSeconds.trim()); + if (!match) throw new Error("--rr-gap-seconds takes or - seconds"); + const min = Number(match[1]); + const max = match[2] === undefined ? min : Number(match[2]); + if (min < MIN_GAP_SECONDS || max > MAX_GAP_SECONDS || max < min) { + throw new Error(`--rr-gap-seconds must be between ${MIN_GAP_SECONDS} and ${MAX_GAP_SECONDS} seconds, min not above max`); + } + return { rrGapMs: [min * 1000, max * 1000] }; +} + +/** One stop and replacement in the sequence; index 0 is the first (K1) kill. */ +export interface RepeatedRestartEntry { + index: number; + signal: "SIGKILL" | "SIGTERM"; + /** Planned wait from the previous replacement's health; null for the first kill. */ + gapMs: number | null; + pid: number | null; + /** `/api/ready` status just before the stop: `recovering` means the stop landed inside recovery. */ + phaseAtStop: string | null; + stopAt: number; + stoppedAt: number | null; + restartedAt: number | null; + replacementPid: number | null; + healthyAt: number | null; +} + +export interface RepeatedRestartRecord { + kind: RepeatedRestartKind; + planned: RestartStep[]; + restarts: RepeatedRestartEntry[]; + /** `/repeated-restarts/.json`. */ + evidencePath?: string; +} + +/** + * The record of an RR fault, built before its first kill: the planned steps and the first (K1) kill. A + * first restart that fails still leaves them in the receipt, so a replay reuses the same gaps. + * Absent planned steps (a fault fired outside a planned task) are drawn with `int`. + */ +export function newRepeatedRestartRecord(input: { kind: RepeatedRestartKind; planned: readonly RestartStep[] | undefined; + firstPid: number | null; startedAt: number; evidencePath: string; int: (min: number, max: number) => number }): RepeatedRestartRecord { + const steps = input.planned?.length ? input.planned : planRestartSteps(input.kind, input.int); + return { kind: input.kind, planned: steps.map(step => ({ ...step })), restarts: [{ index: 0, signal: "SIGKILL", gapMs: null, + pid: input.firstPid, phaseAtStop: "ready", stopAt: input.startedAt, stoppedAt: null, restartedAt: null, replacementPid: null, healthyAt: null }], + evidencePath: input.evidencePath }; +} + +/** Fills the first kill's times from the receipt, whether its replacement became healthy or the fault failed. */ +export function recordFirstRestart(record: RepeatedRestartRecord, receipt: { stoppedAt: number | null; restartedAt: number | null; + healthyAt: number | null }, replacementPid: number | null): void { + const first = record.restarts[0]!; + first.stoppedAt ??= receipt.stoppedAt; + first.restartedAt ??= receipt.restartedAt; + first.healthyAt ??= receipt.healthyAt; + first.replacementPid ??= replacementPid; +} + +export interface RepeatedRestartHooks

{ + now(): number; + sleep(ms: number): Promise; + cancelled(): boolean; + pidOf(server: P): number | null; + /** The server's exit status once it has exited on its own, else null. */ + exitStatus(server: P): { code: number | null; signal: string | null } | null; + readPhase(): Promise; + /** Stop the server with `signal` and wait until it no longer answers health checks. */ + stop(server: P, signal: "SIGKILL" | "SIGTERM"): Promise; + start(): Promise

; + /** Resolves with the time the replacement reported healthy. */ + waitHealthy(server: P): Promise; + onEntry?(entry: RepeatedRestartEntry, stage: "stopping" | "healthy"): void; +} + +/** + * Runs the planned restarts after the first kill's replacement reported healthy at `healthyAt`. + * Each gap runs from the previous replacement's health. Every entry is pushed to `record.restarts` + * before its stop, so a failure part way still leaves the evidence of what fired. + */ +export async function runRepeatedRestarts

(input: { + record: RepeatedRestartRecord; + steps: readonly RestartStep[]; + server: P; + healthyAt: number; + hooks: RepeatedRestartHooks

; +}): Promise<{ server: P; healthyAt: number }> { + const { record, steps, hooks } = input; + let server = input.server; + let healthyAt = input.healthyAt; + for (const step of steps) { + const index = record.restarts.length; + const wait = healthyAt + step.gapMs - hooks.now(); + if (wait > 0) await hooks.sleep(wait); + if (hooks.cancelled()) throw new Error(`Harness shutdown interrupted repeated restart ${index}`); + const exited = hooks.exitStatus(server); + if (exited) throw new Error(`Paperclip server exited on its own before repeated restart ${index} (code=${exited.code} signal=${exited.signal})`); + const phaseAtStop = await hooks.readPhase(); + const entry: RepeatedRestartEntry = { index, signal: step.signal, gapMs: step.gapMs, pid: hooks.pidOf(server), phaseAtStop, + stopAt: hooks.now(), stoppedAt: null, restartedAt: null, replacementPid: null, healthyAt: null }; + record.restarts.push(entry); + hooks.onEntry?.(entry, "stopping"); + await hooks.stop(server, step.signal); + entry.stoppedAt = hooks.now(); + if (hooks.cancelled()) throw new Error(`Harness shutdown interrupted repeated restart ${index}`); + entry.restartedAt = hooks.now(); + server = await hooks.start(); + entry.replacementPid = hooks.pidOf(server); + healthyAt = await hooks.waitHealthy(server); + entry.healthyAt = healthyAt; + hooks.onEntry?.(entry, "healthy"); + } + return { server, healthyAt }; +} diff --git a/tests/restart-chaos/scheduler.ts b/tests/restart-chaos/scheduler.ts index a1fdca3d39b..54c3514d5c3 100644 --- a/tests/restart-chaos/scheduler.ts +++ b/tests/restart-chaos/scheduler.ts @@ -2,6 +2,7 @@ import { isProviderFault, withProviderFaultRules } from "./provider-faults.js"; import { takesJourneyTriggers } from "./journeys.js"; import { workloadDefinitions } from "./workloads.js"; import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js"; +import { isRepeatedRestartFault, planRestartSteps, type RestartStep } from "./repeated-restarts.js"; import type { AdapterId, FaultKind, ScheduledFault, StormFault, StormPlan, TaskFaultKind, TaskPlan, TriggerId, WavePlan, WaveOptions, WorkloadId } from "./types.js"; import type { ReplayRef } from "./config.js"; @@ -93,7 +94,7 @@ function pinnedFireAfterMs(faultAfterMs: number | undefined, drawn: number): num return faultAfterMs ?? drawn; } -export function planTasks(options: Pick): TaskPlan[] { +export function planTasks(options: Pick): TaskPlan[] { const rng = createRng(options.seed); const triggers = options.triggers?.length ? options.triggers : ["time" as const]; for (const trigger of triggers) { @@ -125,6 +126,8 @@ export function planTasks(options: Pick, taskId?: string): ScheduledFault[] { + trigger?: TriggerId; triggerDelayMs?: number; repeatedRestart?: { planned: RestartStep[] } }>, taskId?: string): ScheduledFault[] { if (createdAt === null) return []; return faults.filter(fault => faultIds.includes(fault.faultId)) .map((fault): ScheduledFault => ({ kind: fault.kind, holdMs: fault.holdMs, atMs: Math.max(0, fault.startedAt - createdAt), + // RR: the replay restarts again after the same gaps. + ...(fault.repeatedRestart ? { restartSteps: fault.repeatedRestart.planned.map(step => ({ ...step })) } : {}), ...(fault.trigger && fault.trigger !== "time" && fault.requestedBy === taskId ? { trigger: fault.trigger, delayMs: fault.triggerDelayMs ?? 0 } : {}) })) .sort((a, b) => a.atMs - b.atMs); @@ -287,7 +292,8 @@ export function taskFaultSchedule(task: TaskPlan): ScheduledFault[] { const restart = task.providerFault?.restart; if (restart) return [{ kind: restart.kind, holdMs: 0, atMs: 0, trigger: "provider", delayMs: restart.afterAppliedMs }]; if (task.fault === "K13" || task.fault === "K0" || isProviderFault(task.fault)) return []; - const planned: ScheduledFault = { kind: task.fault, holdMs: task.holdMs, atMs: task.fireAfterMs }; + const planned: ScheduledFault = { kind: task.fault, holdMs: task.holdMs, atMs: task.fireAfterMs, + ...(task.restartSteps ? { restartSteps: task.restartSteps } : {}) }; const trigger = task.trigger ?? "time"; if (trigger === "time") return [planned]; const delayMs = task.triggerDelayMs ?? 0; diff --git a/tests/restart-chaos/supervisor.ts b/tests/restart-chaos/supervisor.ts index e3adf831f1b..ff7d1f85a9f 100644 --- a/tests/restart-chaos/supervisor.ts +++ b/tests/restart-chaos/supervisor.ts @@ -16,7 +16,7 @@ import { execFileSync, spawn, type ChildProcess } from "node:child_process"; import { randomBytes } from "node:crypto"; import { createWriteStream, readFileSync, statSync, writeFileSync, type WriteStream } from "node:fs"; -import { mkdir, readFile, symlink } from "node:fs/promises"; +import { mkdir, readFile, symlink, writeFile } from "node:fs/promises"; import { createRequire } from "node:module"; import os from "node:os"; import path from "node:path"; @@ -35,6 +35,8 @@ import { runnerE2ETypeScriptProcessArgs } from "../runner-e2e/web-server-command import type { MatrixExecution } from "../runner-e2e/types.js"; import { DEFAULT_TIME_AGE_MS, ProcessFreezer, ageWaveDatabase, databaseUrlFromConfig, runFreezeFault, type AgingShift, type TimeFaultRecord } from "./time-faults.js"; import type { Timeline } from "./timeline.js"; +import { isRepeatedRestartFault, newRepeatedRestartRecord, recordFirstRestart, runRepeatedRestarts, type RepeatedRestartKind, + type RepeatedRestartRecord, type RestartStep } from "./repeated-restarts.js"; import type { FaultKind, FaultReceipt, TriggerId } from "./types.js"; /** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */ @@ -46,7 +48,7 @@ const READY_TIMEOUT_MS = 300_000; const STOP_TIMEOUT_MS = 30_000; export const FAULT_SIGNAL: Record = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM", - TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL" }; + TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL" }; /** The checkout's commit, for the K14 receipt. */ function checkoutSha(root: string): string | null { @@ -400,15 +402,16 @@ export class ChaosSupervisor { } /** Faults are serialized: one control plane, one outage at a time. */ - fault(kind: FaultKind, holdMs: number, requestedBy: string, trigger?: { trigger: TriggerId; delayMs: number }): Promise { + fault(kind: FaultKind, holdMs: number, requestedBy: string, trigger?: { trigger: TriggerId; delayMs: number }, + restartSteps?: readonly RestartStep[]): Promise { const requestedAt = Date.now(); - const run = this.faultChain.then(() => this.executeFault(kind, holdMs, requestedBy, requestedAt, trigger)); + const run = this.faultChain.then(() => this.executeFault(kind, holdMs, requestedBy, requestedAt, trigger, restartSteps)); this.faultChain = run.catch(() => undefined); return run; } private async executeFault(kind: FaultKind, holdMs: number, requestedBy: string, requestedAt: number, - trigger?: { trigger: TriggerId; delayMs: number }): Promise { + trigger?: { trigger: TriggerId; delayMs: number }, restartSteps?: readonly RestartStep[]): Promise { const previous = this.child; this.faultCount += 1; const receipt: FaultReceipt = { @@ -418,6 +421,8 @@ export class ChaosSupervisor { requestedAt, startedAt: Date.now(), stoppedAt: null, restartedAt: null, healthyAt: null, readyAt: null, phases: [], serverLogOffsetAtStart: this.serverLogOffset(), serverLogOffsetAtReady: null, error: null, }; + // RR: the planned steps and the first kill are recorded before it, so a failed first restart keeps them for replay. + if (isRepeatedRestartFault(kind)) receipt.repeatedRestart = this.repeatedRestartRecord(kind, receipt, restartSteps, previous?.pid ?? null); this.faultInProgress = true; this.serverReady = false; this.timeline.add("fault_started", { faultId: receipt.faultId, fault: kind, signal: receipt.signal, loadAvg1m: receipt.loadAvg1m, holdMs, requestedBy, previousPid: previous?.pid }); @@ -463,6 +468,7 @@ export class ChaosSupervisor { if (!this.child) throw new Error("K6 crash loop has no replacement server to kill"); receipt.healthyAt = await this.waitForHealth(await this.crashLoop(this.child, receipt), receipt.phases); } + if (receipt.repeatedRestart) receipt.healthyAt = await this.repeatedRestarts(receipt.repeatedRestart, receipt); receipt.readyAt = await this.waitForReadyPhase(receipt.phases); receipt.serverLogOffsetAtReady = this.serverLogOffset(); this.timeline.add("fault_ready", { faultId: receipt.faultId, replacementPid: this.child?.pid ?? outage?.replacementPid, @@ -472,6 +478,7 @@ export class ChaosSupervisor { this.timeline.add("fault_failed", { faultId: receipt.faultId, error: receipt.error }); throw Object.assign(new Error(`Fault ${receipt.faultId} failed: ${receipt.error}`), { receipt }); } finally { + if (receipt.repeatedRestart) await this.writeRepeatedRestartEvidence(receipt); this.faultInProgress = false; } return receipt; @@ -532,6 +539,64 @@ export class ChaosSupervisor { return upgrade; } + /** RR1–RR3: the planned steps and the first kill (as K1), recorded before that kill. */ + private repeatedRestartRecord(kind: RepeatedRestartKind, receipt: FaultReceipt, planned: readonly RestartStep[] | undefined, + firstPid: number | null): RepeatedRestartRecord { + return newRepeatedRestartRecord({ kind, planned, firstPid, startedAt: receipt.startedAt, + evidencePath: path.join(path.dirname(this.paths.serverLogPath), "repeated-restarts", `${receipt.faultId}.json`), + int: (min, max) => min + Math.floor(Math.random() * (max - min + 1)) }); + } + + /** + * RR1–RR3: the first kill has a healthy replacement. Stop it again after each planned gap, while it + * is still recovering, and start another. The sequence is one receipt: one fault window. + */ + private async repeatedRestarts(record: RepeatedRestartRecord, receipt: FaultReceipt): Promise { + const first = this.child; + recordFirstRestart(record, receipt, first?.pid ?? null); + if (!first || receipt.healthyAt === null) throw new Error(`${record.kind} has no healthy replacement after its first kill`); + const result = await runRepeatedRestarts({ record, steps: record.planned, server: first, healthyAt: receipt.healthyAt, hooks: { + now: () => Date.now(), + sleep: ms => delay(ms), + cancelled: () => this.shuttingDown, + pidOf: server => server.pid ?? null, + exitStatus: server => this.exited(server) ? { code: server.exitCode, signal: server.signalCode } : null, + readPhase: () => this.readPhase(), + stop: async (server, signal) => { + if (signal === "SIGKILL") await this.stopper.crashForRestart(server); + else await this.stopper(server, "SIGTERM"); + if (this.child === server) this.child = null; + await this.waitForHealthToStop(); + }, + start: () => this.spawnServer(), + waitHealthy: server => this.waitForHealth(server, receipt.phases), + onEntry: (entry, stage) => { + if (stage === "stopping") { + this.timeline.add("repeated_restart_stop", { faultId: receipt.faultId, index: entry.index, signal: entry.signal, gapMs: entry.gapMs, + pid: entry.pid, phaseAtStop: entry.phaseAtStop }); + this.log.write(`\n[restart-chaos] ${receipt.faultId} ${record.kind} restart ${entry.index} ${entry.signal} pid=${entry.pid} phase=${entry.phaseAtStop} gap=${entry.gapMs}ms\n`); + } else { + this.timeline.add("repeated_restart_healthy", { faultId: receipt.faultId, index: entry.index, replacementPid: entry.replacementPid, + downMs: (entry.healthyAt ?? 0) - entry.stopAt }); + } + }, + } }); + return result.healthyAt; + } + + /** `/repeated-restarts/.json`, written whether the sequence passed or failed; faults.json carries the same record. */ + private async writeRepeatedRestartEvidence(receipt: FaultReceipt): Promise { + const record = receipt.repeatedRestart!; + // A first kill that failed before its replacement was healthy still records the times it reached. + recordFirstRestart(record, receipt, null); + try { + await mkdir(path.dirname(record.evidencePath!), { recursive: true }); + await writeFile(record.evidencePath!, `${JSON.stringify({ faultId: receipt.faultId, requestedBy: receipt.requestedBy, error: receipt.error, ...record }, null, 2)}\n`, { mode: 0o600 }); + } catch (error) { + this.log.write(`[restart-chaos] ${receipt.faultId} could not write repeated-restart evidence: ${error instanceof Error ? error.message : String(error)}\n`); + } + } + private async waitForCrashLoopPhase(child: ChildProcess, target: (typeof CRASH_LOOP_TARGETS)[number], phases: FaultReceipt["phases"]): Promise { const deadline = Date.now() + READY_TIMEOUT_MS; diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index 0791fb889c0..d468301e086 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -1,13 +1,15 @@ /** Shared types for the control-plane restart chaos harness (plan §5). */ import type { ProviderFaultEvent, ProviderFaultKind, ProviderFaultRule, ProviderMessage } from "./provider-faults.js"; import type { QuestionArtifacts, QuestionRecord } from "./question-handoff.js"; +import type { RepeatedRestartRecord, RestartStep } from "./repeated-restarts.js"; import type { TimeFaultRecord } from "./time-faults.js"; /** * K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`). * TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage. + * RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers. */ -export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3"] as const; +export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3"] as const; export type FaultKind = (typeof FAULT_KINDS)[number]; /** * A task's planned fault. "K13" (restart storm) tasks have none of their own; the wave's storm crosses them. @@ -112,6 +114,8 @@ export interface WaveOptions { /** `--provider-fault-restart K1`: each PV1/PV2 task also kills -9 the control plane after its fault applied. */ providerFaultRestart?: "K1"; providerFaultRestartAfterMs?: number; + /** RR1–RR3: the gap from a replacement's health to the next stop (`--rr-gap-seconds`); absent keeps each kind's default. */ + rrGapMs?: [number, number]; } export interface StormFault { @@ -149,6 +153,8 @@ export interface TaskPlan { replayFaults?: ScheduledFault[]; /** PV1–PV3 tasks: the provider message rule armed for this task's nonce. */ providerFault?: ProviderFaultRule; + /** RR1–RR3: the seeded restarts after the first kill. */ + restartSteps?: RestartStep[]; } export interface ScheduledFault { @@ -156,6 +162,8 @@ export interface ScheduledFault { atMs: number; trigger?: TriggerId; delayMs?: number; + /** RR1–RR3: the restarts after the first kill. */ + restartSteps?: RestartStep[]; } export interface WavePlan { @@ -254,6 +262,8 @@ export interface FaultReceipt { upgrade?: { fromRoot: string; fromSha: string | null; toRoot: string; toSha: string | null; switched: boolean }; /** TM1–TM3: what was frozen and which rows were aged. */ timeFault?: TimeFaultRecord; + /** RR1–RR3: every stop and replacement of the sequence, the first kill included. */ + repeatedRestart?: RepeatedRestartRecord; serverLogOffsetAtStart: number; serverLogOffsetAtReady: number | null; error: string | null;