test(restart-chaos): add repeated restarts (RR1-RR3): a restart inside the previous restart's recovery (#20480)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart chaos harness (`tests/restart-chaos/`) tests if agent
tasks survive control-plane restarts.
> - Every existing fault restarts the server once per task.
> - Real incidents on 2026-11-08 and 2026-11-09 were deploy loops. The
server restarted again 5–60 s after it came back, while recovery of
interrupted runs was still in progress.
> - No wave puts a second restart inside the first restart's recovery
window.
> - This pull request adds a fault family, repeated restarts (RR1–RR3).
It does not change product code.
> - The benefit is that waves now test recovery that is itself
interrupted.

## Linked Issues or Issue Description

**What is the problem?**
The chaos harness restarts the server only once per fault. Production
deploy loops restart it again while it recovers, and no wave tests that.

**What is the solution?**
Add three repeated-restart faults. Each one starts as a K1 and then
stops the replacement again after a seeded gap from its health.

**Why is it needed?**
Recovery of interrupted runs can itself be interrupted. A second restart
in that window can leave a lease, a claim or a wake half done.

Refs #20458 (the TM fault family this one follows).

## What Changed

- Add `RR1`, `RR2` and `RR3` to `--faults` (`repeated-restarts.ts`):
- **RR1 double crash**: K1, then one more `SIGKILL` 3–20 s after the
replacement reports healthy.
- **RR2 crash, then graceful**: K1, then one `SIGTERM` (as K2) 3–20 s
after health.
- **RR3 crash loop**: 3–4 `SIGKILL`s in all, 10–30 s after each health.
The last replacement stays up.
- Add `--rr-gap-seconds <n>` or `<min>-<max>` (1–300). It replaces every
RR gap. RR faults do not combine with `--storm-minutes`.
- The gaps are seeded per task (`restartSteps` in `plan.json`, also in
`--matrix` plans). They are drawn only for RR tasks, so the plans of
existing seeds do not change. `--replay` restarts again after the
recorded gaps.
- The supervisor runs the whole sequence as one fault receipt. The
grader therefore sees one fault window, from the first kill to the last
replacement's ready. Crossing, R1's outage allowance, R5, the T-oracle,
task deadlines and the recovery speed budget treat it as one fault. No
check is relaxed.
- Each stop and replacement is recorded with its signal, gap, PID, the
`/api/ready` status just before the stop, and its stopped, restarted and
healthy times. The record goes to
`<wave>/repeated-restarts/<faultId>.json` (also written when the
sequence fails part way), to the receipt (`repeatedRestart` in
`faults.json` and in each failure's `startup-phases.json`), and to the
timeline (`repeated_restart_stop`, `repeated_restart_healthy`).
- Small registry additions: `types.ts`, `config.ts`, `scheduler.ts`,
`journey-matrix.ts`, `main.ts`, `evidence.ts`, `supervisor.ts`, and the
README section next to TM.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
33 files, 402 tests pass. The new `repeated-restarts.test.ts` covers
option parsing, seeded planning for each kind, unchanged plans for other
faults, steps in the task schedule, the matrix and replay, the
stop-and-replace loop on a fake clock (gaps measured from health,
`SIGTERM` for RR2, an exit during a gap, shutdown), and one crossing
window for the sequence.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- `pnpm verify` asks for broader coverage (the README is a global CI
input), so required CI supplies it.
- Live waves run from master after merge, as the merge rule for
test-only harness PRs allows. Their results go to a follow-up comment.

## Risks

- Harness-only change; no product code changes. Waves without RR faults
are unchanged: the plan draws, the fault receipt and the grader paths
are the same.
- During an RR sequence the server stays "not ready" for the harness, so
no other fault fires and the observer does not click Retry until the
last replacement is ready. This is the same rule as for K6.
- A replacement that exits on its own during a gap fails the fault as a
harness error, with the stops so far recorded. The wave then recovers
the server as after any failed fault.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (shell, file edits) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>

tests/restart-chaos/README.md

@@ -759,6 +759,48 @@ when the board decides after a day, and the approved subtask must exist exactly
 - Times kept outside Postgres are not aged: in-memory timers and the runner's own state files.
   Neither are times inside `text` columns or HMACs computed over a time.
 
+## Repeated restarts (RR1–RR3, `--rr-gap-seconds`)
+
+```sh
+# Each kind across running tasks.
+node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \
+  --adapters codex --workloads A1,B5,C1 --faults RR1,RR2,RR3 --concurrency 3 --tasks 6 --seed 401
+# A second kill while an approval is pending.
+node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \
+  --adapters codex --workloads G1 --faults RR1 --triggers approval --concurrency 2 --tasks 2 --seed 402
+```
+
+Every other fault restarts the server once. Real deploy loops restart it again 5–60 s after it comes
+back, while it is still recovering the runs the first restart interrupted. An RR fault starts as a K1
+(`SIGKILL` the server tree, start a replacement at once). After each planned gap, measured from the
+moment the replacement reports healthy, it stops that replacement and starts another one.
+
+| Fault | Restarts | Default gap after health |
+| --- | --- | --- |
+| RR1 double crash | K1, then one more `SIGKILL` | 3–20 s |
+| RR2 crash, then graceful | K1, then one `SIGTERM` (as K2) | 3–20 s |
+| RR3 crash loop | 3–4 `SIGKILL`s in all; the last replacement stays up | 10–30 s |
+
+The gaps are seeded per task (`restartSteps` in `plan.json`), and drawn only for RR tasks, so the plans
+of existing seeds do not change. `--rr-gap-seconds <n>` or `<min>-<max>` (1–300) replaces every RR
+gap: for example `--rr-gap-seconds 5` for a stop that always lands in recovery, or `30-60` for a slow
+deploy loop. RR faults do not combine with `--storm-minutes`, whose faults carry no planned steps.
+
+**Evidence.** Each stop and replacement (the first kill included) is recorded with its signal, planned
+gap, stopped PID, the `/api/ready` status just before the stop (`recovering` means it landed inside
+recovery), and the stopped, restarted and healthy times:
+- in `<wave>/repeated-restarts/<faultId>.json`, also written when the sequence fails part way (the planned steps are recorded before the first kill, so a replay of a failed sequence keeps its gaps);
+- in the fault receipt (`repeatedRestart` in `faults.json`, and in each failure's `startup-phases.json`);
+- in the timeline (`repeated_restart_stop`, `repeated_restart_healthy`).
+
+**Grading.** The whole sequence is one fault receipt, so it is one fault window: from the first kill to
+the last replacement's ready. Crossing, R1's outage allowance, R5's ready clock, the T-oracle, task
+deadlines and the restart-recovery speed budget treat it as one fault. No check is relaxed: the task
+must reach its correct terminal state with no Retry (R3) and no duplicated work (R4). The observer does
+not click Retry and no other fault fires until the last replacement is ready. A `--replay` of an RR
+task restarts again after the same gaps. A replacement that exits on its own during a gap fails the
+fault as a harness error, with the stops so far recorded.
+
 ## Fault timing (`--triggers`)
 
 Each task samples one trigger with its cell. `time` (default) fires at the
@@ -787,7 +829,7 @@ re-arms the task's own triggered faults instead of using their offsets.
 
 | File | Role |
 | --- | --- |
-| `supervisor.ts` | Owns the server. Reuses `tests/runner-e2e` process code (config, ports, env isolation, wrapper-owned Postgres, process-tree stopper, `replaceController`). Faults: K1 SIGKILL + immediate restart, K2 SIGTERM + restart, K4 SIGKILL + 2–5 min hold, K6 crash loop. The controller runtime cache is kept (discarding it is K11). |
+| `supervisor.ts` | Owns the server. Reuses `tests/runner-e2e` process code (config, ports, env isolation, wrapper-owned Postgres, process-tree stopper, `replaceController`). Faults: K1 SIGKILL + immediate restart, K2 SIGTERM + restart, K4 SIGKILL + 2–5 min hold, K6 crash loop, RR1–RR3 repeated restarts. The controller runtime cache is kept (discarding it is K11). |
 | `seed.ts` | Chaos Lab company, encrypted provider secrets, native Codex / ACPX Claude / OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project (git repo: python CLI, one failing test, `bench.sh` ≈ 4 min) with isolated git-worktree workspaces. It first turns on the instance `enableIsolatedWorkspaces` setting; without it the server ignores the project policy and every repo task shares the primary checkout. |
 | `scheduler.ts` | Seeded plan: workload × adapter × fault cells and a random fire time per task. Saved to `results/<wave>/plan.json`. `--replay` reruns one task alone with the same choices and fires the recorded sequence of faults that crossed it (other tasks' faults included) at the same offsets from launch. |
 | `observer.ts` | Polls each issue and its runs. When a task rests needing Retry it calls `POST /api/issues/:id/retry` once (once more only if a later fault started after that click), with the body `useTaskRetry.ts` sends. Collects T-oracle artifacts and the R6 sweep. |
@@ -803,6 +845,7 @@ re-arms the task's own triggered faults instead of using their offsets.
 | `acpx-provider-faults.ts`, `provider-shim/acpx-*` | PV1–PV3 for Claude (ACPX): the stand-in runner package, the server hook, the sidecar/provider tap, and `--provider-fault-restart`. |
 | `question-handoff.ts` | Q1 saved question: the board's answer step and its oracle. |
 | `time-faults.ts`, `time-faults-config.ts` | TM1–TM3: process freezing, database aging with recorded rows, and the time-corrected API the grader reads through. |
+| `repeated-restarts.ts` | RR1–RR3: seeded restart steps, `--rr-gap-seconds`, and the stop-and-replace loop the supervisor runs. |
 | `daytona-sweep.ts` | Daytona R6: read-only reconciliation of this wave's labelled sandboxes with leases; cleanup only through Paperclip. |
 
 ## Output (gitignored)

tests/restart-chaos/config.ts

@@ -5,6 +5,7 @@ import { applyJourneyOptions } from "./journey-matrix.js";
 import { takesJourneyTriggers } from "./journeys.js";
 import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js";
 import { parseTimeFaultOptions } from "./time-faults-config.js";
+import { parseRepeatedRestartOptions } from "./repeated-restarts.js";
 import { PROVIDER_FAULT_KINDS, PROVIDER_FAULT_WORKLOADS, isProviderFault, parseProviderDelays, parseProviderMessage, type ProviderFaultKind } from "./provider-faults.js";
 import { PROVIDER_FAULT_ADAPTERS, parseProviderFaultRestart } from "./acpx-provider-faults.js";
 import {
@@ -116,13 +117,14 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
       "time-age-hours": { type: "string" },
       "provider-fault-restart": { type: "string" },
       "provider-fault-restart-after-seconds": { type: "string" },
+      "rr-gap-seconds": { type: "string" },
     },
   });
   const keep = values.keep === true;
   const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000);
   const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000);
   if (values.replay !== undefined) {
-    for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds"] as const) {
+    for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds"] as const) {
       if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`);
     }
     return { mode: "replay", ref: parseReplayRef(values.replay), keep,
@@ -220,6 +222,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
     throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults");
   }
   Object.assign(options, parseTimeFaultOptions(options, values["time-age-hours"]));
+  Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"]));
   const upgradeFrom = values["upgrade-from"];
   if (options.faults.includes("K14") !== (upgradeFrom !== undefined)) {
     throw new Error("K14 (upgrade mid-run) and --upgrade-from <older checkout> go together: pass both or neither");

tests/restart-chaos/evidence.ts

@@ -83,7 +83,8 @@ export async function writeEvidenceBundle(input: {
   await write("task-record.json", record);
   await write("faults.json", relevant.length ? relevant : input.faults);
   await write("startup-phases.json", (relevant.length ? relevant : input.faults).map(fault => ({ faultId: fault.faultId, kind: fault.kind,
-    startedAt: fault.startedAt, stoppedAt: fault.stoppedAt, restartedAt: fault.restartedAt, healthyAt: fault.healthyAt, readyAt: fault.readyAt, phases: fault.phases })));
+    startedAt: fault.startedAt, stoppedAt: fault.stoppedAt, restartedAt: fault.restartedAt, healthyAt: fault.healthyAt, readyAt: fault.readyAt, phases: fault.phases,
+    ...(fault.repeatedRestart ? { repeatedRestarts: fault.repeatedRestart.restarts } : {}) })));
   await write("timeline.json", input.timeline);
   if (input.artifacts) await write("oracle-artifacts.json", input.artifacts);
   await attempt("server-log", async () => write("server-log-window.txt", await logWindow(input.serverLogPath, relevant.length ? relevant : input.faults)));

tests/restart-chaos/journey-matrix.ts

@@ -14,6 +14,7 @@ import { WORKLOAD_TRIGGERS } from "./journeys.js";
 import { withProviderFaultRules } from "./provider-faults.js";
 import { createRng, k4HoldRange, TRIGGER_DELAY_RANGE_MS } from "./scheduler.js";
 import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js";
+import { isRepeatedRestartFault, planRestartSteps } from "./repeated-restarts.js";
 import { workloadDefinitions } from "./workloads.js";
 import type { AdapterId, TaskFaultKind, TaskPlan, TriggerId, WaveOptions, WavePlan, WorkloadId } from "./types.js";
 
@@ -41,7 +42,7 @@ export function matrixMayLaunch(next: Pick<TaskPlan, "fault">, inFlight: Readonl
   return inFlight.every(task => (task.fault === "K0") === baseline);
 }
 
-export function planMatrixTasks(options: Pick<WaveOptions, "seed" | "workloads" | "adapters" | "faults" | "providerFaults" | "k4HoldMaxMs" | "faultAfterMs">): TaskPlan[] {
+export function planMatrixTasks(options: Pick<WaveOptions, "seed" | "workloads" | "adapters" | "faults" | "providerFaults" | "k4HoldMaxMs" | "faultAfterMs" | "rrGapMs">): TaskPlan[] {
   const rng = createRng((options.seed ^ 0x6a6d7478) >>> 0);
   return matrixCells(options).map((cell, index) => {
     const definition = workloadDefinitions[cell.workload];
@@ -57,6 +58,7 @@ export function planMatrixTasks(options: Pick<WaveOptions, "seed" | "workloads"
       // `--fault-after-seconds` pins time-fired restarts, as it does outside the matrix.
       fireAfterMs: cell.fault === "K0" || cell.fault.startsWith("PV") ? 0 : cell.trigger === "time" ? options.faultAfterMs ?? drawn : drawn,
       holdMs: cell.fault === "K4" ? rng.int(...k4HoldRange(options.k4HoldMaxMs)) : isTimeFault(cell.fault) ? timeFaultHoldMs(cell.fault, rng.int) : 0,
+      ...(isRepeatedRestartFault(cell.fault) ? { restartSteps: planRestartSteps(cell.fault, rng.int, options.rrGapMs) } : {}),
       followUpAfterMs: definition.followUpWindowMs ? rng.int(definition.followUpWindowMs[0], definition.followUpWindowMs[1]) : null,
       nonce: rng.hex(8),
     };

tests/restart-chaos/main.ts

@@ -39,6 +39,7 @@ import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurr
 import { chaosLifecycleOverrides, seedChaosLab } from "./seed.js";
 import { ChaosSupervisor } from "./supervisor.js";
 import { TimeShiftedApi } from "./time-faults.js";
+import type { RestartStep } from "./repeated-restarts.js";
 import { isBrokenStdoutError, markTimelineEchoBroken, Timeline } from "./timeline.js";
 import { guardedRunnerOutputs, worktreeFingerprint } from "./wave-guard.js";
 import type { FaultKind, FaultReceipt, OracleArtifacts, TriggerId, WavePlan } from "./types.js";
@@ -355,9 +356,9 @@ async function main(): Promise<number> {
     if (storm) timeline.add("storm_started", { durationMs: storm.plan.durationMs, plannedFaults: storm.plan.faults.length });
     // Faults are serialized; each one hits every task in flight when it starts.
     const fireFault = (kind: FaultKind, holdMs: number, requestedBy: string, exposed: readonly TaskObserver[],
-      trigger?: { trigger: TriggerId; delayMs: number }, onSettled?: () => void) => {
+      trigger?: { trigger: TriggerId; delayMs: number }, onSettled?: () => void, restartSteps?: RestartStep[]) => {
       faultsQueued += 1;
-      faultWork.push(supervisor.fault(kind, holdMs, requestedBy, trigger).finally(() => { faultsQueued -= 1; onSettled?.(); }).then(receipt => {
+      faultWork.push(supervisor.fault(kind, holdMs, requestedBy, trigger, restartSteps).finally(() => { faultsQueued -= 1; onSettled?.(); }).then(receipt => {
         faults.push(receipt);
         checkWorktree(receipt.faultId);
         for (const candidate of exposed) {
@@ -498,7 +499,7 @@ async function main(): Promise<number> {
         const trigger = due.trigger && due.trigger !== "time" ? { trigger: due.trigger, delayMs: due.delayMs ?? 0 } : undefined;
         if (trigger) timeline.add("trigger_fired", { taskId: observer.plan.taskId, fault: due.kind, ...trigger });
         // Inside a storm a triggered fault is a restart too: the next storm gap runs from its settle.
-        fireFault(due.kind, due.holdMs, observer.plan.taskId, exposed, trigger, storm ? () => storm.settled(Date.now()) : undefined);
+        fireFault(due.kind, due.holdMs, observer.plan.taskId, exposed, trigger, storm ? () => storm.settled(Date.now()) : undefined, due.restartSteps);
       }
       if (storm && !faultsQueued && !supervisor.faultInProgress && supervisor.serverReady) {
         const due = storm.due(Date.now());

tests/restart-chaos/repeated-restarts.test.ts

@@ -0,0 +1,233 @@
+import { describe, expect, it } from "vitest";
+import { parseCommand } from "./config.js";
+import { crossingFaults } from "./grader.js";
+import { planMatrixWave } from "./journey-matrix.js";
+import { createRng, observedFaultSchedule, planTasks, taskFaultSchedule } from "./scheduler.js";
+import {
+  RR3_KILL_RANGE, RR_GAP_RANGE_MS, newRepeatedRestartRecord, planRestartSteps, recordFirstRestart, runRepeatedRestarts, type RepeatedRestartRecord, type RestartStep,
+} from "./repeated-restarts.js";
+import type { FaultReceipt, TaskRecord } from "./types.js";
+
+describe("repeated restart options", () => {
+  it("accepts RR faults with each kind's default gaps", () => {
+    const parsed = parseCommand(["--faults", "RR1,RR2,RR3", "--adapters", "codex", "--workloads", "A1", "--seed", "1"]);
+    if (parsed.mode !== "wave") throw new Error("expected a wave");
+    expect(parsed.options.faults).toEqual(["RR1", "RR2", "RR3"]);
+    expect(parsed.options.rrGapMs).toBeUndefined();
+  });
+
+  it("takes --rr-gap-seconds as one value or a range, only with an RR fault", () => {
+    const one = parseCommand(["--faults", "RR1", "--rr-gap-seconds", "5", "--workloads", "A1", "--seed", "1"]);
+    expect(one.mode === "wave" && one.options.rrGapMs).toEqual([5_000, 5_000]);
+    const range = parseCommand(["--faults", "RR3", "--rr-gap-seconds", "10-40", "--workloads", "A1", "--seed", "1"]);
+    expect(range.mode === "wave" && range.options.rrGapMs).toEqual([10_000, 40_000]);
+    expect(() => parseCommand(["--faults", "K1", "--rr-gap-seconds", "5", "--workloads", "A1"])).toThrow(/add RR1, RR2 or RR3/);
+    expect(() => parseCommand(["--faults", "RR1", "--rr-gap-seconds", "0", "--workloads", "A1"])).toThrow(/between 1 and 300/);
+    expect(() => parseCommand(["--faults", "RR1", "--rr-gap-seconds", "20-10", "--workloads", "A1"])).toThrow(/min not above max/);
+    expect(() => parseCommand(["--faults", "RR1", "--rr-gap-seconds", "5s", "--workloads", "A1"])).toThrow(/<seconds> or <min>-<max>/);
+  });
+
+  it("refuses a storm, whose faults carry no planned restarts", () => {
+    expect(() => parseCommand(["--faults", "RR1", "--storm-minutes", "10"])).toThrow(/storm/);
+  });
+
+  it("refuses --rr-gap-seconds with --replay", () => {
+    expect(() => parseCommand(["--replay", "w-20261110T000000-s1/t01", "--rr-gap-seconds", "5"])).toThrow(/--rr-gap-seconds/);
+  });
+});
+
+describe("repeated restart planning", () => {
+  const within = (value: number, [min, max]: readonly [number, number]) => value >= min && value <= max;
+
+  it("plans one SIGKILL (RR1), one SIGTERM (RR2) or 2-3 more SIGKILLs (RR3) after the first kill", () => {
+    for (let seed = 0; seed < 50; seed += 1) {
+      const rng = createRng(seed);
+      const rr1 = planRestartSteps("RR1", rng.int);
+      expect(rr1).toHaveLength(1);
+      expect(rr1[0]!.signal).toBe("SIGKILL");
+      expect(within(rr1[0]!.gapMs, RR_GAP_RANGE_MS.RR1)).toBe(true);
+      const rr2 = planRestartSteps("RR2", rng.int);
+      expect(rr2.map(step => step.signal)).toEqual(["SIGTERM"]);
+      expect(within(rr2[0]!.gapMs, RR_GAP_RANGE_MS.RR2)).toBe(true);
+      const rr3 = planRestartSteps("RR3", rng.int);
+      expect(within(rr3.length + 1, RR3_KILL_RANGE)).toBe(true);
+      expect(rr3.every(step => step.signal === "SIGKILL" && within(step.gapMs, RR_GAP_RANGE_MS.RR3))).toBe(true);
+    }
+  });
+
+  it("draws both RR3 lengths over seeds", () => {
+    const lengths = new Set(Array.from({ length: 40 }, (_, seed) => planRestartSteps("RR3", createRng(seed).int).length));
+    expect([...lengths].sort()).toEqual([2, 3]);
+  });
+
+  it("uses the --rr-gap-seconds range for every kind", () => {
+    const steps = planRestartSteps("RR3", createRng(3).int, [7_000, 7_000]);
+    expect(steps.every(step => step.gapMs === 7_000)).toBe(true);
+  });
+
+  it("gives RR tasks seeded restart steps and leaves the plans of other faults unchanged", () => {
+    const options = { seed: 11, tasks: 9, workloads: ["A1" as const], adapters: ["codex" as const] };
+    const plain = planTasks({ ...options, faults: ["K1", "K2"] });
+    expect(plain.every(task => task.restartSteps === undefined)).toBe(true);
+    expect(planTasks({ ...options, faults: ["K1", "K2"], rrGapMs: [5_000, 5_000] })).toEqual(plain);
+    const repeated = planTasks({ ...options, faults: ["RR1", "RR2", "RR3"] });
+    expect(repeated.every(task => (task.restartSteps?.length ?? 0) >= 1 && task.holdMs === 0)).toBe(true);
+    expect(planTasks({ ...options, faults: ["RR1", "RR2", "RR3"] })).toEqual(repeated);
+  });
+
+  it("carries the steps into the task's fault schedule and the matrix plan", () => {
+    const [task] = planTasks({ seed: 5, tasks: 1, workloads: ["A1"], adapters: ["codex"], faults: ["RR3"] });
+    expect(taskFaultSchedule(task!)).toEqual([{ kind: "RR3", holdMs: 0, atMs: task!.fireAfterMs, restartSteps: task!.restartSteps }]);
+    const parsed = parseCommand(["--matrix", "--workloads", "Q2", "--adapters", "codex", "--faults", "RR1", "--seed", "4"]);
+    if (parsed.mode !== "wave") throw new Error("expected a wave");
+    const matrix = planMatrixWave("w-matrix", parsed.options);
+    const faulted = matrix.tasks.filter(cell => cell.fault === "RR1");
+    expect(faulted.length).toBeGreaterThan(0);
+    expect(faulted.every(cell => cell.restartSteps?.length === 1)).toBe(true);
+  });
+
+  it("replays the recorded steps of a fault that crossed the task", () => {
+    const planned: RestartStep[] = [{ signal: "SIGKILL", gapMs: 12_000 }, { signal: "SIGKILL", gapMs: 25_000 }];
+    const schedule = observedFaultSchedule(1_000, ["f01"], [{ faultId: "f01", kind: "RR3", holdMs: 0, startedAt: 31_000,
+      repeatedRestart: { planned } }]);
+    expect(schedule).toEqual([{ kind: "RR3", holdMs: 0, atMs: 30_000, restartSteps: planned }]);
+  });
+});
+
+/** A fake server tree on a fake clock: each start takes `bootMs` to be healthy. */
+function fakeHooks(options: { bootMs?: number; stopMs?: number; exitDuringGap?: number; cancelAfterStops?: number } = {}) {
+  let clock = 100_000;
+  let nextPid = 200;
+  const log: string[] = [];
+  const exited = new Set<number>();
+  let stops = 0;
+  const hooks = {
+    now: () => clock,
+    sleep: async (ms: number) => { log.push(`sleep ${ms}`); clock += ms; },
+    cancelled: () => options.cancelAfterStops !== undefined && stops >= options.cancelAfterStops,
+    pidOf: (server: { pid: number }) => server.pid,
+    exitStatus: (server: { pid: number }) => exited.has(server.pid) ? { code: 1, signal: null } : null,
+    readPhase: async () => "recovering",
+    stop: async (server: { pid: number }, signal: "SIGKILL" | "SIGTERM") => {
+      stops += 1;
+      log.push(`stop ${server.pid} ${signal}`);
+      clock += options.stopMs ?? 500;
+      exited.add(server.pid);
+    },
+    start: async () => {
+      const server = { pid: nextPid++ };
+      log.push(`start ${server.pid}`);
+      if (options.exitDuringGap === server.pid) exited.add(server.pid);
+      return server;
+    },
+    waitHealthy: async () => { clock += options.bootMs ?? 8_000; return clock; },
+  };
+  return { hooks, log, now: () => clock };
+}
+
+function record(kind: "RR1" | "RR2" | "RR3", planned: RestartStep[]): RepeatedRestartRecord {
+  return { kind, planned, restarts: [] };
+}
+
+describe("the record built before the first kill", () => {
+  const planned: RestartStep[] = [{ signal: "SIGKILL", gapMs: 12_000 }, { signal: "SIGKILL", gapMs: 25_000 }];
+  const build = (steps: RestartStep[] | undefined) => newRepeatedRestartRecord({ kind: "RR3", planned: steps, firstPid: 41, startedAt: 5_000,
+    evidencePath: "/wave/repeated-restarts/f01.json", int: createRng(9).int });
+
+  it("keeps the planned steps when the first replacement never becomes healthy", () => {
+    const rr = build(planned);
+    // The first restart failed: the server stopped and a replacement was spawned, but it never reported healthy.
+    recordFirstRestart(rr, { stoppedAt: 5_400, restartedAt: 5_400, healthyAt: null }, null);
+    expect(rr.planned).toEqual(planned);
+    expect(rr.planned).not.toBe(planned);
+    expect(rr.restarts).toEqual([{ index: 0, signal: "SIGKILL", gapMs: null, pid: 41, phaseAtStop: "ready", stopAt: 5_000,
+      stoppedAt: 5_400, restartedAt: 5_400, replacementPid: null, healthyAt: null }]);
+    // A replay of the failed fault fires the same gaps.
+    expect(observedFaultSchedule(1_000, ["f01"], [{ faultId: "f01", kind: "RR3", holdMs: 0, startedAt: 5_000, repeatedRestart: rr }])[0]!.restartSteps)
+      .toEqual(planned);
+  });
+
+  it("draws a sequence of its kind when no steps were planned", () => {
+    expect(build(undefined).planned).toEqual(planRestartSteps("RR3", createRng(9).int));
+  });
+
+  it("does not overwrite first-kill times already recorded", () => {
+    const rr = build(planned);
+    recordFirstRestart(rr, { stoppedAt: 5_400, restartedAt: 5_400, healthyAt: 13_000 }, 77);
+    recordFirstRestart(rr, { stoppedAt: 9_999, restartedAt: 9_999, healthyAt: 99_999 }, null);
+    expect(rr.restarts[0]).toMatchObject({ stoppedAt: 5_400, restartedAt: 5_400, healthyAt: 13_000, replacementPid: 77 });
+  });
+});
+
+describe("runRepeatedRestarts", () => {
+  it("stops each replacement its gap after it reported healthy, then starts the next", async () => {
+    const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 10_000 }, { signal: "SIGKILL", gapMs: 20_000 }, { signal: "SIGKILL", gapMs: 15_000 }];
+    const fake = fakeHooks({ bootMs: 6_000, stopMs: 400 });
+    const rr = record("RR3", steps);
+    const result = await runRepeatedRestarts({ record: rr, steps, server: { pid: 100 }, healthyAt: fake.now(), hooks: fake.hooks });
+    expect(fake.log).toEqual([
+      "sleep 10000", "stop 100 SIGKILL", "start 200",
+      "sleep 20000", "stop 200 SIGKILL", "start 201",
+      "sleep 15000", "stop 201 SIGKILL", "start 202",
+    ]);
+    expect(result.server).toEqual({ pid: 202 });
+    expect(rr.restarts.map(entry => [entry.index, entry.pid, entry.replacementPid, entry.phaseAtStop])).toEqual([
+      [0, 100, 200, "recovering"], [1, 200, 201, "recovering"], [2, 201, 202, "recovering"]]);
+    for (const [index, entry] of rr.restarts.entries()) {
+      const previousHealthy = index === 0 ? 100_000 : rr.restarts[index - 1]!.healthyAt!;
+      expect(entry.stopAt - previousHealthy).toBe(steps[index]!.gapMs);
+      expect(entry.stoppedAt! - entry.stopAt).toBe(400);
+      expect(entry.healthyAt! - entry.restartedAt!).toBe(6_000);
+    }
+    expect(result.healthyAt).toBe(rr.restarts[2]!.healthyAt);
+  });
+
+  it("sends SIGTERM for a graceful step (RR2)", async () => {
+    const steps: RestartStep[] = [{ signal: "SIGTERM", gapMs: 3_000 }];
+    const fake = fakeHooks();
+    const rr = record("RR2", steps);
+    await runRepeatedRestarts({ record: rr, steps, server: { pid: 100 }, healthyAt: fake.now(), hooks: fake.hooks });
+    expect(fake.log).toEqual(["sleep 3000", "stop 100 SIGTERM", "start 200"]);
+    expect(rr.restarts[0]!.signal).toBe("SIGTERM");
+  });
+
+  it("does not wait when the gap already passed (the health check returned late)", async () => {
+    const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 3_000 }];
+    const fake = fakeHooks();
+    await runRepeatedRestarts({ record: record("RR1", steps), steps, server: { pid: 100 }, healthyAt: fake.now() - 5_000, hooks: fake.hooks });
+    expect(fake.log).toEqual(["stop 100 SIGKILL", "start 200"]);
+  });
+
+  it("fails, with what fired recorded, when a replacement exits on its own during a gap", async () => {
+    const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 10_000 }, { signal: "SIGKILL", gapMs: 10_000 }];
+    const fake = fakeHooks({ exitDuringGap: 200 });
+    const rr = record("RR3", steps);
+    await expect(runRepeatedRestarts({ record: rr, steps, server: { pid: 100 }, healthyAt: fake.now(), hooks: fake.hooks }))
+      .rejects.toThrow(/exited on its own before repeated restart 1/);
+    expect(rr.restarts).toHaveLength(1);
+    expect(fake.log).not.toContain("stop 200 SIGKILL");
+  });
+
+  it("stops at harness shutdown without starting another replacement", async () => {
+    const steps: RestartStep[] = [{ signal: "SIGKILL", gapMs: 1_000 }, { signal: "SIGKILL", gapMs: 1_000 }];
+    const fake = fakeHooks({ cancelAfterStops: 1 });
+    await expect(runRepeatedRestarts({ record: record("RR3", steps), steps, server: { pid: 100 }, healthyAt: fake.now(), hooks: fake.hooks }))
+      .rejects.toThrow(/shutdown/);
+    expect(fake.log).toEqual(["sleep 1000", "stop 100 SIGKILL"]);
+  });
+});
+
+describe("grading a repeated restart as one fault", () => {
+  it("crosses a task with one fault window from the first kill to the last replacement's ready", () => {
+    const receipt = {
+      faultId: "f01", kind: "RR3", requestedBy: "t01", signal: "SIGKILL", holdMs: 0, requestedAt: 10_000, startedAt: 10_000,
+      stoppedAt: 10_500, restartedAt: 10_500, healthyAt: 90_000, readyAt: 95_000, phases: [], serverLogOffsetAtStart: 0,
+      serverLogOffsetAtReady: 0, error: null,
+    } satisfies FaultReceipt;
+    // A run that started between the first and the last kill overlaps the one window.
+    const task = { plan: { taskId: "t02" }, createdAt: 50_000, restAt: 200_000, samples: [{ at: 60_000, reachable: true, serverReady: false,
+      runs: [{ id: "r1", status: "succeeded", createdAt: new Date(50_000).toISOString(), startedAt: null,
+        finishedAt: new Date(70_000).toISOString(), errorCode: null, retryOfRunId: null, wakeCommentIds: [], livenessState: null }] }] } as unknown as TaskRecord;
+    expect(crossingFaults(task, [receipt])).toEqual(["f01"]);
+  });
+});

tests/restart-chaos/repeated-restarts.ts

@@ -0,0 +1,172 @@
+/**
+ * Repeated restarts (RR1–RR3): a restart that lands while the server is still recovering from the
+ * previous one, as in a deploy loop. Each RR fault starts as a K1 (SIGKILL the server tree, start a
+ * replacement at once). Then, after each planned gap from the moment the replacement reports
+ * healthy, it stops the replacement again and starts another one:
+ *
+ * - RR1 double crash: one more SIGKILL.
+ * - RR2 crash, then graceful: one SIGTERM (as K2).
+ * - RR3 crash loop: 3–4 SIGKILLs in all, then the last replacement stays up.
+ *
+ * The whole sequence is one fault receipt, so the grader sees one fault window from the first kill
+ * to the last replacement's ready. Planning, option parsing and the step loop live here with no
+ * process code, so they are unit-tested; the supervisor supplies the process hooks.
+ */
+import type { WaveOptions } from "./types.js";
+
+export const REPEATED_RESTART_KINDS = ["RR1", "RR2", "RR3"] as const;
+export type RepeatedRestartKind = (typeof REPEATED_RESTART_KINDS)[number];
+
+export function isRepeatedRestartFault(kind: string): kind is RepeatedRestartKind {
+  return (REPEATED_RESTART_KINDS as readonly string[]).includes(kind);
+}
+
+/** One restart after the first kill: wait `gapMs` from the previous replacement's health, then stop it with `signal`. */
+export interface RestartStep { signal: "SIGKILL" | "SIGTERM"; gapMs: number }
+
+/** Default gaps, from health to the next stop: inside the recovery window (RR1, RR2), or a loop of deploys (RR3). */
+export const RR_GAP_RANGE_MS: Record<RepeatedRestartKind, readonly [number, number]> = {
+  RR1: [3_000, 20_000],
+  RR2: [3_000, 20_000],
+  RR3: [10_000, 30_000],
+};
+
+/** RR3: how many kills in all, the first one included. */
+export const RR3_KILL_RANGE: readonly [number, number] = [3, 4];
+
+/** The seeded restarts that follow the first kill. `gapRangeMs` (`--rr-gap-seconds`) replaces the default gaps. */
+export function planRestartSteps(kind: RepeatedRestartKind, int: (min: number, max: number) => number,
+  gapRangeMs?: readonly [number, number]): RestartStep[] {
+  const range = gapRangeMs ?? RR_GAP_RANGE_MS[kind];
+  if (kind === "RR1") return [{ signal: "SIGKILL", gapMs: int(...range) }];
+  if (kind === "RR2") return [{ signal: "SIGTERM", gapMs: int(...range) }];
+  const kills = int(...RR3_KILL_RANGE);
+  const steps: RestartStep[] = [];
+  for (let index = 1; index < kills; index += 1) steps.push({ signal: "SIGKILL", gapMs: int(...range) });
+  return steps;
+}
+
+const MIN_GAP_SECONDS = 1;
+const MAX_GAP_SECONDS = 300;
+
+/** `--rr-gap-seconds <n>` or `<min>-<max>` (1–300 s): the gap from a replacement's health to the next stop. */
+export function parseRepeatedRestartOptions(options: Pick<WaveOptions, "faults" | "stormMinutes">,
+  rawGapSeconds: string | undefined): Pick<WaveOptions, "rrGapMs"> {
+  const repeated = options.faults.some(isRepeatedRestartFault);
+  if (rawGapSeconds !== undefined && !repeated) throw new Error("--rr-gap-seconds sets the RR restart gap; add RR1, RR2 or RR3 to --faults");
+  if (!repeated) return {};
+  // A storm fault carries no planned restart steps, and a storm already restarts again and again.
+  if (options.stormMinutes) throw new Error("Repeated restarts (RR*) are planned per task; do not combine them with --storm-minutes");
+  if (rawGapSeconds === undefined) return {};
+  const match = /^([0-9]+)(?:-([0-9]+))?$/.exec(rawGapSeconds.trim());
+  if (!match) throw new Error("--rr-gap-seconds takes <seconds> or <min>-<max> seconds");
+  const min = Number(match[1]);
+  const max = match[2] === undefined ? min : Number(match[2]);
+  if (min < MIN_GAP_SECONDS || max > MAX_GAP_SECONDS || max < min) {
+    throw new Error(`--rr-gap-seconds must be between ${MIN_GAP_SECONDS} and ${MAX_GAP_SECONDS} seconds, min not above max`);
+  }
+  return { rrGapMs: [min * 1000, max * 1000] };
+}
+
+/** One stop and replacement in the sequence; index 0 is the first (K1) kill. */
+export interface RepeatedRestartEntry {
+  index: number;
+  signal: "SIGKILL" | "SIGTERM";
+  /** Planned wait from the previous replacement's health; null for the first kill. */
+  gapMs: number | null;
+  pid: number | null;
+  /** `/api/ready` status just before the stop: `recovering` means the stop landed inside recovery. */
+  phaseAtStop: string | null;
+  stopAt: number;
+  stoppedAt: number | null;
+  restartedAt: number | null;
+  replacementPid: number | null;
+  healthyAt: number | null;
+}
+
+export interface RepeatedRestartRecord {
+  kind: RepeatedRestartKind;
+  planned: RestartStep[];
+  restarts: RepeatedRestartEntry[];
+  /** `<wave>/repeated-restarts/<faultId>.json`. */
+  evidencePath?: string;
+}
+
+/**
+ * The record of an RR fault, built before its first kill: the planned steps and the first (K1) kill. A
+ * first restart that fails still leaves them in the receipt, so a replay reuses the same gaps.
+ * Absent planned steps (a fault fired outside a planned task) are drawn with `int`.
+ */
+export function newRepeatedRestartRecord(input: { kind: RepeatedRestartKind; planned: readonly RestartStep[] | undefined;
+  firstPid: number | null; startedAt: number; evidencePath: string; int: (min: number, max: number) => number }): RepeatedRestartRecord {
+  const steps = input.planned?.length ? input.planned : planRestartSteps(input.kind, input.int);
+  return { kind: input.kind, planned: steps.map(step => ({ ...step })), restarts: [{ index: 0, signal: "SIGKILL", gapMs: null,
+    pid: input.firstPid, phaseAtStop: "ready", stopAt: input.startedAt, stoppedAt: null, restartedAt: null, replacementPid: null, healthyAt: null }],
+    evidencePath: input.evidencePath };
+}
+
+/** Fills the first kill's times from the receipt, whether its replacement became healthy or the fault failed. */
+export function recordFirstRestart(record: RepeatedRestartRecord, receipt: { stoppedAt: number | null; restartedAt: number | null;
+  healthyAt: number | null }, replacementPid: number | null): void {
+  const first = record.restarts[0]!;
+  first.stoppedAt ??= receipt.stoppedAt;
+  first.restartedAt ??= receipt.restartedAt;
+  first.healthyAt ??= receipt.healthyAt;
+  first.replacementPid ??= replacementPid;
+}
+
+export interface RepeatedRestartHooks<P> {
+  now(): number;
+  sleep(ms: number): Promise<void>;
+  cancelled(): boolean;
+  pidOf(server: P): number | null;
+  /** The server's exit status once it has exited on its own, else null. */
+  exitStatus(server: P): { code: number | null; signal: string | null } | null;
+  readPhase(): Promise<string>;
+  /** Stop the server with `signal` and wait until it no longer answers health checks. */
+  stop(server: P, signal: "SIGKILL" | "SIGTERM"): Promise<void>;
+  start(): Promise<P>;
+  /** Resolves with the time the replacement reported healthy. */
+  waitHealthy(server: P): Promise<number>;
+  onEntry?(entry: RepeatedRestartEntry, stage: "stopping" | "healthy"): void;
+}
+
+/**
+ * Runs the planned restarts after the first kill's replacement reported healthy at `healthyAt`.
+ * Each gap runs from the previous replacement's health. Every entry is pushed to `record.restarts`
+ * before its stop, so a failure part way still leaves the evidence of what fired.
+ */
+export async function runRepeatedRestarts<P>(input: {
+  record: RepeatedRestartRecord;
+  steps: readonly RestartStep[];
+  server: P;
+  healthyAt: number;
+  hooks: RepeatedRestartHooks<P>;
+}): Promise<{ server: P; healthyAt: number }> {
+  const { record, steps, hooks } = input;
+  let server = input.server;
+  let healthyAt = input.healthyAt;
+  for (const step of steps) {
+    const index = record.restarts.length;
+    const wait = healthyAt + step.gapMs - hooks.now();
+    if (wait > 0) await hooks.sleep(wait);
+    if (hooks.cancelled()) throw new Error(`Harness shutdown interrupted repeated restart ${index}`);
+    const exited = hooks.exitStatus(server);
+    if (exited) throw new Error(`Paperclip server exited on its own before repeated restart ${index} (code=${exited.code} signal=${exited.signal})`);
+    const phaseAtStop = await hooks.readPhase();
+    const entry: RepeatedRestartEntry = { index, signal: step.signal, gapMs: step.gapMs, pid: hooks.pidOf(server), phaseAtStop,
+      stopAt: hooks.now(), stoppedAt: null, restartedAt: null, replacementPid: null, healthyAt: null };
+    record.restarts.push(entry);
+    hooks.onEntry?.(entry, "stopping");
+    await hooks.stop(server, step.signal);
+    entry.stoppedAt = hooks.now();
+    if (hooks.cancelled()) throw new Error(`Harness shutdown interrupted repeated restart ${index}`);
+    entry.restartedAt = hooks.now();
+    server = await hooks.start();
+    entry.replacementPid = hooks.pidOf(server);
+    healthyAt = await hooks.waitHealthy(server);
+    entry.healthyAt = healthyAt;
+    hooks.onEntry?.(entry, "healthy");
+  }
+  return { server, healthyAt };
+}

tests/restart-chaos/scheduler.ts

@@ -2,6 +2,7 @@ import { isProviderFault, withProviderFaultRules } from "./provider-faults.js";
 import { takesJourneyTriggers } from "./journeys.js";
 import { workloadDefinitions } from "./workloads.js";
 import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js";
+import { isRepeatedRestartFault, planRestartSteps, type RestartStep } from "./repeated-restarts.js";
 import type { AdapterId, FaultKind, ScheduledFault, StormFault, StormPlan, TaskFaultKind, TaskPlan, TriggerId, WavePlan, WaveOptions, WorkloadId } from "./types.js";
 import type { ReplayRef } from "./config.js";
 
@@ -93,7 +94,7 @@ function pinnedFireAfterMs(faultAfterMs: number | undefined, drawn: number): num
   return faultAfterMs ?? drawn;
 }
 
-export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloads" | "adapters" | "faults" | "triggers" | "stormMinutes" | "k4HoldMaxMs" | "faultAfterMs" | "providerFaults">): TaskPlan[] {
+export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloads" | "adapters" | "faults" | "triggers" | "stormMinutes" | "k4HoldMaxMs" | "faultAfterMs" | "providerFaults" | "rrGapMs">): TaskPlan[] {
   const rng = createRng(options.seed);
   const triggers = options.triggers?.length ? options.triggers : ["time" as const];
   for (const trigger of triggers) {
@@ -125,6 +126,8 @@ export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloa
       ...(trigger === "time" ? {} : { trigger, triggerDelayMs: rng.int(...TRIGGER_DELAY_RANGE_MS[trigger]) }),
       fireAfterMs: cell.fault === "K13" ? 0 : pinnedFireAfterMs(options.faultAfterMs, rng.int(fireMin, fireMax)),
       holdMs: cell.fault === "K4" ? rng.int(...k4HoldRange(options.k4HoldMaxMs)) : isTimeFault(cell.fault) ? timeFaultHoldMs(cell.fault, rng.int) : 0,
+      // Drawn only for RR tasks, so the plans of existing seeds do not change.
+      ...(isRepeatedRestartFault(cell.fault) ? { restartSteps: planRestartSteps(cell.fault, rng.int, options.rrGapMs) } : {}),
       followUpAfterMs: definition.followUpWindowMs ? rng.int(definition.followUpWindowMs[0], definition.followUpWindowMs[1]) : null,
       nonce: rng.hex(8),
     });
@@ -270,10 +273,12 @@ export function replayRecordFor(taskId: string, graded: { createdAt: number | nu
 /** Faults that actually crossed a task, as offsets from its launch. */
 export function observedFaultSchedule(createdAt: number | null, faultIds: readonly string[],
   faults: ReadonlyArray<{ faultId: string; kind: FaultKind; holdMs: number; startedAt: number; requestedBy?: string;
-    trigger?: TriggerId; triggerDelayMs?: number }>, taskId?: string): ScheduledFault[] {
+    trigger?: TriggerId; triggerDelayMs?: number; repeatedRestart?: { planned: RestartStep[] } }>, taskId?: string): ScheduledFault[] {
   if (createdAt === null) return [];
   return faults.filter(fault => faultIds.includes(fault.faultId))
     .map((fault): ScheduledFault => ({ kind: fault.kind, holdMs: fault.holdMs, atMs: Math.max(0, fault.startedAt - createdAt),
+      // RR: the replay restarts again after the same gaps.
+      ...(fault.repeatedRestart ? { restartSteps: fault.repeatedRestart.planned.map(step => ({ ...step })) } : {}),
       ...(fault.trigger && fault.trigger !== "time" && fault.requestedBy === taskId
         ? { trigger: fault.trigger, delayMs: fault.triggerDelayMs ?? 0 } : {}) }))
     .sort((a, b) => a.atMs - b.atMs);
@@ -287,7 +292,8 @@ export function taskFaultSchedule(task: TaskPlan): ScheduledFault[] {
   const restart = task.providerFault?.restart;
   if (restart) return [{ kind: restart.kind, holdMs: 0, atMs: 0, trigger: "provider", delayMs: restart.afterAppliedMs }];
   if (task.fault === "K13" || task.fault === "K0" || isProviderFault(task.fault)) return [];
-  const planned: ScheduledFault = { kind: task.fault, holdMs: task.holdMs, atMs: task.fireAfterMs };
+  const planned: ScheduledFault = { kind: task.fault, holdMs: task.holdMs, atMs: task.fireAfterMs,
+    ...(task.restartSteps ? { restartSteps: task.restartSteps } : {}) };
   const trigger = task.trigger ?? "time";
   if (trigger === "time") return [planned];
   const delayMs = task.triggerDelayMs ?? 0;

tests/restart-chaos/supervisor.ts

@@ -16,7 +16,7 @@
 import { execFileSync, spawn, type ChildProcess } from "node:child_process";
 import { randomBytes } from "node:crypto";
 import { createWriteStream, readFileSync, statSync, writeFileSync, type WriteStream } from "node:fs";
-import { mkdir, readFile, symlink } from "node:fs/promises";
+import { mkdir, readFile, symlink, writeFile } from "node:fs/promises";
 import { createRequire } from "node:module";
 import os from "node:os";
 import path from "node:path";
@@ -35,6 +35,8 @@ import { runnerE2ETypeScriptProcessArgs } from "../runner-e2e/web-server-command
 import type { MatrixExecution } from "../runner-e2e/types.js";
 import { DEFAULT_TIME_AGE_MS, ProcessFreezer, ageWaveDatabase, databaseUrlFromConfig, runFreezeFault, type AgingShift, type TimeFaultRecord } from "./time-faults.js";
 import type { Timeline } from "./timeline.js";
+import { isRepeatedRestartFault, newRepeatedRestartRecord, recordFirstRestart, runRepeatedRestarts, type RepeatedRestartKind,
+  type RepeatedRestartRecord, type RestartStep } from "./repeated-restarts.js";
 import type { FaultKind, FaultReceipt, TriggerId } from "./types.js";
 
 /** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */
@@ -46,7 +48,7 @@ const READY_TIMEOUT_MS = 300_000;
 const STOP_TIMEOUT_MS = 30_000;
 
 export const FAULT_SIGNAL: Record<FaultKind, "SIGKILL" | "SIGTERM" | "SIGSTOP"> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM",
-  TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL" };
+  TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL" };
 
 /** The checkout's commit, for the K14 receipt. */
 function checkoutSha(root: string): string | null {
@@ -400,15 +402,16 @@ export class ChaosSupervisor {
   }
 
   /** Faults are serialized: one control plane, one outage at a time. */
-  fault(kind: FaultKind, holdMs: number, requestedBy: string, trigger?: { trigger: TriggerId; delayMs: number }): Promise<FaultReceipt> {
+  fault(kind: FaultKind, holdMs: number, requestedBy: string, trigger?: { trigger: TriggerId; delayMs: number },
+    restartSteps?: readonly RestartStep[]): Promise<FaultReceipt> {
     const requestedAt = Date.now();
-    const run = this.faultChain.then(() => this.executeFault(kind, holdMs, requestedBy, requestedAt, trigger));
+    const run = this.faultChain.then(() => this.executeFault(kind, holdMs, requestedBy, requestedAt, trigger, restartSteps));
     this.faultChain = run.catch(() => undefined);
     return run;
   }
 
   private async executeFault(kind: FaultKind, holdMs: number, requestedBy: string, requestedAt: number,
-    trigger?: { trigger: TriggerId; delayMs: number }): Promise<FaultReceipt> {
+    trigger?: { trigger: TriggerId; delayMs: number }, restartSteps?: readonly RestartStep[]): Promise<FaultReceipt> {
     const previous = this.child;
     this.faultCount += 1;
     const receipt: FaultReceipt = {
@@ -418,6 +421,8 @@ export class ChaosSupervisor {
       requestedAt, startedAt: Date.now(), stoppedAt: null, restartedAt: null, healthyAt: null, readyAt: null,
       phases: [], serverLogOffsetAtStart: this.serverLogOffset(), serverLogOffsetAtReady: null, error: null,
     };
+    // RR: the planned steps and the first kill are recorded before it, so a failed first restart keeps them for replay.
+    if (isRepeatedRestartFault(kind)) receipt.repeatedRestart = this.repeatedRestartRecord(kind, receipt, restartSteps, previous?.pid ?? null);
     this.faultInProgress = true;
     this.serverReady = false;
     this.timeline.add("fault_started", { faultId: receipt.faultId, fault: kind, signal: receipt.signal, loadAvg1m: receipt.loadAvg1m, holdMs, requestedBy, previousPid: previous?.pid });
@@ -463,6 +468,7 @@ export class ChaosSupervisor {
         if (!this.child) throw new Error("K6 crash loop has no replacement server to kill");
         receipt.healthyAt = await this.waitForHealth(await this.crashLoop(this.child, receipt), receipt.phases);
       }
+      if (receipt.repeatedRestart) receipt.healthyAt = await this.repeatedRestarts(receipt.repeatedRestart, receipt);
       receipt.readyAt = await this.waitForReadyPhase(receipt.phases);
       receipt.serverLogOffsetAtReady = this.serverLogOffset();
       this.timeline.add("fault_ready", { faultId: receipt.faultId, replacementPid: this.child?.pid ?? outage?.replacementPid,
@@ -472,6 +478,7 @@ export class ChaosSupervisor {
       this.timeline.add("fault_failed", { faultId: receipt.faultId, error: receipt.error });
       throw Object.assign(new Error(`Fault ${receipt.faultId} failed: ${receipt.error}`), { receipt });
     } finally {
+      if (receipt.repeatedRestart) await this.writeRepeatedRestartEvidence(receipt);
       this.faultInProgress = false;
     }
     return receipt;
@@ -532,6 +539,64 @@ export class ChaosSupervisor {
     return upgrade;
   }
 
+  /** RR1–RR3: the planned steps and the first kill (as K1), recorded before that kill. */
+  private repeatedRestartRecord(kind: RepeatedRestartKind, receipt: FaultReceipt, planned: readonly RestartStep[] | undefined,
+    firstPid: number | null): RepeatedRestartRecord {
+    return newRepeatedRestartRecord({ kind, planned, firstPid, startedAt: receipt.startedAt,
+      evidencePath: path.join(path.dirname(this.paths.serverLogPath), "repeated-restarts", `${receipt.faultId}.json`),
+      int: (min, max) => min + Math.floor(Math.random() * (max - min + 1)) });
+  }
+
+  /**
+   * RR1–RR3: the first kill has a healthy replacement. Stop it again after each planned gap, while it
+   * is still recovering, and start another. The sequence is one receipt: one fault window.
+   */
+  private async repeatedRestarts(record: RepeatedRestartRecord, receipt: FaultReceipt): Promise<number> {
+    const first = this.child;
+    recordFirstRestart(record, receipt, first?.pid ?? null);
+    if (!first || receipt.healthyAt === null) throw new Error(`${record.kind} has no healthy replacement after its first kill`);
+    const result = await runRepeatedRestarts({ record, steps: record.planned, server: first, healthyAt: receipt.healthyAt, hooks: {
+      now: () => Date.now(),
+      sleep: ms => delay(ms),
+      cancelled: () => this.shuttingDown,
+      pidOf: server => server.pid ?? null,
+      exitStatus: server => this.exited(server) ? { code: server.exitCode, signal: server.signalCode } : null,
+      readPhase: () => this.readPhase(),
+      stop: async (server, signal) => {
+        if (signal === "SIGKILL") await this.stopper.crashForRestart(server);
+        else await this.stopper(server, "SIGTERM");
+        if (this.child === server) this.child = null;
+        await this.waitForHealthToStop();
+      },
+      start: () => this.spawnServer(),
+      waitHealthy: server => this.waitForHealth(server, receipt.phases),
+      onEntry: (entry, stage) => {
+        if (stage === "stopping") {
+          this.timeline.add("repeated_restart_stop", { faultId: receipt.faultId, index: entry.index, signal: entry.signal, gapMs: entry.gapMs,
+            pid: entry.pid, phaseAtStop: entry.phaseAtStop });
+          this.log.write(`\n[restart-chaos] ${receipt.faultId} ${record.kind} restart ${entry.index} ${entry.signal} pid=${entry.pid} phase=${entry.phaseAtStop} gap=${entry.gapMs}ms\n`);
+        } else {
+          this.timeline.add("repeated_restart_healthy", { faultId: receipt.faultId, index: entry.index, replacementPid: entry.replacementPid,
+            downMs: (entry.healthyAt ?? 0) - entry.stopAt });
+        }
+      },
+    } });
+    return result.healthyAt;
+  }
+
+  /** `<wave>/repeated-restarts/<faultId>.json`, written whether the sequence passed or failed; faults.json carries the same record. */
+  private async writeRepeatedRestartEvidence(receipt: FaultReceipt): Promise<void> {
+    const record = receipt.repeatedRestart!;
+    // A first kill that failed before its replacement was healthy still records the times it reached.
+    recordFirstRestart(record, receipt, null);
+    try {
+      await mkdir(path.dirname(record.evidencePath!), { recursive: true });
+      await writeFile(record.evidencePath!, `${JSON.stringify({ faultId: receipt.faultId, requestedBy: receipt.requestedBy, error: receipt.error, ...record }, null, 2)}\n`, { mode: 0o600 });
+    } catch (error) {
+      this.log.write(`[restart-chaos] ${receipt.faultId} could not write repeated-restart evidence: ${error instanceof Error ? error.message : String(error)}\n`);
+    }
+  }
+
   private async waitForCrashLoopPhase(child: ChildProcess, target: (typeof CRASH_LOOP_TARGETS)[number],
     phases: FaultReceipt["phases"]): Promise<string> {
     const deadline = Date.now() + READY_TIMEOUT_MS;

tests/restart-chaos/types.ts

@@ -1,13 +1,15 @@
 /** Shared types for the control-plane restart chaos harness (plan §5). */
 import type { ProviderFaultEvent, ProviderFaultKind, ProviderFaultRule, ProviderMessage } from "./provider-faults.js";
 import type { QuestionArtifacts, QuestionRecord } from "./question-handoff.js";
+import type { RepeatedRestartRecord, RestartStep } from "./repeated-restarts.js";
 import type { TimeFaultRecord } from "./time-faults.js";
 
 /**
  * K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`).
  * TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage.
+ * RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers.
  */
-export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3"] as const;
+export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3"] as const;
 export type FaultKind = (typeof FAULT_KINDS)[number];
 /**
  * A task's planned fault. "K13" (restart storm) tasks have none of their own; the wave's storm crosses them.
@@ -112,6 +114,8 @@ export interface WaveOptions {
   /** `--provider-fault-restart K1`: each PV1/PV2 task also kills -9 the control plane after its fault applied. */
   providerFaultRestart?: "K1";
   providerFaultRestartAfterMs?: number;
+  /** RR1–RR3: the gap from a replacement's health to the next stop (`--rr-gap-seconds`); absent keeps each kind's default. */
+  rrGapMs?: [number, number];
 }
 
 export interface StormFault {
@@ -149,6 +153,8 @@ export interface TaskPlan {
   replayFaults?: ScheduledFault[];
   /** PV1–PV3 tasks: the provider message rule armed for this task's nonce. */
   providerFault?: ProviderFaultRule;
+  /** RR1–RR3: the seeded restarts after the first kill. */
+  restartSteps?: RestartStep[];
 }
 
 export interface ScheduledFault {
@@ -156,6 +162,8 @@ export interface ScheduledFault {
   atMs: number;
   trigger?: TriggerId;
   delayMs?: number;
+  /** RR1–RR3: the restarts after the first kill. */
+  restartSteps?: RestartStep[];
 }
 
 export interface WavePlan {
@@ -254,6 +262,8 @@ export interface FaultReceipt {
   upgrade?: { fromRoot: string; fromSha: string | null; toRoot: string; toSha: string | null; switched: boolean };
   /** TM1–TM3: what was frozen and which rows were aged. */
   timeFault?: TimeFaultRecord;
+  /** RR1–RR3: every stop and replacement of the sequence, the first kill included. */
+  repeatedRestart?: RepeatedRestartRecord;
   serverLogOffsetAtStart: number;
   serverLogOffsetAtReady: number | null;
   error: string | null;