test(chaos): fire planned faults at a fixed time after launch with --fault-after-seconds (#20345)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness fires each task's fault at a seeded time
inside the workload's window (R1: 20–90 s after launch)
> - With a Paperclip-sized remote workspace on Daytona, preparation
(sandbox clone and the initial snapshot save) takes longer than 90 s
> - Every restart in a Daytona R1 wave therefore lands before the run
resolves its runtime mode, so only the pre-resolution recovery path is
ever tested
> - This pull request adds `--fault-after-seconds`, which fires every
planned fault at a fixed time after launch while keeping the rest of the
seeded plan
> - The benefit is that one seed can test a restart both during
preparation and after native resolution

## Linked Issues or Issue Description

**What happened?**
Daytona R1 waves w-20261109T163514-s9913 and w-20261109T185009-s9950
restarted the controller at 43 s and 84 s. Both runs were still
preparing their remote workspace, so the native recovery path for a
real-size workspace was never exercised.

**Expected behavior**
An operator can choose when the fault fires relative to the task launch.

**Actual behavior**
The fault time could only come from the seeded workload window.

## What Changed

- `--fault-after-seconds <n>` (5–3600) sets `faultAfterMs`. `planTasks`
still draws the seeded fire time, so nonces, holds and triggers are
unchanged, but it uses `faultAfterMs` for every planned fault except
K13.
- The option is rejected with storms and crash-point waves, which have
no planned fault times.
- README: one line under Options.

## Verification

- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 259
passed. The new scheduler test fails before the change. It checks that
every planned fault fires at 150 s, that the rest of the plan matches
the same seed without the option, and that out-of-range or storm use is
rejected.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.

## Risks

- A fixed fault time can land after a fast task finishes, in which case
the fault crosses no work for that task. The trigger is recorded as
missed, as for any late fault.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tests/restart-chaos/README.md

@@ -25,6 +25,8 @@ Options: `--triggers time,comment,retry,queued,tool,result` (fault timing, below
 `--workloads A1,B1,B5,C1`, `--budget-cents` (company hard cap,
 default 3000), `--poll-ms` (2000–5000), `--keep` (keep the temporary instance),
 `--k4-hold-max-seconds` (raise the seeded K4 hold upper end from 300 s, up to the 600 s outage cap).
+`--fault-after-seconds <n>` fires every planned fault `n` seconds after its task launches (5–3600; with `--triggers time`), keeping
+the rest of the seeded plan, for example to restart a Daytona R1 task after its workspace preparation.
 
 Each wave's temporary instance (embedded Postgres, workspaces, often 5–13 GB) is
 deleted when the wave ends, pass or fail, including after an interrupt. Evidence

tests/restart-chaos/config.ts

@@ -95,6 +95,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
       "storm-minutes": { type: "string" },
       lifecycle: { type: "string" },
       "k4-hold-max-seconds": { type: "string" },
+      "fault-after-seconds": { type: "string" },
       "crash-point": { type: "string" },
       "crash-point-hold-ms": { type: "string" },
       "upgrade-from": { type: "string" },
@@ -108,7 +109,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
   const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000);
   const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000);
   if (values.replay !== undefined) {
-    for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds"] as const) {
+    for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "fault-after-seconds"] as const) {
       if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`);
     }
     return { mode: "replay", ref: parseReplayRef(values.replay), keep,
@@ -163,10 +164,18 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
     // Seeded K4 hold upper end; the controller-outage cap is 600 s (#19797).
     ...(values["k4-hold-max-seconds"] !== undefined
       ? { k4HoldMaxMs: integer(values["k4-hold-max-seconds"], "k4-hold-max-seconds", 300, 120, 600) * 1000 } : {}),
+    ...(values["fault-after-seconds"] !== undefined
+      ? { faultAfterMs: integer(values["fault-after-seconds"], "fault-after-seconds", 60, 5, 3_600) * 1000 } : {}),
     ...(crashPoint ? { crashPoint } : {}),
     ...(crashPoint && values["crash-point-hold-ms"] !== undefined
       ? { crashPointHoldMs: integer(values["crash-point-hold-ms"], "crash-point-hold-ms", 0, 0, 600_000) } : {}),
   };
+  if (values["fault-after-seconds"] !== undefined && (stormMinutes || crashPoint)) {
+    throw new Error("--fault-after-seconds sets planned fault times; a storm or crash-point wave has none");
+  }
+  if (values["fault-after-seconds"] !== undefined && (options.triggers ?? []).some(trigger => trigger !== "time")) {
+    throw new Error("--fault-after-seconds fires faults by time; use it with --triggers time");
+  }
   if (!crashPoint && values["crash-point-hold-ms"] !== undefined) throw new Error("--crash-point-hold-ms applies only to --crash-point");
   const upgradeFrom = values["upgrade-from"];
   if (options.faults.includes("K14") !== (upgradeFrom !== undefined)) {

tests/restart-chaos/scheduler.test.ts

@@ -36,6 +36,28 @@ describe("scheduler determinism", () => {
     }
   });
 
+  it("fires every planned fault at --fault-after-seconds without changing the rest of the seeded plan", () => {
+    const base = parseCommand(["--faults", "K1,K2", "--tasks", "6", "--seed", "77"]);
+    const pinned = parseCommand(["--faults", "K1,K2", "--tasks", "6", "--seed", "77", "--fault-after-seconds", "150"]);
+    if (base.mode !== "wave" || pinned.mode !== "wave") throw new Error("expected a wave");
+    expect(pinned.options.faultAfterMs).toBe(150_000);
+    const planned = planTasks(pinned.options);
+    expect(planned.map(task => task.fireAfterMs)).toEqual(Array(6).fill(150_000));
+    expect(planned.map(({ fireAfterMs: _fire, ...rest }) => rest)).toEqual(planTasks(base.options).map(({ fireAfterMs: _fire, ...rest }) => rest));
+    expect(() => parseCommand(["--fault-after-seconds", "2"])).toThrow(/fault-after-seconds/);
+    expect(() => parseCommand(["--storm-minutes", "10", "--fault-after-seconds", "60"])).toThrow(/storm/);
+    expect(() => parseCommand(["--triggers", "time,comment", "--fault-after-seconds", "60"])).toThrow(/--triggers time/);
+  });
+
+  it("keeps the plans existing seeds produce for triggered and storm waves", () => {
+    const triggered = planTasks({ seed: 4242, tasks: 4, workloads: ["A1", "C1"], adapters: ["codex", "claude"], faults: ["K1", "K2"],
+      triggers: ["time", "comment", "result"] } as never);
+    expect(triggered.map(task => [task.trigger ?? "time", task.triggerDelayMs ?? null, task.fireAfterMs, task.nonce])).toEqual([
+      ["time", null, 36292, "d3a12313"], ["time", null, 14380, "37697e51"], ["comment", 4272, 23204, "a5d0f179"], ["time", null, 51015, "b9f3ef15"]]);
+    const storm = planTasks({ seed: 4242, tasks: 3, workloads: ["A1"], adapters: ["codex"], faults: ["K1"], triggers: ["time"], stormMinutes: 10 } as never);
+    expect(storm.map(task => [task.fault, task.fireAfterMs, task.nonce])).toEqual([["K13", 0, "84e8a446"], ["K13", 0, "59bc9f5c"], ["K13", 0, "d3a12313"]]);
+  });
+
   it("raises the K4 hold upper end with --k4-hold-max-seconds, up to the 600 s outage cap", () => {
     const command = parseCommand(["--faults", "K4", "--k4-hold-max-seconds", "600", "--seed", "5"]);
     if (command.mode !== "wave") throw new Error("expected a wave");

tests/restart-chaos/scheduler.ts

@@ -80,7 +80,11 @@ function stormCellFaults(options: Pick<WaveOptions, "faults" | "stormMinutes">,
  * Pairwise-friendly sampling: cycle a seeded shuffle of the full
  * workload × adapter × fault product so small waves still spread across cells.
  */
-export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloads" | "adapters" | "faults" | "triggers" | "stormMinutes" | "k4HoldMaxMs">): TaskPlan[] {
+function pinnedFireAfterMs(faultAfterMs: number | undefined, drawn: number): number {
+  return faultAfterMs ?? drawn;
+}
+
+export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloads" | "adapters" | "faults" | "triggers" | "stormMinutes" | "k4HoldMaxMs" | "faultAfterMs">): TaskPlan[] {
   const rng = createRng(options.seed);
   const triggers = options.triggers?.length ? options.triggers : ["time" as const];
   for (const trigger of triggers) {
@@ -110,7 +114,7 @@ export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloa
       index,
       ...cellRest,
       ...(trigger === "time" ? {} : { trigger, triggerDelayMs: rng.int(...TRIGGER_DELAY_RANGE_MS[trigger]) }),
-      fireAfterMs: cell.fault === "K13" ? 0 : rng.int(fireMin, fireMax),
+      fireAfterMs: cell.fault === "K13" ? 0 : pinnedFireAfterMs(options.faultAfterMs, rng.int(fireMin, fireMax)),
       holdMs: cell.fault === "K4" ? rng.int(...k4HoldRange(options.k4HoldMaxMs)) : 0,
       followUpAfterMs: definition.followUpWindowMs ? rng.int(definition.followUpWindowMs[0], definition.followUpWindowMs[1]) : null,
       nonce: rng.hex(8),

tests/restart-chaos/types.ts

@@ -40,6 +40,7 @@ export interface WaveOptions {
   daytonaReuseLease?: boolean;
   /** `--k4-hold-max-seconds`: upper end of the seeded K4 hold (default 300 s, cap 600 s). */
   k4HoldMaxMs?: number;
+  faultAfterMs?: number;
   /**
    * Known product signatures this wave deliberately masks. "rc06" keeps the
    * company budget amount but disables its hard stop, so a stuck