diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 04bb9cd4d41..3619db989c4 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -538,7 +538,9 @@ Codex app-server is late, missing or repeated. `--faults` takes them next to res governed wait (a saved question or approval) or a board Stop sends. `interrupt_ack` picks the acknowledgement of that interruption. Each rule applies once per task, to the first matching message. Provider faults run only with `--env local --adapters codex,claude` (either or both) and only with -workloads that interrupt a turn (Q1, S1, G1). A PV1 task passes `provider_fault_applied` only when the +workloads that interrupt a turn (Q1, S1). G1 does not: its agent asks for approval through `call_api` +and ends its own turn, so no rule ever applied there (waves w-20261110T050407-s206, w-20261110T063253-s208, +w-20261110T085554-s211) and the board waited for a restart that never came. A PV1 task passes `provider_fault_applied` only when the held message was released; a provider that exited first turned the delay into a drop. For Claude, the runner settles a cancelled turn about 3–4 s after the cancel without waiting for the provider's prompt response, and it later closes the sidecar with the provider. A Claude PV1 delay @@ -546,8 +548,7 @@ longer than that is therefore a drop (wave w-20261110T040202-s202: 18 s held, ne still passed its Q1 oracle). Sweep Claude PV1 below that window, for example `--provider-fault-delays 1,3`. -G1 grades the board decision after a restart, so G1 with a provider fault needs -`--provider-fault-restart K1`. On a board Stop (S1) the runner ends Codex without waiting for its terminal: +On a board Stop (S1) the runner ends Codex without waiting for its terminal: in wave w-20261110T050407-s206 Codex exited 1.5 s after `turn/interrupt` with no `turn/completed`, so a `terminal` rule never applies there and the task fails `provider_fault_applied`. Use `interrupt_ack` for S1. An S1 task with a provider fault clicks Stop on the first poll that shows a live provider turn @@ -570,10 +571,28 @@ after its fault applied, through the `provider` trigger. That delay must be shor so the message is still held, and the wave runs with `--concurrency 1`, because every K1 kills the whole control plane. K1 kills the runner with its provider and the held message, so the late or missing terminal straddles the restart: the replacement control plane must settle the saved wait without it. Planned restart faults (`--faults K1,PV2`) are still rejected, and PV3 does not combine with -a restart, since a duplicate holds nothing. Such a task passes when: +a restart, since a duplicate holds nothing. -- `provider_fault_applied`: the fault applied once, the K1 started after it, and for PV1 no release - came before the K1; +`--provider-fault-restart` also takes K2 (SIGTERM, a graceful restart whose shutdown drain meets the +held or missing message) and K4 (kill -9, then the control plane stays down for 120 s, the low end of +the seeded K4 range). For a restart, `provider_fault_applied` requires that the held message was not +released before the old control plane stopped (`stoppedAt`), not only before the fault started. A K2 drain took 15–20 s in wave w-20261110T092735-s213, so a PV1 that must straddle a K2 needs a +longer delay, for example `--provider-fault-delays 40`. A comma list such as `K1,K2,K4` cycles across the PV1/PV2 tasks in plan order, +so one wave covers several restart kinds. `--provider-fault-message` takes `WORKLOAD=message` entries +next to the default, for example `S1=interrupt_ack,terminal`: S1 tasks fault the interrupt +acknowledgement (Codex ends a Stop without its terminal) while Q1 tasks fault the terminal. + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \ + --adapters codex --workloads Q1,S1 --faults PV1,PV2 --provider-fault-restart K2,K4 --provider-fault-delays 40 \ + --provider-fault-message S1=interrupt_ack,terminal --concurrency 1 --tasks 4 --seed 211 +``` + +Such a task passes when: + +- `provider_fault_applied`: the fault applied once, the restart started after it, the held message was + not released before the old control plane stopped, and it was not discarded before the restart started + (a discard during a K2 shutdown is the restart stopping the provider); - the ordinary grade passes: R1–R6 for a restart-crossing task and the workload's T-oracle. For Q1 that means one question, the asking run `succeeded`, no failed run, nothing started before the answer, and the brief carries the answer. @@ -942,7 +961,11 @@ re-arms the task's own triggered faults instead of using their offsets. - Speed: an aggregate p50/p95 row can breach only with at least 5 samples; a smaller row is reported as `insufficient sample (n < 5)`. Every cold or warm run over its p95 budget, and every restart recovery whose Paperclip share exceeds - its p95 budget, still fails the wave on its own. + its p95 budget, still fails the wave on its own. `speed.json` keeps each graded + run's own split (`runs`): `bootMs` (Paperclip: claim to the provider accepting + the turn), `providerFirstActivityMs` (turn accepted to first visible activity), + and the spans before the turn, so a breach stays attributable after the + temporary instance is deleted. - Faults are serialized; each one hits every task running at that moment. - A `todo` or `in_progress` task with no live run and no named wait is not a rest. The observer keeps watching it until it resumes or passes the R5 stall limit. - R5 also fails a run that stays `running` with no live evidence (no new run event, `lastUsefulActionAt` or `execution.lastConfirmedActivityAt`, and no start) for more than 3 minutes after the server is ready. A `running` status alone is not proof of life. A quiet run is still live while a tool it started has not ended (for example a 4-minute `bench.sh`) or while its recorded local runner process still runs with the same PID and start time. diff --git a/tests/restart-chaos/acpx-provider-faults.test.ts b/tests/restart-chaos/acpx-provider-faults.test.ts index ae27be62b10..ed8f48f8cd2 100644 --- a/tests/restart-chaos/acpx-provider-faults.test.ts +++ b/tests/restart-chaos/acpx-provider-faults.test.ts @@ -235,14 +235,31 @@ describe("provider faults across a restart", () => { } }); + it("cycles K1, K2 and K4 restarts across PV tasks and gives each workload its own message", () => { + const command = parseCommand(["--adapters", "codex", "--workloads", "S1,Q1", "--faults", "PV1,PV2", "--provider-fault-restart", "K2,K4,K1", + "--provider-fault-message", "S1=interrupt_ack,terminal", "--tasks", "4", "--concurrency", "1", "--seed", "9"]); + if (command.mode !== "wave") throw new Error("expected a wave"); + expect(command.options).toMatchObject({ providerFaultRestart: "K2", providerFaultRestartKinds: ["K2", "K4", "K1"], + providerFaultMessage: "terminal", providerFaultMessageByWorkload: { S1: "interrupt_ack" } }); + const plan = planWave("w-20261110T000000-s9", command.options); + expect(plan.tasks.map(planned => planned.providerFault?.restart?.kind)).toEqual(["K2", "K4", "K1", "K2"]); + for (const planned of plan.tasks) { + expect(planned.providerFault?.message).toBe(planned.workload === "S1" ? "interrupt_ack" : "terminal"); + const restart = planned.providerFault!.restart!; + expect(taskFaultSchedule(planned)).toEqual([{ kind: restart.kind, holdMs: restart.kind === "K4" ? 120_000 : 0, atMs: 0, trigger: "provider", delayMs: 3_000 }]); + } + expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-message", "Q2=terminal"])).toThrow(/not one of/); + expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-message", "terminal,interrupt_ack"])).toThrow(/one default/); + }); + it("rejects restarts that cannot straddle the fault", () => { expect(() => parseCommand(["--adapters", "claude", "--workloads", "Q1", "--faults", "PV3", "--provider-fault-restart", "K1"])).toThrow(/PV1 or PV2/); expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV1", "--provider-fault-delays", "5", "--provider-fault-restart", "K1", "--provider-fault-restart-after-seconds", "6"])).toThrow(/shorter than every PV1 delay/); - expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-restart", "K2"])).toThrow(/K1/); + expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-restart", "K6"])).toThrow(/K1, K2, K4/); expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "K1", "--provider-fault-restart", "K1"])).toThrow(/provider fault/); - expect(() => parseCommand(["--adapters", "codex", "--workloads", "G1", "--faults", "PV2"])).toThrow(/G1 grades the board decision after a restart/); - expect(parseCommand(["--adapters", "codex", "--workloads", "G1", "--faults", "PV2", "--provider-fault-restart", "K1", "--concurrency", "1"]).mode).toBe("wave"); + // G1's agent ends its own turn after asking for approval: no interrupt, so no provider rule can apply. + expect(() => parseCommand(["--adapters", "codex", "--workloads", "G1", "--faults", "PV2", "--provider-fault-restart", "K1", "--concurrency", "1"])).toThrow(/remove G1/); expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-restart", "K1", "--concurrency", "2"])).toThrow(/--concurrency 1/); expect(parseProviderFaultRestart(undefined, undefined, ["PV2"], undefined)).toEqual({}); expect(() => parseProviderFaultRestart(undefined, "3", ["PV2"], undefined)).toThrow(/needs --provider-fault-restart/); @@ -257,6 +274,17 @@ describe("provider faults across a restart", () => { expect(withProviderFaultCheck(pass, task("aa", rule), [applied, { ...applied, event: "released", at: 2_000 }], 4_000).checks.at(-1)) .toMatchObject({ ok: false, detail: expect.stringContaining("releasedBeforeRestart") }); expect(withProviderFaultCheck(pass, task("aa", rule), [applied], 500).verdict).toBe("fail"); + // K2: a release inside the graceful drain (after the SIGTERM, before the old process stopped) reached + // the old control plane, so the message straddled nothing. + const k2: ProviderFaultRule = { ...rule, restart: { kind: "K2", afterAppliedMs: 3_000 } }; + expect(withProviderFaultCheck(pass, task("aa", k2), [applied, { ...applied, event: "released", at: 10_000 }], 4_000, 20_000).checks.at(-1)) + .toMatchObject({ ok: false, detail: expect.stringContaining("releasedBeforeRestart") }); + expect(withProviderFaultCheck(pass, task("aa", k2), [applied, { ...applied, event: "released", at: 25_000 }], 4_000, 20_000).verdict).toBe("pass"); + // A discard inside the K2 shutdown is its cleanup stopping the provider with the message still held: + // that straddles. Only a provider that exited before the restart started took the message first. + expect(withProviderFaultCheck(pass, task("aa", k2), [applied, { ...applied, event: "held_message_discarded", at: 10_000 }], 4_000, 20_000).verdict).toBe("pass"); + expect(withProviderFaultCheck(pass, task("aa", k2), [applied, { ...applied, event: "held_message_discarded", at: 2_000 }], 4_000, 20_000).checks.at(-1)) + .toMatchObject({ ok: false, detail: expect.stringContaining("discardedBeforeRestart") }); // A provider that closed first took the held message with it before the K1: nothing straddled the restart. expect(withProviderFaultCheck(pass, task("aa", rule), [applied, { ...applied, event: "held_message_discarded", at: 3_000 }], 4_000).checks.at(-1)) .toMatchObject({ ok: false, detail: expect.stringContaining("discardedBeforeRestart") }); diff --git a/tests/restart-chaos/acpx-provider-faults.ts b/tests/restart-chaos/acpx-provider-faults.ts index c004cd85812..0770158d18e 100644 --- a/tests/restart-chaos/acpx-provider-faults.ts +++ b/tests/restart-chaos/acpx-provider-faults.ts @@ -27,7 +27,7 @@ import { chmod, mkdir, readdir, realpath, rm, symlink, writeFile } from "node:fs/promises"; import { existsSync } from "node:fs"; import path from "node:path"; -import { DEFAULT_PROVIDER_DELAY_MS, DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS, type ProviderFaultKind } from "./provider-faults.js"; +import { DEFAULT_PROVIDER_DELAY_MS, DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS, PROVIDER_RESTART_KINDS, type ProviderFaultKind, type ProviderRestartKind } from "./provider-faults.js"; import type { AdapterId, WaveOptions } from "./types.js"; export const ACPX_SIDECAR_HOOK = path.join(import.meta.dirname, "provider-shim", "acpx-sidecar-hook.mjs"); @@ -119,12 +119,15 @@ export function withAcpxSidecarHook(nodeOptions: string | undefined): string { * not PV3: a duplicate has nothing in flight for a restart to cross. */ export function parseProviderFaultRestart(raw: string | undefined, afterRaw: string | undefined, kinds: readonly ProviderFaultKind[], - delaysMs: readonly number[] | undefined): Pick { + delaysMs: readonly number[] | undefined): Pick { if (raw === undefined) { - if (afterRaw !== undefined) throw new Error("--provider-fault-restart-after-seconds needs --provider-fault-restart K1"); + if (afterRaw !== undefined) throw new Error("--provider-fault-restart-after-seconds needs --provider-fault-restart (K1, K2 or K4)"); return {}; } - if (raw !== "K1") throw new Error("--provider-fault-restart supports K1 (kill -9 the control plane) only"); + const restartKinds = raw.split(",").map(value => value.trim()).filter(Boolean) as ProviderRestartKind[]; + if (!restartKinds.length || restartKinds.some(kind => !(PROVIDER_RESTART_KINDS as readonly string[]).includes(kind))) { + throw new Error(`--provider-fault-restart takes ${PROVIDER_RESTART_KINDS.join(", ")} (comma-separated kinds cycle across PV1/PV2 tasks)`); + } if (kinds.includes("PV3")) throw new Error("--provider-fault-restart combines with PV1 or PV2; a PV3 duplicate holds nothing across a restart"); let afterMs = DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS; if (afterRaw !== undefined) { @@ -137,5 +140,6 @@ export function parseProviderFaultRestart(raw: string | undefined, afterRaw: str if (shortest !== null && afterMs >= shortest) { throw new Error(`--provider-fault-restart-after-seconds (${afterMs / 1000}s) must be shorter than every PV1 delay (shortest ${shortest / 1000}s)`); } - return { providerFaultRestart: "K1", providerFaultRestartAfterMs: afterMs }; + return { providerFaultRestart: restartKinds[0]!, providerFaultRestartAfterMs: afterMs, + ...(restartKinds.length > 1 ? { providerFaultRestartKinds: restartKinds } : {}) }; } diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index d68740f5951..d8f702f1f00 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -7,7 +7,7 @@ import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js"; import { parseTimeFaultOptions } from "./time-faults-config.js"; import { parseStorageTrigger } from "./storage-trigger.js"; import { parseRepeatedRestartOptions } from "./repeated-restarts.js"; -import { PROVIDER_FAULT_KINDS, PROVIDER_FAULT_WORKLOADS, isProviderFault, parseProviderDelays, parseProviderMessage, type ProviderFaultKind } from "./provider-faults.js"; +import { PROVIDER_FAULT_KINDS, PROVIDER_FAULT_WORKLOADS, isProviderFault, parseProviderDelays, parseProviderMessages, type ProviderFaultKind } from "./provider-faults.js"; import { PROVIDER_FAULT_ADAPTERS, parseProviderFaultRestart } from "./acpx-provider-faults.js"; import { ADAPTER_IDS, ENVIRONMENT_KINDS, FAULT_KINDS, TRIGGER_IDS, WORKLOAD_IDS, @@ -209,17 +209,16 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC // Every planned task gets the fault; only these workloads send the turn/interrupt the rules target. const idle = options.workloads.filter(id => !PROVIDER_FAULT_WORKLOADS.includes(id)); if (idle.length) throw new Error(`Provider message faults (PV*) need workloads that interrupt a provider turn (${PROVIDER_FAULT_WORKLOADS.join(", ")}); remove ${idle.join(", ")} from --workloads`); - const message = parseProviderMessage(values["provider-fault-message"]); - if (message) options.providerFaultMessage = message; + const messages = parseProviderMessages(values["provider-fault-message"]); + if (messages.message) options.providerFaultMessage = messages.message; + if (messages.byWorkload) options.providerFaultMessageByWorkload = messages.byWorkload; const delays = parseProviderDelays(values["provider-fault-delays"]); if (delays && !providerFaults.includes("PV1")) throw new Error("--provider-fault-delays sets PV1 delays; add PV1 to --faults"); if (delays) options.providerFaultDelaysMs = delays; Object.assign(options, parseProviderFaultRestart(values["provider-fault-restart"], values["provider-fault-restart-after-seconds"], providerFaults, delays)); // Each K1 kills the whole control plane: a second task in flight would lose its held message to the // first task's restart and miss its own, so a restart wave runs one task at a time. - if (options.providerFaultRestart && options.concurrency !== 1) throw new Error("--provider-fault-restart kills the whole control plane per task; run it with --concurrency 1"); - // G1's oracle requires the board's decision to come after a restart (board_decided_after_restart). - if (options.workloads.includes("G1") && !options.providerFaultRestart) throw new Error("G1 grades the board decision after a restart; with provider faults add --provider-fault-restart K1"); + if (options.providerFaultRestart && options.concurrency !== 1) throw new Error("--provider-fault-restart restarts the whole control plane per task; run it with --concurrency 1"); } else if (values["provider-fault-message"] !== undefined || values["provider-fault-delays"] !== undefined || values["provider-fault-restart"] !== undefined || values["provider-fault-restart-after-seconds"] !== undefined) { throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults"); diff --git a/tests/restart-chaos/main.ts b/tests/restart-chaos/main.ts index ba95ece9264..7e88ae8f67b 100644 --- a/tests/restart-chaos/main.ts +++ b/tests/restart-chaos/main.ts @@ -611,7 +611,8 @@ async function main(): Promise { : { verdict: "fail" as const, checks: [{ name: "artifacts_collected", ok: false, detail: observer.record.launchError ?? "collection failed" }] }; // A provider-fault task whose fault never happened tested nothing. const restart = faults.find(receipt => receipt.requestedBy === observer.plan.taskId && receipt.trigger === "provider"); - const oracle = withProviderFaultCheck(graded, observer.plan, artifacts?.providerFaultEvents ?? [], restart?.startedAt ?? null); + const oracle = withProviderFaultCheck(graded, observer.plan, artifacts?.providerFaultEvents ?? [], restart?.startedAt ?? null, + restart?.stoppedAt ?? restart?.startedAt ?? null); const grade = gradeTask(observer.record, faults, observer.definition.budgetMs, oracle, universalByIssue ? { runMessages: allRunMessages, recoveryActions: universalByIssue.get(observer.record.issueId ?? "")?.recoveryActions ?? [] } : undefined); let evidence: string | null = null; diff --git a/tests/restart-chaos/provider-faults.ts b/tests/restart-chaos/provider-faults.ts index 17b0f210527..3c444dd3535 100644 --- a/tests/restart-chaos/provider-faults.ts +++ b/tests/restart-chaos/provider-faults.ts @@ -19,7 +19,7 @@ import { chmod, copyFile, mkdir, readFile, rename, symlink, writeFile } from "no import { readFileSync, realpathSync } from "node:fs"; import { createRequire } from "node:module"; import path from "node:path"; -import type { OracleResult, TaskPlan, WaveOptions, WorkloadId } from "./types.js"; +import { K4_MIN_HOLD_MS, type OracleResult, type TaskPlan, type WaveOptions, type WorkloadId } from "./types.js"; export const PROVIDER_FAULT_KINDS = ["PV1", "PV2", "PV3"] as const; export type ProviderFaultKind = (typeof PROVIDER_FAULT_KINDS)[number]; @@ -29,6 +29,12 @@ export const PROVIDER_FAULT_ACTIONS: Record> } { + if (raw === undefined) return {}; + const result: { message?: ProviderMessage; byWorkload?: Partial> } = {}; + const entries = raw.split(",").map(value => value.trim()).filter(Boolean); + if (!entries.length) throw new Error(`--provider-fault-message must be one of ${PROVIDER_MESSAGES.join(", ")}`); + for (const entry of entries) { + const [left, right] = entry.includes("=") ? entry.split("=", 2) as [string, string] : [undefined, entry]; + if (!(PROVIDER_MESSAGES as readonly string[]).includes(right)) throw new Error(`--provider-fault-message must be one of ${PROVIDER_MESSAGES.join(", ")}`); + if (left === undefined) { + if (result.message) throw new Error("--provider-fault-message takes one default message"); + result.message = right as ProviderMessage; + } else { + if (!PROVIDER_FAULT_WORKLOADS.includes(left as WorkloadId)) throw new Error(`--provider-fault-message: ${left} is not one of ${PROVIDER_FAULT_WORKLOADS.join(", ")}`); + result.byWorkload = { ...result.byWorkload, [left]: right as ProviderMessage }; + } + } + return result; } /** Attach each provider-fault task's rule. Restart-fault tasks are unchanged. */ export function withProviderFaultRules(tasks: TaskPlan[], options: Pick): TaskPlan[] { + | "providerFaultRestart" | "providerFaultRestartAfterMs" | "providerFaultRestartKinds" | "providerFaultMessageByWorkload">): TaskPlan[] { const delays = options.providerFaultDelaysMs?.length ? options.providerFaultDelaysMs : [DEFAULT_PROVIDER_DELAY_MS]; + const restarts = options.providerFaultRestartKinds?.length ? options.providerFaultRestartKinds + : options.providerFaultRestart ? [options.providerFaultRestart] : []; let pv1 = 0; + let restarted = 0; return tasks.map(task => { if (!isProviderFault(task.fault)) return task; - const rule: ProviderFaultRule = { kind: task.fault, message: options.providerFaultMessage ?? "terminal", + const restartKind = restarts.length && task.fault !== "PV3" ? restarts[restarted++ % restarts.length]! : null; + const rule: ProviderFaultRule = { kind: task.fault, + message: options.providerFaultMessageByWorkload?.[task.workload] ?? options.providerFaultMessage ?? "terminal", ...(task.fault === "PV1" ? { delayMs: delays[pv1++ % delays.length]! } : {}), - ...(options.providerFaultRestart && task.fault !== "PV3" - ? { restart: { kind: options.providerFaultRestart, afterAppliedMs: options.providerFaultRestartAfterMs ?? DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS } } : {}) }; + ...(restartKind + ? { restart: { kind: restartKind, afterAppliedMs: options.providerFaultRestartAfterMs ?? DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS, + ...(restartKind === "K4" ? { holdMs: PROVIDER_FAULT_K4_HOLD_MS } : {}) } } : {}) }; return { ...task, providerFault: rule, fireAfterMs: 0, holdMs: 0 }; }); } export function describeProviderFault(rule: ProviderFaultRule): string { return `${rule.kind} ${PROVIDER_FAULT_ACTIONS[rule.kind]} ${rule.message}${rule.delayMs !== undefined ? ` ${rule.delayMs}ms` : ""}` - + (rule.restart ? ` +${rule.restart.kind}@${rule.restart.afterAppliedMs}ms` : ""); + + (rule.restart ? ` +${rule.restart.kind}@${rule.restart.afterAppliedMs}ms${rule.restart.holdMs ? `/hold${rule.restart.holdMs}ms` : ""}` : ""); } /** The Codex executable product code would resolve for this checkout (mirrors codex-command.ts). */ @@ -211,13 +250,17 @@ export async function readProviderFaultEvents(logPath: string, nonce?: string): * A provider-fault task passes only if its fault really happened: otherwise the scenario was not * exercised (for example the turn ended before any interrupt), and a pass would prove nothing. * - * With `--provider-fault-restart K1` the fault must also straddle the restart: the K1 started after - * the fault applied and, for PV1, before the held message was released or discarded. The K1 kills the runner - * with the held message, so a PV1 task then needs no release. Whether the task itself recovered + * With `--provider-fault-restart` the fault must also straddle the restart: the restart started after + * the fault applied, the held message was not released before the old control plane had stopped + * (`restartStoppedAt`), and it was not discarded before the restart started. A K2 drains gracefully + * between its start and its stop, so a message released in that drain reached the old process and + * straddled nothing. A discard after the start is the restart's own cleanup stopping the provider with + * the message still held, which is the straddle. The kill takes the held message with the runner, so a + * PV1 task then needs no release. Whether the task itself recovered * is the ordinary grade (R1–R6 and its T-oracle), which this check never replaces. */ export function withProviderFaultCheck(oracle: OracleResult, plan: TaskPlan, events: readonly ProviderFaultEvent[], - restartStartedAt: number | null = null): OracleResult { + restartStartedAt: number | null = null, restartStoppedAt: number | null = restartStartedAt): OracleResult { if (!plan.providerFault) return oracle; const rule = plan.providerFault; const applied = events.filter(event => event.event === "applied" && event.kind === rule.kind && event.message === rule.message); @@ -229,8 +272,11 @@ export function withProviderFaultCheck(oracle: OracleResult, plan: TaskPlan, eve let restartDetail = ""; if (rule.restart) { const appliedAt = applied[0]?.at ?? null; - // Released or discarded before the K1: the message was already gone, so nothing straddled the restart. - const releasedBeforeRestart = restartStartedAt !== null && released.some(event => event.at < restartStartedAt); + // Released before the old control plane stopped (a K2 drain included): the old process got the + // message, so nothing straddled the restart. Discarded before the restart started: the provider + // exited on its own first. A discard after the start is the restart's cleanup and still straddles. + const stoppedAt = restartStoppedAt ?? restartStartedAt; + const releasedBeforeRestart = stoppedAt !== null && released.some(event => event.at < stoppedAt); const discardedBeforeRestart = restartStartedAt !== null && discarded.some(event => event.at < restartStartedAt); exercised = applied.length === 1 && appliedAt !== null && restartStartedAt !== null && restartStartedAt >= appliedAt && !releasedBeforeRestart && !discardedBeforeRestart; diff --git a/tests/restart-chaos/scheduler.ts b/tests/restart-chaos/scheduler.ts index 54c3514d5c3..44f7fa806bf 100644 --- a/tests/restart-chaos/scheduler.ts +++ b/tests/restart-chaos/scheduler.ts @@ -3,7 +3,7 @@ import { takesJourneyTriggers } from "./journeys.js"; import { workloadDefinitions } from "./workloads.js"; import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js"; import { isRepeatedRestartFault, planRestartSteps, type RestartStep } from "./repeated-restarts.js"; -import type { AdapterId, FaultKind, ScheduledFault, StormFault, StormPlan, TaskFaultKind, TaskPlan, TriggerId, WavePlan, WaveOptions, WorkloadId } from "./types.js"; +import { K4_MIN_HOLD_MS, type AdapterId, type FaultKind, type ScheduledFault, type StormFault, type StormPlan, type TaskFaultKind, type TaskPlan, type TriggerId, type WavePlan, type WaveOptions, type WorkloadId } from "./types.js"; import type { ReplayRef } from "./config.js"; /** mulberry32: small, fast and fully determined by its 32-bit seed. */ @@ -31,7 +31,7 @@ export function createRng(seed: number) { }; } -export const K4_HOLD_RANGE_MS: readonly [number, number] = [120_000, 300_000]; +export const K4_HOLD_RANGE_MS: readonly [number, number] = [K4_MIN_HOLD_MS, 300_000]; /** The seeded K4 hold range; `--k4-hold-max-seconds` raises its upper end * (up to the 600 s controller-outage cap) to reach the 7-minute reconnect @@ -290,7 +290,7 @@ export function taskFaultSchedule(task: TaskPlan): ScheduledFault[] { // A storm crosses K13 tasks; a provider fault (PV*) and a matrix baseline (K0) restart nothing, unless // `--provider-fault-restart` asked for a K1 once the provider fault applied (the `provider` trigger). const restart = task.providerFault?.restart; - if (restart) return [{ kind: restart.kind, holdMs: 0, atMs: 0, trigger: "provider", delayMs: restart.afterAppliedMs }]; + if (restart) return [{ kind: restart.kind, holdMs: restart.holdMs ?? 0, atMs: 0, trigger: "provider", delayMs: restart.afterAppliedMs }]; if (task.fault === "K13" || task.fault === "K0" || isProviderFault(task.fault)) return []; const planned: ScheduledFault = { kind: task.fault, holdMs: task.holdMs, atMs: task.fireAfterMs, ...(task.restartSteps ? { restartSteps: task.restartSteps } : {}) }; diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index 1cafd3b54aa..8d745327e0e 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -1,5 +1,5 @@ /** Shared types for the control-plane restart chaos harness (plan §5). */ -import type { ProviderFaultEvent, ProviderFaultKind, ProviderFaultRule, ProviderMessage } from "./provider-faults.js"; +import type { ProviderFaultEvent, ProviderFaultKind, ProviderFaultRule, ProviderMessage, ProviderRestartKind } from "./provider-faults.js"; import type { QuestionArtifacts, QuestionRecord } from "./question-handoff.js"; import type { RepeatedRestartRecord, RestartStep } from "./repeated-restarts.js"; import type { TimeFaultRecord } from "./time-faults.js"; @@ -11,6 +11,8 @@ import type { TimeFaultRecord } from "./time-faults.js"; */ export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3"] as const; export type FaultKind = (typeof FAULT_KINDS)[number]; +/** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */ +export const K4_MIN_HOLD_MS = 120_000; /** * A task's planned fault. "K13" (restart storm) tasks have none of their own; the wave's storm crosses them. * PV1–PV3 (provider message faults, provider-faults.ts) restart nothing: the task's rule is armed in the Codex shim. @@ -102,6 +104,8 @@ export interface WaveOptions { /** Provider message faults from `--faults` (PV1 delay, PV2 drop, PV3 duplicate); see provider-faults.ts. */ providerFaults?: ProviderFaultKind[]; providerFaultMessage?: ProviderMessage; + /** `--provider-fault-message S1=interrupt_ack,...`: a workload's own message, over `providerFaultMessage`. */ + providerFaultMessageByWorkload?: Partial>; /** PV1 delays, cycled across tasks (`--provider-fault-delays`). */ providerFaultDelaysMs?: number[]; /** `--db-pressure `: harness-owned writers touching the wave's running run rows back to back (contention.ts). */ @@ -112,8 +116,10 @@ export interface WaveOptions { serverFrom?: string; /** TM2/TM3: how far every database timestamp is shifted back (`--time-age-hours`, default 26 h). */ timeAgeMs?: number; - /** `--provider-fault-restart K1`: each PV1/PV2 task also kills -9 the control plane after its fault applied. */ - providerFaultRestart?: "K1"; + /** `--provider-fault-restart K1`: each PV1/PV2 task also restarts the control plane after its fault applied. */ + providerFaultRestart?: ProviderRestartKind; + /** `--provider-fault-restart K2,K4,K1`: restart kinds cycled across PV1/PV2 tasks (the first is `providerFaultRestart`). */ + providerFaultRestartKinds?: ProviderRestartKind[]; providerFaultRestartAfterMs?: number; /** RR1–RR3: the gap from a replacement's health to the next stop (`--rr-gap-seconds`); absent keeps each kind's default. */ rrGapMs?: [number, number]; diff --git a/tests/restart-chaos/wave-speed.test.ts b/tests/restart-chaos/wave-speed.test.ts index 34a7a046f67..cd7d4ff7aeb 100644 --- a/tests/restart-chaos/wave-speed.test.ts +++ b/tests/restart-chaos/wave-speed.test.ts @@ -38,6 +38,10 @@ describe("wave speed grade", () => { events: [span("comment.to_run_created", 0, 4_000), span("heartbeat.queue", 4_000, 100), span("runner.turn.submit", 4_100, 900), span("task.time_to_first_agent_activity", 0, 9_000, { anchor: "comment" }), span("task.time_to_first_token", 0, 40_000, { anchor: "comment" })] }; expect(firstActivityMs(withActivity)).toBe(5_000); + // Per-run detail splits Paperclip's boot from the provider's wait for first activity. + expect(gradeWaveSpeed("local", [withActivity], [], 0).runs).toEqual([{ runId: "a", adapter: "codex", start: "cold", + firstActivityMs: 5_000, bootMs: 900, queueMs: 100, providerFirstActivityMs: 4_000, + preTurnSteps: { "heartbeat.queue": 100, "runner.turn.submit": 900 }, breach: false }]); expect(firstActivityMs(run("b", "codex", 7_000))).toBe(7_000); const speed = gradeWaveSpeed("local", [withActivity], [], 0); expect(speed.rows.find(row => row.metric === "time_to_first_activity")).toMatchObject({ breaches: [], measured: { p50: 5_000 } }); diff --git a/tests/restart-chaos/wave-speed.ts b/tests/restart-chaos/wave-speed.ts index ea351f52d1a..ce5f5fb76a6 100644 --- a/tests/restart-chaos/wave-speed.ts +++ b/tests/restart-chaos/wave-speed.ts @@ -31,6 +31,17 @@ export interface WaveSpeed { singleRunBreaches: number; breaches: number; errors: string[]; + /** + * Every graded (non-restart) run's own timing, so a breach can be attributed after the instance is + * gone: `bootMs` is Paperclip's share (claim to the provider accepting the turn), `providerFirstActivityMs` + * the provider's (turn accepted to first visible activity), and `preTurnSteps` the spans before the turn. + */ + runs?: SpeedRunDetail[]; +} + +export interface SpeedRunDetail { + runId: string; adapter: string; start: "cold" | "warm"; firstActivityMs: number | null; bootMs: number | null; + queueMs: number | null; providerFirstActivityMs: number | null; preTurnSteps: Record; breach: boolean; } const BUDGET_ADAPTER: Partial> = { codex: "codex", claude: "claude", opencode: "opencode" }; @@ -237,25 +248,45 @@ export function gradeWaveSpeed(environment: EnvironmentKind, runs: readonly Base rows.push({ adapter, metric, start: "restart", measured, budget, breaches: breachesOf(measured, budget) }); } } - const singleRunBreaches = runs.filter(run => run.phase !== "restart" && run.phase !== "verify").filter(run => { - const start = runWarmth(run); - const activity = firstActivityMs(run), boot = bootWithoutProvisionMs(run); - const activityBudget = budgetFor(environment, run.adapter, "time_to_first_activity", start); - const bootBudget = budgetFor(environment, run.adapter, "agent_boot", start); - return (activity !== null && activityBudget !== null && activity > activityBudget.p95Ms) - || (boot !== null && bootBudget !== null && boot > bootBudget.p95Ms); - }).length + samples.filter(sample => { + const singleRunBreaches = runs.filter(run => run.phase !== "restart" && run.phase !== "verify") + .filter(run => runOverBudget(environment, run)).length + samples.filter(sample => { const budget = budgetFor(environment, sample.adapter, "restart_recovery_control_plane", "restart"); return !sample.interruptedByFault && sample.controlPlaneMs !== null && budget !== null && sample.controlPlaneMs > budget.p95Ms; }).length; const breaches = rows.reduce((total, row) => total + row.breaches.length, 0) + singleRunBreaches; - return { environment, rows, + const details = runs.filter(run => run.phase !== "restart" && run.phase !== "verify").map(run => runDetail(environment, run)); + return { environment, rows, runs: details, recoveryWithoutProviderEvent: samples.filter(sample => sample.latencyMs === null && !sample.interruptedByFault).length, recoveryInterruptedByFault: samples.filter(sample => sample.interruptedByFault).length, runBudgetExceeded, singleRunBreaches, breaches, errors, outageAdjustedSpans: outage.adjusted, outageRemovedMs: outage.removedMs, storageFaultWindows: storageWindows.length, storageFaultRemovedMs: storageRemovedMs }; } +/** One cold or warm run over its own p95 budget (first activity or boot): a single-run breach. */ +function runOverBudget(environment: EnvironmentKind, run: BaselineRun): boolean { + const start = runWarmth(run); + const activity = firstActivityMs(run), boot = bootWithoutProvisionMs(run); + const activityBudget = budgetFor(environment, run.adapter, "time_to_first_activity", start); + const bootBudget = budgetFor(environment, run.adapter, "agent_boot", start); + return (activity !== null && activityBudget !== null && activity > activityBudget.p95Ms) + || (boot !== null && bootBudget !== null && boot > bootBudget.p95Ms); +} + +function runDetail(environment: EnvironmentKind, run: BaselineRun): SpeedRunDetail { + const start = runWarmth(run); + const metrics = runMetrics(run); + const spans = runSpans(run); + const queue = spans.find(span => span.name === "heartbeat.queue") ?? null; + const submitted = spans.find(span => span.name === "runner.turn.submit") ?? spans.find(span => span.name === "runner.session.resume") ?? null; + const activity = spans.find(span => span.name === "task.time_to_first_agent_activity") ?? null; + const turnAt = submitted ? submitted.startOffsetMs + submitted.durationMs : null; + const activityAt = activity ? activity.startOffsetMs + activity.durationMs : null; + return { runId: run.runId, adapter: run.adapter, start, firstActivityMs: firstActivityMs(run), bootMs: bootWithoutProvisionMs(run), queueMs: queue?.durationMs ?? null, + providerFirstActivityMs: turnAt !== null && activityAt !== null ? Math.max(0, activityAt - turnAt) : null, + preTurnSteps: metrics.preTurnSteps, + breach: runOverBudget(environment, run) }; +} + async function runEventsWithRetry(api: ChaosApi, runId: string, attempts = 3): Promise { let lastError: unknown; for (let attempt = 1; attempt <= attempts; attempt += 1) {