mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
test(restart-chaos): provider faults across K2/K4 restarts, per-workload message, per-run speed split (#20515)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos`) proves that runs recover when the control plane restarts > - PR #20494 fixed a saved governed wait whose provider terminal was late or lost across a restart. The harness proved it only with a K1 (kill -9) restart > - The harness could not restart with K2 (graceful SIGTERM) or K4 (outage hold) after a provider fault, so the graceful shutdown drain was not tested > - S1 needs the `interrupt_ack` message and Q1 needs `terminal`, but one wave could set only one message > - G1 was listed as a provider-fault workload, but its agent ends its own turn. No rule ever applies, and each G1 task waits until its deadline > - Speed breaches could not be explained after a wave, because the instance database is deleted > - This pull request adds K2/K4 provider-fault restarts, a message for each workload, removes G1, and keeps a speed split for each run > - The benefit is that one wave can test #20494 across graceful and held restarts. Also, a cold-start breach now shows whether Paperclip or the provider was slow ## Linked Issues or Issue Description Refs #20494 **What happened?** The harness accepted only `--provider-fault-restart K1`. A wave with `--workloads G1` and a provider fault planned tasks that could not apply the fault. Waves w-20261110T050407-s206, w-20261110T063253-s208 and w-20261110T085554-s211 show `applied=0` for G1 with Codex and Claude. The board then waited for a restart that did not come. **What did you expect?** A provider fault can cross every restart kind that the harness supports. The harness rejects a workload that does not interrupt a provider turn. **Steps to reproduce** Run `--adapters codex --workloads G1 --faults PV1 --provider-fault-restart K1 --concurrency 1`. The Codex shim logs `bound` and then `exited` with code 0. It logs no `interrupt_requested`. The task fails `R1:no_rest_within_budget`. ## What Changed - `--provider-fault-restart` accepts K1, K2 and K4. A comma list (for example `K4,K4,K2,K2`) cycles across PV1/PV2 tasks in plan order. K4 holds the control plane down for 120 s. - `--provider-fault-message` accepts `WORKLOAD=message` entries next to the default, for example `S1=interrupt_ack,terminal`. - G1 is removed from `PROVIDER_FAULT_WORKLOADS`. The config check that told G1 to add a restart is removed, because G1 is now rejected. - `speed.json` keeps a `runs` list. Each entry has `bootMs` (Paperclip share), `queueMs`, `providerFirstActivityMs` (turn accepted to first activity) and the pre-turn spans. - README documents the new flags, the G1 limit, and the speed split. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 434 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave w-20261110T092735-s213 (Codex, seed 213, concurrency 1). All 4 tasks pass R1–R6, T and `provider_fault_applied`: - t01 S1 PV2 drop `interrupt_ack` + K4 (120 s hold) - t02 Q1 PV1 delay `terminal` 18 s + K4 (120 s hold) - t03 S1 PV1 delay `interrupt_ack` 18 s + K2 - t04 Q1 PV2 drop `terminal` + K2 - That wave wrote the new `runs` split in `speed.json`. Cold boot was 1.3–1.7 s and provider first activity was 2.0–2.6 s. ## Risks - Low risk. Only the test harness changes. Product code does not change. - Old `plan.json` files keep `providerFaultRestart: "K1"`. That value is still valid, so replays of old waves do not change. - A wave that used G1 with provider faults now fails at parse time. Before, it ran and failed at the deadline. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed
1 parent
289b402c26
commit
c2423e30e8
10 files changed
+200
-58
No files matched your search
@@ -538,7 +538,9 @@ Codex app-server is late, missing or repeated. `--faults` takes them next to res
|
||||
governed wait (a saved question or approval) or a board Stop sends. `interrupt_ack` picks the
|
||||
acknowledgement of that interruption. Each rule applies once per task, to the first matching message.
|
||||
Provider faults run only with `--env local --adapters codex,claude` (either or both) and only with
|
||||
workloads that interrupt a turn (Q1, S1, G1). A PV1 task passes `provider_fault_applied` only when the
|
||||
workloads that interrupt a turn (Q1, S1). G1 does not: its agent asks for approval through `call_api`
|
||||
and ends its own turn, so no rule ever applied there (waves w-20261110T050407-s206, w-20261110T063253-s208,
|
||||
w-20261110T085554-s211) and the board waited for a restart that never came. A PV1 task passes `provider_fault_applied` only when the
|
||||
held message was released; a provider that exited first turned the delay into a drop.
|
||||
For Claude, the runner settles a cancelled turn about 3–4 s after the cancel without waiting for the
|
||||
provider's prompt response, and it later closes the sidecar with the provider. A Claude PV1 delay
|
||||
@@ -546,8 +548,7 @@ longer than that is therefore a drop (wave w-20261110T040202-s202: 18 s held, ne
|
||||
still passed its Q1 oracle). Sweep Claude PV1 below that window, for example
|
||||
`--provider-fault-delays 1,3`.
|
||||
|
||||
G1 grades the board decision after a restart, so G1 with a provider fault needs
|
||||
`--provider-fault-restart K1`. On a board Stop (S1) the runner ends Codex without waiting for its terminal:
|
||||
On a board Stop (S1) the runner ends Codex without waiting for its terminal:
|
||||
in wave w-20261110T050407-s206 Codex exited 1.5 s after `turn/interrupt` with no `turn/completed`, so a
|
||||
`terminal` rule never applies there and the task fails `provider_fault_applied`. Use `interrupt_ack` for S1.
|
||||
An S1 task with a provider fault clicks Stop on the first poll that shows a live provider turn
|
||||
@@ -570,10 +571,28 @@ after its fault applied, through the `provider` trigger. That delay must be shor
|
||||
so the message is still held, and the wave runs with `--concurrency 1`, because every K1 kills the whole control plane. K1 kills the runner with its provider and the held message, so the late
|
||||
or missing terminal straddles the restart: the replacement control plane must settle the saved wait
|
||||
without it. Planned restart faults (`--faults K1,PV2`) are still rejected, and PV3 does not combine with
|
||||
a restart, since a duplicate holds nothing. Such a task passes when:
|
||||
a restart, since a duplicate holds nothing.
|
||||
|
||||
- `provider_fault_applied`: the fault applied once, the K1 started after it, and for PV1 no release
|
||||
came before the K1;
|
||||
`--provider-fault-restart` also takes K2 (SIGTERM, a graceful restart whose shutdown drain meets the
|
||||
held or missing message) and K4 (kill -9, then the control plane stays down for 120 s, the low end of
|
||||
the seeded K4 range). For a restart, `provider_fault_applied` requires that the held message was not
|
||||
released before the old control plane stopped (`stoppedAt`), not only before the fault started. A K2 drain took 15–20 s in wave w-20261110T092735-s213, so a PV1 that must straddle a K2 needs a
|
||||
longer delay, for example `--provider-fault-delays 40`. A comma list such as `K1,K2,K4` cycles across the PV1/PV2 tasks in plan order,
|
||||
so one wave covers several restart kinds. `--provider-fault-message` takes `WORKLOAD=message` entries
|
||||
next to the default, for example `S1=interrupt_ack,terminal`: S1 tasks fault the interrupt
|
||||
acknowledgement (Codex ends a Stop without its terminal) while Q1 tasks fault the terminal.
|
||||
|
||||
```sh
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \
|
||||
--adapters codex --workloads Q1,S1 --faults PV1,PV2 --provider-fault-restart K2,K4 --provider-fault-delays 40 \
|
||||
--provider-fault-message S1=interrupt_ack,terminal --concurrency 1 --tasks 4 --seed 211
|
||||
```
|
||||
|
||||
Such a task passes when:
|
||||
|
||||
- `provider_fault_applied`: the fault applied once, the restart started after it, the held message was
|
||||
not released before the old control plane stopped, and it was not discarded before the restart started
|
||||
(a discard during a K2 shutdown is the restart stopping the provider);
|
||||
- the ordinary grade passes: R1–R6 for a restart-crossing task and the workload's T-oracle. For Q1
|
||||
that means one question, the asking run `succeeded`, no failed run, nothing started before the
|
||||
answer, and the brief carries the answer.
|
||||
@@ -942,7 +961,11 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
- Speed: an aggregate p50/p95 row can breach only with at least 5 samples; a
|
||||
smaller row is reported as `insufficient sample (n < 5)`. Every cold or warm run
|
||||
over its p95 budget, and every restart recovery whose Paperclip share exceeds
|
||||
its p95 budget, still fails the wave on its own.
|
||||
its p95 budget, still fails the wave on its own. `speed.json` keeps each graded
|
||||
run's own split (`runs`): `bootMs` (Paperclip: claim to the provider accepting
|
||||
the turn), `providerFirstActivityMs` (turn accepted to first visible activity),
|
||||
and the spans before the turn, so a breach stays attributable after the
|
||||
temporary instance is deleted.
|
||||
- Faults are serialized; each one hits every task running at that moment.
|
||||
- A `todo` or `in_progress` task with no live run and no named wait is not a rest. The observer keeps watching it until it resumes or passes the R5 stall limit.
|
||||
- R5 also fails a run that stays `running` with no live evidence (no new run event, `lastUsefulActionAt` or `execution.lastConfirmedActivityAt`, and no start) for more than 3 minutes after the server is ready. A `running` status alone is not proof of life. A quiet run is still live while a tool it started has not ended (for example a 4-minute `bench.sh`) or while its recorded local runner process still runs with the same PID and start time.
|
||||
|
||||
@@ -235,14 +235,31 @@ describe("provider faults across a restart", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("cycles K1, K2 and K4 restarts across PV tasks and gives each workload its own message", () => {
|
||||
const command = parseCommand(["--adapters", "codex", "--workloads", "S1,Q1", "--faults", "PV1,PV2", "--provider-fault-restart", "K2,K4,K1",
|
||||
"--provider-fault-message", "S1=interrupt_ack,terminal", "--tasks", "4", "--concurrency", "1", "--seed", "9"]);
|
||||
if (command.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(command.options).toMatchObject({ providerFaultRestart: "K2", providerFaultRestartKinds: ["K2", "K4", "K1"],
|
||||
providerFaultMessage: "terminal", providerFaultMessageByWorkload: { S1: "interrupt_ack" } });
|
||||
const plan = planWave("w-20261110T000000-s9", command.options);
|
||||
expect(plan.tasks.map(planned => planned.providerFault?.restart?.kind)).toEqual(["K2", "K4", "K1", "K2"]);
|
||||
for (const planned of plan.tasks) {
|
||||
expect(planned.providerFault?.message).toBe(planned.workload === "S1" ? "interrupt_ack" : "terminal");
|
||||
const restart = planned.providerFault!.restart!;
|
||||
expect(taskFaultSchedule(planned)).toEqual([{ kind: restart.kind, holdMs: restart.kind === "K4" ? 120_000 : 0, atMs: 0, trigger: "provider", delayMs: 3_000 }]);
|
||||
}
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-message", "Q2=terminal"])).toThrow(/not one of/);
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-message", "terminal,interrupt_ack"])).toThrow(/one default/);
|
||||
});
|
||||
|
||||
it("rejects restarts that cannot straddle the fault", () => {
|
||||
expect(() => parseCommand(["--adapters", "claude", "--workloads", "Q1", "--faults", "PV3", "--provider-fault-restart", "K1"])).toThrow(/PV1 or PV2/);
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV1", "--provider-fault-delays", "5",
|
||||
"--provider-fault-restart", "K1", "--provider-fault-restart-after-seconds", "6"])).toThrow(/shorter than every PV1 delay/);
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-restart", "K2"])).toThrow(/K1/);
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-restart", "K6"])).toThrow(/K1, K2, K4/);
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "K1", "--provider-fault-restart", "K1"])).toThrow(/provider fault/);
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "G1", "--faults", "PV2"])).toThrow(/G1 grades the board decision after a restart/);
|
||||
expect(parseCommand(["--adapters", "codex", "--workloads", "G1", "--faults", "PV2", "--provider-fault-restart", "K1", "--concurrency", "1"]).mode).toBe("wave");
|
||||
// G1's agent ends its own turn after asking for approval: no interrupt, so no provider rule can apply.
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "G1", "--faults", "PV2", "--provider-fault-restart", "K1", "--concurrency", "1"])).toThrow(/remove G1/);
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q1", "--faults", "PV2", "--provider-fault-restart", "K1", "--concurrency", "2"])).toThrow(/--concurrency 1/);
|
||||
expect(parseProviderFaultRestart(undefined, undefined, ["PV2"], undefined)).toEqual({});
|
||||
expect(() => parseProviderFaultRestart(undefined, "3", ["PV2"], undefined)).toThrow(/needs --provider-fault-restart/);
|
||||
@@ -257,6 +274,17 @@ describe("provider faults across a restart", () => {
|
||||
expect(withProviderFaultCheck(pass, task("aa", rule), [applied, { ...applied, event: "released", at: 2_000 }], 4_000).checks.at(-1))
|
||||
.toMatchObject({ ok: false, detail: expect.stringContaining("releasedBeforeRestart") });
|
||||
expect(withProviderFaultCheck(pass, task("aa", rule), [applied], 500).verdict).toBe("fail");
|
||||
// K2: a release inside the graceful drain (after the SIGTERM, before the old process stopped) reached
|
||||
// the old control plane, so the message straddled nothing.
|
||||
const k2: ProviderFaultRule = { ...rule, restart: { kind: "K2", afterAppliedMs: 3_000 } };
|
||||
expect(withProviderFaultCheck(pass, task("aa", k2), [applied, { ...applied, event: "released", at: 10_000 }], 4_000, 20_000).checks.at(-1))
|
||||
.toMatchObject({ ok: false, detail: expect.stringContaining("releasedBeforeRestart") });
|
||||
expect(withProviderFaultCheck(pass, task("aa", k2), [applied, { ...applied, event: "released", at: 25_000 }], 4_000, 20_000).verdict).toBe("pass");
|
||||
// A discard inside the K2 shutdown is its cleanup stopping the provider with the message still held:
|
||||
// that straddles. Only a provider that exited before the restart started took the message first.
|
||||
expect(withProviderFaultCheck(pass, task("aa", k2), [applied, { ...applied, event: "held_message_discarded", at: 10_000 }], 4_000, 20_000).verdict).toBe("pass");
|
||||
expect(withProviderFaultCheck(pass, task("aa", k2), [applied, { ...applied, event: "held_message_discarded", at: 2_000 }], 4_000, 20_000).checks.at(-1))
|
||||
.toMatchObject({ ok: false, detail: expect.stringContaining("discardedBeforeRestart") });
|
||||
// A provider that closed first took the held message with it before the K1: nothing straddled the restart.
|
||||
expect(withProviderFaultCheck(pass, task("aa", rule), [applied, { ...applied, event: "held_message_discarded", at: 3_000 }], 4_000).checks.at(-1))
|
||||
.toMatchObject({ ok: false, detail: expect.stringContaining("discardedBeforeRestart") });
|
||||
|
||||
@@ -27,7 +27,7 @@
|
||||
import { chmod, mkdir, readdir, realpath, rm, symlink, writeFile } from "node:fs/promises";
|
||||
import { existsSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { DEFAULT_PROVIDER_DELAY_MS, DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS, type ProviderFaultKind } from "./provider-faults.js";
|
||||
import { DEFAULT_PROVIDER_DELAY_MS, DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS, PROVIDER_RESTART_KINDS, type ProviderFaultKind, type ProviderRestartKind } from "./provider-faults.js";
|
||||
import type { AdapterId, WaveOptions } from "./types.js";
|
||||
|
||||
export const ACPX_SIDECAR_HOOK = path.join(import.meta.dirname, "provider-shim", "acpx-sidecar-hook.mjs");
|
||||
@@ -119,12 +119,15 @@ export function withAcpxSidecarHook(nodeOptions: string | undefined): string {
|
||||
* not PV3: a duplicate has nothing in flight for a restart to cross.
|
||||
*/
|
||||
export function parseProviderFaultRestart(raw: string | undefined, afterRaw: string | undefined, kinds: readonly ProviderFaultKind[],
|
||||
delaysMs: readonly number[] | undefined): Pick<WaveOptions, "providerFaultRestart" | "providerFaultRestartAfterMs"> {
|
||||
delaysMs: readonly number[] | undefined): Pick<WaveOptions, "providerFaultRestart" | "providerFaultRestartAfterMs" | "providerFaultRestartKinds"> {
|
||||
if (raw === undefined) {
|
||||
if (afterRaw !== undefined) throw new Error("--provider-fault-restart-after-seconds needs --provider-fault-restart K1");
|
||||
if (afterRaw !== undefined) throw new Error("--provider-fault-restart-after-seconds needs --provider-fault-restart (K1, K2 or K4)");
|
||||
return {};
|
||||
}
|
||||
if (raw !== "K1") throw new Error("--provider-fault-restart supports K1 (kill -9 the control plane) only");
|
||||
const restartKinds = raw.split(",").map(value => value.trim()).filter(Boolean) as ProviderRestartKind[];
|
||||
if (!restartKinds.length || restartKinds.some(kind => !(PROVIDER_RESTART_KINDS as readonly string[]).includes(kind))) {
|
||||
throw new Error(`--provider-fault-restart takes ${PROVIDER_RESTART_KINDS.join(", ")} (comma-separated kinds cycle across PV1/PV2 tasks)`);
|
||||
}
|
||||
if (kinds.includes("PV3")) throw new Error("--provider-fault-restart combines with PV1 or PV2; a PV3 duplicate holds nothing across a restart");
|
||||
let afterMs = DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS;
|
||||
if (afterRaw !== undefined) {
|
||||
@@ -137,5 +140,6 @@ export function parseProviderFaultRestart(raw: string | undefined, afterRaw: str
|
||||
if (shortest !== null && afterMs >= shortest) {
|
||||
throw new Error(`--provider-fault-restart-after-seconds (${afterMs / 1000}s) must be shorter than every PV1 delay (shortest ${shortest / 1000}s)`);
|
||||
}
|
||||
return { providerFaultRestart: "K1", providerFaultRestartAfterMs: afterMs };
|
||||
return { providerFaultRestart: restartKinds[0]!, providerFaultRestartAfterMs: afterMs,
|
||||
...(restartKinds.length > 1 ? { providerFaultRestartKinds: restartKinds } : {}) };
|
||||
}
|
||||
@@ -7,7 +7,7 @@ import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js";
|
||||
import { parseTimeFaultOptions } from "./time-faults-config.js";
|
||||
import { parseStorageTrigger } from "./storage-trigger.js";
|
||||
import { parseRepeatedRestartOptions } from "./repeated-restarts.js";
|
||||
import { PROVIDER_FAULT_KINDS, PROVIDER_FAULT_WORKLOADS, isProviderFault, parseProviderDelays, parseProviderMessage, type ProviderFaultKind } from "./provider-faults.js";
|
||||
import { PROVIDER_FAULT_KINDS, PROVIDER_FAULT_WORKLOADS, isProviderFault, parseProviderDelays, parseProviderMessages, type ProviderFaultKind } from "./provider-faults.js";
|
||||
import { PROVIDER_FAULT_ADAPTERS, parseProviderFaultRestart } from "./acpx-provider-faults.js";
|
||||
import {
|
||||
ADAPTER_IDS, ENVIRONMENT_KINDS, FAULT_KINDS, TRIGGER_IDS, WORKLOAD_IDS,
|
||||
@@ -209,17 +209,16 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
// Every planned task gets the fault; only these workloads send the turn/interrupt the rules target.
|
||||
const idle = options.workloads.filter(id => !PROVIDER_FAULT_WORKLOADS.includes(id));
|
||||
if (idle.length) throw new Error(`Provider message faults (PV*) need workloads that interrupt a provider turn (${PROVIDER_FAULT_WORKLOADS.join(", ")}); remove ${idle.join(", ")} from --workloads`);
|
||||
const message = parseProviderMessage(values["provider-fault-message"]);
|
||||
if (message) options.providerFaultMessage = message;
|
||||
const messages = parseProviderMessages(values["provider-fault-message"]);
|
||||
if (messages.message) options.providerFaultMessage = messages.message;
|
||||
if (messages.byWorkload) options.providerFaultMessageByWorkload = messages.byWorkload;
|
||||
const delays = parseProviderDelays(values["provider-fault-delays"]);
|
||||
if (delays && !providerFaults.includes("PV1")) throw new Error("--provider-fault-delays sets PV1 delays; add PV1 to --faults");
|
||||
if (delays) options.providerFaultDelaysMs = delays;
|
||||
Object.assign(options, parseProviderFaultRestart(values["provider-fault-restart"], values["provider-fault-restart-after-seconds"], providerFaults, delays));
|
||||
// Each K1 kills the whole control plane: a second task in flight would lose its held message to the
|
||||
// first task's restart and miss its own, so a restart wave runs one task at a time.
|
||||
if (options.providerFaultRestart && options.concurrency !== 1) throw new Error("--provider-fault-restart kills the whole control plane per task; run it with --concurrency 1");
|
||||
// G1's oracle requires the board's decision to come after a restart (board_decided_after_restart).
|
||||
if (options.workloads.includes("G1") && !options.providerFaultRestart) throw new Error("G1 grades the board decision after a restart; with provider faults add --provider-fault-restart K1");
|
||||
if (options.providerFaultRestart && options.concurrency !== 1) throw new Error("--provider-fault-restart restarts the whole control plane per task; run it with --concurrency 1");
|
||||
} else if (values["provider-fault-message"] !== undefined || values["provider-fault-delays"] !== undefined
|
||||
|| values["provider-fault-restart"] !== undefined || values["provider-fault-restart-after-seconds"] !== undefined) {
|
||||
throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults");
|
||||
|
||||
@@ -611,7 +611,8 @@ async function main(): Promise<number> {
|
||||
: { verdict: "fail" as const, checks: [{ name: "artifacts_collected", ok: false, detail: observer.record.launchError ?? "collection failed" }] };
|
||||
// A provider-fault task whose fault never happened tested nothing.
|
||||
const restart = faults.find(receipt => receipt.requestedBy === observer.plan.taskId && receipt.trigger === "provider");
|
||||
const oracle = withProviderFaultCheck(graded, observer.plan, artifacts?.providerFaultEvents ?? [], restart?.startedAt ?? null);
|
||||
const oracle = withProviderFaultCheck(graded, observer.plan, artifacts?.providerFaultEvents ?? [], restart?.startedAt ?? null,
|
||||
restart?.stoppedAt ?? restart?.startedAt ?? null);
|
||||
const grade = gradeTask(observer.record, faults, observer.definition.budgetMs, oracle, universalByIssue ? {
|
||||
runMessages: allRunMessages, recoveryActions: universalByIssue.get(observer.record.issueId ?? "")?.recoveryActions ?? [] } : undefined);
|
||||
let evidence: string | null = null;
|
||||
|
||||
@@ -19,7 +19,7 @@ import { chmod, copyFile, mkdir, readFile, rename, symlink, writeFile } from "no
|
||||
import { readFileSync, realpathSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import path from "node:path";
|
||||
import type { OracleResult, TaskPlan, WaveOptions, WorkloadId } from "./types.js";
|
||||
import { K4_MIN_HOLD_MS, type OracleResult, type TaskPlan, type WaveOptions, type WorkloadId } from "./types.js";
|
||||
|
||||
export const PROVIDER_FAULT_KINDS = ["PV1", "PV2", "PV3"] as const;
|
||||
export type ProviderFaultKind = (typeof PROVIDER_FAULT_KINDS)[number];
|
||||
@@ -29,6 +29,12 @@ export const PROVIDER_FAULT_ACTIONS: Record<ProviderFaultKind, "delay" | "drop"
|
||||
export const PROVIDER_MESSAGES = ["terminal", "interrupt_ack"] as const;
|
||||
export type ProviderMessage = (typeof PROVIDER_MESSAGES)[number];
|
||||
|
||||
/** Restarts a provider fault may straddle: K1 kill -9, K2 SIGTERM graceful restart, K4 kill -9 then an outage hold. */
|
||||
export const PROVIDER_RESTART_KINDS = ["K1", "K2", "K4"] as const;
|
||||
export type ProviderRestartKind = (typeof PROVIDER_RESTART_KINDS)[number];
|
||||
/** K4 outage hold after a provider fault: the low end of the seeded K4 range. */
|
||||
export const PROVIDER_FAULT_K4_HOLD_MS = K4_MIN_HOLD_MS;
|
||||
|
||||
/** PV1 default: the late terminal of PR #20330 (the old 15 s governed-wait drain against Codex's 30 s command window). */
|
||||
export const DEFAULT_PROVIDER_DELAY_MS = 18_000;
|
||||
/** Bound on a PV1 delay: beyond 10 minutes a message is effectively dropped (PV2). */
|
||||
@@ -40,13 +46,13 @@ export interface ProviderFaultRule {
|
||||
/** PV1 only. */
|
||||
delayMs?: number;
|
||||
/**
|
||||
* `--provider-fault-restart K1` (PV1, PV2): kill -9 the control plane this long after the fault
|
||||
* applied, so the held or dropped message straddles a restart.
|
||||
* `--provider-fault-restart K1|K2|K4` (PV1, PV2): restart the control plane this long after the fault
|
||||
* applied, so the held or dropped message straddles a restart. `holdMs` is the K4 outage.
|
||||
*/
|
||||
restart?: { kind: "K1"; afterAppliedMs: number };
|
||||
restart?: { kind: ProviderRestartKind; afterAppliedMs: number; holdMs?: number };
|
||||
}
|
||||
|
||||
/** Default gap between a provider fault applying and its K1: well inside the default 18 s PV1 hold. */
|
||||
/** Default gap between a provider fault applying and its restart: well inside the default 18 s PV1 hold. */
|
||||
export const DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS = 3_000;
|
||||
|
||||
/** One shim log line (JSONL, `provider-faults.jsonl` in the wave directory). */
|
||||
@@ -63,8 +69,13 @@ export interface ProviderFaultEvent {
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
/** Workloads whose run sends turn/interrupt: a saved question (Q1), a board Stop (S1), an approval wait (G1). */
|
||||
export const PROVIDER_FAULT_WORKLOADS: readonly WorkloadId[] = ["Q1", "S1", "G1"];
|
||||
/**
|
||||
* Workloads whose run sends turn/interrupt: a saved question (Q1) and a board Stop (S1). Not G1: its agent
|
||||
* asks for approval through `call_api` and ends its own turn, so nothing interrupts the provider and no
|
||||
* rule ever applies (waves w-20261110T050407-s206, w-20261110T063253-s208 and w-20261110T085554-s211:
|
||||
* applied=0 for Codex and Claude, and the board then waits for a restart that never comes).
|
||||
*/
|
||||
export const PROVIDER_FAULT_WORKLOADS: readonly WorkloadId[] = ["Q1", "S1"];
|
||||
|
||||
export function isProviderFault(value: string): value is ProviderFaultKind {
|
||||
return (PROVIDER_FAULT_KINDS as readonly string[]).includes(value);
|
||||
@@ -87,29 +98,57 @@ export function parseProviderDelays(raw: string | undefined): number[] | undefin
|
||||
}
|
||||
|
||||
export function parseProviderMessage(raw: string | undefined): ProviderMessage | undefined {
|
||||
if (raw === undefined) return undefined;
|
||||
if (!(PROVIDER_MESSAGES as readonly string[]).includes(raw)) throw new Error(`--provider-fault-message must be one of ${PROVIDER_MESSAGES.join(", ")}`);
|
||||
return raw as ProviderMessage;
|
||||
return parseProviderMessages(raw).message;
|
||||
}
|
||||
|
||||
/**
|
||||
* `--provider-fault-message terminal` or `--provider-fault-message S1=interrupt_ack,terminal`: a bare
|
||||
* message is the wave default; `WORKLOAD=message` overrides it for that workload's tasks (Codex ends a
|
||||
* Stop without its terminal, so S1 needs `interrupt_ack` while Q1 keeps `terminal`).
|
||||
*/
|
||||
export function parseProviderMessages(raw: string | undefined): { message?: ProviderMessage; byWorkload?: Partial<Record<WorkloadId, ProviderMessage>> } {
|
||||
if (raw === undefined) return {};
|
||||
const result: { message?: ProviderMessage; byWorkload?: Partial<Record<WorkloadId, ProviderMessage>> } = {};
|
||||
const entries = raw.split(",").map(value => value.trim()).filter(Boolean);
|
||||
if (!entries.length) throw new Error(`--provider-fault-message must be one of ${PROVIDER_MESSAGES.join(", ")}`);
|
||||
for (const entry of entries) {
|
||||
const [left, right] = entry.includes("=") ? entry.split("=", 2) as [string, string] : [undefined, entry];
|
||||
if (!(PROVIDER_MESSAGES as readonly string[]).includes(right)) throw new Error(`--provider-fault-message must be one of ${PROVIDER_MESSAGES.join(", ")}`);
|
||||
if (left === undefined) {
|
||||
if (result.message) throw new Error("--provider-fault-message takes one default message");
|
||||
result.message = right as ProviderMessage;
|
||||
} else {
|
||||
if (!PROVIDER_FAULT_WORKLOADS.includes(left as WorkloadId)) throw new Error(`--provider-fault-message: ${left} is not one of ${PROVIDER_FAULT_WORKLOADS.join(", ")}`);
|
||||
result.byWorkload = { ...result.byWorkload, [left]: right as ProviderMessage };
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Attach each provider-fault task's rule. Restart-fault tasks are unchanged. */
|
||||
export function withProviderFaultRules(tasks: TaskPlan[], options: Pick<WaveOptions, "providerFaultMessage" | "providerFaultDelaysMs"
|
||||
| "providerFaultRestart" | "providerFaultRestartAfterMs">): TaskPlan[] {
|
||||
| "providerFaultRestart" | "providerFaultRestartAfterMs" | "providerFaultRestartKinds" | "providerFaultMessageByWorkload">): TaskPlan[] {
|
||||
const delays = options.providerFaultDelaysMs?.length ? options.providerFaultDelaysMs : [DEFAULT_PROVIDER_DELAY_MS];
|
||||
const restarts = options.providerFaultRestartKinds?.length ? options.providerFaultRestartKinds
|
||||
: options.providerFaultRestart ? [options.providerFaultRestart] : [];
|
||||
let pv1 = 0;
|
||||
let restarted = 0;
|
||||
return tasks.map(task => {
|
||||
if (!isProviderFault(task.fault)) return task;
|
||||
const rule: ProviderFaultRule = { kind: task.fault, message: options.providerFaultMessage ?? "terminal",
|
||||
const restartKind = restarts.length && task.fault !== "PV3" ? restarts[restarted++ % restarts.length]! : null;
|
||||
const rule: ProviderFaultRule = { kind: task.fault,
|
||||
message: options.providerFaultMessageByWorkload?.[task.workload] ?? options.providerFaultMessage ?? "terminal",
|
||||
...(task.fault === "PV1" ? { delayMs: delays[pv1++ % delays.length]! } : {}),
|
||||
...(options.providerFaultRestart && task.fault !== "PV3"
|
||||
? { restart: { kind: options.providerFaultRestart, afterAppliedMs: options.providerFaultRestartAfterMs ?? DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS } } : {}) };
|
||||
...(restartKind
|
||||
? { restart: { kind: restartKind, afterAppliedMs: options.providerFaultRestartAfterMs ?? DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS,
|
||||
...(restartKind === "K4" ? { holdMs: PROVIDER_FAULT_K4_HOLD_MS } : {}) } } : {}) };
|
||||
return { ...task, providerFault: rule, fireAfterMs: 0, holdMs: 0 };
|
||||
});
|
||||
}
|
||||
|
||||
export function describeProviderFault(rule: ProviderFaultRule): string {
|
||||
return `${rule.kind} ${PROVIDER_FAULT_ACTIONS[rule.kind]} ${rule.message}${rule.delayMs !== undefined ? ` ${rule.delayMs}ms` : ""}`
|
||||
+ (rule.restart ? ` +${rule.restart.kind}@${rule.restart.afterAppliedMs}ms` : "");
|
||||
+ (rule.restart ? ` +${rule.restart.kind}@${rule.restart.afterAppliedMs}ms${rule.restart.holdMs ? `/hold${rule.restart.holdMs}ms` : ""}` : "");
|
||||
}
|
||||
|
||||
/** The Codex executable product code would resolve for this checkout (mirrors codex-command.ts). */
|
||||
@@ -211,13 +250,17 @@ export async function readProviderFaultEvents(logPath: string, nonce?: string):
|
||||
* A provider-fault task passes only if its fault really happened: otherwise the scenario was not
|
||||
* exercised (for example the turn ended before any interrupt), and a pass would prove nothing.
|
||||
*
|
||||
* With `--provider-fault-restart K1` the fault must also straddle the restart: the K1 started after
|
||||
* the fault applied and, for PV1, before the held message was released or discarded. The K1 kills the runner
|
||||
* with the held message, so a PV1 task then needs no release. Whether the task itself recovered
|
||||
* With `--provider-fault-restart` the fault must also straddle the restart: the restart started after
|
||||
* the fault applied, the held message was not released before the old control plane had stopped
|
||||
* (`restartStoppedAt`), and it was not discarded before the restart started. A K2 drains gracefully
|
||||
* between its start and its stop, so a message released in that drain reached the old process and
|
||||
* straddled nothing. A discard after the start is the restart's own cleanup stopping the provider with
|
||||
* the message still held, which is the straddle. The kill takes the held message with the runner, so a
|
||||
* PV1 task then needs no release. Whether the task itself recovered
|
||||
* is the ordinary grade (R1–R6 and its T-oracle), which this check never replaces.
|
||||
*/
|
||||
export function withProviderFaultCheck(oracle: OracleResult, plan: TaskPlan, events: readonly ProviderFaultEvent[],
|
||||
restartStartedAt: number | null = null): OracleResult {
|
||||
restartStartedAt: number | null = null, restartStoppedAt: number | null = restartStartedAt): OracleResult {
|
||||
if (!plan.providerFault) return oracle;
|
||||
const rule = plan.providerFault;
|
||||
const applied = events.filter(event => event.event === "applied" && event.kind === rule.kind && event.message === rule.message);
|
||||
@@ -229,8 +272,11 @@ export function withProviderFaultCheck(oracle: OracleResult, plan: TaskPlan, eve
|
||||
let restartDetail = "";
|
||||
if (rule.restart) {
|
||||
const appliedAt = applied[0]?.at ?? null;
|
||||
// Released or discarded before the K1: the message was already gone, so nothing straddled the restart.
|
||||
const releasedBeforeRestart = restartStartedAt !== null && released.some(event => event.at < restartStartedAt);
|
||||
// Released before the old control plane stopped (a K2 drain included): the old process got the
|
||||
// message, so nothing straddled the restart. Discarded before the restart started: the provider
|
||||
// exited on its own first. A discard after the start is the restart's cleanup and still straddles.
|
||||
const stoppedAt = restartStoppedAt ?? restartStartedAt;
|
||||
const releasedBeforeRestart = stoppedAt !== null && released.some(event => event.at < stoppedAt);
|
||||
const discardedBeforeRestart = restartStartedAt !== null && discarded.some(event => event.at < restartStartedAt);
|
||||
exercised = applied.length === 1 && appliedAt !== null && restartStartedAt !== null && restartStartedAt >= appliedAt
|
||||
&& !releasedBeforeRestart && !discardedBeforeRestart;
|
||||
|
||||
@@ -3,7 +3,7 @@ import { takesJourneyTriggers } from "./journeys.js";
|
||||
import { workloadDefinitions } from "./workloads.js";
|
||||
import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js";
|
||||
import { isRepeatedRestartFault, planRestartSteps, type RestartStep } from "./repeated-restarts.js";
|
||||
import type { AdapterId, FaultKind, ScheduledFault, StormFault, StormPlan, TaskFaultKind, TaskPlan, TriggerId, WavePlan, WaveOptions, WorkloadId } from "./types.js";
|
||||
import { K4_MIN_HOLD_MS, type AdapterId, type FaultKind, type ScheduledFault, type StormFault, type StormPlan, type TaskFaultKind, type TaskPlan, type TriggerId, type WavePlan, type WaveOptions, type WorkloadId } from "./types.js";
|
||||
import type { ReplayRef } from "./config.js";
|
||||
|
||||
/** mulberry32: small, fast and fully determined by its 32-bit seed. */
|
||||
@@ -31,7 +31,7 @@ export function createRng(seed: number) {
|
||||
};
|
||||
}
|
||||
|
||||
export const K4_HOLD_RANGE_MS: readonly [number, number] = [120_000, 300_000];
|
||||
export const K4_HOLD_RANGE_MS: readonly [number, number] = [K4_MIN_HOLD_MS, 300_000];
|
||||
|
||||
/** The seeded K4 hold range; `--k4-hold-max-seconds` raises its upper end
|
||||
* (up to the 600 s controller-outage cap) to reach the 7-minute reconnect
|
||||
@@ -290,7 +290,7 @@ export function taskFaultSchedule(task: TaskPlan): ScheduledFault[] {
|
||||
// A storm crosses K13 tasks; a provider fault (PV*) and a matrix baseline (K0) restart nothing, unless
|
||||
// `--provider-fault-restart` asked for a K1 once the provider fault applied (the `provider` trigger).
|
||||
const restart = task.providerFault?.restart;
|
||||
if (restart) return [{ kind: restart.kind, holdMs: 0, atMs: 0, trigger: "provider", delayMs: restart.afterAppliedMs }];
|
||||
if (restart) return [{ kind: restart.kind, holdMs: restart.holdMs ?? 0, atMs: 0, trigger: "provider", delayMs: restart.afterAppliedMs }];
|
||||
if (task.fault === "K13" || task.fault === "K0" || isProviderFault(task.fault)) return [];
|
||||
const planned: ScheduledFault = { kind: task.fault, holdMs: task.holdMs, atMs: task.fireAfterMs,
|
||||
...(task.restartSteps ? { restartSteps: task.restartSteps } : {}) };
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
/** Shared types for the control-plane restart chaos harness (plan §5). */
|
||||
import type { ProviderFaultEvent, ProviderFaultKind, ProviderFaultRule, ProviderMessage } from "./provider-faults.js";
|
||||
import type { ProviderFaultEvent, ProviderFaultKind, ProviderFaultRule, ProviderMessage, ProviderRestartKind } from "./provider-faults.js";
|
||||
import type { QuestionArtifacts, QuestionRecord } from "./question-handoff.js";
|
||||
import type { RepeatedRestartRecord, RestartStep } from "./repeated-restarts.js";
|
||||
import type { TimeFaultRecord } from "./time-faults.js";
|
||||
@@ -11,6 +11,8 @@ import type { TimeFaultRecord } from "./time-faults.js";
|
||||
*/
|
||||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3"] as const;
|
||||
export type FaultKind = (typeof FAULT_KINDS)[number];
|
||||
/** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */
|
||||
export const K4_MIN_HOLD_MS = 120_000;
|
||||
/**
|
||||
* A task's planned fault. "K13" (restart storm) tasks have none of their own; the wave's storm crosses them.
|
||||
* PV1–PV3 (provider message faults, provider-faults.ts) restart nothing: the task's rule is armed in the Codex shim.
|
||||
@@ -102,6 +104,8 @@ export interface WaveOptions {
|
||||
/** Provider message faults from `--faults` (PV1 delay, PV2 drop, PV3 duplicate); see provider-faults.ts. */
|
||||
providerFaults?: ProviderFaultKind[];
|
||||
providerFaultMessage?: ProviderMessage;
|
||||
/** `--provider-fault-message S1=interrupt_ack,...`: a workload's own message, over `providerFaultMessage`. */
|
||||
providerFaultMessageByWorkload?: Partial<Record<WorkloadId, ProviderMessage>>;
|
||||
/** PV1 delays, cycled across tasks (`--provider-fault-delays`). */
|
||||
providerFaultDelaysMs?: number[];
|
||||
/** `--db-pressure <writers>`: harness-owned writers touching the wave's running run rows back to back (contention.ts). */
|
||||
@@ -112,8 +116,10 @@ export interface WaveOptions {
|
||||
serverFrom?: string;
|
||||
/** TM2/TM3: how far every database timestamp is shifted back (`--time-age-hours`, default 26 h). */
|
||||
timeAgeMs?: number;
|
||||
/** `--provider-fault-restart K1`: each PV1/PV2 task also kills -9 the control plane after its fault applied. */
|
||||
providerFaultRestart?: "K1";
|
||||
/** `--provider-fault-restart K1`: each PV1/PV2 task also restarts the control plane after its fault applied. */
|
||||
providerFaultRestart?: ProviderRestartKind;
|
||||
/** `--provider-fault-restart K2,K4,K1`: restart kinds cycled across PV1/PV2 tasks (the first is `providerFaultRestart`). */
|
||||
providerFaultRestartKinds?: ProviderRestartKind[];
|
||||
providerFaultRestartAfterMs?: number;
|
||||
/** RR1–RR3: the gap from a replacement's health to the next stop (`--rr-gap-seconds`); absent keeps each kind's default. */
|
||||
rrGapMs?: [number, number];
|
||||
|
||||
@@ -38,6 +38,10 @@ describe("wave speed grade", () => {
|
||||
events: [span("comment.to_run_created", 0, 4_000), span("heartbeat.queue", 4_000, 100), span("runner.turn.submit", 4_100, 900),
|
||||
span("task.time_to_first_agent_activity", 0, 9_000, { anchor: "comment" }), span("task.time_to_first_token", 0, 40_000, { anchor: "comment" })] };
|
||||
expect(firstActivityMs(withActivity)).toBe(5_000);
|
||||
// Per-run detail splits Paperclip's boot from the provider's wait for first activity.
|
||||
expect(gradeWaveSpeed("local", [withActivity], [], 0).runs).toEqual([{ runId: "a", adapter: "codex", start: "cold",
|
||||
firstActivityMs: 5_000, bootMs: 900, queueMs: 100, providerFirstActivityMs: 4_000,
|
||||
preTurnSteps: { "heartbeat.queue": 100, "runner.turn.submit": 900 }, breach: false }]);
|
||||
expect(firstActivityMs(run("b", "codex", 7_000))).toBe(7_000);
|
||||
const speed = gradeWaveSpeed("local", [withActivity], [], 0);
|
||||
expect(speed.rows.find(row => row.metric === "time_to_first_activity")).toMatchObject({ breaches: [], measured: { p50: 5_000 } });
|
||||
|
||||
@@ -31,6 +31,17 @@ export interface WaveSpeed {
|
||||
singleRunBreaches: number;
|
||||
breaches: number;
|
||||
errors: string[];
|
||||
/**
|
||||
* Every graded (non-restart) run's own timing, so a breach can be attributed after the instance is
|
||||
* gone: `bootMs` is Paperclip's share (claim to the provider accepting the turn), `providerFirstActivityMs`
|
||||
* the provider's (turn accepted to first visible activity), and `preTurnSteps` the spans before the turn.
|
||||
*/
|
||||
runs?: SpeedRunDetail[];
|
||||
}
|
||||
|
||||
export interface SpeedRunDetail {
|
||||
runId: string; adapter: string; start: "cold" | "warm"; firstActivityMs: number | null; bootMs: number | null;
|
||||
queueMs: number | null; providerFirstActivityMs: number | null; preTurnSteps: Record<string, number>; breach: boolean;
|
||||
}
|
||||
|
||||
const BUDGET_ADAPTER: Partial<Record<AdapterId, SpeedBudgetAdapter>> = { codex: "codex", claude: "claude", opencode: "opencode" };
|
||||
@@ -237,25 +248,45 @@ export function gradeWaveSpeed(environment: EnvironmentKind, runs: readonly Base
|
||||
rows.push({ adapter, metric, start: "restart", measured, budget, breaches: breachesOf(measured, budget) });
|
||||
}
|
||||
}
|
||||
const singleRunBreaches = runs.filter(run => run.phase !== "restart" && run.phase !== "verify").filter(run => {
|
||||
const start = runWarmth(run);
|
||||
const activity = firstActivityMs(run), boot = bootWithoutProvisionMs(run);
|
||||
const activityBudget = budgetFor(environment, run.adapter, "time_to_first_activity", start);
|
||||
const bootBudget = budgetFor(environment, run.adapter, "agent_boot", start);
|
||||
return (activity !== null && activityBudget !== null && activity > activityBudget.p95Ms)
|
||||
|| (boot !== null && bootBudget !== null && boot > bootBudget.p95Ms);
|
||||
}).length + samples.filter(sample => {
|
||||
const singleRunBreaches = runs.filter(run => run.phase !== "restart" && run.phase !== "verify")
|
||||
.filter(run => runOverBudget(environment, run)).length + samples.filter(sample => {
|
||||
const budget = budgetFor(environment, sample.adapter, "restart_recovery_control_plane", "restart");
|
||||
return !sample.interruptedByFault && sample.controlPlaneMs !== null && budget !== null && sample.controlPlaneMs > budget.p95Ms;
|
||||
}).length;
|
||||
const breaches = rows.reduce((total, row) => total + row.breaches.length, 0) + singleRunBreaches;
|
||||
return { environment, rows,
|
||||
const details = runs.filter(run => run.phase !== "restart" && run.phase !== "verify").map(run => runDetail(environment, run));
|
||||
return { environment, rows, runs: details,
|
||||
recoveryWithoutProviderEvent: samples.filter(sample => sample.latencyMs === null && !sample.interruptedByFault).length,
|
||||
recoveryInterruptedByFault: samples.filter(sample => sample.interruptedByFault).length,
|
||||
runBudgetExceeded, singleRunBreaches, breaches, errors, outageAdjustedSpans: outage.adjusted, outageRemovedMs: outage.removedMs,
|
||||
storageFaultWindows: storageWindows.length, storageFaultRemovedMs: storageRemovedMs };
|
||||
}
|
||||
|
||||
/** One cold or warm run over its own p95 budget (first activity or boot): a single-run breach. */
|
||||
function runOverBudget(environment: EnvironmentKind, run: BaselineRun): boolean {
|
||||
const start = runWarmth(run);
|
||||
const activity = firstActivityMs(run), boot = bootWithoutProvisionMs(run);
|
||||
const activityBudget = budgetFor(environment, run.adapter, "time_to_first_activity", start);
|
||||
const bootBudget = budgetFor(environment, run.adapter, "agent_boot", start);
|
||||
return (activity !== null && activityBudget !== null && activity > activityBudget.p95Ms)
|
||||
|| (boot !== null && bootBudget !== null && boot > bootBudget.p95Ms);
|
||||
}
|
||||
|
||||
function runDetail(environment: EnvironmentKind, run: BaselineRun): SpeedRunDetail {
|
||||
const start = runWarmth(run);
|
||||
const metrics = runMetrics(run);
|
||||
const spans = runSpans(run);
|
||||
const queue = spans.find(span => span.name === "heartbeat.queue") ?? null;
|
||||
const submitted = spans.find(span => span.name === "runner.turn.submit") ?? spans.find(span => span.name === "runner.session.resume") ?? null;
|
||||
const activity = spans.find(span => span.name === "task.time_to_first_agent_activity") ?? null;
|
||||
const turnAt = submitted ? submitted.startOffsetMs + submitted.durationMs : null;
|
||||
const activityAt = activity ? activity.startOffsetMs + activity.durationMs : null;
|
||||
return { runId: run.runId, adapter: run.adapter, start, firstActivityMs: firstActivityMs(run), bootMs: bootWithoutProvisionMs(run), queueMs: queue?.durationMs ?? null,
|
||||
providerFirstActivityMs: turnAt !== null && activityAt !== null ? Math.max(0, activityAt - turnAt) : null,
|
||||
preTurnSteps: metrics.preTurnSteps,
|
||||
breach: runOverBudget(environment, run) };
|
||||
}
|
||||
|
||||
async function runEventsWithRetry(api: ChaosApi, runId: string, attempts = 3): Promise<BaselineEvent[]> {
|
||||
let lastError: unknown;
|
||||
for (let attempt = 1; attempt <= attempts; attempt += 1) {
|
||||
|
||||
Reference in new issue
Block a user