mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
test(chaos): close review-thread gaps in speed grading, S1 stops, readiness and stall reads (#20244)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades how real tasks survive control-plane restarts, and its grades decide which product fixes come next > - Reviewers left 46 unresolved threads on ten merged harness PRs (#20071, #20094, #20106, #20111, #20119, #20147, #20158, #20161, #20178, #20194) > - Several are real harness defects: false speed breaches from one-sample rows, a codex-only wave that cannot start, event paging that can loop forever, an S1 task failed for a Stop the harness never sent, and a readiness wait that aborts on a transient lifecycle error > - Wave S1 t05 (w-20261109T091247-s9806) hit the S1 defect: the run finished before the stop window, so neither the Stop nor the continue comment was sent, and the task was graded `done_too_late Infinity` > - This pull request fixes the valid findings and adds the missing tests > - The benefit is that a failed task points at the product, not at the harness ## Linked Issues or Issue Description Refs #20071, #20094, #20111, #20147, #20158, #20161, #20178, #20194 (review threads). The security finding from #20111 was fixed separately in #20228. **What happened?** Harness defects found in review and in waves: - Aggregate p50/p95 rows breached at n = 1–3, which caused most marginal speed failures. - `--adapters codex` with the default workloads failed at parse because M1 needs two adapters. - `runEvents` paged forever if a full page had no `seq`. - A timed-out task whose last sample was `done` was graded `done_too_late Infinity`, and could pass when a fault had no ready time. - S1 waited forever for a running run to stop; it also clicked Stop while another task's fault had the server down or recovering. - The readiness wait aborted on any `lifecycleError`, including the transient one lifecycle `defer` sets before a 60 s retry, and hid read errors as "not ready". Daytona used the 240 s local timeout. - One failed parent read could hide an M1 stall for the whole 10 s check window. ## What Changed - Speed: aggregate rows breach only from 5 samples (`insufficient sample (n < 5)` otherwise). Restart recoveries over the control-plane p95 budget now count in the per-run check, so a single slow recovery still fails. - Config: the default workloads skip M1 when fewer than two adapters are given; an explicit `--workloads M1` with one adapter still fails. - `runEvents` throws when a full page has no later `seq`; the run's speed read fails instead of hanging. - `gradeR1` fails a timed-out task as `no_rest_within_budget` before checking its status. - S1: the Stop waits for a ready server with no fault in progress. If the run already finished when the window opens, the harness records a missed Stop, skips the stop-triggered fault, still posts the continue comment, and the oracle checks only the continue. - Readiness: only a terminal lifecycle state (rejected, terminating, cleaning up, terminated) aborts at once. Any other lifecycle error is logged when it changes (`seed_agent_lifecycle_error`) and the wait continues until ready or the timeout, which names each agent's last lifecycle or read error. The agent API exposes neither `lifecycleOperation.retryRequired` nor a retry-claim timestamp, so a deferred retry cannot be told apart from a parked failure earlier. Daytona waits up to 10 min. - M1: a failed parent read retries on the next poll. - Child speed collection moved into `childSpeedTasksInTurn` so its one-at-a-time order is tested. - Tests for each change, plus the gaps reviewers named: the stop trigger's arming, stall-wake parsing, and `seedChaosLab`'s readiness wiring. - README: S1 missed Stop, R1 timeout rule, speed sample minimum. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 230 passed. Each new test failed before its fix, such as the paging test running out of memory in the loop and the stop-trigger test firing a fault after a missed Stop. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - Wave evidence for the defects: S1 t05 in w-20261109T091247-s9806 (missed Stop); codex warm agent_boot p50 breach at n = 3 in w-20261109T104527-s9901. ## Risks - Aggregate rows with fewer than 5 samples no longer fail a wave. Per-run p95 checks still catch individual slow runs and recoveries, but a consistently slightly slow p50 in a small wave is now only reported. - A lifecycle error that persists without `retryRequired` now waits for the readiness timeout (4 min local, 10 min Daytona) instead of failing at once. The timeout message names the error. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1b09920a64
commit
610b54f551
15 files changed
+255
-32
No files matched your search
@@ -115,7 +115,10 @@ S1 is the board's Stop button. While the task's run is running, the harness canc
|
||||
comments that it stopped the run on purpose and asks the agent to continue with an extra marker
|
||||
line. The oracle checks that the stopped run ended `cancelled`, that nothing started between the
|
||||
stop and the continue comment (recovery must not restart work the operator stopped), and that the
|
||||
finished brief has both markers. S1 is not a default workload.
|
||||
finished brief has both markers. The Stop waits for a ready server with no fault in progress. If the
|
||||
run already finished when the stop window opens, the harness records a missed Stop
|
||||
(`task_stop_missed`), skips the stop-triggered fault, still posts the continue comment, and the
|
||||
oracle checks only the continue. S1 is not a default workload.
|
||||
|
||||
## Code change in a real repository (R1, `--workspace-repo`)
|
||||
|
||||
@@ -306,6 +309,12 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
`RESTART_CHAOS_ALLOW_UNSANDBOXED_ORACLE=1` is set inside a disposable VM.
|
||||
- R1 allows the workload budget plus outage time. When a Retry was needed R1 is
|
||||
`n/a` and R3 decides; a task that rests unfinished with no Retry fails both.
|
||||
A task that never rested within its deadline fails R1 as `no_rest_within_budget`,
|
||||
whatever its last status.
|
||||
- Speed: an aggregate p50/p95 row can breach only with at least 5 samples; a
|
||||
smaller row is reported as `insufficient sample (n < 5)`. Every cold or warm run
|
||||
over its p95 budget, and every restart recovery whose Paperclip share exceeds
|
||||
its p95 budget, still fails the wave on its own.
|
||||
- Faults are serialized; each one hits every task running at that moment.
|
||||
- A `todo` or `in_progress` task with no live run and no named wait is not a rest. The observer keeps watching it until it resumes or passes the R5 stall limit.
|
||||
- R5 also fails a run that stays `running` with no live evidence (no new run event, `lastUsefulActionAt` or `execution.lastConfirmedActivityAt`, and no start) for more than 3 minutes after the server is ready. A `running` status alone is not proof of life. A quiet run is still live while a tool it started has not ended (for example a 4-minute `bench.sh`) or while its recorded local runner process still runs with the same PID and start time.
|
||||
|
||||
@@ -136,11 +136,13 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
}
|
||||
}
|
||||
if (lifecycle !== undefined && lifecycle !== "per_turn" && lifecycle !== "warm" && lifecycle !== "default") throw new Error("--lifecycle must be per_turn, warm or default");
|
||||
const adapters = list<AdapterId>(values.adapters, ADAPTER_IDS, "adapters", ["codex", "claude"]);
|
||||
const options: WaveOptions = {
|
||||
env,
|
||||
adapters: list<AdapterId>(values.adapters, ADAPTER_IDS, "adapters", ["codex", "claude"]),
|
||||
adapters,
|
||||
// The governance scenarios (G1, H1) are opt-in: H1 needs a wave of its own.
|
||||
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1")),
|
||||
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1"
|
||||
&& (id !== "M1" || adapters.length >= 2))),
|
||||
// A crash-point wave schedules no fault; K1 is the restart its self-kill gets.
|
||||
faults: list<FaultKind>(values.faults, FAULT_KINDS, "faults", stormMinutes ? STORM_DEFAULT_FAULTS : crashPoint ? ["K1"] : ["K1", "K2", "K4"]),
|
||||
triggers: list<TriggerId>(values.triggers, TRIGGER_IDS, "triggers", ["time"]),
|
||||
|
||||
@@ -6,7 +6,7 @@ import { BUDGET_PAUSE_HOLD_MS, BUDGET_RESUME_HEADROOM_CENTS, REST_CONFIRM_MS, Ta
|
||||
import { planTasks, replayOnCurrentBuild, triggerFitsWorkload } from "./scheduler.js";
|
||||
import type { ChaosLab } from "./seed.js";
|
||||
import { Timeline } from "./timeline.js";
|
||||
import type { OracleArtifacts, TaskPlan, TaskRecord, TaskSample, WavePlan } from "./types.js";
|
||||
import type { FaultReceipt, OracleArtifacts, TaskPlan, TaskRecord, TaskSample, WavePlan } from "./types.js";
|
||||
import { g1ApprovalTitle, g1ChildTitle, g1Decision, workloadDefinitions } from "./workloads.js";
|
||||
|
||||
process.env.RESTART_CHAOS_QUIET = "1";
|
||||
@@ -127,6 +127,9 @@ describe("G1: approval-gated action across a restart", () => {
|
||||
createdAt: 0, restAt: 60_000, samples: [] } as unknown as TaskRecord;
|
||||
expect(gradeR1(record, 15 * 60_000, []).verdict).toBe("pass");
|
||||
expect(gradeR1({ ...record, finalStatus: "done" }, 15 * 60_000, []).verdict).toBe("fail");
|
||||
const unrested = gradeR1({ ...record, timedOut: true, restAt: null }, 15 * 60_000,
|
||||
[{ startedAt: 1_000, readyAt: null, healthyAt: null } as unknown as FaultReceipt]);
|
||||
expect(unrested).toEqual({ verdict: "fail", detail: "no_rest_within_budget (last status cancelled)" });
|
||||
});
|
||||
|
||||
it("counts a redacted approval title as the task's own", () => {
|
||||
|
||||
@@ -247,6 +247,11 @@ describe("T-oracles", () => {
|
||||
expect(restarted.checks.find(check => check.name === "nothing_ran_after_stop")).toMatchObject({ ok: false, detail: "continue=true runs=r2" });
|
||||
expect(workloadDefinitions.S1.grade("abcd1234", { ...base, stop: { ...base.stop!, runStatus: "succeeded" } }).verdict).toBe("fail");
|
||||
expect(workloadDefinitions.S1.grade("abcd1234", { ...base, stop: undefined }).verdict).toBe("fail");
|
||||
const missed = workloadDefinitions.S1.grade("abcd1234", { ...base, stop: { runId: null, runStatus: null, runsStartedBeforeContinue: [], continuePosted: true, missed: true } });
|
||||
expect(missed.verdict).toBe("pass");
|
||||
expect(missed.checks.find(check => check.name === "stop_took_effect")).toMatchObject({ ok: true, detail: "missed: the run finished before the stop window" });
|
||||
expect(workloadDefinitions.S1.grade("abcd1234", { ...base, documents: { brief: "CHAOS-abcd1234\n- one\n- two\n- three\n" },
|
||||
stop: { runId: null, runStatus: null, runsStartedBeforeContinue: [], continuePosted: true, missed: true } }).verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("grades B1 by running the tests and the CLI", () => {
|
||||
|
||||
@@ -61,8 +61,9 @@ export function gradeR1(record: TaskRecord, budgetMs: number, faults: readonly F
|
||||
if (record.launchError) return { verdict: "fail", detail: `launch_failed: ${record.launchError}` };
|
||||
if (record.retry) return { verdict: "n/a", detail: "task rested needing Retry; graded under R3" };
|
||||
const expected = expectedFinalStatus(record);
|
||||
if (record.timedOut) return { verdict: "fail", detail: `no_rest_within_budget (last status ${record.finalStatus ?? "unknown"})` };
|
||||
if (record.finalStatus !== expected) {
|
||||
return { verdict: "fail", detail: record.timedOut ? `no_rest_within_budget (last status ${record.finalStatus ?? "unknown"})` : `rested_not_done (status ${record.finalStatus ?? "unknown"}, no Retry offered)` };
|
||||
return { verdict: "fail", detail: `rested_not_done (status ${record.finalStatus ?? "unknown"}, no Retry offered)` };
|
||||
}
|
||||
const allowed = budgetMs + outageOverlapMs(record, faults);
|
||||
const took = (record.restAt ?? Number.POSITIVE_INFINITY) - record.createdAt!;
|
||||
|
||||
@@ -22,7 +22,7 @@ import { gradeTask } from "./grader.js";
|
||||
import { checkPlanInvariants, summarizeViolations, type InvariantCheck } from "./invariants.js";
|
||||
import { appendLedger, ledgerLine, writeSummary, type LedgerLine } from "./ledger.js";
|
||||
import { collectWaveSpeed, speedMarkdown } from "./wave-speed.js";
|
||||
import { TaskObserver } from "./observer.js";
|
||||
import { TaskObserver, childSpeedTasksInTurn } from "./observer.js";
|
||||
import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurrentBuild, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js";
|
||||
import { chaosLifecycleOverrides, seedChaosLab } from "./seed.js";
|
||||
import { ChaosSupervisor } from "./supervisor.js";
|
||||
@@ -469,8 +469,7 @@ async function main(): Promise<number> {
|
||||
}
|
||||
await writeFile(path.join(waveDir, "faults.json"), `${JSON.stringify(faults, null, 2)}\n`, { mode: 0o600 });
|
||||
const speed = await (async () => {
|
||||
const childTasks: Awaited<ReturnType<TaskObserver["childSpeedTasks"]>> = [];
|
||||
for (const observer of launched) childTasks.push(...await observer.childSpeedTasks());
|
||||
const childTasks = await childSpeedTasksInTurn(launched);
|
||||
return collectWaveSpeed(api, plan.options.env, [...launched.flatMap(observer => observer.record.issueId
|
||||
? [{ issueId: observer.record.issueId, adapter: observer.plan.adapter, samples: observer.record.samples }] : []), ...childTasks], faults);
|
||||
})().catch(error => { timeline.add("wave_speed_failed", { error: String(error) }); anyFailure = true; return null; });
|
||||
|
||||
@@ -4,7 +4,7 @@ import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import type { ChaosApi } from "./api.js";
|
||||
import { CHILD_WAKE_GRACE_MS, REST_CONFIRM_MS, STALL_GIVE_UP_MS, STOP_CONTINUE_AFTER_MS, TaskObserver, childrenPending, listWorkspaceFiles } from "./observer.js";
|
||||
import { CHILD_WAKE_GRACE_MS, REST_CONFIRM_MS, STALL_GIVE_UP_MS, STOP_CONTINUE_AFTER_MS, TaskObserver, childSpeedTasksInTurn, childrenPending, listWorkspaceFiles } from "./observer.js";
|
||||
import { gradeTask } from "./grader.js";
|
||||
import { createFixtureRepository, type ChaosLab } from "./seed.js";
|
||||
import { Timeline } from "./timeline.js";
|
||||
@@ -142,6 +142,31 @@ describe("TaskObserver", () => {
|
||||
expect(childRetries).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("M1: rereads a parent whose read failed on the next poll, and reads each stall wake from the parent's activity", async () => {
|
||||
const base = fakeApi({ "parentIds=": () => [{ id: "c2", parentId: "i1", status: "blocked" }], "/documents/": () => null, "/comments": () => [],
|
||||
"/workspace-operations": () => [], "/api/issues/c2": () => ({ id: "c2", status: "blocked", interruption: { canRetry: false } }),
|
||||
"/activity": () => [{ action: "issue.blocker_stalled_wake", details: { stalledIssueId: "c2", recoveryActionId: "a1" } },
|
||||
{ action: "issue.updated", details: {} }, { action: "issue.blocker_stalled_wake", details: {} }] }, "blocked");
|
||||
let failNextStallRead = false;
|
||||
const api = { ...base.api, async get(route: string, timeoutMs?: number) {
|
||||
if (route === "/api/issues/i1") {
|
||||
if (timeoutMs === undefined && failNextStallRead) { failNextStallRead = false; throw new Error("503"); }
|
||||
return { id: "i1", status: "blocked", stalledDependencies: [{ stalledIssueId: "c2", relation: "child", recoveryAction: { id: "a1", ownerType: "board" } }] };
|
||||
}
|
||||
return base.api.get(route);
|
||||
}, post: base.api.post.bind(base.api) } as unknown as ChaosApi;
|
||||
const observer = new TaskObserver({ ...plan, workload: "M1", followUpAfterMs: null }, api, lab, new Timeline(null), ready);
|
||||
await observer.launch();
|
||||
const start = observer.record.createdAt!;
|
||||
failNextStallRead = true;
|
||||
await observer.poll(start + 1_000);
|
||||
expect(observer.record.childStalls ?? []).toEqual([]);
|
||||
await observer.poll(start + 2_000);
|
||||
expect(observer.record.childStalls).toEqual([{ childIssueId: "c2", recoveryActionId: "a1" }]);
|
||||
const artifacts = await observer.collectArtifacts();
|
||||
expect(artifacts.stallWakes).toEqual([{ stalledIssueId: "c2", recoveryActionId: "a1" }, { stalledIssueId: null, recoveryActionId: null }]);
|
||||
});
|
||||
|
||||
it("S1: stops the running run as the board would, then posts the continue comment only after it settles", async () => {
|
||||
let runStatus = "running";
|
||||
const base = fakeApi({ "/runs": () => [{ id: "r1", status: runStatus, createdAt: new Date(0).toISOString(), finishedAt: null }] }, "in_progress");
|
||||
@@ -164,6 +189,67 @@ describe("TaskObserver", () => {
|
||||
expect(posted.filter(route => route.endsWith("/cancel"))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("S1: holds the Stop while another task's fault has the server down or recovering", async () => {
|
||||
const base = fakeApi({ "/runs": () => [{ id: "r1", status: "running", createdAt: new Date(0).toISOString(), finishedAt: null }] }, "in_progress");
|
||||
const posted: string[] = [];
|
||||
const api = { ...base.api, get: base.api.get.bind(base.api), async post(route: string) {
|
||||
posted.push(route);
|
||||
return route.endsWith("/issues") ? { id: "i1", identifier: "CHA-1" } : { id: "r1" };
|
||||
} } as unknown as ChaosApi;
|
||||
const supervisor = { serverReady: true, faultInProgress: false };
|
||||
const observer = new TaskObserver({ ...plan, workload: "S1", followUpAfterMs: 1_000 }, api, lab, new Timeline(null), supervisor);
|
||||
await observer.launch();
|
||||
supervisor.faultInProgress = true;
|
||||
await observer.poll(observer.record.createdAt! + 2_000);
|
||||
expect(posted.filter(route => route.endsWith("/cancel"))).toEqual([]);
|
||||
supervisor.faultInProgress = false;
|
||||
await observer.poll(observer.record.createdAt! + 3_000);
|
||||
expect(posted.filter(route => route.endsWith("/cancel"))).toEqual(["/api/heartbeat-runs/r1/cancel"]);
|
||||
});
|
||||
|
||||
it("S1: records a missed Stop and still posts the continue comment when the run finished before the stop window", async () => {
|
||||
const base = fakeApi({ "/runs": () => [{ id: "r1", status: "succeeded", createdAt: new Date(0).toISOString(), finishedAt: new Date(1).toISOString() }],
|
||||
"/comments": () => [], "/documents/": () => null, "/workspace-operations": () => [] }, "done");
|
||||
const posted: string[] = [];
|
||||
const api = { ...base.api, get: base.api.get.bind(base.api), async post(route: string) {
|
||||
posted.push(route);
|
||||
return route.endsWith("/issues") ? { id: "i1", identifier: "CHA-1" } : { id: "comment-1" };
|
||||
} } as unknown as ChaosApi;
|
||||
const timeline = new Timeline(null);
|
||||
const observer = new TaskObserver({ ...plan, workload: "S1", followUpAfterMs: 1_000 }, api, lab, timeline, ready);
|
||||
await observer.launch();
|
||||
const start = observer.record.createdAt!;
|
||||
await observer.poll(start + 2_000);
|
||||
expect(observer.record.stop).toMatchObject({ runId: null, missed: true });
|
||||
expect(timeline.entries.some(entry => entry.kind === "task_stop_missed")).toBe(true);
|
||||
await observer.poll(start + 2_000 + STOP_CONTINUE_AFTER_MS + 1);
|
||||
expect(observer.record.followUp).toMatchObject({ commentId: "comment-1" });
|
||||
expect(posted.filter(route => route.endsWith("/cancel"))).toEqual([]);
|
||||
expect((await observer.collectArtifacts()).stop).toMatchObject({ missed: true, continuePosted: true, runsStartedBeforeContinue: [] });
|
||||
});
|
||||
|
||||
it("S1: arms a stop-triggered fault at the Stop click plus its delay, and misses it when the Stop was missed", async () => {
|
||||
const running = fakeApi({ "/runs": () => [{ id: "r1", status: "running", createdAt: new Date(0).toISOString(), finishedAt: null }] }, "in_progress");
|
||||
const stopPlan: TaskPlan = { ...plan, workload: "S1", followUpAfterMs: 1_000, trigger: "stop", triggerDelayMs: 500 };
|
||||
const observer = new TaskObserver(stopPlan, running.api, lab, new Timeline(null), ready);
|
||||
await observer.launch();
|
||||
const start = observer.record.createdAt!;
|
||||
expect(observer.faultDue(start + 5_000)).toBeNull();
|
||||
await observer.poll(start + 2_000);
|
||||
const clicked = observer.record.stop!.at;
|
||||
expect(observer.faultDue(clicked + 499)).toBeNull();
|
||||
expect(observer.faultDue(clicked + 500)).toMatchObject({ trigger: "stop", kind: "K1" });
|
||||
|
||||
const finished = fakeApi({ "/runs": () => [{ id: "r1", status: "succeeded", createdAt: new Date(0).toISOString(), finishedAt: new Date(1).toISOString() }] }, "done");
|
||||
const timeline = new Timeline(null);
|
||||
const missed = new TaskObserver(stopPlan, finished.api, lab, timeline, ready);
|
||||
await missed.launch();
|
||||
await missed.poll(missed.record.createdAt! + 2_000);
|
||||
expect(missed.record.stop).toMatchObject({ missed: true });
|
||||
expect(missed.faultDue(missed.record.createdAt! + 2_600)).toBeNull();
|
||||
expect(missed.record.triggerMissed).toEqual(["stop"]);
|
||||
});
|
||||
|
||||
it("S1: does not count the continue comment's own run as a run after the stop, even when it starts before the comment returns", async () => {
|
||||
let runs: Array<Record<string, unknown>> = [{ id: "r1", status: "running", createdAt: new Date(0).toISOString(), finishedAt: null }];
|
||||
const base = fakeApi({ "/runs": () => runs, "/comments": () => [], "/documents/": () => null, "/workspace-operations": () => [] }, "in_progress");
|
||||
@@ -294,3 +380,18 @@ describe("childrenPending", () => {
|
||||
expect(childrenPending([{ status: "done" }, { status: "blocked" }])).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("childSpeedTasksInTurn", () => {
|
||||
it("reads each observer's subtasks one at a time so a large wave does not flood the API", async () => {
|
||||
let inFlight = 0, peak = 0;
|
||||
const observers = [1, 2, 3].map(id => ({ childSpeedTasks: async () => {
|
||||
inFlight += 1; peak = Math.max(peak, inFlight);
|
||||
await new Promise(resolve => setTimeout(resolve, 5));
|
||||
inFlight -= 1;
|
||||
return [{ issueId: `c${id}`, adapter: "codex" as const, samples: [] }];
|
||||
} }));
|
||||
expect((await childSpeedTasksInTurn(observers)).map(task => task.issueId)).toEqual(["c1", "c2", "c3"]);
|
||||
expect(peak).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -52,6 +52,14 @@ export const CHILD_WAKE_GRACE_MS = 60_000;
|
||||
export const CHILD_STALL_CHECK_MS = 10_000;
|
||||
export const STOP_CONTINUE_AFTER_MS = 20_000;
|
||||
|
||||
type ChildSpeedTask = { issueId: string; adapter: AdapterId; samples: Array<{ runs?: RunSample[] }> };
|
||||
|
||||
export async function childSpeedTasksInTurn(observers: ReadonlyArray<{ childSpeedTasks(): Promise<ChildSpeedTask[]> }>): Promise<ChildSpeedTask[]> {
|
||||
const tasks: ChildSpeedTask[] = [];
|
||||
for (const observer of observers) tasks.push(...await observer.childSpeedTasks());
|
||||
return tasks;
|
||||
}
|
||||
|
||||
export function childrenPending(children: ReadonlyArray<{ status: string | null }> | null): boolean {
|
||||
return !children || children.some(child => !["done", "cancelled", "blocked"].includes(child.status ?? ""));
|
||||
}
|
||||
@@ -348,7 +356,7 @@ export class TaskObserver {
|
||||
case "stop": {
|
||||
const stop = this.record.stop;
|
||||
if (!stop) return null;
|
||||
return stop.error ? "missed" : stop.at;
|
||||
return stop.error || stop.missed ? "missed" : stop.at;
|
||||
}
|
||||
case "result": {
|
||||
// B8: the run that received the follow-up has accepted its result but
|
||||
@@ -501,10 +509,15 @@ export class TaskObserver {
|
||||
}
|
||||
|
||||
private async maybeStop(sample: TaskSample, now: number): Promise<boolean> {
|
||||
if (!this.definition.stopBeforeFollowUp || this.record.stop || this.plan.followUpAfterMs === null || !sample.reachable) return false;
|
||||
if (!this.definition.stopBeforeFollowUp || this.record.stop || this.plan.followUpAfterMs === null || !sample.serverReady) return false;
|
||||
if (now < this.record.createdAt! + this.plan.followUpAfterMs) return false;
|
||||
const running = (sample.runs ?? []).find(run => run.status === "running");
|
||||
if (!running) return false;
|
||||
if (!running) {
|
||||
if (!isResting(sample)) return false;
|
||||
this.record.stop = { at: now, runId: null, httpStatus: null, error: null, missed: true };
|
||||
this.timeline.add("task_stop_missed", { taskId: this.plan.taskId, status: sample.issueStatus ?? null });
|
||||
return true;
|
||||
}
|
||||
const stop: NonNullable<TaskRecord["stop"]> = { at: now, runId: running.id, httpStatus: null, error: null };
|
||||
try {
|
||||
await this.api.post(`/api/heartbeat-runs/${running.id}/cancel`, { cancellationRequestId: randomUUID() });
|
||||
@@ -588,6 +601,7 @@ export class TaskObserver {
|
||||
.catch(() => null);
|
||||
const blocked = (Array.isArray(rows) ? rows : []).filter(row => row.parentId === this.record.issueId && row.status === "blocked").map(row => String(row.id));
|
||||
const parent = await this.api.get<Json>(`/api/issues/${this.record.issueId}`).catch(() => null);
|
||||
if (!parent) this.childStallCheckedAt = null;
|
||||
const stalled = Array.isArray(parent?.stalledDependencies) ? parent.stalledDependencies as Json[] : [];
|
||||
for (const dependency of stalled) {
|
||||
if (dependency.relation !== "child" || typeof dependency.stalledIssueId !== "string") continue;
|
||||
@@ -611,8 +625,8 @@ export class TaskObserver {
|
||||
const startedBeforeContinue = [...runs.values()].filter(run => run.id !== stop.runId && run.createdAt
|
||||
&& !(continueCommentId && (run.wakeCommentIds ?? []).includes(continueCommentId))
|
||||
&& Date.parse(run.createdAt) > stop.at && (continueAt === null || Date.parse(run.createdAt) < continueAt)).map(run => run.id);
|
||||
return { runId: stop.runId, runStatus: runs.get(stop.runId)?.status ?? null, runsStartedBeforeContinue: startedBeforeContinue,
|
||||
continuePosted: Boolean(this.record.followUp?.commentId) };
|
||||
return { runId: stop.runId, runStatus: stop.runId ? runs.get(stop.runId)?.status ?? null : null, runsStartedBeforeContinue: startedBeforeContinue,
|
||||
continuePosted: Boolean(this.record.followUp?.commentId), ...(stop.missed ? { missed: true } : {}) };
|
||||
}
|
||||
|
||||
private async retryBlockedChildren(childIds: readonly string[], now: number): Promise<boolean> {
|
||||
@@ -805,7 +819,7 @@ export class TaskObserver {
|
||||
runsStartedWhilePaused: governance.runsStartedWhilePaused });
|
||||
}
|
||||
|
||||
async childSpeedTasks(): Promise<Array<{ issueId: string; adapter: AdapterId; samples: Array<{ runs?: RunSample[] }> }>> {
|
||||
async childSpeedTasks(): Promise<ChildSpeedTask[]> {
|
||||
if (!this.definition.collect.childDocuments || !this.record.issueId) return [];
|
||||
const rows = await this.api.get<Json[]>(`/api/companies/${this.lab.companyId}/issues?view=index&parentIds=${encodeURIComponent(this.record.issueId)}&limit=20`);
|
||||
const tasks: Array<{ issueId: string; adapter: AdapterId; samples: Array<{ runs?: RunSample[] }> }> = [];
|
||||
|
||||
@@ -42,6 +42,11 @@ describe("scheduler determinism", () => {
|
||||
expect(Math.max(...holds)).toBeGreaterThan(K4_HOLD_RANGE_MS[1]);
|
||||
const storm = planStorm(5, 120, ["K4"], 600_000);
|
||||
expect(Math.max(...storm.faults.map(fault => fault.holdMs))).toBeGreaterThan(K4_HOLD_RANGE_MS[1]);
|
||||
const single = parseCommand(["--adapters", "codex", "--seed", "5"]), both = parseCommand(["--seed", "5"]);
|
||||
if (single.mode !== "wave" || both.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(single.options.workloads).not.toContain("M1");
|
||||
expect(both.options.workloads).toContain("M1");
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "M1"])).toThrow(/two agents/);
|
||||
expect(() => parseCommand(["--k4-hold-max-seconds", "601"])).toThrow(/between 120 and 600/);
|
||||
expect(() => parseCommand(["--replay", "w-20261106T101112-s42/t03", "--k4-hold-max-seconds", "400"])).toThrow(/do not combine/);
|
||||
});
|
||||
|
||||
@@ -48,6 +48,18 @@ describe("seedChaosLab workspace isolation", () => {
|
||||
const seed = (api: ChaosApi) => seedChaosLab({ api, env: "local", adapters: ["codex"], concurrency: 2,
|
||||
budgetCents: 3_000, fixtureRoot, waveId: "w-test", timeline: new Timeline(null) });
|
||||
|
||||
it("waits for each seeded agent to reach lifecycleState ready only when the wave asks for it", async () => {
|
||||
const waiting = fakeApi(body => ({ ...body }));
|
||||
const timeline = new Timeline(null);
|
||||
await seedChaosLab({ api: waiting.api, env: "local", adapters: ["codex"], concurrency: 2, budgetCents: 3_000,
|
||||
fixtureRoot: path.join(fixtureRoot, "waiting"), waveId: "w-test", timeline, waitForReadyAgents: true });
|
||||
expect(waiting.calls.some(call => call.method === "GET" && call.route.startsWith("/api/agents/"))).toBe(true);
|
||||
expect(timeline.entries.filter(entry => entry.kind === "seed_agent_ready")).toEqual([expect.objectContaining({ adapter: "codex" })]);
|
||||
const skipping = fakeApi(body => ({ ...body }));
|
||||
await seed(skipping.api);
|
||||
expect(skipping.calls.some(call => call.method === "GET" && call.route.startsWith("/api/agents/"))).toBe(false);
|
||||
});
|
||||
|
||||
// RC-21: the project's git_worktree policy is gated behind the instance's
|
||||
// isolated-workspaces setting. Without it every repo task ran in the shared
|
||||
// primary checkout, so concurrent tasks wrote into one tree.
|
||||
@@ -157,9 +169,26 @@ describe("waitForAgentsReady", () => {
|
||||
await expect(waitForAgentsReady(stuck, { codex: { id: "a3" } }, new Timeline(null), 60_000, 1, () => ++polls > 2)).rejects.toThrow(/interrupted/);
|
||||
});
|
||||
|
||||
it("keeps waiting through a deferred verification error and passes once the retry makes the agent ready", async () => {
|
||||
let reads = 0;
|
||||
const api = { get: async () => (++reads < 3 ? { lifecycleState: "verifying", lifecycleError: "host busy" } : { lifecycleState: "ready" }) } as unknown as ChaosApi;
|
||||
const timeline = new Timeline(null);
|
||||
await waitForAgentsReady(api, { codex: { id: "a1" } }, timeline, 5_000, 1);
|
||||
expect(timeline.entries.filter(entry => entry.kind === "seed_agent_lifecycle_error")).toEqual([expect.objectContaining({ adapter: "codex", error: "host busy" })]);
|
||||
});
|
||||
|
||||
it("names the last read error when an agent never becomes ready", async () => {
|
||||
const broken = { get: async () => { throw new Error("401 unauthorized"); } } as unknown as ChaosApi;
|
||||
await expect(waitForAgentsReady(broken, { codex: { id: "a1" } }, new Timeline(null), 20, 5)).rejects.toThrow(/codex \(last read error: 401 unauthorized\)/);
|
||||
const deferred = { get: async () => ({ lifecycleState: "verifying", lifecycleError: "host busy" }) } as unknown as ChaosApi;
|
||||
await expect(waitForAgentsReady(deferred, { codex: { id: "a1" } }, new Timeline(null), 20, 5)).rejects.toThrow(/codex \(last lifecycle error: host busy\)/);
|
||||
});
|
||||
|
||||
it("fails the wave when an agent fails verification or never becomes ready", async () => {
|
||||
const failed = { get: async () => ({ lifecycleState: "verifying", lifecycleError: "hello probe failed" }) } as unknown as ChaosApi;
|
||||
await expect(waitForAgentsReady(failed, { claude: { id: "a2" } }, new Timeline(null), 5_000, 1)).rejects.toThrow(/hello probe failed/);
|
||||
const parked = { get: async () => ({ lifecycleState: "verifying", lifecycleError: "hello probe failed" }) } as unknown as ChaosApi;
|
||||
await expect(waitForAgentsReady(parked, { claude: { id: "a2" } }, new Timeline(null), 20, 5)).rejects.toThrow(/claude \(last lifecycle error: hello probe failed\)/);
|
||||
const rejected = { get: async () => ({ lifecycleState: "rejected", lifecycleError: "hire rejected" }) } as unknown as ChaosApi;
|
||||
await expect(waitForAgentsReady(rejected, { claude: { id: "a2" } }, new Timeline(null), 5_000, 1)).rejects.toThrow(/hire rejected/);
|
||||
const stuck = { get: async () => ({ lifecycleState: "verifying" }) } as unknown as ChaosApi;
|
||||
await expect(waitForAgentsReady(stuck, { claude: { id: "a3" } }, new Timeline(null), 20, 5)).rejects.toThrow(/never reached lifecycleState ready/);
|
||||
});
|
||||
|
||||
@@ -21,25 +21,43 @@ export interface ChaosLab {
|
||||
agents: Record<AdapterId, { id: string; name: string; body?: Record<string, unknown> }>;
|
||||
}
|
||||
|
||||
export const DAYTONA_AGENT_READY_TIMEOUT_MS = 10 * 60_000;
|
||||
const TERMINAL_LIFECYCLE_STATES = new Set(["rejected", "terminating", "cleaning_up", "terminated"]);
|
||||
|
||||
/** Create the `chaos-sandbox` git repository: python CLI, one failing test, bench.sh. */
|
||||
export async function waitForAgentsReady(api: ChaosApi, agents: Partial<Record<string, { id: string; createdAt?: number }>>, timeline: Timeline,
|
||||
timeoutMs = 4 * 60_000, pollMs = 1_000, cancelled: () => boolean = () => false): Promise<void> {
|
||||
const started = Date.now();
|
||||
const pending = new Map(Object.entries(agents).flatMap(([adapter, agent]) => agent ? [[adapter, agent] as const] : []));
|
||||
const lastProblem = new Map<string, string>();
|
||||
while (pending.size) {
|
||||
if (cancelled()) throw new Error("Harness interrupted while waiting for agents to finish verification");
|
||||
for (const [adapter, { id: agentId, createdAt }] of [...pending]) {
|
||||
const agent = await api.get<Record<string, unknown>>(`/api/agents/${agentId}`).catch(() => null);
|
||||
let agent: Record<string, unknown>;
|
||||
try {
|
||||
agent = await api.get<Record<string, unknown>>(`/api/agents/${agentId}`);
|
||||
} catch (error) {
|
||||
lastProblem.set(adapter, `last read error: ${error instanceof Error ? error.message : String(error)}`.slice(0, 300));
|
||||
continue;
|
||||
}
|
||||
const lifecycleError = agent?.lifecycleError ? String(agent.lifecycleError).slice(0, 300) : null;
|
||||
if (agent?.lifecycleState === "ready") {
|
||||
timeline.add("seed_agent_ready", { adapter, agentId, readyAfterMs: Date.now() - (createdAt ?? started) });
|
||||
pending.delete(adapter);
|
||||
} else if (agent?.lifecycleError) {
|
||||
throw new Error(`Agent ${adapter} (${agentId}) failed lifecycle verification: ${String(agent.lifecycleError).slice(0, 300)}`);
|
||||
} else if (lifecycleError) {
|
||||
if (TERMINAL_LIFECYCLE_STATES.has(String(agent.lifecycleState))) {
|
||||
throw new Error(`Agent ${adapter} (${agentId}) failed lifecycle verification: ${lifecycleError}`);
|
||||
}
|
||||
if (lastProblem.get(adapter) !== `last lifecycle error: ${lifecycleError}`) {
|
||||
timeline.add("seed_agent_lifecycle_error", { adapter, agentId, error: lifecycleError });
|
||||
}
|
||||
lastProblem.set(adapter, `last lifecycle error: ${lifecycleError}`);
|
||||
}
|
||||
}
|
||||
if (!pending.size) return;
|
||||
if (Date.now() - started > timeoutMs) {
|
||||
throw new Error(`Agents never reached lifecycleState ready within ${Math.round(timeoutMs / 1000)} s: ${[...pending.keys()].join(", ")}`);
|
||||
const names = [...pending.keys()].map(adapter => lastProblem.has(adapter) ? `${adapter} (${lastProblem.get(adapter)})` : adapter);
|
||||
throw new Error(`Agents never reached lifecycleState ready within ${Math.round(timeoutMs / 1000)} s: ${names.join(", ")}`);
|
||||
}
|
||||
await new Promise(resolve => setTimeout(resolve, pollMs));
|
||||
}
|
||||
@@ -242,7 +260,8 @@ export async function seedChaosLab(input: {
|
||||
}
|
||||
if (input.waitForReadyAgents) {
|
||||
await waitForAgentsReady(api, Object.fromEntries(Object.entries(agents).map(([adapter, agent]) =>
|
||||
[adapter, { id: agent.id, createdAt: agentCreatedAt.get(adapter) }])), timeline, input.agentReadyTimeoutMs, 1_000, input.cancelled);
|
||||
[adapter, { id: agent.id, createdAt: agentCreatedAt.get(adapter) }])), timeline,
|
||||
input.agentReadyTimeoutMs ?? (input.env === "daytona" ? DAYTONA_AGENT_READY_TIMEOUT_MS : undefined), 1_000, input.cancelled);
|
||||
}
|
||||
|
||||
// The project's git_worktree policy only takes effect when the instance has
|
||||
|
||||
@@ -221,7 +221,7 @@ export interface OracleArtifacts {
|
||||
workspacePath: string | null;
|
||||
/** B5: distinct `.bench/progress` values the harness itself saw while the task was live. */
|
||||
benchProgressObserved?: number;
|
||||
stop?: { runId: string; runStatus: string | null; runsStartedBeforeContinue: string[]; continuePosted: boolean };
|
||||
stop?: { runId: string | null; runStatus: string | null; runsStartedBeforeContinue: string[]; continuePosted: boolean; missed?: boolean };
|
||||
children?: Array<{ id: string; title?: string | null; status: string | null; assigneeAgentId: string | null; documents: Record<string, string | null> }>;
|
||||
childStalls?: Array<{ childIssueId: string; recoveryActionId: string | null }>;
|
||||
stallWakes?: Array<{ stalledIssueId: string | null; recoveryActionId: string | null }>;
|
||||
@@ -283,7 +283,7 @@ export interface TaskRecord {
|
||||
deadlineAt: number | null;
|
||||
timedOut: boolean;
|
||||
followUp: { at: number; sentAt?: number; postedAt?: number; commentId: string | null; error: string | null; duringRun?: boolean } | null;
|
||||
stop?: { at: number; runId: string; httpStatus: number | null; error: string | null };
|
||||
stop?: { at: number; runId: string | null; httpStatus: number | null; error: string | null; missed?: boolean };
|
||||
faultIds: string[];
|
||||
sweep: RestSweep | null;
|
||||
/** Workspace files present just before the first fault that hit this task (R2). */
|
||||
|
||||
@@ -21,9 +21,12 @@ describe("wave speed grade", () => {
|
||||
|
||||
const slow = gradeWaveSpeed("local", [run("a", "codex", 40_000), run("b", "codex", 40_000,
|
||||
[{ eventType: "run.performance.budget_exceeded", createdAt: "2026-11-08T00:00:00.000Z", payload: {} }])], [], 1);
|
||||
expect(slow.rows.find(row => row.metric === "time_to_first_activity")?.breaches).toEqual(["p50", "p95"]);
|
||||
expect(slow.rows.find(row => row.metric === "time_to_first_activity")?.breaches).toEqual([]);
|
||||
expect(slow.rows.find(row => row.metric === "ttft")).toMatchObject({ budget: null, breaches: [] });
|
||||
expect(slow).toMatchObject({ breaches: 4, singleRunBreaches: 2, runBudgetExceeded: 1 });
|
||||
expect(slow).toMatchObject({ breaches: 2, singleRunBreaches: 2, runBudgetExceeded: 1 });
|
||||
const slowFive = gradeWaveSpeed("local", ["a", "b", "c", "d", "e"].map(id => run(id, "codex", 40_000)), [], 0);
|
||||
expect(slowFive.rows.find(row => row.metric === "time_to_first_activity")?.breaches).toEqual(["p50", "p95"]);
|
||||
expect(slowFive).toMatchObject({ breaches: 7, singleRunBreaches: 5 });
|
||||
const heldButFast = gradeWaveSpeed("local", [run("c", "codex", 4_000,
|
||||
[{ eventType: "run.performance.budget_exceeded", createdAt: "2026-11-08T00:00:00.000Z", payload: {} }])], [], 1);
|
||||
expect(heldButFast).toMatchObject({ breaches: 0, singleRunBreaches: 0, runBudgetExceeded: 1 });
|
||||
@@ -46,7 +49,8 @@ describe("wave speed grade", () => {
|
||||
events: [{ eventType: "turn.started", createdAt: new Date(10_000 + 30_000).toISOString() }] };
|
||||
const speed = gradeWaveSpeed("local", [recovered], [{ kind: "K1", readyAt: 10_000, runIds: ["r"] }], 0);
|
||||
// Nothing reloaded, so the whole latency is Paperclip's share and is judged.
|
||||
expect(speed.rows.find(row => row.metric === "restart_recovery_control_plane")).toMatchObject({ adapter: "claude", breaches: ["p50", "p95"] });
|
||||
expect(speed.rows.find(row => row.metric === "restart_recovery_control_plane")).toMatchObject({ adapter: "claude", breaches: [] });
|
||||
expect(speed).toMatchObject({ singleRunBreaches: 1, breaches: 1 });
|
||||
expect(speed.rows.find(row => row.metric === "restart_recovery")).toMatchObject({ budget: null, breaches: [], measured: { p50: 30_000 } });
|
||||
});
|
||||
|
||||
@@ -92,6 +96,25 @@ describe("wave speed grade", () => {
|
||||
expect(breachesOf({ n: 1, p50: 99_000, p95: 99_000, max: 99_000 }, null)).toEqual([]);
|
||||
});
|
||||
|
||||
it("judges aggregate p50 and p95 only from five samples and reports smaller rows as an insufficient sample", () => {
|
||||
const budget = { p50Ms: 1_000, p95Ms: 2_000 };
|
||||
expect(breachesOf({ n: 4, p50: 9_000, p95: 9_000, max: 9_000 }, budget)).toEqual([]);
|
||||
expect(breachesOf({ n: 5, p50: 9_000, p95: 9_000, max: 9_000 }, budget)).toEqual(["p50", "p95"]);
|
||||
const speed = gradeWaveSpeed("local", [], [], 0);
|
||||
speed.rows.push({ adapter: "codex", metric: "agent_boot", start: "cold", measured: { n: 2, p50: 9_000, p95: 9_000, max: 9_000 },
|
||||
budget, breaches: [] });
|
||||
expect(speedMarkdown(speed)).toContain("| insufficient sample (n < 5) |");
|
||||
});
|
||||
|
||||
it("fails a run's speed read instead of paging forever when a full page has no seq", async () => {
|
||||
let reads = 0;
|
||||
const api = { get: async () => { reads += 1; return Array.from({ length: 500 }, () => ({ eventType: "x", createdAt: new Date(0).toISOString() })); } } as unknown as ChaosApi;
|
||||
const speed = await collectWaveSpeed(api, "local", [{ issueId: "i", adapter: "codex", samples: [{ runs: [{ id: "r", status: "succeeded",
|
||||
createdAt: new Date(0).toISOString(), finishedAt: null }] }] as never }], []);
|
||||
expect(speed.errors).toEqual([expect.stringMatching(/^r: .*seq/)]);
|
||||
expect(reads).toBe(3);
|
||||
});
|
||||
|
||||
const traced = (trace: number, spans: Array<[string, number, number]>): BaselineRun => ({
|
||||
runId: "t", adapter: "claude", environment: "local", phase: "cold",
|
||||
events: spans.map(([name, offset, duration]) => ({ eventType: "run.performance.span",
|
||||
|
||||
@@ -46,8 +46,10 @@ export function budgetFor(environment: EnvironmentKind, adapter: string, metric:
|
||||
return byStart[start === "warm" ? "warm" : "cold"];
|
||||
}
|
||||
|
||||
export const MIN_AGGREGATE_SAMPLES = 5;
|
||||
|
||||
export function breachesOf(measured: Distribution, budget: SpeedBudget | null): Array<"p50" | "p95"> {
|
||||
if (!budget) return [];
|
||||
if (!budget || measured.n < MIN_AGGREGATE_SAMPLES) return [];
|
||||
const out: Array<"p50" | "p95"> = [];
|
||||
if (measured.p50 !== null && measured.p50 > budget.p50Ms) out.push("p50");
|
||||
if (measured.p95 !== null && measured.p95 > budget.p95Ms) out.push("p95");
|
||||
@@ -211,6 +213,9 @@ export function gradeWaveSpeed(environment: EnvironmentKind, runs: readonly Base
|
||||
const bootBudget = budgetFor(environment, run.adapter, "agent_boot", start);
|
||||
return (activity !== null && activityBudget !== null && activity > activityBudget.p95Ms)
|
||||
|| (boot !== null && bootBudget !== null && boot > bootBudget.p95Ms);
|
||||
}).length + samples.filter(sample => {
|
||||
const budget = budgetFor(environment, sample.adapter, "restart_recovery_control_plane", "restart");
|
||||
return !sample.interruptedByFault && sample.controlPlaneMs !== null && budget !== null && sample.controlPlaneMs > budget.p95Ms;
|
||||
}).length;
|
||||
const breaches = rows.reduce((total, row) => total + row.breaches.length, 0) + singleRunBreaches;
|
||||
return { environment, rows,
|
||||
@@ -237,7 +242,9 @@ async function runEvents(api: ChaosApi, runId: string): Promise<BaselineEvent[]>
|
||||
const page = await api.get<Json[]>(`/api/heartbeat-runs/${runId}/events?afterSeq=${afterSeq}&limit=500`, 15_000);
|
||||
for (const row of page) events.push({ eventType: String(row.eventType), createdAt: String(row.createdAt), payload: (row.payload ?? null) as Json | null });
|
||||
if (page.length < 500) return events;
|
||||
afterSeq = Number(page[page.length - 1]!.seq);
|
||||
const next = Number(page[page.length - 1]!.seq);
|
||||
if (!Number.isFinite(next) || next <= afterSeq) throw new Error(`run events page after seq ${afterSeq} has no later seq`);
|
||||
afterSeq = next;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -282,16 +289,21 @@ export async function collectWaveSpeed(api: ChaosApi, environment: EnvironmentKi
|
||||
return gradeWaveSpeed(environment, runs, faultRuns(faults, timing), runBudgetExceeded, errors, outage);
|
||||
}
|
||||
|
||||
function breachCell(row: SpeedRow): string {
|
||||
if (row.breaches.length) return row.breaches.join(", ");
|
||||
return row.budget && row.measured.n > 0 && row.measured.n < MIN_AGGREGATE_SAMPLES ? `insufficient sample (n < ${MIN_AGGREGATE_SAMPLES})` : "—";
|
||||
}
|
||||
|
||||
export function speedMarkdown(speed: WaveSpeed): string {
|
||||
const s = (value: number | null) => value === null ? "—" : `${(value / 1000).toFixed(1)} s`;
|
||||
const lines = [`## Speed (${speed.environment})`, "",
|
||||
`Breaches: ${speed.breaches} (aggregate p50/p95 rows plus ${speed.singleRunBreaches} runs over the p95 budget after outage removal; the server logged ${speed.runBudgetExceeded} budget_exceeded events before removal).`,
|
||||
`Breaches: ${speed.breaches} (aggregate p50/p95 rows with at least ${MIN_AGGREGATE_SAMPLES} samples plus ${speed.singleRunBreaches} runs or restart recoveries over the p95 budget after outage removal; the server logged ${speed.runBudgetExceeded} budget_exceeded events before removal).`,
|
||||
`Restart samples with no provider event after ready: ${speed.recoveryWithoutProviderEvent}. Interrupted by a later fault (not graded): ${speed.recoveryInterruptedByFault}.`,
|
||||
`Outage time removed from ${speed.outageAdjustedSpans} spans: ${s(speed.outageRemovedMs)} (fault start to server ready; budgets judge the rest).`, "",
|
||||
"| Adapter | Metric | Start | n | p50 | p95 | Budget p50 / p95 | Breach |", "|---|---|---|---|---|---|---|---|"];
|
||||
for (const row of speed.rows) {
|
||||
lines.push(`| ${row.adapter} | ${row.metric} | ${row.start} | ${row.measured.n} | ${s(row.measured.p50)} | ${s(row.measured.p95)} | `
|
||||
+ `${row.budget ? `${s(row.budget.p50Ms)} / ${s(row.budget.p95Ms)}` : "no budget"} | ${row.breaches.join(", ") || "—"} |`);
|
||||
+ `${row.budget ? `${s(row.budget.p50Ms)} / ${s(row.budget.p95Ms)}` : "no budget"} | ${breachCell(row)} |`);
|
||||
}
|
||||
if (speed.errors.length) lines.push("", `Speed grade incomplete: ${speed.errors.length} run logs not read; the wave fails.`);
|
||||
return `${lines.join("\n")}\n`;
|
||||
|
||||
@@ -270,7 +270,8 @@ export const workloadDefinitions: Record<WorkloadId, WorkloadDefinition> = {
|
||||
return result([
|
||||
statusCheck(artifacts),
|
||||
...briefChecks(artifacts.documents.brief, [briefMarker(nonce), continueMarker(nonce)]),
|
||||
{ name: "stop_took_effect", ok: stop?.runStatus === "cancelled", detail: `run=${stop?.runId ?? "none"} status=${stop?.runStatus ?? "none"}` },
|
||||
{ name: "stop_took_effect", ok: Boolean(stop?.missed) || stop?.runStatus === "cancelled",
|
||||
detail: stop?.missed ? "missed: the run finished before the stop window" : `run=${stop?.runId ?? "none"} status=${stop?.runStatus ?? "none"}` },
|
||||
{ name: "nothing_ran_after_stop", ok: Boolean(stop) && stop!.continuePosted && stop!.runsStartedBeforeContinue.length === 0,
|
||||
detail: stop ? `continue=${stop.continuePosted} runs=${stop.runsStartedBeforeContinue.join(",") || "none"}` : "not stopped" },
|
||||
]);
|
||||
|
||||
Reference in new issue
Block a user