mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
test(restart-chaos): staging class R cells CR1-CR3 and sandbox capture on every Daytona wave (#20909)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) reproduces staging failures before users meet them > - Staging fails with `runner_remote_process_identity_unavailable` when the remote Runner or its provider is gone after a deploy, a stall or a re-wake, and nobody has seen the sandbox side > - The harness had no cell for a provider-only death or a plugin RPC stall, and its sandbox capture ran only on K15 waves > - This pull request adds the staging class R cells CR1–CR3, and captures the sandbox side on every Daytona wave > - The benefit is that a wave that reproduces the refusal also keeps the evidence of why the Runner or provider died ## Linked Issues or Issue Description **What is the problem?** The harness cannot answer why a remote runnerd or its provider dies. Its capture ran only around K15 restarts. It recorded no process start times, sockets or provider log files. It had no fault that kills only the provider or stalls a plugin RPC. **What should change?** Add cells for each staging shape, and capture the sandbox before, during and after each one, on every Daytona wave. **Context** Refs #20907 (keeps runnerd's stderr in the sandbox, and reports it on failed recovery). Refs #20893 (W2 re-wake cells). ## What Changed - `class-r.ts` (new) adds three faults, selected with `--faults`: - **CR1:** a K15 redeploy with a held outage. `--cr1-hold-seconds`, 0–600, default 360. - **CR2:** SIGKILL of only the provider process under the run's `paperclip-runnerd`, inside its sandbox. Runnerd stays alive. By default the controller stays up; `--cr2-hold-seconds` adds a K15 outage. - **CR3:** SIGSTOP of the server's Daytona plugin worker. `--cr3-stall-seconds`, 41–600, default 45. Each CR task must pass the `class_r_fault_applied` check. - `daytona-runner-logs.ts` adds `CLASS_R_PROBE`. It reads process start times and `wchan`, sockets, the runtime file listing, and the tails of the Runner's and the provider's state and log files. It also adds `executeInWaveSandboxes`, the fixed-script runner that CR2 uses. - `main.ts` and `supervisor.ts`: - every Daytona wave captures with the class R probe when a task rests and at wave end; - CR cells also capture before the fault, after the stop, before the restart, and every 15 s for 180 s after ready; - `--sandbox-watch-seconds` captures on a clock for the whole wave. - The README has a new "Staging class R cells" section. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 49 files, 600 tests pass, including the new `class-r.test.ts`. It covers option parsing, planning, the kill command's quoting and its refusal of anything but a run id, plugin worker selection, the watch schedule and the applied check. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Daytona waves run from this branch plus #20907. The results are in the follow-up comment. ## Risks - This change is limited to the harness. CR2 is its only in-sandbox mutation: a SIGKILL of the provider, run by a fixed script only in sandboxes labelled with the wave's company and environment. The script never signals runnerd. - More captures add Daytona toolbox calls on every Daytona wave. Each capture has a 45 s deadline, and a failed capture is recorded and never fails the wave. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, Daytona waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed
1 parent
48d23d1530
commit
0bf3236bc8
12 files changed
+793
-20
No files matched your search
@@ -425,6 +425,76 @@ for a while after the new one was up. Plain K15 restarts on the same instance ho
|
||||
Pass is the usual grade: no Retry (R3), no stuck record (R6), and the oracle. Staging's terminal errors here
|
||||
were `ENOENT` on a runtime asset, `Agent directory operation is already active` after a 9.5-minute
|
||||
`controller_lease_held` wait.
|
||||
### Staging class R cells (CR1–CR3, `class-r.ts`)
|
||||
|
||||
```sh
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \
|
||||
--adapters opencode,claude --workloads T1 --faults CR1 --concurrency 1 --tasks 2 --seed 9501
|
||||
```
|
||||
|
||||
Staging failed with `runner_remote_process_identity_unavailable` in 9 runs. In 8 of them:
|
||||
|
||||
1. A deploy kept the controller away for 5–7 minutes.
|
||||
2. The Runner in the Daytona sandbox survived, and the new controller adopted it.
|
||||
3. About 40 s later, `session.snapshot` failed because the provider process was dead or unresponsive.
|
||||
|
||||
The ninth run had no deploy. A plugin `environmentExecute` RPC stalled for more than 40 s. These cells
|
||||
reproduce each part. The class is named R on staging. The cells are named CR1–CR3 because the R1 workload
|
||||
and the R1–R6 grading rules already exist.
|
||||
|
||||
| Fault | What it does | Option |
|
||||
| --- | --- | --- |
|
||||
| CR1 | The K15 redeploy: SIGTERM, then a held outage, then the next deploy generation. | `--cr1-hold-seconds <n>` or `<min>-<max>`, 0–600. Default 360. |
|
||||
| CR2 | SIGKILL of the provider process only, inside the task's sandbox. The Runner stays alive. | `--cr2-hold-seconds <n>`, 0–600. Default 0: the controller stays up. With a hold, a K15 redeploy holds the controller down, and the kill happens right after the stop. |
|
||||
| CR3 | SIGSTOP of the server's Daytona plugin worker and its children. Every plugin RPC, including `environmentExecute`, stalls. There is no deploy. | `--cr3-stall-seconds <n>`, 41–600. Default 45. |
|
||||
|
||||
- CR2 kills only the processes under the run's `paperclip-runnerd` whose arguments match the adapter's
|
||||
provider command. For Claude (ACPX) that is `--output-format stream-json`. For OpenCode it is
|
||||
`serve --hostname 127.0.0.1 --port`. The Runner is found by the run id in its listen path.
|
||||
- CR2 is the only class R action that changes anything in a sandbox. It runs one fixed script, only in
|
||||
sandboxes that carry the wave's company and environment labels.
|
||||
- The run id comes from the task's last sample before the fault. Each kill's output is saved to
|
||||
`daytona-runner/provider-kill-<task>-<ms>.json`.
|
||||
- Each CR task must pass `class_r_fault_applied`. The fault must not have failed, CR2 must have killed a
|
||||
provider process, and CR3's stall must have ended. A cell whose fault never applied tested nothing.
|
||||
- Use T1, so that the fault lands while a tool runs, as it did on staging. Use `--concurrency 1` for CR3,
|
||||
because the stall hits every task's plugin RPCs.
|
||||
|
||||
**Sandbox capture.** Every Daytona wave, not only a CR wave, runs the class R probe (`CLASS_R_PROBE`) in
|
||||
every started wave sandbox. Staging's Runner loss also reproduces with no fault at all (W2 re-wake). Its
|
||||
output goes to `results/<wave>/daytona-runner/<label>/<sandbox>.txt`. The probe reads:
|
||||
|
||||
- every process with its start time (`lstart`), elapsed time and kernel wait (`wchan`);
|
||||
- TCP sockets with their owners;
|
||||
- the runtime directory listing, with the size and last write of each file;
|
||||
- the tail of every log, state or transcript file the Runner or provider keeps. This includes
|
||||
`runner-state.json`, the ACPX state files, OpenCode `opencode/log`, and Claude `debug` and `projects`
|
||||
transcripts.
|
||||
|
||||
The probe runs at these times:
|
||||
|
||||
| Label | When |
|
||||
| --- | --- |
|
||||
| `<fault>-before` | Just before the fault, with a 20 s deadline. The fault's `startedAt` is taken after it. |
|
||||
| `<fault>-stopped`, `<fault>-before_start` | CR1 and CR2 with a hold: right after the stop, and just before the next deploy starts. |
|
||||
| `<fault>-ready+<n>s` | Every 15 s for 180 s after the controller is ready again. It stops after 60 s if the task has rested. Staging failed about 40 s after adoption, so this window covers it. |
|
||||
| `<task>-rest` | When any task rests, pass or fail, before the product releases the sandbox. |
|
||||
| `watch+<n>s` | With `--sandbox-watch-seconds <n>` (10–600), on that clock for the whole wave. This catches a Runner exit that no fault announces. |
|
||||
| `wave-end` | Before the sweep. |
|
||||
|
||||
Before PR #20907, a remote Runner wrote its stdout and stderr to `/dev/null`. Only a panic or a graceful
|
||||
error exit reached `diagnostics/runnerd.stderr.log`. Since #20907, its stderr goes to the bounded
|
||||
`diagnostics/runnerd.process-stderr.log`, and the probe reads both files. The probe never stops, starts or
|
||||
deletes a sandbox. An archived or stopped sandbox is recorded by its state alone.
|
||||
|
||||
**Results on master `6154b9485e` with #20907** (image `paperclip-daytona-runner@sha256:39df7406…`):
|
||||
|
||||
| Wave | Cell | Result |
|
||||
| --- | --- | --- |
|
||||
| `w-20261111T192658-s5201` | W2 re-wake, `--faults K0 --daytona-warm`, Codex, 10 tasks | `runner_remote_process_identity_unavailable` twice, with no fault. The follow-up run launched a new runnerd on the warm session state. That runnerd stopped itself: `paperclip-runnerd: controller sequence must be contiguous: expected 13, received 11`. The new controller's journal ended before the predecessor's `runner.drain` and `runner.suspend` (seq 11 and 12). |
|
||||
| `w-20261111T202055-s9502` | CR1 75 s and CR2, OpenCode and Claude | No refusal. The Runner survived every outage. A killed provider failed the turn with a clear code: Claude `ACP agent reported a terminal connection failure`, OpenCode `opencode_server_exited`. |
|
||||
| `w-20261111T204144-s9503` | CR1 360 s twice, OpenCode and Claude | No refusal. The Runner and the provider survived both outages. |
|
||||
| `w-20261111T210055-s9504` | CR3 45 s, OpenCode and Claude | Not the staging code, a separate failure. `runnerd process supervision failed: RPC call "environmentExecute" timed out after 40000ms` failed the run as `notification_transport_failed`. The close path suspended a healthy Runner (`lifecycle: suspended`, no stderr). Retry worked. |
|
||||
|
||||
## Absolute symlink in the workspace (F1)
|
||||
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
CR1_DEFAULT_HOLD_SECONDS, PROVIDER_KILL_SCRIPT, classRHoldMs, classRRedeploys, findDaytonaPluginWorker, killedPidsFromOutput,
|
||||
parseClassROptions, providerKillCommand, runPluginStall, watchLabels, withClassRCheck, type ClassRRecord,
|
||||
} from "./class-r.js";
|
||||
import { executeInWaveSandboxes } from "./daytona-runner-logs.js";
|
||||
import { parseCommand } from "./config.js";
|
||||
import { redeployWave } from "./redeploy.js";
|
||||
import { planTasks } from "./scheduler.js";
|
||||
import { CLASS_R_PROBE } from "./daytona-runner-logs.js";
|
||||
import type { WaveOptions } from "./types.js";
|
||||
|
||||
const IMAGE = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"a".repeat(64)}`;
|
||||
const RUN = "c4373803-ef67-4dcc-ad20-ea330aac80c1";
|
||||
const COMPANY = "11111111-1111-4111-8111-111111111111";
|
||||
const ENVIRONMENT = "22222222-2222-4222-8222-222222222222";
|
||||
|
||||
describe("class R options", () => {
|
||||
const base = { env: "daytona" as const, stormMinutes: undefined, crashPoint: undefined };
|
||||
it("defaults CR1 to a 6-minute hold, CR2 to no outage and CR3 to a 45 s stall", () => {
|
||||
expect(parseClassROptions({ ...base, faults: ["CR1", "CR2", "CR3"] }, {})).toEqual({
|
||||
cr1HoldMs: [CR1_DEFAULT_HOLD_SECONDS * 1000, CR1_DEFAULT_HOLD_SECONDS * 1000], cr2HoldMs: [0, 0], cr3StallMs: [45_000, 45_000] });
|
||||
});
|
||||
it("takes a seconds value or a range", () => {
|
||||
expect(parseClassROptions({ ...base, faults: ["CR1"] }, { cr1: "300-420" })).toEqual({ cr1HoldMs: [300_000, 420_000] });
|
||||
expect(parseClassROptions({ ...base, faults: ["CR2"] }, { cr2: "90" })).toEqual({ cr2HoldMs: [90_000, 90_000] });
|
||||
});
|
||||
it("refuses a CR3 stall that is not longer than 40 s, and a hold above the outage cap", () => {
|
||||
expect(() => parseClassROptions({ ...base, faults: ["CR3"] }, { cr3: "40" })).toThrow(/cr3-stall-seconds must be between 41/);
|
||||
expect(() => parseClassROptions({ ...base, faults: ["CR1"] }, { cr1: "601" })).toThrow(/between 0 and 600/);
|
||||
});
|
||||
it("needs Daytona, its own fault for each option and no storm or crash point", () => {
|
||||
expect(() => parseClassROptions({ ...base, env: "local", faults: ["CR1"] }, {})).toThrow(/--env daytona/);
|
||||
expect(() => parseClassROptions({ ...base, faults: ["K15"] }, { cr1: "60" })).toThrow(/add CR1 to --faults/);
|
||||
expect(() => parseClassROptions({ ...base, stormMinutes: 10, faults: ["CR1"] }, {})).toThrow(/storm/);
|
||||
expect(() => parseClassROptions({ ...base, faults: ["CR1", "MS1"] }, {})).toThrow(/MS1-MS3/);
|
||||
expect(parseClassROptions({ ...base, faults: ["K1"] }, {})).toEqual({});
|
||||
});
|
||||
it("parses from the command line", () => {
|
||||
const parsed = parseCommand(["--env", "daytona", "--daytona-image", IMAGE, "--adapters", "opencode,claude", "--workloads", "T1",
|
||||
"--faults", "CR1", "--cr1-hold-seconds", "75", "--tasks", "2", "--seed", "1"]);
|
||||
expect("options" in parsed && parsed.options.cr1HoldMs).toEqual([75_000, 75_000]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("class R planning", () => {
|
||||
const options: WaveOptions = { env: "daytona", adapters: ["opencode", "claude"], workloads: ["T1"], faults: ["CR1", "CR2", "CR3"],
|
||||
concurrency: 1, tasks: 6, seed: 7, budgetCents: 3000, pollMs: 2000, cr1HoldMs: [360_000, 360_000], cr2HoldMs: [0, 0], cr3StallMs: [45_000, 45_000] };
|
||||
it("gives each cell its own hold", () => {
|
||||
const holds = Object.fromEntries(planTasks(options).map(task => [task.fault, task.holdMs]));
|
||||
expect(holds).toEqual({ CR1: 360_000, CR2: 0, CR3: 45_000 });
|
||||
});
|
||||
it("draws a hold only for a range", () => {
|
||||
expect(classRHoldMs("CR1", { cr1HoldMs: [300_000, 420_000] }, () => 333_000)).toBe(333_000);
|
||||
expect(classRHoldMs("K2", {}, () => { throw new Error("no draw"); })).toBe(0);
|
||||
});
|
||||
it("redeploys for CR1 and for CR2 with a hold", () => {
|
||||
expect(classRRedeploys("CR1", 0)).toBe(true);
|
||||
expect(classRRedeploys("CR2", 0)).toBe(false);
|
||||
expect(classRRedeploys("CR2", 60_000)).toBe(true);
|
||||
expect(classRRedeploys("CR3", 45_000)).toBe(false);
|
||||
expect(redeployWave({ options: { ...options, faults: ["CR1"] }, tasks: [] })).toBe(true);
|
||||
// A default CR2 keeps the controller up, so it gets no deploy identity; a held CR2 does.
|
||||
const cr2 = (holdMs: number) => planTasks({ ...options, faults: ["CR2"], tasks: 1, cr2HoldMs: [holdMs, holdMs] });
|
||||
expect(redeployWave({ options: { ...options, faults: ["CR2"] }, tasks: cr2(0) })).toBe(false);
|
||||
expect(redeployWave({ options: { ...options, faults: ["CR2"] }, tasks: cr2(60_000) })).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("CR2 provider kill", () => {
|
||||
it("quotes the fixed script and passes only a run id and the adapter's pattern", () => {
|
||||
const command = providerKillCommand(RUN, "opencode");
|
||||
expect(command.startsWith("sh -c '")).toBe(true);
|
||||
expect(command.endsWith(`paperclip-chaos-cr2 '${RUN}' ' serve --hostname 127.0.0.1 --port '`)).toBe(true);
|
||||
expect(PROVIDER_KILL_SCRIPT).toContain("kill -KILL $targets");
|
||||
// A survivor fails the fault instead of being reported as killed.
|
||||
expect(PROVIDER_KILL_SCRIPT.indexOf("exit 5")).toBeLessThan(PROVIDER_KILL_SCRIPT.indexOf("== killed"));
|
||||
// Never the Runner itself.
|
||||
expect(PROVIDER_KILL_SCRIPT).toContain('test "$p" = "$runner" && continue');
|
||||
});
|
||||
it("refuses anything but a run id", () => {
|
||||
expect(() => providerKillCommand("x'; rm -rf /", "claude")).toThrow(/needs a run id/);
|
||||
});
|
||||
it("reads the killed PIDs from the script's output", () => {
|
||||
expect(killedPidsFromOutput("== runner 484\n== killed 603 741 at 2026-11-11T03:00:29.000Z\n")).toEqual([603, 741]);
|
||||
expect(killedPidsFromOutput("== runner 484\n")).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("CR3 plugin worker", () => {
|
||||
const rows = [
|
||||
{ pid: 10, ppid: 1, command: "node server/src/index.ts" },
|
||||
{ pid: 11, ppid: 10, command: "node /x/packages/plugins/sandbox-providers/daytona/dist/worker.js" },
|
||||
{ pid: 12, ppid: 11, command: "node /x/packages/plugins/sandbox-providers/daytona/dist/child.js" },
|
||||
{ pid: 13, ppid: 10, command: "node /x/packages/plugins/other/dist/worker.js" },
|
||||
{ pid: 20, ppid: 1, command: "node /elsewhere/daytona/dist/worker.js" },
|
||||
];
|
||||
const descendants = (root: number) => {
|
||||
const out: number[] = []; const queue = [root];
|
||||
while (queue.length) { const pid = queue.shift()!; for (const row of rows) if (row.ppid === pid) { out.push(row.pid); queue.push(row.pid); } }
|
||||
return out;
|
||||
};
|
||||
it("finds the server's own Daytona worker, not its children or another server's", () => {
|
||||
expect(findDaytonaPluginWorker(rows, 10, descendants)).toEqual({ pid: 11, command: rows[1]!.command });
|
||||
});
|
||||
it("finds nothing when there is no single worker", () => {
|
||||
expect(findDaytonaPluginWorker(rows.filter(row => row.pid !== 11 && row.pid !== 12), 10, descendants)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("CR3 stall", () => {
|
||||
const rows = [{ pid: 10, ppid: 1, command: "node server" }, { pid: 11, ppid: 10, command: "node /x/daytona/dist/worker.js" }];
|
||||
const base = { serverPid: 10, holdMs: 30, cancelled: () => false, record: () => {}, table: () => rows, sleep: async () => {} };
|
||||
it("starts the stall clock only once the worker itself is stopped", async () => {
|
||||
const thawed: string[] = [];
|
||||
const stall = await runPluginStall({ ...base, freezer: { freeze: () => [11], thaw: () => { thawed.push("thaw"); } } });
|
||||
expect(stall.error).toBeNull();
|
||||
expect(stall.frozenAt).not.toBeNull();
|
||||
expect(stall.thawedAt! - stall.frozenAt!).toBeGreaterThanOrEqual(30);
|
||||
expect(thawed).toEqual(["thaw"]);
|
||||
});
|
||||
it("fails when the worker could not be stopped or is missing", async () => {
|
||||
const missed = await runPluginStall({ ...base, freezer: { freeze: () => [], thaw: () => {} } });
|
||||
expect(missed.error).toMatch(/could not stop/);
|
||||
expect(missed.frozenAt).toBeNull();
|
||||
expect((await runPluginStall({ ...base, table: () => rows.slice(0, 1), freezer: { freeze: () => [], thaw: () => {} } })).error).toMatch(/no single/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("CR2 sandbox search", () => {
|
||||
const list = async function* () {
|
||||
yield { id: "a", state: "started", labels: { "paperclip-company-id": COMPANY, "paperclip-environment-id": ENVIRONMENT } };
|
||||
};
|
||||
it("returns at its deadline when a Daytona lookup never answers", async () => {
|
||||
const started = Date.now();
|
||||
const result = await executeInWaveSandboxes({ apiKey: "", companyId: COMPANY, environmentId: ENVIRONMENT, command: "true", secrets: [],
|
||||
done: () => true, client: { list, get: async () => new Promise<never>(() => {}) } as never, deadlineMs: 1_500 });
|
||||
const elapsed = Date.now() - started;
|
||||
expect(result.error).toMatch(/deadline 1500 ms/);
|
||||
expect(elapsed).toBeGreaterThanOrEqual(1_400);
|
||||
expect(elapsed).toBeLessThan(5_000);
|
||||
});
|
||||
it("never starts a command once the deadline has passed, even when the lookup answers late", async () => {
|
||||
let commands = 0;
|
||||
const sandbox = { process: { executeCommand: async () => { commands += 1; return { exitCode: 0, result: "" }; } } };
|
||||
const get = () => new Promise(resolve => setTimeout(() => resolve(sandbox), 1_800));
|
||||
const result = await executeInWaveSandboxes({ apiKey: "", companyId: COMPANY, environmentId: ENVIRONMENT, command: "true", secrets: [],
|
||||
done: () => true, client: { list, get } as never, deadlineMs: 1_500 });
|
||||
expect(result.error).toMatch(/deadline/);
|
||||
await new Promise(resolve => setTimeout(resolve, 600));
|
||||
expect(commands).toBe(0);
|
||||
expect(result.attempts).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("class R evidence", () => {
|
||||
it("watches the window staging failed in", () => {
|
||||
const labels = watchLabels("f01");
|
||||
expect(labels[0]).toEqual({ label: "f01-ready+0s", atMs: 0 });
|
||||
expect(labels.some(entry => entry.atMs === 45_000)).toBe(true);
|
||||
expect(labels.at(-1)!.atMs).toBe(180_000);
|
||||
});
|
||||
it("captures start times, sockets and provider files", () => {
|
||||
expect(CLASS_R_PROBE).toContain("lstart");
|
||||
expect(CLASS_R_PROBE).toContain("ss -tanp");
|
||||
expect(CLASS_R_PROBE).toContain("opencode/log");
|
||||
expect(CLASS_R_PROBE).not.toContain("'");
|
||||
});
|
||||
});
|
||||
|
||||
describe("class_r_fault_applied", () => {
|
||||
const oracle: { verdict: "pass" | "fail"; checks: Array<{ name: string; ok: boolean; detail?: string }> } = { verdict: "pass", checks: [{ name: "brief", ok: true }] };
|
||||
const classR = (record: Partial<ClassRRecord>): ClassRRecord => ({ kind: "CR2", redeploy: false, captures: [], ...record });
|
||||
it("passes a CR2 task only when a provider was killed", () => {
|
||||
const killed = { requestedBy: "t01", kind: "CR2", error: null, classR: classR({ providerKill: { runId: RUN, adapter: "claude", at: 1,
|
||||
sandboxId: "s", killedPids: [603], attempts: [], error: null } }) };
|
||||
expect(withClassRCheck(oracle, { taskId: "t01", fault: "CR2" }, [killed]).verdict).toBe("pass");
|
||||
const missed = { ...killed, classR: classR({ providerKill: { ...killed.classR.providerKill!, killedPids: [], error: "no running run for the task" } }) };
|
||||
const graded = withClassRCheck(oracle, { taskId: "t01", fault: "CR2" }, [missed]);
|
||||
expect(graded.verdict).toBe("fail");
|
||||
expect(graded.checks.at(-1)!.detail).toContain("no running run");
|
||||
});
|
||||
it("passes a CR3 task only when the stall ended", () => {
|
||||
const stall = { workerPid: 11, command: "node", frozenPids: [11], frozenAt: 1_000, thawedAt: 46_000, error: null };
|
||||
expect(withClassRCheck(oracle, { taskId: "t01", fault: "CR3" }, [{ requestedBy: "t01", kind: "CR3", error: null,
|
||||
classR: classR({ kind: "CR3", pluginStall: stall }) }]).checks.at(-1)!.detail).toContain("45000ms");
|
||||
});
|
||||
it("fails a CR task whose fault never fired, and leaves other tasks alone", () => {
|
||||
expect(withClassRCheck(oracle, { taskId: "t01", fault: "CR1" }, []).verdict).toBe("fail");
|
||||
expect(withClassRCheck(oracle, { taskId: "t01", fault: "K2" }, [])).toBe(oracle);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,279 @@
|
||||
/**
|
||||
* Class R cells (CR1–CR3): the staging failure `runner_remote_process_identity_unavailable`
|
||||
* (doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md, class R).
|
||||
*
|
||||
* On staging, 8 of 9 such runs crossed a deploy that kept the controller away for 5–7 minutes. The Runner in
|
||||
* the Daytona sandbox survived and was adopted, but about 40 s later `session.snapshot` failed because the
|
||||
* provider process inside the sandbox was dead or unresponsive ("ACPX request snapshot requires the surviving
|
||||
* provider"). The ninth run had no deploy: a plugin `environmentExecute` RPC stalled for more than 40 s.
|
||||
*
|
||||
* - CR1: a K15 redeploy whose outage is held `--cr1-hold-seconds` (default 360 s), then the next deploy.
|
||||
* - CR2: only the provider process inside the task's sandbox is killed with SIGKILL; the Runner stays alive.
|
||||
* The controller stays up, or with `--cr2-hold-seconds <n>` a K15 redeploy holds it down for `n` seconds and
|
||||
* the provider is killed right after the controller stopped.
|
||||
* - CR3: the server's Daytona plugin worker is stopped (SIGSTOP) for `--cr3-stall-seconds` (default 45 s,
|
||||
* more than 40), so every plugin RPC, `environmentExecute` included, stalls with no deploy.
|
||||
*
|
||||
* Every CR cell also captures the sandbox side (daytona-runner-logs.ts, extended probe): just before the fault,
|
||||
* right after the stop, before the next deploy starts, every few seconds for the first minutes after the
|
||||
* controller is ready again, when the task rests and at wave end. The Runner itself discards its own stdout and
|
||||
* stderr in a remote sandbox (`REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT` sends them to /dev/null), so these captures,
|
||||
* the Runner's durable state and the provider's own files are the only sandbox-side record.
|
||||
*
|
||||
* Harness only: no product code changes. The one in-sandbox mutation is CR2's SIGKILL of the provider
|
||||
* process, run by a fixed script in sandboxes labelled with this wave's company and environment.
|
||||
*/
|
||||
import type { AdapterId, FaultKind, FaultReceipt, WaveOptions } from "./types.js";
|
||||
import { ProcessFreezer, descendantPids, readProcessTable, type ProcessRow } from "./time-faults.js";
|
||||
|
||||
export const CLASS_R_FAULT_KINDS = ["CR1", "CR2", "CR3"] as const;
|
||||
export type ClassRFaultKind = (typeof CLASS_R_FAULT_KINDS)[number];
|
||||
|
||||
export function isClassRFault(kind: string): kind is ClassRFaultKind {
|
||||
return (CLASS_R_FAULT_KINDS as readonly string[]).includes(kind);
|
||||
}
|
||||
|
||||
/** Staging deploys kept the controller away for 5–7 minutes. */
|
||||
export const CR1_DEFAULT_HOLD_SECONDS = 360;
|
||||
/** Staging's ninth run stalled one plugin RPC for more than 40 s. */
|
||||
export const CR3_DEFAULT_STALL_SECONDS = 45;
|
||||
export const CR3_MIN_STALL_SECONDS = 41;
|
||||
/** The controller-outage cap (tests/runner-e2e/controller-outage.ts), as for K15. */
|
||||
export const CLASS_R_MAX_HOLD_SECONDS = 600;
|
||||
|
||||
function parseRange(raw: string, flag: string, min: number, max: number): [number, number] {
|
||||
const match = /^([0-9]+)(?:-([0-9]+))?$/.exec(raw.trim());
|
||||
if (!match) throw new Error(`--${flag} takes <seconds> or <min>-<max> seconds`);
|
||||
const low = Number(match[1]);
|
||||
const high = match[2] === undefined ? low : Number(match[2]);
|
||||
if (low < min || high > max || high < low) throw new Error(`--${flag} must be between ${min} and ${max} seconds, min not above max`);
|
||||
return [low * 1000, high * 1000];
|
||||
}
|
||||
|
||||
/**
|
||||
* `--cr1-hold-seconds`, `--cr2-hold-seconds` and `--cr3-stall-seconds`. Class R cells need Daytona: the fault and
|
||||
* the evidence are both inside the remote sandbox. They do not join a storm, a crash-point wave or MS1–MS3.
|
||||
*/
|
||||
export function parseClassROptions(options: Pick<WaveOptions, "env" | "faults" | "stormMinutes" | "crashPoint">,
|
||||
values: { cr1?: string; cr2?: string; cr3?: string }): Pick<WaveOptions, "cr1HoldMs" | "cr2HoldMs" | "cr3StallMs"> {
|
||||
const kinds = options.faults.filter(isClassRFault);
|
||||
const given = { CR1: values.cr1, CR2: values.cr2, CR3: values.cr3 } as const;
|
||||
const flags = { CR1: "cr1-hold-seconds", CR2: "cr2-hold-seconds", CR3: "cr3-stall-seconds" } as const;
|
||||
for (const kind of CLASS_R_FAULT_KINDS) {
|
||||
if (given[kind] !== undefined && !kinds.includes(kind)) throw new Error(`--${flags[kind]} sets ${kind}; add ${kind} to --faults`);
|
||||
}
|
||||
if (!kinds.length) return {};
|
||||
if (options.env !== "daytona") throw new Error("Class R faults (CR1-CR3) act on the remote Daytona sandbox; run them with --env daytona");
|
||||
if (options.stormMinutes) throw new Error("Class R faults (CR1-CR3) are planned per task; do not combine them with --storm-minutes");
|
||||
if (options.crashPoint) throw new Error("Class R faults (CR1-CR3) do not combine with --crash-point");
|
||||
if (options.faults.some(kind => kind.startsWith("MS"))) throw new Error("Class R faults (CR1-CR3) do not combine with MS1-MS3");
|
||||
return {
|
||||
...(kinds.includes("CR1") ? { cr1HoldMs: parseRange(values.cr1 ?? String(CR1_DEFAULT_HOLD_SECONDS), flags.CR1, 0, CLASS_R_MAX_HOLD_SECONDS) } : {}),
|
||||
...(kinds.includes("CR2") ? { cr2HoldMs: parseRange(values.cr2 ?? "0", flags.CR2, 0, CLASS_R_MAX_HOLD_SECONDS) } : {}),
|
||||
...(kinds.includes("CR3") ? { cr3StallMs: parseRange(values.cr3 ?? String(CR3_DEFAULT_STALL_SECONDS), flags.CR3, CR3_MIN_STALL_SECONDS, CLASS_R_MAX_HOLD_SECONDS) } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
/** The seeded hold of one class R cell; draws only for CR cells, so other seeds keep their plans. */
|
||||
export function classRHoldMs(kind: FaultKind, options: Pick<WaveOptions, "cr1HoldMs" | "cr2HoldMs" | "cr3StallMs">,
|
||||
int: (min: number, max: number) => number): number {
|
||||
const range = kind === "CR1" ? options.cr1HoldMs ?? [CR1_DEFAULT_HOLD_SECONDS * 1000, CR1_DEFAULT_HOLD_SECONDS * 1000]
|
||||
: kind === "CR2" ? options.cr2HoldMs ?? [0, 0]
|
||||
: kind === "CR3" ? options.cr3StallMs ?? [CR3_DEFAULT_STALL_SECONDS * 1000, CR3_DEFAULT_STALL_SECONDS * 1000] : null;
|
||||
if (!range) return 0;
|
||||
return range[0] === range[1] ? range[0] : int(range[0], range[1]);
|
||||
}
|
||||
|
||||
/** CR1, and CR2 with a hold, replace the controller as a deploy does (K15). */
|
||||
/** Class R hooks (main.ts sets them on CR waves): sandbox captures and CR2's provider kill. None may throw. */
|
||||
export interface ClassRHooks {
|
||||
/** Captures every wave sandbox under `label`; returns the label for the receipt. */
|
||||
capture(label: string, deadlineMs?: number): Promise<string>;
|
||||
/** CR2: SIGKILL the provider process of the task's current run inside its sandbox. */
|
||||
killProvider(requestedBy: string): Promise<ProviderKillRecord>;
|
||||
/** After the controller is ready again: keep capturing through the window staging failed in. */
|
||||
watchAfterReady(receipt: FaultReceipt): void;
|
||||
}
|
||||
|
||||
export function classRRedeploys(kind: FaultKind, holdMs: number): boolean {
|
||||
return kind === "CR1" || (kind === "CR2" && holdMs > 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* The provider process each adapter's Runner supervises, matched against the full argument list of the Runner's
|
||||
* descendants. Observed in K15 sandbox captures (w-20261111T022637-s9311): Claude runs through the ACPX sidecar
|
||||
* as `<claude> --output-format stream-json ...`; OpenCode runs as `<opencode> serve --hostname 127.0.0.1 --port N`.
|
||||
*/
|
||||
export const PROVIDER_PROCESS_PATTERN: Record<AdapterId, string> = {
|
||||
claude: "--output-format stream-json",
|
||||
opencode: " serve --hostname 127.0.0.1 --port ",
|
||||
codex: " app-server",
|
||||
};
|
||||
|
||||
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
|
||||
|
||||
/**
|
||||
* CR2's fixed in-sandbox script. Arguments: `$1` the run id the Runner listens for, `$2` the provider pattern.
|
||||
* It finds the `paperclip-runnerd` whose listen path names this run, walks its descendants, SIGKILLs only the
|
||||
* ones whose arguments contain the pattern, and prints the process table with start times before and after.
|
||||
* Exit 3: no Runner for this run here (another task's sandbox). Exit 4: the Runner has no provider process.
|
||||
* Exit 5: a selected process was still alive after the signal; `== killed` is printed only for confirmed deaths.
|
||||
*/
|
||||
export const PROVIDER_KILL_SCRIPT = [
|
||||
"set -u",
|
||||
"run_id=$1; pattern=$2",
|
||||
'runner=""; for p in $(ps -eo pid= 2>/dev/null); do a=$(tr "\\000" " " < /proc/$p/cmdline 2>/dev/null) || continue;'
|
||||
+ ' case "$a" in *paperclip-runnerd*"/connect/$run_id"*) runner=$p; break;; esac; done',
|
||||
'test -n "$runner" || exit 3',
|
||||
'all=$runner; frontier=$runner; while test -n "$frontier"; do next=""; for p in $frontier; do next="$next $(ps -o pid= --ppid "$p" 2>/dev/null | tr "\\n" " ")"; done;'
|
||||
+ ' frontier=$(echo $next); all="$all $frontier"; done',
|
||||
'targets=""; for p in $all; do test "$p" = "$runner" && continue; a=$(tr "\\000" " " < /proc/$p/cmdline 2>/dev/null) || continue;'
|
||||
+ ' case "$a" in *"$pattern"*) targets="$targets $p";; esac; done',
|
||||
'echo "== runner $runner"',
|
||||
'echo "== before $(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)"; ps -o pid,ppid,sid,lstart,etimes,stat,wchan:24,args -p "$(echo $all | tr " " ",")" 2>/dev/null | cut -c1-300',
|
||||
'test -n "$targets" || exit 4',
|
||||
'kill -KILL $targets 2>/dev/null',
|
||||
// Only a process that is gone counts: a failed signal or a survivor fails the fault (exit 5).
|
||||
'alive=""; wait_ticks=0; while test "$wait_ticks" -lt 20; do alive=""; for p in $targets; do'
|
||||
+ ' if kill -0 "$p" 2>/dev/null && ! grep -qs "^State:[[:space:]]*Z" /proc/$p/status; then alive="$alive $p"; fi; done;'
|
||||
+ ' test -z "$alive" && break; wait_ticks=$((wait_ticks + 1)); sleep 0.1; done',
|
||||
'if test -n "$alive"; then echo "== survived$alive"; exit 5; fi',
|
||||
'echo "== killed$targets at $(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)"',
|
||||
"sleep 1",
|
||||
'echo "== after $(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)"; ps -o pid,ppid,sid,lstart,etimes,stat,wchan:24,args -p "$(echo $all | tr " " ",")" 2>/dev/null | cut -c1-300',
|
||||
"exit 0",
|
||||
].join("\n");
|
||||
|
||||
/** The command CR2 sends through the Daytona toolbox: the fixed script, its two arguments single-quoted. */
|
||||
export function providerKillCommand(runId: string, adapter: AdapterId): string {
|
||||
if (!UUID.test(runId)) throw new Error(`CR2 needs a run id, got ${JSON.stringify(runId)}`);
|
||||
const quote = (value: string) => `'${value.replace(/'/g, `'\\''`)}'`;
|
||||
return `sh -c ${quote(PROVIDER_KILL_SCRIPT)} paperclip-chaos-cr2 ${quote(runId)} ${quote(PROVIDER_PROCESS_PATTERN[adapter])}`;
|
||||
}
|
||||
|
||||
/** What one CR2 kill found and did, per sandbox. */
|
||||
export interface ProviderKillRecord {
|
||||
runId: string | null;
|
||||
adapter: AdapterId;
|
||||
at: number;
|
||||
/** The sandbox whose Runner serves the run, and the PIDs killed there. */
|
||||
sandboxId: string | null;
|
||||
killedPids: number[];
|
||||
/** Per sandbox: the script's exit code and output (3 means no Runner for the run there). */
|
||||
attempts: Array<{ sandboxId: string; exitCode: number | null; output: string; error: string | null }>;
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
export function killedPidsFromOutput(output: string): number[] {
|
||||
const line = /^== killed((?:\s+[0-9]+)+) at /m.exec(output);
|
||||
return line ? line[1]!.trim().split(/\s+/).map(Number) : [];
|
||||
}
|
||||
|
||||
/** What CR3 stopped: the Daytona plugin worker and its children, for how long. */
|
||||
export interface PluginStallRecord {
|
||||
workerPid: number | null;
|
||||
command: string | null;
|
||||
frozenPids: number[];
|
||||
frozenAt: number | null;
|
||||
thawedAt: number | null;
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
/** The Daytona sandbox-provider plugin worker among the server's descendants (a forked `node .../worker`). */
|
||||
/**
|
||||
* CR3: stop the server's Daytona plugin worker and its children for the hold, then continue them. The stall clock
|
||||
* starts only once the worker itself is stopped; a worker that is missing or could not be stopped is an error.
|
||||
*/
|
||||
export async function runPluginStall(input: {
|
||||
serverPid: number; holdMs: number; cancelled: () => boolean; record: (event: string, data: Record<string, unknown>) => void;
|
||||
freezer?: Pick<ProcessFreezer, "freeze" | "thaw">; table?: () => ProcessRow[]; sleep?: (ms: number) => Promise<void>;
|
||||
}): Promise<PluginStallRecord> {
|
||||
const stall: PluginStallRecord = { workerPid: null, command: null, frozenPids: [], frozenAt: null, thawedAt: null, error: null };
|
||||
const rows = (input.table ?? readProcessTable)();
|
||||
const worker = findDaytonaPluginWorker(rows, input.serverPid, root => descendantPids(root, rows));
|
||||
if (!worker) return { ...stall, error: "CR3 found no single Daytona plugin worker under the server" };
|
||||
stall.workerPid = worker.pid;
|
||||
stall.command = worker.command;
|
||||
const freezer = input.freezer ?? new ProcessFreezer();
|
||||
const sleep = input.sleep ?? ((ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms)));
|
||||
try {
|
||||
stall.frozenPids = freezer.freeze(worker.pid, true);
|
||||
if (!stall.frozenPids.includes(worker.pid)) {
|
||||
stall.error = `CR3 could not stop the Daytona plugin worker ${worker.pid}`;
|
||||
return stall;
|
||||
}
|
||||
stall.frozenAt = Date.now();
|
||||
input.record("plugin_worker_frozen", { pids: stall.frozenPids, holdMs: input.holdMs });
|
||||
while (Date.now() - stall.frozenAt < input.holdMs) {
|
||||
if (input.cancelled()) { stall.error = "Harness shutdown interrupted the plugin stall"; break; }
|
||||
await sleep(Math.min(1_000, input.holdMs - (Date.now() - stall.frozenAt)));
|
||||
}
|
||||
} finally {
|
||||
freezer.thaw();
|
||||
if (stall.frozenAt !== null) {
|
||||
stall.thawedAt = Date.now();
|
||||
input.record("plugin_worker_thawed", { frozenMs: stall.thawedAt - stall.frozenAt });
|
||||
}
|
||||
}
|
||||
return stall;
|
||||
}
|
||||
|
||||
export function findDaytonaPluginWorker(rows: ReadonlyArray<{ pid: number; ppid: number; command: string }>, serverPid: number,
|
||||
descendants: (root: number) => number[]): { pid: number; command: string } | null {
|
||||
const under = new Set(descendants(serverPid));
|
||||
const matches = rows.filter(row => under.has(row.pid) && /daytona/i.test(row.command) && /\bnode\b|\.m?js\b/.test(row.command)
|
||||
&& !/paperclip-runnerd/.test(row.command));
|
||||
// The worker itself, not a child it spawned: its parent is not another match.
|
||||
const ids = new Set(matches.map(row => row.pid));
|
||||
const roots = matches.filter(row => !ids.has(row.ppid));
|
||||
return roots.length === 1 ? { pid: roots[0]!.pid, command: roots[0]!.command.slice(0, 300) } : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* After the controller is ready again, staging failed about 40 s later. Capture every this many ms until the
|
||||
* window ends or the task rests, so the moment the provider is found dead lands between two captures.
|
||||
*/
|
||||
export const CLASS_R_WATCH_INTERVAL_MS = 15_000;
|
||||
export const CLASS_R_WATCH_WINDOW_MS = 180_000;
|
||||
/** The capture right before a fault runs while the controller is still up; it must not delay the fault long. */
|
||||
export const CLASS_R_PRE_FAULT_DEADLINE_MS = 20_000;
|
||||
|
||||
/** The labels of one watch series: `<fault>-ready+<seconds>s`. */
|
||||
export function watchLabels(faultId: string, windowMs = CLASS_R_WATCH_WINDOW_MS, intervalMs = CLASS_R_WATCH_INTERVAL_MS): Array<{ label: string; atMs: number }> {
|
||||
const out: Array<{ label: string; atMs: number }> = [];
|
||||
for (let at = 0; at <= windowMs; at += intervalMs) out.push({ label: `${faultId}-ready+${Math.round(at / 1000)}s`, atMs: at });
|
||||
return out;
|
||||
}
|
||||
|
||||
/** A class R fault's receipt: what the fault itself did, beside the restart fields. */
|
||||
export interface ClassRRecord {
|
||||
kind: ClassRFaultKind;
|
||||
redeploy: boolean;
|
||||
providerKill?: ProviderKillRecord;
|
||||
pluginStall?: PluginStallRecord;
|
||||
captures: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* CR2 and CR3 only test something when their fault applied: a provider was killed, or the plugin worker was
|
||||
* stopped for the whole stall. A CR task without that fails `class_r_fault_applied`.
|
||||
*/
|
||||
export function withClassRCheck<T extends { verdict: "pass" | "fail"; checks: Array<{ name: string; ok: boolean; detail?: string }> }>(
|
||||
oracle: T, plan: { taskId: string; fault: string }, faults: ReadonlyArray<{ requestedBy: string; kind: string; classR?: ClassRRecord; error: string | null }>): T {
|
||||
if (!isClassRFault(plan.fault)) return oracle;
|
||||
const own = faults.filter(fault => fault.requestedBy === plan.taskId && fault.kind === plan.fault);
|
||||
let ok = own.length > 0 && own.every(fault => fault.error === null);
|
||||
let detail = own.length ? own.map(fault => fault.error ? `error=${fault.error}` : "ok").join(",") : "fault never fired";
|
||||
if (plan.fault === "CR2") {
|
||||
const kills = own.map(fault => fault.classR?.providerKill);
|
||||
ok &&= kills.length > 0 && kills.every(kill => (kill?.killedPids.length ?? 0) > 0 && kill?.error === null);
|
||||
detail += `; killed ${kills.map(kill => kill?.killedPids.join("+") || `none(${kill?.error ?? "?"})`).join(",")}`;
|
||||
}
|
||||
if (plan.fault === "CR3") {
|
||||
const stalls = own.map(fault => fault.classR?.pluginStall);
|
||||
ok &&= stalls.length > 0 && stalls.every(stall => stall?.frozenAt != null && stall.thawedAt != null);
|
||||
detail += `; stalled ${stalls.map(stall => stall?.frozenAt != null && stall.thawedAt != null ? `${stall.thawedAt - stall.frozenAt}ms` : `no(${stall?.error ?? "?"})`).join(",")}`;
|
||||
}
|
||||
const checks = [...oracle.checks, { name: "class_r_fault_applied", ok, detail }];
|
||||
return { ...oracle, checks, verdict: checks.every(check => check.ok) ? "pass" : "fail" };
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import { parseK15DeployShape, parseK15HoldSeconds } from "./redeploy.js";
|
||||
import { validateRecoveryStep } from "./recovery-step-faults.js";
|
||||
import { parseLongTurnMinutes } from "./long-turn.js";
|
||||
import { isRewakeWorkload } from "./rewake.js";
|
||||
import { parseClassROptions } from "./class-r.js";
|
||||
import { parseTimeFaultOptions } from "./time-faults-config.js";
|
||||
import { parseStorageTrigger } from "./storage-trigger.js";
|
||||
import { parseNativeTrigger, parseNativeTriggerOutage } from "./native-trigger.js";
|
||||
@@ -145,13 +146,17 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
"tm1-hold-seconds": { type: "string" },
|
||||
"fail-recovery-step": { type: "string" },
|
||||
"long-turn-minutes": { type: "string" },
|
||||
"sandbox-watch-seconds": { type: "string" },
|
||||
"cr1-hold-seconds": { type: "string" },
|
||||
"cr2-hold-seconds": { type: "string" },
|
||||
"cr3-stall-seconds": { type: "string" },
|
||||
},
|
||||
});
|
||||
const keep = values.keep === true;
|
||||
const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000);
|
||||
const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000);
|
||||
if (values.replay !== undefined) {
|
||||
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "k15-overlap-seconds", "k15-fresh-home", "tm1-hold-seconds", "fail-recovery-step", "long-turn-minutes"] as const) {
|
||||
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "k15-overlap-seconds", "k15-fresh-home", "tm1-hold-seconds", "fail-recovery-step", "long-turn-minutes", "cr1-hold-seconds", "cr2-hold-seconds", "cr3-stall-seconds", "sandbox-watch-seconds"] as const) {
|
||||
if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`);
|
||||
}
|
||||
return { mode: "replay", ref: parseReplayRef(values.replay), keep,
|
||||
@@ -251,6 +256,10 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
if (baseline && (stormMinutes || crashPoint)) throw new Error("--faults K0 plans cells with no fault; a storm or crash-point wave has its own faults");
|
||||
Object.assign(options, validateRecoveryStep(options, values["fail-recovery-step"]));
|
||||
Object.assign(options, parseLongTurnMinutes(values["long-turn-minutes"]));
|
||||
if (values["sandbox-watch-seconds"] !== undefined) {
|
||||
if (env !== "daytona") throw new Error("--sandbox-watch-seconds captures Daytona sandboxes; run it with --env daytona");
|
||||
options.sandboxWatchMs = integer(values["sandbox-watch-seconds"], "sandbox-watch-seconds", 30, 10, 600) * 1000;
|
||||
}
|
||||
// The matrix planner builds its own task plans; a long turn would silently drop out of them.
|
||||
if (options.longTurnMinutes && values.matrix) throw new Error("--long-turn-minutes is not a --matrix option; run it as a planned wave");
|
||||
if (providerFaults.length) {
|
||||
@@ -309,6 +318,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"], values["rr-stop-on"]));
|
||||
Object.assign(options, parseK15HoldSeconds(options, values["k15-hold-seconds"]));
|
||||
Object.assign(options, parseK15DeployShape(options, values["k15-overlap-seconds"], values["k15-fresh-home"] === true));
|
||||
Object.assign(options, parseClassROptions(options, { cr1: values["cr1-hold-seconds"], cr2: values["cr2-hold-seconds"], cr3: values["cr3-stall-seconds"] }));
|
||||
const upgradeFrom = values["upgrade-from"];
|
||||
if (options.faults.includes("K14") !== (upgradeFrom !== undefined)) {
|
||||
throw new Error("K14 (upgrade mid-run) and --upgrade-from <older checkout> go together: pass both or neither");
|
||||
|
||||
@@ -29,6 +29,36 @@ export const RUNNER_LOG_PROBE = [
|
||||
+ ' echo "== file $f $(stat -c "%s bytes, modified %y" "$f" 2>/dev/null)"; tail -c 24576 "$f"; echo; done',
|
||||
].join("; ");
|
||||
|
||||
/**
|
||||
* Class R (class-r.ts): what killed or wedged the provider. On top of the Runner probe:
|
||||
* - every process with its start time (`lstart`) and what it is blocked on (`wchan`), so a replaced process
|
||||
* shows a new start time and a wedged one shows its wait;
|
||||
* - TCP sockets with their owners, so a provider blocked on a Paperclip callback shows an open connection;
|
||||
* - the Runner's runtime directory listing (size and last write of each file);
|
||||
* - the tail of every log, transcript or state file the Runner and the provider keep under the runtime root and
|
||||
* the usual provider log homes (OpenCode `opencode/log`, Claude `debug` and `projects` transcripts, ACPX state).
|
||||
* A remote Runner's own stdout and stderr go to /dev/null, so these files are the only sandbox-side record.
|
||||
*/
|
||||
export const CLASS_R_PROBE = [
|
||||
'echo "== observed_at $(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)"',
|
||||
'echo "== uptime"; cat /proc/uptime 2>/dev/null',
|
||||
'echo "== processes"; ps -eo pid,ppid,sid,lstart,etimes,stat,wchan:24,rss,args 2>/dev/null | cut -c1-400 | head -n 160',
|
||||
'echo "== sockets"; (ss -tanp 2>/dev/null || netstat -tanp 2>/dev/null) | head -n 80',
|
||||
'echo "== memory"; free -m 2>/dev/null',
|
||||
'echo "== disk"; df -h / /tmp 2>/dev/null',
|
||||
'echo "== oom"; dmesg 2>/dev/null | grep -iE "oom|killed process" | tail -n 20',
|
||||
// Every Runner runtime root: the shared checkout's and each isolated task workspace's (`.paperclip-workspaces/<id>`).
|
||||
'roots=$(find /home/daytona -xdev -maxdepth 5 -type d -name .paperclip-runtime -prune -print 2>/dev/null | head -n 20)',
|
||||
'echo "== runtime roots"; echo "$roots"',
|
||||
'echo "== runtime files"; for root in $roots; do find "$root" -xdev -maxdepth 9 \\( -name node_modules -o -name provider-pack \\) -prune -o -type f'
|
||||
+ ' -printf "%TY-%Tm-%TdT%TH:%TM:%TS %10s %p\\n" 2>/dev/null; done | sort | tail -n 200',
|
||||
'for f in $(find $roots /tmp /home/daytona/.local/share/opencode /home/daytona/.claude -xdev -maxdepth 10'
|
||||
+ ' \\( -name node_modules -o -name provider-pack \\) -prune -o -type f'
|
||||
+ ' \\( -name "*.log" -o -name "*.stderr" -o -name "*state*.json" -o -name "*.identity"'
|
||||
+ ' -o -path "*opencode/log/*" -o -path "*/debug/*.txt" -o -path "*/projects/*.jsonl" \\) -mmin -240 -print 2>/dev/null | head -n 60); do'
|
||||
+ ' echo "== file $f $(stat -c "%s bytes, modified %y" "$f" 2>/dev/null)"; tail -c 32768 "$f"; echo; done',
|
||||
].join("; ");
|
||||
|
||||
/** The whole capture (client, listing and every probe) gets this long; the outage it runs in must not grow. */
|
||||
export const RUNNER_LOG_CAPTURE_DEADLINE_MS = 45_000;
|
||||
|
||||
@@ -66,7 +96,10 @@ export function probeableSandbox(sandbox: DaytonaSandboxRow): boolean {
|
||||
export async function captureWaveRunnerLogs(input: {
|
||||
apiKey: string; companyId: string; environmentId: string; outDir: string; label: string; secrets: readonly string[];
|
||||
client?: ProbeClient; timeoutSeconds?: number; deadlineMs?: number;
|
||||
/** The fixed probe to run: the Runner probe (default) or the class R probe. */
|
||||
probe?: "runner" | "class_r";
|
||||
}): Promise<RunnerLogCapture> {
|
||||
const probe = input.probe === "class_r" ? CLASS_R_PROBE : RUNNER_LOG_PROBE;
|
||||
// An unusable label must not throw out of the restart path: fall back to a fixed directory and record why.
|
||||
let label = "capture";
|
||||
let labelError: string | null = null;
|
||||
@@ -118,7 +151,9 @@ export async function captureWaveRunnerLogs(input: {
|
||||
try {
|
||||
const sandbox = await client.get(row.id);
|
||||
if (!live()) { if (!expired) entry.error = "deadline"; continue; }
|
||||
const response = await sandbox.process.executeCommand(`sh -c '${RUNNER_LOG_PROBE.replace(/'/g, `'\\''`)}'`, undefined, {}, input.timeoutSeconds ?? 20);
|
||||
// The class R probe reads more files; give it longer, still inside the shared capture deadline.
|
||||
const response = await sandbox.process.executeCommand(`sh -c '${probe.replace(/'/g, `'\\''`)}'`, undefined, {},
|
||||
input.timeoutSeconds ?? (input.probe === "class_r" ? 40 : 20));
|
||||
// Evidence that arrives after the deadline was taken after the outage window: never write it.
|
||||
if (!live()) { if (!expired) entry.error = "deadline"; continue; }
|
||||
entry.exitCode = response.exitCode;
|
||||
@@ -135,3 +170,59 @@ export async function captureWaveRunnerLogs(input: {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export interface SandboxCommandAttempt { sandboxId: string; exitCode: number | null; output: string; error: string | null }
|
||||
|
||||
/**
|
||||
* Runs one fixed command (a fixed script with validated arguments, built by the caller) in the started sandboxes
|
||||
* of this wave, one at a time, until `done` accepts a result. Only CR2 (class-r.ts) uses it; its script acts only
|
||||
* in the sandbox whose Runner serves the task's run. Failures are recorded, never thrown.
|
||||
*
|
||||
* The whole search shares one deadline (default 60 s), because CR2 runs it inside a fault, sometimes while the
|
||||
* controller is down. Once the deadline passes no new command starts, and the result reports `deadline`.
|
||||
*/
|
||||
export async function executeInWaveSandboxes(input: {
|
||||
apiKey: string; companyId: string; environmentId: string; command: string; secrets: readonly string[];
|
||||
done: (attempt: SandboxCommandAttempt) => boolean;
|
||||
client?: ProbeClient; timeoutSeconds?: number; deadlineMs?: number;
|
||||
}): Promise<{ attempts: SandboxCommandAttempt[]; error: string | null }> {
|
||||
const attempts: SandboxCommandAttempt[] = [];
|
||||
const deadlineMs = input.deadlineMs ?? 60_000;
|
||||
const deadlineAt = Date.now() + deadlineMs;
|
||||
const remainingSeconds = () => Math.floor((deadlineAt - Date.now()) / 1000);
|
||||
let timer: NodeJS.Timeout | undefined;
|
||||
const deadline = new Promise<"deadline">(resolve => { timer = setTimeout(() => resolve("deadline"), deadlineMs); });
|
||||
let expired = false;
|
||||
const search = async (): Promise<string | null> => {
|
||||
const client = input.client ?? await createRemoteFixtureClient(input.apiKey) as unknown as ProbeClient;
|
||||
const wave = { companyId: input.companyId, environmentId: input.environmentId };
|
||||
const rows: DaytonaSandboxRow[] = [];
|
||||
for await (const row of client.list({ labels: { "paperclip-company-id": wave.companyId, "paperclip-environment-id": wave.environmentId } })) rows.push(row);
|
||||
for (const row of selectWaveSandboxes(rows, wave).filter(probeableSandbox)) {
|
||||
// Never start a command (a kill) after the deadline.
|
||||
if (expired || remainingSeconds() < 1) return `deadline ${deadlineMs} ms exceeded`;
|
||||
let attempt: SandboxCommandAttempt;
|
||||
try {
|
||||
const sandbox = await client.get(row.id);
|
||||
if (expired || remainingSeconds() < 1) return `deadline ${deadlineMs} ms exceeded`;
|
||||
const response = await sandbox.process.executeCommand(input.command, undefined, {},
|
||||
Math.min(input.timeoutSeconds ?? 30, remainingSeconds()));
|
||||
attempt = { sandboxId: row.id, exitCode: response.exitCode,
|
||||
output: redactKnownSecrets(String(response.result ?? ""), input.secrets).slice(-16_384), error: null };
|
||||
} catch (error) {
|
||||
attempt = { sandboxId: row.id, exitCode: null, output: "", error: error instanceof Error ? error.message : String(error) };
|
||||
}
|
||||
if (expired) return `deadline ${deadlineMs} ms exceeded`;
|
||||
attempts.push(attempt);
|
||||
if (input.done(attempt)) return null;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
try {
|
||||
const outcome = await Promise.race([search().catch((error: unknown) => error instanceof Error ? error.message : String(error)), deadline]);
|
||||
if (outcome === "deadline") { expired = true; return { attempts: [...attempts], error: `deadline ${deadlineMs} ms exceeded` }; }
|
||||
return { attempts, error: outcome };
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import { WORKLOAD_TRIGGERS } from "./journeys.js";
|
||||
import { withProviderFaultRules } from "./provider-faults.js";
|
||||
import { createRng, k4HoldRange, TRIGGER_DELAY_RANGE_MS } from "./scheduler.js";
|
||||
import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js";
|
||||
import { classRHoldMs, isClassRFault } from "./class-r.js";
|
||||
import { isMultiServerFault, multiServerHoldMs } from "./multi-server.js";
|
||||
import { isRepeatedRestartFault, planRestartSteps } from "./repeated-restarts.js";
|
||||
import { workloadDefinitions } from "./workloads.js";
|
||||
@@ -44,7 +45,7 @@ export function matrixMayLaunch(next: Pick<TaskPlan, "fault">, inFlight: Readonl
|
||||
return inFlight.every(task => (task.fault === "K0") === baseline);
|
||||
}
|
||||
|
||||
export function planMatrixTasks(options: Pick<WaveOptions, "seed" | "workloads" | "adapters" | "faults" | "providerFaults" | "k4HoldMaxMs" | "k15HoldMs" | "tm1HoldMs" | "faultAfterMs" | "rrGapMs">): TaskPlan[] {
|
||||
export function planMatrixTasks(options: Pick<WaveOptions, "seed" | "workloads" | "adapters" | "faults" | "providerFaults" | "k4HoldMaxMs" | "k15HoldMs" | "tm1HoldMs" | "faultAfterMs" | "rrGapMs" | "cr1HoldMs" | "cr2HoldMs" | "cr3StallMs">): TaskPlan[] {
|
||||
const rng = createRng((options.seed ^ 0x6a6d7478) >>> 0);
|
||||
return matrixCells(options).map((cell, index) => {
|
||||
const definition = workloadDefinitions[cell.workload];
|
||||
@@ -62,7 +63,8 @@ export function planMatrixTasks(options: Pick<WaveOptions, "seed" | "workloads"
|
||||
holdMs: cell.fault === "K4" ? rng.int(...k4HoldRange(options.k4HoldMaxMs))
|
||||
: isTimeFault(cell.fault) ? timeFaultHoldMs(cell.fault, rng.int, options.tm1HoldMs)
|
||||
: isMultiServerFault(cell.fault) ? multiServerHoldMs(cell.fault, rng.int)
|
||||
: cell.fault === "K15" ? k15HoldMs(options.k15HoldMs, rng.int) : 0,
|
||||
: cell.fault === "K15" ? k15HoldMs(options.k15HoldMs, rng.int)
|
||||
: isClassRFault(cell.fault) ? classRHoldMs(cell.fault, options, rng.int) : 0,
|
||||
...(isRepeatedRestartFault(cell.fault) ? { restartSteps: planRestartSteps(cell.fault, rng.int, options.rrGapMs) } : {}),
|
||||
followUpAfterMs: definition.followUpWindowMs ? rng.int(definition.followUpWindowMs[0], definition.followUpWindowMs[1]) : null,
|
||||
nonce: rng.hex(8),
|
||||
|
||||
@@ -42,7 +42,8 @@ import { matrixMayLaunch, planMatrixWave } from "./journey-matrix.js";
|
||||
import { redeployWave } from "./redeploy.js";
|
||||
import { withT1CrossingCheck } from "./workloads.js";
|
||||
import { parseRecoveryStepReceipts, withRecoveryStepCheck } from "./recovery-step-faults.js";
|
||||
import { captureWaveRunnerLogs } from "./daytona-runner-logs.js";
|
||||
import { captureWaveRunnerLogs, executeInWaveSandboxes } from "./daytona-runner-logs.js";
|
||||
import { isClassRFault, killedPidsFromOutput, providerKillCommand, watchLabels, withClassRCheck, type ProviderKillRecord } from "./class-r.js";
|
||||
import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurrentBuild, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js";
|
||||
import { chaosLifecycleOverrides, seedChaosLab } from "./seed.js";
|
||||
import { ChaosSupervisor } from "./supervisor.js";
|
||||
@@ -261,10 +262,15 @@ async function main(): Promise<number> {
|
||||
let sweeper: DaytonaWaveSweeper | null = null;
|
||||
let sandboxSweep: DaytonaSweepResult | null = null;
|
||||
let runnerLogTarget: { companyId: string; environmentId: string } | null = null;
|
||||
const captureRunnerLogs = async (label: string) => {
|
||||
// CR1–CR3 get the class R hooks (class-r.ts). Every Daytona wave captures its sandboxes with the class R probe
|
||||
// (daytona-runner-logs.ts): staging's Runner loss also reproduces with no fault at all (W2 re-wake).
|
||||
const classRWave = plan.tasks.some(task => isClassRFault(task.fault) || (task.replayFaults?.some(fault => isClassRFault(fault.kind)) ?? false));
|
||||
const sandboxEvidence = plan.options.env === "daytona";
|
||||
const captureRunnerLogs = async (label: string, deadlineMs?: number) => {
|
||||
if (!runnerLogTarget) return;
|
||||
const capture = await captureWaveRunnerLogs({ apiKey: process.env.DAYTONA_API_KEY!, ...runnerLogTarget,
|
||||
outDir: path.join(waveDir, "daytona-runner"), label, secrets: knownSecretValues() });
|
||||
outDir: path.join(waveDir, "daytona-runner"), label, secrets: knownSecretValues(),
|
||||
probe: "class_r", ...(deadlineMs ? { deadlineMs } : {}) });
|
||||
timeline.add("runner_logs_captured", { label: capture.label, error: capture.error,
|
||||
sandboxes: capture.sandboxes.map(sandbox => `${sandbox.id}:${sandbox.state ?? "?"}${sandbox.file ? ":probed" : ""}${sandbox.error ? `:error=${sandbox.error.slice(0, 120)}` : ""}`) });
|
||||
};
|
||||
@@ -318,6 +324,58 @@ async function main(): Promise<number> {
|
||||
timeline.add("wave_spend", { spend });
|
||||
console.info(`Wave spend: ${JSON.stringify(spend)}`);
|
||||
};
|
||||
// CR2 and the class R watch read the task's latest run from the observers (set once they exist).
|
||||
let classRObservers: TaskObserver[] = [];
|
||||
const killTaskProvider = async (owned: { companyId: string; environmentId: string }, taskId: string): Promise<ProviderKillRecord> => {
|
||||
const observer = classRObservers.find(candidate => candidate.plan.taskId === taskId);
|
||||
const adapter = observer?.plan.adapter ?? "claude";
|
||||
// The latest sample was taken just before the fault (snapshotBeforeFault); the server may already be down.
|
||||
const runs = [...(observer?.record.samples ?? [])].reverse().find(sample => sample.runs?.length)?.runs ?? [];
|
||||
const runId = runs.find(run => run.status === "running")?.id ?? null;
|
||||
const record: ProviderKillRecord = { runId, adapter, at: Date.now(), sandboxId: null, killedPids: [], attempts: [], error: null };
|
||||
if (!runId) record.error = "no running run for the task";
|
||||
else {
|
||||
const result = await executeInWaveSandboxes({ apiKey: process.env.DAYTONA_API_KEY!, ...owned, command: providerKillCommand(runId, adapter),
|
||||
secrets: knownSecretValues(), done: attempt => attempt.exitCode === 0 || attempt.exitCode === 4 || attempt.exitCode === 5 });
|
||||
record.attempts = result.attempts;
|
||||
// Exit 0 is printed only when every selected provider process is confirmed gone.
|
||||
const hit = result.attempts.find(attempt => attempt.exitCode === 0);
|
||||
record.sandboxId = hit?.sandboxId ?? null;
|
||||
record.killedPids = hit ? killedPidsFromOutput(hit.output) : [];
|
||||
record.error = result.error ?? (hit ? null : result.attempts.some(attempt => attempt.exitCode === 5)
|
||||
? "a provider process survived the kill" : result.attempts.some(attempt => attempt.exitCode === 4)
|
||||
? "the run's Runner had no provider process" : "no sandbox runs a Runner for the run");
|
||||
}
|
||||
timeline.add("class_r_provider_killed", { taskId, runId, adapter, sandboxId: record.sandboxId, killedPids: record.killedPids, error: record.error,
|
||||
attempts: record.attempts.map(attempt => `${attempt.sandboxId}:${attempt.exitCode ?? attempt.error?.slice(0, 80)}`) });
|
||||
await writeFile(path.join(waveDir, "daytona-runner", `provider-kill-${taskId}-${record.at}.json`), `${JSON.stringify(record, null, 2)}\n`, { mode: 0o600 })
|
||||
.catch(() => undefined);
|
||||
return record;
|
||||
};
|
||||
// After a class R fault the controller is ready again; staging found the provider gone about 40 s later.
|
||||
const watchClassR = async (receipt: FaultReceipt) => {
|
||||
const readyAt = receipt.readyAt ?? Date.now();
|
||||
for (const { label, atMs } of watchLabels(receipt.faultId)) {
|
||||
const owner = classRObservers.find(candidate => candidate.plan.taskId === receipt.requestedBy);
|
||||
// Keep watching through the staging window even after a fast rest; stop early only past it.
|
||||
if (atMs > 60_000 && (owner?.done ?? true)) break;
|
||||
if (interrupted) break;
|
||||
const wait = readyAt + atMs - Date.now();
|
||||
if (wait > 0) await delay(wait);
|
||||
await captureRunnerLogs(label);
|
||||
receipt.classR?.captures.push(label);
|
||||
}
|
||||
};
|
||||
let sandboxWatchStopped = false;
|
||||
const watchSandboxes = async (intervalMs: number) => {
|
||||
const startedAt = Date.now();
|
||||
while (!sandboxWatchStopped && !interrupted) {
|
||||
const started = Date.now();
|
||||
await captureRunnerLogs(`watch+${Math.round((started - startedAt) / 1000)}s`);
|
||||
const wait = intervalMs - (Date.now() - started);
|
||||
for (let left = wait; left > 0 && !sandboxWatchStopped && !interrupted; left -= 1_000) await delay(Math.min(1_000, left));
|
||||
}
|
||||
};
|
||||
let faultFailures = 0;
|
||||
let faultsQueued = 0;
|
||||
try {
|
||||
@@ -382,10 +440,17 @@ async function main(): Promise<number> {
|
||||
onEnvironment: owned => {
|
||||
sweeper = new DaytonaWaveSweeper({ apiKey: process.env.DAYTONA_API_KEY!, companyId: owned.companyId, environmentId: owned.environmentId, api });
|
||||
// K15 on Daytona: the remote Runner's logs, identity and state while the controller is down (daytona-runner-logs.ts).
|
||||
if (redeployWave(plan)) {
|
||||
if (sandboxEvidence) {
|
||||
runnerLogTarget = { companyId: owned.companyId, environmentId: owned.environmentId };
|
||||
supervisor.redeployEvidence = (faultId, phase) => captureRunnerLogs(`${faultId}-${phase}`);
|
||||
if (redeployWave(plan)) supervisor.redeployEvidence = (faultId, phase) => captureRunnerLogs(`${faultId}-${phase}`);
|
||||
// `--sandbox-watch-seconds`: capture on a clock too, for Runner exits that no fault announces (W2 re-wake).
|
||||
if (plan.options.sandboxWatchMs) faultWork.push(watchSandboxes(plan.options.sandboxWatchMs));
|
||||
}
|
||||
if (classRWave) supervisor.classR = {
|
||||
capture: async (label, deadlineMs) => { await captureRunnerLogs(label, deadlineMs); return label; },
|
||||
killProvider: requestedBy => killTaskProvider(owned, requestedBy),
|
||||
watchAfterReady: receipt => { faultWork.push(watchClassR(receipt)); },
|
||||
};
|
||||
} });
|
||||
spendTarget = { api, companyId: lab.companyId };
|
||||
// --db-pressure: contend with the wave's running run rows from outside the server (contention.ts).
|
||||
@@ -399,6 +464,7 @@ async function main(): Promise<number> {
|
||||
: new BoardUi(supervisor.baseUrl, undefined, path.join(waveDir, "ui-retry-diagnostics"));
|
||||
const observers = plan.tasks.map(task => new TaskObserver(task, api, lab, timeline, supervisor, protectedRoots, board));
|
||||
for (const observer of observers) if (isDiskFullFault(observer.plan.fault)) observer.diskFullSeam = diskFullSeam;
|
||||
classRObservers = observers;
|
||||
const pending = [...observers];
|
||||
const launched: TaskObserver[] = [];
|
||||
const active: TaskObserver[] = [];
|
||||
@@ -633,10 +699,14 @@ async function main(): Promise<number> {
|
||||
}
|
||||
}
|
||||
for (let index = active.length - 1; index >= 0; index -= 1) {
|
||||
if (active[index]!.done) { active.splice(index, 1); lastDoneAt = Date.now(); }
|
||||
if (!active[index]!.done) continue;
|
||||
// Daytona: the sandboxes as the task rests, before the product releases them (pass or fail).
|
||||
if (runnerLogTarget) faultWork.push(captureRunnerLogs(`${active[index]!.plan.taskId}-rest`));
|
||||
active.splice(index, 1); lastDoneAt = Date.now();
|
||||
}
|
||||
await delay(1_000);
|
||||
}
|
||||
sandboxWatchStopped = true;
|
||||
await Promise.all(faultWork);
|
||||
if (storageTrigger) {
|
||||
const shortfall = storageTriggerShortfall(storageTrigger.finish(faults));
|
||||
@@ -717,8 +787,10 @@ async function main(): Promise<number> {
|
||||
: withT1CrossingCheck(oracle, observer.plan, artifacts?.documents.brief, faults);
|
||||
// `--fail-recovery-step`: the step failed in the replacement, and no attempt of the task failed (recovery-step-faults.ts).
|
||||
const armedStep = faults.find(receipt => receipt.recoveryStep);
|
||||
const toolOracle = withRecoveryStepCheck(t1Oracle, armedStep?.recoveryStep ? { faultId: armedStep.faultId, ...armedStep.recoveryStep } : null,
|
||||
const stepOracle = withRecoveryStepCheck(t1Oracle, armedStep?.recoveryStep ? { faultId: armedStep.faultId, ...armedStep.recoveryStep } : null,
|
||||
observer.record.faultIds, parseRecoveryStepReceipts(await readFile(recoveryStepReceipts, "utf8").catch(() => "")), artifacts?.runs ?? []);
|
||||
// CR1–CR3: the task's own fault applied (CR2 killed a provider, CR3 stalled the plugin worker; class-r.ts).
|
||||
const toolOracle = withClassRCheck(stepOracle, observer.plan, faults);
|
||||
const grade = gradeTask(observer.record, faults, observer.definition.budgetMs, toolOracle, universalByIssue ? {
|
||||
runMessages: allRunMessages, recoveryActions: universalByIssue.get(observer.record.issueId ?? "")?.recoveryActions ?? [] } : undefined);
|
||||
let evidence: string | null = null;
|
||||
@@ -798,6 +870,7 @@ async function main(): Promise<number> {
|
||||
harnessError = error;
|
||||
timeline.add("harness_error", { error: error instanceof Error ? error.stack ?? error.message : String(error) });
|
||||
} finally {
|
||||
sandboxWatchStopped = true;
|
||||
await stopPressure().catch(error => timeline.add("db_pressure_stop_failed", { error: String(error) }));
|
||||
// Best effort for an interrupted or failed wave; the server may already be down.
|
||||
if (supervisor.serverReady) await recordWaveSpend().catch(() => undefined);
|
||||
|
||||
@@ -21,8 +21,12 @@ import type { WavePlan } from "./types.js";
|
||||
|
||||
/** Whether any server of this wave can be redeployed: a planned, storm or replayed K15. */
|
||||
export function redeployWave(plan: Pick<WavePlan, "options" | "tasks" | "storm">): boolean {
|
||||
return plan.options.faults.includes("K15") || (plan.storm?.faults.some(fault => fault.kind === "K15") ?? false)
|
||||
|| plan.tasks.some(task => task.fault === "K15" || (task.replayFaults?.some(fault => fault.kind === "K15") ?? false));
|
||||
// CR1 always redeploys; CR2 only when it holds the controller down (class-r.ts). Planned holds decide CR2.
|
||||
const redeploys = (kind: string, holdMs: number) => kind === "K15" || kind === "CR1" || (kind === "CR2" && holdMs > 0);
|
||||
return plan.options.faults.some(kind => kind === "K15" || kind === "CR1")
|
||||
|| (plan.storm?.faults.some(fault => redeploys(fault.kind, fault.holdMs)) ?? false)
|
||||
|| plan.tasks.some(task => redeploys(task.fault, task.holdMs)
|
||||
|| (task.replayFaults?.some(fault => redeploys(fault.kind, fault.holdMs)) ?? false));
|
||||
}
|
||||
|
||||
/** `--k15-hold-seconds` upper bound: the controller-outage cap (tests/runner-e2e/controller-outage.ts). */
|
||||
|
||||
@@ -7,6 +7,7 @@ import { isTimeFault, timeFaultHoldMs } from "./time-faults-config.js";
|
||||
import { isMultiServerFault, multiServerHoldMs } from "./multi-server.js";
|
||||
import { isRepeatedRestartFault, planRestartSteps, type RestartStep } from "./repeated-restarts.js";
|
||||
import { k15HoldMs } from "./redeploy.js";
|
||||
import { classRHoldMs, isClassRFault } from "./class-r.js";
|
||||
import { K4_MIN_HOLD_MS, type AdapterId, type FaultKind, type ScheduledFault, type StormFault, type StormPlan, type TaskFaultKind, type TaskPlan, type TriggerId, type WavePlan, type WaveOptions, type WorkloadId } from "./types.js";
|
||||
import type { ReplayRef } from "./config.js";
|
||||
|
||||
@@ -98,7 +99,7 @@ function pinnedFireAfterMs(faultAfterMs: number | undefined, drawn: number): num
|
||||
return faultAfterMs ?? drawn;
|
||||
}
|
||||
|
||||
export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloads" | "adapters" | "faults" | "triggers" | "stormMinutes" | "k4HoldMaxMs" | "k15HoldMs" | "tm1HoldMs" | "faultAfterMs" | "providerFaults" | "diskFullFaults" | "sessionFaults" | "rrGapMs" | "baseline">): TaskPlan[] {
|
||||
export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloads" | "adapters" | "faults" | "triggers" | "stormMinutes" | "k4HoldMaxMs" | "k15HoldMs" | "tm1HoldMs" | "faultAfterMs" | "providerFaults" | "diskFullFaults" | "sessionFaults" | "rrGapMs" | "baseline" | "cr1HoldMs" | "cr2HoldMs" | "cr3StallMs">): TaskPlan[] {
|
||||
const rng = createRng(options.seed);
|
||||
const triggers = options.triggers?.length ? options.triggers : ["time" as const];
|
||||
for (const trigger of triggers) {
|
||||
@@ -131,7 +132,8 @@ export function planTasks(options: Pick<WaveOptions, "seed" | "tasks" | "workloa
|
||||
fireAfterMs: cell.fault === "K13" || cell.fault === "K0" ? 0 : pinnedFireAfterMs(options.faultAfterMs, rng.int(fireMin, fireMax)),
|
||||
holdMs: cell.fault === "K4" ? rng.int(...k4HoldRange(options.k4HoldMaxMs)) : isTimeFault(cell.fault) ? timeFaultHoldMs(cell.fault, rng.int, options.tm1HoldMs)
|
||||
: isMultiServerFault(cell.fault) ? multiServerHoldMs(cell.fault, rng.int)
|
||||
: cell.fault === "K15" ? k15HoldMs(options.k15HoldMs, rng.int) : 0,
|
||||
: cell.fault === "K15" ? k15HoldMs(options.k15HoldMs, rng.int)
|
||||
: isClassRFault(cell.fault) ? classRHoldMs(cell.fault, options, rng.int) : 0,
|
||||
// Drawn only for RR tasks, so the plans of existing seeds do not change.
|
||||
...(isRepeatedRestartFault(cell.fault) ? { restartSteps: planRestartSteps(cell.fault, rng.int, options.rrGapMs) } : {}),
|
||||
followUpAfterMs: definition.followUpWindowMs ? rng.int(definition.followUpWindowMs[0], definition.followUpWindowMs[1]) : null,
|
||||
|
||||
@@ -41,6 +41,7 @@ import type { FaultKind, FaultReceipt, TriggerId } from "./types.js";
|
||||
import { isMultiServerFault, runMultiServerFault, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow } from "./multi-server.js";
|
||||
import { generationHome, nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs, redeployReceipt, type RedeployIdentity } from "./redeploy.js";
|
||||
import { startEdgeProxy, type EdgeProxy } from "./edge-proxy.js";
|
||||
import { CLASS_R_PRE_FAULT_DEADLINE_MS, classRRedeploys, isClassRFault, runPluginStall, type ClassRHooks } from "./class-r.js";
|
||||
|
||||
/** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */
|
||||
const HEALTH_TIMEOUT_MS = (() => {
|
||||
@@ -53,7 +54,10 @@ const STOP_TIMEOUT_MS = 30_000;
|
||||
export const FAULT_SIGNAL: Record<FaultKind, FaultReceipt["signal"]> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM", K15: "SIGTERM",
|
||||
TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL",
|
||||
// MS1 leaves the primary alone; MS2 freezes it; MS3 stops it gracefully (multi-server.ts).
|
||||
MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM" };
|
||||
MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM",
|
||||
// CR1 stops the server like K15; CR2 signals only the sandbox's provider (SIGTERM to the server with a hold);
|
||||
// CR3 stops the Daytona plugin worker, not the server (class-r.ts).
|
||||
CR1: "SIGTERM", CR2: "none", CR3: "SIGSTOP" };
|
||||
|
||||
/** The checkout's commit, for the K14 receipt. */
|
||||
function checkoutSha(root: string): string | null {
|
||||
@@ -177,6 +181,8 @@ export class ChaosSupervisor {
|
||||
/** `--fail-recovery-step`: "<point>:<nth>", armed in fail mode when the first restart fault starts its replacement. */
|
||||
private pendingRecoveryStep: string | null = null;
|
||||
private recoveryStepWatch: NodeJS.Timeout | null = null;
|
||||
/** CR1–CR3 waves: sandbox captures and the provider kill (class-r.ts). */
|
||||
classR: ClassRHooks | null = null;
|
||||
/** Wave secrets, kept across a K14 switch as an in-place upgrade keeps its configuration. */
|
||||
private readonly secrets = { PAPERCLIP_AGENT_JWT_SECRET: secret(), PAPERCLIP_DECISION_SIGNING_SECRET: secret(),
|
||||
PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: secret(), BETTER_AUTH_SECRET: secret() };
|
||||
@@ -493,13 +499,38 @@ export class ChaosSupervisor {
|
||||
this.serverReady = false;
|
||||
this.timeline.add("fault_started", { faultId: receipt.faultId, fault: kind, signal: receipt.signal, loadAvg1m: receipt.loadAvg1m, holdMs, requestedBy, previousPid: previous?.pid });
|
||||
this.log.write(`\n[restart-chaos] ${receipt.faultId} ${kind} ${receipt.signal} hold=${holdMs}ms requestedBy=${requestedBy} at ${new Date().toISOString()}\n`);
|
||||
// CR1 and CR2 with a hold replace the controller as a K15 deploy does (class-r.ts).
|
||||
const redeploys = kind === "K15" || classRRedeploys(kind, holdMs);
|
||||
try {
|
||||
if (!previous?.pid || this.exited(previous)) throw new Error("No live Paperclip server to fault");
|
||||
let outage: ControllerOutageReceipt | undefined;
|
||||
if (isClassRFault(kind)) {
|
||||
if (!this.classR) throw new Error(`${kind} needs the class R hooks of a Daytona wave (main.ts)`);
|
||||
receipt.classR = { kind, redeploy: redeploys, captures: [] };
|
||||
if (kind === "CR2" && holdMs > 0) receipt.signal = "SIGTERM";
|
||||
// The sandbox just before the fault, while the controller still holds it; the fault starts after it.
|
||||
receipt.classR.captures.push(await this.classR.capture(`${receipt.faultId}-before`, CLASS_R_PRE_FAULT_DEADLINE_MS));
|
||||
receipt.startedAt = Date.now();
|
||||
}
|
||||
if (kind === "TM1" || kind === "TM2") await this.freeze(kind, previous, holdMs, receipt);
|
||||
else if (isMultiServerFault(kind)) await this.multiServer(kind, previous, holdMs, receipt);
|
||||
// `--k15-overlap-seconds`: the next deploy boots beside the old server; its hold is the overlap.
|
||||
else if (kind === "K15" && this.deployShape.overlap) await this.rollingRedeploy(previous, holdMs, receipt);
|
||||
else if (kind === "CR3") {
|
||||
// The server keeps running; its Daytona plugin worker is stopped for the hold (class-r.ts).
|
||||
const stall = await runPluginStall({ serverPid: previous.pid, holdMs, cancelled: () => this.shuttingDown,
|
||||
record: (event, data) => this.timeline.add(event, { faultId: receipt.faultId, ...data }) });
|
||||
receipt.classR!.pluginStall = stall;
|
||||
receipt.stoppedAt = stall.frozenAt;
|
||||
receipt.restartedAt = stall.thawedAt;
|
||||
if (stall.error) throw new Error(stall.error);
|
||||
receipt.healthyAt = await this.waitForHealth(previous, receipt.phases);
|
||||
}
|
||||
else if (kind === "CR2" && !redeploys) {
|
||||
// The controller stays up: only the provider inside the task's sandbox dies.
|
||||
receipt.classR!.providerKill = await this.classR!.killProvider(requestedBy);
|
||||
receipt.healthyAt = Date.now();
|
||||
}
|
||||
else outage = await replaceController({
|
||||
requestId: `chaos-${receipt.faultId}-${requestedBy}`,
|
||||
previousPid: previous.pid,
|
||||
@@ -518,6 +549,11 @@ export class ChaosSupervisor {
|
||||
discardCache: async () => {},
|
||||
// K15: capture the sandbox side as soon as the controller is gone (redeployEvidence).
|
||||
...(kind === "K15" && this.redeployEvidence ? { afterStopped: async () => { await this.redeployEvidence?.(receipt.faultId, "stopped"); } } : {}),
|
||||
// CR1/CR2: the sandbox as the controller left it; CR2 then kills the provider while the controller is away.
|
||||
...(receipt.classR ? { afterStopped: async () => {
|
||||
receipt.classR!.captures.push(await this.classR!.capture(`${receipt.faultId}-stopped`));
|
||||
if (kind === "CR2") receipt.classR!.providerKill = await this.classR!.killProvider(requestedBy);
|
||||
} } : {}),
|
||||
// TM3: the server is down; every database timestamp moves back by the age before the replacement starts.
|
||||
...(kind === "TM3" ? { afterStopped: async () => {
|
||||
const timeFault: TimeFaultRecord = { kind };
|
||||
@@ -529,9 +565,10 @@ export class ChaosSupervisor {
|
||||
// K14: the replacement is the current build, on the same database and instance directory.
|
||||
if (kind === "K14") receipt.upgrade = await this.switchToCurrentBuild(receipt.faultId);
|
||||
// K15: the replacement is the next deploy: a new process space and deployment id on the same volume.
|
||||
if (kind === "K15") {
|
||||
if (redeploys) {
|
||||
// After the hold, before the next deploy starts: what its recovery will find in the sandboxes.
|
||||
await this.redeployEvidence?.(receipt.faultId, "before_start");
|
||||
if (receipt.classR) receipt.classR.captures.push(await this.classR!.capture(`${receipt.faultId}-before_start`));
|
||||
else await this.redeployEvidence?.(receipt.faultId, "before_start");
|
||||
receipt.redeploy = await this.advanceDeploy(receipt.faultId);
|
||||
}
|
||||
this.timeline.add("fault_restarting", { faultId: receipt.faultId });
|
||||
@@ -560,6 +597,8 @@ export class ChaosSupervisor {
|
||||
receipt.serverLogOffsetAtReady = this.serverLogOffset();
|
||||
this.timeline.add("fault_ready", { faultId: receipt.faultId, replacementPid: this.child?.pid ?? outage?.replacementPid,
|
||||
downMs: (receipt.restartedAt ?? 0) - (receipt.stoppedAt ?? 0), readyAfterMs: receipt.readyAt - receipt.startedAt, phases: receipt.phases });
|
||||
// Class R: staging failed about 40 s after adoption; watch the sandbox through that window (not awaited).
|
||||
if (receipt.classR) this.classR?.watchAfterReady(receipt);
|
||||
} catch (error) {
|
||||
receipt.error = error instanceof Error ? error.message : String(error);
|
||||
this.timeline.add("fault_failed", { faultId: receipt.faultId, error: receipt.error });
|
||||
|
||||
@@ -178,7 +178,7 @@ export function descendantPids(rootPid: number, table: readonly ProcessRow[]): n
|
||||
return out;
|
||||
}
|
||||
|
||||
function readProcessTable(): ProcessRow[] {
|
||||
export function readProcessTable(): ProcessRow[] {
|
||||
return parseProcessTable(execFileSync("ps", ["-A", "-o", "pid=,ppid=,command="], { encoding: "utf8", maxBuffer: 16 * 1024 * 1024 }));
|
||||
}
|
||||
|
||||
|
||||
@@ -12,8 +12,9 @@ import type { TimeFaultRecord } from "./time-faults.js";
|
||||
* TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage.
|
||||
* RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers.
|
||||
* MS1–MS3 (two servers, one database, multi-server.ts): a second server beside the first.
|
||||
* CR1–CR3 (staging class R, class-r.ts): a long-held redeploy, a provider-only kill in the sandbox, a plugin RPC stall.
|
||||
*/
|
||||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "K15", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const;
|
||||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "K15", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3", "CR1", "CR2", "CR3"] as const;
|
||||
export type FaultKind = (typeof FAULT_KINDS)[number];
|
||||
/** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */
|
||||
export const K4_MIN_HOLD_MS = 120_000;
|
||||
@@ -94,6 +95,14 @@ export interface WaveOptions {
|
||||
failRecoveryStep?: string;
|
||||
/** `--long-turn-minutes`: every task's first turn does this much real tool work before its workload (long-turn.ts). */
|
||||
longTurnMinutes?: number;
|
||||
/** `--sandbox-watch-seconds`: on Daytona, capture every wave sandbox on this clock (daytona-runner-logs.ts class R probe). */
|
||||
sandboxWatchMs?: number;
|
||||
/** `--cr1-hold-seconds`: the CR1 outage range (class-r.ts); absent on a CR1 wave means 360 s. */
|
||||
cr1HoldMs?: [number, number];
|
||||
/** `--cr2-hold-seconds`: a K15 outage around CR2's provider kill; absent or 0 keeps the controller up. */
|
||||
cr2HoldMs?: [number, number];
|
||||
/** `--cr3-stall-seconds`: how long CR3 stops the Daytona plugin worker; absent means 45 s. */
|
||||
cr3StallMs?: [number, number];
|
||||
/** `--tm1-hold-seconds`: the seeded TM1 stall range (time-faults-config.ts); absent uses 95–150 s. */
|
||||
tm1HoldMs?: [number, number];
|
||||
faultAfterMs?: number;
|
||||
@@ -330,6 +339,8 @@ export interface FaultReceipt {
|
||||
repeatedRestart?: RepeatedRestartRecord;
|
||||
/** MS1–MS3: the second server and what happened to the first. */
|
||||
multiServer?: import("./multi-server.js").MultiServerRecord;
|
||||
/** CR1–CR3: the provider kill or plugin stall and the sandbox captures taken around the fault (class-r.ts). */
|
||||
classR?: import("./class-r.js").ClassRRecord;
|
||||
serverLogOffsetAtStart: number;
|
||||
serverLogOffsetAtReady: number | null;
|
||||
error: string | null;
|
||||
|
||||
Reference in new issue
Block a user