diff --git a/doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md b/doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md new file mode 100644 index 00000000000..0e6b0bb4f3c --- /dev/null +++ b/doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md @@ -0,0 +1,86 @@ +# Restart chaos: config and aged-state faults (CF1–CF4) + +Status: CF1–CF3 are implemented in `tests/restart-chaos/config-faults.ts` and `config-wave.ts`. CF4 is designed here but not built. + +## Why + +Every chaos wave starts from a new instance that has default, valid settings. On +2026-11-09, production failures came from state that a new instance never has: + +- **#20269.** A project's explicit setup command named `scripts/provision-worktree.sh`, + a script that had been renamed. Workspace preparation failed before the provider ran. +- **#20361.** An OpenRouter binding selected the responsible user's personal account + default, but that default did not exist. Verification retried forever, and the task + stayed Queued. + +## Oracle (CF1–CF4) + +"The task completes" is not the oracle. A task passes when all of these hold: + +- **Bounded rest.** Within a bound, the task either completes or rests where the task + page names the broken setting. +- **No silent waiting.** No run retries unchanged configuration forever. Nothing stays + Queued without a live owner or a named wait. +- **Restart safety.** The rest survives a K1 or K2 restart. +- **Repair.** After the harness repairs the setting and uses the action the task offers + (card, Retry, or agent setup Retry), the task completes. + +See the README section "Config faults" for the exact checks. + +## CF4: aged instance (design) + +**Goal.** Start the wave on an instance that an older release created. That release saves +tasks and settings, including CF1–CF3 breaks and parked tasks. Then switch the same +database and instance directory to the current build. Grade with the CF oracle. + +**Reuse.** K14 already has the switching machinery: + +- `ChaosSupervisor.create({ upgradeFrom })` starts the server from a prepared older + checkout. +- `supervisor.fault("K14")` stops that server gracefully and starts the current checkout + against the same database, port, secrets and instance home. +- The prepared checkout is made with `git worktree add ` and + `node scripts/agent-preflight.mjs`, as in the K14 section of the README. + +Build CF4 on these. Do not add a second upgrade path. + +**Changes to `config-wave.ts`** (about half a day): + +1. **New flag.** Add `--upgrade-from `. Check that the directory is prepared, the + same way `main.ts preflight` does. +2. **Seed and break on the older build.** The older build runs `seedChaosLab`, creates + each task's resources, applies the break, and launches the task. Use only routes that + both builds have. Where a route differs, use the older build's route. +3. **Barrier.** Wait until every task has finished its broken phase, so it has rested or + reached its bound. The parked state is then saved by the older build. +4. **One switch.** Fire `K14` once for the wave. This replaces each task's own K1/K2. + Re-check the rest on the current build, using the same check as after a restart. +5. **Repair on the current build.** Run repair and the offered action on the current + build, then grade. +6. **Record both commits.** Write both commits into the summary. The K14 receipt already + includes them. + +**What it covers that CF1–CF3 do not:** + +- Older-release run state is resumed by the current build: + - setup cards created by the older build + - lifecycle operations whose old retry policy was "retry every 60 s" + - interruptions projected by older code +- Settings written by older validators that newer validators would reject. A + `refused_on_write` setting can still exist in an aged database. CF4 is the only way to + reach that case without writing directly to the database. + +**Commit choice.** Use the commit one or two releases back. Also use the commit just +before a known fix. For example, the merge base of #20361 makes the old build save a +CF2 queued-without-card state, which the fixed build must then recover. + +**Cost and limits.** + +- A prepared older checkout takes about 3 minutes with the shared preflight cache, plus + about 2 GB of disk. +- Waves stay at 4 tasks or fewer and a concurrency of 2 or less. +- It runs locally only, as K14 does. + +**Not designed here.** Months-old data shapes, such as rows from migrations long since +squashed. A seeded fixture database taken from an older release would cover them. +Capture one only when an upgrade incident shows the need. diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 7466974a64e..994805d9ce5 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -173,6 +173,54 @@ receipt names both commits (`upgrade` in `faults.json`, `upgrade_switched` in th timeline). K14 is local-only, because a Daytona runner must match its image, and it is never part of a storm's default faults. +## Config faults (CF1–CF3, `config-wave.ts`) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts \ + --faults CF1,CF3 --adapters codex --tasks 4 --concurrency 1 --restarts K1,K2 --seed 61 +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF2 --tasks 1 --seed 62 +``` + +A fresh wave only has valid settings. Config faults break one saved setting that refers +to another object. Each task gets its own agent, credential secret and project, so one +task's break never affects another task. The harness breaks the setting through the +board routes a user would use, launches an A1 brief task on that agent and project, and +then checks four things: + +1. **Bounded rest.** Within `--park-seconds` (default 420) the task either completes, or + rests on something that waits for a person: a pending card, an interruption that + offers Retry or a user action, parked agent setup, or a blocked task. An unstarted + Queued run counts as a rest only with a card or parked setup that says why it waits. +2. **Actionable message.** While the task rests, the task page names the broken setting: + the interruption, the execution status, cards, the agent's setup status, or a notice. + A message that appears only in the run's own error text fails as + `message_only_in_run_detail`. +3. **No loop, no ownerless queue.** At most 3 runs start while the setting is broken. + Agent setup does not say it "will retry automatically" while the configuration is + unchanged. No Queued run sits unstarted for more than 3 minutes without a named wait. +4. **Restart and repair.** A K1 or K2 restart fires while the task rests (`--restarts`, + alternating), and the task must rest again. The harness then repairs the setting and + uses the action the task offers: complete the connection card, click Retry, or retry + the agent's setup. The task must then finish within `--repair-seconds`, and the A1 + brief oracle must pass. + +A break that the server refuses on write passes as `refused_on_write`. + +| Fault | Setting broken | +| --- | --- | +| CF1 | Project `workspaceStrategy.provisionCommand` set to `bash ./scripts/provision-worktree.sh`, a script that does not exist (as in #20269) | +| CF2 | OpenCode agent `runtimeConfig.aiConnection` selects the responsible user's personal OpenRouter default, which was never created (as in #20361). Repair connects a personal account | +| CF3 | Seeded walk over `provision_command`, `ai_default` (opencode only), `agent_secret` (secret deleted), `workspace_cwd` (path removed), `base_ref` (missing branch), `model_id` (unknown model). Pin a set with `--references` | + +Waves are capped at 4 tasks and a concurrency of 2. A wave can contain only one +`ai_default` task, because the product has no route that clears a default once a repair +creates it. Results go to `results/-cf/` (`SUMMARY.md`, `.config-grade.json`, +`timeline.jsonl`, `server.log`). The instance is deleted when the wave ends. + +Settings that CF3 does not break, and why, are listed in `CONFIG_REFERENCES_NOT_BROKEN` +and in every wave summary. CF4 (an instance aged by an older release) is designed in +[`doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`](../../doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md). + ## Cloud storage (`--storage s3`, `--storage-faults`) ```sh @@ -386,6 +434,7 @@ re-arms the task's own triggered faults instead of using their offsets. | `scheduler.ts` | Seeded plan: workload × adapter × fault cells and a random fire time per task. Saved to `results//plan.json`. `--replay` reruns one task alone with the same choices and fires the recorded sequence of faults that crossed it (other tasks' faults included) at the same offsets from launch. | | `observer.ts` | Polls each issue and its runs. When a task rests needing Retry it calls `POST /api/issues/:id/retry` once (once more only if a later fault started after that click), with the body `useTaskRetry.ts` sends. Collects T-oracle artifacts and the R6 sweep. | | `grader.ts` | Pure R1–R6 grading over the samples. | +| `config-faults.ts`, `config-wave.ts` | Config faults: the reference catalog, seeded plan and oracle (pure), and the wave runner that breaks, restarts, repairs and grades. | | `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, with their T-oracles. | | `evidence.ts`, `ledger.ts` | Evidence bundle for each failure and the JSONL ledger plus `SUMMARY.md`. | | `daytona-image.ts` | Docker-free extraction of the image's runnerd and provider pack, cached per digest, and the controller/image content-ID check. | diff --git a/tests/restart-chaos/config-faults.test.ts b/tests/restart-chaos/config-faults.test.ts new file mode 100644 index 00000000000..e1bbdc1ecfe --- /dev/null +++ b/tests/restart-chaos/config-faults.test.ts @@ -0,0 +1,178 @@ +import { describe, expect, it } from "vitest"; +import { + CONFIG_REFERENCE_IDS, CONFIG_REFERENCES, RestartGate, gradeConfigTask, isWriteRefusal, parkState, planConfigWave, referencesFor, taskLevelTexts, validateConfigPlan, + type ConfigSnapshot, type ConfigTaskPlan, type ConfigTaskRecord, type ConfigWaveOptions, +} from "./config-faults.js"; +import { parseConfigWaveArgs } from "./config-wave.js"; + +const base: ConfigWaveOptions = { seed: 7, faults: ["CF1", "CF3"], adapters: ["codex"], tasks: 4, concurrency: 1, restarts: ["K1", "K2"], + parkBoundMs: 420_000, repairBudgetMs: 900_000 }; + +function snapshot(at: number, patch: Partial = {}): ConfigSnapshot { + return { at, reachable: true, issueStatus: "in_progress", runs: [], interruption: null, execution: null, cards: [], agent: null, notices: [], ...patch }; +} + +function record(plan: Partial, patch: Partial = {}): ConfigTaskRecord { + return { + plan: { taskId: "t01", index: 0, fault: "CF1", reference: "provision_command", adapter: "codex", restart: "K1", nonce: "abcd1234", ...plan }, + issueId: "issue", agentId: "agent", projectId: "project", launchedAt: 0, refusedOnWrite: null, harnessError: null, + brokenSnapshots: [], parkedAt: null, parkReason: null, restart: null, afterRestartSnapshots: [], afterRestartParked: null, repair: null, + doneAt: null, finalStatus: null, brief: { ok: true, detail: "brief ok" }, ...patch, + }; +} + +const failedRun = (id: string, createdAt: number, error: string) => ({ id, status: "failed", createdAt, startedAt: createdAt, errorCode: "setup_failed", error }); + +describe("planConfigWave", () => { + it("is seeded, walks distinct CF3 references and pins CF2 to opencode", () => { + const tasks = planConfigWave({ ...base, faults: ["CF3"], tasks: 4 }); + expect(planConfigWave({ ...base, faults: ["CF3"], tasks: 4 })).toEqual(tasks); + expect(new Set(tasks.map(task => task.reference)).size).toBe(4); + expect(tasks.every(task => referencesFor("CF3", "codex").includes(task.reference))).toBe(true); + expect(planConfigWave({ ...base, faults: ["CF2"], tasks: 1 })[0]).toMatchObject({ adapter: "opencode", reference: "ai_default" }); + }); + + it("crosses each fault kind with both restart kinds", () => { + const tasks = planConfigWave({ ...base, faults: ["CF1", "CF2"], tasks: 3 }); + expect(tasks.map(task => `${task.fault}/${task.restart}`)).toEqual(["CF1/K1", "CF2/K1", "CF1/K2"]); + }); + + it("offers the account default only to adapters that carry the binding", () => { + expect(referencesFor("CF3", "codex")).not.toContain("ai_default"); + expect(referencesFor("CF3", "opencode")).toEqual([...CONFIG_REFERENCE_IDS]); + }); + + it("refuses a second account-default task in one wave", () => { + expect(() => validateConfigPlan(planConfigWave({ ...base, faults: ["CF2"], tasks: 2 }))).toThrow(/Only one ai_default/); + }); +}); + +describe("write refusals and the restart gate", () => { + it("treats only validation statuses as a product refusal", () => { + expect([400, 422].every(isWriteRefusal)).toBe(true); + expect([403, 404, 409, 429, null].some(isWriteRefusal)).toBe(false); + }); + + it("holds writes while a restart runs and waits for writes in flight", async () => { + const gate = new RestartGate(); + const events: string[] = []; + let finishWrite!: () => void; + const first = gate.write(() => new Promise(resolve => { finishWrite = () => { events.push("write1-end"); resolve(); }; })); + const restart = gate.restart(async () => { events.push("restart"); await Promise.resolve(); events.push("restart-end"); }); + const second = gate.write(async () => { events.push("write2"); }); + await Promise.resolve(); + expect(events).toEqual([]); + finishWrite(); + await Promise.all([first, restart, second]); + expect(events).toEqual(["write1-end", "restart", "restart-end", "write2"]); + }); +}); + +describe("parseConfigWaveArgs", () => { + it("keeps waves small", () => { + expect(() => parseConfigWaveArgs(["--tasks", "5"])).toThrow(/tasks/); + expect(() => parseConfigWaveArgs(["--concurrency", "3"])).toThrow(/concurrency/); + expect(parseConfigWaveArgs(["--faults", "CF2", "--seed", "3"]).options).toMatchObject({ faults: ["CF2"], seed: 3, concurrency: 1 }); + }); +}); + +describe("parkState", () => { + it("does not treat an unstarted queued run without a named wait as a rest", () => { + const queued = snapshot(100_000, { runs: [{ id: "r1", status: "queued", createdAt: 0, startedAt: null, errorCode: null, error: null }] }); + expect(parkState(queued, 100_000, 30_000)).toEqual({ kind: "active", reason: "queued_unstarted" }); + const withCard = { ...queued, cards: [{ id: "c1", kind: "connection_intent", status: "pending", serviceSlug: "openrouter", agentSetup: true }] }; + expect(parkState(withCard, 100_000, 30_000)).toEqual({ kind: "parked", reason: "queued_with_card" }); + }); + + it("waits for runs to settle before calling an interruption a rest", () => { + const failed = snapshot(40_000, { runs: [failedRun("r1", 20_000, "x")], interruption: { state: "failed", canRetry: true, userAction: null, workspaceFailure: null, observedRunId: "r1" } }); + expect(parkState(failed, 40_000, 30_000).kind).toBe("active"); + expect(parkState(failed, 60_000, 30_000)).toEqual({ kind: "parked", reason: "interruption" }); + }); +}); + +describe("gradeConfigTask", () => { + const options = { parkBoundMs: 420_000 }; + const generic = { state: "failed", canRetry: true, userAction: null, workspaceFailure: null, observedRunId: "r1" }; + + it("fails CF1 when the command is named only on the run page (master today)", () => { + const parked = snapshot(90_000, { runs: [failedRun("r1", 10_000, 'Execution workspace provision command "bash ./scripts/provision-worktree.sh" failed')], interruption: generic }); + const grade = gradeConfigTask(record({}, { brokenSnapshots: [parked], parkedAt: 90_000, parkReason: "interruption", + restart: { kind: "K1", faultId: "f00", firedAt: 95_000, readyAt: 120_000, error: null }, afterRestartSnapshots: [parked], afterRestartParked: true, + repair: { at: 200_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 400_000 }), options); + expect(grade.pass).toBe(false); + expect(grade.checks.find(check => check.name === "actionable_message")?.detail).toMatch(/^message_only_in_run_detail/); + expect(grade.signature).toBe("CF1/provision_command:message_only_in_run_detail"); + }); + + it("passes when the task names the setting, survives the restart and completes after repair", () => { + const parked = snapshot(90_000, { runs: [failedRun("r1", 10_000, "x")], + interruption: { ...generic, workspaceFailure: "The project workspace path is not available" } }); + const grade = gradeConfigTask(record({ fault: "CF3", reference: "workspace_cwd" }, { brokenSnapshots: [parked], parkedAt: 90_000, parkReason: "interruption", + restart: { kind: "K2", faultId: "f00", firedAt: 95_000, readyAt: 120_000, error: null }, afterRestartSnapshots: [parked], afterRestartParked: true, + repair: { at: 200_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 400_000 }), options); + expect(grade).toMatchObject({ pass: true, outcome: "parked", signature: null }); + }); + + it("fails CF2 on a queued run with automatic setup retries and no card (the #20361 bug)", () => { + const stuck = snapshot(400_000, { issueStatus: "todo", runs: [{ id: "r1", status: "queued", createdAt: 5_000, startedAt: null, errorCode: null, error: null }], + execution: { label: "Queued", cause: null, nextAction: null }, + agent: { lifecycleState: "verifying", retryRequired: false, + lifecycleError: "Execution configuration check failed. Paperclip could not complete this step. It will retry automatically." } }); + const grade = gradeConfigTask(record({ fault: "CF2", reference: "ai_default", adapter: "opencode" }, { brokenSnapshots: [stuck], + restart: { kind: "K1", faultId: "f00", firedAt: 420_000, readyAt: 450_000, error: null }, afterRestartSnapshots: [stuck], afterRestartParked: false, + repair: { at: 600_000, action: "agent setup Retry", detail: "", error: null }, finalStatus: "done", doneAt: 700_000 }), options); + expect(grade.pass).toBe(false); + expect(grade.outcome).toBe("never_rested"); + expect(grade.signature).toBe("CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner"); + }); + + it("passes CF2 when the task shows the connection card and parked setup", () => { + const parked = snapshot(60_000, { issueStatus: "todo", runs: [{ id: "r1", status: "queued", createdAt: 5_000, startedAt: null, errorCode: null, error: null }], + cards: [{ id: "c1", kind: "connection_intent", status: "pending", serviceSlug: "openrouter", agentSetup: true }], + agent: { lifecycleState: "verifying", retryRequired: true, + lifecycleError: "Connect an AI account and choose the responsible user's personal default for this provider. Automatic verification is paused." } }); + expect(taskLevelTexts(parked)).toContain("Connect openrouter (agent setup)"); + const grade = gradeConfigTask(record({ fault: "CF2", reference: "ai_default", adapter: "opencode" }, { brokenSnapshots: [parked], parkedAt: 60_000, + parkReason: "queued_with_card", restart: { kind: "K1", faultId: "f00", firedAt: 70_000, readyAt: 90_000, error: null }, afterRestartSnapshots: [parked], + afterRestartParked: true, repair: { at: 100_000, action: "completed connection card c1", detail: "", error: null }, finalStatus: "done", doneAt: 200_000 }), options); + expect(grade.pass).toBe(true); + }); + + it("fails a retry loop over unchanged configuration", () => { + const runs = [1, 2, 3, 4].map(index => failedRun(`r${index}`, index * 10_000, "x")); + const grade = gradeConfigTask(record({}, { brokenSnapshots: [snapshot(400_000, { runs, interruption: { ...generic, workspaceFailure: "provision command failed" } })], + parkedAt: 400_000, repair: { at: 500_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 600_000 }), options); + expect(grade.checks.find(check => check.name === "no_retry_loop")?.verdict).toBe("fail"); + }); + + it("fails when a restart replaces the named message with a generic Retry", () => { + const named = snapshot(90_000, { runs: [failedRun("r1", 10_000, "x")], + interruption: { ...generic, workspaceFailure: "The project workspace path is not available" } }); + const generic2 = snapshot(200_000, { runs: [failedRun("r1", 10_000, "x"), failedRun("r2", 130_000, "x")], interruption: { ...generic, observedRunId: "r2" } }); + const grade = gradeConfigTask(record({ fault: "CF3", reference: "workspace_cwd" }, { brokenSnapshots: [named], parkedAt: 90_000, parkReason: "interruption", + restart: { kind: "K1", faultId: "f00", firedAt: 95_000, readyAt: 120_000, error: null }, afterRestartSnapshots: [generic2], afterRestartParked: true, + repair: { at: 300_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 400_000 }), options); + expect(grade.signature).toBe("CF3/workspace_cwd:message_lost_after_restart"); + }); + + it("counts runs started after the restart toward the retry loop", () => { + const parked = snapshot(90_000, { runs: [failedRun("r1", 10_000, "x")], interruption: { ...generic, workspaceFailure: "provision command failed" } }); + const looping = snapshot(300_000, { runs: [1, 2, 3, 4].map(index => failedRun(`r${index}`, index * 10_000, "x")), + interruption: { ...generic, workspaceFailure: "provision command failed" } }); + const grade = gradeConfigTask(record({}, { brokenSnapshots: [parked], parkedAt: 90_000, restart: { kind: "K2", faultId: "f00", firedAt: 95_000, readyAt: 120_000, error: null }, + afterRestartSnapshots: [looping], afterRestartParked: true, repair: { at: 400_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 500_000 }), options); + expect(grade.checks.find(check => check.name === "no_retry_loop")?.verdict).toBe("fail"); + }); + + it("does not take generic credential or model failures as naming the setting", () => { + expect(CONFIG_REFERENCES.agent_secret.names("abcd1234").test("Invalid credentials")).toBe(false); + expect(CONFIG_REFERENCES.agent_secret.names("abcd1234").test('Secret "OPENAI_API_KEY" was deleted')).toBe(true); + expect(CONFIG_REFERENCES.model_id.names("abcd1234").test("Model request failed")).toBe(false); + expect(CONFIG_REFERENCES.model_id.names("abcd1234").test("userAction:choose_model")).toBe(true); + }); + + it("counts a write-time refusal as a pass", () => { + expect(gradeConfigTask(record({}, { refusedOnWrite: "422 Secret not found" }), options)).toMatchObject({ pass: true, outcome: "refused_on_write" }); + }); +}); diff --git a/tests/restart-chaos/config-faults.ts b/tests/restart-chaos/config-faults.ts new file mode 100644 index 00000000000..e65a20928be --- /dev/null +++ b/tests/restart-chaos/config-faults.ts @@ -0,0 +1,412 @@ +/** + * Config and aged-state fault family (CF*): pure catalog, seeded plan and oracle. + * + * A fresh chaos instance only ever has valid settings. Production also has + * saved settings that point at something that no longer exists (a renamed + * setup script, an account default that was never chosen, a deleted secret). + * Each CF task breaks one such reference on its own agent or project, and the + * oracle is not "the task completes". It is: + * + * 1. within a bound the task completes, or rests where the user sees a + * specific message that names the broken setting (task-level surfaces); + * 2. no run retries unchanged configuration forever, and nothing sits + * Queued with no live owner and no named wait; + * 3. the parked state survives a K1/K2 restart; + * 4. after the harness repairs the setting and uses the offered action + * (complete the connection card, Retry), the task completes. + * + * I/O lives in config-wave.ts; this file is credential-free and unit tested. + */ +import { createRng } from "./scheduler.js"; +import type { AdapterId } from "./types.js"; + +export const CONFIG_FAULT_KINDS = ["CF1", "CF2", "CF3"] as const; +export type ConfigFaultKind = (typeof CONFIG_FAULT_KINDS)[number]; +export const CONFIG_RESTART_KINDS = ["K1", "K2"] as const; +export type ConfigRestartKind = (typeof CONFIG_RESTART_KINDS)[number]; + +/** A saved setting that refers to another object, and how a user would recognise a message about it. */ +export interface ConfigReference { + id: ConfigReferenceId; + owner: "agent" | "project"; + /** Field path as a user or the API names it. */ + field: string; + refersTo: "script" | "account_default" | "credential" | "workspace_path" | "git_ref" | "model"; + /** Adapters whose agents carry this setting; absent means every adapter. */ + adapters?: readonly AdapterId[]; + /** An actionable message names this setting (case-insensitive). The nonce placeholder is filled per task. */ + names: (nonce: string) => RegExp; + label: string; +} + +export const CONFIG_REFERENCE_IDS = ["provision_command", "ai_default", "agent_secret", "workspace_cwd", "base_ref", "model_id"] as const; +export type ConfigReferenceId = (typeof CONFIG_REFERENCE_IDS)[number]; + +/** The stale setup script CF1 points at: the path #20269 renamed in the Paperclip repository. */ +export const STALE_PROVISION_COMMAND = "bash ./scripts/provision-worktree.sh"; + +export const CONFIG_REFERENCES: Record = { + provision_command: { + id: "provision_command", owner: "project", field: "executionWorkspacePolicy.workspaceStrategy.provisionCommand", + refersTo: "script", label: "Project workspace setup command points at a missing script", + names: () => /provision-worktree\.sh|(setup|provision) command/i, + }, + ai_default: { + id: "ai_default", owner: "agent", field: "runtimeConfig.aiConnection (responsible_user default)", + refersTo: "account_default", adapters: ["opencode"], label: "OpenRouter binding selects a personal account default that does not exist", + names: () => /openrouter|personal default|connect (an |your )?(ai )?account/i, + }, + agent_secret: { + id: "agent_secret", owner: "agent", field: "adapterConfig.env. (secret_ref)", + refersTo: "credential", label: "Agent credential secret was deleted", + names: nonce => new RegExp(`\\bsecret\\b|[A-Z]+_API_KEY|_${nonce}\\b`, "i"), + }, + workspace_cwd: { + id: "workspace_cwd", owner: "project", field: "project workspace cwd", + refersTo: "workspace_path", label: "Project workspace path no longer exists", + names: nonce => new RegExp(`project workspace|workspace path|workspace folder|chaos-missing-${nonce}`, "i"), + }, + base_ref: { + id: "base_ref", owner: "project", field: "executionWorkspacePolicy.workspaceStrategy.baseRef", + refersTo: "git_ref", label: "Worktree base ref names a branch that no longer exists", + names: nonce => new RegExp(`base ref|base branch|chaos-missing-ref-${nonce}`, "i"), + }, + model_id: { + id: "model_id", owner: "agent", field: "adapterConfig.model", + refersTo: "model", label: "Agent model id no longer exists", + names: nonce => new RegExp(`chaos-missing-model-${nonce}|choose_model|(unknown|unsupported|invalid|unavailable) model|model[^.]{0,40}(not found|unknown|unsupported|does not exist|not available|rejected)`, "i"), + }, +}; + +/** + * Settings that reference another object but that CF3 does not break, and why. + * Reported with every wave so the coverage gap stays visible. + */ +export const CONFIG_REFERENCES_NOT_BROKEN: Record = { + "agent skills (paperclipSkillSync.desiredSkills)": "deleting a skill an agent uses is refused (422) and unknown keys are rejected on write", + "agent defaultEnvironmentId": "the FK is ON DELETE SET NULL and an instance has one local environment, so a local wave cannot leave it dangling", + "project executionWorkspacePolicy.environmentId": "not read when choosing a run's environment, so breaking it changes nothing", + "agent instructionsFilePath": "native runner agents use an instructions bundle; a missing file is only a warning for local adapters", + "agent adapterConfig.cwd": "created on demand; project tasks use the execution workspace instead", +}; + +/** Which references a CF kind may break for an adapter. */ +export function referencesFor(kind: ConfigFaultKind, adapter: AdapterId): ConfigReferenceId[] { + if (kind === "CF1") return ["provision_command"]; + if (kind === "CF2") return ["ai_default"]; + return CONFIG_REFERENCE_IDS.filter(id => !CONFIG_REFERENCES[id].adapters || CONFIG_REFERENCES[id].adapters!.includes(adapter)); +} + +export interface ConfigTaskPlan { + taskId: string; + index: number; + fault: ConfigFaultKind; + reference: ConfigReferenceId; + adapter: AdapterId; + /** Restart fired while the task is parked (or at the park bound when it never parks). */ + restart: ConfigRestartKind; + nonce: string; +} + +export interface ConfigWaveOptions { + seed: number; + faults: ConfigFaultKind[]; + adapters: AdapterId[]; + tasks: number; + concurrency: number; + restarts: ConfigRestartKind[]; + /** CF3 only: break exactly these references, in turn (default: seeded over every applicable reference). */ + references?: ConfigReferenceId[]; + parkBoundMs: number; + repairBudgetMs: number; +} + +/** + * Seeded plan. CF2's reference lives on an OpenCode agent, so CF2 tasks always + * use the opencode adapter. CF3 walks a seeded shuffle of the applicable + * references so a small wave still covers distinct settings. + */ +export function planConfigWave(options: ConfigWaveOptions): ConfigTaskPlan[] { + const rng = createRng(options.seed ^ 0x0cf0cf); + const tasks: ConfigTaskPlan[] = []; + const cf3Queue: Partial> = {}; + for (let index = 0; index < options.tasks; index += 1) { + const fault = options.faults[index % options.faults.length]!; + const adapter: AdapterId = fault === "CF2" ? "opencode" : options.adapters[index % options.adapters.length]!; + let reference: ConfigReferenceId; + if (fault === "CF3") { + const allowed = referencesFor("CF3", adapter).filter(id => !options.references || options.references.includes(id)); + if (!allowed.length) throw new Error(`CF3 has no reference to break for adapter ${adapter}`); + let queue = cf3Queue[adapter]; + if (!queue?.length) queue = cf3Queue[adapter] = rng.shuffle(allowed); + reference = queue.shift()!; + } else { + reference = referencesFor(fault, adapter)[0]!; + } + tasks.push({ taskId: `t${String(index + 1).padStart(2, "0")}`, index, fault, reference, adapter, + // Alternate restart kinds per fault kind, so two tasks of one fault cross both K1 and K2. + restart: options.restarts[tasks.filter(earlier => earlier.fault === fault).length % options.restarts.length]!, nonce: rng.hex(8) }); + } + return tasks; +} + +/** + * The responsible user's account default is company-wide: once one task's + * repair connects an account, a later task's binding is no longer broken, and + * the product has no route to clear a default. One ai_default task per wave. + */ +export function validateConfigPlan(tasks: readonly ConfigTaskPlan[]): void { + const aiDefault = tasks.filter(task => task.reference === "ai_default"); + if (aiDefault.length > 1) { + throw new Error(`Only one ai_default (CF2) task fits in a wave; got ${aiDefault.map(task => task.taskId).join(", ")}. Lower --tasks or use --references.`); + } +} + +/** One poll of everything a user could see about the task, plus run state. */ +export interface ConfigSnapshot { + at: number; + reachable: boolean; + issueStatus: string | null; + runs: Array<{ id: string; status: string; createdAt: number | null; startedAt: number | null; errorCode: string | null; error: string | null }>; + interruption: { state: string | null; canRetry: boolean; userAction: string | null; workspaceFailure: string | null; observedRunId: string | null } | null; + execution: { label: string | null; cause: string | null; nextAction: string | null } | null; + /** Pending thread interactions on the task (connection cards and similar). */ + cards: Array<{ id: string; kind: string; status: string; serviceSlug: string | null; agentSetup: boolean }>; + agent: { lifecycleState: string | null; lifecycleError: string | null; retryRequired: boolean } | null; + /** Non-agent comments on the task (recovery notices). */ + notices: string[]; +} + +/** Text a user sees on the task page: the interruption, the execution status, cards, agent setup status and notices. */ +export function taskLevelTexts(snapshot: ConfigSnapshot): string[] { + const texts: string[] = []; + const interruption = snapshot.interruption; + if (interruption?.workspaceFailure) texts.push(interruption.workspaceFailure); + if (interruption?.userAction) texts.push(`userAction:${interruption.userAction}`); + if (snapshot.execution?.label) texts.push(snapshot.execution.label); + if (snapshot.execution?.nextAction) texts.push(snapshot.execution.nextAction); + for (const card of snapshot.cards) { + if (card.status !== "pending") continue; + texts.push(card.kind === "connection_intent" ? `Connect ${card.serviceSlug ?? "an account"}${card.agentSetup ? " (agent setup)" : ""}` : card.kind); + } + if (snapshot.agent?.lifecycleError) texts.push(snapshot.agent.lifecycleError); + texts.push(...snapshot.notices); + return texts; +} + +/** Run error text: visible only on the run's own page, not on the task. */ +export function runDetailTexts(snapshot: ConfigSnapshot): string[] { + return snapshot.runs.flatMap(run => run.error ? [run.error] : []); +} + +export function namingTexts(texts: readonly string[], pattern: RegExp): string[] { + return texts.filter(text => pattern.test(text)); +} + +export type ParkState = + | { kind: "done" } + | { kind: "active"; reason: string } + | { kind: "parked"; reason: string }; + +/** + * Whether the task rests. A rest needs no running run, no run created in the + * last `settleMs`, and something on the task that waits on a person: a pending + * card, an interruption offering Retry or a user action, an agent whose setup + * needs repair, or a blocked/cancelled task. An unstarted queued run counts as + * a rest only with a pending card or parked agent setup naming why it waits. + */ +export function parkState(snapshot: ConfigSnapshot, now: number, settleMs: number): ParkState { + if (snapshot.issueStatus === "done") return { kind: "done" }; + if (snapshot.runs.some(run => run.status === "running")) return { kind: "active", reason: "run_running" }; + const newest = Math.max(0, ...snapshot.runs.map(run => run.createdAt ?? 0)); + if (newest && now - newest < settleMs) return { kind: "active", reason: "run_recently_created" }; + const pendingCard = snapshot.cards.some(card => card.status === "pending"); + const setupParked = snapshot.agent?.retryRequired === true; + const queued = snapshot.runs.some(run => run.status === "queued" && run.startedAt === null); + if (queued) return pendingCard || setupParked ? { kind: "parked", reason: pendingCard ? "queued_with_card" : "queued_setup_parked" } + : { kind: "active", reason: "queued_unstarted" }; + if (pendingCard) return { kind: "parked", reason: "card_pending" }; + if (snapshot.interruption && (snapshot.interruption.canRetry || snapshot.interruption.userAction)) return { kind: "parked", reason: "interruption" }; + if (setupParked) return { kind: "parked", reason: "agent_setup_parked" }; + if (snapshot.issueStatus === "blocked" || snapshot.issueStatus === "cancelled") return { kind: "parked", reason: `issue_${snapshot.issueStatus}` }; + return { kind: "active", reason: "no_named_wait" }; +} + +export interface ConfigTaskRecord { + plan: ConfigTaskPlan; + issueId: string | null; + agentId: string | null; + projectId: string | null; + launchedAt: number | null; + /** The break was refused on write (validation): the product prevents the stale reference. */ + refusedOnWrite: string | null; + /** The harness itself could not set the task up (not a product verdict). */ + harnessError: string | null; + brokenSnapshots: ConfigSnapshot[]; + parkedAt: number | null; + parkReason: string | null; + restart: { kind: ConfigRestartKind; faultId: string | null; firedAt: number; readyAt: number | null; error: string | null } | null; + /** Every poll after the restart, while the setting is still broken. */ + afterRestartSnapshots: ConfigSnapshot[]; + /** The task rested again (or completed) after the restart. */ + afterRestartParked: boolean | null; + /** Last snapshot after the repair (evidence when the task never completes). */ + afterRepair?: ConfigSnapshot | null; + repair: { at: number; action: string; detail: string; error: string | null } | null; + doneAt: number | null; + finalStatus: string | null; + /** A1 brief oracle result after repair. */ + brief: { ok: boolean; detail: string } | null; +} + +/** Status codes that mean the server validated the request and refused the stale value. */ +export function isWriteRefusal(status: number | null): boolean { + return status === 400 || status === 422; +} + +/** + * Restarts and harness writes must not overlap: a write sent while the server + * is down fails, and the task would be graded on a harness error. Writes share + * the gate; a restart waits for writes in flight and holds new ones. + */ +export class RestartGate { + private writers = 0; + private restarting: Promise | null = null; + private idle: Array<() => void> = []; + + async write(work: () => Promise): Promise { + while (this.restarting) await this.restarting; + this.writers += 1; + try { return await work(); } finally { + this.writers -= 1; + if (!this.writers) for (const wake of this.idle.splice(0)) wake(); + } + } + + async restart(work: () => Promise): Promise { + while (this.restarting) await this.restarting; + let release!: () => void; + this.restarting = new Promise(resolve => { release = resolve; }); + try { + while (this.writers) await new Promise(resolve => this.idle.push(resolve)); + return await work(); + } finally { + this.restarting = null; + release(); + } + } +} + +export interface ConfigCheck { name: string; verdict: "pass" | "fail" | "n/a"; detail: string } +export interface ConfigGrade { pass: boolean; outcome: "completed" | "parked" | "refused_on_write" | "never_rested" | "harness_error"; checks: ConfigCheck[]; signature: string | null } + +/** Runs created after this many attempts of unchanged configuration are a retry loop. */ +export const MAX_BROKEN_RUNS = 3; +/** An unstarted queued run older than this with no named wait has no live owner. */ +export const OWNERLESS_QUEUE_MS = 3 * 60_000; + +export function gradeConfigTask(record: ConfigTaskRecord, options: { parkBoundMs: number }): ConfigGrade { + const checks: ConfigCheck[] = []; + const add = (name: string, verdict: ConfigCheck["verdict"], detail: string) => checks.push({ name, verdict, detail }); + if (record.harnessError) { + return { pass: false, outcome: "harness_error", checks: [{ name: "harness", verdict: "fail", detail: record.harnessError }], signature: "harness_error" }; + } + if (record.refusedOnWrite) { + return { pass: true, outcome: "refused_on_write", checks: [{ name: "refused_on_write", verdict: "pass", detail: record.refusedOnWrite }], signature: null }; + } + const reference = CONFIG_REFERENCES[record.plan.reference]; + const pattern = reference.names(record.plan.nonce); + const last = record.brokenSnapshots.filter(snapshot => snapshot.reachable).at(-1) ?? null; + const afterRestart = record.afterRestartSnapshots.filter(snapshot => snapshot.reachable); + // Every sample while the setting was broken, before and after the restart. + const whileBroken = [...record.brokenSnapshots, ...afterRestart]; + const lastBroken = afterRestart.at(-1) ?? last; + const completedWhileBroken = last?.issueStatus === "done"; + const outcome: ConfigGrade["outcome"] = completedWhileBroken ? "completed" : record.parkedAt !== null ? "parked" : "never_rested"; + + // 1. Bounded rest. + if (completedWhileBroken) add("bounded_rest", "pass", "completed despite the broken setting"); + else if (record.parkedAt !== null && record.launchedAt !== null) { + add("bounded_rest", record.parkedAt - record.launchedAt <= options.parkBoundMs ? "pass" : "fail", + `parked (${record.parkReason}) after ${Math.round((record.parkedAt - record.launchedAt) / 1000)} s`); + } else add("bounded_rest", "fail", `no rest within ${Math.round(options.parkBoundMs / 1000)} s (last: ${last ? parkState(last, last.at, 0).kind : "unreachable"})`); + + // 2. Actionable message naming the setting, on the task. + if (completedWhileBroken) add("actionable_message", "n/a", "completed"); + else { + const restingSnapshots = record.parkedAt !== null ? record.brokenSnapshots.filter(snapshot => snapshot.at >= record.parkedAt!) : record.brokenSnapshots; + const named = [...new Set(restingSnapshots.flatMap(snapshot => namingTexts(taskLevelTexts(snapshot), pattern)))]; + const seen = [...new Set(restingSnapshots.flatMap(taskLevelTexts))]; + const runOnly = [...new Set(restingSnapshots.flatMap(snapshot => namingTexts(runDetailTexts(snapshot), pattern)))]; + if (named.length) add("actionable_message", "pass", truncate(named[0]!)); + else add("actionable_message", "fail", `${runOnly.length ? "message_only_in_run_detail: " + truncate(runOnly[0]!) + "; " : ""}user saw: ${seen.map(text => truncate(text)).join(" | ") || "nothing"}`); + } + + // 3. No retry loop over unchanged configuration. + const brokenRuns = new Set(whileBroken.flatMap(snapshot => snapshot.runs.map(run => run.id))); + const lifecycleRetrying = whileBroken.some(snapshot => snapshot.agent?.lifecycleError && /retry automatically/i.test(snapshot.agent.lifecycleError) + && snapshot.agent.retryRequired !== true); + if (brokenRuns.size > MAX_BROKEN_RUNS) add("no_retry_loop", "fail", `retry_loop: ${brokenRuns.size} runs while the setting stayed broken`); + else if (lifecycleRetrying && !completedWhileBroken) add("no_retry_loop", "fail", "lifecycle_retrying_unchanged_config: agent setup says it will retry automatically"); + else add("no_retry_loop", "pass", `${brokenRuns.size} run(s) while broken`); + + // 4. Nothing queued without a live owner. + const ownerless = lastBroken ? ownerlessQueuedRun(lastBroken, OWNERLESS_QUEUE_MS) : null; + add("no_ownerless_queue", ownerless ? "fail" : "pass", ownerless ? `queued_without_live_owner: run ${ownerless} unstarted with no named wait` : "none"); + + // 5. Parked state survives the restart. + if (!record.restart) add("survives_restart", "n/a", "no restart fired"); + else if (record.restart.error) add("survives_restart", "fail", `restart failed: ${record.restart.error}`); + else if (completedWhileBroken) add("survives_restart", "n/a", "completed before the restart"); + else if (record.parkedAt === null) add("survives_restart", "n/a", "never parked"); + else if (!record.afterRestartParked) { + const final = afterRestart.at(-1); + add("survives_restart", "fail", `rest_lost_after_restart: ${record.restart.kind} while parked; then ${final ? stateLabel(parkState(final, final.at, 0)) : "unreachable"}`); + } else { + // The rest must still name the setting: a restart that leaves only a generic Retry fails. + const final = afterRestart.at(-1); + const named = final ? namingTexts(taskLevelTexts(final), pattern) : []; + const namedBefore = checks.find(check => check.name === "actionable_message")?.verdict === "pass"; + if (final?.issueStatus === "done" || named.length || !namedBefore) { + add("survives_restart", "pass", `${record.restart.kind} while parked; parked again${named.length ? `: ${truncate(named[0]!)}` : ""}`); + } else { + add("survives_restart", "fail", `message_lost_after_restart: ${record.restart.kind}; user then saw: ${final ? taskLevelTexts(final).map(text => truncate(text)).join(" | ") || "nothing" : "nothing"}`); + } + } + + // 6. Repair plus the offered action completes the task. + if (completedWhileBroken) add("repair_completes", "n/a", "completed while broken"); + else if (!record.repair) add("repair_completes", "fail", "repair not attempted"); + else if (record.repair.error) add("repair_completes", "fail", `repair failed: ${record.repair.error}`); + else add("repair_completes", record.finalStatus === "done" ? "pass" : "fail", + `${record.repair.action}; final status ${record.finalStatus ?? "unknown"}${record.doneAt ? ` after ${Math.round((record.doneAt - record.repair.at) / 1000)} s` : ""}`); + + // 7. The work itself (A1 brief). + if (record.brief) add("brief_oracle", record.brief.ok ? "pass" : "fail", record.brief.detail); + else add("brief_oracle", "fail", "not collected"); + + const failed = checks.filter(check => check.verdict === "fail"); + return { pass: failed.length === 0, outcome, checks, + signature: failed.length ? `${record.plan.fault}/${record.plan.reference}:${failed.map(failureCode).join("+")}` : null }; +} + +/** A check's detail starting with a snake_case code names the failure; otherwise the check name does. */ +function failureCode(check: ConfigCheck): string { + const match = /^([a-z][a-z0-9_]+):/.exec(check.detail); + return match ? match[1]! : check.name; +} + +export function ownerlessQueuedRun(snapshot: ConfigSnapshot, olderThanMs: number): string | null { + if (snapshot.cards.some(card => card.status === "pending") || snapshot.agent?.retryRequired) return null; + const stale = snapshot.runs.find(run => run.status === "queued" && run.startedAt === null && run.createdAt !== null && snapshot.at - run.createdAt > olderThanMs); + return stale?.id ?? null; +} + +function stateLabel(state: ParkState): string { + return state.kind === "done" ? "done" : `${state.kind} (${state.reason})`; +} + +function truncate(text: string, max = 220): string { + const flat = text.replace(/\s+/g, " ").trim(); + return flat.length > max ? `${flat.slice(0, max - 1)}…` : flat; +} diff --git a/tests/restart-chaos/config-wave.ts b/tests/restart-chaos/config-wave.ts new file mode 100644 index 00000000000..e205c5c025c --- /dev/null +++ b/tests/restart-chaos/config-wave.ts @@ -0,0 +1,504 @@ +/** + * Config and aged-state fault waves (CF1–CF3). See config-faults.ts for the + * oracle and README.md ("Config faults") for usage. + * + * node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts \ + * --faults CF1,CF3 --adapters codex --tasks 4 --concurrency 1 --restarts K1,K2 --seed 61 + * + * Each task gets its own agent, credential secret and project, so breaking + * one task's setting never touches another task. The harness breaks the + * setting through the board routes a user would use, launches an A1 brief + * task, watches it rest, restarts the control plane while it is parked, + * repairs the setting, uses the offered action, and grades the result. + */ +import { spawnSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { existsSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { parseArgs } from "node:util"; +import { setTimeout as delay } from "node:timers/promises"; +import { releaseInstance } from "../runner-e2e/instance-retention.js"; +import { ChaosApi, ChaosApiError } from "./api.js"; +import { + CONFIG_FAULT_KINDS, CONFIG_REFERENCE_IDS, CONFIG_REFERENCES, CONFIG_REFERENCES_NOT_BROKEN, CONFIG_RESTART_KINDS, + RestartGate, STALE_PROVISION_COMMAND, gradeConfigTask, isWriteRefusal, parkState, planConfigWave, validateConfigPlan, + type ConfigGrade, type ConfigSnapshot, type ConfigTaskPlan, type ConfigTaskRecord, type ConfigWaveOptions, +} from "./config-faults.js"; +import { CREDENTIAL_FOR_ADAPTER, waveIdFor } from "./config.js"; +import { knownSecretValues, redactKnownSecrets } from "./evidence.js"; +import { seedChaosLab, waitForAgentsReady, type ChaosLab } from "./seed.js"; +import { ChaosSupervisor } from "./supervisor.js"; +import { Timeline } from "./timeline.js"; +import { guardedRunnerOutputs, worktreeFingerprint } from "./wave-guard.js"; +import { ADAPTER_IDS } from "./types.js"; +import { workloadDefinitions } from "./workloads.js"; + +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); +const resultsRoot = path.resolve(process.env.RESTART_CHAOS_RESULTS_DIR ?? path.join(import.meta.dirname, "results")); +const POLL_MS = 4_000; +/** No run created for this long before a rest counts. */ +const SETTLE_MS = 30_000; +/** After the restart, how long the task may take to rest again. */ +const RESTART_REPARK_MS = 150_000; +type Json = Record; + +export function parseConfigWaveArgs(argv: readonly string[]): { options: ConfigWaveOptions; keep: boolean } { + const { values } = parseArgs({ args: argv[0] === "--" ? argv.slice(1) : [...argv], strict: true, options: { + faults: { type: "string" }, adapters: { type: "string" }, tasks: { type: "string" }, concurrency: { type: "string" }, + restarts: { type: "string" }, seed: { type: "string" }, references: { type: "string" }, + "park-seconds": { type: "string" }, "repair-seconds": { type: "string" }, keep: { type: "boolean" }, + } }); + const list = (raw: string | undefined, allowed: readonly T[], name: string, fallback: T[]): T[] => { + if (raw === undefined) return fallback; + const items = raw.split(",").map(item => item.trim()).filter(Boolean); + for (const item of items) if (!allowed.includes(item as T)) throw new Error(`Unknown ${name} "${item}"; allowed: ${allowed.join(", ")}`); + if (!items.length) throw new Error(`--${name} must not be empty`); + return [...new Set(items)] as T[]; + }; + const int = (raw: string | undefined, name: string, fallback: number, min: number, max: number) => { + if (raw === undefined) return fallback; + const value = Number(raw); + if (!Number.isSafeInteger(value) || value < min || value > max) throw new Error(`--${name} must be an integer between ${min} and ${max}`); + return value; + }; + const options: ConfigWaveOptions = { + faults: list(values.faults, CONFIG_FAULT_KINDS, "faults", ["CF1", "CF3"]), + adapters: list(values.adapters, ADAPTER_IDS, "adapters", ["codex"]), + tasks: int(values.tasks, "tasks", 2, 1, 4), + // Disk and the one-wave-per-machine rule keep these waves small. + concurrency: int(values.concurrency, "concurrency", 1, 1, 2), + restarts: list(values.restarts, CONFIG_RESTART_KINDS, "restarts", ["K1", "K2"]), + seed: int(values.seed, "seed", Date.now() % 100_000, 0, 2 ** 31 - 1), + ...(values.references ? { references: list(values.references, CONFIG_REFERENCE_IDS, "references", []) } : {}), + parkBoundMs: int(values["park-seconds"], "park-seconds", 420, 60, 1_800) * 1_000, + repairBudgetMs: int(values["repair-seconds"], "repair-seconds", 900, 120, 3_600) * 1_000, + }; + return { options, keep: values.keep === true }; +} + +function preflight(tasks: readonly ConfigTaskPlan[]) { + const missing = [...new Set(tasks.map(task => CREDENTIAL_FOR_ADAPTER[task.adapter]))].filter(key => !process.env[key]); + if (missing.length) throw new Error(`Missing credentials ${missing.join(", ")}. Run: set -a; source ~/.secrets; set +a`); + const runner = process.env.PAPERCLIP_RUNNER_BINARY ?? path.join(repositoryRoot, "packages/paperclip-runner/runner/target/debug/paperclip-runnerd"); + if (!existsSync(runner)) throw new Error(`Runner binary missing at ${runner}. Run: node scripts/agent-preflight.mjs`); + const build = spawnSync(process.execPath, [path.join(repositoryRoot, "packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs")], + { cwd: repositoryRoot, stdio: "inherit" }); + if (build.status !== 0) throw new Error("Verified provider entrypoint build failed"); +} + +/** A task's own agent, credential and project: the objects its broken setting refers to. */ +interface TaskResources { + agentId: string; + agentBody: Json; + credentialKey: string; + secretId: string | null; + projectId: string; + workspaceId: string; + basePolicy: Json; + /** CF2 repair: the OpenRouter account the harness connected. */ + connectionId?: string; +} + +function basePolicy(lab: ChaosLab): Json { + return { enabled: true, defaultMode: "isolated_workspace", allowIssueOverride: false, environmentId: lab.environmentId, + workspaceStrategy: { type: "git_worktree" } }; +} + +const secretRef = (secretId: string) => ({ type: "secret_ref", secretId, version: "latest" }); + +async function createTaskResources(api: ChaosApi, lab: ChaosLab, task: ConfigTaskPlan, timeline: Timeline): Promise { + const source = lab.agents[task.adapter]?.body; + if (!source) throw new Error(`Seeded lab has no ${task.adapter} agent body to clone`); + const credentialKey = CREDENTIAL_FOR_ADAPTER[task.adapter]; + const adapterConfig = { ...((source.adapterConfig ?? {}) as Json) }; + const env = { ...((adapterConfig.env ?? {}) as Json) }; + const runtimeConfig = { ...((source.runtimeConfig ?? {}) as Json) }; + let secretId: string | null = null; + if (task.reference === "ai_default") { + // The responsible user's personal default; the fresh company has no OpenRouter account at all. + delete env[credentialKey]; + runtimeConfig.aiConnection = { provider: "openrouter", method: "api_key", mode: "responsible_user" }; + } else { + const secret = await api.post<{ id: string }>(`/api/companies/${lab.companyId}/secrets`, { + // Secret keys are unique per company; the agent's env var name stays the provider's. + name: `Chaos ${credentialKey} ${task.taskId}-${task.nonce}`, key: `${credentialKey}_${task.taskId}_${task.nonce}`, value: process.env[credentialKey], + description: `Restart chaos config fault ${task.taskId}` }); + secretId = secret.id; + env[credentialKey] = secretRef(secret.id); + } + const agentBody: Json = { ...source, name: `Chaos ${task.adapter} ${task.taskId}`, adapterConfig: { ...adapterConfig, env }, runtimeConfig }; + const agent = await api.post<{ id: string }>(`/api/companies/${lab.companyId}/agents`, agentBody); + const policy = basePolicy(lab); + const project = await api.post<{ id: string }>(`/api/companies/${lab.companyId}/projects`, { + name: `cf-${task.taskId}-${task.nonce}`, description: `Restart chaos config fault ${task.fault}/${task.reference}`, + executionWorkspacePolicy: policy, + workspace: { name: "Primary", sourceType: "local_path", cwd: lab.fixtureRepositoryPath, isPrimary: true }, + }); + const full = await api.get(`/api/projects/${project.id}`); + const workspaceId = full.primaryWorkspace?.id ?? full.workspaces?.[0]?.id; + if (!workspaceId) throw new Error(`Project ${project.id} has no primary workspace`); + timeline.add("cf_task_resources", { taskId: task.taskId, agentId: agent.id, projectId: project.id, workspaceId, secretId }); + return { agentId: agent.id, agentBody, credentialKey, secretId, projectId: project.id, workspaceId, basePolicy: policy }; +} + +/** Break the task's reference through the board routes. Returns a refusal when the server rejected the stale value on write. */ +async function breakReference(api: ChaosApi, lab: ChaosLab, task: ConfigTaskPlan, resources: TaskResources): Promise { + try { + switch (task.reference) { + case "provision_command": + await api.patch(`/api/projects/${resources.projectId}`, { executionWorkspacePolicy: { ...resources.basePolicy, + workspaceStrategy: { type: "git_worktree", provisionCommand: STALE_PROVISION_COMMAND } } }); + return null; + case "base_ref": + await api.patch(`/api/projects/${resources.projectId}`, { executionWorkspacePolicy: { ...resources.basePolicy, + workspaceStrategy: { type: "git_worktree", baseRef: `chaos-missing-ref-${task.nonce}` } } }); + return null; + case "workspace_cwd": + await api.patch(`/api/projects/${resources.projectId}/workspaces/${resources.workspaceId}`, + { cwd: path.join(path.dirname(lab.fixtureRepositoryPath), `chaos-missing-${task.nonce}`) }); + return null; + case "agent_secret": + await api.delete(`/api/secrets/${resources.secretId}`); + return null; + case "model_id": + await api.patch(`/api/agents/${resources.agentId}`, { adapterConfig: { model: `chaos-missing-model-${task.nonce}` } }); + return null; + case "ai_default": + // Broken at creation: the binding selects a personal default the company never had. + return null; + } + } catch (error) { + // Only a validation refusal proves the product rejects the stale value; a 403, 404 or 429 is a harness failure. + if (error instanceof ChaosApiError && isWriteRefusal(error.status)) return error.message.slice(0, 400); + throw error; + } +} + +/** Put the setting back the way a user would. */ +async function repairReference(api: ChaosApi, lab: ChaosLab, task: ConfigTaskPlan, resources: TaskResources): Promise { + switch (task.reference) { + case "provision_command": + case "base_ref": + await api.patch(`/api/projects/${resources.projectId}`, { executionWorkspacePolicy: resources.basePolicy }); + return "restored the project workspace policy without the stale value"; + case "workspace_cwd": + await api.patch(`/api/projects/${resources.projectId}/workspaces/${resources.workspaceId}`, { cwd: lab.fixtureRepositoryPath }); + return "restored the project workspace path"; + case "agent_secret": { + const secret = await api.post<{ id: string }>(`/api/companies/${lab.companyId}/secrets`, { + name: `Chaos ${resources.credentialKey} ${task.taskId}-${task.nonce} repaired`, key: `${resources.credentialKey}_${task.taskId}_${task.nonce}_repaired`, + value: process.env[resources.credentialKey], description: "Restart chaos config fault repair" }); + await api.patch(`/api/agents/${resources.agentId}`, { adapterConfig: { env: { [resources.credentialKey]: secretRef(secret.id) } } }); + return "created a new secret and pointed the agent at it"; + } + case "model_id": + await api.patch(`/api/agents/${resources.agentId}`, { adapterConfig: { model: resources.agentBody.adapterConfig?.model } }); + return `restored model ${String(resources.agentBody.adapterConfig?.model)}`; + case "ai_default": { + const created = await api.post<{ connectionId: string; grantId: string }>(`/api/companies/${lab.companyId}/ai-connections`, { + provider: "openrouter", method: "api_key", name: `Chaos OpenRouter ${task.nonce}`, ownership: "personal", + // The account must be installed for the agent; without it selection refuses "not permitted for this agent". + apiKey: process.env.OPENROUTER_API_KEY, agentIds: [resources.agentId], allAgents: false }, 60_000); + resources.connectionId = created.connectionId; + return "connected a personal OpenRouter account for the agent (it becomes the default)"; + } + } +} + +async function snapshot(api: ChaosApi, issueId: string, agentId: string, errors: Map): Promise { + const at = Date.now(); + try { + const [issue, runs, execution, interactions, agent, comments] = await Promise.all([ + api.get(`/api/issues/${issueId}`, 5_000), + api.get(`/api/issues/${issueId}/runs`, 5_000), + api.get(`/api/issues/${issueId}/execution`, 5_000).catch(() => null), + api.get(`/api/issues/${issueId}/interactions`, 5_000).catch(() => []), + api.get(`/api/agents/${agentId}`, 5_000).catch(() => null), + api.get(`/api/issues/${issueId}/comments`, 5_000).catch(() => []), + ]); + const runSamples: ConfigSnapshot["runs"] = []; + for (const run of runs ?? []) { + const id = String(run.runId ?? run.id); + const status = String(run.status); + let error: string | null = typeof run.error === "string" ? run.error : null; + if (!error && !["queued", "running"].includes(status)) { + if (!errors.has(id)) errors.set(id, await api.get(`/api/heartbeat-runs/${id}`, 5_000).then(row => row?.error ?? null).catch(() => null)); + error = errors.get(id) ?? null; + } + runSamples.push({ id, status, createdAt: time(run.createdAt), startedAt: time(run.startedAt), errorCode: run.errorCode ?? null, error }); + } + const interruption = issue.interruption as Json | null | undefined; + const projection = (execution?.execution ?? null) as Json | null; + return { + at, reachable: true, issueStatus: String(issue.status), runs: runSamples, + interruption: interruption ? { state: interruption.state ?? null, canRetry: interruption.canRetry === true, userAction: interruption.userAction ?? null, + workspaceFailure: interruption.workspaceFailure?.message ?? null, observedRunId: interruption.observedRunId ?? null } : null, + execution: projection ? { label: projection.label ?? null, cause: projection.cause ?? null, nextAction: projection.nextAction ?? null } : null, + cards: (Array.isArray(interactions) ? interactions : []).map(card => ({ id: String(card.id), kind: String(card.kind), status: String(card.status), + serviceSlug: card.payload?.serviceSlug ?? null, agentSetup: card.payload?.agentSetup === true })), + agent: agent ? { lifecycleState: agent.lifecycleState ?? null, lifecycleError: agent.lifecycleError ?? null, + // The agent response redacts lifecycleOperation; parked setup also says so in its message. + retryRequired: agent.lifecycleOperation?.retryRequired === true + || /automatic verification is paused|choose retry now/i.test(String(agent.lifecycleError ?? "")) } : null, + notices: (Array.isArray(comments) ? comments : []).filter(comment => !comment.authorAgentId).map(comment => String(comment.body ?? "")).slice(-5), + }; + } catch { + return { at, reachable: false, issueStatus: null, runs: [], interruption: null, execution: null, cards: [], agent: null, notices: [] }; + } +} + +function time(value: unknown): number | null { + const at = typeof value === "string" ? Date.parse(value) : Number.NaN; + return Number.isFinite(at) ? at : null; +} + +/** Use what the task offers: complete a pending connection card, then Retry, then the agent's setup Retry. */ +async function offeredAction(api: ChaosApi, record: ConfigTaskRecord, resources: TaskResources, current: ConfigSnapshot): Promise { + const card = current.cards.find(candidate => candidate.status === "pending" && candidate.kind === "connection_intent"); + if (card && resources.connectionId) { + await api.post(`/api/connection-intents/${card.id}/complete`, { connectionId: resources.connectionId }); + return `completed connection card ${card.id}`; + } + if (current.interruption?.canRetry && current.interruption.observedRunId) { + const response = await api.post(`/api/issues/${record.issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId: current.interruption.observedRunId }); + return `clicked Retry (accepted=${String(response?.accepted)})`; + } + if (current.agent && current.agent.lifecycleState !== "ready") { + await api.post(`/api/agents/${resources.agentId}/lifecycle/retry`, {}); + return `agent setup Retry (lifecycle ${current.agent.lifecycleState})`; + } + return "no offered action; waited"; +} + +/** Set by SIGINT/SIGTERM: tasks stop at their next step and the wave still reaches cleanup. */ +let interrupted = false; +function checkInterrupted() { + if (interrupted) throw new Error("interrupted"); +} + +interface TaskInput { + api: ChaosApi; lab: ChaosLab; task: ConfigTaskPlan; supervisor: ChaosSupervisor; timeline: Timeline; + options: ConfigWaveOptions; gate: RestartGate; onRestart: (faultId: string) => void; +} + +async function runTask(input: TaskInput): Promise { + const { api, lab, task, timeline, gate } = input; + const record: ConfigTaskRecord = { plan: task, issueId: null, agentId: null, projectId: null, launchedAt: null, refusedOnWrite: null, + harnessError: null, brokenSnapshots: [], parkedAt: null, parkReason: null, restart: null, afterRestartSnapshots: [], afterRestartParked: null, + repair: null, doneAt: null, finalStatus: null, brief: null }; + const errors = new Map(); + let resources: TaskResources; + try { + checkInterrupted(); + resources = await gate.write(() => createTaskResources(api, lab, task, timeline)); + record.agentId = resources.agentId; + record.projectId = resources.projectId; + // A working agent first, so the only broken thing is the setting under test (CF2's agent cannot verify by design). + if (task.reference !== "ai_default") { + await waitForAgentsReady(api, { [task.taskId]: { id: resources.agentId } }, timeline, undefined, 1_000, () => interrupted); + } + checkInterrupted(); + record.refusedOnWrite = await gate.write(() => breakReference(api, lab, task, resources)); + timeline.add("cf_broken", { taskId: task.taskId, fault: task.fault, reference: task.reference, refusedOnWrite: record.refusedOnWrite }); + if (record.refusedOnWrite) return record; + const a1 = workloadDefinitions.A1; + const issue = await gate.write(() => api.post(`/api/companies/${lab.companyId}/issues`, { + title: `Chaos ${task.fault} ${task.reference} ${task.nonce}`, description: a1.description(task.nonce), status: "todo", priority: "medium", + assigneeAgentId: resources.agentId, projectId: resources.projectId })); + record.issueId = String(issue.id); + record.launchedAt = Date.now(); + timeline.add("cf_launched", { taskId: task.taskId, issueId: issue.id, identifier: issue.identifier ?? null }); + } catch (error) { + record.harnessError = `setup: ${error instanceof Error ? error.message : String(error)}`.slice(0, 600); + return record; + } + try { + await observeTask(input, record, resources, errors); + } catch (error) { + record.harnessError = (error instanceof Error ? error.message : String(error)).slice(0, 600); + } + return record; +} + +async function observeTask(input: TaskInput, record: ConfigTaskRecord, resources: TaskResources, errors: Map): Promise { + const { api, lab, task, supervisor, timeline, options, gate } = input; + const issueId = record.issueId!; + + // Broken phase: until done, parked (stable across two polls) or the bound. + let parkedCandidate: number | null = null; + while (Date.now() - record.launchedAt! < options.parkBoundMs) { + await delay(POLL_MS); + checkInterrupted(); + if (!supervisor.serverReady) continue; + const current = await snapshot(api, issueId, resources.agentId, errors); + record.brokenSnapshots.push(current); + if (!current.reachable) continue; + const state = parkState(current, current.at, SETTLE_MS); + if (state.kind === "done") break; + if (state.kind === "parked") { + parkedCandidate ??= current.at; + if (current.at - parkedCandidate >= 2 * POLL_MS) { record.parkedAt = parkedCandidate; record.parkReason = state.reason; break; } + } else parkedCandidate = null; + } + timeline.add("cf_broken_phase_end", { taskId: task.taskId, parkedAt: record.parkedAt, parkReason: record.parkReason, + lastStatus: record.brokenSnapshots.at(-1)?.issueStatus ?? null }); + + const completedWhileBroken = record.brokenSnapshots.at(-1)?.issueStatus === "done"; + if (!completedWhileBroken) { + // Restart while parked (or at the bound). The gate holds every task's writes while the server is down. + checkInterrupted(); + await gate.restart(async () => { + const restart: NonNullable = { kind: task.restart, faultId: null, firedAt: Date.now(), readyAt: null, error: null }; + record.restart = restart; + try { + const receipt = await supervisor.fault(task.restart, 0, task.taskId); + restart.faultId = receipt.faultId; + restart.readyAt = receipt.readyAt; + restart.error = receipt.error; + input.onRestart(receipt.faultId); + } catch (error) { + restart.error = error instanceof Error ? error.message : String(error); + } + }); + // After the restart: the task must rest again, still naming the setting. + const repark = Date.now(); + let lastParked: number | null = null; + while (Date.now() - repark < RESTART_REPARK_MS) { + await delay(POLL_MS); + checkInterrupted(); + if (!supervisor.serverReady) continue; + const current = await snapshot(api, issueId, resources.agentId, errors); + if (!current.reachable) continue; + record.afterRestartSnapshots.push(current); + const state = parkState(current, current.at, SETTLE_MS); + if (state.kind === "done") { record.afterRestartParked = true; break; } + if (state.kind === "parked") { + lastParked ??= current.at; + if (current.at - lastParked >= 2 * POLL_MS) { record.afterRestartParked = true; break; } + } else lastParked = null; + } + record.afterRestartParked ??= false; + timeline.add("cf_after_restart", { taskId: task.taskId, restart: record.restart, parkedAgain: record.afterRestartParked }); + + // Repair, then the offered action. + checkInterrupted(); + const repairAt = Date.now(); + try { + const { detail, action } = await gate.write(async () => { + const detail = await repairReference(api, lab, task, resources); + await delay(3_000); + const current = await snapshot(api, issueId, resources.agentId, errors); + const action = current.issueStatus === "done" ? "none (already done)" : await offeredAction(api, record, resources, current); + return { detail, action }; + }); + record.repair = { at: repairAt, action, detail, error: null }; + } catch (error) { + record.repair = { at: repairAt, action: "repair", detail: "", error: error instanceof Error ? error.message.slice(0, 400) : String(error) }; + } + timeline.add("cf_repaired", { taskId: task.taskId, repair: record.repair }); + } + + // Completion after repair, bounded. + const deadline = Date.now() + options.repairBudgetMs; + while (Date.now() < deadline) { + checkInterrupted(); + const current = await snapshot(api, issueId, resources.agentId, errors); + if (current.reachable) { + record.finalStatus = current.issueStatus; + record.afterRepair = current; + if (current.issueStatus === "done") { record.doneAt = Date.now(); break; } + } + await delay(POLL_MS * 2); + } + const brief = await api.get(`/api/issues/${issueId}/documents/brief`).then(row => typeof row?.body === "string" ? row.body : null).catch(() => null); + const oracle = workloadDefinitions.A1.grade(task.nonce, { issueStatus: record.finalStatus, documents: { brief }, agentComments: [], files: {}, commands: {}, workspacePath: null }); + record.brief = { ok: oracle.verdict === "pass", detail: oracle.checks.filter(check => !check.ok).map(check => `${check.name}${check.detail ? ` (${check.detail})` : ""}`).join("; ") || "brief ok" }; +} + +export function configSummaryMarkdown(waveId: string, gitSha: string, rows: Array<{ record: ConfigTaskRecord; grade: ConfigGrade }>): string { + const lines = [`# Config fault wave ${waveId}`, "", `Commit ${gitSha}`, "", + "| Task | Fault | Setting | Adapter | Restart | Outcome | Pass | Signature |", "| --- | --- | --- | --- | --- | --- | --- | --- |"]; + for (const { record, grade } of rows) { + lines.push(`| ${record.plan.taskId} | ${record.plan.fault} | ${record.plan.reference} | ${record.plan.adapter} | ${record.plan.restart} | ${grade.outcome} | ${grade.pass ? "pass" : "FAIL"} | ${grade.signature ?? ""} |`); + } + lines.push("", "## Checks", ""); + for (const { record, grade } of rows) { + lines.push(`### ${record.plan.taskId} ${record.plan.fault}/${record.plan.reference}: ${CONFIG_REFERENCES[record.plan.reference].label}`, ""); + for (const check of grade.checks) lines.push(`- ${check.verdict === "pass" ? "pass" : check.verdict === "n/a" ? "n/a " : "FAIL"} ${check.name}: ${check.detail}`); + lines.push(""); + } + lines.push("## Not broken by CF3", "", ...Object.entries(CONFIG_REFERENCES_NOT_BROKEN).map(([name, why]) => `- ${name}: ${why}`), ""); + return lines.join("\n"); +} + +async function main(): Promise { + const { options, keep } = parseConfigWaveArgs(process.argv.slice(2)); + const tasks = planConfigWave(options); + validateConfigPlan(tasks); + const waveId = `${waveIdFor(options.seed)}-cf`; + const waveDir = path.join(resultsRoot, waveId); + await mkdir(waveDir, { recursive: true, mode: 0o700 }); + await writeFile(path.join(waveDir, "config-plan.json"), `${JSON.stringify({ waveId, options, tasks }, null, 2)}\n`, { mode: 0o600 }); + const timeline = new Timeline(path.join(waveDir, "timeline.jsonl")); + const gitSha = spawnSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8" }).stdout.trim(); + timeline.add("cf_wave_planned", { waveId, gitSha, tasks: tasks.map(task => `${task.taskId}:${task.adapter}/${task.fault}/${task.reference}/${task.restart}`) }); + preflight(tasks); + // Every restart re-runs the server from this checkout: a build or checkout change mid-wave would mix builds. + const guardedOutputs = guardedRunnerOutputs(repositoryRoot); + const startFingerprint = worktreeFingerprint(repositoryRoot, guardedOutputs); + let worktreeChangedAt: string | null = null; + const checkWorktree = (at: string) => { + if (worktreeChangedAt || worktreeFingerprint(repositoryRoot, guardedOutputs) === startFingerprint) return; + worktreeChangedAt = at; + timeline.add("worktree_changed_mid_wave", { at }); + }; + const onSignal = (signal: NodeJS.Signals) => { if (!interrupted) { interrupted = true; timeline.add("harness_interrupted", { signal }); } }; + process.on("SIGINT", onSignal); + process.on("SIGTERM", onSignal); + const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline }); + let stopFailed = false; + let harnessError: unknown = null; + const rows: Array<{ record: ConfigTaskRecord; grade: ConfigGrade }> = []; + try { + await supervisor.start({}); + const api = new ChaosApi(supervisor.baseUrl); + const lab = await seedChaosLab({ api, env: "local", waitForReadyAgents: true, adapters: [...new Set(tasks.map(task => task.adapter))], + concurrency: options.concurrency, budgetCents: 3_000, fixtureRoot: path.join(supervisor.paths.temporaryRoot, "fixtures"), waveId, timeline }); + const gate = new RestartGate(); + const pending = [...tasks]; + const records: ConfigTaskRecord[] = []; + await Promise.all(Array.from({ length: options.concurrency }, async () => { + while (pending.length && !interrupted) { + const task = pending.shift()!; + records.push(await runTask({ api, lab, task, supervisor, timeline, options, gate, onRestart: checkWorktree })); + } + })); + checkWorktree("wave_end"); + if (worktreeChangedAt) throw new Error(`The worktree changed during the wave (first seen after ${worktreeChangedAt}); later restarts may have run different code`); + if (interrupted) throw new Error("Harness interrupted before the wave finished"); + for (const record of records.sort((left, right) => left.plan.index - right.plan.index)) { + const grade = gradeConfigTask(record, { parkBoundMs: options.parkBoundMs }); + rows.push({ record, grade }); + await writeFile(path.join(waveDir, `${record.plan.taskId}.config-grade.json`), + `${redactKnownSecrets(JSON.stringify({ record, grade }, null, 2), knownSecretValues())}\n`, { mode: 0o600 }); + timeline.add("cf_task_graded", { taskId: record.plan.taskId, pass: grade.pass, outcome: grade.outcome, signature: grade.signature }); + } + } catch (error) { + harnessError = error; + timeline.add("harness_error", { error: error instanceof Error ? error.stack ?? error.message : String(error) }); + } finally { + try { await supervisor.stop(); } catch (error) { stopFailed = true; harnessError ??= error; } + await releaseInstance(supervisor.paths.temporaryRoot, { keep, stopFailed, owner: waveId }) + .catch(error => timeline.add("instance_cleanup_failed", { error: String(error) })); + } + const summary = configSummaryMarkdown(waveId, gitSha, rows); + await writeFile(path.join(waveDir, "SUMMARY.md"), summary, { mode: 0o600 }); + console.info(`\n${summary}\nWave ${waveId}: results ${waveDir}`); + if (harnessError) { console.error(harnessError); return 2; } + return rows.every(row => row.grade.pass) && rows.length === tasks.length ? 0 : 1; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === path.resolve(import.meta.filename)) { + process.exit(await main().catch(error => { console.error(error); return 2; })); +}