mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
test(restart-chaos): add config fault waves (CF1-CF3) and the CF4 design (#20446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) starts an isolated control plane, runs real agent tasks and restarts the server during them > - Every wave starts from a new instance with valid default settings and one valid provider > - On 2026-11-09 production failed on state that a new instance never has: a setup command that named a renamed script (#20269), and an account default that did not exist (#20361) > - This pull request adds a "config fault" family (CF1–CF3): the harness breaks one saved reference on a task's own agent or project, restarts the server while the task is parked, repairs the setting and grades the result > - The benefit is that the harness now catches "stale saved setting" failures, including a run that retries forever or a task that stays Queued with no explanation ## Linked Issues or Issue Description Refs #20269 Refs #20361 **What existing behavior does this improve?** The restart chaos harness only tests new instances with valid settings. It cannot see failures that come from saved settings that point at a missing object. **Subsystem affected** `tests/restart-chaos/` (test harness only). No product code changes. **Current behavior** No wave breaks a saved setting. The CF2 failure (#20361) and the CF1 failure (#20269) were found in production, not by the harness. **Proposed behavior** `config-wave.ts` runs CF1–CF3. The oracle is not "the task completes". A task passes when all of these are true: - it completes, or it rests on a message that names the broken setting on the task page within a bound; - no run retries unchanged configuration forever; - no task stays Queued with no owner; - the rest survives a K1 or K2 restart; - after repair and the offered action (connection card, Retry, or agent setup Retry), the task completes. ## What Changed - `tests/restart-chaos/config-faults.ts` (new, pure): the catalog of broken references, the seeded plan, `parkState`, the task-level message check and `gradeConfigTask`. CF1 is a missing setup script. CF2 is a missing personal OpenRouter default. CF3 is a seeded walk over a deleted secret, a missing workspace path, a missing base ref, an unknown model, plus CF1 and CF2. `CONFIG_REFERENCES_NOT_BROKEN` lists the settings CF3 cannot break, and why. - `tests/restart-chaos/config-wave.ts` (new): the wave runner. Each task gets its own agent, credential secret and project. The runner breaks the setting through board routes, launches an A1 brief task, polls, fires K1 or K2 while the task is parked, repairs the setting, uses the offered action and grades the task. It reuses the existing supervisor, seed and A1 oracle. Waves are limited to 4 tasks and a concurrency of 2. The instance is deleted at the end. - `tests/restart-chaos/config-faults.test.ts` (new): unit tests for the plan, the arguments, `parkState` and the grader. These include the CF2 bug signature and the CF1 "message only on the run page" signature. - `tests/restart-chaos/README.md`: a new "Config faults" section and a row in the pieces table. - `doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`: the CF4 design (an instance aged by an older release, built on the existing K14 `--upgrade-from` machinery). CF4 is not built in this pull request. - There are no edits to `main.ts`, `types.ts`, `workloads.ts`, `invariants.ts` or `grader.ts`. Other agents work on those files in parallel. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 24 files, 291 tests pass (before the last unit test was added). The config-fault suite has 13 tests, and all pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Live waves (local, isolated instance, one wave at a time): - **CF2 red before #20361** (base `8004ad069e`, before #20361 merged): - Waves `w-20261110T000602-s201-cf` and `w-20261110T004818-s202-cf` fail with `CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner`. - The user sees `Queued`, a `resume` user action, and "Execution configuration check failed. Paperclip could not complete this step. It will retry automatically." There is no card. - A debug run with logging added to the lifecycle failure log (not committed) showed that the underlying error is `ai_connection_default_missing`. Verification retried it every 60 s, more than 20 times. - After the harness connects an account that is installed for the agent, the task completes 35 s later on its own. - The CF2 run after #20361, and the CF1 and CF3 waves on current master, are added below as they finish. ## Risks - Low risk. The change is test harness code and docs only. - A wave costs provider tokens only when the repaired task runs (one A1 brief). - The CF2 repair creates a real OpenRouter account in the isolated instance. The server validates the key live. - The oracle reads the task page through the API (interruption, execution status, cards, agent setup status and notices). It does not render the board UI. A message that appears only in the run's own error text is graded `message_only_in_run_detail`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, subagents). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed
1 parent
0991d9bda1
commit
1b52f359dd
5 files changed
+1229
No files matched your search
@@ -0,0 +1,86 @@
|
||||
# Restart chaos: config and aged-state faults (CF1–CF4)
|
||||
|
||||
Status: CF1–CF3 are implemented in `tests/restart-chaos/config-faults.ts` and `config-wave.ts`. CF4 is designed here but not built.
|
||||
|
||||
## Why
|
||||
|
||||
Every chaos wave starts from a new instance that has default, valid settings. On
|
||||
2026-11-09, production failures came from state that a new instance never has:
|
||||
|
||||
- **#20269.** A project's explicit setup command named `scripts/provision-worktree.sh`,
|
||||
a script that had been renamed. Workspace preparation failed before the provider ran.
|
||||
- **#20361.** An OpenRouter binding selected the responsible user's personal account
|
||||
default, but that default did not exist. Verification retried forever, and the task
|
||||
stayed Queued.
|
||||
|
||||
## Oracle (CF1–CF4)
|
||||
|
||||
"The task completes" is not the oracle. A task passes when all of these hold:
|
||||
|
||||
- **Bounded rest.** Within a bound, the task either completes or rests where the task
|
||||
page names the broken setting.
|
||||
- **No silent waiting.** No run retries unchanged configuration forever. Nothing stays
|
||||
Queued without a live owner or a named wait.
|
||||
- **Restart safety.** The rest survives a K1 or K2 restart.
|
||||
- **Repair.** After the harness repairs the setting and uses the action the task offers
|
||||
(card, Retry, or agent setup Retry), the task completes.
|
||||
|
||||
See the README section "Config faults" for the exact checks.
|
||||
|
||||
## CF4: aged instance (design)
|
||||
|
||||
**Goal.** Start the wave on an instance that an older release created. That release saves
|
||||
tasks and settings, including CF1–CF3 breaks and parked tasks. Then switch the same
|
||||
database and instance directory to the current build. Grade with the CF oracle.
|
||||
|
||||
**Reuse.** K14 already has the switching machinery:
|
||||
|
||||
- `ChaosSupervisor.create({ upgradeFrom })` starts the server from a prepared older
|
||||
checkout.
|
||||
- `supervisor.fault("K14")` stops that server gracefully and starts the current checkout
|
||||
against the same database, port, secrets and instance home.
|
||||
- The prepared checkout is made with `git worktree add <dir> <commit>` and
|
||||
`node scripts/agent-preflight.mjs`, as in the K14 section of the README.
|
||||
|
||||
Build CF4 on these. Do not add a second upgrade path.
|
||||
|
||||
**Changes to `config-wave.ts`** (about half a day):
|
||||
|
||||
1. **New flag.** Add `--upgrade-from <dir>`. Check that the directory is prepared, the
|
||||
same way `main.ts preflight` does.
|
||||
2. **Seed and break on the older build.** The older build runs `seedChaosLab`, creates
|
||||
each task's resources, applies the break, and launches the task. Use only routes that
|
||||
both builds have. Where a route differs, use the older build's route.
|
||||
3. **Barrier.** Wait until every task has finished its broken phase, so it has rested or
|
||||
reached its bound. The parked state is then saved by the older build.
|
||||
4. **One switch.** Fire `K14` once for the wave. This replaces each task's own K1/K2.
|
||||
Re-check the rest on the current build, using the same check as after a restart.
|
||||
5. **Repair on the current build.** Run repair and the offered action on the current
|
||||
build, then grade.
|
||||
6. **Record both commits.** Write both commits into the summary. The K14 receipt already
|
||||
includes them.
|
||||
|
||||
**What it covers that CF1–CF3 do not:**
|
||||
|
||||
- Older-release run state is resumed by the current build:
|
||||
- setup cards created by the older build
|
||||
- lifecycle operations whose old retry policy was "retry every 60 s"
|
||||
- interruptions projected by older code
|
||||
- Settings written by older validators that newer validators would reject. A
|
||||
`refused_on_write` setting can still exist in an aged database. CF4 is the only way to
|
||||
reach that case without writing directly to the database.
|
||||
|
||||
**Commit choice.** Use the commit one or two releases back. Also use the commit just
|
||||
before a known fix. For example, the merge base of #20361 makes the old build save a
|
||||
CF2 queued-without-card state, which the fixed build must then recover.
|
||||
|
||||
**Cost and limits.**
|
||||
|
||||
- A prepared older checkout takes about 3 minutes with the shared preflight cache, plus
|
||||
about 2 GB of disk.
|
||||
- Waves stay at 4 tasks or fewer and a concurrency of 2 or less.
|
||||
- It runs locally only, as K14 does.
|
||||
|
||||
**Not designed here.** Months-old data shapes, such as rows from migrations long since
|
||||
squashed. A seeded fixture database taken from an older release would cover them.
|
||||
Capture one only when an upgrade incident shows the need.
|
||||
@@ -173,6 +173,54 @@ receipt names both commits (`upgrade` in `faults.json`, `upgrade_switched` in th
|
||||
timeline). K14 is local-only, because a Daytona runner must match its image, and
|
||||
it is never part of a storm's default faults.
|
||||
|
||||
## Config faults (CF1–CF3, `config-wave.ts`)
|
||||
|
||||
```sh
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts \
|
||||
--faults CF1,CF3 --adapters codex --tasks 4 --concurrency 1 --restarts K1,K2 --seed 61
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF2 --tasks 1 --seed 62
|
||||
```
|
||||
|
||||
A fresh wave only has valid settings. Config faults break one saved setting that refers
|
||||
to another object. Each task gets its own agent, credential secret and project, so one
|
||||
task's break never affects another task. The harness breaks the setting through the
|
||||
board routes a user would use, launches an A1 brief task on that agent and project, and
|
||||
then checks four things:
|
||||
|
||||
1. **Bounded rest.** Within `--park-seconds` (default 420) the task either completes, or
|
||||
rests on something that waits for a person: a pending card, an interruption that
|
||||
offers Retry or a user action, parked agent setup, or a blocked task. An unstarted
|
||||
Queued run counts as a rest only with a card or parked setup that says why it waits.
|
||||
2. **Actionable message.** While the task rests, the task page names the broken setting:
|
||||
the interruption, the execution status, cards, the agent's setup status, or a notice.
|
||||
A message that appears only in the run's own error text fails as
|
||||
`message_only_in_run_detail`.
|
||||
3. **No loop, no ownerless queue.** At most 3 runs start while the setting is broken.
|
||||
Agent setup does not say it "will retry automatically" while the configuration is
|
||||
unchanged. No Queued run sits unstarted for more than 3 minutes without a named wait.
|
||||
4. **Restart and repair.** A K1 or K2 restart fires while the task rests (`--restarts`,
|
||||
alternating), and the task must rest again. The harness then repairs the setting and
|
||||
uses the action the task offers: complete the connection card, click Retry, or retry
|
||||
the agent's setup. The task must then finish within `--repair-seconds`, and the A1
|
||||
brief oracle must pass.
|
||||
|
||||
A break that the server refuses on write passes as `refused_on_write`.
|
||||
|
||||
| Fault | Setting broken |
|
||||
| --- | --- |
|
||||
| CF1 | Project `workspaceStrategy.provisionCommand` set to `bash ./scripts/provision-worktree.sh`, a script that does not exist (as in #20269) |
|
||||
| CF2 | OpenCode agent `runtimeConfig.aiConnection` selects the responsible user's personal OpenRouter default, which was never created (as in #20361). Repair connects a personal account |
|
||||
| CF3 | Seeded walk over `provision_command`, `ai_default` (opencode only), `agent_secret` (secret deleted), `workspace_cwd` (path removed), `base_ref` (missing branch), `model_id` (unknown model). Pin a set with `--references` |
|
||||
|
||||
Waves are capped at 4 tasks and a concurrency of 2. A wave can contain only one
|
||||
`ai_default` task, because the product has no route that clears a default once a repair
|
||||
creates it. Results go to `results/<wave>-cf/` (`SUMMARY.md`, `<task>.config-grade.json`,
|
||||
`timeline.jsonl`, `server.log`). The instance is deleted when the wave ends.
|
||||
|
||||
Settings that CF3 does not break, and why, are listed in `CONFIG_REFERENCES_NOT_BROKEN`
|
||||
and in every wave summary. CF4 (an instance aged by an older release) is designed in
|
||||
[`doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`](../../doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md).
|
||||
|
||||
## Cloud storage (`--storage s3`, `--storage-faults`)
|
||||
|
||||
```sh
|
||||
@@ -386,6 +434,7 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
| `scheduler.ts` | Seeded plan: workload × adapter × fault cells and a random fire time per task. Saved to `results/<wave>/plan.json`. `--replay` reruns one task alone with the same choices and fires the recorded sequence of faults that crossed it (other tasks' faults included) at the same offsets from launch. |
|
||||
| `observer.ts` | Polls each issue and its runs. When a task rests needing Retry it calls `POST /api/issues/:id/retry` once (once more only if a later fault started after that click), with the body `useTaskRetry.ts` sends. Collects T-oracle artifacts and the R6 sweep. |
|
||||
| `grader.ts` | Pure R1–R6 grading over the samples. |
|
||||
| `config-faults.ts`, `config-wave.ts` | Config faults: the reference catalog, seeded plan and oracle (pure), and the wave runner that breaks, restarts, repairs and grades. |
|
||||
| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, with their T-oracles. |
|
||||
| `evidence.ts`, `ledger.ts` | Evidence bundle for each failure and the JSONL ledger plus `SUMMARY.md`. |
|
||||
| `daytona-image.ts` | Docker-free extraction of the image's runnerd and provider pack, cached per digest, and the controller/image content-ID check. |
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
CONFIG_REFERENCE_IDS, CONFIG_REFERENCES, RestartGate, gradeConfigTask, isWriteRefusal, parkState, planConfigWave, referencesFor, taskLevelTexts, validateConfigPlan,
|
||||
type ConfigSnapshot, type ConfigTaskPlan, type ConfigTaskRecord, type ConfigWaveOptions,
|
||||
} from "./config-faults.js";
|
||||
import { parseConfigWaveArgs } from "./config-wave.js";
|
||||
|
||||
const base: ConfigWaveOptions = { seed: 7, faults: ["CF1", "CF3"], adapters: ["codex"], tasks: 4, concurrency: 1, restarts: ["K1", "K2"],
|
||||
parkBoundMs: 420_000, repairBudgetMs: 900_000 };
|
||||
|
||||
function snapshot(at: number, patch: Partial<ConfigSnapshot> = {}): ConfigSnapshot {
|
||||
return { at, reachable: true, issueStatus: "in_progress", runs: [], interruption: null, execution: null, cards: [], agent: null, notices: [], ...patch };
|
||||
}
|
||||
|
||||
function record(plan: Partial<ConfigTaskPlan>, patch: Partial<ConfigTaskRecord> = {}): ConfigTaskRecord {
|
||||
return {
|
||||
plan: { taskId: "t01", index: 0, fault: "CF1", reference: "provision_command", adapter: "codex", restart: "K1", nonce: "abcd1234", ...plan },
|
||||
issueId: "issue", agentId: "agent", projectId: "project", launchedAt: 0, refusedOnWrite: null, harnessError: null,
|
||||
brokenSnapshots: [], parkedAt: null, parkReason: null, restart: null, afterRestartSnapshots: [], afterRestartParked: null, repair: null,
|
||||
doneAt: null, finalStatus: null, brief: { ok: true, detail: "brief ok" }, ...patch,
|
||||
};
|
||||
}
|
||||
|
||||
const failedRun = (id: string, createdAt: number, error: string) => ({ id, status: "failed", createdAt, startedAt: createdAt, errorCode: "setup_failed", error });
|
||||
|
||||
describe("planConfigWave", () => {
|
||||
it("is seeded, walks distinct CF3 references and pins CF2 to opencode", () => {
|
||||
const tasks = planConfigWave({ ...base, faults: ["CF3"], tasks: 4 });
|
||||
expect(planConfigWave({ ...base, faults: ["CF3"], tasks: 4 })).toEqual(tasks);
|
||||
expect(new Set(tasks.map(task => task.reference)).size).toBe(4);
|
||||
expect(tasks.every(task => referencesFor("CF3", "codex").includes(task.reference))).toBe(true);
|
||||
expect(planConfigWave({ ...base, faults: ["CF2"], tasks: 1 })[0]).toMatchObject({ adapter: "opencode", reference: "ai_default" });
|
||||
});
|
||||
|
||||
it("crosses each fault kind with both restart kinds", () => {
|
||||
const tasks = planConfigWave({ ...base, faults: ["CF1", "CF2"], tasks: 3 });
|
||||
expect(tasks.map(task => `${task.fault}/${task.restart}`)).toEqual(["CF1/K1", "CF2/K1", "CF1/K2"]);
|
||||
});
|
||||
|
||||
it("offers the account default only to adapters that carry the binding", () => {
|
||||
expect(referencesFor("CF3", "codex")).not.toContain("ai_default");
|
||||
expect(referencesFor("CF3", "opencode")).toEqual([...CONFIG_REFERENCE_IDS]);
|
||||
});
|
||||
|
||||
it("refuses a second account-default task in one wave", () => {
|
||||
expect(() => validateConfigPlan(planConfigWave({ ...base, faults: ["CF2"], tasks: 2 }))).toThrow(/Only one ai_default/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("write refusals and the restart gate", () => {
|
||||
it("treats only validation statuses as a product refusal", () => {
|
||||
expect([400, 422].every(isWriteRefusal)).toBe(true);
|
||||
expect([403, 404, 409, 429, null].some(isWriteRefusal)).toBe(false);
|
||||
});
|
||||
|
||||
it("holds writes while a restart runs and waits for writes in flight", async () => {
|
||||
const gate = new RestartGate();
|
||||
const events: string[] = [];
|
||||
let finishWrite!: () => void;
|
||||
const first = gate.write(() => new Promise<void>(resolve => { finishWrite = () => { events.push("write1-end"); resolve(); }; }));
|
||||
const restart = gate.restart(async () => { events.push("restart"); await Promise.resolve(); events.push("restart-end"); });
|
||||
const second = gate.write(async () => { events.push("write2"); });
|
||||
await Promise.resolve();
|
||||
expect(events).toEqual([]);
|
||||
finishWrite();
|
||||
await Promise.all([first, restart, second]);
|
||||
expect(events).toEqual(["write1-end", "restart", "restart-end", "write2"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseConfigWaveArgs", () => {
|
||||
it("keeps waves small", () => {
|
||||
expect(() => parseConfigWaveArgs(["--tasks", "5"])).toThrow(/tasks/);
|
||||
expect(() => parseConfigWaveArgs(["--concurrency", "3"])).toThrow(/concurrency/);
|
||||
expect(parseConfigWaveArgs(["--faults", "CF2", "--seed", "3"]).options).toMatchObject({ faults: ["CF2"], seed: 3, concurrency: 1 });
|
||||
});
|
||||
});
|
||||
|
||||
describe("parkState", () => {
|
||||
it("does not treat an unstarted queued run without a named wait as a rest", () => {
|
||||
const queued = snapshot(100_000, { runs: [{ id: "r1", status: "queued", createdAt: 0, startedAt: null, errorCode: null, error: null }] });
|
||||
expect(parkState(queued, 100_000, 30_000)).toEqual({ kind: "active", reason: "queued_unstarted" });
|
||||
const withCard = { ...queued, cards: [{ id: "c1", kind: "connection_intent", status: "pending", serviceSlug: "openrouter", agentSetup: true }] };
|
||||
expect(parkState(withCard, 100_000, 30_000)).toEqual({ kind: "parked", reason: "queued_with_card" });
|
||||
});
|
||||
|
||||
it("waits for runs to settle before calling an interruption a rest", () => {
|
||||
const failed = snapshot(40_000, { runs: [failedRun("r1", 20_000, "x")], interruption: { state: "failed", canRetry: true, userAction: null, workspaceFailure: null, observedRunId: "r1" } });
|
||||
expect(parkState(failed, 40_000, 30_000).kind).toBe("active");
|
||||
expect(parkState(failed, 60_000, 30_000)).toEqual({ kind: "parked", reason: "interruption" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("gradeConfigTask", () => {
|
||||
const options = { parkBoundMs: 420_000 };
|
||||
const generic = { state: "failed", canRetry: true, userAction: null, workspaceFailure: null, observedRunId: "r1" };
|
||||
|
||||
it("fails CF1 when the command is named only on the run page (master today)", () => {
|
||||
const parked = snapshot(90_000, { runs: [failedRun("r1", 10_000, 'Execution workspace provision command "bash ./scripts/provision-worktree.sh" failed')], interruption: generic });
|
||||
const grade = gradeConfigTask(record({}, { brokenSnapshots: [parked], parkedAt: 90_000, parkReason: "interruption",
|
||||
restart: { kind: "K1", faultId: "f00", firedAt: 95_000, readyAt: 120_000, error: null }, afterRestartSnapshots: [parked], afterRestartParked: true,
|
||||
repair: { at: 200_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 400_000 }), options);
|
||||
expect(grade.pass).toBe(false);
|
||||
expect(grade.checks.find(check => check.name === "actionable_message")?.detail).toMatch(/^message_only_in_run_detail/);
|
||||
expect(grade.signature).toBe("CF1/provision_command:message_only_in_run_detail");
|
||||
});
|
||||
|
||||
it("passes when the task names the setting, survives the restart and completes after repair", () => {
|
||||
const parked = snapshot(90_000, { runs: [failedRun("r1", 10_000, "x")],
|
||||
interruption: { ...generic, workspaceFailure: "The project workspace path is not available" } });
|
||||
const grade = gradeConfigTask(record({ fault: "CF3", reference: "workspace_cwd" }, { brokenSnapshots: [parked], parkedAt: 90_000, parkReason: "interruption",
|
||||
restart: { kind: "K2", faultId: "f00", firedAt: 95_000, readyAt: 120_000, error: null }, afterRestartSnapshots: [parked], afterRestartParked: true,
|
||||
repair: { at: 200_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 400_000 }), options);
|
||||
expect(grade).toMatchObject({ pass: true, outcome: "parked", signature: null });
|
||||
});
|
||||
|
||||
it("fails CF2 on a queued run with automatic setup retries and no card (the #20361 bug)", () => {
|
||||
const stuck = snapshot(400_000, { issueStatus: "todo", runs: [{ id: "r1", status: "queued", createdAt: 5_000, startedAt: null, errorCode: null, error: null }],
|
||||
execution: { label: "Queued", cause: null, nextAction: null },
|
||||
agent: { lifecycleState: "verifying", retryRequired: false,
|
||||
lifecycleError: "Execution configuration check failed. Paperclip could not complete this step. It will retry automatically." } });
|
||||
const grade = gradeConfigTask(record({ fault: "CF2", reference: "ai_default", adapter: "opencode" }, { brokenSnapshots: [stuck],
|
||||
restart: { kind: "K1", faultId: "f00", firedAt: 420_000, readyAt: 450_000, error: null }, afterRestartSnapshots: [stuck], afterRestartParked: false,
|
||||
repair: { at: 600_000, action: "agent setup Retry", detail: "", error: null }, finalStatus: "done", doneAt: 700_000 }), options);
|
||||
expect(grade.pass).toBe(false);
|
||||
expect(grade.outcome).toBe("never_rested");
|
||||
expect(grade.signature).toBe("CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner");
|
||||
});
|
||||
|
||||
it("passes CF2 when the task shows the connection card and parked setup", () => {
|
||||
const parked = snapshot(60_000, { issueStatus: "todo", runs: [{ id: "r1", status: "queued", createdAt: 5_000, startedAt: null, errorCode: null, error: null }],
|
||||
cards: [{ id: "c1", kind: "connection_intent", status: "pending", serviceSlug: "openrouter", agentSetup: true }],
|
||||
agent: { lifecycleState: "verifying", retryRequired: true,
|
||||
lifecycleError: "Connect an AI account and choose the responsible user's personal default for this provider. Automatic verification is paused." } });
|
||||
expect(taskLevelTexts(parked)).toContain("Connect openrouter (agent setup)");
|
||||
const grade = gradeConfigTask(record({ fault: "CF2", reference: "ai_default", adapter: "opencode" }, { brokenSnapshots: [parked], parkedAt: 60_000,
|
||||
parkReason: "queued_with_card", restart: { kind: "K1", faultId: "f00", firedAt: 70_000, readyAt: 90_000, error: null }, afterRestartSnapshots: [parked],
|
||||
afterRestartParked: true, repair: { at: 100_000, action: "completed connection card c1", detail: "", error: null }, finalStatus: "done", doneAt: 200_000 }), options);
|
||||
expect(grade.pass).toBe(true);
|
||||
});
|
||||
|
||||
it("fails a retry loop over unchanged configuration", () => {
|
||||
const runs = [1, 2, 3, 4].map(index => failedRun(`r${index}`, index * 10_000, "x"));
|
||||
const grade = gradeConfigTask(record({}, { brokenSnapshots: [snapshot(400_000, { runs, interruption: { ...generic, workspaceFailure: "provision command failed" } })],
|
||||
parkedAt: 400_000, repair: { at: 500_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 600_000 }), options);
|
||||
expect(grade.checks.find(check => check.name === "no_retry_loop")?.verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("fails when a restart replaces the named message with a generic Retry", () => {
|
||||
const named = snapshot(90_000, { runs: [failedRun("r1", 10_000, "x")],
|
||||
interruption: { ...generic, workspaceFailure: "The project workspace path is not available" } });
|
||||
const generic2 = snapshot(200_000, { runs: [failedRun("r1", 10_000, "x"), failedRun("r2", 130_000, "x")], interruption: { ...generic, observedRunId: "r2" } });
|
||||
const grade = gradeConfigTask(record({ fault: "CF3", reference: "workspace_cwd" }, { brokenSnapshots: [named], parkedAt: 90_000, parkReason: "interruption",
|
||||
restart: { kind: "K1", faultId: "f00", firedAt: 95_000, readyAt: 120_000, error: null }, afterRestartSnapshots: [generic2], afterRestartParked: true,
|
||||
repair: { at: 300_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 400_000 }), options);
|
||||
expect(grade.signature).toBe("CF3/workspace_cwd:message_lost_after_restart");
|
||||
});
|
||||
|
||||
it("counts runs started after the restart toward the retry loop", () => {
|
||||
const parked = snapshot(90_000, { runs: [failedRun("r1", 10_000, "x")], interruption: { ...generic, workspaceFailure: "provision command failed" } });
|
||||
const looping = snapshot(300_000, { runs: [1, 2, 3, 4].map(index => failedRun(`r${index}`, index * 10_000, "x")),
|
||||
interruption: { ...generic, workspaceFailure: "provision command failed" } });
|
||||
const grade = gradeConfigTask(record({}, { brokenSnapshots: [parked], parkedAt: 90_000, restart: { kind: "K2", faultId: "f00", firedAt: 95_000, readyAt: 120_000, error: null },
|
||||
afterRestartSnapshots: [looping], afterRestartParked: true, repair: { at: 400_000, action: "clicked Retry", detail: "", error: null }, finalStatus: "done", doneAt: 500_000 }), options);
|
||||
expect(grade.checks.find(check => check.name === "no_retry_loop")?.verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("does not take generic credential or model failures as naming the setting", () => {
|
||||
expect(CONFIG_REFERENCES.agent_secret.names("abcd1234").test("Invalid credentials")).toBe(false);
|
||||
expect(CONFIG_REFERENCES.agent_secret.names("abcd1234").test('Secret "OPENAI_API_KEY" was deleted')).toBe(true);
|
||||
expect(CONFIG_REFERENCES.model_id.names("abcd1234").test("Model request failed")).toBe(false);
|
||||
expect(CONFIG_REFERENCES.model_id.names("abcd1234").test("userAction:choose_model")).toBe(true);
|
||||
});
|
||||
|
||||
it("counts a write-time refusal as a pass", () => {
|
||||
expect(gradeConfigTask(record({}, { refusedOnWrite: "422 Secret not found" }), options)).toMatchObject({ pass: true, outcome: "refused_on_write" });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,412 @@
|
||||
/**
|
||||
* Config and aged-state fault family (CF*): pure catalog, seeded plan and oracle.
|
||||
*
|
||||
* A fresh chaos instance only ever has valid settings. Production also has
|
||||
* saved settings that point at something that no longer exists (a renamed
|
||||
* setup script, an account default that was never chosen, a deleted secret).
|
||||
* Each CF task breaks one such reference on its own agent or project, and the
|
||||
* oracle is not "the task completes". It is:
|
||||
*
|
||||
* 1. within a bound the task completes, or rests where the user sees a
|
||||
* specific message that names the broken setting (task-level surfaces);
|
||||
* 2. no run retries unchanged configuration forever, and nothing sits
|
||||
* Queued with no live owner and no named wait;
|
||||
* 3. the parked state survives a K1/K2 restart;
|
||||
* 4. after the harness repairs the setting and uses the offered action
|
||||
* (complete the connection card, Retry), the task completes.
|
||||
*
|
||||
* I/O lives in config-wave.ts; this file is credential-free and unit tested.
|
||||
*/
|
||||
import { createRng } from "./scheduler.js";
|
||||
import type { AdapterId } from "./types.js";
|
||||
|
||||
export const CONFIG_FAULT_KINDS = ["CF1", "CF2", "CF3"] as const;
|
||||
export type ConfigFaultKind = (typeof CONFIG_FAULT_KINDS)[number];
|
||||
export const CONFIG_RESTART_KINDS = ["K1", "K2"] as const;
|
||||
export type ConfigRestartKind = (typeof CONFIG_RESTART_KINDS)[number];
|
||||
|
||||
/** A saved setting that refers to another object, and how a user would recognise a message about it. */
|
||||
export interface ConfigReference {
|
||||
id: ConfigReferenceId;
|
||||
owner: "agent" | "project";
|
||||
/** Field path as a user or the API names it. */
|
||||
field: string;
|
||||
refersTo: "script" | "account_default" | "credential" | "workspace_path" | "git_ref" | "model";
|
||||
/** Adapters whose agents carry this setting; absent means every adapter. */
|
||||
adapters?: readonly AdapterId[];
|
||||
/** An actionable message names this setting (case-insensitive). The nonce placeholder is filled per task. */
|
||||
names: (nonce: string) => RegExp;
|
||||
label: string;
|
||||
}
|
||||
|
||||
export const CONFIG_REFERENCE_IDS = ["provision_command", "ai_default", "agent_secret", "workspace_cwd", "base_ref", "model_id"] as const;
|
||||
export type ConfigReferenceId = (typeof CONFIG_REFERENCE_IDS)[number];
|
||||
|
||||
/** The stale setup script CF1 points at: the path #20269 renamed in the Paperclip repository. */
|
||||
export const STALE_PROVISION_COMMAND = "bash ./scripts/provision-worktree.sh";
|
||||
|
||||
export const CONFIG_REFERENCES: Record<ConfigReferenceId, ConfigReference> = {
|
||||
provision_command: {
|
||||
id: "provision_command", owner: "project", field: "executionWorkspacePolicy.workspaceStrategy.provisionCommand",
|
||||
refersTo: "script", label: "Project workspace setup command points at a missing script",
|
||||
names: () => /provision-worktree\.sh|(setup|provision) command/i,
|
||||
},
|
||||
ai_default: {
|
||||
id: "ai_default", owner: "agent", field: "runtimeConfig.aiConnection (responsible_user default)",
|
||||
refersTo: "account_default", adapters: ["opencode"], label: "OpenRouter binding selects a personal account default that does not exist",
|
||||
names: () => /openrouter|personal default|connect (an |your )?(ai )?account/i,
|
||||
},
|
||||
agent_secret: {
|
||||
id: "agent_secret", owner: "agent", field: "adapterConfig.env.<provider key> (secret_ref)",
|
||||
refersTo: "credential", label: "Agent credential secret was deleted",
|
||||
names: nonce => new RegExp(`\\bsecret\\b|[A-Z]+_API_KEY|_${nonce}\\b`, "i"),
|
||||
},
|
||||
workspace_cwd: {
|
||||
id: "workspace_cwd", owner: "project", field: "project workspace cwd",
|
||||
refersTo: "workspace_path", label: "Project workspace path no longer exists",
|
||||
names: nonce => new RegExp(`project workspace|workspace path|workspace folder|chaos-missing-${nonce}`, "i"),
|
||||
},
|
||||
base_ref: {
|
||||
id: "base_ref", owner: "project", field: "executionWorkspacePolicy.workspaceStrategy.baseRef",
|
||||
refersTo: "git_ref", label: "Worktree base ref names a branch that no longer exists",
|
||||
names: nonce => new RegExp(`base ref|base branch|chaos-missing-ref-${nonce}`, "i"),
|
||||
},
|
||||
model_id: {
|
||||
id: "model_id", owner: "agent", field: "adapterConfig.model",
|
||||
refersTo: "model", label: "Agent model id no longer exists",
|
||||
names: nonce => new RegExp(`chaos-missing-model-${nonce}|choose_model|(unknown|unsupported|invalid|unavailable) model|model[^.]{0,40}(not found|unknown|unsupported|does not exist|not available|rejected)`, "i"),
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Settings that reference another object but that CF3 does not break, and why.
|
||||
* Reported with every wave so the coverage gap stays visible.
|
||||
*/
|
||||
export const CONFIG_REFERENCES_NOT_BROKEN: Record<string, string> = {
|
||||
"agent skills (paperclipSkillSync.desiredSkills)": "deleting a skill an agent uses is refused (422) and unknown keys are rejected on write",
|
||||
"agent defaultEnvironmentId": "the FK is ON DELETE SET NULL and an instance has one local environment, so a local wave cannot leave it dangling",
|
||||
"project executionWorkspacePolicy.environmentId": "not read when choosing a run's environment, so breaking it changes nothing",
|
||||
"agent instructionsFilePath": "native runner agents use an instructions bundle; a missing file is only a warning for local adapters",
|
||||
"agent adapterConfig.cwd": "created on demand; project tasks use the execution workspace instead",
|
||||
};
|
||||
|
||||
/** Which references a CF kind may break for an adapter. */
|
||||
export function referencesFor(kind: ConfigFaultKind, adapter: AdapterId): ConfigReferenceId[] {
|
||||
if (kind === "CF1") return ["provision_command"];
|
||||
if (kind === "CF2") return ["ai_default"];
|
||||
return CONFIG_REFERENCE_IDS.filter(id => !CONFIG_REFERENCES[id].adapters || CONFIG_REFERENCES[id].adapters!.includes(adapter));
|
||||
}
|
||||
|
||||
export interface ConfigTaskPlan {
|
||||
taskId: string;
|
||||
index: number;
|
||||
fault: ConfigFaultKind;
|
||||
reference: ConfigReferenceId;
|
||||
adapter: AdapterId;
|
||||
/** Restart fired while the task is parked (or at the park bound when it never parks). */
|
||||
restart: ConfigRestartKind;
|
||||
nonce: string;
|
||||
}
|
||||
|
||||
export interface ConfigWaveOptions {
|
||||
seed: number;
|
||||
faults: ConfigFaultKind[];
|
||||
adapters: AdapterId[];
|
||||
tasks: number;
|
||||
concurrency: number;
|
||||
restarts: ConfigRestartKind[];
|
||||
/** CF3 only: break exactly these references, in turn (default: seeded over every applicable reference). */
|
||||
references?: ConfigReferenceId[];
|
||||
parkBoundMs: number;
|
||||
repairBudgetMs: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Seeded plan. CF2's reference lives on an OpenCode agent, so CF2 tasks always
|
||||
* use the opencode adapter. CF3 walks a seeded shuffle of the applicable
|
||||
* references so a small wave still covers distinct settings.
|
||||
*/
|
||||
export function planConfigWave(options: ConfigWaveOptions): ConfigTaskPlan[] {
|
||||
const rng = createRng(options.seed ^ 0x0cf0cf);
|
||||
const tasks: ConfigTaskPlan[] = [];
|
||||
const cf3Queue: Partial<Record<AdapterId, ConfigReferenceId[]>> = {};
|
||||
for (let index = 0; index < options.tasks; index += 1) {
|
||||
const fault = options.faults[index % options.faults.length]!;
|
||||
const adapter: AdapterId = fault === "CF2" ? "opencode" : options.adapters[index % options.adapters.length]!;
|
||||
let reference: ConfigReferenceId;
|
||||
if (fault === "CF3") {
|
||||
const allowed = referencesFor("CF3", adapter).filter(id => !options.references || options.references.includes(id));
|
||||
if (!allowed.length) throw new Error(`CF3 has no reference to break for adapter ${adapter}`);
|
||||
let queue = cf3Queue[adapter];
|
||||
if (!queue?.length) queue = cf3Queue[adapter] = rng.shuffle(allowed);
|
||||
reference = queue.shift()!;
|
||||
} else {
|
||||
reference = referencesFor(fault, adapter)[0]!;
|
||||
}
|
||||
tasks.push({ taskId: `t${String(index + 1).padStart(2, "0")}`, index, fault, reference, adapter,
|
||||
// Alternate restart kinds per fault kind, so two tasks of one fault cross both K1 and K2.
|
||||
restart: options.restarts[tasks.filter(earlier => earlier.fault === fault).length % options.restarts.length]!, nonce: rng.hex(8) });
|
||||
}
|
||||
return tasks;
|
||||
}
|
||||
|
||||
/**
|
||||
* The responsible user's account default is company-wide: once one task's
|
||||
* repair connects an account, a later task's binding is no longer broken, and
|
||||
* the product has no route to clear a default. One ai_default task per wave.
|
||||
*/
|
||||
export function validateConfigPlan(tasks: readonly ConfigTaskPlan[]): void {
|
||||
const aiDefault = tasks.filter(task => task.reference === "ai_default");
|
||||
if (aiDefault.length > 1) {
|
||||
throw new Error(`Only one ai_default (CF2) task fits in a wave; got ${aiDefault.map(task => task.taskId).join(", ")}. Lower --tasks or use --references.`);
|
||||
}
|
||||
}
|
||||
|
||||
/** One poll of everything a user could see about the task, plus run state. */
|
||||
export interface ConfigSnapshot {
|
||||
at: number;
|
||||
reachable: boolean;
|
||||
issueStatus: string | null;
|
||||
runs: Array<{ id: string; status: string; createdAt: number | null; startedAt: number | null; errorCode: string | null; error: string | null }>;
|
||||
interruption: { state: string | null; canRetry: boolean; userAction: string | null; workspaceFailure: string | null; observedRunId: string | null } | null;
|
||||
execution: { label: string | null; cause: string | null; nextAction: string | null } | null;
|
||||
/** Pending thread interactions on the task (connection cards and similar). */
|
||||
cards: Array<{ id: string; kind: string; status: string; serviceSlug: string | null; agentSetup: boolean }>;
|
||||
agent: { lifecycleState: string | null; lifecycleError: string | null; retryRequired: boolean } | null;
|
||||
/** Non-agent comments on the task (recovery notices). */
|
||||
notices: string[];
|
||||
}
|
||||
|
||||
/** Text a user sees on the task page: the interruption, the execution status, cards, agent setup status and notices. */
|
||||
export function taskLevelTexts(snapshot: ConfigSnapshot): string[] {
|
||||
const texts: string[] = [];
|
||||
const interruption = snapshot.interruption;
|
||||
if (interruption?.workspaceFailure) texts.push(interruption.workspaceFailure);
|
||||
if (interruption?.userAction) texts.push(`userAction:${interruption.userAction}`);
|
||||
if (snapshot.execution?.label) texts.push(snapshot.execution.label);
|
||||
if (snapshot.execution?.nextAction) texts.push(snapshot.execution.nextAction);
|
||||
for (const card of snapshot.cards) {
|
||||
if (card.status !== "pending") continue;
|
||||
texts.push(card.kind === "connection_intent" ? `Connect ${card.serviceSlug ?? "an account"}${card.agentSetup ? " (agent setup)" : ""}` : card.kind);
|
||||
}
|
||||
if (snapshot.agent?.lifecycleError) texts.push(snapshot.agent.lifecycleError);
|
||||
texts.push(...snapshot.notices);
|
||||
return texts;
|
||||
}
|
||||
|
||||
/** Run error text: visible only on the run's own page, not on the task. */
|
||||
export function runDetailTexts(snapshot: ConfigSnapshot): string[] {
|
||||
return snapshot.runs.flatMap(run => run.error ? [run.error] : []);
|
||||
}
|
||||
|
||||
export function namingTexts(texts: readonly string[], pattern: RegExp): string[] {
|
||||
return texts.filter(text => pattern.test(text));
|
||||
}
|
||||
|
||||
export type ParkState =
|
||||
| { kind: "done" }
|
||||
| { kind: "active"; reason: string }
|
||||
| { kind: "parked"; reason: string };
|
||||
|
||||
/**
|
||||
* Whether the task rests. A rest needs no running run, no run created in the
|
||||
* last `settleMs`, and something on the task that waits on a person: a pending
|
||||
* card, an interruption offering Retry or a user action, an agent whose setup
|
||||
* needs repair, or a blocked/cancelled task. An unstarted queued run counts as
|
||||
* a rest only with a pending card or parked agent setup naming why it waits.
|
||||
*/
|
||||
export function parkState(snapshot: ConfigSnapshot, now: number, settleMs: number): ParkState {
|
||||
if (snapshot.issueStatus === "done") return { kind: "done" };
|
||||
if (snapshot.runs.some(run => run.status === "running")) return { kind: "active", reason: "run_running" };
|
||||
const newest = Math.max(0, ...snapshot.runs.map(run => run.createdAt ?? 0));
|
||||
if (newest && now - newest < settleMs) return { kind: "active", reason: "run_recently_created" };
|
||||
const pendingCard = snapshot.cards.some(card => card.status === "pending");
|
||||
const setupParked = snapshot.agent?.retryRequired === true;
|
||||
const queued = snapshot.runs.some(run => run.status === "queued" && run.startedAt === null);
|
||||
if (queued) return pendingCard || setupParked ? { kind: "parked", reason: pendingCard ? "queued_with_card" : "queued_setup_parked" }
|
||||
: { kind: "active", reason: "queued_unstarted" };
|
||||
if (pendingCard) return { kind: "parked", reason: "card_pending" };
|
||||
if (snapshot.interruption && (snapshot.interruption.canRetry || snapshot.interruption.userAction)) return { kind: "parked", reason: "interruption" };
|
||||
if (setupParked) return { kind: "parked", reason: "agent_setup_parked" };
|
||||
if (snapshot.issueStatus === "blocked" || snapshot.issueStatus === "cancelled") return { kind: "parked", reason: `issue_${snapshot.issueStatus}` };
|
||||
return { kind: "active", reason: "no_named_wait" };
|
||||
}
|
||||
|
||||
export interface ConfigTaskRecord {
|
||||
plan: ConfigTaskPlan;
|
||||
issueId: string | null;
|
||||
agentId: string | null;
|
||||
projectId: string | null;
|
||||
launchedAt: number | null;
|
||||
/** The break was refused on write (validation): the product prevents the stale reference. */
|
||||
refusedOnWrite: string | null;
|
||||
/** The harness itself could not set the task up (not a product verdict). */
|
||||
harnessError: string | null;
|
||||
brokenSnapshots: ConfigSnapshot[];
|
||||
parkedAt: number | null;
|
||||
parkReason: string | null;
|
||||
restart: { kind: ConfigRestartKind; faultId: string | null; firedAt: number; readyAt: number | null; error: string | null } | null;
|
||||
/** Every poll after the restart, while the setting is still broken. */
|
||||
afterRestartSnapshots: ConfigSnapshot[];
|
||||
/** The task rested again (or completed) after the restart. */
|
||||
afterRestartParked: boolean | null;
|
||||
/** Last snapshot after the repair (evidence when the task never completes). */
|
||||
afterRepair?: ConfigSnapshot | null;
|
||||
repair: { at: number; action: string; detail: string; error: string | null } | null;
|
||||
doneAt: number | null;
|
||||
finalStatus: string | null;
|
||||
/** A1 brief oracle result after repair. */
|
||||
brief: { ok: boolean; detail: string } | null;
|
||||
}
|
||||
|
||||
/** Status codes that mean the server validated the request and refused the stale value. */
|
||||
export function isWriteRefusal(status: number | null): boolean {
|
||||
return status === 400 || status === 422;
|
||||
}
|
||||
|
||||
/**
|
||||
* Restarts and harness writes must not overlap: a write sent while the server
|
||||
* is down fails, and the task would be graded on a harness error. Writes share
|
||||
* the gate; a restart waits for writes in flight and holds new ones.
|
||||
*/
|
||||
export class RestartGate {
|
||||
private writers = 0;
|
||||
private restarting: Promise<void> | null = null;
|
||||
private idle: Array<() => void> = [];
|
||||
|
||||
async write<T>(work: () => Promise<T>): Promise<T> {
|
||||
while (this.restarting) await this.restarting;
|
||||
this.writers += 1;
|
||||
try { return await work(); } finally {
|
||||
this.writers -= 1;
|
||||
if (!this.writers) for (const wake of this.idle.splice(0)) wake();
|
||||
}
|
||||
}
|
||||
|
||||
async restart<T>(work: () => Promise<T>): Promise<T> {
|
||||
while (this.restarting) await this.restarting;
|
||||
let release!: () => void;
|
||||
this.restarting = new Promise<void>(resolve => { release = resolve; });
|
||||
try {
|
||||
while (this.writers) await new Promise<void>(resolve => this.idle.push(resolve));
|
||||
return await work();
|
||||
} finally {
|
||||
this.restarting = null;
|
||||
release();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export interface ConfigCheck { name: string; verdict: "pass" | "fail" | "n/a"; detail: string }
|
||||
export interface ConfigGrade { pass: boolean; outcome: "completed" | "parked" | "refused_on_write" | "never_rested" | "harness_error"; checks: ConfigCheck[]; signature: string | null }
|
||||
|
||||
/** Runs created after this many attempts of unchanged configuration are a retry loop. */
|
||||
export const MAX_BROKEN_RUNS = 3;
|
||||
/** An unstarted queued run older than this with no named wait has no live owner. */
|
||||
export const OWNERLESS_QUEUE_MS = 3 * 60_000;
|
||||
|
||||
export function gradeConfigTask(record: ConfigTaskRecord, options: { parkBoundMs: number }): ConfigGrade {
|
||||
const checks: ConfigCheck[] = [];
|
||||
const add = (name: string, verdict: ConfigCheck["verdict"], detail: string) => checks.push({ name, verdict, detail });
|
||||
if (record.harnessError) {
|
||||
return { pass: false, outcome: "harness_error", checks: [{ name: "harness", verdict: "fail", detail: record.harnessError }], signature: "harness_error" };
|
||||
}
|
||||
if (record.refusedOnWrite) {
|
||||
return { pass: true, outcome: "refused_on_write", checks: [{ name: "refused_on_write", verdict: "pass", detail: record.refusedOnWrite }], signature: null };
|
||||
}
|
||||
const reference = CONFIG_REFERENCES[record.plan.reference];
|
||||
const pattern = reference.names(record.plan.nonce);
|
||||
const last = record.brokenSnapshots.filter(snapshot => snapshot.reachable).at(-1) ?? null;
|
||||
const afterRestart = record.afterRestartSnapshots.filter(snapshot => snapshot.reachable);
|
||||
// Every sample while the setting was broken, before and after the restart.
|
||||
const whileBroken = [...record.brokenSnapshots, ...afterRestart];
|
||||
const lastBroken = afterRestart.at(-1) ?? last;
|
||||
const completedWhileBroken = last?.issueStatus === "done";
|
||||
const outcome: ConfigGrade["outcome"] = completedWhileBroken ? "completed" : record.parkedAt !== null ? "parked" : "never_rested";
|
||||
|
||||
// 1. Bounded rest.
|
||||
if (completedWhileBroken) add("bounded_rest", "pass", "completed despite the broken setting");
|
||||
else if (record.parkedAt !== null && record.launchedAt !== null) {
|
||||
add("bounded_rest", record.parkedAt - record.launchedAt <= options.parkBoundMs ? "pass" : "fail",
|
||||
`parked (${record.parkReason}) after ${Math.round((record.parkedAt - record.launchedAt) / 1000)} s`);
|
||||
} else add("bounded_rest", "fail", `no rest within ${Math.round(options.parkBoundMs / 1000)} s (last: ${last ? parkState(last, last.at, 0).kind : "unreachable"})`);
|
||||
|
||||
// 2. Actionable message naming the setting, on the task.
|
||||
if (completedWhileBroken) add("actionable_message", "n/a", "completed");
|
||||
else {
|
||||
const restingSnapshots = record.parkedAt !== null ? record.brokenSnapshots.filter(snapshot => snapshot.at >= record.parkedAt!) : record.brokenSnapshots;
|
||||
const named = [...new Set(restingSnapshots.flatMap(snapshot => namingTexts(taskLevelTexts(snapshot), pattern)))];
|
||||
const seen = [...new Set(restingSnapshots.flatMap(taskLevelTexts))];
|
||||
const runOnly = [...new Set(restingSnapshots.flatMap(snapshot => namingTexts(runDetailTexts(snapshot), pattern)))];
|
||||
if (named.length) add("actionable_message", "pass", truncate(named[0]!));
|
||||
else add("actionable_message", "fail", `${runOnly.length ? "message_only_in_run_detail: " + truncate(runOnly[0]!) + "; " : ""}user saw: ${seen.map(text => truncate(text)).join(" | ") || "nothing"}`);
|
||||
}
|
||||
|
||||
// 3. No retry loop over unchanged configuration.
|
||||
const brokenRuns = new Set(whileBroken.flatMap(snapshot => snapshot.runs.map(run => run.id)));
|
||||
const lifecycleRetrying = whileBroken.some(snapshot => snapshot.agent?.lifecycleError && /retry automatically/i.test(snapshot.agent.lifecycleError)
|
||||
&& snapshot.agent.retryRequired !== true);
|
||||
if (brokenRuns.size > MAX_BROKEN_RUNS) add("no_retry_loop", "fail", `retry_loop: ${brokenRuns.size} runs while the setting stayed broken`);
|
||||
else if (lifecycleRetrying && !completedWhileBroken) add("no_retry_loop", "fail", "lifecycle_retrying_unchanged_config: agent setup says it will retry automatically");
|
||||
else add("no_retry_loop", "pass", `${brokenRuns.size} run(s) while broken`);
|
||||
|
||||
// 4. Nothing queued without a live owner.
|
||||
const ownerless = lastBroken ? ownerlessQueuedRun(lastBroken, OWNERLESS_QUEUE_MS) : null;
|
||||
add("no_ownerless_queue", ownerless ? "fail" : "pass", ownerless ? `queued_without_live_owner: run ${ownerless} unstarted with no named wait` : "none");
|
||||
|
||||
// 5. Parked state survives the restart.
|
||||
if (!record.restart) add("survives_restart", "n/a", "no restart fired");
|
||||
else if (record.restart.error) add("survives_restart", "fail", `restart failed: ${record.restart.error}`);
|
||||
else if (completedWhileBroken) add("survives_restart", "n/a", "completed before the restart");
|
||||
else if (record.parkedAt === null) add("survives_restart", "n/a", "never parked");
|
||||
else if (!record.afterRestartParked) {
|
||||
const final = afterRestart.at(-1);
|
||||
add("survives_restart", "fail", `rest_lost_after_restart: ${record.restart.kind} while parked; then ${final ? stateLabel(parkState(final, final.at, 0)) : "unreachable"}`);
|
||||
} else {
|
||||
// The rest must still name the setting: a restart that leaves only a generic Retry fails.
|
||||
const final = afterRestart.at(-1);
|
||||
const named = final ? namingTexts(taskLevelTexts(final), pattern) : [];
|
||||
const namedBefore = checks.find(check => check.name === "actionable_message")?.verdict === "pass";
|
||||
if (final?.issueStatus === "done" || named.length || !namedBefore) {
|
||||
add("survives_restart", "pass", `${record.restart.kind} while parked; parked again${named.length ? `: ${truncate(named[0]!)}` : ""}`);
|
||||
} else {
|
||||
add("survives_restart", "fail", `message_lost_after_restart: ${record.restart.kind}; user then saw: ${final ? taskLevelTexts(final).map(text => truncate(text)).join(" | ") || "nothing" : "nothing"}`);
|
||||
}
|
||||
}
|
||||
|
||||
// 6. Repair plus the offered action completes the task.
|
||||
if (completedWhileBroken) add("repair_completes", "n/a", "completed while broken");
|
||||
else if (!record.repair) add("repair_completes", "fail", "repair not attempted");
|
||||
else if (record.repair.error) add("repair_completes", "fail", `repair failed: ${record.repair.error}`);
|
||||
else add("repair_completes", record.finalStatus === "done" ? "pass" : "fail",
|
||||
`${record.repair.action}; final status ${record.finalStatus ?? "unknown"}${record.doneAt ? ` after ${Math.round((record.doneAt - record.repair.at) / 1000)} s` : ""}`);
|
||||
|
||||
// 7. The work itself (A1 brief).
|
||||
if (record.brief) add("brief_oracle", record.brief.ok ? "pass" : "fail", record.brief.detail);
|
||||
else add("brief_oracle", "fail", "not collected");
|
||||
|
||||
const failed = checks.filter(check => check.verdict === "fail");
|
||||
return { pass: failed.length === 0, outcome, checks,
|
||||
signature: failed.length ? `${record.plan.fault}/${record.plan.reference}:${failed.map(failureCode).join("+")}` : null };
|
||||
}
|
||||
|
||||
/** A check's detail starting with a snake_case code names the failure; otherwise the check name does. */
|
||||
function failureCode(check: ConfigCheck): string {
|
||||
const match = /^([a-z][a-z0-9_]+):/.exec(check.detail);
|
||||
return match ? match[1]! : check.name;
|
||||
}
|
||||
|
||||
export function ownerlessQueuedRun(snapshot: ConfigSnapshot, olderThanMs: number): string | null {
|
||||
if (snapshot.cards.some(card => card.status === "pending") || snapshot.agent?.retryRequired) return null;
|
||||
const stale = snapshot.runs.find(run => run.status === "queued" && run.startedAt === null && run.createdAt !== null && snapshot.at - run.createdAt > olderThanMs);
|
||||
return stale?.id ?? null;
|
||||
}
|
||||
|
||||
function stateLabel(state: ParkState): string {
|
||||
return state.kind === "done" ? "done" : `${state.kind} (${state.reason})`;
|
||||
}
|
||||
|
||||
function truncate(text: string, max = 220): string {
|
||||
const flat = text.replace(/\s+/g, " ").trim();
|
||||
return flat.length > max ? `${flat.slice(0, max - 1)}…` : flat;
|
||||
}
|
||||
@@ -0,0 +1,504 @@
|
||||
/**
|
||||
* Config and aged-state fault waves (CF1–CF3). See config-faults.ts for the
|
||||
* oracle and README.md ("Config faults") for usage.
|
||||
*
|
||||
* node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts \
|
||||
* --faults CF1,CF3 --adapters codex --tasks 4 --concurrency 1 --restarts K1,K2 --seed 61
|
||||
*
|
||||
* Each task gets its own agent, credential secret and project, so breaking
|
||||
* one task's setting never touches another task. The harness breaks the
|
||||
* setting through the board routes a user would use, launches an A1 brief
|
||||
* task, watches it rest, restarts the control plane while it is parked,
|
||||
* repairs the setting, uses the offered action, and grades the result.
|
||||
*/
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { existsSync } from "node:fs";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { parseArgs } from "node:util";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { releaseInstance } from "../runner-e2e/instance-retention.js";
|
||||
import { ChaosApi, ChaosApiError } from "./api.js";
|
||||
import {
|
||||
CONFIG_FAULT_KINDS, CONFIG_REFERENCE_IDS, CONFIG_REFERENCES, CONFIG_REFERENCES_NOT_BROKEN, CONFIG_RESTART_KINDS,
|
||||
RestartGate, STALE_PROVISION_COMMAND, gradeConfigTask, isWriteRefusal, parkState, planConfigWave, validateConfigPlan,
|
||||
type ConfigGrade, type ConfigSnapshot, type ConfigTaskPlan, type ConfigTaskRecord, type ConfigWaveOptions,
|
||||
} from "./config-faults.js";
|
||||
import { CREDENTIAL_FOR_ADAPTER, waveIdFor } from "./config.js";
|
||||
import { knownSecretValues, redactKnownSecrets } from "./evidence.js";
|
||||
import { seedChaosLab, waitForAgentsReady, type ChaosLab } from "./seed.js";
|
||||
import { ChaosSupervisor } from "./supervisor.js";
|
||||
import { Timeline } from "./timeline.js";
|
||||
import { guardedRunnerOutputs, worktreeFingerprint } from "./wave-guard.js";
|
||||
import { ADAPTER_IDS } from "./types.js";
|
||||
import { workloadDefinitions } from "./workloads.js";
|
||||
|
||||
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
|
||||
const resultsRoot = path.resolve(process.env.RESTART_CHAOS_RESULTS_DIR ?? path.join(import.meta.dirname, "results"));
|
||||
const POLL_MS = 4_000;
|
||||
/** No run created for this long before a rest counts. */
|
||||
const SETTLE_MS = 30_000;
|
||||
/** After the restart, how long the task may take to rest again. */
|
||||
const RESTART_REPARK_MS = 150_000;
|
||||
type Json = Record<string, any>;
|
||||
|
||||
export function parseConfigWaveArgs(argv: readonly string[]): { options: ConfigWaveOptions; keep: boolean } {
|
||||
const { values } = parseArgs({ args: argv[0] === "--" ? argv.slice(1) : [...argv], strict: true, options: {
|
||||
faults: { type: "string" }, adapters: { type: "string" }, tasks: { type: "string" }, concurrency: { type: "string" },
|
||||
restarts: { type: "string" }, seed: { type: "string" }, references: { type: "string" },
|
||||
"park-seconds": { type: "string" }, "repair-seconds": { type: "string" }, keep: { type: "boolean" },
|
||||
} });
|
||||
const list = <T extends string>(raw: string | undefined, allowed: readonly T[], name: string, fallback: T[]): T[] => {
|
||||
if (raw === undefined) return fallback;
|
||||
const items = raw.split(",").map(item => item.trim()).filter(Boolean);
|
||||
for (const item of items) if (!allowed.includes(item as T)) throw new Error(`Unknown ${name} "${item}"; allowed: ${allowed.join(", ")}`);
|
||||
if (!items.length) throw new Error(`--${name} must not be empty`);
|
||||
return [...new Set(items)] as T[];
|
||||
};
|
||||
const int = (raw: string | undefined, name: string, fallback: number, min: number, max: number) => {
|
||||
if (raw === undefined) return fallback;
|
||||
const value = Number(raw);
|
||||
if (!Number.isSafeInteger(value) || value < min || value > max) throw new Error(`--${name} must be an integer between ${min} and ${max}`);
|
||||
return value;
|
||||
};
|
||||
const options: ConfigWaveOptions = {
|
||||
faults: list(values.faults, CONFIG_FAULT_KINDS, "faults", ["CF1", "CF3"]),
|
||||
adapters: list(values.adapters, ADAPTER_IDS, "adapters", ["codex"]),
|
||||
tasks: int(values.tasks, "tasks", 2, 1, 4),
|
||||
// Disk and the one-wave-per-machine rule keep these waves small.
|
||||
concurrency: int(values.concurrency, "concurrency", 1, 1, 2),
|
||||
restarts: list(values.restarts, CONFIG_RESTART_KINDS, "restarts", ["K1", "K2"]),
|
||||
seed: int(values.seed, "seed", Date.now() % 100_000, 0, 2 ** 31 - 1),
|
||||
...(values.references ? { references: list(values.references, CONFIG_REFERENCE_IDS, "references", []) } : {}),
|
||||
parkBoundMs: int(values["park-seconds"], "park-seconds", 420, 60, 1_800) * 1_000,
|
||||
repairBudgetMs: int(values["repair-seconds"], "repair-seconds", 900, 120, 3_600) * 1_000,
|
||||
};
|
||||
return { options, keep: values.keep === true };
|
||||
}
|
||||
|
||||
function preflight(tasks: readonly ConfigTaskPlan[]) {
|
||||
const missing = [...new Set(tasks.map(task => CREDENTIAL_FOR_ADAPTER[task.adapter]))].filter(key => !process.env[key]);
|
||||
if (missing.length) throw new Error(`Missing credentials ${missing.join(", ")}. Run: set -a; source ~/.secrets; set +a`);
|
||||
const runner = process.env.PAPERCLIP_RUNNER_BINARY ?? path.join(repositoryRoot, "packages/paperclip-runner/runner/target/debug/paperclip-runnerd");
|
||||
if (!existsSync(runner)) throw new Error(`Runner binary missing at ${runner}. Run: node scripts/agent-preflight.mjs`);
|
||||
const build = spawnSync(process.execPath, [path.join(repositoryRoot, "packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs")],
|
||||
{ cwd: repositoryRoot, stdio: "inherit" });
|
||||
if (build.status !== 0) throw new Error("Verified provider entrypoint build failed");
|
||||
}
|
||||
|
||||
/** A task's own agent, credential and project: the objects its broken setting refers to. */
|
||||
interface TaskResources {
|
||||
agentId: string;
|
||||
agentBody: Json;
|
||||
credentialKey: string;
|
||||
secretId: string | null;
|
||||
projectId: string;
|
||||
workspaceId: string;
|
||||
basePolicy: Json;
|
||||
/** CF2 repair: the OpenRouter account the harness connected. */
|
||||
connectionId?: string;
|
||||
}
|
||||
|
||||
function basePolicy(lab: ChaosLab): Json {
|
||||
return { enabled: true, defaultMode: "isolated_workspace", allowIssueOverride: false, environmentId: lab.environmentId,
|
||||
workspaceStrategy: { type: "git_worktree" } };
|
||||
}
|
||||
|
||||
const secretRef = (secretId: string) => ({ type: "secret_ref", secretId, version: "latest" });
|
||||
|
||||
async function createTaskResources(api: ChaosApi, lab: ChaosLab, task: ConfigTaskPlan, timeline: Timeline): Promise<TaskResources> {
|
||||
const source = lab.agents[task.adapter]?.body;
|
||||
if (!source) throw new Error(`Seeded lab has no ${task.adapter} agent body to clone`);
|
||||
const credentialKey = CREDENTIAL_FOR_ADAPTER[task.adapter];
|
||||
const adapterConfig = { ...((source.adapterConfig ?? {}) as Json) };
|
||||
const env = { ...((adapterConfig.env ?? {}) as Json) };
|
||||
const runtimeConfig = { ...((source.runtimeConfig ?? {}) as Json) };
|
||||
let secretId: string | null = null;
|
||||
if (task.reference === "ai_default") {
|
||||
// The responsible user's personal default; the fresh company has no OpenRouter account at all.
|
||||
delete env[credentialKey];
|
||||
runtimeConfig.aiConnection = { provider: "openrouter", method: "api_key", mode: "responsible_user" };
|
||||
} else {
|
||||
const secret = await api.post<{ id: string }>(`/api/companies/${lab.companyId}/secrets`, {
|
||||
// Secret keys are unique per company; the agent's env var name stays the provider's.
|
||||
name: `Chaos ${credentialKey} ${task.taskId}-${task.nonce}`, key: `${credentialKey}_${task.taskId}_${task.nonce}`, value: process.env[credentialKey],
|
||||
description: `Restart chaos config fault ${task.taskId}` });
|
||||
secretId = secret.id;
|
||||
env[credentialKey] = secretRef(secret.id);
|
||||
}
|
||||
const agentBody: Json = { ...source, name: `Chaos ${task.adapter} ${task.taskId}`, adapterConfig: { ...adapterConfig, env }, runtimeConfig };
|
||||
const agent = await api.post<{ id: string }>(`/api/companies/${lab.companyId}/agents`, agentBody);
|
||||
const policy = basePolicy(lab);
|
||||
const project = await api.post<{ id: string }>(`/api/companies/${lab.companyId}/projects`, {
|
||||
name: `cf-${task.taskId}-${task.nonce}`, description: `Restart chaos config fault ${task.fault}/${task.reference}`,
|
||||
executionWorkspacePolicy: policy,
|
||||
workspace: { name: "Primary", sourceType: "local_path", cwd: lab.fixtureRepositoryPath, isPrimary: true },
|
||||
});
|
||||
const full = await api.get<Json>(`/api/projects/${project.id}`);
|
||||
const workspaceId = full.primaryWorkspace?.id ?? full.workspaces?.[0]?.id;
|
||||
if (!workspaceId) throw new Error(`Project ${project.id} has no primary workspace`);
|
||||
timeline.add("cf_task_resources", { taskId: task.taskId, agentId: agent.id, projectId: project.id, workspaceId, secretId });
|
||||
return { agentId: agent.id, agentBody, credentialKey, secretId, projectId: project.id, workspaceId, basePolicy: policy };
|
||||
}
|
||||
|
||||
/** Break the task's reference through the board routes. Returns a refusal when the server rejected the stale value on write. */
|
||||
async function breakReference(api: ChaosApi, lab: ChaosLab, task: ConfigTaskPlan, resources: TaskResources): Promise<string | null> {
|
||||
try {
|
||||
switch (task.reference) {
|
||||
case "provision_command":
|
||||
await api.patch(`/api/projects/${resources.projectId}`, { executionWorkspacePolicy: { ...resources.basePolicy,
|
||||
workspaceStrategy: { type: "git_worktree", provisionCommand: STALE_PROVISION_COMMAND } } });
|
||||
return null;
|
||||
case "base_ref":
|
||||
await api.patch(`/api/projects/${resources.projectId}`, { executionWorkspacePolicy: { ...resources.basePolicy,
|
||||
workspaceStrategy: { type: "git_worktree", baseRef: `chaos-missing-ref-${task.nonce}` } } });
|
||||
return null;
|
||||
case "workspace_cwd":
|
||||
await api.patch(`/api/projects/${resources.projectId}/workspaces/${resources.workspaceId}`,
|
||||
{ cwd: path.join(path.dirname(lab.fixtureRepositoryPath), `chaos-missing-${task.nonce}`) });
|
||||
return null;
|
||||
case "agent_secret":
|
||||
await api.delete(`/api/secrets/${resources.secretId}`);
|
||||
return null;
|
||||
case "model_id":
|
||||
await api.patch(`/api/agents/${resources.agentId}`, { adapterConfig: { model: `chaos-missing-model-${task.nonce}` } });
|
||||
return null;
|
||||
case "ai_default":
|
||||
// Broken at creation: the binding selects a personal default the company never had.
|
||||
return null;
|
||||
}
|
||||
} catch (error) {
|
||||
// Only a validation refusal proves the product rejects the stale value; a 403, 404 or 429 is a harness failure.
|
||||
if (error instanceof ChaosApiError && isWriteRefusal(error.status)) return error.message.slice(0, 400);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/** Put the setting back the way a user would. */
|
||||
async function repairReference(api: ChaosApi, lab: ChaosLab, task: ConfigTaskPlan, resources: TaskResources): Promise<string> {
|
||||
switch (task.reference) {
|
||||
case "provision_command":
|
||||
case "base_ref":
|
||||
await api.patch(`/api/projects/${resources.projectId}`, { executionWorkspacePolicy: resources.basePolicy });
|
||||
return "restored the project workspace policy without the stale value";
|
||||
case "workspace_cwd":
|
||||
await api.patch(`/api/projects/${resources.projectId}/workspaces/${resources.workspaceId}`, { cwd: lab.fixtureRepositoryPath });
|
||||
return "restored the project workspace path";
|
||||
case "agent_secret": {
|
||||
const secret = await api.post<{ id: string }>(`/api/companies/${lab.companyId}/secrets`, {
|
||||
name: `Chaos ${resources.credentialKey} ${task.taskId}-${task.nonce} repaired`, key: `${resources.credentialKey}_${task.taskId}_${task.nonce}_repaired`,
|
||||
value: process.env[resources.credentialKey], description: "Restart chaos config fault repair" });
|
||||
await api.patch(`/api/agents/${resources.agentId}`, { adapterConfig: { env: { [resources.credentialKey]: secretRef(secret.id) } } });
|
||||
return "created a new secret and pointed the agent at it";
|
||||
}
|
||||
case "model_id":
|
||||
await api.patch(`/api/agents/${resources.agentId}`, { adapterConfig: { model: resources.agentBody.adapterConfig?.model } });
|
||||
return `restored model ${String(resources.agentBody.adapterConfig?.model)}`;
|
||||
case "ai_default": {
|
||||
const created = await api.post<{ connectionId: string; grantId: string }>(`/api/companies/${lab.companyId}/ai-connections`, {
|
||||
provider: "openrouter", method: "api_key", name: `Chaos OpenRouter ${task.nonce}`, ownership: "personal",
|
||||
// The account must be installed for the agent; without it selection refuses "not permitted for this agent".
|
||||
apiKey: process.env.OPENROUTER_API_KEY, agentIds: [resources.agentId], allAgents: false }, 60_000);
|
||||
resources.connectionId = created.connectionId;
|
||||
return "connected a personal OpenRouter account for the agent (it becomes the default)";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function snapshot(api: ChaosApi, issueId: string, agentId: string, errors: Map<string, string | null>): Promise<ConfigSnapshot> {
|
||||
const at = Date.now();
|
||||
try {
|
||||
const [issue, runs, execution, interactions, agent, comments] = await Promise.all([
|
||||
api.get<Json>(`/api/issues/${issueId}`, 5_000),
|
||||
api.get<Json[]>(`/api/issues/${issueId}/runs`, 5_000),
|
||||
api.get<Json | null>(`/api/issues/${issueId}/execution`, 5_000).catch(() => null),
|
||||
api.get<Json[]>(`/api/issues/${issueId}/interactions`, 5_000).catch(() => []),
|
||||
api.get<Json>(`/api/agents/${agentId}`, 5_000).catch(() => null),
|
||||
api.get<Json[]>(`/api/issues/${issueId}/comments`, 5_000).catch(() => []),
|
||||
]);
|
||||
const runSamples: ConfigSnapshot["runs"] = [];
|
||||
for (const run of runs ?? []) {
|
||||
const id = String(run.runId ?? run.id);
|
||||
const status = String(run.status);
|
||||
let error: string | null = typeof run.error === "string" ? run.error : null;
|
||||
if (!error && !["queued", "running"].includes(status)) {
|
||||
if (!errors.has(id)) errors.set(id, await api.get<Json>(`/api/heartbeat-runs/${id}`, 5_000).then(row => row?.error ?? null).catch(() => null));
|
||||
error = errors.get(id) ?? null;
|
||||
}
|
||||
runSamples.push({ id, status, createdAt: time(run.createdAt), startedAt: time(run.startedAt), errorCode: run.errorCode ?? null, error });
|
||||
}
|
||||
const interruption = issue.interruption as Json | null | undefined;
|
||||
const projection = (execution?.execution ?? null) as Json | null;
|
||||
return {
|
||||
at, reachable: true, issueStatus: String(issue.status), runs: runSamples,
|
||||
interruption: interruption ? { state: interruption.state ?? null, canRetry: interruption.canRetry === true, userAction: interruption.userAction ?? null,
|
||||
workspaceFailure: interruption.workspaceFailure?.message ?? null, observedRunId: interruption.observedRunId ?? null } : null,
|
||||
execution: projection ? { label: projection.label ?? null, cause: projection.cause ?? null, nextAction: projection.nextAction ?? null } : null,
|
||||
cards: (Array.isArray(interactions) ? interactions : []).map(card => ({ id: String(card.id), kind: String(card.kind), status: String(card.status),
|
||||
serviceSlug: card.payload?.serviceSlug ?? null, agentSetup: card.payload?.agentSetup === true })),
|
||||
agent: agent ? { lifecycleState: agent.lifecycleState ?? null, lifecycleError: agent.lifecycleError ?? null,
|
||||
// The agent response redacts lifecycleOperation; parked setup also says so in its message.
|
||||
retryRequired: agent.lifecycleOperation?.retryRequired === true
|
||||
|| /automatic verification is paused|choose retry now/i.test(String(agent.lifecycleError ?? "")) } : null,
|
||||
notices: (Array.isArray(comments) ? comments : []).filter(comment => !comment.authorAgentId).map(comment => String(comment.body ?? "")).slice(-5),
|
||||
};
|
||||
} catch {
|
||||
return { at, reachable: false, issueStatus: null, runs: [], interruption: null, execution: null, cards: [], agent: null, notices: [] };
|
||||
}
|
||||
}
|
||||
|
||||
function time(value: unknown): number | null {
|
||||
const at = typeof value === "string" ? Date.parse(value) : Number.NaN;
|
||||
return Number.isFinite(at) ? at : null;
|
||||
}
|
||||
|
||||
/** Use what the task offers: complete a pending connection card, then Retry, then the agent's setup Retry. */
|
||||
async function offeredAction(api: ChaosApi, record: ConfigTaskRecord, resources: TaskResources, current: ConfigSnapshot): Promise<string> {
|
||||
const card = current.cards.find(candidate => candidate.status === "pending" && candidate.kind === "connection_intent");
|
||||
if (card && resources.connectionId) {
|
||||
await api.post(`/api/connection-intents/${card.id}/complete`, { connectionId: resources.connectionId });
|
||||
return `completed connection card ${card.id}`;
|
||||
}
|
||||
if (current.interruption?.canRetry && current.interruption.observedRunId) {
|
||||
const response = await api.post<Json>(`/api/issues/${record.issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId: current.interruption.observedRunId });
|
||||
return `clicked Retry (accepted=${String(response?.accepted)})`;
|
||||
}
|
||||
if (current.agent && current.agent.lifecycleState !== "ready") {
|
||||
await api.post(`/api/agents/${resources.agentId}/lifecycle/retry`, {});
|
||||
return `agent setup Retry (lifecycle ${current.agent.lifecycleState})`;
|
||||
}
|
||||
return "no offered action; waited";
|
||||
}
|
||||
|
||||
/** Set by SIGINT/SIGTERM: tasks stop at their next step and the wave still reaches cleanup. */
|
||||
let interrupted = false;
|
||||
function checkInterrupted() {
|
||||
if (interrupted) throw new Error("interrupted");
|
||||
}
|
||||
|
||||
interface TaskInput {
|
||||
api: ChaosApi; lab: ChaosLab; task: ConfigTaskPlan; supervisor: ChaosSupervisor; timeline: Timeline;
|
||||
options: ConfigWaveOptions; gate: RestartGate; onRestart: (faultId: string) => void;
|
||||
}
|
||||
|
||||
async function runTask(input: TaskInput): Promise<ConfigTaskRecord> {
|
||||
const { api, lab, task, timeline, gate } = input;
|
||||
const record: ConfigTaskRecord = { plan: task, issueId: null, agentId: null, projectId: null, launchedAt: null, refusedOnWrite: null,
|
||||
harnessError: null, brokenSnapshots: [], parkedAt: null, parkReason: null, restart: null, afterRestartSnapshots: [], afterRestartParked: null,
|
||||
repair: null, doneAt: null, finalStatus: null, brief: null };
|
||||
const errors = new Map<string, string | null>();
|
||||
let resources: TaskResources;
|
||||
try {
|
||||
checkInterrupted();
|
||||
resources = await gate.write(() => createTaskResources(api, lab, task, timeline));
|
||||
record.agentId = resources.agentId;
|
||||
record.projectId = resources.projectId;
|
||||
// A working agent first, so the only broken thing is the setting under test (CF2's agent cannot verify by design).
|
||||
if (task.reference !== "ai_default") {
|
||||
await waitForAgentsReady(api, { [task.taskId]: { id: resources.agentId } }, timeline, undefined, 1_000, () => interrupted);
|
||||
}
|
||||
checkInterrupted();
|
||||
record.refusedOnWrite = await gate.write(() => breakReference(api, lab, task, resources));
|
||||
timeline.add("cf_broken", { taskId: task.taskId, fault: task.fault, reference: task.reference, refusedOnWrite: record.refusedOnWrite });
|
||||
if (record.refusedOnWrite) return record;
|
||||
const a1 = workloadDefinitions.A1;
|
||||
const issue = await gate.write(() => api.post<Json>(`/api/companies/${lab.companyId}/issues`, {
|
||||
title: `Chaos ${task.fault} ${task.reference} ${task.nonce}`, description: a1.description(task.nonce), status: "todo", priority: "medium",
|
||||
assigneeAgentId: resources.agentId, projectId: resources.projectId }));
|
||||
record.issueId = String(issue.id);
|
||||
record.launchedAt = Date.now();
|
||||
timeline.add("cf_launched", { taskId: task.taskId, issueId: issue.id, identifier: issue.identifier ?? null });
|
||||
} catch (error) {
|
||||
record.harnessError = `setup: ${error instanceof Error ? error.message : String(error)}`.slice(0, 600);
|
||||
return record;
|
||||
}
|
||||
try {
|
||||
await observeTask(input, record, resources, errors);
|
||||
} catch (error) {
|
||||
record.harnessError = (error instanceof Error ? error.message : String(error)).slice(0, 600);
|
||||
}
|
||||
return record;
|
||||
}
|
||||
|
||||
async function observeTask(input: TaskInput, record: ConfigTaskRecord, resources: TaskResources, errors: Map<string, string | null>): Promise<void> {
|
||||
const { api, lab, task, supervisor, timeline, options, gate } = input;
|
||||
const issueId = record.issueId!;
|
||||
|
||||
// Broken phase: until done, parked (stable across two polls) or the bound.
|
||||
let parkedCandidate: number | null = null;
|
||||
while (Date.now() - record.launchedAt! < options.parkBoundMs) {
|
||||
await delay(POLL_MS);
|
||||
checkInterrupted();
|
||||
if (!supervisor.serverReady) continue;
|
||||
const current = await snapshot(api, issueId, resources.agentId, errors);
|
||||
record.brokenSnapshots.push(current);
|
||||
if (!current.reachable) continue;
|
||||
const state = parkState(current, current.at, SETTLE_MS);
|
||||
if (state.kind === "done") break;
|
||||
if (state.kind === "parked") {
|
||||
parkedCandidate ??= current.at;
|
||||
if (current.at - parkedCandidate >= 2 * POLL_MS) { record.parkedAt = parkedCandidate; record.parkReason = state.reason; break; }
|
||||
} else parkedCandidate = null;
|
||||
}
|
||||
timeline.add("cf_broken_phase_end", { taskId: task.taskId, parkedAt: record.parkedAt, parkReason: record.parkReason,
|
||||
lastStatus: record.brokenSnapshots.at(-1)?.issueStatus ?? null });
|
||||
|
||||
const completedWhileBroken = record.brokenSnapshots.at(-1)?.issueStatus === "done";
|
||||
if (!completedWhileBroken) {
|
||||
// Restart while parked (or at the bound). The gate holds every task's writes while the server is down.
|
||||
checkInterrupted();
|
||||
await gate.restart(async () => {
|
||||
const restart: NonNullable<ConfigTaskRecord["restart"]> = { kind: task.restart, faultId: null, firedAt: Date.now(), readyAt: null, error: null };
|
||||
record.restart = restart;
|
||||
try {
|
||||
const receipt = await supervisor.fault(task.restart, 0, task.taskId);
|
||||
restart.faultId = receipt.faultId;
|
||||
restart.readyAt = receipt.readyAt;
|
||||
restart.error = receipt.error;
|
||||
input.onRestart(receipt.faultId);
|
||||
} catch (error) {
|
||||
restart.error = error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
});
|
||||
// After the restart: the task must rest again, still naming the setting.
|
||||
const repark = Date.now();
|
||||
let lastParked: number | null = null;
|
||||
while (Date.now() - repark < RESTART_REPARK_MS) {
|
||||
await delay(POLL_MS);
|
||||
checkInterrupted();
|
||||
if (!supervisor.serverReady) continue;
|
||||
const current = await snapshot(api, issueId, resources.agentId, errors);
|
||||
if (!current.reachable) continue;
|
||||
record.afterRestartSnapshots.push(current);
|
||||
const state = parkState(current, current.at, SETTLE_MS);
|
||||
if (state.kind === "done") { record.afterRestartParked = true; break; }
|
||||
if (state.kind === "parked") {
|
||||
lastParked ??= current.at;
|
||||
if (current.at - lastParked >= 2 * POLL_MS) { record.afterRestartParked = true; break; }
|
||||
} else lastParked = null;
|
||||
}
|
||||
record.afterRestartParked ??= false;
|
||||
timeline.add("cf_after_restart", { taskId: task.taskId, restart: record.restart, parkedAgain: record.afterRestartParked });
|
||||
|
||||
// Repair, then the offered action.
|
||||
checkInterrupted();
|
||||
const repairAt = Date.now();
|
||||
try {
|
||||
const { detail, action } = await gate.write(async () => {
|
||||
const detail = await repairReference(api, lab, task, resources);
|
||||
await delay(3_000);
|
||||
const current = await snapshot(api, issueId, resources.agentId, errors);
|
||||
const action = current.issueStatus === "done" ? "none (already done)" : await offeredAction(api, record, resources, current);
|
||||
return { detail, action };
|
||||
});
|
||||
record.repair = { at: repairAt, action, detail, error: null };
|
||||
} catch (error) {
|
||||
record.repair = { at: repairAt, action: "repair", detail: "", error: error instanceof Error ? error.message.slice(0, 400) : String(error) };
|
||||
}
|
||||
timeline.add("cf_repaired", { taskId: task.taskId, repair: record.repair });
|
||||
}
|
||||
|
||||
// Completion after repair, bounded.
|
||||
const deadline = Date.now() + options.repairBudgetMs;
|
||||
while (Date.now() < deadline) {
|
||||
checkInterrupted();
|
||||
const current = await snapshot(api, issueId, resources.agentId, errors);
|
||||
if (current.reachable) {
|
||||
record.finalStatus = current.issueStatus;
|
||||
record.afterRepair = current;
|
||||
if (current.issueStatus === "done") { record.doneAt = Date.now(); break; }
|
||||
}
|
||||
await delay(POLL_MS * 2);
|
||||
}
|
||||
const brief = await api.get<Json>(`/api/issues/${issueId}/documents/brief`).then(row => typeof row?.body === "string" ? row.body : null).catch(() => null);
|
||||
const oracle = workloadDefinitions.A1.grade(task.nonce, { issueStatus: record.finalStatus, documents: { brief }, agentComments: [], files: {}, commands: {}, workspacePath: null });
|
||||
record.brief = { ok: oracle.verdict === "pass", detail: oracle.checks.filter(check => !check.ok).map(check => `${check.name}${check.detail ? ` (${check.detail})` : ""}`).join("; ") || "brief ok" };
|
||||
}
|
||||
|
||||
export function configSummaryMarkdown(waveId: string, gitSha: string, rows: Array<{ record: ConfigTaskRecord; grade: ConfigGrade }>): string {
|
||||
const lines = [`# Config fault wave ${waveId}`, "", `Commit ${gitSha}`, "",
|
||||
"| Task | Fault | Setting | Adapter | Restart | Outcome | Pass | Signature |", "| --- | --- | --- | --- | --- | --- | --- | --- |"];
|
||||
for (const { record, grade } of rows) {
|
||||
lines.push(`| ${record.plan.taskId} | ${record.plan.fault} | ${record.plan.reference} | ${record.plan.adapter} | ${record.plan.restart} | ${grade.outcome} | ${grade.pass ? "pass" : "FAIL"} | ${grade.signature ?? ""} |`);
|
||||
}
|
||||
lines.push("", "## Checks", "");
|
||||
for (const { record, grade } of rows) {
|
||||
lines.push(`### ${record.plan.taskId} ${record.plan.fault}/${record.plan.reference}: ${CONFIG_REFERENCES[record.plan.reference].label}`, "");
|
||||
for (const check of grade.checks) lines.push(`- ${check.verdict === "pass" ? "pass" : check.verdict === "n/a" ? "n/a " : "FAIL"} ${check.name}: ${check.detail}`);
|
||||
lines.push("");
|
||||
}
|
||||
lines.push("## Not broken by CF3", "", ...Object.entries(CONFIG_REFERENCES_NOT_BROKEN).map(([name, why]) => `- ${name}: ${why}`), "");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function main(): Promise<number> {
|
||||
const { options, keep } = parseConfigWaveArgs(process.argv.slice(2));
|
||||
const tasks = planConfigWave(options);
|
||||
validateConfigPlan(tasks);
|
||||
const waveId = `${waveIdFor(options.seed)}-cf`;
|
||||
const waveDir = path.join(resultsRoot, waveId);
|
||||
await mkdir(waveDir, { recursive: true, mode: 0o700 });
|
||||
await writeFile(path.join(waveDir, "config-plan.json"), `${JSON.stringify({ waveId, options, tasks }, null, 2)}\n`, { mode: 0o600 });
|
||||
const timeline = new Timeline(path.join(waveDir, "timeline.jsonl"));
|
||||
const gitSha = spawnSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8" }).stdout.trim();
|
||||
timeline.add("cf_wave_planned", { waveId, gitSha, tasks: tasks.map(task => `${task.taskId}:${task.adapter}/${task.fault}/${task.reference}/${task.restart}`) });
|
||||
preflight(tasks);
|
||||
// Every restart re-runs the server from this checkout: a build or checkout change mid-wave would mix builds.
|
||||
const guardedOutputs = guardedRunnerOutputs(repositoryRoot);
|
||||
const startFingerprint = worktreeFingerprint(repositoryRoot, guardedOutputs);
|
||||
let worktreeChangedAt: string | null = null;
|
||||
const checkWorktree = (at: string) => {
|
||||
if (worktreeChangedAt || worktreeFingerprint(repositoryRoot, guardedOutputs) === startFingerprint) return;
|
||||
worktreeChangedAt = at;
|
||||
timeline.add("worktree_changed_mid_wave", { at });
|
||||
};
|
||||
const onSignal = (signal: NodeJS.Signals) => { if (!interrupted) { interrupted = true; timeline.add("harness_interrupted", { signal }); } };
|
||||
process.on("SIGINT", onSignal);
|
||||
process.on("SIGTERM", onSignal);
|
||||
const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline });
|
||||
let stopFailed = false;
|
||||
let harnessError: unknown = null;
|
||||
const rows: Array<{ record: ConfigTaskRecord; grade: ConfigGrade }> = [];
|
||||
try {
|
||||
await supervisor.start({});
|
||||
const api = new ChaosApi(supervisor.baseUrl);
|
||||
const lab = await seedChaosLab({ api, env: "local", waitForReadyAgents: true, adapters: [...new Set(tasks.map(task => task.adapter))],
|
||||
concurrency: options.concurrency, budgetCents: 3_000, fixtureRoot: path.join(supervisor.paths.temporaryRoot, "fixtures"), waveId, timeline });
|
||||
const gate = new RestartGate();
|
||||
const pending = [...tasks];
|
||||
const records: ConfigTaskRecord[] = [];
|
||||
await Promise.all(Array.from({ length: options.concurrency }, async () => {
|
||||
while (pending.length && !interrupted) {
|
||||
const task = pending.shift()!;
|
||||
records.push(await runTask({ api, lab, task, supervisor, timeline, options, gate, onRestart: checkWorktree }));
|
||||
}
|
||||
}));
|
||||
checkWorktree("wave_end");
|
||||
if (worktreeChangedAt) throw new Error(`The worktree changed during the wave (first seen after ${worktreeChangedAt}); later restarts may have run different code`);
|
||||
if (interrupted) throw new Error("Harness interrupted before the wave finished");
|
||||
for (const record of records.sort((left, right) => left.plan.index - right.plan.index)) {
|
||||
const grade = gradeConfigTask(record, { parkBoundMs: options.parkBoundMs });
|
||||
rows.push({ record, grade });
|
||||
await writeFile(path.join(waveDir, `${record.plan.taskId}.config-grade.json`),
|
||||
`${redactKnownSecrets(JSON.stringify({ record, grade }, null, 2), knownSecretValues())}\n`, { mode: 0o600 });
|
||||
timeline.add("cf_task_graded", { taskId: record.plan.taskId, pass: grade.pass, outcome: grade.outcome, signature: grade.signature });
|
||||
}
|
||||
} catch (error) {
|
||||
harnessError = error;
|
||||
timeline.add("harness_error", { error: error instanceof Error ? error.stack ?? error.message : String(error) });
|
||||
} finally {
|
||||
try { await supervisor.stop(); } catch (error) { stopFailed = true; harnessError ??= error; }
|
||||
await releaseInstance(supervisor.paths.temporaryRoot, { keep, stopFailed, owner: waveId })
|
||||
.catch(error => timeline.add("instance_cleanup_failed", { error: String(error) }));
|
||||
}
|
||||
const summary = configSummaryMarkdown(waveId, gitSha, rows);
|
||||
await writeFile(path.join(waveDir, "SUMMARY.md"), summary, { mode: 0o600 });
|
||||
console.info(`\n${summary}\nWave ${waveId}: results ${waveDir}`);
|
||||
if (harnessError) { console.error(harnessError); return 2; }
|
||||
return rows.every(row => row.grade.pass) && rows.length === tasks.length ? 0 : 1;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === path.resolve(import.meta.filename)) {
|
||||
process.exit(await main().catch(error => { console.error(error); return 2; }));
|
||||
}
|
||||
Reference in new issue
Block a user