From 051834bdc866960787128818d4dc69ee44b571f9 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 7 Nov 2026 23:41:15 -0600 Subject: [PATCH] feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - Random restart-chaos waves found these 20 ms–1 s windows only by luck > - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable write boundaries > - Nothing yet runs a task, kills the server at a chosen point, restarts it, and checks the plan's invariants > - This pull request adds that on-demand suite, as a local command and as a manual CI workflow (not a required check, per Decision 2) > - The benefit is a repeatable work list: which boundary breaks which invariant, for Phases 2–4 ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only reconciler used for I1–I7). Stacked on #19938 (call sites), which is stacked on #19937 (helper). **Problem** The named crash points cannot be exercised end to end. No command arms one point, runs real agent tasks across it, restarts, and grades the result. **Expected behavior** `node scripts/crash-points.mjs --points --kinds --adapters ` runs one wave per selected point and reports each point as pass, fail (with failing R rules and violated invariants), not reached, or harness error. A `workflow_dispatch` workflow runs the same thing on demand. ## What Changed - **Runtime arming.** `PAPERCLIP_CRASH_POINT_CONTROL=`, in the shared helper and the runner mirror. Each hit reads ` [:]`, and a new token resets the count. It is test mode only: `NODE_ENV=production` ignores it, and the disarmed path is still one comparison. - **`tests/restart-chaos/main.ts --crash-point :,:,...`** (config, types, ledger): - one booted server and no Vite UI; - per point: take an invariant baseline, launch the point's batch, arm the point, then on rest disarm it and check invariants as violations that are new since the baseline; - every task is graded R1–R6 and T; - `crash-point.json` (v2) holds one entry per point. - **`tests/restart-chaos/supervisor.ts`:** - every spawn gets the control file; - a SIGKILL whose receipt names the child's pid is restarted like K1: the control file is emptied first, the owned tree is killed, and a replacement starts. A failed tree cleanup fails the fault; - the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a `.paperclip/.env` above the checkout otherwise suppressed all scheduling); - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`. - **`tests/restart-chaos/invariants.ts`:** - the Phase 1 report-only pass with `record: false`: two real-clock passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7; - spool kinds are read from the wave's instance; - an unfinished pass is a harness error; - each violation carries its wake and run state as evidence. - **`tests/restart-chaos/crash-points.ts` and `scripts/crash-points.mjs`:** - select points by glob, kind and environment, and run them in one wave; - a per-point outcome: pass, fail, not_reached or harness_error; - `SUMMARY.md` with a violation evidence column. - **`.github/workflows/crash-points.yml`**, manual only, under the paid-workflow contract: - a credential-free `authorize` job; - a `suite` job in `runner-e2e-paid` whose first step reauthorizes; - a frozen install with `--ignore-scripts` and no shared cache; - only the selected adapters' keys; - the workflow is added to both security-test allowlists. - **Docs:** "Crash-point suite" in `tests/restart-chaos/README.md`, and "Built (Phase 5)" plus the first work list in the plan. ## Verification - Focused tests: - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`; - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 tests, including runtime arming in a real child process; - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72 tests; - `tests/runner-e2e/workflow-security.test.ts`: 19 tests. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Loop line: question "how long is one crash-point loop?" | loop: `node scripts/crash-points.mjs --points ` (codex A1) | elapsed 2.7 min end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s, rest about 2 min | result: one point alone takes about 3 min. Before this change it took about 15 min (one cold boot per point, Vite UI, host at load 77). - Full local run, 14 points, codex A1, 11.8 min in all: - every point was reached (the server killed itself at each one, and the replacement came back); - 11 pass; - `workspace.finalize.after_start_before_copyback` and `workspace.finalize.after_copyback_before_done` fail I2 and R6: a `workspace_finalize` operation stays running after the run succeeded; - `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its evidence shows the wake failed with "…retrying once", and the retry delivered the work, so this is the known refusal-regex false positive. - The 6 Daytona-only points were not run. ## Risks - Test harness and an on-demand workflow only. No product code changes. - `.github/**` needs a CODEOWNER review. The workflow uses provider keys, so it refuses non-default branches and actors that are not on the allowlist, like `runner-full-stack-e2e.yml`. - `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only crash-point waves. A chaos instance is never a worktree dev instance, so scheduling must not be suppressed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .github/workflows/crash-points.yml | 203 +++++++++++++++++ ...07-durable-ownership-and-reconciliation.md | 9 + packages/paperclip-runner/src/crash-point.ts | 50 +++-- packages/shared/src/crash-points.test.ts | 35 +++ packages/shared/src/crash-points.ts | 53 ++++- .../release-verify-workflow.test.mjs | 2 + scripts/crash-points.mjs | 17 ++ tests/restart-chaos/README.md | 90 ++++++++ tests/restart-chaos/config.ts | 16 +- tests/restart-chaos/crash-points.test.ts | 107 +++++++++ tests/restart-chaos/crash-points.ts | 211 ++++++++++++++++++ tests/restart-chaos/invariants.ts | 132 +++++++++++ tests/restart-chaos/ledger.ts | 3 + tests/restart-chaos/main.ts | 116 +++++++++- tests/restart-chaos/observer.ts | 40 +++- tests/restart-chaos/supervisor.ts | 117 +++++++++- tests/restart-chaos/types.ts | 9 + tests/runner-e2e/workflow-security.test.ts | 3 + 18 files changed, 1187 insertions(+), 26 deletions(-) create mode 100644 .github/workflows/crash-points.yml create mode 100644 scripts/crash-points.mjs create mode 100644 tests/restart-chaos/crash-points.test.ts create mode 100644 tests/restart-chaos/crash-points.ts create mode 100644 tests/restart-chaos/invariants.ts diff --git a/.github/workflows/crash-points.yml b/.github/workflows/crash-points.yml new file mode 100644 index 00000000000..988834749c0 --- /dev/null +++ b/.github/workflows/crash-points.yml @@ -0,0 +1,203 @@ +name: Crash-point suite + +# On-demand only (durable ownership plan, Decision 2): never a required PR +# check. Runs tests/restart-chaos/crash-points.ts on the default branch, which +# kills an isolated control plane at each selected named crash point, restarts +# it and asserts plan invariants I1–I8 plus restart-chaos graders R1–R6. +# Provider keys come from the runner-e2e-paid environment, so the code under +# test must be CODEOWNER-reviewed: the default branch only, allowlisted actors. + +on: + workflow_dispatch: + inputs: + points: + description: "Comma-separated crash-point globs (blank = every point reachable in env)" + required: false + default: "" + type: string + kinds: + description: "Comma-separated ownership kind ids, e.g. native_executor_claim (blank = all)" + required: false + default: "" + type: string + adapters: + description: "Comma-separated adapters: codex, claude, opencode" + required: false + default: codex + type: string + workloads: + description: "Comma-separated restart-chaos workloads: A1, B1, B5, C1" + required: false + default: A1 + type: string + nth: + description: "Fire at the nth hit of each point" + required: false + default: "1" + type: string + +concurrency: + group: crash-points-${{ github.ref }} + cancel-in-progress: false + +jobs: + # No secrets here: refuse the wrong branch, actor or input before the paid job starts. + authorize: + name: Authorize crash-point run + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: read + steps: + - name: Require default branch and allowlisted numeric actor IDs + env: + GH_TOKEN: ${{ github.token }} + REF: ${{ github.ref }} + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + ACTOR: ${{ github.actor }} + ACTOR_ID: ${{ github.actor_id }} + TRIGGERING_ACTOR: ${{ github.triggering_actor }} + ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} + run: | + set -euo pipefail + if [ "$REF" != "refs/heads/$DEFAULT_BRANCH" ]; then + echo "The crash-point suite uses provider keys and may run only from the default branch." >&2 + exit 1 + fi + jq -e 'type == "array" and length > 0 and all(.[]; type == "number" and . > 0 and floor == .)' <<< "${ALLOWED_ACTOR_IDS:-}" >/dev/null + for attempt in 1 2 3; do + if triggering_actor_id="$(timeout 30s gh api "users/$TRIGGERING_ACTOR" --jq .id)"; then break; fi + if [ "$attempt" = 3 ]; then exit 1; fi + sleep "$((attempt * 2))" + done + if [ "$triggering_actor_id" != "$ACTOR_ID" ] && [ "$TRIGGERING_ACTOR" = "$ACTOR" ]; then + echo "GitHub actor identity contexts disagree; refusing the paid run." >&2 + exit 1 + fi + for candidate in "$triggering_actor_id" "$ACTOR_ID"; do + jq -e --argjson candidate "$candidate" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null + done + + - name: Validate inputs + env: + POINTS: ${{ inputs.points }} + KINDS: ${{ inputs.kinds }} + ADAPTERS: ${{ inputs.adapters }} + WORKLOADS: ${{ inputs.workloads }} + NTH: ${{ inputs.nth }} + run: | + set -euo pipefail + [[ "$POINTS" =~ ^[a-z0-9_.*?,]*$ ]] || { echo "points: lower-case names, '.', '_', '*', '?' and ','" >&2; exit 1; } + [[ "$KINDS" =~ ^[a-z0-9_,]*$ ]] || { echo "kinds: lower-case ids and ','" >&2; exit 1; } + [[ "$ADAPTERS" =~ ^(codex|claude|opencode)(,(codex|claude|opencode))*$ ]] || { echo "adapters: codex, claude, opencode" >&2; exit 1; } + [[ "$WORKLOADS" =~ ^(A1|B1|B5|C1)(,(A1|B1|B5|C1))*$ ]] || { echo "workloads: A1, B1, B5, C1" >&2; exit 1; } + [[ "$NTH" =~ ^[1-9][0-9]{0,2}$ ]] || { echo "nth: 1-999" >&2; exit 1; } + + suite: + name: Crash points (${{ inputs.points || 'all' }}) + needs: authorize + runs-on: ubuntu-latest + timeout-minutes: 300 + permissions: + contents: read + environment: + name: runner-e2e-paid + steps: + - name: Reauthorize paid execution before provider access + env: + GH_TOKEN: ${{ github.token }} + REF: ${{ github.ref }} + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + ACTOR: ${{ github.actor }} + ACTOR_ID: ${{ github.actor_id }} + TRIGGERING_ACTOR: ${{ github.triggering_actor }} + ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }} + run: | + set -euo pipefail + if [ "$REF" != "refs/heads/$DEFAULT_BRANCH" ]; then + echo "The crash-point suite uses provider keys and may run only from the default branch." >&2 + exit 1 + fi + jq -e 'type == "array" and length > 0 and all(.[]; type == "number" and . > 0 and floor == .)' <<< "${ALLOWED_ACTOR_IDS:-}" >/dev/null + for attempt in 1 2 3; do + if triggering_actor_id="$(timeout 30s gh api "users/$TRIGGERING_ACTOR" --jq .id)"; then break; fi + if [ "$attempt" = 3 ]; then exit 1; fi + sleep "$((attempt * 2))" + done + if [ "$triggering_actor_id" != "$ACTOR_ID" ] && [ "$TRIGGERING_ACTOR" = "$ACTOR" ]; then + echo "GitHub actor identity contexts disagree; refusing the paid run." >&2 + exit 1 + fi + for candidate in "$triggering_actor_id" "$ACTOR_ID"; do + jq -e --argjson candidate "$candidate" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null + done + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false + + - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6 + with: + version: 9.15.4 + + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 + with: + node-version: 24 + + # No shared dependency cache and no dependency lifecycle scripts in a + # secret-bearing job (tests/runner-e2e/SECURITY.md); the repo's own + # postinstall step runs explicitly. + - run: pnpm install --frozen-lockfile --ignore-scripts + - run: node scripts/link-plugin-dev-sdk.mjs + + - name: Build native runner + run: | + set -euo pipefail + pnpm --filter @paperclipai/paperclip-runner build:typescript + pnpm --filter @paperclipai/paperclip-runner build:runner-binaries + + - name: List selected crash points + env: + POINTS: ${{ inputs.points }} + KINDS: ${{ inputs.kinds }} + run: node scripts/crash-points.mjs --list ${POINTS:+--points "$POINTS"} ${KINDS:+--kinds "$KINDS"} + + - name: Run crash-point suite + env: + POINTS: ${{ inputs.points }} + KINDS: ${{ inputs.kinds }} + ADAPTERS: ${{ inputs.adapters }} + WORKLOADS: ${{ inputs.workloads }} + NTH: ${{ inputs.nth }} + OPENAI_API_KEY: ${{ contains(inputs.adapters, 'codex') && secrets.OPENAI_API_KEY || '' }} + ANTHROPIC_API_KEY: ${{ contains(inputs.adapters, 'claude') && secrets.ANTHROPIC_API_KEY || '' }} + OPENROUTER_API_KEY: ${{ contains(inputs.adapters, 'opencode') && secrets.OPENROUTER_API_KEY || '' }} + # The hosted runner is a disposable VM; B1 oracles run agent code without Seatbelt there. + RESTART_CHAOS_ALLOW_UNSANDBOXED_ORACLE: "1" + RESTART_CHAOS_RESULTS_DIR: ${{ runner.temp }}/crash-point-results + run: | + set -euo pipefail + node scripts/crash-points.mjs ${POINTS:+--points "$POINTS"} ${KINDS:+--kinds "$KINDS"} \ + --adapters "$ADAPTERS" --workloads "$WORKLOADS" --nth "$NTH" --seed "$((GITHUB_RUN_ID % 2147483647))" + + - name: Summarize + if: always() + run: | + shopt -s nullglob + for summary in "$RUNNER_TEMP"/crash-point-results/crash-points/*/SUMMARY.md; do cat "$summary" >> "$GITHUB_STEP_SUMMARY"; done + + # Summaries, grades and crash-point receipts only: server logs and + # evidence bundles can hold provider output and stay on the runner. + - name: Upload results + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: crash-points-${{ github.run_id }} + path: | + ${{ runner.temp }}/crash-point-results/crash-points/*/SUMMARY.md + ${{ runner.temp }}/crash-point-results/crash-points/*/summary.json + ${{ runner.temp }}/crash-point-results/crash-points/*/w-*/crash-point.json + ${{ runner.temp }}/crash-point-results/crash-points/*/w-*/t*.grade.json + ${{ runner.temp }}/crash-point-results/crash-points/*/w-*/timeline.jsonl + retention-days: 14 + if-no-files-found: warn diff --git a/doc/plans/2026-11-07-durable-ownership-and-reconciliation.md b/doc/plans/2026-11-07-durable-ownership-and-reconciliation.md index ce9e9d3dc93..4465aa2f154 100644 --- a/doc/plans/2026-11-07-durable-ownership-and-reconciliation.md +++ b/doc/plans/2026-11-07-durable-ownership-and-reconciliation.md @@ -223,6 +223,15 @@ The runner stamps `generation` and `turnId` on every event. Driver and session c - Keep the chaos harness as a soak test for timing interactions the enumeration misses. Its R1–R6 graders should map onto I1–I7. - The random waves found 20 ms–1 s windows by luck. Enumeration finds them on purpose, and it finds the windows we haven't hit yet. +**Built (Phase 5).** +- `crashPoint(name)` in `packages/shared/src/crash-points.ts`, with the named points in `crash-point-registry.ts`. `PAPERCLIP_CRASH_POINT=[:]` arms one point; `NODE_ENV=production` ignores it. Disarmed, a call costs about 4 ns. In-process tests use `armCrashPointForTest`. +- 20 points at the boundaries in the inventory's write-ahead table, from `sandbox.create.after_provider_before_lease` to `accounting.closure.after_commit_before_spool_unlink`. `server/src/__tests__/crash-point-call-sites.test.ts` requires a call site for each one. +- `node scripts/crash-points.mjs [--points ] [--kinds ] [--adapters …]` boots one instance and arms each point in turn at runtime (`PAPERCLIP_CRASH_POINT_CONTROL`). It asserts R1–R6 plus I1–I4 and I7 per point, from the Phase 1 reconciler. I5, I6 and I8 are reported as not measured until sections F and G land. The manual CI run is `.github/workflows/crash-points.yml`. +- First full local run (2026-11-08, 14 local points, codex A1, 11.8 min): 11 pass. Work list: + - `workspace.finalize.after_start_before_copyback` and `…after_copyback_before_done` leave a `workspace_finalize` operation running after the run succeeded (I2 and R6, kind `native_workspace_finalize_attempt`). + - `wake.claim.after_run_running_before_wakeup_claimed` reports I4 on a wake whose retry delivered the work: the known refusal-regex false positive, which section E removes. +- How to add a point and how to read the results: `tests/restart-chaos/README.md`, "Crash-point suite". + ## Phases | Phase | Scope | Exit criteria | diff --git a/packages/paperclip-runner/src/crash-point.ts b/packages/paperclip-runner/src/crash-point.ts index bfa45677208..97117dcb507 100644 --- a/packages/paperclip-runner/src/crash-point.ts +++ b/packages/paperclip-runner/src/crash-point.ts @@ -8,40 +8,64 @@ * Disarmed (always under NODE_ENV=production) a call is one comparison. * runnerd never sees the variable: its environment is an allowlist. */ -import { appendFileSync, existsSync } from "node:fs"; +import { appendFileSync, existsSync, readFileSync } from "node:fs"; export type RunnerCrashPointName = | "runner.process.after_spawn_before_identity" | "runner.turn_start.after_runner_accept_before_controller_checkpoint" | "runner.turn_start.after_controller_journal_before_db_checkpoint"; -type Armed = { name: string; nth: number; hits: number; receiptPath: string | null; gatePath: string | null }; +type Armed = { + name: string; nth: number; hits: number; receiptPath: string | null; gatePath: string | null; + /** Runtime arming (PAPERCLIP_CRASH_POINT_CONTROL): read " [:]" at each hit. */ + controlPath: string | null; token: string | null; +}; + +function parseSpec(raw: string): { name: string; nth: number } | null { + const separator = raw.lastIndexOf(":"); + const name = separator === -1 ? raw : raw.slice(0, separator); + const nth = separator === -1 ? 1 : Number(raw.slice(separator + 1)); + return name && Number.isInteger(nth) && nth >= 1 ? { name, nth } : null; +} function armFromEnvironment(): Armed | null { const env = typeof process === "undefined" ? undefined : process.env; - const raw = env?.PAPERCLIP_CRASH_POINT?.trim(); - if (!env || !raw || env.NODE_ENV === "production") return null; - const separator = raw.lastIndexOf(":"); - const name = separator === -1 ? raw : raw.slice(0, separator); + if (!env || env.NODE_ENV === "production") return null; + const receiptPath = env.PAPERCLIP_CRASH_POINT_RECEIPT?.trim() || null; + const controlPath = env.PAPERCLIP_CRASH_POINT_CONTROL?.trim() || null; + if (controlPath) return { name: "", nth: 1, hits: 0, receiptPath, gatePath: null, controlPath, token: null }; + const raw = env.PAPERCLIP_CRASH_POINT?.trim(); + if (!raw) return null; + const spec = parseSpec(raw); // The shared helper validates the name; this copy only arms its own names. - if (!name.startsWith("runner.")) return null; - const nth = separator === -1 ? 1 : Number(raw.slice(separator + 1)); - if (!Number.isInteger(nth) || nth < 1) return null; - return { name, nth, hits: 0, receiptPath: env.PAPERCLIP_CRASH_POINT_RECEIPT?.trim() || null, - gatePath: env.PAPERCLIP_CRASH_POINT_GATE?.trim() || null }; + if (!spec || !spec.name.startsWith("runner.")) return null; + return { ...spec, hits: 0, receiptPath, gatePath: env.PAPERCLIP_CRASH_POINT_GATE?.trim() || null, controlPath: null, token: null }; } const armed: Armed | null = armFromEnvironment(); +function readControl(current: Armed): boolean { + let text: string; + try { text = readFileSync(current.controlPath!, "utf8"); } catch { return false; } + const [token, raw, ...rest] = text.trim().split(/\s+/); + const spec = token && raw && rest.length === 0 ? parseSpec(raw) : null; + if (!token || !spec) return false; + if (token !== current.token) Object.assign(current, { token, name: spec.name, nth: spec.nth, hits: 0 }); + return true; +} + export function crashPoint(name: RunnerCrashPointName): void { - if (armed === null || armed.name !== name) return; + if (armed === null) return; + if (armed.controlPath && !readControl(armed)) return; + if (armed.name !== name) return; if (armed.gatePath !== null) { if (!existsSync(armed.gatePath)) return; armed.gatePath = null; } armed.hits += 1; if (armed.hits !== armed.nth) return; - const receipt = { point: name, hit: armed.hits, pid: process.pid, at: new Date().toISOString() }; + const receipt = { point: name, hit: armed.hits, pid: process.pid, at: new Date().toISOString(), + ...(armed.token ? { token: armed.token } : {}) }; if (armed.receiptPath) { try { appendFileSync(armed.receiptPath, `${JSON.stringify(receipt)}\n`); } catch { /* the log line still records it */ } } diff --git a/packages/shared/src/crash-points.test.ts b/packages/shared/src/crash-points.test.ts index 9d745e28f5e..804a3acfd7a 100644 --- a/packages/shared/src/crash-points.test.ts +++ b/packages/shared/src/crash-points.test.ts @@ -11,6 +11,7 @@ import { armCrashPointForTest, armedCrashPoint, crashPoint, + parseCrashPointControl, parseCrashPointSpec, type CrashPointName, } from "./crash-points.js"; @@ -50,6 +51,15 @@ describe("parseCrashPointSpec", () => { }); }); +describe("parseCrashPointControl", () => { + it("reads a token and a spec, and treats anything else as disarmed", () => { + expect(parseCrashPointControl(`t1 ${first}:3\n`)).toEqual({ token: "t1", spec: { name: first, nth: 3 } }); + expect(parseCrashPointControl("")).toBeNull(); + expect(parseCrashPointControl(first)).toBeNull(); + expect(parseCrashPointControl("t1 sandbox.create.after_typo")).toBeNull(); + }); +}); + describe("crashPoint in a test process", () => { it("does nothing while disarmed", () => { expect(armedCrashPoint()).toBeNull(); @@ -150,6 +160,31 @@ describe("crashPoint armed by the environment", { timeout: 60_000 }, () => { } }); + it("arms at runtime from the control file and resets the count on a new token", () => { + const dir = mkdtempSync(path.join(os.tmpdir(), "crash-point-control-")); + try { + const control = path.join(dir, "control"); + const receipt = path.join(dir, "receipt.jsonl"); + const hit = `crashPoint(${JSON.stringify(first)});`; + const source = [ + `const { crashPoint } = await import(${JSON.stringify(path.join(import.meta.dirname, "crash-points.ts"))});`, + `const { writeFileSync } = await import("node:fs");`, + `${hit} process.stdout.write("absent\\n");`, + `writeFileSync(${JSON.stringify(control)}, "t1 ${first}:2\\n");`, + `${hit} process.stdout.write("t1-one\\n");`, + `writeFileSync(${JSON.stringify(control)}, "t2 ${first}:2\\n");`, + `${hit} process.stdout.write("t2-one\\n");`, + `${hit} process.stdout.write("t2-two\\n");`, + ].join("\n"); + const result = runChild({ PAPERCLIP_CRASH_POINT_CONTROL: control, PAPERCLIP_CRASH_POINT_RECEIPT: receipt }, source); + expect(result.signal).toBe("SIGKILL"); + expect(result.stdout).toBe("absent\nt1-one\nt2-one\n"); + expect(JSON.parse(readFileSync(receipt, "utf8").trim())).toMatchObject({ point: first, hit: 2, token: "t2" }); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + it("stays disarmed in production", () => { const result = runChild({ PAPERCLIP_CRASH_POINT: first, NODE_ENV: "production" }); expect(result.signal).toBeNull(); diff --git a/packages/shared/src/crash-points.ts b/packages/shared/src/crash-points.ts index 06d430e50f6..0ad6fdaeb6b 100644 --- a/packages/shared/src/crash-points.ts +++ b/packages/shared/src/crash-points.ts @@ -14,6 +14,12 @@ * PAPERCLIP_CRASH_POINT_GATE= optional; hits before this file exists are not counted, * so setup work (seeding, probes) cannot fire the point * + * Runtime arming (one booted server, many points), instead of PAPERCLIP_CRASH_POINT: + * + * PAPERCLIP_CRASH_POINT_CONTROL= each hit reads " [:]" from the file; + * no file or an empty one means disarmed. Hits count per + * token, so writing a new token re-arms with a fresh count. + * * The variable is read once, when this module is first imported. It is ignored * when `NODE_ENV=production`, and an unknown name throws at import so a typo * cannot silently arm nothing. Disarmed, `crashPoint` is a single comparison @@ -27,7 +33,7 @@ * This is test instrumentation, not Telemetry or Observability (AGENTS.md §5.7): * nothing is recorded or sent anywhere unless a test arms a point. */ -import { appendFileSync, existsSync } from "node:fs"; +import { appendFileSync, existsSync, readFileSync } from "node:fs"; import { CRASH_POINTS, isCrashPointName, type CrashPointName } from "./crash-point-registry.js"; export { CRASH_POINTS, CRASH_POINT_NAME_PATTERN, isCrashPointName } from "./crash-point-registry.js"; @@ -36,6 +42,7 @@ export type { CrashPointDefinition, CrashPointInvariant, CrashPointName } from " export const CRASH_POINT_ENV = "PAPERCLIP_CRASH_POINT"; export const CRASH_POINT_RECEIPT_ENV = "PAPERCLIP_CRASH_POINT_RECEIPT"; export const CRASH_POINT_GATE_ENV = "PAPERCLIP_CRASH_POINT_GATE"; +export const CRASH_POINT_CONTROL_ENV = "PAPERCLIP_CRASH_POINT_CONTROL"; /** Printed to stderr just before the kill; the suite greps server logs for it. */ export const CRASH_POINT_LOG_PREFIX = "[crash-point]"; @@ -46,6 +53,8 @@ export type CrashPointReceipt = { hit: number; pid: number; at: string; + /** Runtime arming only: the control file token the hit counted under. */ + token?: string; }; /** Thrown by a point armed with `armCrashPointForTest`. */ @@ -79,11 +88,33 @@ type Armed = { receiptPath: string | null; /** Hits count only once this file exists; null when ungated. */ gatePath: string | null; + /** Runtime arming: the control file read at each hit; null for a fixed spec. */ + controlPath?: string | null; + token?: string | null; }; +/** Parses a control file line, " [:]"; null when empty or malformed. */ +export function parseCrashPointControl(text: string): { token: string; spec: CrashPointSpec } | null { + const [token, raw, ...rest] = text.trim().split(/\s+/); + if (!token || !raw || rest.length > 0) return null; + try { + const spec = parseCrashPointSpec(raw); + return spec ? { token, spec } : null; + } catch { + return null; + } +} + function armFromEnvironment(): Armed | null { const env = typeof process === "undefined" ? undefined : process.env; if (!env || env.NODE_ENV === "production") return null; + const controlPath = env[CRASH_POINT_CONTROL_ENV]?.trim(); + if (controlPath) { + process.stderr.write(`${CRASH_POINT_LOG_PREFIX} runtime arming from ${controlPath} pid=${process.pid}\n`); + // The spec is a placeholder; each hit reads the real one from the control file. + return { spec: { name: "" as CrashPointName, nth: 1 }, hits: 0, mode: "kill", + receiptPath: env[CRASH_POINT_RECEIPT_ENV]?.trim() || null, gatePath: null, controlPath, token: null }; + } const spec = parseCrashPointSpec(env[CRASH_POINT_ENV]); if (!spec) return null; process.stderr.write(`${CRASH_POINT_LOG_PREFIX} armed ${spec.name}:${spec.nth} pid=${process.pid}\n`); @@ -103,10 +134,27 @@ let armed: Armed | null = armFromEnvironment(); */ export function crashPoint(name: CrashPointName): void { if (armed === null) return; + if (armed.controlPath) { + if (!readControl(armed)) return; + } if (armed.spec.name !== name) return; reach(armed); } +/** Runtime arming: loads the control file's spec; a new token resets the count. */ +function readControl(current: Armed): boolean { + let text: string; + try { text = readFileSync(current.controlPath!, "utf8"); } catch { return false; } + const control = parseCrashPointControl(text); + if (!control) return false; + if (control.token !== current.token) { + current.token = control.token; + current.spec = control.spec; + current.hits = 0; + } + return true; +} + function reach(current: Armed): void { if (current.gatePath !== null) { if (!existsSync(current.gatePath)) return; @@ -117,6 +165,7 @@ function reach(current: Armed): void { if (current.mode === "throw") throw new CrashPointReached(current.spec.name, current.hits); const receipt: CrashPointReceipt = { point: current.spec.name, hit: current.hits, pid: process.pid, at: new Date().toISOString(), + ...(current.token ? { token: current.token } : {}), }; if (current.receiptPath) { try { appendFileSync(current.receiptPath, `${JSON.stringify(receipt)}\n`); } catch { /* the log line below still records it */ } @@ -142,5 +191,5 @@ export function armCrashPointForTest(spec: string | CrashPointSpec): () => void /** The armed point and its hit count so far, or null. For tests and diagnostics. */ export function armedCrashPoint(): { name: CrashPointName; nth: number; hits: number } | null { - return armed ? { name: armed.spec.name, nth: armed.spec.nth, hits: armed.hits } : null; + return armed && armed.spec.name ? { name: armed.spec.name, nth: armed.spec.nth, hits: armed.hits } : null; } diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index f7beb9ceb02..677ff0bc812 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -655,6 +655,8 @@ test("Runner eval workflows pin actions and gate paid live execution", () => { "runner-protocol-live-evals.yml", "chat-catalog-testops.yml", "chat-catalog-testops-case.yml", + // On-demand crash-point suite (durable ownership plan, Phase 5). + "crash-points.yml", ]; const paidWorkflowNameSet = new Set(paidWorkflowNames); const providerSecretReference = diff --git a/scripts/crash-points.mjs b/scripts/crash-points.mjs new file mode 100644 index 00000000000..612355807de --- /dev/null +++ b/scripts/crash-points.mjs @@ -0,0 +1,17 @@ +#!/usr/bin/env node +// On-demand crash-point suite (durable ownership plan, Phase 5). +// Usage: node scripts/crash-points.mjs [--list] [--points ] [--kinds ] +// [--adapters codex,claude,opencode] [--workloads A1,B1,B5,C1] [--nth N] [--env local|daytona] +// See tests/restart-chaos/README.md, "Crash-point suite". Load provider keys +// into this process only: set -a; source ~/.secrets; set +a +import { spawnSync } from "node:child_process"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const result = spawnSync(process.execPath, [ + path.join(root, "cli/node_modules/tsx/dist/cli.mjs"), + path.join(root, "tests/restart-chaos/crash-points.ts"), + ...process.argv.slice(2), +], { cwd: root, stdio: "inherit" }); +process.exit(result.status ?? 2); diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 04e8f0065bf..5a60b99c571 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -206,3 +206,93 @@ re-arms the task's own triggered faults instead of using their offsets. - The wave fingerprints the checkout and `packages/paperclip-runner/dist` at start and after every fault. A build, install or branch switch during a wave would make later restarts run different code, so it fails the wave as a harness error (`worktree_changed_mid_wave`). The baseline is taken after the harness's own setup (preflight rebuilds provider entrypoints) and covers the tracked diff, untracked file contents, the runner `dist` and the runnerd binary. Build before the wave, never during it. - `faults.json` and `task-faults.json` (each task's launch time and the faults that crossed it) are written after every fault, so `--replay` of a task in an interrupted or wedged wave, or a storm task, re-fires the faults that crossed it at the same offsets. - A fault that fails to complete fails the wave as a harness error (exit 2), even when recovery brings the server back. + +## Crash-point suite (`scripts/crash-points.mjs`) + +Plan: [durable ownership, "Testing: enumerate crash points"](../../doc/plans/2026-11-07-durable-ownership-and-reconciliation.md). +Random waves find a 20 ms–1 s window between two durable writes by luck. The +crash-point suite kills the control plane at each named window on purpose. + +```sh +set -a; source ~/.secrets; set +a # provider keys, process env only +node scripts/crash-points.mjs --list # selected points, kinds, invariants +node scripts/crash-points.mjs --points 'runner.*,native.*' --adapters codex --workloads A1 +node scripts/crash-points.mjs --kinds wake_request_claimed --nth 2 +``` + +Options: `--points ` (`*`, `?`), `--kinds ` +(ids from `server/src/services/recovery/ownership-registry.ts`), `--adapters` +(default `codex`), `--workloads` (default `A1`), `--nth` (fire at the nth hit, +default 1), `--env local|daytona` (a point runs only in the environments its +registry entry lists), plus `--daytona-image`, `--daytona-cpu`, `--budget-cents`, +`--poll-ms`, `--lifecycle` and `--keep`, passed to each wave. + +The suite runs one wave, `main.ts --crash-point :,:,...`: + +1. One isolated control plane boots once, with `PAPERCLIP_CRASH_POINT_CONTROL` + set to an empty control file (nothing armed) and without the Vite UI. +2. For each point in turn: invariants are read as a baseline, the harness + arms the point by writing ` :` to the control file, and + the point's batch launches (one task per adapter × workload). Creating an + assigned issue already queues its wake, so the point is armed first. +3. At the nth hit the server appends a receipt and SIGKILLs itself. The + supervisor treats it as K1: it empties the control file, kills what is + left of the server's process tree, starts a replacement, and waits for ready. +4. When the batch rests, the point is disarmed. `invariants.ts` runs the + Phase 1 report-only reconciler against the wave's database + (`record: false`), and the point is charged only with violations that are + new since its baseline: + - two passes at the real clock for I1 (owned) and I2 (settled); + - one pass with the clock moved 15 minutes ahead for I3 (admission), I4 + (exactly-once intents) and I7 (visible next action). + Each violation carries its wake and run state as evidence. I5, I6 and I8 + are reported as not measured, with the reconciler's reasons; R3 and R1 + cover part of I5 and I8. +5. Every task is graded R1–R6 and T at the end, as in any wave. + +The first full local run (14 points, codex A1) took 11.8 minutes: one boot, +then about 40–60 seconds per point. One point alone takes about 3 minutes, +boot included. + +Limits: the server kills itself without warning, so there is no pre-fault +workspace snapshot and R2 checks workspace failures but not lost files +(`notMeasured` in `crash-point.json`). `--replay` of a crash-point task fires +the recorded K1 restart at its offset instead of re-arming the point; to +reproduce a point, rerun the suite for it. + +Results: `results/crash-points//SUMMARY.md` and `summary.json`, plus the +wave directory with `crash-point.json` (per point: receipt, task grades, +violations with evidence). Outcomes: `pass`, `fail` (a task failed R1–R6/T or an invariant was +violated, or the wave exited 1 for a wave-level reason such as a Daytona +sandbox left after cleanup), `not_reached` (the chosen workloads never crossed the point: pick a +workload or `--nth` that does) and `harness_error`. Exit code 0, 1, 3 and 2. + +**CI.** `.github/workflows/crash-points.yml` runs the same command on demand +(`workflow_dispatch`, inputs `points`, `kinds`, `adapters`, `workloads`, `nth`). +It is not a required PR check. It runs only from the default branch, for +allowlisted actors, in the `runner-e2e-paid` environment, and uploads summaries +and grades only (no server logs). + +### Add a crash point + +1. Add an entry to `packages/shared/src/crash-point-registry.ts`. Name it + `.._` after the gap it marks, for + example `sandbox.create.after_provider_before_lease`. Give a description of + what is durable and what is not at that instant, the environments that reach + it, the ownable kind ids it affects and the invariants at risk. +2. Call `crashPoint("")` at the gap: after the first durable write (or + outside effect) returns and before the next one starts. Server code imports + `crashPoint` from `@paperclipai/shared/crash-points`. Controller code in + `packages/paperclip-runner` uses `src/crash-point.ts` (add the name to its + union too). Do not put a point inside a transaction; the gap is between + commits. +3. `server/src/__tests__/crash-point-call-sites.test.ts` fails until every + registered point has a call site and every kind id exists. +4. Run the point: `node scripts/crash-points.mjs --points `. A + `not_reached` result means the workload never crosses it. + +Disarmed, a point costs one comparison (about 4 ns), and the arming variables +(`PAPERCLIP_CRASH_POINT`, `PAPERCLIP_CRASH_POINT_CONTROL`) are ignored under +`NODE_ENV=production`, so points may sit on the path before `turn.start`. +In-process tests use `armCrashPointForTest(":")`, which throws +`CrashPointReached` at the hit instead of killing the test runner. diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index 11a724d478d..0cdce2b67de 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -88,13 +88,14 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC "storm-minutes": { type: "string" }, lifecycle: { type: "string" }, "k4-hold-max-seconds": { type: "string" }, + "crash-point": { type: "string" }, }, }); const keep = values.keep === true; const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000); const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000); if (values.replay !== undefined) { - for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "storm-minutes", "lifecycle", "k4-hold-max-seconds"] as const) { + for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point"] as const) { if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`); } return { mode: "replay", ref: parseReplayRef(values.replay), keep, @@ -116,12 +117,22 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC // pool that keeps `--concurrency` tasks in flight until the storm ends. const stormMinutes = values["storm-minutes"] === undefined ? undefined : integer(values["storm-minutes"], "storm-minutes", 60, 1, 240); const lifecycle = values.lifecycle; + const crashPoint = values["crash-point"]?.trim(); + if (crashPoint !== undefined) { + if (!/^[a-z0-9_.]+(?::[1-9][0-9]*)?(?:,[a-z0-9_.]+(?::[1-9][0-9]*)?)*$/.test(crashPoint)) { + throw new Error("--crash-point expects [:][,[:]...]"); + } + for (const key of ["faults", "triggers", "storm-minutes"] as const) { + if (values[key] !== undefined) throw new Error(`--crash-point replaces scheduled faults; do not combine it with --${key}`); + } + } if (lifecycle !== undefined && lifecycle !== "per_turn" && lifecycle !== "warm" && lifecycle !== "default") throw new Error("--lifecycle must be per_turn, warm or default"); const options: WaveOptions = { env, adapters: list(values.adapters, ADAPTER_IDS, "adapters", ["codex", "claude"]), workloads: list(values.workloads, workloadChoices, "workloads", workloadChoices), - faults: list(values.faults, FAULT_KINDS, "faults", stormMinutes ? FAULT_KINDS : ["K1", "K2", "K4"]), + // A crash-point wave schedules no fault; K1 is the restart its self-kill gets. + faults: list(values.faults, FAULT_KINDS, "faults", stormMinutes ? FAULT_KINDS : crashPoint ? ["K1"] : ["K1", "K2", "K4"]), triggers: list(values.triggers, TRIGGER_IDS, "triggers", ["time"]), concurrency: integer(values.concurrency, "concurrency", 4, 1, 20), tasks: integer(values.tasks, "tasks", stormMinutes ? 200 : 8, 1, 200), @@ -135,6 +146,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC // Seeded K4 hold upper end; the controller-outage cap is 600 s (#19797). ...(values["k4-hold-max-seconds"] !== undefined ? { k4HoldMaxMs: integer(values["k4-hold-max-seconds"], "k4-hold-max-seconds", 300, 120, 600) * 1000 } : {}), + ...(crashPoint ? { crashPoint } : {}), }; if (env !== "daytona" && values["daytona-cpu"] !== undefined) throw new Error("--daytona-cpu applies only to --env daytona"); if (options.daytonaCpu !== undefined) { diff --git a/tests/restart-chaos/crash-points.test.ts b/tests/restart-chaos/crash-points.test.ts new file mode 100644 index 00000000000..deaf4291158 --- /dev/null +++ b/tests/restart-chaos/crash-points.test.ts @@ -0,0 +1,107 @@ +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import type { CrashPointDefinition, CrashPointName } from "../../packages/shared/src/crash-point-registry.js"; +import { classifyPoint, globToRegExp, parseSuiteArgs, renderSummary, selectPoints, type CrashPointEntry } from "./crash-points.js"; +import { parseCommand } from "./config.js"; +import { readCrashPointReceipt } from "./supervisor.js"; + +const definition = (kinds: string[], environments: Array<"local" | "daytona"> = ["local", "daytona"]): CrashPointDefinition => ({ + description: "test point description long enough", kinds, invariants: ["I2"], environments, +}); +const registry: Record = { + "runner.startup.after_spawn": definition(["native_startup_context"]), + "sandbox.create.after_provider_before_lease": definition(["sandbox_creation_intent"], ["daytona"]), + "wake.request.after_claim_before_run": definition(["wake_request_claimed"]), +}; +const point = "runner.startup.after_spawn" as CrashPointName; + +describe("selectPoints", () => { + it("filters by glob, kind and environment", () => { + expect(selectPoints(registry, { points: [], kinds: [], env: "local" })) + .toEqual(["runner.startup.after_spawn", "wake.request.after_claim_before_run"]); + expect(selectPoints(registry, { points: ["runner.*"], kinds: [], env: "local" })).toEqual(["runner.startup.after_spawn"]); + expect(selectPoints(registry, { points: [], kinds: ["sandbox_creation_intent"], env: "daytona" })) + .toEqual(["sandbox.create.after_provider_before_lease"]); + expect(selectPoints(registry, { points: [], kinds: ["sandbox_creation_intent"], env: "local" })).toEqual([]); + }); + + it("matches globs literally apart from * and ?", () => { + expect(globToRegExp("runner.*").test("runnerXstartup")).toBe(false); + expect(globToRegExp("*.after_?pawn").test("runner.startup.after_spawn")).toBe(true); + }); +}); + +describe("parseSuiteArgs", () => { + it("defaults to one local codex A1 task per point", () => { + expect(parseSuiteArgs([])).toMatchObject({ env: "local", adapters: ["codex"], workloads: ["A1"], nth: 1, list: false, passthrough: [] }); + }); + + it("passes harness options through and refuses unknown values", () => { + expect(parseSuiteArgs(["--", "--keep", "--budget-cents", "500"]).passthrough).toEqual(["--budget-cents", "500", "--keep"]); + expect(() => parseSuiteArgs(["--adapters", "gpt"])).toThrow(/Unknown adapter/); + expect(() => parseSuiteArgs(["--nth", "0"])).toThrow(/positive integer/); + }); +}); + +describe("classifyPoint", () => { + const entry = (overrides: Partial = {}): CrashPointEntry => ({ + point: `${point}:1`, reached: { name: point, hit: 1, pid: 42 }, armedAt: 0, restedAt: 120_000, + tasks: [{ taskId: "t01", adapter: "codex", workload: "A1", pass: true, signature: null, rules: { R1: "pass" } }], + invariants: { violations: [], byInvariant: {}, errors: [] }, + ...overrides, + }); + + it("passes a reached point with no failed task or new violation, timed from arming to rest", () => { + expect(classifyPoint(point, 0, entry())).toMatchObject({ outcome: "pass", durationMs: 120_000 }); + }); + + it("fails on a new invariant violation or a failed task, even when not reached", () => { + const violation = { rule: "I2_settled", kind: "native_executor_claim", reason: "owner_run_terminal", recordId: "r1" }; + expect(classifyPoint(point, 1, entry({ invariants: { violations: [violation], byInvariant: { I2: 1 }, errors: [] } }))) + .toMatchObject({ outcome: "fail", failedInvariants: { I2: 1 } }); + expect(classifyPoint(point, 1, entry({ reached: null, + tasks: [{ taskId: "t01", adapter: "codex", workload: "A1", pass: false, signature: "x", rules: { R1: "fail" } }] })).outcome).toBe("fail"); + }); + + it("separates unreached, unrun and unchecked points", () => { + expect(classifyPoint(point, 0, entry({ reached: null })).outcome).toBe("not_reached"); + expect(classifyPoint(point, 2, null)).toMatchObject({ outcome: "harness_error", error: "not run (wave exited 2)" }); + expect(classifyPoint(point, 0, entry({ invariants: null })).outcome).toBe("harness_error"); + expect(classifyPoint(point, 0, entry({ invariants: { violations: [], byInvariant: {}, errors: ["I3 pass did not finish"] } })).outcome) + .toBe("harness_error"); + }); + + it("renders a summary row per point", () => { + const result = { ...classifyPoint(point, 0, entry()), waveDir: null }; + expect(renderSummary("cp-x", parseSuiteArgs([]), [result])).toContain("| `runner.startup.after_spawn` | pass | — | — | 2.0 |"); + }); +}); + +describe("readCrashPointReceipt", () => { + it("returns the receipt line of the given pid only", () => { + const dir = mkdtempSync(path.join(os.tmpdir(), "crash-receipt-")); + try { + const file = path.join(dir, "receipts.jsonl"); + writeFileSync(file, `${JSON.stringify({ point, hit: 1, pid: 7 })}\n{"partial`); + expect(readCrashPointReceipt(file, 7)).toEqual({ name: point, hit: 1, pid: 7 }); + expect(readCrashPointReceipt(file, 8)).toBeNull(); + expect(readCrashPointReceipt(path.join(dir, "missing"), 7)).toBeNull(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); + +describe("main.ts --crash-point", () => { + it("plans a K1-only wave with the point and refuses scheduled-fault options", () => { + const parsed = parseCommand(["--crash-point", `${point}:2`, "--tasks", "1", "--seed", "1"]); + expect(parsed).toMatchObject({ mode: "wave", options: { crashPoint: `${point}:2`, faults: ["K1"] } }); + expect(() => parseCommand(["--crash-point", point, "--faults", "K2"])).toThrow(/replaces scheduled faults/); + expect(() => parseCommand(["--crash-point", "Bad Name"])).toThrow(/\[:\]/); + expect(parseCommand(["--crash-point", `${point}:1,${point}:2`, "--tasks", "2"])) + .toMatchObject({ options: { crashPoint: `${point}:1,${point}:2` } }); + expect(() => parseCommand(["--replay", "w-20261107T000000-s1/t01", "--crash-point", point])).toThrow(/--replay/); + }); +}); diff --git a/tests/restart-chaos/crash-points.ts b/tests/restart-chaos/crash-points.ts new file mode 100644 index 00000000000..b5481b89493 --- /dev/null +++ b/tests/restart-chaos/crash-points.ts @@ -0,0 +1,211 @@ +/** + * On-demand crash-point suite (durable ownership plan, Phase 5). + * + * pnpm crash-points --list + * pnpm crash-points --points 'runner.*' --adapters codex + * pnpm crash-points --kinds sandbox_creation_intent --env daytona --daytona-image + * + * It runs one restart-chaos wave (main.ts `--crash-point a,b,...`): one + * isolated control plane boots once, then each selected point in turn gets a + * batch of tasks (one per adapter × workload), is armed at runtime, SIGKILLs + * the server at its nth hit, and the server restarts like K1. When the batch + * rests, plan invariants I1–I8 from the Phase 1 report-only reconciler + * (invariants.ts) are checked as violations new since the point was armed. + * Every task is graded R1–R6 at the end. + * Results: tests/restart-chaos/results/crash-points//SUMMARY.md. + * + * Exit: 0 every reached point passed, 1 a point failed, 2 a harness error, + * 3 a selected point was never reached by the chosen workloads. + */ +import { spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, readdir, readFile, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { parseArgs } from "node:util"; +import { CRASH_POINTS, type CrashPointDefinition, type CrashPointName } from "../../packages/shared/src/crash-point-registry.js"; +import { knownSecretValues, redactKnownSecrets } from "./evidence.js"; +import { ADAPTER_IDS, WORKLOAD_IDS, type AdapterId, type WorkloadId } from "./types.js"; + +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); +const resultsRoot = path.resolve(process.env.RESTART_CHAOS_RESULTS_DIR ?? path.join(import.meta.dirname, "results")); + +export type PointOutcome = "pass" | "fail" | "not_reached" | "harness_error"; + +export interface PointResult { + point: CrashPointName; + outcome: PointOutcome; + exitCode: number | null; + waveDir: string | null; + reached: { name: string; hit: number; pid: number } | null; + failedInvariants: Record; + failedTasks: Array<{ taskId: string; adapter: string; workload: string; signature: string | null; rules: Record }>; + violations: Array<{ rule: string; kind: string; reason: string; recordId: string; evidence?: Record }>; + durationMs: number; + error?: string; +} + +export interface SuiteOptions { + points: string[]; + kinds: string[]; + adapters: AdapterId[]; + workloads: WorkloadId[]; + env: "local" | "daytona"; + nth: number; + seed: number; + list: boolean; + passthrough: string[]; +} + +export function globToRegExp(glob: string): RegExp { + const escaped = glob.trim().replace(/[.+^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*").replace(/\?/g, "."); + return new RegExp(`^${escaped}$`); +} + +/** Points matching any `--points` glob and any `--kinds` id, reachable in `env`. */ +export function selectPoints(registry: Record, options: Pick): CrashPointName[] { + const globs = options.points.length ? options.points.map(globToRegExp) : [/.*/]; + return (Object.keys(registry) as CrashPointName[]).filter(name => { + const definition = registry[name]!; + if (!globs.some(glob => glob.test(name))) return false; + if (options.kinds.length && !definition.kinds.some(kind => options.kinds.includes(kind))) return false; + return definition.environments.includes(options.env); + }); +} + +function csv(value: string | undefined): string[] { + return value ? value.split(",").map(part => part.trim()).filter(Boolean) : []; +} + +export function parseSuiteArgs(argv: readonly string[]): SuiteOptions { + const args = argv[0] === "--" ? argv.slice(1) : [...argv]; + const { values } = parseArgs({ args, strict: true, options: { + points: { type: "string" }, kinds: { type: "string" }, adapters: { type: "string" }, workloads: { type: "string" }, + env: { type: "string" }, nth: { type: "string" }, seed: { type: "string" }, list: { type: "boolean" }, + "daytona-image": { type: "string" }, "daytona-cpu": { type: "string" }, "budget-cents": { type: "string" }, + "poll-ms": { type: "string" }, keep: { type: "boolean" }, lifecycle: { type: "string" }, + } }); + const env = values.env ?? "local"; + if (env !== "local" && env !== "daytona") throw new Error("--env must be local or daytona"); + const adapters = csv(values.adapters ?? "codex") as AdapterId[]; + for (const adapter of adapters) if (!ADAPTER_IDS.includes(adapter)) throw new Error(`Unknown adapter "${adapter}"`); + const workloads = csv(values.workloads ?? "A1") as WorkloadId[]; + for (const workload of workloads) if (!WORKLOAD_IDS.includes(workload)) throw new Error(`Unknown workload "${workload}"`); + const nth = Number(values.nth ?? "1"); + if (!Number.isInteger(nth) || nth < 1) throw new Error("--nth must be a positive integer"); + const seed = Number(values.seed ?? "1"); + if (!Number.isInteger(seed) || seed < 0) throw new Error("--seed must be a non-negative integer"); + const passthrough: string[] = []; + for (const key of ["daytona-image", "daytona-cpu", "budget-cents", "poll-ms", "lifecycle"] as const) { + if (values[key] !== undefined) passthrough.push(`--${key}`, values[key]!); + } + if (values.keep) passthrough.push("--keep"); + return { points: csv(values.points), kinds: csv(values.kinds), adapters, workloads, env, nth, seed, list: values.list === true, passthrough }; +} + +/** One point's entry in crash-point.json (schema v2). */ +export type CrashPointEntry = { + point: string; + reached: { name: string; hit: number; pid: number } | null; + armedAt: number; + restedAt: number | null; + tasks: Array<{ taskId: string; adapter: string; workload: string; pass: boolean; signature: string | null; rules: Record }>; + invariants: { violations: Array<{ rule: string; kind: string; reason: string; recordId: string; evidence?: Record }>; byInvariant: Record; errors: string[] } | null; +}; + +/** Classifies one point from its entry; a point the wave never reached has none. */ +export function classifyPoint(point: CrashPointName, exitCode: number | null, entry: CrashPointEntry | null): Omit { + const base = { point, exitCode, reached: entry?.reached ?? null, failedInvariants: entry?.invariants?.byInvariant ?? {}, + failedTasks: (entry?.tasks ?? []).filter(task => !task.pass).map(({ pass: _pass, ...task }) => task), + violations: (entry?.invariants?.violations ?? []).map(({ rule, kind, reason, recordId, evidence }) => + ({ rule, kind, reason, recordId, ...(evidence ? { evidence } : {}) })), + durationMs: entry?.restedAt ? entry.restedAt - entry.armedAt : 0 }; + if (!entry) return { ...base, outcome: "harness_error", error: `not run (wave exited ${exitCode})` }; + if (base.failedTasks.length > 0 || base.violations.length > 0) return { ...base, outcome: "fail" }; + if (entry.invariants?.errors.length) return { ...base, outcome: "harness_error", error: entry.invariants.errors.join("; ") }; + if (!entry.invariants) return { ...base, outcome: "harness_error", error: "no invariant check" }; + if (!entry.reached) return { ...base, outcome: "not_reached" }; + return { ...base, outcome: "pass" }; +} + +/** One wave for every selected point: one boot, one point at a time. */ +async function runSuite(points: readonly CrashPointName[], options: SuiteOptions, runDir: string): Promise<{ results: PointResult[]; exitCode: number | null; waveDir: string | null }> { + const cells = options.adapters.length * options.workloads.length; + const args = [path.join(repositoryRoot, "cli/node_modules/tsx/dist/cli.mjs"), path.join(import.meta.dirname, "main.ts"), + "--env", options.env, "--adapters", options.adapters.join(","), "--workloads", options.workloads.join(","), + "--tasks", String(cells * points.length), "--concurrency", String(cells), "--seed", String(options.seed), + "--crash-point", points.map(point => `${point}:${options.nth}`).join(","), ...options.passthrough]; + console.info(`\n[crash-points] ${points.length} point(s) × ${cells} task(s), results ${runDir}`); + const exitCode = await new Promise(resolve => { + const child = spawn(process.execPath, args, { cwd: repositoryRoot, stdio: "inherit", + env: { ...process.env, RESTART_CHAOS_RESULTS_DIR: runDir, + // The wave boots a fresh instance (all migrations); allow a loaded host. + RESTART_CHAOS_HEALTH_TIMEOUT_MS: process.env.RESTART_CHAOS_HEALTH_TIMEOUT_MS ?? "600000" } }); + child.once("exit", code => resolve(code)); + child.once("error", () => resolve(null)); + }); + const waves = (await readdir(runDir, { withFileTypes: true }).catch(() => [])) + .filter(entry => entry.isDirectory() && entry.name.startsWith("w-")).map(entry => entry.name).sort(); + const waveDir = waves.length ? path.join(runDir, waves[waves.length - 1]!) : null; + const filePath = waveDir ? path.join(waveDir, "crash-point.json") : null; + const file = filePath && existsSync(filePath) ? JSON.parse(await readFile(filePath, "utf8")) as { points?: CrashPointEntry[] } : null; + const entries = new Map((file?.points ?? []).map(entry => [entry.point.replace(/:\d+$/, ""), entry])); + return { results: points.map(point => ({ ...classifyPoint(point, exitCode, entries.get(point) ?? null), waveDir })), exitCode, waveDir }; +} + +export function renderSummary(runId: string, options: SuiteOptions, results: readonly PointResult[]): string { + const lines = [`# Crash-point suite ${runId}`, "", + `env ${options.env}, adapters ${options.adapters.join(",")}, workloads ${options.workloads.join(",")}, nth ${options.nth}, seed ${options.seed}`, "", + "| Point | Outcome | Invariants violated | Failed tasks (rules) | Minutes (armed → rest) |", "|---|---|---|---|---|"]; + for (const result of results) { + const invariants = Object.entries(result.failedInvariants).map(([id, count]) => `${id}×${count}`).join(", ") || "—"; + const tasks = result.failedTasks.map(task => `${task.taskId} ${task.adapter}/${task.workload} ` + + `(${Object.entries(task.rules).filter(([, verdict]) => verdict === "fail").map(([id]) => id).join(",") || task.signature || "T"})`).join("; ") || "—"; + lines.push(`| \`${result.point}\` | ${result.outcome}${result.error ? ` (${result.error})` : ""} | ${invariants} | ${tasks} | ${(result.durationMs / 60_000).toFixed(1)} |`); + } + const violations = results.flatMap(result => result.violations.map(violation => ({ point: result.point, ...violation }))); + if (violations.length) { + lines.push("", "## Violations", "", "| Point | Rule | Kind | Reason | Record | Evidence |", "|---|---|---|---|---|---|"); + for (const violation of violations) { + const evidence = violation.evidence ? `\`${JSON.stringify(violation.evidence).replace(/\|/g, "\\|")}\`` : "—"; + lines.push(`| \`${violation.point}\` | ${violation.rule} | ${violation.kind} | ${violation.reason} | ${violation.recordId} | ${evidence} |`); + } + } + return `${lines.join("\n")}\n`; +} + +async function main(): Promise { + const options = parseSuiteArgs(process.argv.slice(2)); + const selected = selectPoints(CRASH_POINTS, options); + if (options.list) { + for (const name of selected) { + const definition = CRASH_POINTS[name] as CrashPointDefinition; + console.info(`${name}\n ${definition.description}\n kinds ${definition.kinds.join(", ") || "—"}; invariants ${definition.invariants.join(", ")}; env ${definition.environments.join(", ")}`); + } + return 0; + } + if (!selected.length) throw new Error(`No crash point matches --points ${options.points.join(",") || "*"} --kinds ${options.kinds.join(",") || "*"} --env ${options.env}`); + const runId = `cp-${new Date().toISOString().replace(/[-:]/g, "").replace(/\.\d{3}Z$/, "")}`; + const runDir = path.join(resultsRoot, "crash-points", runId); + await mkdir(runDir, { recursive: true, mode: 0o700 }); + const startedAt = Date.now(); + const { results, exitCode } = await runSuite(selected, options, runDir); + const summary = renderSummary(runId, options, results) + + `\nWave exit ${exitCode}; ${((Date.now() - startedAt) / 60_000).toFixed(1)} min in all (one boot, then one point at a time).\n`; + // Violation evidence carries run and wake error text; CI uploads both files. + const secrets = knownSecretValues(); + await writeFile(path.join(runDir, "SUMMARY.md"), redactKnownSecrets(summary, secrets), { mode: 0o600 }); + await writeFile(path.join(runDir, "summary.json"), + `${redactKnownSecrets(JSON.stringify({ runId, options, exitCode, results }, null, 2), secrets)}\n`, { mode: 0o600 }); + console.info(`\n${redactKnownSecrets(summary, secrets)}\nResults: ${runDir}`); + // Exit 1 with every point clean is a wave-level failure (for example a Daytona leak); see the wave timeline. + if (exitCode === 2 || exitCode === null) return 2; + if (results.some(result => result.outcome === "harness_error")) return 2; + if (results.some(result => result.outcome === "fail") || exitCode === 1) return 1; + if (results.some(result => result.outcome === "not_reached")) return 3; + return 0; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { + process.exit(await main().catch(error => { console.error(error); return 2; })); +} diff --git a/tests/restart-chaos/invariants.ts b/tests/restart-chaos/invariants.ts new file mode 100644 index 00000000000..8af8fde2cb2 --- /dev/null +++ b/tests/restart-chaos/invariants.ts @@ -0,0 +1,132 @@ +/** + * Plan invariants I1–I8 for one wave (durable ownership plan, "Invariants"). + * + * Runs the Phase 1 report-only ownership reconciler against the wave's own + * database after every task rested. Nothing is written: `record: false`. + * + * I1, I2 two passes at the real clock: a record with no owner stamp (I1), + * or whose owner was proven gone on both passes and is still in + * progress (I2: "settled within one reconciler pass"). + * I3, I4, I7 one more pass with the clock moved past every grace window + * (accounting closure 10 min, next action 5 min). The tasks are at + * rest, so anything still waiting then would wait forever. + * I5, I6, I8 not measurable by the reconciler yet; reported as coverage + * gaps with its own reasons. The chaos graders cover part of them: + * R3 (one Retry at most) bounds I5 and R1 (finished in budget) I8. + */ +import { createDb, type Db } from "../../packages/db/src/index.js"; +import { + createOwnershipReportState, + runOwnershipReconciliationPass, + type InvariantViolation, +} from "../../server/src/services/recovery/ownership-invariant-report.js"; + +export type PlanInvariant = "I1" | "I2" | "I3" | "I4" | "I5" | "I6" | "I7" | "I8"; + +/** A violation with the state of its record and run, so a failure is diagnosable after the instance is gone. */ +export type EvidencedViolation = InvariantViolation & { evidence?: Record }; + +export interface InvariantCheck { + violations: EvidencedViolation[]; + byInvariant: Partial>; + inProgress: Record; + notMeasured: Partial>; + errors: string[]; +} + +/** Grace windows the shifted pass must exceed (ownership-invariant-report.ts). */ +export const GRACE_SHIFT_MS = 15 * 60_000; + +export const invariantOf = (violation: Pick): PlanInvariant => violation.rule.slice(0, 2) as PlanInvariant; + +export function summarizeViolations(violations: readonly InvariantViolation[]): Partial> { + const counts: Partial> = {}; + for (const violation of violations) counts[invariantOf(violation)] = (counts[invariantOf(violation)] ?? 0) + 1; + return counts; +} + +/** + * Local-disk kinds (usage-receipt and orphan-cleanup spools) resolve their + * directories from the environment when listed. Point them at the wave's + * instance, not the operator's, for the duration of the check. + */ +async function withWaveInstance(instance: { paperclipHome: string; instanceId: string }, run: () => Promise): Promise { + const keys = ["PAPERCLIP_HOME", "PAPERCLIP_INSTANCE_ID", "SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR"] as const; + const saved = Object.fromEntries(keys.map(key => [key, process.env[key]])); + process.env.PAPERCLIP_HOME = instance.paperclipHome; + process.env.PAPERCLIP_INSTANCE_ID = instance.instanceId; + delete process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR; + try { return await run(); } finally { + for (const key of keys) { + if (saved[key] === undefined) delete process.env[key]; else process.env[key] = saved[key]; + } + } +} + +const clip = (value: unknown) => typeof value === "string" ? value.slice(0, 300) : value ?? null; + +async function withEvidence(db: Db, violations: InvariantViolation[]): Promise { + if (!violations.length) return violations; + const wakeIds = violations.filter(violation => violation.kind === "wake_request" || violation.kind === "wake_request_claimed") + .map(violation => violation.recordId); + const runIds = [...new Set(violations.flatMap(violation => violation.runId ? [violation.runId] : []))]; + type Row = Record; + const wakes = (wakeIds.length ? await db.$client.unsafe( + "select id::text, status, reason, error, run_id::text as \"runId\" from agent_wakeup_requests where id::text = any($1)", [wakeIds]) : []) as Row[]; + const runs = (runIds.length ? await db.$client.unsafe( + "select id::text, status, error_code as \"errorCode\", error from heartbeat_runs where id::text = any($1)", [runIds]) : []) as Row[]; + const wakeById = new Map(wakes.map(wake => [wake.id, wake])); + const runById = new Map(runs.map(run => [run.id, run])); + return violations.map(violation => { + const wake = wakeById.get(violation.recordId); + const run = violation.runId ? runById.get(violation.runId) : undefined; + const evidence = { + ...(wake ? { wake: { status: wake.status, reason: clip(wake.reason), error: clip(wake.error), runId: wake.runId } } : {}), + ...(run ? { run: { status: run.status, errorCode: run.errorCode, error: clip(run.error) } } : {}), + }; + return Object.keys(evidence).length ? { ...violation, evidence } : violation; + }); +} + +export async function checkPlanInvariants(input: { + connectionString: string; companyId: string | null; paperclipHome: string; instanceId: string; now?: Date; +}): Promise { + const db = createDb(input.connectionString); + const errors: string[] = []; + try { + const now = input.now ?? new Date(); + const pass = (state: ReturnType, at: Date) => + withWaveInstance(input, () => runOwnershipReconciliationPass(db, { state, now: at, record: false })); + const state = createOwnershipReportState(); + // I2 needs the same record proven gone on both passes: a kind that failed + // to list on the first pass has only one proof, so report it as incomplete. + const first = await pass(state, now); + for (const kind of first.kinds) if (kind.error) errors.push(`${kind.kind} (first I1/I2 pass): ${kind.error}`); + if (first.retryNeeded) errors.push("the first I1/I2 pass did not finish (retryNeeded)"); + const second = await pass(state, new Date(now.getTime() + 1_000)); + // A failed I3/I4/I7 query is only logged by the pass and sets retryNeeded. + // Retry once; a pass that still could not finish is an incomplete check. + let shifted = await pass(createOwnershipReportState(), new Date(now.getTime() + GRACE_SHIFT_MS)); + if (shifted.retryNeeded) shifted = await pass(createOwnershipReportState(), new Date(now.getTime() + GRACE_SHIFT_MS)); + for (const kind of [...second.kinds, ...shifted.kinds]) if (kind.error) errors.push(`${kind.kind}: ${kind.error}`); + if (second.retryNeeded) errors.push("the I1/I2 pass did not finish (retryNeeded)"); + if (shifted.retryNeeded) errors.push("the I3/I4/I7 pass did not finish after one retry (retryNeeded)"); + const ownCompany = (violation: InvariantViolation) => input.companyId === null || violation.companyId === null + || violation.companyId === input.companyId; + const fromClock = second.violations.filter(violation => violation.rule === "I1_owned" || violation.rule === "I2_settled"); + const fromShift = shifted.violations.filter(violation => violation.rule !== "I1_owned" && violation.rule !== "I2_settled"); + const violations = await withEvidence(db, [...fromClock, ...fromShift].filter(ownCompany)); + const inProgress = Object.fromEntries(second.kinds.filter(kind => kind.inProgress > 0) + .map(kind => [kind.kind, { inProgress: kind.inProgress, ...kind.verdicts }])); + return { + violations, byInvariant: summarizeViolations(violations), inProgress, errors, + notMeasured: { + I5: second.coverageGaps.I5_work_based_budgets ?? "not measured", + I6: second.coverageGaps.I6_generation_fencing ?? "not measured", + I8: second.coverageGaps.I8_fast_turns ?? "not measured", + }, + }; + } finally { + await db.$client.end({ timeout: 5 }).catch(() => undefined); + } +} diff --git a/tests/restart-chaos/ledger.ts b/tests/restart-chaos/ledger.ts index 57a5c7ea87f..e843d49e2f5 100644 --- a/tests/restart-chaos/ledger.ts +++ b/tests/restart-chaos/ledger.ts @@ -32,6 +32,8 @@ export interface LedgerLine { recordedAt: string; /** Known signatures masked for this wave, e.g. ["rc06"]. */ masks?: string[]; + /** Crash-point suite: the named point this wave armed. */ + crashPoint?: string; } export function ledgerLine(plan: WavePlan, record: TaskRecord, grade: TaskGrade, gitSha: string, evidence: string | null, @@ -49,6 +51,7 @@ export function ledgerLine(plan: WavePlan, record: TaskRecord, grade: TaskGrade, signature: grade.signature, retried: Boolean(record.retry), evidence, replay: replayCommand(plan.waveId, record.plan.taskId), recordedAt: new Date().toISOString(), ...(plan.options.masks?.length ? { masks: [...plan.options.masks] } : {}), + ...(plan.options.crashPoint ? { crashPoint: plan.options.crashPoint } : {}), }; } diff --git a/tests/restart-chaos/main.ts b/tests/restart-chaos/main.ts index 8ca2a0e1a28..9d9d83581bf 100644 --- a/tests/restart-chaos/main.ts +++ b/tests/restart-chaos/main.ts @@ -7,7 +7,7 @@ * node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay / */ import { execFileSync, spawnSync } from "node:child_process"; -import { existsSync, writeFileSync } from "node:fs"; +import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; @@ -17,8 +17,9 @@ import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from " import { assertControllerMatchesImage, prepareDaytonaImageArtifacts } from "./daytona-image.js"; import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js"; import { DaytonaWaveSweeper, leaksForTask, unattributedLeaks, type DaytonaSweepResult } from "./daytona-sweep.js"; -import { knownSecretValues, writeEvidenceBundle } from "./evidence.js"; +import { knownSecretValues, redactKnownSecrets, writeEvidenceBundle } from "./evidence.js"; import { gradeTask } from "./grader.js"; +import { checkPlanInvariants, summarizeViolations, type InvariantCheck } from "./invariants.js"; import { appendLedger, ledgerLine, writeSummary, type LedgerLine } from "./ledger.js"; import { TaskObserver } from "./observer.js"; import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js"; @@ -135,11 +136,33 @@ async function main(): Promise { tasks: plan.tasks.map(task => `${task.taskId}:${task.adapter}/${task.workload}/${task.fault}${task.trigger ? `^${task.trigger}+${task.triggerDelayMs ?? 0}ms` : ""}@${task.fireAfterMs}ms${task.holdMs ? ` hold ${task.holdMs}ms` : ""}`), ...(plan.storm ? { storm: { durationMs: plan.storm.durationMs, faults: plan.storm.faults.map(fault => `${fault.kind}+${fault.gapMs}ms${fault.holdMs ? ` hold ${fault.holdMs}ms` : ""}`) } } : {}) }); preflight(plan); - const serverEnvironment = plan.options.env === "daytona" ? await prepareDaytona(plan, timeline) : {}; + const serverEnvironment: Record = plan.options.env === "daytona" ? await prepareDaytona(plan, timeline) : {}; + // The suite drives the API only; the Vite dev UI only slows every boot and restart. + if (plan.options.crashPoint && !plan.replayOf) serverEnvironment.SERVE_UI = "false"; // Baseline only after the harness's own build steps (preflight rewrites the runner's dist). startFingerprint = worktreeFingerprint(repositoryRoot, guardedOutputs); const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline }); + // Crash-point suite: one booted server, one point at a time. Each point gets + // its own batch of tasks; the point is armed once the batch launched and the + // server kills itself at its nth hit (restarted like K1). Invariants are + // checked when the batch rests, as violations new since the point was armed. + // A replay fires the recorded faults (the crash point's K1 restart included) + // at their offsets; re-arming the point could miss its nth hit with fewer tasks. + const crashPoint = plan.replayOf ? null : plan.options.crashPoint ?? null; + const crashPoints = crashPoint ? crashPoint.split(",") : []; + const batchSize = crashPoints.length ? Math.max(1, Math.floor(plan.tasks.length / crashPoints.length)) : 0; + type PointResult = { point: string; token: string; taskIds: string[]; armedAt: number; restedAt: number | null; + fault: FaultReceipt | null; invariants: InvariantCheck | null }; + const pointResults: Array }> = []; + let currentPoint: (PointResult & { baseline: Set }) | null = null; + let databaseUrl: string | null = null; + const violationKey = (violation: { rule: string; kind: string; recordId: string }) => `${violation.rule}:${violation.kind}:${violation.recordId}`; + if (crashPoint) { + if (plan.tasks.length !== batchSize * crashPoints.length) throw new Error(`--tasks must be a multiple of the ${crashPoints.length} crash point(s)`); + supervisor.useCrashPointControl({ controlPath: path.join(supervisor.paths.temporaryRoot, "crash-point.control"), + receiptPath: path.join(waveDir, "crash-point-receipts.jsonl") }); + } let sweeper: DaytonaWaveSweeper | null = null; let sandboxSweep: DaytonaSweepResult | null = null; let interrupted = false; @@ -231,6 +254,62 @@ async function main(): Promise { })); }; + supervisor.onCrashPoint = receipt => { + const owner = pointResults.find(result => result.token === receipt.crashPoint?.token); + if (owner) owner.fault = receipt; + faults.push(receipt); + if (receipt.error) { faultFailures += 1; timeline.add("fault_error", { taskId: "crash-point", error: receipt.error }); } + for (const candidate of active.filter(observer => !observer.done)) { + candidate.noteFault(receipt.faultId, receipt.startedAt); + candidate.extendDeadline((receipt.readyAt ?? Date.now()) - receipt.startedAt); + taskFaultLog[candidate.plan.taskId] = { createdAt: candidate.record.createdAt, faultIds: [...candidate.record.faultIds] }; + } + checkWorktree(receipt.faultId); + void persistFaultLog(); + }; + + const invariantsNow = async () => { + if (!databaseUrl) { + const config = JSON.parse(readFileSync(supervisor.paths.configPath, "utf8")) as { database?: { connectionString?: string } }; + databaseUrl = config.database?.connectionString ?? null; + if (!databaseUrl) throw new Error("Crash-point wave: the instance config has no database connection string"); + } + return checkPlanInvariants({ connectionString: databaseUrl, companyId: lab.companyId, + paperclipHome: supervisor.paths.home, instanceId: "default" }); + }; + const startCrashPoint = async (batch: TaskObserver[]) => { + const index = pointResults.length; + const spec = crashPoints[index]!; + // Violations already present belong to earlier points, not this one. + const before = await invariantsNow(); + const baseline = new Set(before.violations.map(violationKey)); + // Stage the batch unassigned (no wake yet), arm the point, then assign: + // no task's work can start before the point is armed, and an early + // crash cannot fail a later task's creation. The previous batch has + // rested, so only this batch's work can hit the point. + for (const observer of batch) await observer.stage(); + const result = { point: spec, token: `p${String(index + 1).padStart(2, "0")}`, taskIds: batch.map(observer => observer.plan.taskId), + armedAt: Date.now(), restedAt: null, fault: null, invariants: null, baseline }; + pointResults.push(result); + supervisor.armCrashPoint(result.token, spec); + for (const observer of batch) { + launched.push(observer); + await observer.release(); + if (!observer.done) active.push(observer); + } + return result; + }; + const finishCrashPoint = async (point: PointResult & { baseline: Set }) => { + supervisor.disarmCrashPoint(); + point.restedAt = Date.now(); + const check = await invariantsNow(); + const fresh = check.violations.filter(violation => !point.baseline.has(violationKey(violation))); + point.invariants = { ...check, violations: fresh, byInvariant: summarizeViolations(fresh) }; + timeline.add("crash_point_settled", { point: point.point, token: point.token, reached: point.fault?.crashPoint ?? null, + seconds: Math.round((point.restedAt - point.armedAt) / 1000), byInvariant: point.invariants.byInvariant, + errors: check.errors.map(error => redactKnownSecrets(error, knownSecretValues())) }); + }; + while (!interrupted && (pending.length || active.length)) { const now = Date.now(); if (storm?.endIfDue(now)) { @@ -241,8 +320,15 @@ async function main(): Promise { // A failed fault or an unexpected exit leaves the server not ready. // Recover within a bound, or fail the wave instead of waiting forever. if (!supervisor.serverReady && !supervisor.faultInProgress) await supervisor.ensureServing(); + if (crashPoint) { + // One point at a time: settle the previous batch, then launch and arm the next. + if (!active.length && supervisor.serverReady && !supervisor.faultInProgress) { + if (currentPoint) { await finishCrashPoint(currentPoint); currentPoint = null; } + if (pending.length) currentPoint = await startCrashPoint(pending.splice(0, batchSize)); + } + } // Recheck the storm clock per launch: ensureServing or a slow launch can cross the storm's end. - while (active.length < plan.options.concurrency && pending.length && supervisor.serverReady && !supervisor.faultInProgress + while (!crashPoint && active.length < plan.options.concurrency && pending.length && supervisor.serverReady && !supervisor.faultInProgress && (!storm || storm.mayLaunch(Date.now()))) { const next = pending.shift()!; launched.push(next); @@ -254,6 +340,8 @@ async function main(): Promise { await observer.poll(now); })); for (const observer of active) { + // A crash-point wave has no scheduled faults: the armed point is the fault. + if (crashPoint) break; if (faultsQueued || supervisor.faultInProgress || !supervisor.serverReady) break; // A storm's drain gets no new fault; an unfired trigger is recorded as missed at rest. if (storm && !storm.mayLaunch(Date.now())) break; @@ -340,6 +428,26 @@ async function main(): Promise { rules: Object.fromEntries(Object.entries(grade.rules).map(([id, rule]) => [id, rule.verdict])), T: grade.T.verdict }); } await writeFile(path.join(waveDir, "faults.json"), `${JSON.stringify(faults, null, 2)}\n`, { mode: 0o600 }); + if (crashPoint) { + if (currentPoint) { await finishCrashPoint(currentPoint); currentPoint = null; } + const byTask = new Map(lines.map(line => [line.taskId, line])); + for (const point of pointResults) { + if (point.invariants?.violations.length) anyFailure = true; + if (point.invariants?.errors.length) harnessError ??= new Error(`Invariant check incomplete at ${point.point}: ${point.invariants.errors.join("; ")}`); + } + // Evidence carries run and wake error text; CI uploads this file. + await writeFile(path.join(waveDir, "crash-point.json"), `${redactKnownSecrets(JSON.stringify({ + schema: "paperclip.restart-chaos.crash-point/v2", gitSha, gitDirty, + // The server kills itself without warning, so no pre-fault workspace + // snapshot exists: R2 checks workspace failures, not lost files. + notMeasured: { R2_lost_files: "no pre-fault workspace snapshot: the crash point fires inside the server" }, + points: pointResults.map(({ baseline: _baseline, ...point }) => ({ + ...(point as PointResult), reached: point.fault?.crashPoint ?? null, + tasks: point.taskIds.flatMap(id => { const line = byTask.get(id); return line ? [{ taskId: id, adapter: line.adapter, + workload: line.workload, pass: line.pass, signature: line.signature, rules: line.rules, ruleDetails: line.ruleDetails, T: line.T }] : []; }), + })), + }, null, 2), knownSecretValues())}\n`, { mode: 0o600 }); + } // R6 at wave level: a leak no task's runs account for (e.g. the setup probe) still fails the wave. if (sandboxSweep) { const unattributed = unattributedLeaks(sandboxSweep, attributed); diff --git a/tests/restart-chaos/observer.ts b/tests/restart-chaos/observer.ts index 5e2901253cf..672dec5f088 100644 --- a/tests/restart-chaos/observer.ts +++ b/tests/restart-chaos/observer.ts @@ -3,6 +3,7 @@ import { randomUUID } from "node:crypto"; import { readdir, readFile } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; import { ChaosApiError, type ChaosApi } from "./api.js"; import { ACTIVE_RUN_STATUSES, isStalledSample, STALL_LIMIT_MS } from "./grader.js"; import type { ChaosLab } from "./seed.js"; @@ -159,15 +160,50 @@ export class TaskObserver { get definition() { return workloadDefinitions[this.plan.workload]; } - async launch(): Promise { + /** + * Crash-point suite: create the issue without an assignee, so no wake is + * queued yet. `release` assigns it once the batch's point is armed. + */ + async stage(): Promise { await this.launch({ assign: false }); } + + /** + * Assigns a staged issue, which queues its wake. An earlier task of the same + * batch can crash the server meanwhile, so retry until the replacement + * answers; setting the same assignee again is idempotent. + */ + async release(deadlineMs = 10 * 60_000): Promise { + if (this.done || !this.record.issueId) return; + const until = Date.now() + deadlineMs; + for (;;) { + try { + await this.api.patch(`/api/issues/${this.record.issueId}`, { assigneeAgentId: this.lab.agents[this.plan.adapter].id }); + break; + } catch (error) { + const transient = error instanceof ChaosApiError && (error.status === null || error.status >= 500); + if (!transient || Date.now() > until) { + this.record.launchError = error instanceof Error ? error.message : String(error); + this.done = true; + this.timeline.add("task_launch_failed", { taskId: this.plan.taskId, error: this.record.launchError }); + return; + } + await delay(1_000); + } + } + this.record.createdAt = Date.now(); + this.record.deadlineAt = this.record.createdAt + this.definition.budgetMs; + this.timeline.add("task_released", { taskId: this.plan.taskId, issueId: this.record.issueId }); + } + + async launch(options: { assign?: boolean } = {}): Promise { const definition = this.definition; + const assign = options.assign !== false; try { const issue = await this.api.post(`/api/companies/${this.lab.companyId}/issues`, { title: definition.title(this.plan.nonce), description: definition.description(this.plan.nonce), status: "todo", priority: "medium", - assigneeAgentId: this.lab.agents[this.plan.adapter].id, + ...(assign ? { assigneeAgentId: this.lab.agents[this.plan.adapter].id } : {}), ...(definition.usesRepository ? { projectId: this.lab.projectId } : {}), }); this.record.issueId = String(issue.id); diff --git a/tests/restart-chaos/supervisor.ts b/tests/restart-chaos/supervisor.ts index d6dfb874484..eadf5919bee 100644 --- a/tests/restart-chaos/supervisor.ts +++ b/tests/restart-chaos/supervisor.ts @@ -15,7 +15,7 @@ */ import { spawn, type ChildProcess } from "node:child_process"; import { randomBytes } from "node:crypto"; -import { createWriteStream, statSync, type WriteStream } from "node:fs"; +import { createWriteStream, readFileSync, statSync, writeFileSync, type WriteStream } from "node:fs"; import { mkdir, readFile, symlink } from "node:fs/promises"; import { createRequire } from "node:module"; import os from "node:os"; @@ -36,7 +36,11 @@ import type { MatrixExecution } from "../runner-e2e/types.js"; import type { Timeline } from "./timeline.js"; import type { FaultKind, FaultReceipt, TriggerId } from "./types.js"; -const HEALTH_TIMEOUT_MS = 180_000; +/** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */ +const HEALTH_TIMEOUT_MS = (() => { + const raw = Number(process.env.RESTART_CHAOS_HEALTH_TIMEOUT_MS); + return Number.isInteger(raw) && raw >= 30_000 && raw <= 1_800_000 ? raw : 180_000; +})(); const READY_TIMEOUT_MS = 300_000; const STOP_TIMEOUT_MS = 30_000; @@ -52,6 +56,30 @@ export function crashLoopShouldFire(target: (typeof CRASH_LOOP_TARGETS)[number], export interface SupervisorPaths { temporaryRoot: string; home: string; configPath: string; serverLogPath: string } +/** + * Crash-point suite: every server spawn reads its armed point from the control + * file at each hit (PAPERCLIP_CRASH_POINT_CONTROL), so one booted server serves + * many points. The supervisor writes " [:]" to arm a point + * and empties the file to disarm, including before a crash's replacement starts. + */ +export interface CrashPointControl { controlPath: string; receiptPath: string } + +/** The receipt line the crash-point helper appended for this pid, if any. */ +export function readCrashPointReceipt(receiptPath: string, pid: number | undefined): { name: string; hit: number; pid: number; token?: string } | null { + let text: string; + try { text = readFileSync(receiptPath, "utf8"); } catch { return null; } + for (const line of text.split("\n").reverse()) { + try { + const parsed = JSON.parse(line) as { point?: unknown; hit?: unknown; pid?: unknown }; + if (pid !== undefined && parsed.pid === pid && typeof parsed.point === "string" && typeof parsed.hit === "number") { + const token = (parsed as { token?: unknown }).token; + return { name: parsed.point, hit: parsed.hit, pid, ...(typeof token === "string" ? { token } : {}) }; + } + } catch { /* partial or blank line */ } + } + return null; +} + const secret = () => randomBytes(48).toString("base64url"); /** Same provider PATH shape as the runner-e2e launcher: pinned Codex first. */ @@ -84,6 +112,9 @@ export class ChaosSupervisor { serverReady = false; faultInProgress = false; unexpectedExits = 0; + private crashPointControl: CrashPointControl | null = null; + /** Called with the receipt of a crash-point self-kill once the replacement is ready (or failed). */ + onCrashPoint: ((receipt: FaultReceipt) => void) | null = null; private constructor(readonly repositoryRoot: string, readonly port: number, readonly paths: SupervisorPaths, private readonly timeline: Timeline) { @@ -139,6 +170,10 @@ export class ChaosSupervisor { PAPERCLIP_STORAGE_LOCAL_DIR: path.join(temporaryRoot, "storage"), PAPERCLIP_SECRETS_PROVIDER: "local_encrypted", PAPERCLIP_SECRETS_STRICT_MODE: "true", PAPERCLIP_DB_BACKUP_ENABLED: "false", PAPERCLIP_DB_BACKUP_DIR: path.join(temporaryRoot, "backups"), PAPERCLIP_OPEN_ON_LISTEN: "false", + // The server loads a .paperclip/.env found above its cwd without overriding + // set values. A worktree-initialized checkout's file sets + // PAPERCLIP_IN_WORKTREE=true, which suppresses all scheduling: no run starts. + PAPERCLIP_IN_WORKTREE: "false", ...extraEnvironment, }; const environment = buildPaperclipServerEnvironment(source, overrides); @@ -154,14 +189,22 @@ export class ChaosSupervisor { private async spawnServer(): Promise { if (this.shuttingDown) throw new Error("Refusing to start Paperclip during harness shutdown"); const entry = path.join(this.repositoryRoot, "tests/runner-e2e/server-entry.ts"); + const control = this.crashPointControl; + const env = control ? { ...this.environment, PAPERCLIP_CRASH_POINT_CONTROL: control.controlPath, + PAPERCLIP_CRASH_POINT_RECEIPT: control.receiptPath } : this.environment; const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(this.repositoryRoot, entry, ["onboard", "--yes", "--run"]), - { cwd: this.repositoryRoot, env: this.environment, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached }); + { cwd: this.repositoryRoot, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached }); this.child = child; + child.stdout?.on("data", chunk => this.log.write(chunk)); child.stderr?.on("data", chunk => this.log.write(chunk)); child.once("error", error => { this.spawnErrors.set(child, error); }); child.once("exit", (code, signal) => { this.log.write(`\n[restart-chaos] server pid=${child.pid} exited code=${code} signal=${signal}\n`); + if (control && signal === "SIGKILL" && !this.expectedStops.has(child) && !this.shuttingDown) { + const hit = readCrashPointReceipt(control.receiptPath, child.pid); + if (hit) { this.restartAfterCrashPoint(child, hit); return; } + } if (!this.expectedStops.has(child) && !this.shuttingDown) { this.unexpectedExits += 1; this.serverReady = false; @@ -229,6 +272,74 @@ export class ChaosSupervisor { throw new Error(`Paperclip did not report ready within ${READY_TIMEOUT_MS}ms`); } + /** Crash-point suite: run every spawn (from `start` on) with runtime arming. */ + useCrashPointControl(control: CrashPointControl): void { + writeFileSync(control.controlPath, "", { mode: 0o600 }); + this.crashPointControl = control; + } + + /** Arms `spec` ("[:]") with a fresh hit count under `token`. */ + armCrashPoint(token: string, spec: string): void { + if (!this.crashPointControl) throw new Error("Crash-point control is not enabled"); + writeFileSync(this.crashPointControl.controlPath, `${token} ${spec}\n`, { mode: 0o600 }); + this.timeline.add("crash_point_armed", { token, spec }); + } + + disarmCrashPoint(): void { + if (this.crashPointControl) writeFileSync(this.crashPointControl.controlPath, "", { mode: 0o600 }); + } + + /** + * The armed server killed itself at its crash point. Treat it like K1: kill + * what is left of its owned process tree (local runners), start a + * replacement without the point armed, and wait for ready. + */ + private restartAfterCrashPoint(previous: ChildProcess, hit: { name: string; hit: number; pid: number; token?: string }): void { + // The replacement reads the same control file: disarm before it starts. + this.disarmCrashPoint(); + this.faultInProgress = true; + this.serverReady = false; + this.faultCount += 1; + const now = Date.now(); + const receipt: FaultReceipt = { + faultId: `f${String(this.faultCount).padStart(2, "0")}`, kind: "K1", requestedBy: "crash-point", signal: "SIGKILL", holdMs: 0, + loadAvg1m: Math.round(os.loadavg()[0]! * 100) / 100, crashPoint: hit, + requestedAt: now, startedAt: now, stoppedAt: null, restartedAt: null, healthyAt: null, readyAt: null, + phases: [], serverLogOffsetAtStart: this.serverLogOffset(), serverLogOffsetAtReady: null, error: null, + }; + this.timeline.add("crash_point_fired", { faultId: receipt.faultId, point: hit.name, hit: hit.hit, pid: hit.pid }); + this.log.write(`\n[restart-chaos] ${receipt.faultId} crash point ${hit.name} hit ${hit.hit} killed pid=${hit.pid}\n`); + let cleanupError: string | null = null; + const run = this.faultChain.then(async () => { + try { + // The leader is already dead; this kills its owned descendants, as K1 does. + // A cleanup that cannot prove the tree stopped fails the fault (and the + // wave); the replacement still starts so the tasks can drain. + await this.stopper.crashForRestart(previous).catch(error => { + cleanupError = `owned process tree cleanup failed: ${error instanceof Error ? error.message : String(error)}`; + this.log.write(`[restart-chaos] crash-point ${cleanupError}\n`); + }); + if (this.child === previous) this.child = null; + await this.waitForHealthToStop(); + receipt.stoppedAt = Date.now(); + receipt.restartedAt = Date.now(); + const child = await this.spawnServer(); + receipt.healthyAt = await this.waitForHealth(child, receipt.phases); + receipt.readyAt = await this.waitForReadyPhase(receipt.phases); + receipt.serverLogOffsetAtReady = this.serverLogOffset(); + this.timeline.add("fault_ready", { faultId: receipt.faultId, replacementPid: child.pid, readyAfterMs: receipt.readyAt - receipt.startedAt }); + } catch (error) { + receipt.error = error instanceof Error ? error.message : String(error); + this.timeline.add("fault_failed", { faultId: receipt.faultId, error: receipt.error }); + } finally { + if (cleanupError && !receipt.error) receipt.error = cleanupError; + this.faultInProgress = false; + this.onCrashPoint?.(receipt); + } + }); + this.faultChain = run.catch(() => undefined); + } + /** Faults are serialized: one control plane, one outage at a time. */ fault(kind: FaultKind, holdMs: number, requestedBy: string, trigger?: { trigger: TriggerId; delayMs: number }): Promise { const requestedAt = Date.now(); diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index e1106d82eea..da5d79b3f18 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -53,6 +53,13 @@ export interface WaveOptions { * keeps the catalog profile's own setting. */ lifecycle?: "per_turn" | "warm" | "default"; + /** + * Crash-point suite (`--crash-point [:][,...]`): no external + * fault is scheduled. One server serves every listed point in turn: each + * point's batch of tasks launches, then the point is armed and the server + * SIGKILLs itself at its nth hit and is restarted like K1. + */ + crashPoint?: string; } export interface StormFault { @@ -182,6 +189,8 @@ export interface FaultReceipt { triggerDelayMs?: number; loadAvg1m?: number; crashLoop?: Array<{ pid: number | null; target: string; phase: string; at: number }>; + /** Crash-point suite: the server killed itself at this named point (no external signal). */ + crashPoint?: { name: string; hit: number; pid: number; token?: string }; serverLogOffsetAtStart: number; serverLogOffsetAtReady: number | null; error: string | null; diff --git a/tests/runner-e2e/workflow-security.test.ts b/tests/runner-e2e/workflow-security.test.ts index 1ede9fed2dc..b1c4b45875b 100644 --- a/tests/runner-e2e/workflow-security.test.ts +++ b/tests/runner-e2e/workflow-security.test.ts @@ -617,6 +617,9 @@ describe("public repository paid workflow security", () => { // private-trust-workflow-security.test.ts checks how these two hold the keys. "chat-catalog-testops.yml", "chat-catalog-testops-case.yml", + // On-demand crash-point suite (durable ownership plan, Phase 5): manual + // dispatch, default branch, allowlisted actors, runner-e2e-paid. + "crash-points.yml", ]); const names = (await readdir(workflowDirectory)).filter((name) => /\.ya?ml$/.test(name),