mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - Random restart-chaos waves found these 20 ms–1 s windows only by luck > - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable write boundaries > - Nothing yet runs a task, kills the server at a chosen point, restarts it, and checks the plan's invariants > - This pull request adds that on-demand suite, as a local command and as a manual CI workflow (not a required check, per Decision 2) > - The benefit is a repeatable work list: which boundary breaks which invariant, for Phases 2–4 ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only reconciler used for I1–I7). Stacked on #19938 (call sites), which is stacked on #19937 (helper). **Problem** The named crash points cannot be exercised end to end. No command arms one point, runs real agent tasks across it, restarts, and grades the result. **Expected behavior** `node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters <list>` runs one wave per selected point and reports each point as pass, fail (with failing R rules and violated invariants), not reached, or harness error. A `workflow_dispatch` workflow runs the same thing on demand. ## What Changed - **Runtime arming.** `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the shared helper and the runner mirror. Each hit reads `<token> <name>[:<nth>]`, and a new token resets the count. It is test mode only: `NODE_ENV=production` ignores it, and the disarmed path is still one comparison. - **`tests/restart-chaos/main.ts --crash-point <a>:<nth>,<b>:<nth>,...`** (config, types, ledger): - one booted server and no Vite UI; - per point: take an invariant baseline, launch the point's batch, arm the point, then on rest disarm it and check invariants as violations that are new since the baseline; - every task is graded R1–R6 and T; - `crash-point.json` (v2) holds one entry per point. - **`tests/restart-chaos/supervisor.ts`:** - every spawn gets the control file; - a SIGKILL whose receipt names the child's pid is restarted like K1: the control file is emptied first, the owned tree is killed, and a replacement starts. A failed tree cleanup fails the fault; - the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a `.paperclip/.env` above the checkout otherwise suppressed all scheduling); - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`. - **`tests/restart-chaos/invariants.ts`:** - the Phase 1 report-only pass with `record: false`: two real-clock passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7; - spool kinds are read from the wave's instance; - an unfinished pass is a harness error; - each violation carries its wake and run state as evidence. - **`tests/restart-chaos/crash-points.ts` and `scripts/crash-points.mjs`:** - select points by glob, kind and environment, and run them in one wave; - a per-point outcome: pass, fail, not_reached or harness_error; - `SUMMARY.md` with a violation evidence column. - **`.github/workflows/crash-points.yml`**, manual only, under the paid-workflow contract: - a credential-free `authorize` job; - a `suite` job in `runner-e2e-paid` whose first step reauthorizes; - a frozen install with `--ignore-scripts` and no shared cache; - only the selected adapters' keys; - the workflow is added to both security-test allowlists. - **Docs:** "Crash-point suite" in `tests/restart-chaos/README.md`, and "Built (Phase 5)" plus the first work list in the plan. ## Verification - Focused tests: - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`; - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 tests, including runtime arming in a real child process; - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72 tests; - `tests/runner-e2e/workflow-security.test.ts`: 19 tests. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Loop line: question "how long is one crash-point loop?" | loop: `node scripts/crash-points.mjs --points <one>` (codex A1) | elapsed 2.7 min end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s, rest about 2 min | result: one point alone takes about 3 min. Before this change it took about 15 min (one cold boot per point, Vite UI, host at load 77). - Full local run, 14 points, codex A1, 11.8 min in all: - every point was reached (the server killed itself at each one, and the replacement came back); - 11 pass; - `workspace.finalize.after_start_before_copyback` and `workspace.finalize.after_copyback_before_done` fail I2 and R6: a `workspace_finalize` operation stays running after the run succeeded; - `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its evidence shows the wake failed with "…retrying once", and the retry delivered the work, so this is the known refusal-regex false positive. - The 6 Daytona-only points were not run. ## Risks - Test harness and an on-demand workflow only. No product code changes. - `.github/**` needs a CODEOWNER review. The workflow uses provider keys, so it refuses non-default branches and actors that are not on the allowlist, like `runner-full-stack-e2e.yml`. - `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only crash-point waves. A chaos instance is never a worktree dev instance, so scheduling must not be suppressed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
5742c42b75
commit
051834bdc8
18 files changed
+1187
-26
No files matched your search
@@ -0,0 +1,203 @@
|
||||
name: Crash-point suite
|
||||
|
||||
# On-demand only (durable ownership plan, Decision 2): never a required PR
|
||||
# check. Runs tests/restart-chaos/crash-points.ts on the default branch, which
|
||||
# kills an isolated control plane at each selected named crash point, restarts
|
||||
# it and asserts plan invariants I1–I8 plus restart-chaos graders R1–R6.
|
||||
# Provider keys come from the runner-e2e-paid environment, so the code under
|
||||
# test must be CODEOWNER-reviewed: the default branch only, allowlisted actors.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
points:
|
||||
description: "Comma-separated crash-point globs (blank = every point reachable in env)"
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
kinds:
|
||||
description: "Comma-separated ownership kind ids, e.g. native_executor_claim (blank = all)"
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
adapters:
|
||||
description: "Comma-separated adapters: codex, claude, opencode"
|
||||
required: false
|
||||
default: codex
|
||||
type: string
|
||||
workloads:
|
||||
description: "Comma-separated restart-chaos workloads: A1, B1, B5, C1"
|
||||
required: false
|
||||
default: A1
|
||||
type: string
|
||||
nth:
|
||||
description: "Fire at the nth hit of each point"
|
||||
required: false
|
||||
default: "1"
|
||||
type: string
|
||||
|
||||
concurrency:
|
||||
group: crash-points-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
# No secrets here: refuse the wrong branch, actor or input before the paid job starts.
|
||||
authorize:
|
||||
name: Authorize crash-point run
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Require default branch and allowlisted numeric actor IDs
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
REF: ${{ github.ref }}
|
||||
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
||||
ACTOR: ${{ github.actor }}
|
||||
ACTOR_ID: ${{ github.actor_id }}
|
||||
TRIGGERING_ACTOR: ${{ github.triggering_actor }}
|
||||
ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "$REF" != "refs/heads/$DEFAULT_BRANCH" ]; then
|
||||
echo "The crash-point suite uses provider keys and may run only from the default branch." >&2
|
||||
exit 1
|
||||
fi
|
||||
jq -e 'type == "array" and length > 0 and all(.[]; type == "number" and . > 0 and floor == .)' <<< "${ALLOWED_ACTOR_IDS:-}" >/dev/null
|
||||
for attempt in 1 2 3; do
|
||||
if triggering_actor_id="$(timeout 30s gh api "users/$TRIGGERING_ACTOR" --jq .id)"; then break; fi
|
||||
if [ "$attempt" = 3 ]; then exit 1; fi
|
||||
sleep "$((attempt * 2))"
|
||||
done
|
||||
if [ "$triggering_actor_id" != "$ACTOR_ID" ] && [ "$TRIGGERING_ACTOR" = "$ACTOR" ]; then
|
||||
echo "GitHub actor identity contexts disagree; refusing the paid run." >&2
|
||||
exit 1
|
||||
fi
|
||||
for candidate in "$triggering_actor_id" "$ACTOR_ID"; do
|
||||
jq -e --argjson candidate "$candidate" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null
|
||||
done
|
||||
|
||||
- name: Validate inputs
|
||||
env:
|
||||
POINTS: ${{ inputs.points }}
|
||||
KINDS: ${{ inputs.kinds }}
|
||||
ADAPTERS: ${{ inputs.adapters }}
|
||||
WORKLOADS: ${{ inputs.workloads }}
|
||||
NTH: ${{ inputs.nth }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
[[ "$POINTS" =~ ^[a-z0-9_.*?,]*$ ]] || { echo "points: lower-case names, '.', '_', '*', '?' and ','" >&2; exit 1; }
|
||||
[[ "$KINDS" =~ ^[a-z0-9_,]*$ ]] || { echo "kinds: lower-case ids and ','" >&2; exit 1; }
|
||||
[[ "$ADAPTERS" =~ ^(codex|claude|opencode)(,(codex|claude|opencode))*$ ]] || { echo "adapters: codex, claude, opencode" >&2; exit 1; }
|
||||
[[ "$WORKLOADS" =~ ^(A1|B1|B5|C1)(,(A1|B1|B5|C1))*$ ]] || { echo "workloads: A1, B1, B5, C1" >&2; exit 1; }
|
||||
[[ "$NTH" =~ ^[1-9][0-9]{0,2}$ ]] || { echo "nth: 1-999" >&2; exit 1; }
|
||||
|
||||
suite:
|
||||
name: Crash points (${{ inputs.points || 'all' }})
|
||||
needs: authorize
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 300
|
||||
permissions:
|
||||
contents: read
|
||||
environment:
|
||||
name: runner-e2e-paid
|
||||
steps:
|
||||
- name: Reauthorize paid execution before provider access
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
REF: ${{ github.ref }}
|
||||
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
||||
ACTOR: ${{ github.actor }}
|
||||
ACTOR_ID: ${{ github.actor_id }}
|
||||
TRIGGERING_ACTOR: ${{ github.triggering_actor }}
|
||||
ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "$REF" != "refs/heads/$DEFAULT_BRANCH" ]; then
|
||||
echo "The crash-point suite uses provider keys and may run only from the default branch." >&2
|
||||
exit 1
|
||||
fi
|
||||
jq -e 'type == "array" and length > 0 and all(.[]; type == "number" and . > 0 and floor == .)' <<< "${ALLOWED_ACTOR_IDS:-}" >/dev/null
|
||||
for attempt in 1 2 3; do
|
||||
if triggering_actor_id="$(timeout 30s gh api "users/$TRIGGERING_ACTOR" --jq .id)"; then break; fi
|
||||
if [ "$attempt" = 3 ]; then exit 1; fi
|
||||
sleep "$((attempt * 2))"
|
||||
done
|
||||
if [ "$triggering_actor_id" != "$ACTOR_ID" ] && [ "$TRIGGERING_ACTOR" = "$ACTOR" ]; then
|
||||
echo "GitHub actor identity contexts disagree; refusing the paid run." >&2
|
||||
exit 1
|
||||
fi
|
||||
for candidate in "$triggering_actor_id" "$ACTOR_ID"; do
|
||||
jq -e --argjson candidate "$candidate" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null
|
||||
done
|
||||
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
||||
with:
|
||||
version: 9.15.4
|
||||
|
||||
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
||||
with:
|
||||
node-version: 24
|
||||
|
||||
# No shared dependency cache and no dependency lifecycle scripts in a
|
||||
# secret-bearing job (tests/runner-e2e/SECURITY.md); the repo's own
|
||||
# postinstall step runs explicitly.
|
||||
- run: pnpm install --frozen-lockfile --ignore-scripts
|
||||
- run: node scripts/link-plugin-dev-sdk.mjs
|
||||
|
||||
- name: Build native runner
|
||||
run: |
|
||||
set -euo pipefail
|
||||
pnpm --filter @paperclipai/paperclip-runner build:typescript
|
||||
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
|
||||
|
||||
- name: List selected crash points
|
||||
env:
|
||||
POINTS: ${{ inputs.points }}
|
||||
KINDS: ${{ inputs.kinds }}
|
||||
run: node scripts/crash-points.mjs --list ${POINTS:+--points "$POINTS"} ${KINDS:+--kinds "$KINDS"}
|
||||
|
||||
- name: Run crash-point suite
|
||||
env:
|
||||
POINTS: ${{ inputs.points }}
|
||||
KINDS: ${{ inputs.kinds }}
|
||||
ADAPTERS: ${{ inputs.adapters }}
|
||||
WORKLOADS: ${{ inputs.workloads }}
|
||||
NTH: ${{ inputs.nth }}
|
||||
OPENAI_API_KEY: ${{ contains(inputs.adapters, 'codex') && secrets.OPENAI_API_KEY || '' }}
|
||||
ANTHROPIC_API_KEY: ${{ contains(inputs.adapters, 'claude') && secrets.ANTHROPIC_API_KEY || '' }}
|
||||
OPENROUTER_API_KEY: ${{ contains(inputs.adapters, 'opencode') && secrets.OPENROUTER_API_KEY || '' }}
|
||||
# The hosted runner is a disposable VM; B1 oracles run agent code without Seatbelt there.
|
||||
RESTART_CHAOS_ALLOW_UNSANDBOXED_ORACLE: "1"
|
||||
RESTART_CHAOS_RESULTS_DIR: ${{ runner.temp }}/crash-point-results
|
||||
run: |
|
||||
set -euo pipefail
|
||||
node scripts/crash-points.mjs ${POINTS:+--points "$POINTS"} ${KINDS:+--kinds "$KINDS"} \
|
||||
--adapters "$ADAPTERS" --workloads "$WORKLOADS" --nth "$NTH" --seed "$((GITHUB_RUN_ID % 2147483647))"
|
||||
|
||||
- name: Summarize
|
||||
if: always()
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
for summary in "$RUNNER_TEMP"/crash-point-results/crash-points/*/SUMMARY.md; do cat "$summary" >> "$GITHUB_STEP_SUMMARY"; done
|
||||
|
||||
# Summaries, grades and crash-point receipts only: server logs and
|
||||
# evidence bundles can hold provider output and stay on the runner.
|
||||
- name: Upload results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: crash-points-${{ github.run_id }}
|
||||
path: |
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/SUMMARY.md
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/summary.json
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/w-*/crash-point.json
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/w-*/t*.grade.json
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/w-*/timeline.jsonl
|
||||
retention-days: 14
|
||||
if-no-files-found: warn
|
||||
@@ -223,6 +223,15 @@ The runner stamps `generation` and `turnId` on every event. Driver and session c
|
||||
- Keep the chaos harness as a soak test for timing interactions the enumeration misses. Its R1–R6 graders should map onto I1–I7.
|
||||
- The random waves found 20 ms–1 s windows by luck. Enumeration finds them on purpose, and it finds the windows we haven't hit yet.
|
||||
|
||||
**Built (Phase 5).**
|
||||
- `crashPoint(name)` in `packages/shared/src/crash-points.ts`, with the named points in `crash-point-registry.ts`. `PAPERCLIP_CRASH_POINT=<name>[:<nth>]` arms one point; `NODE_ENV=production` ignores it. Disarmed, a call costs about 4 ns. In-process tests use `armCrashPointForTest`.
|
||||
- 20 points at the boundaries in the inventory's write-ahead table, from `sandbox.create.after_provider_before_lease` to `accounting.closure.after_commit_before_spool_unlink`. `server/src/__tests__/crash-point-call-sites.test.ts` requires a call site for each one.
|
||||
- `node scripts/crash-points.mjs [--points <glob>] [--kinds <id>] [--adapters …]` boots one instance and arms each point in turn at runtime (`PAPERCLIP_CRASH_POINT_CONTROL`). It asserts R1–R6 plus I1–I4 and I7 per point, from the Phase 1 reconciler. I5, I6 and I8 are reported as not measured until sections F and G land. The manual CI run is `.github/workflows/crash-points.yml`.
|
||||
- First full local run (2026-11-08, 14 local points, codex A1, 11.8 min): 11 pass. Work list:
|
||||
- `workspace.finalize.after_start_before_copyback` and `…after_copyback_before_done` leave a `workspace_finalize` operation running after the run succeeded (I2 and R6, kind `native_workspace_finalize_attempt`).
|
||||
- `wake.claim.after_run_running_before_wakeup_claimed` reports I4 on a wake whose retry delivered the work: the known refusal-regex false positive, which section E removes.
|
||||
- How to add a point and how to read the results: `tests/restart-chaos/README.md`, "Crash-point suite".
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Scope | Exit criteria |
|
||||
|
||||
@@ -8,40 +8,64 @@
|
||||
* Disarmed (always under NODE_ENV=production) a call is one comparison.
|
||||
* runnerd never sees the variable: its environment is an allowlist.
|
||||
*/
|
||||
import { appendFileSync, existsSync } from "node:fs";
|
||||
import { appendFileSync, existsSync, readFileSync } from "node:fs";
|
||||
|
||||
export type RunnerCrashPointName =
|
||||
| "runner.process.after_spawn_before_identity"
|
||||
| "runner.turn_start.after_runner_accept_before_controller_checkpoint"
|
||||
| "runner.turn_start.after_controller_journal_before_db_checkpoint";
|
||||
|
||||
type Armed = { name: string; nth: number; hits: number; receiptPath: string | null; gatePath: string | null };
|
||||
type Armed = {
|
||||
name: string; nth: number; hits: number; receiptPath: string | null; gatePath: string | null;
|
||||
/** Runtime arming (PAPERCLIP_CRASH_POINT_CONTROL): read "<token> <name>[:<nth>]" at each hit. */
|
||||
controlPath: string | null; token: string | null;
|
||||
};
|
||||
|
||||
function parseSpec(raw: string): { name: string; nth: number } | null {
|
||||
const separator = raw.lastIndexOf(":");
|
||||
const name = separator === -1 ? raw : raw.slice(0, separator);
|
||||
const nth = separator === -1 ? 1 : Number(raw.slice(separator + 1));
|
||||
return name && Number.isInteger(nth) && nth >= 1 ? { name, nth } : null;
|
||||
}
|
||||
|
||||
function armFromEnvironment(): Armed | null {
|
||||
const env = typeof process === "undefined" ? undefined : process.env;
|
||||
const raw = env?.PAPERCLIP_CRASH_POINT?.trim();
|
||||
if (!env || !raw || env.NODE_ENV === "production") return null;
|
||||
const separator = raw.lastIndexOf(":");
|
||||
const name = separator === -1 ? raw : raw.slice(0, separator);
|
||||
if (!env || env.NODE_ENV === "production") return null;
|
||||
const receiptPath = env.PAPERCLIP_CRASH_POINT_RECEIPT?.trim() || null;
|
||||
const controlPath = env.PAPERCLIP_CRASH_POINT_CONTROL?.trim() || null;
|
||||
if (controlPath) return { name: "", nth: 1, hits: 0, receiptPath, gatePath: null, controlPath, token: null };
|
||||
const raw = env.PAPERCLIP_CRASH_POINT?.trim();
|
||||
if (!raw) return null;
|
||||
const spec = parseSpec(raw);
|
||||
// The shared helper validates the name; this copy only arms its own names.
|
||||
if (!name.startsWith("runner.")) return null;
|
||||
const nth = separator === -1 ? 1 : Number(raw.slice(separator + 1));
|
||||
if (!Number.isInteger(nth) || nth < 1) return null;
|
||||
return { name, nth, hits: 0, receiptPath: env.PAPERCLIP_CRASH_POINT_RECEIPT?.trim() || null,
|
||||
gatePath: env.PAPERCLIP_CRASH_POINT_GATE?.trim() || null };
|
||||
if (!spec || !spec.name.startsWith("runner.")) return null;
|
||||
return { ...spec, hits: 0, receiptPath, gatePath: env.PAPERCLIP_CRASH_POINT_GATE?.trim() || null, controlPath: null, token: null };
|
||||
}
|
||||
|
||||
const armed: Armed | null = armFromEnvironment();
|
||||
|
||||
function readControl(current: Armed): boolean {
|
||||
let text: string;
|
||||
try { text = readFileSync(current.controlPath!, "utf8"); } catch { return false; }
|
||||
const [token, raw, ...rest] = text.trim().split(/\s+/);
|
||||
const spec = token && raw && rest.length === 0 ? parseSpec(raw) : null;
|
||||
if (!token || !spec) return false;
|
||||
if (token !== current.token) Object.assign(current, { token, name: spec.name, nth: spec.nth, hits: 0 });
|
||||
return true;
|
||||
}
|
||||
|
||||
export function crashPoint(name: RunnerCrashPointName): void {
|
||||
if (armed === null || armed.name !== name) return;
|
||||
if (armed === null) return;
|
||||
if (armed.controlPath && !readControl(armed)) return;
|
||||
if (armed.name !== name) return;
|
||||
if (armed.gatePath !== null) {
|
||||
if (!existsSync(armed.gatePath)) return;
|
||||
armed.gatePath = null;
|
||||
}
|
||||
armed.hits += 1;
|
||||
if (armed.hits !== armed.nth) return;
|
||||
const receipt = { point: name, hit: armed.hits, pid: process.pid, at: new Date().toISOString() };
|
||||
const receipt = { point: name, hit: armed.hits, pid: process.pid, at: new Date().toISOString(),
|
||||
...(armed.token ? { token: armed.token } : {}) };
|
||||
if (armed.receiptPath) {
|
||||
try { appendFileSync(armed.receiptPath, `${JSON.stringify(receipt)}\n`); } catch { /* the log line still records it */ }
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
armCrashPointForTest,
|
||||
armedCrashPoint,
|
||||
crashPoint,
|
||||
parseCrashPointControl,
|
||||
parseCrashPointSpec,
|
||||
type CrashPointName,
|
||||
} from "./crash-points.js";
|
||||
@@ -50,6 +51,15 @@ describe("parseCrashPointSpec", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseCrashPointControl", () => {
|
||||
it("reads a token and a spec, and treats anything else as disarmed", () => {
|
||||
expect(parseCrashPointControl(`t1 ${first}:3\n`)).toEqual({ token: "t1", spec: { name: first, nth: 3 } });
|
||||
expect(parseCrashPointControl("")).toBeNull();
|
||||
expect(parseCrashPointControl(first)).toBeNull();
|
||||
expect(parseCrashPointControl("t1 sandbox.create.after_typo")).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("crashPoint in a test process", () => {
|
||||
it("does nothing while disarmed", () => {
|
||||
expect(armedCrashPoint()).toBeNull();
|
||||
@@ -150,6 +160,31 @@ describe("crashPoint armed by the environment", { timeout: 60_000 }, () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("arms at runtime from the control file and resets the count on a new token", () => {
|
||||
const dir = mkdtempSync(path.join(os.tmpdir(), "crash-point-control-"));
|
||||
try {
|
||||
const control = path.join(dir, "control");
|
||||
const receipt = path.join(dir, "receipt.jsonl");
|
||||
const hit = `crashPoint(${JSON.stringify(first)});`;
|
||||
const source = [
|
||||
`const { crashPoint } = await import(${JSON.stringify(path.join(import.meta.dirname, "crash-points.ts"))});`,
|
||||
`const { writeFileSync } = await import("node:fs");`,
|
||||
`${hit} process.stdout.write("absent\\n");`,
|
||||
`writeFileSync(${JSON.stringify(control)}, "t1 ${first}:2\\n");`,
|
||||
`${hit} process.stdout.write("t1-one\\n");`,
|
||||
`writeFileSync(${JSON.stringify(control)}, "t2 ${first}:2\\n");`,
|
||||
`${hit} process.stdout.write("t2-one\\n");`,
|
||||
`${hit} process.stdout.write("t2-two\\n");`,
|
||||
].join("\n");
|
||||
const result = runChild({ PAPERCLIP_CRASH_POINT_CONTROL: control, PAPERCLIP_CRASH_POINT_RECEIPT: receipt }, source);
|
||||
expect(result.signal).toBe("SIGKILL");
|
||||
expect(result.stdout).toBe("absent\nt1-one\nt2-one\n");
|
||||
expect(JSON.parse(readFileSync(receipt, "utf8").trim())).toMatchObject({ point: first, hit: 2, token: "t2" });
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("stays disarmed in production", () => {
|
||||
const result = runChild({ PAPERCLIP_CRASH_POINT: first, NODE_ENV: "production" });
|
||||
expect(result.signal).toBeNull();
|
||||
|
||||
@@ -14,6 +14,12 @@
|
||||
* PAPERCLIP_CRASH_POINT_GATE=<file> optional; hits before this file exists are not counted,
|
||||
* so setup work (seeding, probes) cannot fire the point
|
||||
*
|
||||
* Runtime arming (one booted server, many points), instead of PAPERCLIP_CRASH_POINT:
|
||||
*
|
||||
* PAPERCLIP_CRASH_POINT_CONTROL=<file> each hit reads "<token> <name>[:<nth>]" from the file;
|
||||
* no file or an empty one means disarmed. Hits count per
|
||||
* token, so writing a new token re-arms with a fresh count.
|
||||
*
|
||||
* The variable is read once, when this module is first imported. It is ignored
|
||||
* when `NODE_ENV=production`, and an unknown name throws at import so a typo
|
||||
* cannot silently arm nothing. Disarmed, `crashPoint` is a single comparison
|
||||
@@ -27,7 +33,7 @@
|
||||
* This is test instrumentation, not Telemetry or Observability (AGENTS.md §5.7):
|
||||
* nothing is recorded or sent anywhere unless a test arms a point.
|
||||
*/
|
||||
import { appendFileSync, existsSync } from "node:fs";
|
||||
import { appendFileSync, existsSync, readFileSync } from "node:fs";
|
||||
import { CRASH_POINTS, isCrashPointName, type CrashPointName } from "./crash-point-registry.js";
|
||||
|
||||
export { CRASH_POINTS, CRASH_POINT_NAME_PATTERN, isCrashPointName } from "./crash-point-registry.js";
|
||||
@@ -36,6 +42,7 @@ export type { CrashPointDefinition, CrashPointInvariant, CrashPointName } from "
|
||||
export const CRASH_POINT_ENV = "PAPERCLIP_CRASH_POINT";
|
||||
export const CRASH_POINT_RECEIPT_ENV = "PAPERCLIP_CRASH_POINT_RECEIPT";
|
||||
export const CRASH_POINT_GATE_ENV = "PAPERCLIP_CRASH_POINT_GATE";
|
||||
export const CRASH_POINT_CONTROL_ENV = "PAPERCLIP_CRASH_POINT_CONTROL";
|
||||
/** Printed to stderr just before the kill; the suite greps server logs for it. */
|
||||
export const CRASH_POINT_LOG_PREFIX = "[crash-point]";
|
||||
|
||||
@@ -46,6 +53,8 @@ export type CrashPointReceipt = {
|
||||
hit: number;
|
||||
pid: number;
|
||||
at: string;
|
||||
/** Runtime arming only: the control file token the hit counted under. */
|
||||
token?: string;
|
||||
};
|
||||
|
||||
/** Thrown by a point armed with `armCrashPointForTest`. */
|
||||
@@ -79,11 +88,33 @@ type Armed = {
|
||||
receiptPath: string | null;
|
||||
/** Hits count only once this file exists; null when ungated. */
|
||||
gatePath: string | null;
|
||||
/** Runtime arming: the control file read at each hit; null for a fixed spec. */
|
||||
controlPath?: string | null;
|
||||
token?: string | null;
|
||||
};
|
||||
|
||||
/** Parses a control file line, "<token> <name>[:<nth>]"; null when empty or malformed. */
|
||||
export function parseCrashPointControl(text: string): { token: string; spec: CrashPointSpec } | null {
|
||||
const [token, raw, ...rest] = text.trim().split(/\s+/);
|
||||
if (!token || !raw || rest.length > 0) return null;
|
||||
try {
|
||||
const spec = parseCrashPointSpec(raw);
|
||||
return spec ? { token, spec } : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function armFromEnvironment(): Armed | null {
|
||||
const env = typeof process === "undefined" ? undefined : process.env;
|
||||
if (!env || env.NODE_ENV === "production") return null;
|
||||
const controlPath = env[CRASH_POINT_CONTROL_ENV]?.trim();
|
||||
if (controlPath) {
|
||||
process.stderr.write(`${CRASH_POINT_LOG_PREFIX} runtime arming from ${controlPath} pid=${process.pid}\n`);
|
||||
// The spec is a placeholder; each hit reads the real one from the control file.
|
||||
return { spec: { name: "" as CrashPointName, nth: 1 }, hits: 0, mode: "kill",
|
||||
receiptPath: env[CRASH_POINT_RECEIPT_ENV]?.trim() || null, gatePath: null, controlPath, token: null };
|
||||
}
|
||||
const spec = parseCrashPointSpec(env[CRASH_POINT_ENV]);
|
||||
if (!spec) return null;
|
||||
process.stderr.write(`${CRASH_POINT_LOG_PREFIX} armed ${spec.name}:${spec.nth} pid=${process.pid}\n`);
|
||||
@@ -103,10 +134,27 @@ let armed: Armed | null = armFromEnvironment();
|
||||
*/
|
||||
export function crashPoint(name: CrashPointName): void {
|
||||
if (armed === null) return;
|
||||
if (armed.controlPath) {
|
||||
if (!readControl(armed)) return;
|
||||
}
|
||||
if (armed.spec.name !== name) return;
|
||||
reach(armed);
|
||||
}
|
||||
|
||||
/** Runtime arming: loads the control file's spec; a new token resets the count. */
|
||||
function readControl(current: Armed): boolean {
|
||||
let text: string;
|
||||
try { text = readFileSync(current.controlPath!, "utf8"); } catch { return false; }
|
||||
const control = parseCrashPointControl(text);
|
||||
if (!control) return false;
|
||||
if (control.token !== current.token) {
|
||||
current.token = control.token;
|
||||
current.spec = control.spec;
|
||||
current.hits = 0;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function reach(current: Armed): void {
|
||||
if (current.gatePath !== null) {
|
||||
if (!existsSync(current.gatePath)) return;
|
||||
@@ -117,6 +165,7 @@ function reach(current: Armed): void {
|
||||
if (current.mode === "throw") throw new CrashPointReached(current.spec.name, current.hits);
|
||||
const receipt: CrashPointReceipt = {
|
||||
point: current.spec.name, hit: current.hits, pid: process.pid, at: new Date().toISOString(),
|
||||
...(current.token ? { token: current.token } : {}),
|
||||
};
|
||||
if (current.receiptPath) {
|
||||
try { appendFileSync(current.receiptPath, `${JSON.stringify(receipt)}\n`); } catch { /* the log line below still records it */ }
|
||||
@@ -142,5 +191,5 @@ export function armCrashPointForTest(spec: string | CrashPointSpec): () => void
|
||||
|
||||
/** The armed point and its hit count so far, or null. For tests and diagnostics. */
|
||||
export function armedCrashPoint(): { name: CrashPointName; nth: number; hits: number } | null {
|
||||
return armed ? { name: armed.spec.name, nth: armed.spec.nth, hits: armed.hits } : null;
|
||||
return armed && armed.spec.name ? { name: armed.spec.name, nth: armed.spec.nth, hits: armed.hits } : null;
|
||||
}
|
||||
@@ -655,6 +655,8 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
|
||||
"runner-protocol-live-evals.yml",
|
||||
"chat-catalog-testops.yml",
|
||||
"chat-catalog-testops-case.yml",
|
||||
// On-demand crash-point suite (durable ownership plan, Phase 5).
|
||||
"crash-points.yml",
|
||||
];
|
||||
const paidWorkflowNameSet = new Set(paidWorkflowNames);
|
||||
const providerSecretReference =
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
#!/usr/bin/env node
|
||||
// On-demand crash-point suite (durable ownership plan, Phase 5).
|
||||
// Usage: node scripts/crash-points.mjs [--list] [--points <glob,...>] [--kinds <id,...>]
|
||||
// [--adapters codex,claude,opencode] [--workloads A1,B1,B5,C1] [--nth N] [--env local|daytona]
|
||||
// See tests/restart-chaos/README.md, "Crash-point suite". Load provider keys
|
||||
// into this process only: set -a; source ~/.secrets; set +a
|
||||
import { spawnSync } from "node:child_process";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const result = spawnSync(process.execPath, [
|
||||
path.join(root, "cli/node_modules/tsx/dist/cli.mjs"),
|
||||
path.join(root, "tests/restart-chaos/crash-points.ts"),
|
||||
...process.argv.slice(2),
|
||||
], { cwd: root, stdio: "inherit" });
|
||||
process.exit(result.status ?? 2);
|
||||
@@ -206,3 +206,93 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
- The wave fingerprints the checkout and `packages/paperclip-runner/dist` at start and after every fault. A build, install or branch switch during a wave would make later restarts run different code, so it fails the wave as a harness error (`worktree_changed_mid_wave`). The baseline is taken after the harness's own setup (preflight rebuilds provider entrypoints) and covers the tracked diff, untracked file contents, the runner `dist` and the runnerd binary. Build before the wave, never during it.
|
||||
- `faults.json` and `task-faults.json` (each task's launch time and the faults that crossed it) are written after every fault, so `--replay` of a task in an interrupted or wedged wave, or a storm task, re-fires the faults that crossed it at the same offsets.
|
||||
- A fault that fails to complete fails the wave as a harness error (exit 2), even when recovery brings the server back.
|
||||
|
||||
## Crash-point suite (`scripts/crash-points.mjs`)
|
||||
|
||||
Plan: [durable ownership, "Testing: enumerate crash points"](../../doc/plans/2026-11-07-durable-ownership-and-reconciliation.md).
|
||||
Random waves find a 20 ms–1 s window between two durable writes by luck. The
|
||||
crash-point suite kills the control plane at each named window on purpose.
|
||||
|
||||
```sh
|
||||
set -a; source ~/.secrets; set +a # provider keys, process env only
|
||||
node scripts/crash-points.mjs --list # selected points, kinds, invariants
|
||||
node scripts/crash-points.mjs --points 'runner.*,native.*' --adapters codex --workloads A1
|
||||
node scripts/crash-points.mjs --kinds wake_request_claimed --nth 2
|
||||
```
|
||||
|
||||
Options: `--points <glob,...>` (`*`, `?`), `--kinds <ownable kind id,...>`
|
||||
(ids from `server/src/services/recovery/ownership-registry.ts`), `--adapters`
|
||||
(default `codex`), `--workloads` (default `A1`), `--nth` (fire at the nth hit,
|
||||
default 1), `--env local|daytona` (a point runs only in the environments its
|
||||
registry entry lists), plus `--daytona-image`, `--daytona-cpu`, `--budget-cents`,
|
||||
`--poll-ms`, `--lifecycle` and `--keep`, passed to each wave.
|
||||
|
||||
The suite runs one wave, `main.ts --crash-point <a>:<nth>,<b>:<nth>,...`:
|
||||
|
||||
1. One isolated control plane boots once, with `PAPERCLIP_CRASH_POINT_CONTROL`
|
||||
set to an empty control file (nothing armed) and without the Vite UI.
|
||||
2. For each point in turn: invariants are read as a baseline, the harness
|
||||
arms the point by writing `<token> <name>:<nth>` to the control file, and
|
||||
the point's batch launches (one task per adapter × workload). Creating an
|
||||
assigned issue already queues its wake, so the point is armed first.
|
||||
3. At the nth hit the server appends a receipt and SIGKILLs itself. The
|
||||
supervisor treats it as K1: it empties the control file, kills what is
|
||||
left of the server's process tree, starts a replacement, and waits for ready.
|
||||
4. When the batch rests, the point is disarmed. `invariants.ts` runs the
|
||||
Phase 1 report-only reconciler against the wave's database
|
||||
(`record: false`), and the point is charged only with violations that are
|
||||
new since its baseline:
|
||||
- two passes at the real clock for I1 (owned) and I2 (settled);
|
||||
- one pass with the clock moved 15 minutes ahead for I3 (admission), I4
|
||||
(exactly-once intents) and I7 (visible next action).
|
||||
Each violation carries its wake and run state as evidence. I5, I6 and I8
|
||||
are reported as not measured, with the reconciler's reasons; R3 and R1
|
||||
cover part of I5 and I8.
|
||||
5. Every task is graded R1–R6 and T at the end, as in any wave.
|
||||
|
||||
The first full local run (14 points, codex A1) took 11.8 minutes: one boot,
|
||||
then about 40–60 seconds per point. One point alone takes about 3 minutes,
|
||||
boot included.
|
||||
|
||||
Limits: the server kills itself without warning, so there is no pre-fault
|
||||
workspace snapshot and R2 checks workspace failures but not lost files
|
||||
(`notMeasured` in `crash-point.json`). `--replay` of a crash-point task fires
|
||||
the recorded K1 restart at its offset instead of re-arming the point; to
|
||||
reproduce a point, rerun the suite for it.
|
||||
|
||||
Results: `results/crash-points/<run>/SUMMARY.md` and `summary.json`, plus the
|
||||
wave directory with `crash-point.json` (per point: receipt, task grades,
|
||||
violations with evidence). Outcomes: `pass`, `fail` (a task failed R1–R6/T or an invariant was
|
||||
violated, or the wave exited 1 for a wave-level reason such as a Daytona
|
||||
sandbox left after cleanup), `not_reached` (the chosen workloads never crossed the point: pick a
|
||||
workload or `--nth` that does) and `harness_error`. Exit code 0, 1, 3 and 2.
|
||||
|
||||
**CI.** `.github/workflows/crash-points.yml` runs the same command on demand
|
||||
(`workflow_dispatch`, inputs `points`, `kinds`, `adapters`, `workloads`, `nth`).
|
||||
It is not a required PR check. It runs only from the default branch, for
|
||||
allowlisted actors, in the `runner-e2e-paid` environment, and uploads summaries
|
||||
and grades only (no server logs).
|
||||
|
||||
### Add a crash point
|
||||
|
||||
1. Add an entry to `packages/shared/src/crash-point-registry.ts`. Name it
|
||||
`<subsystem>.<record>.<before|after>_<step>` after the gap it marks, for
|
||||
example `sandbox.create.after_provider_before_lease`. Give a description of
|
||||
what is durable and what is not at that instant, the environments that reach
|
||||
it, the ownable kind ids it affects and the invariants at risk.
|
||||
2. Call `crashPoint("<name>")` at the gap: after the first durable write (or
|
||||
outside effect) returns and before the next one starts. Server code imports
|
||||
`crashPoint` from `@paperclipai/shared/crash-points`. Controller code in
|
||||
`packages/paperclip-runner` uses `src/crash-point.ts` (add the name to its
|
||||
union too). Do not put a point inside a transaction; the gap is between
|
||||
commits.
|
||||
3. `server/src/__tests__/crash-point-call-sites.test.ts` fails until every
|
||||
registered point has a call site and every kind id exists.
|
||||
4. Run the point: `node scripts/crash-points.mjs --points <name>`. A
|
||||
`not_reached` result means the workload never crosses it.
|
||||
|
||||
Disarmed, a point costs one comparison (about 4 ns), and the arming variables
|
||||
(`PAPERCLIP_CRASH_POINT`, `PAPERCLIP_CRASH_POINT_CONTROL`) are ignored under
|
||||
`NODE_ENV=production`, so points may sit on the path before `turn.start`.
|
||||
In-process tests use `armCrashPointForTest("<name>:<nth>")`, which throws
|
||||
`CrashPointReached` at the hit instead of killing the test runner.
|
||||
@@ -88,13 +88,14 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
"storm-minutes": { type: "string" },
|
||||
lifecycle: { type: "string" },
|
||||
"k4-hold-max-seconds": { type: "string" },
|
||||
"crash-point": { type: "string" },
|
||||
},
|
||||
});
|
||||
const keep = values.keep === true;
|
||||
const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000);
|
||||
const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000);
|
||||
if (values.replay !== undefined) {
|
||||
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "storm-minutes", "lifecycle", "k4-hold-max-seconds"] as const) {
|
||||
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point"] as const) {
|
||||
if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`);
|
||||
}
|
||||
return { mode: "replay", ref: parseReplayRef(values.replay), keep,
|
||||
@@ -116,12 +117,22 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
// pool that keeps `--concurrency` tasks in flight until the storm ends.
|
||||
const stormMinutes = values["storm-minutes"] === undefined ? undefined : integer(values["storm-minutes"], "storm-minutes", 60, 1, 240);
|
||||
const lifecycle = values.lifecycle;
|
||||
const crashPoint = values["crash-point"]?.trim();
|
||||
if (crashPoint !== undefined) {
|
||||
if (!/^[a-z0-9_.]+(?::[1-9][0-9]*)?(?:,[a-z0-9_.]+(?::[1-9][0-9]*)?)*$/.test(crashPoint)) {
|
||||
throw new Error("--crash-point expects <name>[:<nth>][,<name>[:<nth>]...]");
|
||||
}
|
||||
for (const key of ["faults", "triggers", "storm-minutes"] as const) {
|
||||
if (values[key] !== undefined) throw new Error(`--crash-point replaces scheduled faults; do not combine it with --${key}`);
|
||||
}
|
||||
}
|
||||
if (lifecycle !== undefined && lifecycle !== "per_turn" && lifecycle !== "warm" && lifecycle !== "default") throw new Error("--lifecycle must be per_turn, warm or default");
|
||||
const options: WaveOptions = {
|
||||
env,
|
||||
adapters: list<AdapterId>(values.adapters, ADAPTER_IDS, "adapters", ["codex", "claude"]),
|
||||
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices),
|
||||
faults: list<FaultKind>(values.faults, FAULT_KINDS, "faults", stormMinutes ? FAULT_KINDS : ["K1", "K2", "K4"]),
|
||||
// A crash-point wave schedules no fault; K1 is the restart its self-kill gets.
|
||||
faults: list<FaultKind>(values.faults, FAULT_KINDS, "faults", stormMinutes ? FAULT_KINDS : crashPoint ? ["K1"] : ["K1", "K2", "K4"]),
|
||||
triggers: list<TriggerId>(values.triggers, TRIGGER_IDS, "triggers", ["time"]),
|
||||
concurrency: integer(values.concurrency, "concurrency", 4, 1, 20),
|
||||
tasks: integer(values.tasks, "tasks", stormMinutes ? 200 : 8, 1, 200),
|
||||
@@ -135,6 +146,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
// Seeded K4 hold upper end; the controller-outage cap is 600 s (#19797).
|
||||
...(values["k4-hold-max-seconds"] !== undefined
|
||||
? { k4HoldMaxMs: integer(values["k4-hold-max-seconds"], "k4-hold-max-seconds", 300, 120, 600) * 1000 } : {}),
|
||||
...(crashPoint ? { crashPoint } : {}),
|
||||
};
|
||||
if (env !== "daytona" && values["daytona-cpu"] !== undefined) throw new Error("--daytona-cpu applies only to --env daytona");
|
||||
if (options.daytonaCpu !== undefined) {
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { CrashPointDefinition, CrashPointName } from "../../packages/shared/src/crash-point-registry.js";
|
||||
import { classifyPoint, globToRegExp, parseSuiteArgs, renderSummary, selectPoints, type CrashPointEntry } from "./crash-points.js";
|
||||
import { parseCommand } from "./config.js";
|
||||
import { readCrashPointReceipt } from "./supervisor.js";
|
||||
|
||||
const definition = (kinds: string[], environments: Array<"local" | "daytona"> = ["local", "daytona"]): CrashPointDefinition => ({
|
||||
description: "test point description long enough", kinds, invariants: ["I2"], environments,
|
||||
});
|
||||
const registry: Record<string, CrashPointDefinition> = {
|
||||
"runner.startup.after_spawn": definition(["native_startup_context"]),
|
||||
"sandbox.create.after_provider_before_lease": definition(["sandbox_creation_intent"], ["daytona"]),
|
||||
"wake.request.after_claim_before_run": definition(["wake_request_claimed"]),
|
||||
};
|
||||
const point = "runner.startup.after_spawn" as CrashPointName;
|
||||
|
||||
describe("selectPoints", () => {
|
||||
it("filters by glob, kind and environment", () => {
|
||||
expect(selectPoints(registry, { points: [], kinds: [], env: "local" }))
|
||||
.toEqual(["runner.startup.after_spawn", "wake.request.after_claim_before_run"]);
|
||||
expect(selectPoints(registry, { points: ["runner.*"], kinds: [], env: "local" })).toEqual(["runner.startup.after_spawn"]);
|
||||
expect(selectPoints(registry, { points: [], kinds: ["sandbox_creation_intent"], env: "daytona" }))
|
||||
.toEqual(["sandbox.create.after_provider_before_lease"]);
|
||||
expect(selectPoints(registry, { points: [], kinds: ["sandbox_creation_intent"], env: "local" })).toEqual([]);
|
||||
});
|
||||
|
||||
it("matches globs literally apart from * and ?", () => {
|
||||
expect(globToRegExp("runner.*").test("runnerXstartup")).toBe(false);
|
||||
expect(globToRegExp("*.after_?pawn").test("runner.startup.after_spawn")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseSuiteArgs", () => {
|
||||
it("defaults to one local codex A1 task per point", () => {
|
||||
expect(parseSuiteArgs([])).toMatchObject({ env: "local", adapters: ["codex"], workloads: ["A1"], nth: 1, list: false, passthrough: [] });
|
||||
});
|
||||
|
||||
it("passes harness options through and refuses unknown values", () => {
|
||||
expect(parseSuiteArgs(["--", "--keep", "--budget-cents", "500"]).passthrough).toEqual(["--budget-cents", "500", "--keep"]);
|
||||
expect(() => parseSuiteArgs(["--adapters", "gpt"])).toThrow(/Unknown adapter/);
|
||||
expect(() => parseSuiteArgs(["--nth", "0"])).toThrow(/positive integer/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("classifyPoint", () => {
|
||||
const entry = (overrides: Partial<CrashPointEntry> = {}): CrashPointEntry => ({
|
||||
point: `${point}:1`, reached: { name: point, hit: 1, pid: 42 }, armedAt: 0, restedAt: 120_000,
|
||||
tasks: [{ taskId: "t01", adapter: "codex", workload: "A1", pass: true, signature: null, rules: { R1: "pass" } }],
|
||||
invariants: { violations: [], byInvariant: {}, errors: [] },
|
||||
...overrides,
|
||||
});
|
||||
|
||||
it("passes a reached point with no failed task or new violation, timed from arming to rest", () => {
|
||||
expect(classifyPoint(point, 0, entry())).toMatchObject({ outcome: "pass", durationMs: 120_000 });
|
||||
});
|
||||
|
||||
it("fails on a new invariant violation or a failed task, even when not reached", () => {
|
||||
const violation = { rule: "I2_settled", kind: "native_executor_claim", reason: "owner_run_terminal", recordId: "r1" };
|
||||
expect(classifyPoint(point, 1, entry({ invariants: { violations: [violation], byInvariant: { I2: 1 }, errors: [] } })))
|
||||
.toMatchObject({ outcome: "fail", failedInvariants: { I2: 1 } });
|
||||
expect(classifyPoint(point, 1, entry({ reached: null,
|
||||
tasks: [{ taskId: "t01", adapter: "codex", workload: "A1", pass: false, signature: "x", rules: { R1: "fail" } }] })).outcome).toBe("fail");
|
||||
});
|
||||
|
||||
it("separates unreached, unrun and unchecked points", () => {
|
||||
expect(classifyPoint(point, 0, entry({ reached: null })).outcome).toBe("not_reached");
|
||||
expect(classifyPoint(point, 2, null)).toMatchObject({ outcome: "harness_error", error: "not run (wave exited 2)" });
|
||||
expect(classifyPoint(point, 0, entry({ invariants: null })).outcome).toBe("harness_error");
|
||||
expect(classifyPoint(point, 0, entry({ invariants: { violations: [], byInvariant: {}, errors: ["I3 pass did not finish"] } })).outcome)
|
||||
.toBe("harness_error");
|
||||
});
|
||||
|
||||
it("renders a summary row per point", () => {
|
||||
const result = { ...classifyPoint(point, 0, entry()), waveDir: null };
|
||||
expect(renderSummary("cp-x", parseSuiteArgs([]), [result])).toContain("| `runner.startup.after_spawn` | pass | — | — | 2.0 |");
|
||||
});
|
||||
});
|
||||
|
||||
describe("readCrashPointReceipt", () => {
|
||||
it("returns the receipt line of the given pid only", () => {
|
||||
const dir = mkdtempSync(path.join(os.tmpdir(), "crash-receipt-"));
|
||||
try {
|
||||
const file = path.join(dir, "receipts.jsonl");
|
||||
writeFileSync(file, `${JSON.stringify({ point, hit: 1, pid: 7 })}\n{"partial`);
|
||||
expect(readCrashPointReceipt(file, 7)).toEqual({ name: point, hit: 1, pid: 7 });
|
||||
expect(readCrashPointReceipt(file, 8)).toBeNull();
|
||||
expect(readCrashPointReceipt(path.join(dir, "missing"), 7)).toBeNull();
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("main.ts --crash-point", () => {
|
||||
it("plans a K1-only wave with the point and refuses scheduled-fault options", () => {
|
||||
const parsed = parseCommand(["--crash-point", `${point}:2`, "--tasks", "1", "--seed", "1"]);
|
||||
expect(parsed).toMatchObject({ mode: "wave", options: { crashPoint: `${point}:2`, faults: ["K1"] } });
|
||||
expect(() => parseCommand(["--crash-point", point, "--faults", "K2"])).toThrow(/replaces scheduled faults/);
|
||||
expect(() => parseCommand(["--crash-point", "Bad Name"])).toThrow(/<name>\[:<nth>\]/);
|
||||
expect(parseCommand(["--crash-point", `${point}:1,${point}:2`, "--tasks", "2"]))
|
||||
.toMatchObject({ options: { crashPoint: `${point}:1,${point}:2` } });
|
||||
expect(() => parseCommand(["--replay", "w-20261107T000000-s1/t01", "--crash-point", point])).toThrow(/--replay/);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,211 @@
|
||||
/**
|
||||
* On-demand crash-point suite (durable ownership plan, Phase 5).
|
||||
*
|
||||
* pnpm crash-points --list
|
||||
* pnpm crash-points --points 'runner.*' --adapters codex
|
||||
* pnpm crash-points --kinds sandbox_creation_intent --env daytona --daytona-image <image@sha256:…>
|
||||
*
|
||||
* It runs one restart-chaos wave (main.ts `--crash-point a,b,...`): one
|
||||
* isolated control plane boots once, then each selected point in turn gets a
|
||||
* batch of tasks (one per adapter × workload), is armed at runtime, SIGKILLs
|
||||
* the server at its nth hit, and the server restarts like K1. When the batch
|
||||
* rests, plan invariants I1–I8 from the Phase 1 report-only reconciler
|
||||
* (invariants.ts) are checked as violations new since the point was armed.
|
||||
* Every task is graded R1–R6 at the end.
|
||||
* Results: tests/restart-chaos/results/crash-points/<run>/SUMMARY.md.
|
||||
*
|
||||
* Exit: 0 every reached point passed, 1 a point failed, 2 a harness error,
|
||||
* 3 a selected point was never reached by the chosen workloads.
|
||||
*/
|
||||
import { spawn } from "node:child_process";
|
||||
import { existsSync } from "node:fs";
|
||||
import { mkdir, readdir, readFile, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { CRASH_POINTS, type CrashPointDefinition, type CrashPointName } from "../../packages/shared/src/crash-point-registry.js";
|
||||
import { knownSecretValues, redactKnownSecrets } from "./evidence.js";
|
||||
import { ADAPTER_IDS, WORKLOAD_IDS, type AdapterId, type WorkloadId } from "./types.js";
|
||||
|
||||
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
|
||||
const resultsRoot = path.resolve(process.env.RESTART_CHAOS_RESULTS_DIR ?? path.join(import.meta.dirname, "results"));
|
||||
|
||||
export type PointOutcome = "pass" | "fail" | "not_reached" | "harness_error";
|
||||
|
||||
export interface PointResult {
|
||||
point: CrashPointName;
|
||||
outcome: PointOutcome;
|
||||
exitCode: number | null;
|
||||
waveDir: string | null;
|
||||
reached: { name: string; hit: number; pid: number } | null;
|
||||
failedInvariants: Record<string, number>;
|
||||
failedTasks: Array<{ taskId: string; adapter: string; workload: string; signature: string | null; rules: Record<string, string> }>;
|
||||
violations: Array<{ rule: string; kind: string; reason: string; recordId: string; evidence?: Record<string, unknown> }>;
|
||||
durationMs: number;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export interface SuiteOptions {
|
||||
points: string[];
|
||||
kinds: string[];
|
||||
adapters: AdapterId[];
|
||||
workloads: WorkloadId[];
|
||||
env: "local" | "daytona";
|
||||
nth: number;
|
||||
seed: number;
|
||||
list: boolean;
|
||||
passthrough: string[];
|
||||
}
|
||||
|
||||
export function globToRegExp(glob: string): RegExp {
|
||||
const escaped = glob.trim().replace(/[.+^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*").replace(/\?/g, ".");
|
||||
return new RegExp(`^${escaped}$`);
|
||||
}
|
||||
|
||||
/** Points matching any `--points` glob and any `--kinds` id, reachable in `env`. */
|
||||
export function selectPoints(registry: Record<string, CrashPointDefinition>, options: Pick<SuiteOptions, "points" | "kinds" | "env">): CrashPointName[] {
|
||||
const globs = options.points.length ? options.points.map(globToRegExp) : [/.*/];
|
||||
return (Object.keys(registry) as CrashPointName[]).filter(name => {
|
||||
const definition = registry[name]!;
|
||||
if (!globs.some(glob => glob.test(name))) return false;
|
||||
if (options.kinds.length && !definition.kinds.some(kind => options.kinds.includes(kind))) return false;
|
||||
return definition.environments.includes(options.env);
|
||||
});
|
||||
}
|
||||
|
||||
function csv(value: string | undefined): string[] {
|
||||
return value ? value.split(",").map(part => part.trim()).filter(Boolean) : [];
|
||||
}
|
||||
|
||||
export function parseSuiteArgs(argv: readonly string[]): SuiteOptions {
|
||||
const args = argv[0] === "--" ? argv.slice(1) : [...argv];
|
||||
const { values } = parseArgs({ args, strict: true, options: {
|
||||
points: { type: "string" }, kinds: { type: "string" }, adapters: { type: "string" }, workloads: { type: "string" },
|
||||
env: { type: "string" }, nth: { type: "string" }, seed: { type: "string" }, list: { type: "boolean" },
|
||||
"daytona-image": { type: "string" }, "daytona-cpu": { type: "string" }, "budget-cents": { type: "string" },
|
||||
"poll-ms": { type: "string" }, keep: { type: "boolean" }, lifecycle: { type: "string" },
|
||||
} });
|
||||
const env = values.env ?? "local";
|
||||
if (env !== "local" && env !== "daytona") throw new Error("--env must be local or daytona");
|
||||
const adapters = csv(values.adapters ?? "codex") as AdapterId[];
|
||||
for (const adapter of adapters) if (!ADAPTER_IDS.includes(adapter)) throw new Error(`Unknown adapter "${adapter}"`);
|
||||
const workloads = csv(values.workloads ?? "A1") as WorkloadId[];
|
||||
for (const workload of workloads) if (!WORKLOAD_IDS.includes(workload)) throw new Error(`Unknown workload "${workload}"`);
|
||||
const nth = Number(values.nth ?? "1");
|
||||
if (!Number.isInteger(nth) || nth < 1) throw new Error("--nth must be a positive integer");
|
||||
const seed = Number(values.seed ?? "1");
|
||||
if (!Number.isInteger(seed) || seed < 0) throw new Error("--seed must be a non-negative integer");
|
||||
const passthrough: string[] = [];
|
||||
for (const key of ["daytona-image", "daytona-cpu", "budget-cents", "poll-ms", "lifecycle"] as const) {
|
||||
if (values[key] !== undefined) passthrough.push(`--${key}`, values[key]!);
|
||||
}
|
||||
if (values.keep) passthrough.push("--keep");
|
||||
return { points: csv(values.points), kinds: csv(values.kinds), adapters, workloads, env, nth, seed, list: values.list === true, passthrough };
|
||||
}
|
||||
|
||||
/** One point's entry in crash-point.json (schema v2). */
|
||||
export type CrashPointEntry = {
|
||||
point: string;
|
||||
reached: { name: string; hit: number; pid: number } | null;
|
||||
armedAt: number;
|
||||
restedAt: number | null;
|
||||
tasks: Array<{ taskId: string; adapter: string; workload: string; pass: boolean; signature: string | null; rules: Record<string, string> }>;
|
||||
invariants: { violations: Array<{ rule: string; kind: string; reason: string; recordId: string; evidence?: Record<string, unknown> }>; byInvariant: Record<string, number>; errors: string[] } | null;
|
||||
};
|
||||
|
||||
/** Classifies one point from its entry; a point the wave never reached has none. */
|
||||
export function classifyPoint(point: CrashPointName, exitCode: number | null, entry: CrashPointEntry | null): Omit<PointResult, "waveDir"> {
|
||||
const base = { point, exitCode, reached: entry?.reached ?? null, failedInvariants: entry?.invariants?.byInvariant ?? {},
|
||||
failedTasks: (entry?.tasks ?? []).filter(task => !task.pass).map(({ pass: _pass, ...task }) => task),
|
||||
violations: (entry?.invariants?.violations ?? []).map(({ rule, kind, reason, recordId, evidence }) =>
|
||||
({ rule, kind, reason, recordId, ...(evidence ? { evidence } : {}) })),
|
||||
durationMs: entry?.restedAt ? entry.restedAt - entry.armedAt : 0 };
|
||||
if (!entry) return { ...base, outcome: "harness_error", error: `not run (wave exited ${exitCode})` };
|
||||
if (base.failedTasks.length > 0 || base.violations.length > 0) return { ...base, outcome: "fail" };
|
||||
if (entry.invariants?.errors.length) return { ...base, outcome: "harness_error", error: entry.invariants.errors.join("; ") };
|
||||
if (!entry.invariants) return { ...base, outcome: "harness_error", error: "no invariant check" };
|
||||
if (!entry.reached) return { ...base, outcome: "not_reached" };
|
||||
return { ...base, outcome: "pass" };
|
||||
}
|
||||
|
||||
/** One wave for every selected point: one boot, one point at a time. */
|
||||
async function runSuite(points: readonly CrashPointName[], options: SuiteOptions, runDir: string): Promise<{ results: PointResult[]; exitCode: number | null; waveDir: string | null }> {
|
||||
const cells = options.adapters.length * options.workloads.length;
|
||||
const args = [path.join(repositoryRoot, "cli/node_modules/tsx/dist/cli.mjs"), path.join(import.meta.dirname, "main.ts"),
|
||||
"--env", options.env, "--adapters", options.adapters.join(","), "--workloads", options.workloads.join(","),
|
||||
"--tasks", String(cells * points.length), "--concurrency", String(cells), "--seed", String(options.seed),
|
||||
"--crash-point", points.map(point => `${point}:${options.nth}`).join(","), ...options.passthrough];
|
||||
console.info(`\n[crash-points] ${points.length} point(s) × ${cells} task(s), results ${runDir}`);
|
||||
const exitCode = await new Promise<number | null>(resolve => {
|
||||
const child = spawn(process.execPath, args, { cwd: repositoryRoot, stdio: "inherit",
|
||||
env: { ...process.env, RESTART_CHAOS_RESULTS_DIR: runDir,
|
||||
// The wave boots a fresh instance (all migrations); allow a loaded host.
|
||||
RESTART_CHAOS_HEALTH_TIMEOUT_MS: process.env.RESTART_CHAOS_HEALTH_TIMEOUT_MS ?? "600000" } });
|
||||
child.once("exit", code => resolve(code));
|
||||
child.once("error", () => resolve(null));
|
||||
});
|
||||
const waves = (await readdir(runDir, { withFileTypes: true }).catch(() => []))
|
||||
.filter(entry => entry.isDirectory() && entry.name.startsWith("w-")).map(entry => entry.name).sort();
|
||||
const waveDir = waves.length ? path.join(runDir, waves[waves.length - 1]!) : null;
|
||||
const filePath = waveDir ? path.join(waveDir, "crash-point.json") : null;
|
||||
const file = filePath && existsSync(filePath) ? JSON.parse(await readFile(filePath, "utf8")) as { points?: CrashPointEntry[] } : null;
|
||||
const entries = new Map((file?.points ?? []).map(entry => [entry.point.replace(/:\d+$/, ""), entry]));
|
||||
return { results: points.map(point => ({ ...classifyPoint(point, exitCode, entries.get(point) ?? null), waveDir })), exitCode, waveDir };
|
||||
}
|
||||
|
||||
export function renderSummary(runId: string, options: SuiteOptions, results: readonly PointResult[]): string {
|
||||
const lines = [`# Crash-point suite ${runId}`, "",
|
||||
`env ${options.env}, adapters ${options.adapters.join(",")}, workloads ${options.workloads.join(",")}, nth ${options.nth}, seed ${options.seed}`, "",
|
||||
"| Point | Outcome | Invariants violated | Failed tasks (rules) | Minutes (armed → rest) |", "|---|---|---|---|---|"];
|
||||
for (const result of results) {
|
||||
const invariants = Object.entries(result.failedInvariants).map(([id, count]) => `${id}×${count}`).join(", ") || "—";
|
||||
const tasks = result.failedTasks.map(task => `${task.taskId} ${task.adapter}/${task.workload} `
|
||||
+ `(${Object.entries(task.rules).filter(([, verdict]) => verdict === "fail").map(([id]) => id).join(",") || task.signature || "T"})`).join("; ") || "—";
|
||||
lines.push(`| \`${result.point}\` | ${result.outcome}${result.error ? ` (${result.error})` : ""} | ${invariants} | ${tasks} | ${(result.durationMs / 60_000).toFixed(1)} |`);
|
||||
}
|
||||
const violations = results.flatMap(result => result.violations.map(violation => ({ point: result.point, ...violation })));
|
||||
if (violations.length) {
|
||||
lines.push("", "## Violations", "", "| Point | Rule | Kind | Reason | Record | Evidence |", "|---|---|---|---|---|---|");
|
||||
for (const violation of violations) {
|
||||
const evidence = violation.evidence ? `\`${JSON.stringify(violation.evidence).replace(/\|/g, "\\|")}\`` : "—";
|
||||
lines.push(`| \`${violation.point}\` | ${violation.rule} | ${violation.kind} | ${violation.reason} | ${violation.recordId} | ${evidence} |`);
|
||||
}
|
||||
}
|
||||
return `${lines.join("\n")}\n`;
|
||||
}
|
||||
|
||||
async function main(): Promise<number> {
|
||||
const options = parseSuiteArgs(process.argv.slice(2));
|
||||
const selected = selectPoints(CRASH_POINTS, options);
|
||||
if (options.list) {
|
||||
for (const name of selected) {
|
||||
const definition = CRASH_POINTS[name] as CrashPointDefinition;
|
||||
console.info(`${name}\n ${definition.description}\n kinds ${definition.kinds.join(", ") || "—"}; invariants ${definition.invariants.join(", ")}; env ${definition.environments.join(", ")}`);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
if (!selected.length) throw new Error(`No crash point matches --points ${options.points.join(",") || "*"} --kinds ${options.kinds.join(",") || "*"} --env ${options.env}`);
|
||||
const runId = `cp-${new Date().toISOString().replace(/[-:]/g, "").replace(/\.\d{3}Z$/, "")}`;
|
||||
const runDir = path.join(resultsRoot, "crash-points", runId);
|
||||
await mkdir(runDir, { recursive: true, mode: 0o700 });
|
||||
const startedAt = Date.now();
|
||||
const { results, exitCode } = await runSuite(selected, options, runDir);
|
||||
const summary = renderSummary(runId, options, results)
|
||||
+ `\nWave exit ${exitCode}; ${((Date.now() - startedAt) / 60_000).toFixed(1)} min in all (one boot, then one point at a time).\n`;
|
||||
// Violation evidence carries run and wake error text; CI uploads both files.
|
||||
const secrets = knownSecretValues();
|
||||
await writeFile(path.join(runDir, "SUMMARY.md"), redactKnownSecrets(summary, secrets), { mode: 0o600 });
|
||||
await writeFile(path.join(runDir, "summary.json"),
|
||||
`${redactKnownSecrets(JSON.stringify({ runId, options, exitCode, results }, null, 2), secrets)}\n`, { mode: 0o600 });
|
||||
console.info(`\n${redactKnownSecrets(summary, secrets)}\nResults: ${runDir}`);
|
||||
// Exit 1 with every point clean is a wave-level failure (for example a Daytona leak); see the wave timeline.
|
||||
if (exitCode === 2 || exitCode === null) return 2;
|
||||
if (results.some(result => result.outcome === "harness_error")) return 2;
|
||||
if (results.some(result => result.outcome === "fail") || exitCode === 1) return 1;
|
||||
if (results.some(result => result.outcome === "not_reached")) return 3;
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
|
||||
process.exit(await main().catch(error => { console.error(error); return 2; }));
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
/**
|
||||
* Plan invariants I1–I8 for one wave (durable ownership plan, "Invariants").
|
||||
*
|
||||
* Runs the Phase 1 report-only ownership reconciler against the wave's own
|
||||
* database after every task rested. Nothing is written: `record: false`.
|
||||
*
|
||||
* I1, I2 two passes at the real clock: a record with no owner stamp (I1),
|
||||
* or whose owner was proven gone on both passes and is still in
|
||||
* progress (I2: "settled within one reconciler pass").
|
||||
* I3, I4, I7 one more pass with the clock moved past every grace window
|
||||
* (accounting closure 10 min, next action 5 min). The tasks are at
|
||||
* rest, so anything still waiting then would wait forever.
|
||||
* I5, I6, I8 not measurable by the reconciler yet; reported as coverage
|
||||
* gaps with its own reasons. The chaos graders cover part of them:
|
||||
* R3 (one Retry at most) bounds I5 and R1 (finished in budget) I8.
|
||||
*/
|
||||
import { createDb, type Db } from "../../packages/db/src/index.js";
|
||||
import {
|
||||
createOwnershipReportState,
|
||||
runOwnershipReconciliationPass,
|
||||
type InvariantViolation,
|
||||
} from "../../server/src/services/recovery/ownership-invariant-report.js";
|
||||
|
||||
export type PlanInvariant = "I1" | "I2" | "I3" | "I4" | "I5" | "I6" | "I7" | "I8";
|
||||
|
||||
/** A violation with the state of its record and run, so a failure is diagnosable after the instance is gone. */
|
||||
export type EvidencedViolation = InvariantViolation & { evidence?: Record<string, unknown> };
|
||||
|
||||
export interface InvariantCheck {
|
||||
violations: EvidencedViolation[];
|
||||
byInvariant: Partial<Record<PlanInvariant, number>>;
|
||||
inProgress: Record<string, { inProgress: number; gone: number; alive: number; unprovable: number }>;
|
||||
notMeasured: Partial<Record<PlanInvariant, string>>;
|
||||
errors: string[];
|
||||
}
|
||||
|
||||
/** Grace windows the shifted pass must exceed (ownership-invariant-report.ts). */
|
||||
export const GRACE_SHIFT_MS = 15 * 60_000;
|
||||
|
||||
export const invariantOf = (violation: Pick<InvariantViolation, "rule">): PlanInvariant => violation.rule.slice(0, 2) as PlanInvariant;
|
||||
|
||||
export function summarizeViolations(violations: readonly InvariantViolation[]): Partial<Record<PlanInvariant, number>> {
|
||||
const counts: Partial<Record<PlanInvariant, number>> = {};
|
||||
for (const violation of violations) counts[invariantOf(violation)] = (counts[invariantOf(violation)] ?? 0) + 1;
|
||||
return counts;
|
||||
}
|
||||
|
||||
/**
|
||||
* Local-disk kinds (usage-receipt and orphan-cleanup spools) resolve their
|
||||
* directories from the environment when listed. Point them at the wave's
|
||||
* instance, not the operator's, for the duration of the check.
|
||||
*/
|
||||
async function withWaveInstance<T>(instance: { paperclipHome: string; instanceId: string }, run: () => Promise<T>): Promise<T> {
|
||||
const keys = ["PAPERCLIP_HOME", "PAPERCLIP_INSTANCE_ID", "SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR"] as const;
|
||||
const saved = Object.fromEntries(keys.map(key => [key, process.env[key]]));
|
||||
process.env.PAPERCLIP_HOME = instance.paperclipHome;
|
||||
process.env.PAPERCLIP_INSTANCE_ID = instance.instanceId;
|
||||
delete process.env.SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR;
|
||||
try { return await run(); } finally {
|
||||
for (const key of keys) {
|
||||
if (saved[key] === undefined) delete process.env[key]; else process.env[key] = saved[key];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const clip = (value: unknown) => typeof value === "string" ? value.slice(0, 300) : value ?? null;
|
||||
|
||||
async function withEvidence(db: Db, violations: InvariantViolation[]): Promise<EvidencedViolation[]> {
|
||||
if (!violations.length) return violations;
|
||||
const wakeIds = violations.filter(violation => violation.kind === "wake_request" || violation.kind === "wake_request_claimed")
|
||||
.map(violation => violation.recordId);
|
||||
const runIds = [...new Set(violations.flatMap(violation => violation.runId ? [violation.runId] : []))];
|
||||
type Row = Record<string, unknown>;
|
||||
const wakes = (wakeIds.length ? await db.$client.unsafe(
|
||||
"select id::text, status, reason, error, run_id::text as \"runId\" from agent_wakeup_requests where id::text = any($1)", [wakeIds]) : []) as Row[];
|
||||
const runs = (runIds.length ? await db.$client.unsafe(
|
||||
"select id::text, status, error_code as \"errorCode\", error from heartbeat_runs where id::text = any($1)", [runIds]) : []) as Row[];
|
||||
const wakeById = new Map(wakes.map(wake => [wake.id, wake]));
|
||||
const runById = new Map(runs.map(run => [run.id, run]));
|
||||
return violations.map(violation => {
|
||||
const wake = wakeById.get(violation.recordId);
|
||||
const run = violation.runId ? runById.get(violation.runId) : undefined;
|
||||
const evidence = {
|
||||
...(wake ? { wake: { status: wake.status, reason: clip(wake.reason), error: clip(wake.error), runId: wake.runId } } : {}),
|
||||
...(run ? { run: { status: run.status, errorCode: run.errorCode, error: clip(run.error) } } : {}),
|
||||
};
|
||||
return Object.keys(evidence).length ? { ...violation, evidence } : violation;
|
||||
});
|
||||
}
|
||||
|
||||
export async function checkPlanInvariants(input: {
|
||||
connectionString: string; companyId: string | null; paperclipHome: string; instanceId: string; now?: Date;
|
||||
}): Promise<InvariantCheck> {
|
||||
const db = createDb(input.connectionString);
|
||||
const errors: string[] = [];
|
||||
try {
|
||||
const now = input.now ?? new Date();
|
||||
const pass = (state: ReturnType<typeof createOwnershipReportState>, at: Date) =>
|
||||
withWaveInstance(input, () => runOwnershipReconciliationPass(db, { state, now: at, record: false }));
|
||||
const state = createOwnershipReportState();
|
||||
// I2 needs the same record proven gone on both passes: a kind that failed
|
||||
// to list on the first pass has only one proof, so report it as incomplete.
|
||||
const first = await pass(state, now);
|
||||
for (const kind of first.kinds) if (kind.error) errors.push(`${kind.kind} (first I1/I2 pass): ${kind.error}`);
|
||||
if (first.retryNeeded) errors.push("the first I1/I2 pass did not finish (retryNeeded)");
|
||||
const second = await pass(state, new Date(now.getTime() + 1_000));
|
||||
// A failed I3/I4/I7 query is only logged by the pass and sets retryNeeded.
|
||||
// Retry once; a pass that still could not finish is an incomplete check.
|
||||
let shifted = await pass(createOwnershipReportState(), new Date(now.getTime() + GRACE_SHIFT_MS));
|
||||
if (shifted.retryNeeded) shifted = await pass(createOwnershipReportState(), new Date(now.getTime() + GRACE_SHIFT_MS));
|
||||
for (const kind of [...second.kinds, ...shifted.kinds]) if (kind.error) errors.push(`${kind.kind}: ${kind.error}`);
|
||||
if (second.retryNeeded) errors.push("the I1/I2 pass did not finish (retryNeeded)");
|
||||
if (shifted.retryNeeded) errors.push("the I3/I4/I7 pass did not finish after one retry (retryNeeded)");
|
||||
const ownCompany = (violation: InvariantViolation) => input.companyId === null || violation.companyId === null
|
||||
|| violation.companyId === input.companyId;
|
||||
const fromClock = second.violations.filter(violation => violation.rule === "I1_owned" || violation.rule === "I2_settled");
|
||||
const fromShift = shifted.violations.filter(violation => violation.rule !== "I1_owned" && violation.rule !== "I2_settled");
|
||||
const violations = await withEvidence(db, [...fromClock, ...fromShift].filter(ownCompany));
|
||||
const inProgress = Object.fromEntries(second.kinds.filter(kind => kind.inProgress > 0)
|
||||
.map(kind => [kind.kind, { inProgress: kind.inProgress, ...kind.verdicts }]));
|
||||
return {
|
||||
violations, byInvariant: summarizeViolations(violations), inProgress, errors,
|
||||
notMeasured: {
|
||||
I5: second.coverageGaps.I5_work_based_budgets ?? "not measured",
|
||||
I6: second.coverageGaps.I6_generation_fencing ?? "not measured",
|
||||
I8: second.coverageGaps.I8_fast_turns ?? "not measured",
|
||||
},
|
||||
};
|
||||
} finally {
|
||||
await db.$client.end({ timeout: 5 }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
@@ -32,6 +32,8 @@ export interface LedgerLine {
|
||||
recordedAt: string;
|
||||
/** Known signatures masked for this wave, e.g. ["rc06"]. */
|
||||
masks?: string[];
|
||||
/** Crash-point suite: the named point this wave armed. */
|
||||
crashPoint?: string;
|
||||
}
|
||||
|
||||
export function ledgerLine(plan: WavePlan, record: TaskRecord, grade: TaskGrade, gitSha: string, evidence: string | null,
|
||||
@@ -49,6 +51,7 @@ export function ledgerLine(plan: WavePlan, record: TaskRecord, grade: TaskGrade,
|
||||
signature: grade.signature, retried: Boolean(record.retry), evidence, replay: replayCommand(plan.waveId, record.plan.taskId),
|
||||
recordedAt: new Date().toISOString(),
|
||||
...(plan.options.masks?.length ? { masks: [...plan.options.masks] } : {}),
|
||||
...(plan.options.crashPoint ? { crashPoint: plan.options.crashPoint } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+112
-4
@@ -7,7 +7,7 @@
|
||||
* node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay <wave>/<task>
|
||||
*/
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import { existsSync, writeFileSync } from "node:fs";
|
||||
import { existsSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
@@ -17,8 +17,9 @@ import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from "
|
||||
import { assertControllerMatchesImage, prepareDaytonaImageArtifacts } from "./daytona-image.js";
|
||||
import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js";
|
||||
import { DaytonaWaveSweeper, leaksForTask, unattributedLeaks, type DaytonaSweepResult } from "./daytona-sweep.js";
|
||||
import { knownSecretValues, writeEvidenceBundle } from "./evidence.js";
|
||||
import { knownSecretValues, redactKnownSecrets, writeEvidenceBundle } from "./evidence.js";
|
||||
import { gradeTask } from "./grader.js";
|
||||
import { checkPlanInvariants, summarizeViolations, type InvariantCheck } from "./invariants.js";
|
||||
import { appendLedger, ledgerLine, writeSummary, type LedgerLine } from "./ledger.js";
|
||||
import { TaskObserver } from "./observer.js";
|
||||
import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js";
|
||||
@@ -135,11 +136,33 @@ async function main(): Promise<number> {
|
||||
tasks: plan.tasks.map(task => `${task.taskId}:${task.adapter}/${task.workload}/${task.fault}${task.trigger ? `^${task.trigger}+${task.triggerDelayMs ?? 0}ms` : ""}@${task.fireAfterMs}ms${task.holdMs ? ` hold ${task.holdMs}ms` : ""}`),
|
||||
...(plan.storm ? { storm: { durationMs: plan.storm.durationMs, faults: plan.storm.faults.map(fault => `${fault.kind}+${fault.gapMs}ms${fault.holdMs ? ` hold ${fault.holdMs}ms` : ""}`) } } : {}) });
|
||||
preflight(plan);
|
||||
const serverEnvironment = plan.options.env === "daytona" ? await prepareDaytona(plan, timeline) : {};
|
||||
const serverEnvironment: Record<string, string> = plan.options.env === "daytona" ? await prepareDaytona(plan, timeline) : {};
|
||||
// The suite drives the API only; the Vite dev UI only slows every boot and restart.
|
||||
if (plan.options.crashPoint && !plan.replayOf) serverEnvironment.SERVE_UI = "false";
|
||||
// Baseline only after the harness's own build steps (preflight rewrites the runner's dist).
|
||||
startFingerprint = worktreeFingerprint(repositoryRoot, guardedOutputs);
|
||||
|
||||
const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline });
|
||||
// Crash-point suite: one booted server, one point at a time. Each point gets
|
||||
// its own batch of tasks; the point is armed once the batch launched and the
|
||||
// server kills itself at its nth hit (restarted like K1). Invariants are
|
||||
// checked when the batch rests, as violations new since the point was armed.
|
||||
// A replay fires the recorded faults (the crash point's K1 restart included)
|
||||
// at their offsets; re-arming the point could miss its nth hit with fewer tasks.
|
||||
const crashPoint = plan.replayOf ? null : plan.options.crashPoint ?? null;
|
||||
const crashPoints = crashPoint ? crashPoint.split(",") : [];
|
||||
const batchSize = crashPoints.length ? Math.max(1, Math.floor(plan.tasks.length / crashPoints.length)) : 0;
|
||||
type PointResult = { point: string; token: string; taskIds: string[]; armedAt: number; restedAt: number | null;
|
||||
fault: FaultReceipt | null; invariants: InvariantCheck | null };
|
||||
const pointResults: Array<PointResult & { baseline?: Set<string> }> = [];
|
||||
let currentPoint: (PointResult & { baseline: Set<string> }) | null = null;
|
||||
let databaseUrl: string | null = null;
|
||||
const violationKey = (violation: { rule: string; kind: string; recordId: string }) => `${violation.rule}:${violation.kind}:${violation.recordId}`;
|
||||
if (crashPoint) {
|
||||
if (plan.tasks.length !== batchSize * crashPoints.length) throw new Error(`--tasks must be a multiple of the ${crashPoints.length} crash point(s)`);
|
||||
supervisor.useCrashPointControl({ controlPath: path.join(supervisor.paths.temporaryRoot, "crash-point.control"),
|
||||
receiptPath: path.join(waveDir, "crash-point-receipts.jsonl") });
|
||||
}
|
||||
let sweeper: DaytonaWaveSweeper | null = null;
|
||||
let sandboxSweep: DaytonaSweepResult | null = null;
|
||||
let interrupted = false;
|
||||
@@ -231,6 +254,62 @@ async function main(): Promise<number> {
|
||||
}));
|
||||
};
|
||||
|
||||
supervisor.onCrashPoint = receipt => {
|
||||
const owner = pointResults.find(result => result.token === receipt.crashPoint?.token);
|
||||
if (owner) owner.fault = receipt;
|
||||
faults.push(receipt);
|
||||
if (receipt.error) { faultFailures += 1; timeline.add("fault_error", { taskId: "crash-point", error: receipt.error }); }
|
||||
for (const candidate of active.filter(observer => !observer.done)) {
|
||||
candidate.noteFault(receipt.faultId, receipt.startedAt);
|
||||
candidate.extendDeadline((receipt.readyAt ?? Date.now()) - receipt.startedAt);
|
||||
taskFaultLog[candidate.plan.taskId] = { createdAt: candidate.record.createdAt, faultIds: [...candidate.record.faultIds] };
|
||||
}
|
||||
checkWorktree(receipt.faultId);
|
||||
void persistFaultLog();
|
||||
};
|
||||
|
||||
const invariantsNow = async () => {
|
||||
if (!databaseUrl) {
|
||||
const config = JSON.parse(readFileSync(supervisor.paths.configPath, "utf8")) as { database?: { connectionString?: string } };
|
||||
databaseUrl = config.database?.connectionString ?? null;
|
||||
if (!databaseUrl) throw new Error("Crash-point wave: the instance config has no database connection string");
|
||||
}
|
||||
return checkPlanInvariants({ connectionString: databaseUrl, companyId: lab.companyId,
|
||||
paperclipHome: supervisor.paths.home, instanceId: "default" });
|
||||
};
|
||||
const startCrashPoint = async (batch: TaskObserver[]) => {
|
||||
const index = pointResults.length;
|
||||
const spec = crashPoints[index]!;
|
||||
// Violations already present belong to earlier points, not this one.
|
||||
const before = await invariantsNow();
|
||||
const baseline = new Set(before.violations.map(violationKey));
|
||||
// Stage the batch unassigned (no wake yet), arm the point, then assign:
|
||||
// no task's work can start before the point is armed, and an early
|
||||
// crash cannot fail a later task's creation. The previous batch has
|
||||
// rested, so only this batch's work can hit the point.
|
||||
for (const observer of batch) await observer.stage();
|
||||
const result = { point: spec, token: `p${String(index + 1).padStart(2, "0")}`, taskIds: batch.map(observer => observer.plan.taskId),
|
||||
armedAt: Date.now(), restedAt: null, fault: null, invariants: null, baseline };
|
||||
pointResults.push(result);
|
||||
supervisor.armCrashPoint(result.token, spec);
|
||||
for (const observer of batch) {
|
||||
launched.push(observer);
|
||||
await observer.release();
|
||||
if (!observer.done) active.push(observer);
|
||||
}
|
||||
return result;
|
||||
};
|
||||
const finishCrashPoint = async (point: PointResult & { baseline: Set<string> }) => {
|
||||
supervisor.disarmCrashPoint();
|
||||
point.restedAt = Date.now();
|
||||
const check = await invariantsNow();
|
||||
const fresh = check.violations.filter(violation => !point.baseline.has(violationKey(violation)));
|
||||
point.invariants = { ...check, violations: fresh, byInvariant: summarizeViolations(fresh) };
|
||||
timeline.add("crash_point_settled", { point: point.point, token: point.token, reached: point.fault?.crashPoint ?? null,
|
||||
seconds: Math.round((point.restedAt - point.armedAt) / 1000), byInvariant: point.invariants.byInvariant,
|
||||
errors: check.errors.map(error => redactKnownSecrets(error, knownSecretValues())) });
|
||||
};
|
||||
|
||||
while (!interrupted && (pending.length || active.length)) {
|
||||
const now = Date.now();
|
||||
if (storm?.endIfDue(now)) {
|
||||
@@ -241,8 +320,15 @@ async function main(): Promise<number> {
|
||||
// A failed fault or an unexpected exit leaves the server not ready.
|
||||
// Recover within a bound, or fail the wave instead of waiting forever.
|
||||
if (!supervisor.serverReady && !supervisor.faultInProgress) await supervisor.ensureServing();
|
||||
if (crashPoint) {
|
||||
// One point at a time: settle the previous batch, then launch and arm the next.
|
||||
if (!active.length && supervisor.serverReady && !supervisor.faultInProgress) {
|
||||
if (currentPoint) { await finishCrashPoint(currentPoint); currentPoint = null; }
|
||||
if (pending.length) currentPoint = await startCrashPoint(pending.splice(0, batchSize));
|
||||
}
|
||||
}
|
||||
// Recheck the storm clock per launch: ensureServing or a slow launch can cross the storm's end.
|
||||
while (active.length < plan.options.concurrency && pending.length && supervisor.serverReady && !supervisor.faultInProgress
|
||||
while (!crashPoint && active.length < plan.options.concurrency && pending.length && supervisor.serverReady && !supervisor.faultInProgress
|
||||
&& (!storm || storm.mayLaunch(Date.now()))) {
|
||||
const next = pending.shift()!;
|
||||
launched.push(next);
|
||||
@@ -254,6 +340,8 @@ async function main(): Promise<number> {
|
||||
await observer.poll(now);
|
||||
}));
|
||||
for (const observer of active) {
|
||||
// A crash-point wave has no scheduled faults: the armed point is the fault.
|
||||
if (crashPoint) break;
|
||||
if (faultsQueued || supervisor.faultInProgress || !supervisor.serverReady) break;
|
||||
// A storm's drain gets no new fault; an unfired trigger is recorded as missed at rest.
|
||||
if (storm && !storm.mayLaunch(Date.now())) break;
|
||||
@@ -340,6 +428,26 @@ async function main(): Promise<number> {
|
||||
rules: Object.fromEntries(Object.entries(grade.rules).map(([id, rule]) => [id, rule.verdict])), T: grade.T.verdict });
|
||||
}
|
||||
await writeFile(path.join(waveDir, "faults.json"), `${JSON.stringify(faults, null, 2)}\n`, { mode: 0o600 });
|
||||
if (crashPoint) {
|
||||
if (currentPoint) { await finishCrashPoint(currentPoint); currentPoint = null; }
|
||||
const byTask = new Map(lines.map(line => [line.taskId, line]));
|
||||
for (const point of pointResults) {
|
||||
if (point.invariants?.violations.length) anyFailure = true;
|
||||
if (point.invariants?.errors.length) harnessError ??= new Error(`Invariant check incomplete at ${point.point}: ${point.invariants.errors.join("; ")}`);
|
||||
}
|
||||
// Evidence carries run and wake error text; CI uploads this file.
|
||||
await writeFile(path.join(waveDir, "crash-point.json"), `${redactKnownSecrets(JSON.stringify({
|
||||
schema: "paperclip.restart-chaos.crash-point/v2", gitSha, gitDirty,
|
||||
// The server kills itself without warning, so no pre-fault workspace
|
||||
// snapshot exists: R2 checks workspace failures, not lost files.
|
||||
notMeasured: { R2_lost_files: "no pre-fault workspace snapshot: the crash point fires inside the server" },
|
||||
points: pointResults.map(({ baseline: _baseline, ...point }) => ({
|
||||
...(point as PointResult), reached: point.fault?.crashPoint ?? null,
|
||||
tasks: point.taskIds.flatMap(id => { const line = byTask.get(id); return line ? [{ taskId: id, adapter: line.adapter,
|
||||
workload: line.workload, pass: line.pass, signature: line.signature, rules: line.rules, ruleDetails: line.ruleDetails, T: line.T }] : []; }),
|
||||
})),
|
||||
}, null, 2), knownSecretValues())}\n`, { mode: 0o600 });
|
||||
}
|
||||
// R6 at wave level: a leak no task's runs account for (e.g. the setup probe) still fails the wave.
|
||||
if (sandboxSweep) {
|
||||
const unattributed = unattributedLeaks(sandboxSweep, attributed);
|
||||
|
||||
@@ -3,6 +3,7 @@ import { randomUUID } from "node:crypto";
|
||||
import { readdir, readFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { ChaosApiError, type ChaosApi } from "./api.js";
|
||||
import { ACTIVE_RUN_STATUSES, isStalledSample, STALL_LIMIT_MS } from "./grader.js";
|
||||
import type { ChaosLab } from "./seed.js";
|
||||
@@ -159,15 +160,50 @@ export class TaskObserver {
|
||||
|
||||
get definition() { return workloadDefinitions[this.plan.workload]; }
|
||||
|
||||
async launch(): Promise<void> {
|
||||
/**
|
||||
* Crash-point suite: create the issue without an assignee, so no wake is
|
||||
* queued yet. `release` assigns it once the batch's point is armed.
|
||||
*/
|
||||
async stage(): Promise<void> { await this.launch({ assign: false }); }
|
||||
|
||||
/**
|
||||
* Assigns a staged issue, which queues its wake. An earlier task of the same
|
||||
* batch can crash the server meanwhile, so retry until the replacement
|
||||
* answers; setting the same assignee again is idempotent.
|
||||
*/
|
||||
async release(deadlineMs = 10 * 60_000): Promise<void> {
|
||||
if (this.done || !this.record.issueId) return;
|
||||
const until = Date.now() + deadlineMs;
|
||||
for (;;) {
|
||||
try {
|
||||
await this.api.patch(`/api/issues/${this.record.issueId}`, { assigneeAgentId: this.lab.agents[this.plan.adapter].id });
|
||||
break;
|
||||
} catch (error) {
|
||||
const transient = error instanceof ChaosApiError && (error.status === null || error.status >= 500);
|
||||
if (!transient || Date.now() > until) {
|
||||
this.record.launchError = error instanceof Error ? error.message : String(error);
|
||||
this.done = true;
|
||||
this.timeline.add("task_launch_failed", { taskId: this.plan.taskId, error: this.record.launchError });
|
||||
return;
|
||||
}
|
||||
await delay(1_000);
|
||||
}
|
||||
}
|
||||
this.record.createdAt = Date.now();
|
||||
this.record.deadlineAt = this.record.createdAt + this.definition.budgetMs;
|
||||
this.timeline.add("task_released", { taskId: this.plan.taskId, issueId: this.record.issueId });
|
||||
}
|
||||
|
||||
async launch(options: { assign?: boolean } = {}): Promise<void> {
|
||||
const definition = this.definition;
|
||||
const assign = options.assign !== false;
|
||||
try {
|
||||
const issue = await this.api.post<Json>(`/api/companies/${this.lab.companyId}/issues`, {
|
||||
title: definition.title(this.plan.nonce),
|
||||
description: definition.description(this.plan.nonce),
|
||||
status: "todo",
|
||||
priority: "medium",
|
||||
assigneeAgentId: this.lab.agents[this.plan.adapter].id,
|
||||
...(assign ? { assigneeAgentId: this.lab.agents[this.plan.adapter].id } : {}),
|
||||
...(definition.usesRepository ? { projectId: this.lab.projectId } : {}),
|
||||
});
|
||||
this.record.issueId = String(issue.id);
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
*/
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { createWriteStream, statSync, type WriteStream } from "node:fs";
|
||||
import { createWriteStream, readFileSync, statSync, writeFileSync, type WriteStream } from "node:fs";
|
||||
import { mkdir, readFile, symlink } from "node:fs/promises";
|
||||
import { createRequire } from "node:module";
|
||||
import os from "node:os";
|
||||
@@ -36,7 +36,11 @@ import type { MatrixExecution } from "../runner-e2e/types.js";
|
||||
import type { Timeline } from "./timeline.js";
|
||||
import type { FaultKind, FaultReceipt, TriggerId } from "./types.js";
|
||||
|
||||
const HEALTH_TIMEOUT_MS = 180_000;
|
||||
/** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */
|
||||
const HEALTH_TIMEOUT_MS = (() => {
|
||||
const raw = Number(process.env.RESTART_CHAOS_HEALTH_TIMEOUT_MS);
|
||||
return Number.isInteger(raw) && raw >= 30_000 && raw <= 1_800_000 ? raw : 180_000;
|
||||
})();
|
||||
const READY_TIMEOUT_MS = 300_000;
|
||||
const STOP_TIMEOUT_MS = 30_000;
|
||||
|
||||
@@ -52,6 +56,30 @@ export function crashLoopShouldFire(target: (typeof CRASH_LOOP_TARGETS)[number],
|
||||
|
||||
export interface SupervisorPaths { temporaryRoot: string; home: string; configPath: string; serverLogPath: string }
|
||||
|
||||
/**
|
||||
* Crash-point suite: every server spawn reads its armed point from the control
|
||||
* file at each hit (PAPERCLIP_CRASH_POINT_CONTROL), so one booted server serves
|
||||
* many points. The supervisor writes "<token> <name>[:<nth>]" to arm a point
|
||||
* and empties the file to disarm, including before a crash's replacement starts.
|
||||
*/
|
||||
export interface CrashPointControl { controlPath: string; receiptPath: string }
|
||||
|
||||
/** The receipt line the crash-point helper appended for this pid, if any. */
|
||||
export function readCrashPointReceipt(receiptPath: string, pid: number | undefined): { name: string; hit: number; pid: number; token?: string } | null {
|
||||
let text: string;
|
||||
try { text = readFileSync(receiptPath, "utf8"); } catch { return null; }
|
||||
for (const line of text.split("\n").reverse()) {
|
||||
try {
|
||||
const parsed = JSON.parse(line) as { point?: unknown; hit?: unknown; pid?: unknown };
|
||||
if (pid !== undefined && parsed.pid === pid && typeof parsed.point === "string" && typeof parsed.hit === "number") {
|
||||
const token = (parsed as { token?: unknown }).token;
|
||||
return { name: parsed.point, hit: parsed.hit, pid, ...(typeof token === "string" ? { token } : {}) };
|
||||
}
|
||||
} catch { /* partial or blank line */ }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
const secret = () => randomBytes(48).toString("base64url");
|
||||
|
||||
/** Same provider PATH shape as the runner-e2e launcher: pinned Codex first. */
|
||||
@@ -84,6 +112,9 @@ export class ChaosSupervisor {
|
||||
serverReady = false;
|
||||
faultInProgress = false;
|
||||
unexpectedExits = 0;
|
||||
private crashPointControl: CrashPointControl | null = null;
|
||||
/** Called with the receipt of a crash-point self-kill once the replacement is ready (or failed). */
|
||||
onCrashPoint: ((receipt: FaultReceipt) => void) | null = null;
|
||||
|
||||
private constructor(readonly repositoryRoot: string, readonly port: number, readonly paths: SupervisorPaths,
|
||||
private readonly timeline: Timeline) {
|
||||
@@ -139,6 +170,10 @@ export class ChaosSupervisor {
|
||||
PAPERCLIP_STORAGE_LOCAL_DIR: path.join(temporaryRoot, "storage"), PAPERCLIP_SECRETS_PROVIDER: "local_encrypted",
|
||||
PAPERCLIP_SECRETS_STRICT_MODE: "true", PAPERCLIP_DB_BACKUP_ENABLED: "false",
|
||||
PAPERCLIP_DB_BACKUP_DIR: path.join(temporaryRoot, "backups"), PAPERCLIP_OPEN_ON_LISTEN: "false",
|
||||
// The server loads a .paperclip/.env found above its cwd without overriding
|
||||
// set values. A worktree-initialized checkout's file sets
|
||||
// PAPERCLIP_IN_WORKTREE=true, which suppresses all scheduling: no run starts.
|
||||
PAPERCLIP_IN_WORKTREE: "false",
|
||||
...extraEnvironment,
|
||||
};
|
||||
const environment = buildPaperclipServerEnvironment(source, overrides);
|
||||
@@ -154,14 +189,22 @@ export class ChaosSupervisor {
|
||||
private async spawnServer(): Promise<ChildProcess> {
|
||||
if (this.shuttingDown) throw new Error("Refusing to start Paperclip during harness shutdown");
|
||||
const entry = path.join(this.repositoryRoot, "tests/runner-e2e/server-entry.ts");
|
||||
const control = this.crashPointControl;
|
||||
const env = control ? { ...this.environment, PAPERCLIP_CRASH_POINT_CONTROL: control.controlPath,
|
||||
PAPERCLIP_CRASH_POINT_RECEIPT: control.receiptPath } : this.environment;
|
||||
const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(this.repositoryRoot, entry, ["onboard", "--yes", "--run"]),
|
||||
{ cwd: this.repositoryRoot, env: this.environment, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached });
|
||||
{ cwd: this.repositoryRoot, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached });
|
||||
this.child = child;
|
||||
|
||||
child.stdout?.on("data", chunk => this.log.write(chunk));
|
||||
child.stderr?.on("data", chunk => this.log.write(chunk));
|
||||
child.once("error", error => { this.spawnErrors.set(child, error); });
|
||||
child.once("exit", (code, signal) => {
|
||||
this.log.write(`\n[restart-chaos] server pid=${child.pid} exited code=${code} signal=${signal}\n`);
|
||||
if (control && signal === "SIGKILL" && !this.expectedStops.has(child) && !this.shuttingDown) {
|
||||
const hit = readCrashPointReceipt(control.receiptPath, child.pid);
|
||||
if (hit) { this.restartAfterCrashPoint(child, hit); return; }
|
||||
}
|
||||
if (!this.expectedStops.has(child) && !this.shuttingDown) {
|
||||
this.unexpectedExits += 1;
|
||||
this.serverReady = false;
|
||||
@@ -229,6 +272,74 @@ export class ChaosSupervisor {
|
||||
throw new Error(`Paperclip did not report ready within ${READY_TIMEOUT_MS}ms`);
|
||||
}
|
||||
|
||||
/** Crash-point suite: run every spawn (from `start` on) with runtime arming. */
|
||||
useCrashPointControl(control: CrashPointControl): void {
|
||||
writeFileSync(control.controlPath, "", { mode: 0o600 });
|
||||
this.crashPointControl = control;
|
||||
}
|
||||
|
||||
/** Arms `spec` ("<name>[:<nth>]") with a fresh hit count under `token`. */
|
||||
armCrashPoint(token: string, spec: string): void {
|
||||
if (!this.crashPointControl) throw new Error("Crash-point control is not enabled");
|
||||
writeFileSync(this.crashPointControl.controlPath, `${token} ${spec}\n`, { mode: 0o600 });
|
||||
this.timeline.add("crash_point_armed", { token, spec });
|
||||
}
|
||||
|
||||
disarmCrashPoint(): void {
|
||||
if (this.crashPointControl) writeFileSync(this.crashPointControl.controlPath, "", { mode: 0o600 });
|
||||
}
|
||||
|
||||
/**
|
||||
* The armed server killed itself at its crash point. Treat it like K1: kill
|
||||
* what is left of its owned process tree (local runners), start a
|
||||
* replacement without the point armed, and wait for ready.
|
||||
*/
|
||||
private restartAfterCrashPoint(previous: ChildProcess, hit: { name: string; hit: number; pid: number; token?: string }): void {
|
||||
// The replacement reads the same control file: disarm before it starts.
|
||||
this.disarmCrashPoint();
|
||||
this.faultInProgress = true;
|
||||
this.serverReady = false;
|
||||
this.faultCount += 1;
|
||||
const now = Date.now();
|
||||
const receipt: FaultReceipt = {
|
||||
faultId: `f${String(this.faultCount).padStart(2, "0")}`, kind: "K1", requestedBy: "crash-point", signal: "SIGKILL", holdMs: 0,
|
||||
loadAvg1m: Math.round(os.loadavg()[0]! * 100) / 100, crashPoint: hit,
|
||||
requestedAt: now, startedAt: now, stoppedAt: null, restartedAt: null, healthyAt: null, readyAt: null,
|
||||
phases: [], serverLogOffsetAtStart: this.serverLogOffset(), serverLogOffsetAtReady: null, error: null,
|
||||
};
|
||||
this.timeline.add("crash_point_fired", { faultId: receipt.faultId, point: hit.name, hit: hit.hit, pid: hit.pid });
|
||||
this.log.write(`\n[restart-chaos] ${receipt.faultId} crash point ${hit.name} hit ${hit.hit} killed pid=${hit.pid}\n`);
|
||||
let cleanupError: string | null = null;
|
||||
const run = this.faultChain.then(async () => {
|
||||
try {
|
||||
// The leader is already dead; this kills its owned descendants, as K1 does.
|
||||
// A cleanup that cannot prove the tree stopped fails the fault (and the
|
||||
// wave); the replacement still starts so the tasks can drain.
|
||||
await this.stopper.crashForRestart(previous).catch(error => {
|
||||
cleanupError = `owned process tree cleanup failed: ${error instanceof Error ? error.message : String(error)}`;
|
||||
this.log.write(`[restart-chaos] crash-point ${cleanupError}\n`);
|
||||
});
|
||||
if (this.child === previous) this.child = null;
|
||||
await this.waitForHealthToStop();
|
||||
receipt.stoppedAt = Date.now();
|
||||
receipt.restartedAt = Date.now();
|
||||
const child = await this.spawnServer();
|
||||
receipt.healthyAt = await this.waitForHealth(child, receipt.phases);
|
||||
receipt.readyAt = await this.waitForReadyPhase(receipt.phases);
|
||||
receipt.serverLogOffsetAtReady = this.serverLogOffset();
|
||||
this.timeline.add("fault_ready", { faultId: receipt.faultId, replacementPid: child.pid, readyAfterMs: receipt.readyAt - receipt.startedAt });
|
||||
} catch (error) {
|
||||
receipt.error = error instanceof Error ? error.message : String(error);
|
||||
this.timeline.add("fault_failed", { faultId: receipt.faultId, error: receipt.error });
|
||||
} finally {
|
||||
if (cleanupError && !receipt.error) receipt.error = cleanupError;
|
||||
this.faultInProgress = false;
|
||||
this.onCrashPoint?.(receipt);
|
||||
}
|
||||
});
|
||||
this.faultChain = run.catch(() => undefined);
|
||||
}
|
||||
|
||||
/** Faults are serialized: one control plane, one outage at a time. */
|
||||
fault(kind: FaultKind, holdMs: number, requestedBy: string, trigger?: { trigger: TriggerId; delayMs: number }): Promise<FaultReceipt> {
|
||||
const requestedAt = Date.now();
|
||||
|
||||
@@ -53,6 +53,13 @@ export interface WaveOptions {
|
||||
* keeps the catalog profile's own setting.
|
||||
*/
|
||||
lifecycle?: "per_turn" | "warm" | "default";
|
||||
/**
|
||||
* Crash-point suite (`--crash-point <name>[:<nth>][,...]`): no external
|
||||
* fault is scheduled. One server serves every listed point in turn: each
|
||||
* point's batch of tasks launches, then the point is armed and the server
|
||||
* SIGKILLs itself at its nth hit and is restarted like K1.
|
||||
*/
|
||||
crashPoint?: string;
|
||||
}
|
||||
|
||||
export interface StormFault {
|
||||
@@ -182,6 +189,8 @@ export interface FaultReceipt {
|
||||
triggerDelayMs?: number;
|
||||
loadAvg1m?: number;
|
||||
crashLoop?: Array<{ pid: number | null; target: string; phase: string; at: number }>;
|
||||
/** Crash-point suite: the server killed itself at this named point (no external signal). */
|
||||
crashPoint?: { name: string; hit: number; pid: number; token?: string };
|
||||
serverLogOffsetAtStart: number;
|
||||
serverLogOffsetAtReady: number | null;
|
||||
error: string | null;
|
||||
|
||||
@@ -617,6 +617,9 @@ describe("public repository paid workflow security", () => {
|
||||
// private-trust-workflow-security.test.ts checks how these two hold the keys.
|
||||
"chat-catalog-testops.yml",
|
||||
"chat-catalog-testops-case.yml",
|
||||
// On-demand crash-point suite (durable ownership plan, Phase 5): manual
|
||||
// dispatch, default branch, allowlisted actors, runner-e2e-paid.
|
||||
"crash-points.yml",
|
||||
]);
|
||||
const names = (await readdir(workflowDirectory)).filter((name) =>
|
||||
/\.ya?ml$/.test(name),
|
||||
|
||||
Reference in new issue
Block a user