mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - Random restart-chaos waves found these 20 ms–1 s windows only by luck > - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable write boundaries > - Nothing yet runs a task, kills the server at a chosen point, restarts it, and checks the plan's invariants > - This pull request adds that on-demand suite, as a local command and as a manual CI workflow (not a required check, per Decision 2) > - The benefit is a repeatable work list: which boundary breaks which invariant, for Phases 2–4 ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only reconciler used for I1–I7). Stacked on #19938 (call sites), which is stacked on #19937 (helper). **Problem** The named crash points cannot be exercised end to end. No command arms one point, runs real agent tasks across it, restarts, and grades the result. **Expected behavior** `node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters <list>` runs one wave per selected point and reports each point as pass, fail (with failing R rules and violated invariants), not reached, or harness error. A `workflow_dispatch` workflow runs the same thing on demand. ## What Changed - **Runtime arming.** `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the shared helper and the runner mirror. Each hit reads `<token> <name>[:<nth>]`, and a new token resets the count. It is test mode only: `NODE_ENV=production` ignores it, and the disarmed path is still one comparison. - **`tests/restart-chaos/main.ts --crash-point <a>:<nth>,<b>:<nth>,...`** (config, types, ledger): - one booted server and no Vite UI; - per point: take an invariant baseline, launch the point's batch, arm the point, then on rest disarm it and check invariants as violations that are new since the baseline; - every task is graded R1–R6 and T; - `crash-point.json` (v2) holds one entry per point. - **`tests/restart-chaos/supervisor.ts`:** - every spawn gets the control file; - a SIGKILL whose receipt names the child's pid is restarted like K1: the control file is emptied first, the owned tree is killed, and a replacement starts. A failed tree cleanup fails the fault; - the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a `.paperclip/.env` above the checkout otherwise suppressed all scheduling); - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`. - **`tests/restart-chaos/invariants.ts`:** - the Phase 1 report-only pass with `record: false`: two real-clock passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7; - spool kinds are read from the wave's instance; - an unfinished pass is a harness error; - each violation carries its wake and run state as evidence. - **`tests/restart-chaos/crash-points.ts` and `scripts/crash-points.mjs`:** - select points by glob, kind and environment, and run them in one wave; - a per-point outcome: pass, fail, not_reached or harness_error; - `SUMMARY.md` with a violation evidence column. - **`.github/workflows/crash-points.yml`**, manual only, under the paid-workflow contract: - a credential-free `authorize` job; - a `suite` job in `runner-e2e-paid` whose first step reauthorizes; - a frozen install with `--ignore-scripts` and no shared cache; - only the selected adapters' keys; - the workflow is added to both security-test allowlists. - **Docs:** "Crash-point suite" in `tests/restart-chaos/README.md`, and "Built (Phase 5)" plus the first work list in the plan. ## Verification - Focused tests: - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`; - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 tests, including runtime arming in a real child process; - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72 tests; - `tests/runner-e2e/workflow-security.test.ts`: 19 tests. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Loop line: question "how long is one crash-point loop?" | loop: `node scripts/crash-points.mjs --points <one>` (codex A1) | elapsed 2.7 min end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s, rest about 2 min | result: one point alone takes about 3 min. Before this change it took about 15 min (one cold boot per point, Vite UI, host at load 77). - Full local run, 14 points, codex A1, 11.8 min in all: - every point was reached (the server killed itself at each one, and the replacement came back); - 11 pass; - `workspace.finalize.after_start_before_copyback` and `workspace.finalize.after_copyback_before_done` fail I2 and R6: a `workspace_finalize` operation stays running after the run succeeded; - `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its evidence shows the wake failed with "…retrying once", and the retry delivered the work, so this is the known refusal-regex false positive. - The 6 Daytona-only points were not run. ## Risks - Test harness and an on-demand workflow only. No product code changes. - `.github/**` needs a CODEOWNER review. The workflow uses provider keys, so it refuses non-default branches and actors that are not on the allowlist, like `runner-full-stack-e2e.yml`. - `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only crash-point waves. A chaos instance is never a worktree dev instance, so scheduling must not be suppressed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
5742c42b75
commit
051834bdc8
18 files changed
+1187
-26
No files matched your search
@@ -0,0 +1,203 @@
|
||||
name: Crash-point suite
|
||||
|
||||
# On-demand only (durable ownership plan, Decision 2): never a required PR
|
||||
# check. Runs tests/restart-chaos/crash-points.ts on the default branch, which
|
||||
# kills an isolated control plane at each selected named crash point, restarts
|
||||
# it and asserts plan invariants I1–I8 plus restart-chaos graders R1–R6.
|
||||
# Provider keys come from the runner-e2e-paid environment, so the code under
|
||||
# test must be CODEOWNER-reviewed: the default branch only, allowlisted actors.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
points:
|
||||
description: "Comma-separated crash-point globs (blank = every point reachable in env)"
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
kinds:
|
||||
description: "Comma-separated ownership kind ids, e.g. native_executor_claim (blank = all)"
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
adapters:
|
||||
description: "Comma-separated adapters: codex, claude, opencode"
|
||||
required: false
|
||||
default: codex
|
||||
type: string
|
||||
workloads:
|
||||
description: "Comma-separated restart-chaos workloads: A1, B1, B5, C1"
|
||||
required: false
|
||||
default: A1
|
||||
type: string
|
||||
nth:
|
||||
description: "Fire at the nth hit of each point"
|
||||
required: false
|
||||
default: "1"
|
||||
type: string
|
||||
|
||||
concurrency:
|
||||
group: crash-points-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
# No secrets here: refuse the wrong branch, actor or input before the paid job starts.
|
||||
authorize:
|
||||
name: Authorize crash-point run
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Require default branch and allowlisted numeric actor IDs
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
REF: ${{ github.ref }}
|
||||
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
||||
ACTOR: ${{ github.actor }}
|
||||
ACTOR_ID: ${{ github.actor_id }}
|
||||
TRIGGERING_ACTOR: ${{ github.triggering_actor }}
|
||||
ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "$REF" != "refs/heads/$DEFAULT_BRANCH" ]; then
|
||||
echo "The crash-point suite uses provider keys and may run only from the default branch." >&2
|
||||
exit 1
|
||||
fi
|
||||
jq -e 'type == "array" and length > 0 and all(.[]; type == "number" and . > 0 and floor == .)' <<< "${ALLOWED_ACTOR_IDS:-}" >/dev/null
|
||||
for attempt in 1 2 3; do
|
||||
if triggering_actor_id="$(timeout 30s gh api "users/$TRIGGERING_ACTOR" --jq .id)"; then break; fi
|
||||
if [ "$attempt" = 3 ]; then exit 1; fi
|
||||
sleep "$((attempt * 2))"
|
||||
done
|
||||
if [ "$triggering_actor_id" != "$ACTOR_ID" ] && [ "$TRIGGERING_ACTOR" = "$ACTOR" ]; then
|
||||
echo "GitHub actor identity contexts disagree; refusing the paid run." >&2
|
||||
exit 1
|
||||
fi
|
||||
for candidate in "$triggering_actor_id" "$ACTOR_ID"; do
|
||||
jq -e --argjson candidate "$candidate" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null
|
||||
done
|
||||
|
||||
- name: Validate inputs
|
||||
env:
|
||||
POINTS: ${{ inputs.points }}
|
||||
KINDS: ${{ inputs.kinds }}
|
||||
ADAPTERS: ${{ inputs.adapters }}
|
||||
WORKLOADS: ${{ inputs.workloads }}
|
||||
NTH: ${{ inputs.nth }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
[[ "$POINTS" =~ ^[a-z0-9_.*?,]*$ ]] || { echo "points: lower-case names, '.', '_', '*', '?' and ','" >&2; exit 1; }
|
||||
[[ "$KINDS" =~ ^[a-z0-9_,]*$ ]] || { echo "kinds: lower-case ids and ','" >&2; exit 1; }
|
||||
[[ "$ADAPTERS" =~ ^(codex|claude|opencode)(,(codex|claude|opencode))*$ ]] || { echo "adapters: codex, claude, opencode" >&2; exit 1; }
|
||||
[[ "$WORKLOADS" =~ ^(A1|B1|B5|C1)(,(A1|B1|B5|C1))*$ ]] || { echo "workloads: A1, B1, B5, C1" >&2; exit 1; }
|
||||
[[ "$NTH" =~ ^[1-9][0-9]{0,2}$ ]] || { echo "nth: 1-999" >&2; exit 1; }
|
||||
|
||||
suite:
|
||||
name: Crash points (${{ inputs.points || 'all' }})
|
||||
needs: authorize
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 300
|
||||
permissions:
|
||||
contents: read
|
||||
environment:
|
||||
name: runner-e2e-paid
|
||||
steps:
|
||||
- name: Reauthorize paid execution before provider access
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
REF: ${{ github.ref }}
|
||||
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
||||
ACTOR: ${{ github.actor }}
|
||||
ACTOR_ID: ${{ github.actor_id }}
|
||||
TRIGGERING_ACTOR: ${{ github.triggering_actor }}
|
||||
ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "$REF" != "refs/heads/$DEFAULT_BRANCH" ]; then
|
||||
echo "The crash-point suite uses provider keys and may run only from the default branch." >&2
|
||||
exit 1
|
||||
fi
|
||||
jq -e 'type == "array" and length > 0 and all(.[]; type == "number" and . > 0 and floor == .)' <<< "${ALLOWED_ACTOR_IDS:-}" >/dev/null
|
||||
for attempt in 1 2 3; do
|
||||
if triggering_actor_id="$(timeout 30s gh api "users/$TRIGGERING_ACTOR" --jq .id)"; then break; fi
|
||||
if [ "$attempt" = 3 ]; then exit 1; fi
|
||||
sleep "$((attempt * 2))"
|
||||
done
|
||||
if [ "$triggering_actor_id" != "$ACTOR_ID" ] && [ "$TRIGGERING_ACTOR" = "$ACTOR" ]; then
|
||||
echo "GitHub actor identity contexts disagree; refusing the paid run." >&2
|
||||
exit 1
|
||||
fi
|
||||
for candidate in "$triggering_actor_id" "$ACTOR_ID"; do
|
||||
jq -e --argjson candidate "$candidate" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null
|
||||
done
|
||||
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
||||
with:
|
||||
version: 9.15.4
|
||||
|
||||
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
||||
with:
|
||||
node-version: 24
|
||||
|
||||
# No shared dependency cache and no dependency lifecycle scripts in a
|
||||
# secret-bearing job (tests/runner-e2e/SECURITY.md); the repo's own
|
||||
# postinstall step runs explicitly.
|
||||
- run: pnpm install --frozen-lockfile --ignore-scripts
|
||||
- run: node scripts/link-plugin-dev-sdk.mjs
|
||||
|
||||
- name: Build native runner
|
||||
run: |
|
||||
set -euo pipefail
|
||||
pnpm --filter @paperclipai/paperclip-runner build:typescript
|
||||
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
|
||||
|
||||
- name: List selected crash points
|
||||
env:
|
||||
POINTS: ${{ inputs.points }}
|
||||
KINDS: ${{ inputs.kinds }}
|
||||
run: node scripts/crash-points.mjs --list ${POINTS:+--points "$POINTS"} ${KINDS:+--kinds "$KINDS"}
|
||||
|
||||
- name: Run crash-point suite
|
||||
env:
|
||||
POINTS: ${{ inputs.points }}
|
||||
KINDS: ${{ inputs.kinds }}
|
||||
ADAPTERS: ${{ inputs.adapters }}
|
||||
WORKLOADS: ${{ inputs.workloads }}
|
||||
NTH: ${{ inputs.nth }}
|
||||
OPENAI_API_KEY: ${{ contains(inputs.adapters, 'codex') && secrets.OPENAI_API_KEY || '' }}
|
||||
ANTHROPIC_API_KEY: ${{ contains(inputs.adapters, 'claude') && secrets.ANTHROPIC_API_KEY || '' }}
|
||||
OPENROUTER_API_KEY: ${{ contains(inputs.adapters, 'opencode') && secrets.OPENROUTER_API_KEY || '' }}
|
||||
# The hosted runner is a disposable VM; B1 oracles run agent code without Seatbelt there.
|
||||
RESTART_CHAOS_ALLOW_UNSANDBOXED_ORACLE: "1"
|
||||
RESTART_CHAOS_RESULTS_DIR: ${{ runner.temp }}/crash-point-results
|
||||
run: |
|
||||
set -euo pipefail
|
||||
node scripts/crash-points.mjs ${POINTS:+--points "$POINTS"} ${KINDS:+--kinds "$KINDS"} \
|
||||
--adapters "$ADAPTERS" --workloads "$WORKLOADS" --nth "$NTH" --seed "$((GITHUB_RUN_ID % 2147483647))"
|
||||
|
||||
- name: Summarize
|
||||
if: always()
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
for summary in "$RUNNER_TEMP"/crash-point-results/crash-points/*/SUMMARY.md; do cat "$summary" >> "$GITHUB_STEP_SUMMARY"; done
|
||||
|
||||
# Summaries, grades and crash-point receipts only: server logs and
|
||||
# evidence bundles can hold provider output and stay on the runner.
|
||||
- name: Upload results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: crash-points-${{ github.run_id }}
|
||||
path: |
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/SUMMARY.md
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/summary.json
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/w-*/crash-point.json
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/w-*/t*.grade.json
|
||||
${{ runner.temp }}/crash-point-results/crash-points/*/w-*/timeline.jsonl
|
||||
retention-days: 14
|
||||
if-no-files-found: warn
|
||||
Reference in new issue
Block a user