mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 21:49:18 +01:00
Devin FoleyandPaperclip
fc06f7f05f
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud deployments require verified artifacts for the merged source commit. > - Cloud readiness and the npm release independently run the same source checks. > - Their shared chaos workflow used only the source ref as its concurrency key. > - One caller could cancel the other caller's required job for the same commit. > - This pull request scopes that key to the caller workflow and source ref. > - Both callers can finish their checks without blocking deployment readiness. ## Linked Issues or Issue Description Refs #13192 and #13205. Searched for related open issues and PRs; no duplicate fix was found. **What happened?** The master push for `398d304e15739d1ee6105633bd8a0e42c929d33f` started Cloud readiness and Release together. GitHub cancelled the Cloud readiness chaos job before it acquired a runner. Its annotation reported a higher-priority waiting request for the same concurrency group. The required readiness gate cannot pass after that cancellation. **Expected behavior** Cloud readiness and Release must each finish source verification for the same SHA. Standalone chaos evals must also have a separate group. **Steps to reproduce** Merge a commit to master while the npm release queue is empty. Both callers reach the reusable chaos workflow with the same source SHA. See [the cancelled job](https://github.com/paperclipai/paperclip/actions/runs/34569569760/job/103168603926). **Paperclip version or commit** `398d304e15739d1ee6105633bd8a0e42c929d33f`. **Deployment mode** GitHub Actions on master. ## What Changed - Add the caller workflow name to the chaos workflow concurrency group. Retain source isolation and cancellation of duplicate calls within the same workflow. - Add a regression test that evaluates the group for Cloud readiness, Release, and standalone evals at the same source SHA. - Document the concurrency boundary in the readiness runbook. ## Verification - `node --test scripts/preview-artifacts.test.mjs scripts/__tests__/release-verify-workflow.test.mjs` passed: 26 tests. - The new regression test fails against the previous concurrency key and passes with this fix. - `actionlint -shellcheck= -pyflakes= .github/workflows/runner-chaos-evals.yml .github/workflows/release-verify.yml .github/workflows/cloud-readiness.yml` passed. - `git diff --check` passed. - The full local typecheck passed for the same application source in #13205. Its macOS general-server test phase had 10,471 passes and 70 failures in seven unchanged application test files: missing Cargo/Runner test binaries, filesystem permissions, timeouts, a port conflict, and a load-test count mismatch. Linux CI test checks passed. The full local build passed with Cargo on PATH. This PR changes workflow configuration, its test, and documentation only. - All CI checks pass on the final head, including typecheck, tests, browser suites, build, and canary dry run. Greptile is 5/5 with no open findings. After merge, verify both callers' chaos jobs complete for the same master SHA and record the resulting readiness time. ## Risks - Two callers may now run chaos tests at the same time. This uses two existing GitHub runners, which is the intended cost of independent verification. - Renaming a caller changes its concurrency group. The fixed prefix keeps this child group separate from caller-level concurrency groups. - The readiness gate continues to require every verification prerequisite. No gate is bypassed. ## Model Used - OpenAI GPT-6 / Codex, with reasoning, repository editing, and command/API tools. Exact serving model ID and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (26 focused workflow/artifact tests) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
86 lines
2.9 KiB
YAML
86 lines
2.9 KiB
YAML
name: Runner Chaos Evals
|
|
|
|
on:
|
|
schedule:
|
|
- cron: "43 7 * * 0"
|
|
workflow_dispatch:
|
|
workflow_call:
|
|
inputs:
|
|
ref:
|
|
description: Commit SHA, branch, or tag to verify before release
|
|
required: false
|
|
type: string
|
|
|
|
concurrency:
|
|
# Reusable calls inherit the caller's workflow name. Cloud readiness and
|
|
# Release verify the same SHA independently and must not cancel each other.
|
|
group: runner-chaos-evals-${{ github.workflow }}-${{ inputs.ref || github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
chaos_and_recovery:
|
|
name: Restart, replay, trace, and recovery faults
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 45
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref || github.sha }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Build eval and Runner contracts
|
|
run: |
|
|
pnpm --filter @paperclipai/paperclip-eval-kernel build
|
|
pnpm --filter @paperclipai/paperclip-runner report:runner-chaos-evals
|
|
|
|
- name: Run Runner fault and replay suites
|
|
run: |
|
|
pnpm --filter @paperclipai/paperclip-runner exec vitest run \
|
|
src/eval/workflow-evals.test.ts \
|
|
src/native-session-runtime.test.ts \
|
|
src/live/live-session.test.ts \
|
|
src/live/turn-stream.test.ts \
|
|
src/protocol/replay-contract.test.ts \
|
|
src/drivers/opencode/mcp-bridge.test.ts \
|
|
src/drivers/acpx/runtime-host.test.ts
|
|
|
|
- name: Build server test dependencies
|
|
run: pnpm --filter @paperclipai/plugin-sdk ensure-build-deps
|
|
|
|
- name: Run server finalization and recovery suites
|
|
run: |
|
|
pnpm --filter @paperclipai/server exec vitest run \
|
|
src/__tests__/native-finalization-recovery.test.ts \
|
|
src/__tests__/heartbeat-process-recovery.test.ts \
|
|
src/__tests__/heartbeat-comment-wake-batching.test.ts \
|
|
src/__tests__/heartbeat-dependency-scheduling.test.ts \
|
|
src/__tests__/provider-trace-store.test.ts \
|
|
src/services/issue-thread-interaction-resolution.test.ts \
|
|
src/services/recovery/successful-run-handoff.test.ts
|
|
|
|
- name: Upload chaos eval bundle
|
|
if: always()
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
name: runner-chaos-evals-${{ github.run_id }}
|
|
path: packages/paperclip-runner/.paperclip-local/evals/workflows/
|
|
retention-days: 30
|
|
if-no-files-found: error
|