mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-11 05:31:46 +02:00
Devin FoleyandPaperclip
974949a39b
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud waits for source verification before deploying a new image. > - The slowest server verification job spends about ten minutes running tests. > - Each job uses one test worker to preserve test isolation. > - This pull request distributes those suites across ten standard hosted runners. > - The benefit is a shorter verification path with the same test coverage. ## Linked Issues or Issue Description **Current behavior** In [readiness run 34572340764](https://github.com/paperclipai/paperclip/actions/runs/34572340764), the slowest server job ran for 638 seconds. Test execution used 594 seconds. This held readiness behind the image job. **Proposed behavior** Use ten general server jobs in the reusable release verification workflow. Keep the three chat jobs and every existing prerequisite. The complete partition test verifies that no server suite is omitted or duplicated. **Reason and benefit** Reduce merge-to-deployable time on the existing runner type. The next longest prerequisite was Runner verification at 526 seconds, so the initial expected total gain is about two minutes rather than a halving of readiness time. Measure actual queue and execution time before claiming a result. Related: #13198 introduced the separate chat lane. #12577 refreshes duration estimates; this change leaves that manifest alone. ## What Changed - Increase the general server matrix from five jobs to ten. - Verify the ten-way partition covers the complete server suite when combined with the chat lane. - Document runner demand and the unchanged local and PR grouping. ## Verification - `node --test scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs`: 29 passed. - `actionlint .github/workflows/release-verify.yml`: passed. - Full local `pnpm -r typecheck` and `pnpm build`: passed. - All latest-head GitHub CI checks passed, including the complete Linux test partition, build, typecheck, and browser gates. Greptile: 5/5 with zero open findings. - [Ten-shard timing probe](https://github.com/paperclipai/paperclip/actions/runs/34606772388): all 16 jobs passed; slowest server job 6m 23s versus 10m 38s in the earlier five-shard sample. This compares the server lane, not total readiness, and is not a controlled same-source A/B. - The full local `pnpm test:run` is also running. It has reproduced previously observed macOS-only failures in unchanged skill-cache and native-session suites; the corresponding Linux CI suites passed. Final local results will be attached separately. No affected-workflow test failed. ## Risks Five additional concurrent jobs per release verification run increase runner demand and repeated setup work. Queueing can offset the gain. Test workers, timeouts, permissions, and readiness requirements stay unchanged. Revert the matrix and its partition test to restore the previous split. ## Model Used OpenAI GPT-6 / Codex, with reasoning, tool use, and code execution. The exact serving model identifier and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the affected workflow tests locally and they pass; full-suite macOS limitations are disclosed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
320 lines
9.8 KiB
YAML
320 lines
9.8 KiB
YAML
name: Release Verify
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
ref:
|
|
description: Commit SHA, branch, or tag to verify
|
|
required: true
|
|
type: string
|
|
|
|
jobs:
|
|
runner_chaos_evals:
|
|
name: Pre-release Runner chaos evals
|
|
uses: ./.github/workflows/runner-chaos-evals.yml
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
typecheck:
|
|
name: Typecheck
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
- name: Validate release package manifest
|
|
run: node ./scripts/release-package-map.mjs check
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Typecheck
|
|
run: pnpm -r typecheck
|
|
|
|
general_tests:
|
|
name: General tests (${{ matrix.group_label }})
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
# Split the long chat file by collected test locations, and balance
|
|
# the remaining server files across ten runners. Normal PR/local
|
|
# invocations retain their complete general-server group.
|
|
- group: general-server-without-chat
|
|
group_label: server (1/10)
|
|
shard_index: 0
|
|
shard_count: 10
|
|
- group: general-server-without-chat
|
|
group_label: server (2/10)
|
|
shard_index: 1
|
|
shard_count: 10
|
|
- group: general-server-without-chat
|
|
group_label: server (3/10)
|
|
shard_index: 2
|
|
shard_count: 10
|
|
- group: general-server-without-chat
|
|
group_label: server (4/10)
|
|
shard_index: 3
|
|
shard_count: 10
|
|
- group: general-server-without-chat
|
|
group_label: server (5/10)
|
|
shard_index: 4
|
|
shard_count: 10
|
|
- group: general-server-without-chat
|
|
group_label: server (6/10)
|
|
shard_index: 5
|
|
shard_count: 10
|
|
- group: general-server-without-chat
|
|
group_label: server (7/10)
|
|
shard_index: 6
|
|
shard_count: 10
|
|
- group: general-server-without-chat
|
|
group_label: server (8/10)
|
|
shard_index: 7
|
|
shard_count: 10
|
|
- group: general-server-without-chat
|
|
group_label: server (9/10)
|
|
shard_index: 8
|
|
shard_count: 10
|
|
- group: general-server-without-chat
|
|
group_label: server (10/10)
|
|
shard_index: 9
|
|
shard_count: 10
|
|
- group: general-chat
|
|
group_label: chat (1/3)
|
|
shard_index: 0
|
|
shard_count: 3
|
|
- group: general-chat
|
|
group_label: chat (2/3)
|
|
shard_index: 1
|
|
shard_count: 3
|
|
- group: general-chat
|
|
group_label: chat (3/3)
|
|
shard_index: 2
|
|
shard_count: 3
|
|
# Keep parity with pr.yml: workspaces-a is split with Vitest's
|
|
# native --shard because the ui project dominates the lane.
|
|
- group: general-workspaces-a
|
|
group_label: workspaces-a (1/2)
|
|
shard_index: 0
|
|
shard_count: 2
|
|
- group: general-workspaces-a
|
|
group_label: workspaces-a (2/2)
|
|
shard_index: 1
|
|
shard_count: 2
|
|
- group: general-workspaces-b
|
|
group_label: workspaces-b
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Run grouped general test suites
|
|
run: |
|
|
if [ -n "${{ matrix.shard_count }}" ]; then
|
|
pnpm test:run:general -- --group '${{ matrix.group }}' \
|
|
--shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
|
|
else
|
|
pnpm test:run:general -- --group '${{ matrix.group }}'
|
|
fi
|
|
|
|
serialized_tests:
|
|
name: Serialized tests (${{ matrix.shard_label }})
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- shard_index: 0
|
|
shard_count: 5
|
|
shard_label: 1/5
|
|
- shard_index: 1
|
|
shard_count: 5
|
|
shard_label: 2/5
|
|
- shard_index: 2
|
|
shard_count: 5
|
|
shard_label: 3/5
|
|
- shard_index: 3
|
|
shard_count: 5
|
|
shard_label: 4/5
|
|
- shard_index: 4
|
|
shard_count: 5
|
|
shard_label: 5/5
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Run serialized server test shard
|
|
run: pnpm test:run:serialized -- --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
|
|
|
|
runner_workflow_evals:
|
|
name: Runner workflow eval scorer contract
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Run deterministic Runner workflow scorer tests
|
|
run: pnpm test:runner-workflow-evals
|
|
|
|
verify_paperclip_runner:
|
|
name: Verify Paperclip Runner
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
- name: Select the pinned Runner Rust toolchain
|
|
working-directory: packages/paperclip-runner
|
|
run: |
|
|
set -euo pipefail
|
|
rustup show
|
|
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
|
|
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
|
|
|
|
- name: Cache Runner Rust dependencies
|
|
# Restore and save only within trusted master-push verification. GitHub
|
|
# isolates branch/PR caches from master; other callers compile afresh.
|
|
if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
|
with:
|
|
workspaces: packages/paperclip-runner/runner -> target
|
|
shared-key: release-runner-v1
|
|
# Rebuild workspace code and rerun every check. Cache only compiled
|
|
# dependencies; never restore installed executables from cargo/bin.
|
|
cache-workspace-crates: false
|
|
cache-bin: false
|
|
# The step guard also restricts restores. Save only after a successful
|
|
# master-push verification of that push's exact commit.
|
|
save-if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Verify Paperclip Runner
|
|
run: pnpm --filter @paperclipai/paperclip-runner check:all
|
|
|
|
build:
|
|
name: Build
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
persist-credentials: false
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Build
|
|
run: pnpm build
|