Files
PaperClipAI/.github/workflows/release-verify.yml
Devin FoleyandPaperclip 4265cb3a2b fix(ci): cache native server integration builds in release verification (#15619)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Release verification must run the same server integration coverage
as PR verification.
> - Three server suites use real Rust Runner binaries.
> - PR checks already run them with the shared Rust dependency cache,
but release checks cold-build them inside ordinary server test shards.
> - A cold Dot Runner build can consume its entire setup deadline before
any test runs.
> - This pull request gives release verification a required cached
native integration lane and preserves every test.

## Linked Issues or Issue Description

**What happened?**

On master commit `1881894973a2b25838d8abed9bd8aeebc3af4441`, [Cloud
readiness run
37837810707](https://github.com/paperclipai/paperclip/actions/runs/37837810707)
failed in server shard 7. Dot Runner's `cargo build --release` exceeded
its 300-second child-process deadline. The other 1,640 tests in that
shard passed. The failed setup then tried to remove an undefined
temporary path and emitted a second error. Cargo output was captured as
an opaque buffer, which obscured build progress.

**What did you expect to happen?**

Build fixture binaries before tests in a lane with the existing Rust
dependency cache. Run all native integration tests and make their result
required for release readiness. A setup failure should keep its original
diagnostic.

**Steps to reproduce**

Run release verification on a clean runner. The existing
`general-server-without-chat` group retains the three Cargo-backed
suites outside the PR workflow. The Dot suite can time out during a cold
release build. Run the new workflow and partition tests against the
previous source to reproduce five routing and coverage failures
deterministically.

**Version / commit**

Observed at `1881894973a2b25838d8abed9bd8aeebc3af4441`; this change is
based on `ed6abbf158b`.

**Deployment mode**

GitHub Actions Release and Cloud readiness verification. No application
runtime or deployment action changes.

Related work: #15581 also edits Dot integration tests for onboarding
behavior. It does not repair release test routing. Searches found no
open PR for this failure.

## What Changed

- Add an explicit server test group that excludes the dedicated chat and
native suites. Preserve the existing PR and local test groups.
- Run all three native server suites in one required matrix lane with
the existing trusted Rust dependency cache.
- Build debug and release fixture binaries in a visible step with a
10-minute limit before tests start. Keep Cargo freshness checks and the
existing test deadlines.
- Stream Cargo diagnostics and clean up safely when Dot setup stops
before creating its temporary directory.
- Test complete, non-overlapping partitions for Release, Cloud
readiness, other callers, and existing PR/local groups. Check cache
restrictions, build order, and the source verification dependency.

## Verification

- Passed 44 workflow and partition tests: `node --test
scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`.
- The same tests fail in five relevant cases against the previous
workflow and selector. They pass after this correction.
- An independent review repeated all 44 tests successfully.
- Passed `actionlint .github/workflows/release-verify.yml`, `git diff
--check`, and a secret scan.
- Passed all 381 workflow policy tests: `node --test
'.github/scripts/tests/*.test.mjs'`.
- Passed full `pnpm build` in a clean worktree.
- Built debug and release fixture binaries, then passed all 32 tests in
the three real native integration suites: `pnpm test:run:general --
--group general-server-native-runner` (49.5 seconds, no skips).
- Passed full `pnpm -r typecheck`.
- Injected a synthetic Cargo setup failure. The suite reports that
failure without the secondary undefined-path cleanup error.
- Exact-head CI completed: 53 successful checks, including all server
shards, both native Runner lanes, build, typecheck, browser tests, and
the canary dry run. Two optional Storybook checks were skipped.
- Started the duplicate local `pnpm test:run` aggregate and stopped it
after exact-head CI passed. No completed local aggregate result is
claimed. All test processes owned by that run exited.
- Greptile scored the final head
`ffe5ab5a28af2dfea9db1c1952c652f070bf52f8` at 5/5 with no review
threads.
- `Superagent Supply Chain Scan` is neutral, not passed: it only
supports exact dependency pin replacements and cannot verify structural
edits to `.github/workflows/release-verify.yml`. It reported no
annotations. Independent source review, Greptile, actionlint, and the
workflow policy tests cover this structural change.
- GitHub still requires a code-owner review for the workflow files.
There are no merge conflicts.

## Risks

- Adds one CI matrix job, which increases concurrent runner demand. The
existing cache writer and trust restrictions stay in place.
- Missing dependencies still compile from the lockfile. A build that
exceeds its step limit fails visibly; failed tests still block source
verification.
- Workflow execution uses the new lane after merge. PR tests validate
the workflow contract and run the existing native test lane.
- The supply chain scanner cannot analyze this workflow structure. Its
neutral result is an explicit coverage limit; it is not counted as a
successful scan.
- No runtime, schema, deployment, publication, credential, or
test-timeout changes.

## Model Used

OpenAI Codex, based on GPT-6, with repository inspection, code
execution, and independent agent review. The exact deployed model ID and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-08 14:33:10 -07:00

443 lines
19 KiB
YAML

name: Release Verify
on:
workflow_call:
inputs:
ref:
description: Commit SHA, branch, or tag to verify
required: true
type: string
# Caller-provided refs may name unmerged PR code. AWS is eligible only when
# the caller runs on canonical master and verifies that event's exact SHA.
# The organization group also restricts these workflow files to master.
jobs:
runner_chaos_evals:
name: Pre-release Runner chaos evals
uses: ./.github/workflows/runner-chaos-evals.yml
with:
ref: ${{ inputs.ref }}
typecheck:
name: Typecheck
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
# rust-cache hashes its absolute cache paths into the entry's version,
# and GitHub only serves an entry whose key and version both match. The
# target directory sits under the checkout, and the checkout root
# differs by runner: /home/runner/_work on the RunsOn fleets that write
# this cache against /home/runner/work on GitHub-hosted runners. Every
# key input agreed, yet all 25 completed pull requests on 2026-09-28
# logged "No cache found" (run 36424309181) and cold-compiled every
# crate for ~4 minutes in four lanes. Point rust-cache at the same
# directory through a checkout-independent path so both layouts hash
# the same version. Keep this block identical in both workflows: the
# reader and the writer must agree or the version matches nothing.
- name: Pin the Runner Rust workspace path
id: runner_rust_workspace
run: |
set -euo pipefail
pinned="$HOME/paperclip-runner-rust"
# A symlink here is removed as a link, never followed into the checkout.
rm -rf "$pinned"
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
echo "path=$pinned" >> "$GITHUB_OUTPUT"
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -euo pipefail
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
- name: Cache typecheck Rust dependencies
# Restore and save only within trusted master-push verification. GitHub
# isolates branch/PR caches from master; other callers compile afresh.
if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
shared-key: release-typecheck-v2
# Rebuild workspace code and rerun every check. Cache only compiled
# dependencies; never restore installed executables from cargo/bin.
cache-workspace-crates: false
cache-bin: false
# The step guard also restricts restores. Save only after a successful
# master-push verification of that push's exact commit.
save-if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
- name: Validate release package manifest
run: node ./scripts/release-package-map.mjs check
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Typecheck
run: pnpm -r typecheck
general_tests:
name: General tests (${{ matrix.group_label }})
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
# Split the long chat file by collected test locations, and balance
# the remaining server files across ten runners. Rust-backed server
# suites run in the cached Runner lane below. Local invocations retain
# their complete general-server group.
- group: general-server-without-chat-or-native-runner
group_label: server (1/10)
shard_index: 0
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (2/10)
shard_index: 1
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (3/10)
shard_index: 2
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (4/10)
shard_index: 3
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (5/10)
shard_index: 4
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (6/10)
shard_index: 5
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (7/10)
shard_index: 6
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (8/10)
shard_index: 7
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (9/10)
shard_index: 8
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (10/10)
shard_index: 9
shard_count: 10
- group: general-chat
group_label: chat (1/3)
shard_index: 0
shard_count: 3
- group: general-chat
group_label: chat (2/3)
shard_index: 1
shard_count: 3
- group: general-chat
group_label: chat (3/3)
shard_index: 2
shard_count: 3
# Keep parity with pr.yml: workspaces-a is split with Vitest's
# native --shard because the ui project dominates the lane.
- group: general-workspaces-a
group_label: workspaces-a (1/2)
shard_index: 0
shard_count: 2
- group: general-workspaces-a
group_label: workspaces-a (2/2)
shard_index: 1
shard_count: 2
- group: general-workspaces-b
group_label: workspaces-b
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Run grouped general test suites
run: |
if [ -n "${{ matrix.shard_count }}" ]; then
pnpm test:run:general -- --group '${{ matrix.group }}' \
--shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
else
pnpm test:run:general -- --group '${{ matrix.group }}'
fi
serialized_tests:
name: Serialized tests (${{ matrix.shard_label }})
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- shard_index: 0
shard_count: 5
shard_label: 1/5
- shard_index: 1
shard_count: 5
shard_label: 2/5
- shard_index: 2
shard_count: 5
shard_label: 3/5
- shard_index: 3
shard_count: 5
shard_label: 4/5
- shard_index: 4
shard_count: 5
shard_label: 5/5
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Run serialized server test shard
run: pnpm test:run:serialized -- --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
runner_workflow_evals:
name: Runner workflow eval scorer contract
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 10
permissions:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Run deterministic Runner workflow scorer tests
run: pnpm test:runner-workflow-evals
verify_paperclip_runner:
name: Verify Paperclip Runner (${{ matrix.lane }})
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- lane: protocol
checks: check:eval-kernel check:protocol
- lane: rust
checks: check:runner check:api-authority
- lane: server-integration
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
# rust-cache hashes its absolute cache paths into the entry's version,
# and GitHub only serves an entry whose key and version both match. The
# target directory sits under the checkout, and the checkout root
# differs by runner: /home/runner/_work on the RunsOn fleets that write
# this cache against /home/runner/work on GitHub-hosted runners. Every
# key input agreed, yet all 25 completed pull requests on 2026-09-28
# logged "No cache found" (run 36424309181) and cold-compiled every
# crate for ~4 minutes in four lanes. Point rust-cache at the same
# directory through a checkout-independent path so both layouts hash
# the same version. Keep this block identical in both workflows: the
# reader and the writer must agree or the version matches nothing.
- name: Pin the Runner Rust workspace path
id: runner_rust_workspace
run: |
set -euo pipefail
pinned="$HOME/paperclip-runner-rust"
# A symlink here is removed as a link, never followed into the checkout.
rm -rf "$pinned"
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
echo "path=$pinned" >> "$GITHUB_OUTPUT"
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -euo pipefail
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
# rust-cache hashes every installed toolchain into the cache key, not
# only the active one. Each runner image also ships its own stable
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
# therefore still computed different keys, and every GitHub-hosted
# pull request missed this cache while the fleet hit it. Remove every
# toolchain except the pin, so the key depends on the pinned compiler
# and the lockfile alone rather than on what the image happens to
# carry. Keep this block identical in both workflows: the reader and
# the writer must agree or the key matches nothing.
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [ -n "$extra_toolchains" ]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Cache Runner Rust dependencies
# Restore and save only within trusted master-push verification. GitHub
# isolates branch/PR caches from master; other callers compile afresh.
if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
shared-key: release-runner-v2
# Rebuild workspace code and rerun every check. Cache only compiled
# dependencies; never restore installed executables from cargo/bin.
cache-workspace-crates: false
cache-bin: false
# All lanes restore the existing dependency cache. Only the Rust
# lane saves it, after warming both release and debug dependencies.
save-if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Build native server test binaries
if: ${{ matrix.lane == 'server-integration' }}
timeout-minutes: 10
working-directory: packages/paperclip-runner
run: |
set -euo pipefail
pnpm build:rust
cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd --bin fake-codex-app-server
- name: Run native server integration suites
if: ${{ matrix.lane == 'server-integration' }}
run: pnpm test:run:general -- --group general-server-native-runner
- name: Verify Paperclip Runner
if: ${{ matrix.lane != 'server-integration' }}
env:
RUNNER_CHECKS: ${{ matrix.checks }}
run: |
set -euo pipefail
for check in $RUNNER_CHECKS; do
pnpm --filter @paperclipai/paperclip-runner "$check"
done
- name: Warm debug dependencies for the shared Runner cache
# Protocol tests need debug binaries. Populate their dependencies in
# the sole cache writer, so a cold save also serves the protocol lane.
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
run: pnpm --filter @paperclipai/paperclip-runner build:rust
build:
name: Build
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
persist-credentials: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Build
run: pnpm build