mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 00:05:49 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI must check changes without making contributors wait for repeated setup. > - A complete CI profile showed 158 serialized Vitest launches and repeated dependency installs. > - A large fixture also transferred 40,000 files to prove two different size boundaries. > - This PR removes repeated controller, install, build, transfer, and wait work. > - It keeps isolated test processes and all useful regression boundaries. > - Paired command benchmarks check speed and test inventories before we claim a benefit. ## Linked Issues or Issue Description Refs #16423 for the complete CI profiling data path. No duplicate optimization PR was found. **Current behavior** Each serialized file starts a new Vitest controller. Each CI lane restores the pnpm store and installs dependencies. Large Git scanning and staging share one 40,000-file fixture. A readiness regression waits over a minute for the default database idle deadline. **Proposed behavior** Use one controller per serialized shard, with a fresh fork for each file. Restore exact installed dependencies and certified shared/SDK build outputs when inputs match. Keep the normal install and build paths on a miss. Separate the 32 MiB Git scan proof from staging above command argument limits. Cross a real database idle deadline with a one-second fixture setting. **Reason and benefit** The paired Linux CI benchmark reduced the complete 566-case serialized shard from 303.43s to 246.96s (18.6%). The Git/database fixture command fell from 156.54s to 33.78s (78.4%). Exact dependency/build restoration reduced setup from 56s to 19s (66.1%). These are measured command/setup times. They do not establish a controlled reduction in the full workflow's queue-to-completion time. **Breaking changes** None to application behavior. Tests still run in full. Test results are not cached. Cache misses preserve the existing lockfile resolution fallback. ## What Changed - Batch serialized files into one isolated, single-worker Vitest invocation per shard. Give each worker its own home, config and instance ID. - Add a real two-order isolation check for processes, environment, globals, timers and filesystem state. - Add exact dependency keys for all manifests, lockfile, workspace settings, patches, installer, runtime, platform, libc, pnpm, store and checkout path. Cache only installed package directories, not source fixtures. - Add a separate content key for the shared/SDK dependency graph. Retain the existing source/output fingerprint verifier and rebuild path. Only an exact canonical master push can populate normal caches. Stale lockfile fallback runs cannot publish entries. - Keep four real Git filename lanes above 32 MiB. Use a separate 4,000-file staging/deletion case above 2 MiB. Check every staged file, unusual names, cleanup and bounded commands. - Configure a real one-second PostgreSQL idle timeout in the readiness fixture. Assert that the backend connection changes and that another connection can write the parent row. - Add an opt-in, unprivileged Linux benchmark under `profile-ci`. It compares the tested merge tree with its actual first parent on one runner. It also removes installed outputs and qualifies a real cache restore using temporary PR-scoped entries. ## Verification - Local Node 24.19.0: unchanged serialized shard 255.69s; candidate 231.44s; all 566 cases pass in both. - Local fixture command: unchanged 189.13s; candidate 43.39s. Candidate scan 18.58s, staging 13.41s, database case 2.80s. All three candidate cases pass. - Negative control: moving readiness back inside the real parent-lock transaction fails the shortened regression at the parent-write assertion. The production source was restored and is absent from the diff. - Worker isolation check passes in both file orders. - 565 CI policy, partition and workflow regression checks pass with local HTTP/IPC access enabled. The first sandboxed attempt was denied listen access; it was not a code failure. - `actionlint` passes for the three changed workflows. - [All 48 main CI jobs passed](https://github.com/paperclipai/paperclip/actions/runs/38745643045) on head `5bec2edb2b194e74352ca697f3fd1d0ac03ce967`, including full typecheck, tests, build, browser E2E and release checks. All 158 serialized files and 3,250 cases passed through 9 controller invocations. Full typecheck/test/build were verified in CI; they were not all repeated locally. - [Paired Linux benchmark passed](https://github.com/paperclipai/paperclip/actions/runs/38745643127) on Node 24.21.0. Its baseline is `57fa6b358de62922cfa9a67766b2a227c434fcdf`; its candidate merge tree is `f68f1b32966090f11647502e31852130c20237cf`. The artifact contains results, exact case inventories, setup times and raw logs. Serialized case identities match exactly. All three fixture contracts execute in both variants. | Measured work | Baseline | Candidate | Reduction | |---|---:|---:|---:| | Complete serialized shard, 566 cases | 303.43s | 246.96s | 18.6% | | Git/database fixture command | 156.54s | 33.78s | 78.4% | | Dependency/build setup, cold vs exact hit | 56s | 19s | 66.1% | - [Honeycomb profile export succeeded on its second attempt](https://github.com/paperclipai/paperclip/actions/runs/38747237615/attempts/2). The first attempt failed on GitHub HTTP 503 while downloading an artifact, before export. Only the export was retried; tests were not repeated. The temporary PR-scoped qualification caches were removed after validation, reclaiming 3.34 GB. Timing artifacts remain available. - Greptile is 5/5 on the current head, with no unresolved review threads. Its P1 checkout-prefix finding was fixed in `5bec2edb2`; the corrected reporter paths and paired comparison passed. Earlier runs were superseded while this was fixed. A copy-batching experiment did not speed up the Git fixture and was replaced by the two-boundary fixture split. ## Risks - A persistent controller retains transform state longer. The local shard's maximum RSS rose from about 1.52 GB to 1.99 GB. Full CI must confirm all shards fit the runner budget. - Materialized caches are larger than metadata caches. The Linux benchmark measured real archive transfer and validation after removing installed outputs. Its hit took 19s versus 56s cold. Normal PR lanes cannot publish cache entries. Exact misses run installation. Canonical cache warming requires a frozen-lockfile install; the existing stale-lockfile fallback remains uncached. - The PR uses the trusted reusable workflow from master. The paired benchmark directly exercises the candidate dependency action; normal workflow adoption still requires this PR to merge. This PR does not change the trust references. The completed main workflow took 11m02 including queue time. No controlled full-workflow latency improvement is claimed before this workflow adopts the cache change. - Only deterministic dependency/build outputs are reused. Broad dependency-based test omission and caching successful test results are deliberately excluded from this change. - The real Runner lease minimum and cold-admission waits remain. They protect protocol boundaries and were not shortened by weakening production limits. ## Model Used OpenAI GPT-6 through Codex, with reasoning, shell execution and code tools. The runtime does not expose a more specific deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
703 lines
34 KiB
YAML
703 lines
34 KiB
YAML
name: Release Verify
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
ref:
|
|
description: Commit SHA, branch, or tag to verify
|
|
required: true
|
|
type: string
|
|
runner_binaries_only:
|
|
description: Build exact-source release daemons without publishing or provider calls
|
|
required: false
|
|
default: false
|
|
type: boolean
|
|
outputs:
|
|
runner_artifact_prefix:
|
|
value: ${{ jobs.runner_binaries.outputs.artifact_prefix }}
|
|
native_consumer_matrix:
|
|
value: ${{ jobs.runner_binary_targets.outputs.native_consumers }}
|
|
|
|
# Caller-provided refs may name unmerged PR code. AWS is eligible only when
|
|
# the caller runs on canonical master and verifies that event's exact SHA.
|
|
# The organization group also restricts these workflow files to master.
|
|
jobs:
|
|
runner_chaos_evals:
|
|
if: ${{ !inputs.runner_binaries_only }}
|
|
name: Pre-release Runner chaos evals
|
|
uses: ./.github/workflows/runner-chaos-evals.yml
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
typecheck:
|
|
if: ${{ !inputs.runner_binaries_only }}
|
|
name: Typecheck
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
package-manager-cache: false
|
|
|
|
# rust-cache hashes its absolute cache paths into the entry's version,
|
|
# and GitHub only serves an entry whose key and version both match. The
|
|
# target directory sits under the checkout, and the checkout root
|
|
# differs by runner: /home/runner/_work on the RunsOn fleets that write
|
|
# this cache against /home/runner/work on GitHub-hosted runners. Every
|
|
# key input agreed, yet all 25 completed pull requests on 2026-09-28
|
|
# logged "No cache found" (run 36424309181) and cold-compiled every
|
|
# crate for ~4 minutes in four lanes. Point rust-cache at the same
|
|
# directory through a checkout-independent path so both layouts hash
|
|
# the same version. Keep this block identical in both workflows: the
|
|
# reader and the writer must agree or the version matches nothing.
|
|
- name: Pin the Runner Rust workspace path
|
|
id: runner_rust_workspace
|
|
run: |
|
|
set -euo pipefail
|
|
pinned="$HOME/paperclip-runner-rust"
|
|
# A symlink here is removed as a link, never followed into the checkout.
|
|
rm -rf "$pinned"
|
|
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
|
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Select the pinned Runner Rust toolchain
|
|
working-directory: packages/paperclip-runner
|
|
run: |
|
|
set -euo pipefail
|
|
rustup show
|
|
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
|
|
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
|
|
|
|
- name: Cache typecheck Rust dependencies
|
|
# Restore and save only within trusted master-push verification. GitHub
|
|
# isolates branch/PR caches from master; other callers compile afresh.
|
|
if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
|
with:
|
|
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
|
shared-key: release-typecheck-v2
|
|
# Rebuild workspace code and rerun every check. Cache only compiled
|
|
# dependencies; never restore installed executables from cargo/bin.
|
|
cache-workspace-crates: false
|
|
cache-bin: false
|
|
# The step guard also restricts restores. Save only after a successful
|
|
# master-push verification of that push's exact commit.
|
|
save-if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
|
|
- name: Validate release package manifest
|
|
run: node ./scripts/release-package-map.mjs check
|
|
|
|
- name: Restore materialized dependencies and shared builds
|
|
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
|
|
uses: ./.github/actions/setup-ci-dependencies
|
|
with:
|
|
save-cache: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
|
|
- name: Install dependencies
|
|
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Typecheck
|
|
run: pnpm -r typecheck
|
|
|
|
general_tests:
|
|
if: ${{ !inputs.runner_binaries_only }}
|
|
name: General tests (${{ matrix.group_label }})
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
# Split the long chat file by collected test locations, and balance
|
|
# the remaining server files across ten runners. Rust-backed server
|
|
# suites run in the cached Runner lane below. Local invocations retain
|
|
# their complete general-server group.
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (1/10)
|
|
shard_index: 0
|
|
shard_count: 10
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (2/10)
|
|
shard_index: 1
|
|
shard_count: 10
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (3/10)
|
|
shard_index: 2
|
|
shard_count: 10
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (4/10)
|
|
shard_index: 3
|
|
shard_count: 10
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (5/10)
|
|
shard_index: 4
|
|
shard_count: 10
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (6/10)
|
|
shard_index: 5
|
|
shard_count: 10
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (7/10)
|
|
shard_index: 6
|
|
shard_count: 10
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (8/10)
|
|
shard_index: 7
|
|
shard_count: 10
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (9/10)
|
|
shard_index: 8
|
|
shard_count: 10
|
|
- group: general-server-without-chat-or-native-runner
|
|
group_label: server (10/10)
|
|
shard_index: 9
|
|
shard_count: 10
|
|
- group: general-chat
|
|
group_label: chat (1/3)
|
|
shard_index: 0
|
|
shard_count: 3
|
|
- group: general-chat
|
|
group_label: chat (2/3)
|
|
shard_index: 1
|
|
shard_count: 3
|
|
- group: general-chat
|
|
group_label: chat (3/3)
|
|
shard_index: 2
|
|
shard_count: 3
|
|
# Keep parity with pr.yml: workspaces-a is split with Vitest's
|
|
# native --shard because the ui project dominates the lane.
|
|
- group: general-workspaces-a
|
|
group_label: workspaces-a (1/2)
|
|
shard_index: 0
|
|
shard_count: 2
|
|
- group: general-workspaces-a
|
|
group_label: workspaces-a (2/2)
|
|
shard_index: 1
|
|
shard_count: 2
|
|
- group: general-workspaces-b
|
|
group_label: workspaces-b
|
|
# Keep parity with pr.yml: the db project runs as its own lane.
|
|
- group: general-workspaces-c
|
|
group_label: workspaces-c
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Run grouped general test suites
|
|
run: |
|
|
if [ -n "${{ matrix.shard_count }}" ]; then
|
|
pnpm test:run:general -- --group '${{ matrix.group }}' \
|
|
--shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
|
|
else
|
|
pnpm test:run:general -- --group '${{ matrix.group }}'
|
|
fi
|
|
|
|
serialized_tests:
|
|
if: ${{ !inputs.runner_binaries_only }}
|
|
name: Serialized tests (${{ matrix.shard_label }})
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- shard_index: 0
|
|
shard_count: 5
|
|
shard_label: 1/5
|
|
- shard_index: 1
|
|
shard_count: 5
|
|
shard_label: 2/5
|
|
- shard_index: 2
|
|
shard_count: 5
|
|
shard_label: 3/5
|
|
- shard_index: 3
|
|
shard_count: 5
|
|
shard_label: 4/5
|
|
- shard_index: 4
|
|
shard_count: 5
|
|
shard_label: 5/5
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Run serialized server test shard
|
|
run: pnpm test:run:serialized -- --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
|
|
|
|
runner_workflow_evals:
|
|
if: ${{ !inputs.runner_binaries_only }}
|
|
name: Runner workflow eval scorer contract
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 10
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Run deterministic Runner workflow scorer tests
|
|
run: pnpm test:runner-workflow-evals
|
|
|
|
verify_paperclip_runner:
|
|
if: ${{ !inputs.runner_binaries_only }}
|
|
name: Verify Paperclip Runner (${{ matrix.lane }})
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- lane: protocol
|
|
checks: check:eval-kernel check:protocol
|
|
- lane: rust
|
|
checks: check:runner check:api-authority
|
|
- lane: server-integration
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
cache: pnpm
|
|
|
|
# rust-cache hashes its absolute cache paths into the entry's version,
|
|
# and GitHub only serves an entry whose key and version both match. The
|
|
# target directory sits under the checkout, and the checkout root
|
|
# differs by runner: /home/runner/_work on the RunsOn fleets that write
|
|
# this cache against /home/runner/work on GitHub-hosted runners. Every
|
|
# key input agreed, yet all 25 completed pull requests on 2026-09-28
|
|
# logged "No cache found" (run 36424309181) and cold-compiled every
|
|
# crate for ~4 minutes in four lanes. Point rust-cache at the same
|
|
# directory through a checkout-independent path so both layouts hash
|
|
# the same version. Keep this block identical in both workflows: the
|
|
# reader and the writer must agree or the version matches nothing.
|
|
- name: Pin the Runner Rust workspace path
|
|
id: runner_rust_workspace
|
|
run: |
|
|
set -euo pipefail
|
|
pinned="$HOME/paperclip-runner-rust"
|
|
# A symlink here is removed as a link, never followed into the checkout.
|
|
rm -rf "$pinned"
|
|
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
|
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Select the pinned Runner Rust toolchain
|
|
working-directory: packages/paperclip-runner
|
|
run: |
|
|
set -euo pipefail
|
|
rustup show
|
|
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
|
|
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
|
|
|
|
# rust-cache hashes every installed toolchain into the cache key, not
|
|
# only the active one. Each runner image also ships its own stable
|
|
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
|
|
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
|
|
# therefore still computed different keys, and every GitHub-hosted
|
|
# pull request missed this cache while the fleet hit it. Remove every
|
|
# toolchain except the pin, so the key depends on the pinned compiler
|
|
# and the lockfile alone rather than on what the image happens to
|
|
# carry. Keep this block identical in both workflows: the reader and
|
|
# the writer must agree or the key matches nothing.
|
|
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
|
|
if [ -n "$extra_toolchains" ]; then
|
|
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|
|
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
|
|
fi
|
|
rustup toolchain list
|
|
|
|
- name: Cache Runner Rust dependencies
|
|
# Restore and save only within trusted master-push verification. GitHub
|
|
# isolates branch/PR caches from master; other callers compile afresh.
|
|
if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
|
with:
|
|
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
|
|
shared-key: release-runner-v2
|
|
# Rebuild workspace code and rerun every check. Cache only compiled
|
|
# dependencies; never restore installed executables from cargo/bin.
|
|
cache-workspace-crates: false
|
|
cache-bin: false
|
|
# All lanes restore the existing dependency cache. Only the Rust
|
|
# lane saves it, after warming both release and debug dependencies.
|
|
save-if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Build native server test binaries
|
|
if: ${{ matrix.lane == 'server-integration' }}
|
|
timeout-minutes: 10
|
|
working-directory: packages/paperclip-runner
|
|
run: |
|
|
set -euo pipefail
|
|
pnpm build:rust
|
|
cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd --bin fake-codex-app-server
|
|
|
|
- name: Run native server integration suites
|
|
if: ${{ matrix.lane == 'server-integration' }}
|
|
run: pnpm test:run:general -- --group general-server-native-runner
|
|
|
|
- name: Verify Paperclip Runner
|
|
if: ${{ matrix.lane != 'server-integration' }}
|
|
env:
|
|
RUNNER_CHECKS: ${{ matrix.checks }}
|
|
run: |
|
|
set -euo pipefail
|
|
for check in $RUNNER_CHECKS; do
|
|
pnpm --filter @paperclipai/paperclip-runner "$check"
|
|
done
|
|
|
|
- name: Warm debug dependencies for the shared Runner cache
|
|
# Protocol tests need debug binaries. Populate their dependencies in
|
|
# the sole cache writer, so a cold save also serves the protocol lane.
|
|
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
run: pnpm --filter @paperclipai/paperclip-runner build:rust
|
|
|
|
# The dependency cache above deliberately leaves the Runner's own crate
|
|
# out, so every pull request build compiled paperclip-runner-core in
|
|
# release mode (28-49s) although most pushes leave its inputs alone.
|
|
# This master push saves that crate's release artifacts for the readers
|
|
# in pr-trusted.yml ("Restore the Runner release crate") under a key
|
|
# made from the compiler, the checkout root and a hash of every input
|
|
# the build reads, so a reader only ever restores output of inputs
|
|
# byte-identical to its own. Same writer gate as the dependency cache:
|
|
# the Rust lane of an own-repository master push. The entry is looked
|
|
# up first; a push that left the inputs alone already has it. On a
|
|
# miss the crate is cleaned and built with the exact command the
|
|
# readers run, so the artifacts carry no test-profile variants. The
|
|
# path list and the key live in scripts/runner-release-crate-cache.mjs
|
|
# and must match the readers'; a guard test holds them together.
|
|
- name: Compute the Runner release crate cache key
|
|
id: runner_release_crate_key
|
|
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Check for this push's Runner release crate
|
|
id: runner_release_crate
|
|
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
lookup-only: true
|
|
path: |
|
|
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
|
|
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
|
|
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
|
|
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
|
|
packages/paperclip-runner/runner/target/release/paperclip-runnerd
|
|
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
|
|
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
|
|
|
|
- name: Build the Runner release crate for its cache
|
|
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha && steps.runner_release_crate.outputs.cache-hit != 'true' }}
|
|
working-directory: packages/paperclip-runner
|
|
run: |
|
|
set -euo pipefail
|
|
cargo clean --release --manifest-path runner/Cargo.toml -p paperclip-runner-core
|
|
pnpm build:binary
|
|
|
|
- name: Save the Runner release crate
|
|
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha && steps.runner_release_crate.outputs.cache-hit != 'true' }}
|
|
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
path: |
|
|
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
|
|
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
|
|
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
|
|
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
|
|
packages/paperclip-runner/runner/target/release/paperclip-runnerd
|
|
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
|
|
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
|
|
|
|
build:
|
|
if: ${{ !inputs.runner_binaries_only }}
|
|
name: Build
|
|
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
|
timeout-minutes: 20
|
|
permissions:
|
|
contents: read
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
persist-credentials: false
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
|
with:
|
|
version: 9.15.4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
|
with:
|
|
node-version: 24
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
|
|
- name: Build
|
|
run: pnpm build
|
|
|
|
runner_binary_targets:
|
|
if: inputs.runner_binaries_only
|
|
name: Read exact-source daemon targets
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 5
|
|
permissions:
|
|
contents: read
|
|
outputs:
|
|
matrix: ${{ steps.targets.outputs.matrix }}
|
|
native_consumers: ${{ steps.targets.outputs.native_consumers }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
persist-credentials: false
|
|
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
- name: Preserve the immutable source's platform contract
|
|
id: targets
|
|
env:
|
|
SOURCE_SHA: ${{ inputs.ref }}
|
|
run: |
|
|
set -euo pipefail
|
|
[[ "$SOURCE_SHA" =~ ^[a-f0-9]{40}$ ]] || exit 1
|
|
[[ "$(git rev-parse HEAD)" == "$SOURCE_SHA" ]] || exit 1
|
|
node --input-type=module <<'NODE'
|
|
import assert from 'node:assert/strict';
|
|
import { execFileSync } from 'node:child_process';
|
|
import { appendFileSync } from 'node:fs';
|
|
const source = process.env.SOURCE_SHA;
|
|
const readBlob = file => {
|
|
const entries = execFileSync('git', ['ls-tree', '-z', source, '--', file], {
|
|
encoding: 'utf8', maxBuffer: 4096,
|
|
}).split('\0').filter(Boolean);
|
|
if (!entries.length) return undefined;
|
|
assert.equal(entries.length, 1, 'Ambiguous exact-source target contract');
|
|
assert.ok(entries[0].startsWith('100644 blob ') && entries[0].endsWith('\t' + file), 'Target contract must be a regular source file');
|
|
return execFileSync('git', ['show', source + ':' + file], { encoding: 'utf8', maxBuffer: 64 * 1024 });
|
|
};
|
|
const data = readBlob('packages/paperclip-runner/release-runner-targets.json');
|
|
let declared;
|
|
if (data !== undefined) {
|
|
const contract = JSON.parse(data);
|
|
assert.ok(contract && contract.schema === 'paperclip.runner.release-targets.v1'
|
|
&& Object.keys(contract).sort().join() === 'schema,targets', 'Unknown exact-source daemon platform contract');
|
|
declared = contract.targets;
|
|
} else {
|
|
const legacy = readBlob('packages/paperclip-runner/scripts/stage-release-runner-binaries.mjs');
|
|
const declarations = [...(legacy ?? '').matchAll(/^[ \t]*export const RELEASE_RUNNER_TARGETS = (?:Object\.freeze\()?(\[[^\r\n]*\])\)?;[ \t]*$/gm)];
|
|
assert.equal(declarations.length, 1, 'Unknown exact-source daemon platform contract');
|
|
declared = JSON.parse(declarations[0][1]);
|
|
}
|
|
const historical = ['darwin-arm64', 'darwin-x64', 'linux-x64'];
|
|
const current = [...historical, 'linux-arm64'];
|
|
assert.ok(Array.isArray(declared), 'Unknown exact-source daemon platform contract');
|
|
const matches = expected => JSON.stringify([...declared].sort()) === JSON.stringify([...expected].sort());
|
|
assert.ok(matches(historical) || matches(current), 'Unknown exact-source daemon platform contract');
|
|
const targets = matches(current) ? current : historical;
|
|
const hosts = { 'linux-x64': targets.length === current.length ? 'ubuntu-24.04' : 'ubuntu-latest', 'linux-arm64': 'ubuntu-24.04-arm', 'darwin-arm64': 'macos-15', 'darwin-x64': 'macos-15-intel' };
|
|
const include = targets.map(target => ({ target, host: hosts[target] }));
|
|
appendFileSync(process.env.GITHUB_OUTPUT, `matrix=${JSON.stringify({ include })}\nnative_consumers=${JSON.stringify({ include: include.filter(row => row.target !== 'linux-x64') })}\n`);
|
|
NODE
|
|
|
|
runner_binaries:
|
|
if: inputs.runner_binaries_only
|
|
needs: runner_binary_targets
|
|
name: Release daemon (${{ matrix.target }})
|
|
runs-on: ${{ matrix.host }}
|
|
timeout-minutes: 30
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix: ${{ fromJSON(needs.runner_binary_targets.outputs.matrix) }}
|
|
outputs:
|
|
artifact_prefix: ${{ steps.identity.outputs.artifact_prefix }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
|
|
with:
|
|
ref: ${{ inputs.ref }}
|
|
persist-credentials: false
|
|
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86
|
|
with:
|
|
version: 9.15.4
|
|
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
|
|
with:
|
|
node-version: 24
|
|
- name: Bind immutable producer source and target
|
|
id: identity
|
|
env:
|
|
SOURCE_SHA: ${{ inputs.ref }}
|
|
TARGET: ${{ matrix.target }}
|
|
run: |
|
|
set -euo pipefail
|
|
[[ "$SOURCE_SHA" =~ ^[a-f0-9]{40}$ ]] || exit 1
|
|
[[ "$(git rev-parse HEAD)" == "$SOURCE_SHA" ]] || exit 1
|
|
[[ "$(node -p 'process.platform + "-" + process.arch')" == "$TARGET" ]] || exit 1
|
|
echo "artifact_prefix=runner-release-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" >> "$GITHUB_OUTPUT"
|
|
- name: Pin native Cargo workspace and toolchain
|
|
id: native_rust_workspace
|
|
working-directory: packages/paperclip-runner
|
|
run: |
|
|
set -euo pipefail
|
|
pinned="$HOME/paperclip-runner-rust"
|
|
if [[ -e "$pinned" && ! -L "$pinned" ]]; then
|
|
echo '::error::Native Cargo cache path must not replace a real directory'
|
|
exit 1
|
|
fi
|
|
rm -f "$pinned"
|
|
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
|
|
echo "path=$pinned" >> "$GITHUB_OUTPUT"
|
|
rustup show
|
|
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
|
|
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
|
|
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
|
|
if [[ -n "$extra_toolchains" ]]; then
|
|
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|
|
|| echo '::notice title=Native Rust cache::could not remove an extra toolchain; the cache key may not match'
|
|
fi
|
|
rustup toolchain list
|
|
- name: Cache native Rust dependencies
|
|
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6
|
|
with:
|
|
workspaces: ${{ steps.native_rust_workspace.outputs.path }} -> target
|
|
shared-key: release-native-v2-${{ matrix.host }}-${{ matrix.target }}-${{ hashFiles('packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/src/**', 'packages/paperclip-runner/*.json') }}
|
|
cache-workspace-crates: false
|
|
cache-bin: false
|
|
cache-on-failure: false
|
|
save-if: ${{ github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
|
- name: Install dependencies
|
|
run: pnpm install --no-frozen-lockfile
|
|
- name: Build TypeScript contract and host daemon concurrently
|
|
run: |
|
|
node --input-type=module <<'NODE'
|
|
import { mkdirSync, readFileSync } from 'node:fs';
|
|
import { join } from 'node:path';
|
|
import { startPrefetch, stopPrefetches } from './scripts/prefetch-child.mjs';
|
|
const root = process.cwd(), logs = join(process.env.RUNNER_TEMP, 'native-build-preparation');
|
|
mkdirSync(logs, { recursive: true });
|
|
const builds = [];
|
|
let failure;
|
|
try {
|
|
for (const [name, args, cwd] of [
|
|
['typescript', ['--filter', '@paperclipai/paperclip-runner', 'build:typescript'], root],
|
|
['native', ['build:binary'], join(root, 'packages/paperclip-runner')],
|
|
]) {
|
|
builds.push(startPrefetch({ name, cmd: 'pnpm', args, cwd, env: process.env, logPath: join(logs, name + '.log') }));
|
|
}
|
|
await Promise.all(builds.map(build => build.done));
|
|
} catch (error) { failure = error; }
|
|
finally {
|
|
await stopPrefetches(builds);
|
|
const results = await Promise.allSettled(builds.map(build => build.done));
|
|
builds.forEach((build, index) => {
|
|
console.log('::group::' + build.name + ' build');
|
|
process.stdout.write(readFileSync(join(logs, build.name + '.log'), 'utf8'));
|
|
console.log('::endgroup::');
|
|
console.log(JSON.stringify({ build: build.name, status: results[index].status,
|
|
exitCode: build.child.exitCode, signal: build.child.signalCode, durationMs: build.durationMs }));
|
|
});
|
|
}
|
|
if (failure) throw failure;
|
|
NODE
|
|
- name: Record actual daemon metadata and immutable bytes
|
|
env:
|
|
SOURCE_SHA: ${{ inputs.ref }}
|
|
TARGET: ${{ matrix.target }}
|
|
run: |
|
|
node packages/paperclip-runner/scripts/stage-release-runner-binaries.mjs record "$SOURCE_SHA" "$TARGET" \
|
|
packages/paperclip-runner/dist/bin/paperclip-runnerd "$RUNNER_TEMP/runner-release/$TARGET"
|
|
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
|
|
with:
|
|
name: ${{ steps.identity.outputs.artifact_prefix }}-${{ matrix.target }}
|
|
path: ${{ runner.temp }}/runner-release/
|
|
if-no-files-found: error
|
|
retention-days: 14
|