Files
PaperClipAI/.github/workflows/release-verify.yml
T
DottaandPaperclip d67e3507b2 feat(runner): qualify Codex platform packages and cache hosted builds (#16028)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution needs a compatible installed daemon and provider
dependency.
> - The Codex-default work found missing platform packages and
clean-install startup failures.
> - Release builds also repeat dependency compilation and run
preparation serially.
> - This prerequisite packages four daemon targets and verifies
installed native startup.
> - It also caches hosted Cargo dependencies and overlaps JavaScript
preparation with compilation.
> - Agent creation defaults and experimental gates stay at the current
master behavior.

## Linked Issues or Issue Description

Refs #15683. Refs #15555. Refs #15422. Refs #15926. Refs #15932.

Extract runtime and installation support from #15683. This prerequisite
has 65 changed files. Creation, onboarding, imports, UI runner selection
and graduation remain in the dependent PR. Feature/access maps and the
Storybook source catalog stay synchronized. Existing agents keep their
execution choices.

## What Changed

- Correct the existing runner test fixtures. Reserve owned credential
quorum listeners through the existing bounded fixture helper. Keep
foreign listeners intact, run actual credential staging once, and
preserve its rejection and unchanged-auth assertions. Reuse the
durable-resume outer-limit correction. Wait for actual warm-completion
and held-authorization callback entry under the existing recovery
deadline and surface early request errors. Preserve every semantic
assertion.
- Retain bounded installation failure diagnostics through the existing
qualification receipt upload. Keep the original error, 10-minute
deadline and all success checks.
- Keep startup ownership until credential preparation and registration
drain. Reject duplicate and post-close startup/resume work. Observe
shutdown during normal and adopted authentication readiness. Preserve
the primary failure and retain private state when cleanup is
unconfirmed.
- Package Linux x64/ARM64 and macOS Apple Silicon/Intel daemons. Verify
target headers, source identities, manifests, and hashes before assembly
and installation.
- Reuse the existing release producer/consumer workflows. Cache Cargo
dependencies by target, OS, compiler, lockfile, and contract inputs.
Rebuild workspace binaries. Permit trusted cache writes only for
canonical exact-source master pushes.
- Overlap JavaScript preparation with native compilation. Join both
commands and failure/cancellation cleanup before artifact verification.
- Preserve shell positional arguments in the shared SSH runner and apply
environment assignments correctly. Reject environment option keys. Six
before/after controls and two real SSH subprocess paths pass with zero
provider calls; this fix lands before both Codex and Claude defaults.
- Admit verified suspended SSH backups during cold controller
continuation, using the same full-scope, integrity and lifecycle checks
as sandbox targets. Two positive SSH cases fail before; all 14
positive/negative matrix cases pass afterward. Active ownership and
corrupt/malformed/nonempty state remain rejected.
- Record measured sandbox platform facts through the existing explicit
environment probe. Bind facts to the company, effective provider
configuration, immutable OCI image and actual captured image binding.
Check the returned cleanup receipt. Reject client-authored proof and
stale or named boot identities. Creation and discovery allocate nothing.
Preserve connection-only behavior.
- Resolve the actual remote Codex target and preserve same-host
development overrides. Preserve runtime artifact verification. Use
actual protocol/model compatibility rather than a numerical CLI version
fence.
- Resolve the installed Codex login dependency on 64-bit Linux and Mac
independently of the daemon libc baseline. Preserve account homes and
safe errors.
- Fix actual clean-install TOML table comparison. Preserve strict
semantic checks, comments, private writes, and malformed-input
rejection. Preserve primary startup failures and bounded cleanup.
- Document GNU libc 2.39+ and stock Ubuntu sandbox prerequisites.
Qualify ARM startup with a scoped distribution profile and unchanged
global restrictions. Preserve Windows and Alpine deferral.
- Preserve selected and saved native Codex reasoning effort in the
existing UI builder, while excluding it from other providers. This small
shared correction prevents import edits higher in the stack from
dropping effort.
- Reuse existing installed-package checks and assertions. Paperclip
tarballs contain no Codex native payloads. Official npm dependencies
supply the consumer's Codex binaries. All Docker and native builds stay
remote.

## Verification

Candidate: `271d434c170e74bdedde65d0b13059a55419c943`. It includes
current master `e168512f7bdbb6af6ea01954c6d7ed80b91e903e` and preserves
prior candidates through ordinary commits.

Current [full
CI](https://github.com/paperclipai/paperclip/actions/runs/38346601686)
is successful on attempt 2. GitHub reports 54 successful checks and four
conditional skips, including required verify and browser aggregates,
workspace typecheck/tests/build, Linux DB controls, both runner shards
and [isolated native
compilation](https://github.com/paperclipai/paperclip/actions/runs/38346601609).
Fresh current-head Greptile is 5/5 with all four threads resolved. The
passing build's clean Grok/Codex report records CI merge source
`7a82a37438e6e448abc7cc5caa60dc2519a97be0`, 18 installed packages,
omitted Codex payloads, official host-only npm dependencies and zero
provider calls. Pi setup and its no-prompt lease verification also pass.
The first attempt's single 60-second Node archive download timeout
remains retained; its DNS/TLS/download cause remains unknown. One
unchanged-source retry was requested through the normal Build-job dialog
with debug logging disabled. GitHub's attempt 2 graph reports the other
CI jobs as well; no second retry or source/deadline relaxation was made.

The [final exact-source package
campaign](https://github.com/paperclipai/paperclip/actions/runs/38347027535)
is successful on literal `271d434c170e74bdedde65d0b13059a55419c943`. It
passes all four native producers, public Linux package assembly and
clean installed consumers on Apple Silicon, Intel Mac and Linux ARM64.
The nine qualification-mode skips are intentional; full CI above covers
their ordinary gates. Native builds and Docker assembly ran remotely.
The predecessor [`dae808` package
campaign](https://github.com/paperclipai/paperclip/actions/runs/38342104102)
remains literal historical evidence. Only tests and the dated record
differ from that source; runtime, package and workflow code is
unchanged. The reused credential fixture correction passes all 42
canonical Node 24 controls with synthetic data and independent
matching-source review, and the current hosted runner shard passes.
Production credential behavior is unchanged.

**Merge hold:** current-source CI, fresh 5/5 review and exact-source
package qualification pass. Actual code-owner approval and normal
protected merge remain pending. No slice has merged or deployed. GitHub
CLI authentication needs refresh; the normal signed-in embedded GitHub
UI provided this status without credential workarounds.

- The master merge preserves both Storybook source-review appendices and
the guarded test-home cleanup. All six existing cleanup controls pass
without skips. Master's isolated-home helper/test/setup are
byte-identical. The two existing whitespace lines in master's unrelated
privacy test remain outside this PR's diff.
- Canonical Node 24 passes all 24 existing public-install helper tests.
Three narrow added cases prove bounded/redacted failure metadata,
symlink exclusions, original exception identity, failed diagnostic
writes and no false failure on success. The Mac installer now uses
`--timing`; at most two 16 KiB tails and 32 structured entries per tail
enter the existing receipt upload. Raw argv, URLs, environment, error
text and stacks are omitted. A failed receipt explicitly remains
FAILED/unqualified/runtime-not-attempted. The 600,000 ms install
deadline and every real package/hash/payload/lock/lifecycle/runtime
oracle are unchanged.
- Canonical Node 24 reproduces the durable-resume outer timeout under
controlled fake latency. Only its outer test limit changes to 15
seconds; internal 500 ms/5 second deadlines and usage/effect assertions
remain. Controlled warm-recovery and before-bootstrap authorization
scheduling delays reproduce their callback polling failures, while the
explicit entry signals pass. Early recovery errors surface unchanged.
Independent review is clear. Historical mock binaries support the
fixture controls; fresh full hosted CI remains decisive. The original
hosted callback failure cause is inconclusive.
- The composed runner/surfaces/server/E2E TypeScript checks pass. Map
checks pass 35 areas/171 capability IDs. Token gates pass. The existing
Storybook catalog passes 223 files/2,210 stories; story IDs and
classifications remain intact. The current master had 25 inherited stale
fingerprints from three changed map recipes. Only those guarded
fingerprints and a source-only review note change; all 92 non-map
story/component/fixture inputs remain unchanged. Existing
classifications are retained. This proves source structure, not visual
or live acceptance.
- The 65-file contribution audit has no code file crossing from below
1,000 lines to above the limit. Earlier comment cleanup preserves
executable emissions, parsed workflows and action pins. Independent
cleanup/catalog and three-file reporting reviews are clear. No new
workflow, suite, grader or local native/Docker build was added.

**Prior `dae808` gate evidence:** fresh Greptile is 5/5 with four
resolved threads, isolated native compilation and the browser aggregate
pass. Its first runner shard passes the previous two corrections but
fails a different held-authorization cell at the implicit one-second
polling deadline. It records 1,672 passes/one failure/three skips. This
candidate applies the verified entry-signal correction without changing
production deadlines or any semantic assertions. Its original hosted
cause remains inconclusive until fresh CI.

**Previous-source evidence, kept literal:** the [exact `16a532` package
campaign](https://github.com/paperclipai/paperclip/actions/runs/38338806076)
passes all four native producers, public Linux package assembly, Apple
Silicon, Intel Mac and Linux ARM64 installed consumers. Its fresh review
is 5/5 with all four threads resolved. Full CI records 52 successful
checks, four conditional skips and the two fixture failures corrected
here. This is not a green full-CI candidate or proof for the new source.

`64c074fb8da1b31f8d6042c4fef48dc4164e759b` passed [full
CI](https://github.com/paperclipai/paperclip/actions/runs/38332726938),
54 checks/four conditional skips, [isolated native
compilation](https://github.com/paperclipai/paperclip/actions/runs/38332726787/job/115085895328),
and fresh Greptile 5/5/all four resolved threads. The hosted transport
suite passed 246 tests and startup trust passed 22, closing that
source's local fake-binary limitation. These are historical after
reconciliation.

The [previous exact-source package
campaign](https://github.com/paperclipai/paperclip/actions/runs/38332921463)
built all four platform daemons and passed public Linux package assembly
plus Apple Silicon and Linux ARM64 installed consumers. **Intel Mac
failed:** its initial scripts-disabled npm install timed out before
lifecycle/runtime assertions. The underlying download/resolution/host
cause is unknown because the old helper deleted the npm logs. This is
the concrete baseline for the diagnostic change. The failed campaign
remains failed; no all-platform signoff or automatic retry is claimed.

The earlier `a200495` [four-target
campaign](https://github.com/paperclipai/paperclip/actions/runs/38283695248)
and [scoped ARM
qualification](https://github.com/paperclipai/paperclip/actions/runs/38291535914)
remain component evidence. Cargo restores were cold misses; no
warm-cache speedup is claimed. Paperclip tarballs contain no Codex
payloads. Official npm dependencies supply Codex on the user's platform.

Current-source full workspace typecheck/tests/build, Linux DB controls,
static/Rust/native tests, all browser shards and exact-source package
consumers must pass. New-agent defaults and final managed onboarding
belong to #15683. Actual private Cloud composition, warm/cold signup,
task/follow-up and rollback compatibility remain later release gates.
Source audits found that ordinary managed onboarding does not run the
saved platform probe and Cloud distributes named snapshots without a
verified final boot-image binding. These dependent release blockers
remain open; prequalified E2E fixtures do not close them. Master adds
task-workspace migrations; the final image/migrator and previous-image
checks must include them. ARM paid task/follow-up and ordinary OS APT
installation are not proven.

## Risks

- Multi-target packages require source and header agreement. Missing or
mismatched artifacts fail explicitly. Existing execution choices remain
unchanged.
- Linux requires GNU libc 2.39+ and compatible sandbox prerequisites.
The scoped hosted ARM preparation does not qualify arbitrary containers
or kernels. Windows, 32-bit ARM, and Alpine remain deferred.
- Only dependency crates are cached. Preview jobs cannot write trusted
caches. Concurrent command cleanup must preserve the primary failure.
- Native-free Paperclip packaging retains official Codex npm
dependencies. No Codex payload bundling, schema migration, automatic
agent migration, or production deployment is included.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and parallel agents. The exact serving model snapshot and
context-window size are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-13 06:24:07 -05:00

696 lines
34 KiB
YAML

name: Release Verify
on:
workflow_call:
inputs:
ref:
description: Commit SHA, branch, or tag to verify
required: true
type: string
runner_binaries_only:
description: Build exact-source release daemons without publishing or provider calls
required: false
default: false
type: boolean
outputs:
runner_artifact_prefix:
value: ${{ jobs.runner_binaries.outputs.artifact_prefix }}
native_consumer_matrix:
value: ${{ jobs.runner_binary_targets.outputs.native_consumers }}
# Caller-provided refs may name unmerged PR code. AWS is eligible only when
# the caller runs on canonical master and verifies that event's exact SHA.
# The organization group also restricts these workflow files to master.
jobs:
runner_chaos_evals:
if: ${{ !inputs.runner_binaries_only }}
name: Pre-release Runner chaos evals
uses: ./.github/workflows/runner-chaos-evals.yml
with:
ref: ${{ inputs.ref }}
typecheck:
if: ${{ !inputs.runner_binaries_only }}
name: Typecheck
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
# rust-cache hashes its absolute cache paths into the entry's version,
# and GitHub only serves an entry whose key and version both match. The
# target directory sits under the checkout, and the checkout root
# differs by runner: /home/runner/_work on the RunsOn fleets that write
# this cache against /home/runner/work on GitHub-hosted runners. Every
# key input agreed, yet all 25 completed pull requests on 2026-09-28
# logged "No cache found" (run 36424309181) and cold-compiled every
# crate for ~4 minutes in four lanes. Point rust-cache at the same
# directory through a checkout-independent path so both layouts hash
# the same version. Keep this block identical in both workflows: the
# reader and the writer must agree or the version matches nothing.
- name: Pin the Runner Rust workspace path
id: runner_rust_workspace
run: |
set -euo pipefail
pinned="$HOME/paperclip-runner-rust"
# A symlink here is removed as a link, never followed into the checkout.
rm -rf "$pinned"
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
echo "path=$pinned" >> "$GITHUB_OUTPUT"
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -euo pipefail
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
- name: Cache typecheck Rust dependencies
# Restore and save only within trusted master-push verification. GitHub
# isolates branch/PR caches from master; other callers compile afresh.
if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
shared-key: release-typecheck-v2
# Rebuild workspace code and rerun every check. Cache only compiled
# dependencies; never restore installed executables from cargo/bin.
cache-workspace-crates: false
cache-bin: false
# The step guard also restricts restores. Save only after a successful
# master-push verification of that push's exact commit.
save-if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
- name: Validate release package manifest
run: node ./scripts/release-package-map.mjs check
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Typecheck
run: pnpm -r typecheck
general_tests:
if: ${{ !inputs.runner_binaries_only }}
name: General tests (${{ matrix.group_label }})
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
# Split the long chat file by collected test locations, and balance
# the remaining server files across ten runners. Rust-backed server
# suites run in the cached Runner lane below. Local invocations retain
# their complete general-server group.
- group: general-server-without-chat-or-native-runner
group_label: server (1/10)
shard_index: 0
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (2/10)
shard_index: 1
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (3/10)
shard_index: 2
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (4/10)
shard_index: 3
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (5/10)
shard_index: 4
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (6/10)
shard_index: 5
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (7/10)
shard_index: 6
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (8/10)
shard_index: 7
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (9/10)
shard_index: 8
shard_count: 10
- group: general-server-without-chat-or-native-runner
group_label: server (10/10)
shard_index: 9
shard_count: 10
- group: general-chat
group_label: chat (1/3)
shard_index: 0
shard_count: 3
- group: general-chat
group_label: chat (2/3)
shard_index: 1
shard_count: 3
- group: general-chat
group_label: chat (3/3)
shard_index: 2
shard_count: 3
# Keep parity with pr.yml: workspaces-a is split with Vitest's
# native --shard because the ui project dominates the lane.
- group: general-workspaces-a
group_label: workspaces-a (1/2)
shard_index: 0
shard_count: 2
- group: general-workspaces-a
group_label: workspaces-a (2/2)
shard_index: 1
shard_count: 2
- group: general-workspaces-b
group_label: workspaces-b
# Keep parity with pr.yml: the db project runs as its own lane.
- group: general-workspaces-c
group_label: workspaces-c
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Run grouped general test suites
run: |
if [ -n "${{ matrix.shard_count }}" ]; then
pnpm test:run:general -- --group '${{ matrix.group }}' \
--shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
else
pnpm test:run:general -- --group '${{ matrix.group }}'
fi
serialized_tests:
if: ${{ !inputs.runner_binaries_only }}
name: Serialized tests (${{ matrix.shard_label }})
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- shard_index: 0
shard_count: 5
shard_label: 1/5
- shard_index: 1
shard_count: 5
shard_label: 2/5
- shard_index: 2
shard_count: 5
shard_label: 3/5
- shard_index: 3
shard_count: 5
shard_label: 4/5
- shard_index: 4
shard_count: 5
shard_label: 5/5
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Run serialized server test shard
run: pnpm test:run:serialized -- --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
runner_workflow_evals:
if: ${{ !inputs.runner_binaries_only }}
name: Runner workflow eval scorer contract
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 10
permissions:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Run deterministic Runner workflow scorer tests
run: pnpm test:runner-workflow-evals
verify_paperclip_runner:
if: ${{ !inputs.runner_binaries_only }}
name: Verify Paperclip Runner (${{ matrix.lane }})
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- lane: protocol
checks: check:eval-kernel check:protocol
- lane: rust
checks: check:runner check:api-authority
- lane: server-integration
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
# rust-cache hashes its absolute cache paths into the entry's version,
# and GitHub only serves an entry whose key and version both match. The
# target directory sits under the checkout, and the checkout root
# differs by runner: /home/runner/_work on the RunsOn fleets that write
# this cache against /home/runner/work on GitHub-hosted runners. Every
# key input agreed, yet all 25 completed pull requests on 2026-09-28
# logged "No cache found" (run 36424309181) and cold-compiled every
# crate for ~4 minutes in four lanes. Point rust-cache at the same
# directory through a checkout-independent path so both layouts hash
# the same version. Keep this block identical in both workflows: the
# reader and the writer must agree or the version matches nothing.
- name: Pin the Runner Rust workspace path
id: runner_rust_workspace
run: |
set -euo pipefail
pinned="$HOME/paperclip-runner-rust"
# A symlink here is removed as a link, never followed into the checkout.
rm -rf "$pinned"
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
echo "path=$pinned" >> "$GITHUB_OUTPUT"
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -euo pipefail
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
# rust-cache hashes every installed toolchain into the cache key, not
# only the active one. Each runner image also ships its own stable
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
# therefore still computed different keys, and every GitHub-hosted
# pull request missed this cache while the fleet hit it. Remove every
# toolchain except the pin, so the key depends on the pinned compiler
# and the lockfile alone rather than on what the image happens to
# carry. Keep this block identical in both workflows: the reader and
# the writer must agree or the key matches nothing.
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [ -n "$extra_toolchains" ]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Cache Runner Rust dependencies
# Restore and save only within trusted master-push verification. GitHub
# isolates branch/PR caches from master; other callers compile afresh.
if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
shared-key: release-runner-v2
# Rebuild workspace code and rerun every check. Cache only compiled
# dependencies; never restore installed executables from cargo/bin.
cache-workspace-crates: false
cache-bin: false
# All lanes restore the existing dependency cache. Only the Rust
# lane saves it, after warming both release and debug dependencies.
save-if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Build native server test binaries
if: ${{ matrix.lane == 'server-integration' }}
timeout-minutes: 10
working-directory: packages/paperclip-runner
run: |
set -euo pipefail
pnpm build:rust
cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd --bin fake-codex-app-server
- name: Run native server integration suites
if: ${{ matrix.lane == 'server-integration' }}
run: pnpm test:run:general -- --group general-server-native-runner
- name: Verify Paperclip Runner
if: ${{ matrix.lane != 'server-integration' }}
env:
RUNNER_CHECKS: ${{ matrix.checks }}
run: |
set -euo pipefail
for check in $RUNNER_CHECKS; do
pnpm --filter @paperclipai/paperclip-runner "$check"
done
- name: Warm debug dependencies for the shared Runner cache
# Protocol tests need debug binaries. Populate their dependencies in
# the sole cache writer, so a cold save also serves the protocol lane.
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
run: pnpm --filter @paperclipai/paperclip-runner build:rust
# The dependency cache above deliberately leaves the Runner's own crate
# out, so every pull request build compiled paperclip-runner-core in
# release mode (28-49s) although most pushes leave its inputs alone.
# This master push saves that crate's release artifacts for the readers
# in pr-trusted.yml ("Restore the Runner release crate") under a key
# made from the compiler, the checkout root and a hash of every input
# the build reads, so a reader only ever restores output of inputs
# byte-identical to its own. Same writer gate as the dependency cache:
# the Rust lane of an own-repository master push. The entry is looked
# up first; a push that left the inputs alone already has it. On a
# miss the crate is cleaned and built with the exact command the
# readers run, so the artifacts carry no test-profile variants. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the readers'; a guard test holds them together.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Check for this push's Runner release crate
id: runner_release_crate
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
lookup-only: true
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Build the Runner release crate for its cache
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha && steps.runner_release_crate.outputs.cache-hit != 'true' }}
working-directory: packages/paperclip-runner
run: |
set -euo pipefail
cargo clean --release --manifest-path runner/Cargo.toml -p paperclip-runner-core
pnpm build:binary
- name: Save the Runner release crate
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha && steps.runner_release_crate.outputs.cache-hit != 'true' }}
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
build:
if: ${{ !inputs.runner_binaries_only }}
name: Build
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.ref }}
persist-credentials: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Build
run: pnpm build
runner_binary_targets:
if: inputs.runner_binaries_only
name: Read exact-source daemon targets
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
outputs:
matrix: ${{ steps.targets.outputs.matrix }}
native_consumers: ${{ steps.targets.outputs.native_consumers }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
with:
ref: ${{ inputs.ref }}
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
with:
node-version: 24
- name: Preserve the immutable source's platform contract
id: targets
env:
SOURCE_SHA: ${{ inputs.ref }}
run: |
set -euo pipefail
[[ "$SOURCE_SHA" =~ ^[a-f0-9]{40}$ ]] || exit 1
[[ "$(git rev-parse HEAD)" == "$SOURCE_SHA" ]] || exit 1
node --input-type=module <<'NODE'
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import { appendFileSync } from 'node:fs';
const source = process.env.SOURCE_SHA;
const readBlob = file => {
const entries = execFileSync('git', ['ls-tree', '-z', source, '--', file], {
encoding: 'utf8', maxBuffer: 4096,
}).split('\0').filter(Boolean);
if (!entries.length) return undefined;
assert.equal(entries.length, 1, 'Ambiguous exact-source target contract');
assert.ok(entries[0].startsWith('100644 blob ') && entries[0].endsWith('\t' + file), 'Target contract must be a regular source file');
return execFileSync('git', ['show', source + ':' + file], { encoding: 'utf8', maxBuffer: 64 * 1024 });
};
const data = readBlob('packages/paperclip-runner/release-runner-targets.json');
let declared;
if (data !== undefined) {
const contract = JSON.parse(data);
assert.ok(contract && contract.schema === 'paperclip.runner.release-targets.v1'
&& Object.keys(contract).sort().join() === 'schema,targets', 'Unknown exact-source daemon platform contract');
declared = contract.targets;
} else {
const legacy = readBlob('packages/paperclip-runner/scripts/stage-release-runner-binaries.mjs');
const declarations = [...(legacy ?? '').matchAll(/^[ \t]*export const RELEASE_RUNNER_TARGETS = (?:Object\.freeze\()?(\[[^\r\n]*\])\)?;[ \t]*$/gm)];
assert.equal(declarations.length, 1, 'Unknown exact-source daemon platform contract');
declared = JSON.parse(declarations[0][1]);
}
const historical = ['darwin-arm64', 'darwin-x64', 'linux-x64'];
const current = [...historical, 'linux-arm64'];
assert.ok(Array.isArray(declared), 'Unknown exact-source daemon platform contract');
const matches = expected => JSON.stringify([...declared].sort()) === JSON.stringify([...expected].sort());
assert.ok(matches(historical) || matches(current), 'Unknown exact-source daemon platform contract');
const targets = matches(current) ? current : historical;
const hosts = { 'linux-x64': targets.length === current.length ? 'ubuntu-24.04' : 'ubuntu-latest', 'linux-arm64': 'ubuntu-24.04-arm', 'darwin-arm64': 'macos-15', 'darwin-x64': 'macos-15-intel' };
const include = targets.map(target => ({ target, host: hosts[target] }));
appendFileSync(process.env.GITHUB_OUTPUT, `matrix=${JSON.stringify({ include })}\nnative_consumers=${JSON.stringify({ include: include.filter(row => row.target !== 'linux-x64') })}\n`);
NODE
runner_binaries:
if: inputs.runner_binaries_only
needs: runner_binary_targets
name: Release daemon (${{ matrix.target }})
runs-on: ${{ matrix.host }}
timeout-minutes: 30
permissions:
contents: read
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.runner_binary_targets.outputs.matrix) }}
outputs:
artifact_prefix: ${{ steps.identity.outputs.artifact_prefix }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
with:
ref: ${{ inputs.ref }}
persist-credentials: false
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86
with:
version: 9.15.4
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
with:
node-version: 24
- name: Bind immutable producer source and target
id: identity
env:
SOURCE_SHA: ${{ inputs.ref }}
TARGET: ${{ matrix.target }}
run: |
set -euo pipefail
[[ "$SOURCE_SHA" =~ ^[a-f0-9]{40}$ ]] || exit 1
[[ "$(git rev-parse HEAD)" == "$SOURCE_SHA" ]] || exit 1
[[ "$(node -p 'process.platform + "-" + process.arch')" == "$TARGET" ]] || exit 1
echo "artifact_prefix=runner-release-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" >> "$GITHUB_OUTPUT"
- name: Pin native Cargo workspace and toolchain
id: native_rust_workspace
working-directory: packages/paperclip-runner
run: |
set -euo pipefail
pinned="$HOME/paperclip-runner-rust"
if [[ -e "$pinned" && ! -L "$pinned" ]]; then
echo '::error::Native Cargo cache path must not replace a real directory'
exit 1
fi
rm -f "$pinned"
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
echo "path=$pinned" >> "$GITHUB_OUTPUT"
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [[ -n "$extra_toolchains" ]]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Native Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Cache native Rust dependencies
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6
with:
workspaces: ${{ steps.native_rust_workspace.outputs.path }} -> target
shared-key: release-native-v2-${{ matrix.host }}-${{ matrix.target }}-${{ hashFiles('packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/src/**', 'packages/paperclip-runner/*.json') }}
cache-workspace-crates: false
cache-bin: false
cache-on-failure: false
save-if: ${{ github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Build TypeScript contract and host daemon concurrently
run: |
node --input-type=module <<'NODE'
import { mkdirSync, readFileSync } from 'node:fs';
import { join } from 'node:path';
import { startPrefetch, stopPrefetches } from './scripts/prefetch-child.mjs';
const root = process.cwd(), logs = join(process.env.RUNNER_TEMP, 'native-build-preparation');
mkdirSync(logs, { recursive: true });
const builds = [];
let failure;
try {
for (const [name, args, cwd] of [
['typescript', ['--filter', '@paperclipai/paperclip-runner', 'build:typescript'], root],
['native', ['build:binary'], join(root, 'packages/paperclip-runner')],
]) {
builds.push(startPrefetch({ name, cmd: 'pnpm', args, cwd, env: process.env, logPath: join(logs, name + '.log') }));
}
await Promise.all(builds.map(build => build.done));
} catch (error) { failure = error; }
finally {
await stopPrefetches(builds);
const results = await Promise.allSettled(builds.map(build => build.done));
builds.forEach((build, index) => {
console.log('::group::' + build.name + ' build');
process.stdout.write(readFileSync(join(logs, build.name + '.log'), 'utf8'));
console.log('::endgroup::');
console.log(JSON.stringify({ build: build.name, status: results[index].status,
exitCode: build.child.exitCode, signal: build.child.signalCode, durationMs: build.durationMs }));
});
}
if (failure) throw failure;
NODE
- name: Record actual daemon metadata and immutable bytes
env:
SOURCE_SHA: ${{ inputs.ref }}
TARGET: ${{ matrix.target }}
run: |
node packages/paperclip-runner/scripts/stage-release-runner-binaries.mjs record "$SOURCE_SHA" "$TARGET" \
packages/paperclip-runner/dist/bin/paperclip-runnerd "$RUNNER_TEMP/runner-release/$TARGET"
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: ${{ steps.identity.outputs.artifact_prefix }}-${{ matrix.target }}
path: ${{ runner.temp }}/runner-release/
if-no-files-found: error
retention-days: 14