Files
PaperClipAI/.github/workflows/pr-trusted.yml
T
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00

1810 lines
96 KiB
YAML

name: Trusted PR CI
# Rollout is intentionally two-step: merge this reusable workflow first, then
# replace pr.yml with an immutable-SHA caller so the active CI definition cannot
# drift from this file.
on:
workflow_call:
permissions:
actions: read
contents: read
pull-requests: read
# Test reporters write local artifacts only. The workflow_run exporter owns
# the Honeycomb credential, so fork PRs cannot execute with it.
env:
PAPERCLIP_CI_PROFILE_DIR: ${{ github.workspace }}/.ci-profile
# Empty keeps the PR default of one Vitest retry (scripts/ci/vitest-retry.mjs).
# Set the CI_VITEST_RETRY repository variable to 0 to switch retries off.
PAPERCLIP_VITEST_RETRY: ${{ vars.CI_VITEST_RETRY }}
concurrency:
group: pr-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
gate:
uses: paperclipai/paperclip/.github/workflows/ci-routing.yml@master
test_selection:
name: Select Vitest files
needs: [gate]
if: >-
needs.gate.outputs.full_ci == 'true' && needs.gate.outputs.plan != '' &&
fromJSON(needs.gate.outputs.plan).mode == 'enabled' &&
fromJSON(needs.gate.outputs.plan).forceReason == null &&
(vars.CI_VITEST_SELECTION == '' || vars.CI_VITEST_SELECTION == 'enabled')
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 10
# An unavailable optimization must not suppress the original test matrix.
continue-on-error: true
outputs:
plan: ${{ steps.publish.outputs.plan }}
env:
CI_PLAN: ${{ needs.gate.outputs.plan }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
with:
persist-credentials: false
# The merge commit and its two parent trees are the complete diff.
fetch-depth: 2
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
with:
node-version: 24
package-manager-cache: false
- name: Check source-diff eligibility
id: eligible
run: |
if [[ -f scripts/ci/select-tests.mjs ]]; then
node scripts/ci/select-tests.mjs --eligible
else
echo 'enabled=false' >> "$GITHUB_OUTPUT"
fi
# Same trusted bootstrap as every other pnpm setup in this workflow:
# Node immediately first, and no npm audit, fund or update-notifier calls.
- name: Setup Node.js for pnpm bootstrap
if: steps.eligible.outputs.enabled == 'true'
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
if: steps.eligible.outputs.enabled == 'true'
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
- name: Restore dependencies for graph analysis
if: steps.eligible.outputs.enabled == 'true'
uses: ./.github/actions/setup-ci-dependencies
- name: Select files and rebalance test lanes
id: select
if: steps.eligible.outputs.enabled == 'true'
timeout-minutes: 5
run: node scripts/ci/select-tests.mjs
- name: Upload validated selection and audit
id: upload_selection
if: steps.select.outputs.selected == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: vitest-selection-${{ github.run_attempt }}
path: |
.ci-test-selection/manifest.json
.ci-test-selection/audit.json
include-hidden-files: true
if-no-files-found: error
retention-days: 7
# Publish only after upload succeeds. Missing/failed/timed-out selectors
# produce no plan, and dependent jobs use the original gate's matrix.
- name: Publish the executable plan
id: publish
if: steps.eligible.outputs.enabled == 'true'
env:
ARTIFACT_ID: ${{ steps.upload_selection.outputs.artifact-id }}
run: |
node --input-type=module <<'NODE'
import assert from 'node:assert/strict';
import { appendFileSync, readFileSync } from 'node:fs';
const plan = JSON.parse(readFileSync('.ci-test-selection/plan.json', 'utf8'));
if (plan.testSelection) {
assert.match(process.env.ARTIFACT_ID ?? '', /^[0-9]+$/);
plan.testSelection.artifactId = process.env.ARTIFACT_ID;
}
appendFileSync(process.env.GITHUB_OUTPUT, `plan=${JSON.stringify(plan)}\n`);
NODE
policy:
needs: [gate]
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 10
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: policy
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Select compatible Node test command
run: |
# A rerun can use this trusted workflow with a pre-feature checkout.
if [[ -f scripts/ci-profile/node-test.mjs ]]; then
echo 'CI_NODE_TEST=scripts/ci-profile/node-test.mjs' >> "$GITHUB_ENV"
else
echo 'CI_NODE_TEST=--test' >> "$GITHUB_ENV"
fi
# Only trusted environments write lock content: the lockfile refresh
# workflow, and Dependabot when the lockfile bytes come from a verified
# Dependabot commit. Run the policy from the current base-branch tip so a
# pull request cannot rewrite the check that judges it.
- name: Block untrusted lockfile edits
id: lockfile_writer
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
HEAD_REF: ${{ github.head_ref }}
run: |
# GitHub does not move pull_request.base.sha forward on every push, so
# it can name an old base-branch commit. The checkout is GitHub's merge
# commit: its first parent is the current base-branch tip, and its
# second parent is the PR head. Load the policy from that tip.
trusted_base="$BASE_SHA"
if [ "$(git rev-parse --verify --quiet "$GITHUB_SHA^2")" = "$HEAD_SHA" ]; then
trusted_base="$(git rev-parse "$GITHUB_SHA^1")"
fi
policy="$RUNNER_TEMP/lockfile-trust.mjs"
if git show "$trusted_base:.github/scripts/lockfile-trust.mjs" > "$policy" 2>/dev/null; then
BASE_SHA="$trusted_base" node "$policy" pr
# Fail closed if the policy did not run.
if ! grep -q '^writer=' "$GITHUB_OUTPUT"; then
echo "::error title=Lockfile policy::lockfile-trust.mjs did not report a lockfile writer."
exit 1
fi
exit 0
fi
# The base predates lockfile-trust.mjs: keep the earlier rule.
echo "writer=none" >> "$GITHUB_OUTPUT"
if [ "$HEAD_REF" = 'chore/refresh-lockfile' ] || [ "$PR_AUTHOR" = 'dependabot[bot]' ]; then
exit 0
fi
changed="$(git diff --name-only "$trusted_base...$HEAD_SHA")"
if printf '%s\n' "$changed" | grep -qx 'pnpm-lock.yaml'; then
echo "Do not commit pnpm-lock.yaml in pull requests. CI owns lockfile updates."
exit 1
fi
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
run_install: false
- name: Validate migration ordering against target branch
run: >-
node .github/scripts/check-pr-migration-order.mjs
"${{ github.event.pull_request.base.sha }}"
"${{ github.event.pull_request.head.sha }}"
- name: Validate Dockerfile deps stage
run: |
node ./scripts/check-docker-deps-stage.mjs
node "$CI_NODE_TEST" ./scripts/docker-deps-postinstall.test.mjs
- name: Validate Node version policy
run: pnpm check:node-version
- name: Reject git push in adapter/runtime code
run: node ./scripts/check-no-git-push.mjs
- name: Test no-git-push check
run: node "$CI_NODE_TEST" ./scripts/check-no-git-push.test.mjs
- name: Validate feature module boundaries
run: pnpm check:module-boundaries
- name: Test feature module boundary check
run: node "$CI_NODE_TEST" ./scripts/check-module-boundaries.test.mjs
- name: Reject raw query-error rendering
run: node scripts/check-query-error-rendering.mjs --enforce --summary
- name: Test query-error rendering check
run: node --test ./scripts/check-query-error-rendering.test.mjs
- name: Test PR quality-gate scripts
run: node "$CI_NODE_TEST" '.github/scripts/tests/*.test.mjs'
- name: Test general-server shard partition
run: node "$CI_NODE_TEST" ./scripts/__tests__/run-vitest-stable-shard.test.mjs
- name: Test e2e shard partition
run: node "$CI_NODE_TEST" ./scripts/__tests__/e2e-shard.test.mjs
- name: Test release verify workflow wiring
run: node "$CI_NODE_TEST" ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/eval-schedule-monitor.test.mjs ./scripts/__tests__/prepare-runner-release-image.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/cloud-readiness-runner-loss.test.mjs ./scripts/standard-image-contract.test.mjs ./scripts/reuse-canary-image.test.mjs
- name: Test standalone package build concurrency
run: node "$CI_NODE_TEST" ./scripts/__tests__/build-standalone-concurrency.test.mjs
- name: Validate release package manifest
run: node ./scripts/release-package-map.mjs check
- name: Verify release package bootstrap for changed manifests
run: |
mapfile -t changed_paths < <(git diff --name-only "${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}")
PAPERCLIP_RELEASE_BOOTSTRAP_BASE_SHA="${{ github.event.pull_request.base.sha }}" \
node ./scripts/check-release-package-bootstrap.mjs "${changed_paths[@]}"
# Each lane resolves a stale lockfile inline; this early check still
# fails fast when the merge tree cannot resolve at all.
- name: Validate dependency resolution
run: pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
typecheck_release_registry:
name: Typecheck + Release Registry
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.typecheck_release_registry.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: Typecheck + Release Registry
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Share the checked-in lockfile key with master. PR merge refs must not
# save full copies of the store or evict the post-merge build caches.
- name: Restore materialized dependencies and shared builds
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
uses: ./.github/actions/setup-ci-dependencies
- name: Locate pnpm store
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
id: pnpm_store
run: |
echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
- name: Restore pnpm store (read only)
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.pnpm_store.outputs.path }}
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Install dependencies
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
# The typecheck lane builds @paperclipai/server, whose build script
# rebuilds the Runner release binary; without the shared Rust cache that
# is a ~3m40s cold compile of all third-party crates (run 35036001734,
# 2026-09-15). Same restore-only contract as Verify Paperclip Runner.
# rust-cache hashes its absolute cache paths into the entry's version,
# and GitHub only serves an entry whose key and version both match. The
# target directory sits under the checkout, and the checkout root
# differs by runner: /home/runner/_work on the RunsOn fleets that write
# this cache against /home/runner/work on GitHub-hosted runners. Every
# key input agreed, yet all 25 completed pull requests on 2026-09-28
# logged "No cache found" (run 36424309181) and cold-compiled every
# crate for ~4 minutes in four lanes. Point rust-cache at the same
# directory through a checkout-independent path so both layouts hash
# the same version. Keep this block identical in both workflows: the
# reader and the writer must agree or the version matches nothing.
- name: Pin the Runner Rust workspace path
id: runner_rust_workspace
run: |
set -euo pipefail
pinned="$HOME/paperclip-runner-rust"
# A symlink here is removed as a link, never followed into the checkout.
rm -rf "$pinned"
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
echo "path=$pinned" >> "$GITHUB_OUTPUT"
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -uo pipefail
# release-verify.yml runs on a single post-merge fleet image; the
# gate here can route to ubuntu-latest or the public PR fleet, so
# this tolerates an image without rustup instead of failing every
# pull request. Without the pin the cache key simply will not match.
if ! command -v rustup >/dev/null 2>&1; then
echo '::notice title=Runner Rust cache::rustup is unavailable; building with the image default toolchain'
exit 0
fi
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
# rust-cache hashes every installed toolchain into the cache key, not
# only the active one. Each runner image also ships its own stable
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
# therefore still computed different keys, and every GitHub-hosted
# pull request missed this cache while the fleet hit it. Remove every
# toolchain except the pin, so the key depends on the pinned compiler
# and the lockfile alone rather than on what the image happens to
# carry. Keep this block identical in both workflows: the reader and
# the writer must agree or the key matches nothing.
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [ -n "$extra_toolchains" ]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Restore Runner Rust dependencies (read only)
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
shared-key: release-runner-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
# Mirror the master writer: these also feed the cache key.
cache-workspace-crates: false
cache-bin: false
# Restore only. Never let a pull request evict master's entry.
save-if: false
# The Runner's own crate is the one thing the entry above leaves out:
# rust-cache drops workspace crates when it saves, and a fresh checkout
# would invalidate cargo's mtime fingerprints for them anyway. So this
# job compiled paperclip-runner-core in release mode on every run,
# 28-49s (runs 38169436532 to 38183149920, 2026-10-11), while 221 of
# the 239 master commits of the preceding week left its inputs alone.
# The master Rust lane in release-verify.yml saves that crate's release
# artifacts (about 70 MB) under a key made from the compiler, the
# checkout root and a hash of every input the build reads. A pull
# request whose merge tree carries byte-identical inputs restores them
# and marks them newer than the checkout, so cargo keeps them; any
# other pull request misses and compiles exactly as before. Exact key
# only: a prefix match would hand cargo another revision's output. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the writer's; a guard test holds them together.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Restore the Runner release crate (read only)
id: runner_release_crate
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-process-table-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_process_table-*
packages/paperclip-runner/runner/target/release/deps/paperclip_process_table-*.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/acpx-profiles.json', 'packages/paperclip-runner/antigravity-distributions.json', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Mark the restored Runner release crate current
if: ${{ steps.runner_release_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-release-crate-cache.mjs mark-current
- name: Typecheck, CLI guidance safety and release registry tests
if: ${{ hashFiles('scripts/ci-typecheck-lane.mjs') != '' }}
run: node scripts/ci-typecheck-lane.mjs
- name: Typecheck a checkout that predates the typecheck lane script
if: ${{ hashFiles('scripts/ci-typecheck-lane.mjs') == '' }}
env:
PAPERCLIP_TYPECHECK_LOW_MEMORY: "1"
run: |
pnpm run typecheck:build-gaps
node scripts/typecheck-test-programs.mjs --low-memory
pnpm --filter @paperclipai/server typecheck
node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations
pnpm exec vitest run --project @paperclipai/server server/src/__tests__/cli-invocation-safety.test.ts
pnpm run test:release-registry
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
general_tests:
name: General tests (${{ matrix.group_label }})
needs: [gate, test_selection]
if: ${{ always() && !cancelled() && needs.gate.result == 'success' && needs.gate.outputs.full_ci == 'true' && ((needs.test_selection.outputs.plan || needs.gate.outputs.plan) == '' || fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan)).jobs.general_tests.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
strategy:
fail-fast: false
max-parallel: ${{ needs.gate.outputs.runner == 'ubuntu-latest' && 4 || 256 }}
matrix: ${{ fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan) != '' && toJSON(fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan)).jobs.general_tests.matrix) || '{"include":[{"group":"general-server-without-chat","group_label":"server (1/12)","shard_index":0,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (2/12)","shard_index":1,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (3/12)","shard_index":2,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (4/12)","shard_index":3,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (5/12)","shard_index":4,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (6/12)","shard_index":5,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (7/12)","shard_index":6,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (8/12)","shard_index":7,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (9/12)","shard_index":8,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (10/12)","shard_index":9,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (11/12)","shard_index":10,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (12/12)","shard_index":11,"shard_count":12,"task":"ci:server"},{"group":"general-chat","group_label":"chat (1/3)","shard_index":0,"shard_count":3,"task":"ci:chat"},{"group":"general-chat","group_label":"chat (2/3)","shard_index":1,"shard_count":3,"task":"ci:chat"},{"group":"general-chat","group_label":"chat (3/3)","shard_index":2,"shard_count":3,"task":"ci:chat"},{"group":"general-workspaces-a","group_label":"workspaces-a (1/2)","shard_index":0,"shard_count":2,"task":"ci:workspaces-a"},{"group":"general-workspaces-a","group_label":"workspaces-a (2/2)","shard_index":1,"shard_count":2,"task":"ci:workspaces-a"},{"group":"general-workspaces-b","group_label":"workspaces-b","task":"ci:workspaces-b"},{"group":"general-workspaces-c","group_label":"workspaces-c","task":"ci:db"}]}') }}
env:
PAPERCLIP_CI_TEST_SELECTION: ${{ matrix.selection && '.ci-test-selection/manifest.json' || '' }}
PAPERCLIP_CI_SELECTION_HASH: ${{ matrix.selection && fromJSON(needs.test_selection.outputs.plan).testSelection.hash || '' }}
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: General tests (${{ matrix.group_label }})
# The reporting-lock policy test skips a Dependabot lockfile, which the
# policy job verifies and the lockfile refresh admits after merge.
PAPERCLIP_CI_PR_AUTHOR: ${{ github.event.pull_request.user.login }}
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Download selected test files
if: matrix.selection
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
artifact-ids: ${{ fromJSON(needs.test_selection.outputs.plan).testSelection.artifactId }}
path: .ci-test-selection
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Share the checked-in lockfile key with master. PR merge refs must not
# save full copies of the store or evict the post-merge build caches.
- name: Restore materialized dependencies and shared builds
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
uses: ./.github/actions/setup-ci-dependencies
- name: Locate pnpm store
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
id: pnpm_store
run: |
echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
- name: Restore pnpm store (read only)
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.pnpm_store.outputs.path }}
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Install dependencies
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
- name: Run grouped general test suites through Turbo
if: hashFiles('turbo.json', '.github/actions/run-ci-task/action.yml') != ''
uses: ./.github/actions/run-ci-task
with:
task: ${{ matrix.task }}
args: ${{ matrix.shard_count && format('["--shard-index","{0}","--shard-count","{1}"]', matrix.shard_index, matrix.shard_count) || '[]' }}
partition: ${{ strategy.job-index || 0 }}
force: ${{ needs.gate.outputs.profile_ci == 'true' || (needs.test_selection.outputs.plan || needs.gate.outputs.plan) == '' || fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan)).mode != 'enabled' || fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan)).forceReason != null }}
- name: Run grouped general test suites
if: hashFiles('turbo.json') == ''
run: |
if [ -n "${{ matrix.shard_count }}" ]; then
pnpm test:run:general -- --group '${{ matrix.group }}' \
--shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
else
pnpm test:run:general -- --group '${{ matrix.group }}'
fi
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
docker_context_integrity:
name: Docker context integrity
needs: gate
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.docker_context_integrity.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 15
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: Docker context integrity
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
# Not every runner the gate can select ships the Buildx plugin —
# the image-build workflows set it up explicitly, so this lane does
# too rather than failing before it checks anything.
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
- name: Set up Docker Compose
uses: docker/setup-compose-action@54042514f505b273907334ae2b9cdbb9a0213c1a # v2
with:
version: v5.5.1
- name: Verify Compose signing secret forwarding
run: node --test scripts/docker-compose-secrets.test.mjs
# Same .dockerignore semantics as the real image builds: a
# context-slimming change that strips a committed build input must
# fail here, on the pull request, instead of failing every
# post-merge image build. (2026-09-04: a new **/*.md ignore rule
# stripped the committed capability contract out of the context;
# every Docker build on master failed its drift check and no cloud
# image published for eight hours while PR CI stayed green.)
- name: Run generated-file drift checks against the Docker build context
run: docker buildx build --file .github/docker-context-checks.Dockerfile .
- name: Verify Tailscale runtime clients
run: docker buildx build --file Dockerfile --target tailscale-client .
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
# Sandbox providers are outside the root workspace, so no other lane runs their
# tests. The `//#ci:sandbox-providers` inputs in turbo.json select this job: the
# providers, the plugin SDK, and the shared package the SDK depends on.
sandbox_provider_tests:
name: Sandbox provider tests
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.sandbox_provider_tests.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 15
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: Sandbox provider tests
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Share the checked-in lockfile key with master. PR merge refs must not
# save full copies of the store or evict the post-merge build caches.
- name: Restore materialized dependencies and shared builds
uses: ./.github/actions/setup-ci-dependencies
# The script installs the providers from their own lockfiles, links the
# in-repo plugin SDK, and runs every provider suite. Daytona's capacity
# tests kill a process at a file-size limit inside the exported
# directory. On a runner that allows core dumps, its core file joins the
# export and changes the transferred file count, so turn core dumps off.
- name: Run sandbox provider tests
run: |
ulimit -c 0
node scripts/test-sandbox-providers.mjs
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
verify_runner_e2e_support:
name: Runner support suite
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.verify_runner_e2e_support.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: Runner support suite
# The paid Runner workflows depend on this suite: workflow security
# policy, evidence oracles, cost receipts and controller recovery. It
# used to run only in a manual Docker Runner check, so master changes
# broke it unseen (#19527, #19529). It needs no provider credentials.
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Share the checked-in lockfile key with master. PR merge refs must not
# save full copies of the store or evict the post-merge build caches.
- name: Restore materialized dependencies and shared builds
uses: ./.github/actions/setup-ci-dependencies
# Builds the plugin SDK first, then runs Vitest and the node:test files.
- name: Run the Runner support suite
run: pnpm test:e2e:runner:unit
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
verify:
name: verify
if: ${{ always() }}
needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run, verify_runner_e2e_support]
runs-on: ${{ needs.gate.outputs.runner || 'ubuntu-latest' }}
timeout-minutes: 5
steps:
- name: Validate every planned suite result
env:
FULL_CI: ${{ needs.gate.outputs.full_ci }}
CI_PLAN: ${{ needs.test_selection.outputs.plan || needs.gate.outputs.plan }}
CI_RESULTS: ${{ toJSON(needs) }}
run: |
node --input-type=module <<'NODE'
import assert from 'node:assert/strict';
const results = JSON.parse(process.env.CI_RESULTS);
assert.equal(results.gate.result, 'success', 'CI routing must succeed');
assert.equal(results.policy.result, 'success', 'Policy must succeed');
assert.ok(['true', 'false'].includes(process.env.FULL_CI), 'Valid scope required');
const plan = process.env.CI_PLAN ? JSON.parse(process.env.CI_PLAN) : null;
if (plan) assert.equal(plan.version, 1, 'Supported task plan required');
if (plan?.testSelection) assert.equal(results.test_selection.result, 'success', 'Published selection must succeed');
for (const job of ["typecheck_release_registry","general_tests","verify_paperclip_runner","build","docker_context_integrity","sandbox_provider_tests","verify_serialized_server","canary_dry_run","verify_runner_e2e_support"]) {
if (plan) assert.equal(typeof plan.jobs[job]?.run, 'boolean', 'Missing suite decision: ' + job);
const selected = process.env.FULL_CI === 'true' && (!plan || plan.jobs[job].run);
assert.equal(results[job].result, selected ? 'success' : 'skipped', job);
}
NODE
verify_paperclip_runner:
name: Verify Paperclip Runner (${{ matrix.lane_label }})
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.verify_paperclip_runner.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
strategy:
fail-fast: false
max-parallel: ${{ needs.gate.outputs.runner == 'ubuntu-latest' && 4 || 256 }}
matrix: ${{ fromJSON(needs.gate.outputs.plan != '' && toJSON(fromJSON(needs.gate.outputs.plan).jobs.verify_paperclip_runner.matrix) || '{"include":[{"lane_label":"static checks","command":"check:static","task":"ci:runner-static"},{"lane_label":"rust","command":"check:runner","release_crate":true,"task":"ci:runner-rust"},{"lane_label":"vitest package (1/3)","command":"test:typescript:vitest --package-lane=1/3","task":"ci:runner-vitest"},{"lane_label":"vitest package (2/3)","command":"test:typescript:vitest --package-lane=2/3","task":"ci:runner-vitest"},{"lane_label":"vitest package (3/3)","command":"test:typescript:vitest --package-lane=3/3","task":"ci:runner-vitest"},{"lane_label":"vitest native server","command":"test:typescript:vitest --native-server","release_crate":true,"task":"ci:runner-vitest"}]}') }}
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: Verify Paperclip Runner (${{ matrix.lane_label }})
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Share the checked-in lockfile key with master. PR merge refs must not
# save full copies of the store or evict the post-merge build caches.
- name: Restore materialized dependencies and shared builds
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
uses: ./.github/actions/setup-ci-dependencies
- name: Locate pnpm store
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
id: pnpm_store
run: |
echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
- name: Restore pnpm store (read only)
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.pnpm_store.outputs.path }}
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Install dependencies
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
# Same restore-only contract as the pnpm store above, for the Rust
# dependency tree: master's post-merge verification is the sole writer
# of release-runner-v2, and PR merge refs must not save branch-scoped
# copies of a ~680MB target directory. Every key input below has to
# match that writer in release-verify.yml exactly or each PR misses and
# recompiles all 313 third-party crates in both profiles. A miss is a
# slow run, never a wrong one.
# rust-cache hashes its absolute cache paths into the entry's version,
# and GitHub only serves an entry whose key and version both match. The
# target directory sits under the checkout, and the checkout root
# differs by runner: /home/runner/_work on the RunsOn fleets that write
# this cache against /home/runner/work on GitHub-hosted runners. Every
# key input agreed, yet all 25 completed pull requests on 2026-09-28
# logged "No cache found" (run 36424309181) and cold-compiled every
# crate for ~4 minutes in four lanes. Point rust-cache at the same
# directory through a checkout-independent path so both layouts hash
# the same version. Keep this block identical in both workflows: the
# reader and the writer must agree or the version matches nothing.
- name: Pin the Runner Rust workspace path
id: runner_rust_workspace
run: |
set -euo pipefail
pinned="$HOME/paperclip-runner-rust"
# A symlink here is removed as a link, never followed into the checkout.
rm -rf "$pinned"
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
echo "path=$pinned" >> "$GITHUB_OUTPUT"
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -uo pipefail
# release-verify.yml runs on a single post-merge fleet image; the
# gate here can route to ubuntu-latest or the public PR fleet, so
# this tolerates an image without rustup instead of failing every
# pull request. Without the pin the cache key simply will not match.
if ! command -v rustup >/dev/null 2>&1; then
echo '::notice title=Runner Rust cache::rustup is unavailable; building with the image default toolchain'
exit 0
fi
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
# rust-cache hashes every installed toolchain into the cache key, not
# only the active one. Each runner image also ships its own stable
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
# therefore still computed different keys, and every GitHub-hosted
# pull request missed this cache while the fleet hit it. Remove every
# toolchain except the pin, so the key depends on the pinned compiler
# and the lockfile alone rather than on what the image happens to
# carry. Keep this block identical in both workflows: the reader and
# the writer must agree or the key matches nothing.
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [ -n "$extra_toolchains" ]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Restore Runner Rust dependencies (read only)
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
shared-key: release-runner-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
# Mirror the master writer: these also feed the cache key.
cache-workspace-crates: false
cache-bin: false
# Restore only. Never let a pull request evict master's entry.
save-if: false
# The Runner's own crate is the one thing the entry above leaves out:
# rust-cache drops workspace crates when it saves, and a fresh checkout
# would invalidate cargo's mtime fingerprints for them anyway. So this
# job compiled paperclip-runner-core in release mode on every run,
# 28-49s (runs 38169436532 to 38183149920, 2026-10-11), while 221 of
# the 239 master commits of the preceding week left its inputs alone.
# The master Rust lane in release-verify.yml saves that crate's release
# artifacts (about 70 MB) under a key made from the compiler, the
# checkout root and a hash of every input the build reads. A pull
# request whose merge tree carries byte-identical inputs restores them
# and marks them newer than the checkout, so cargo keeps them; any
# other pull request misses and compiles exactly as before. Exact key
# only: a prefix match would hand cargo another revision's output. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the writer's; a guard test holds them together.
#
# Only the lanes that build the release crate restore it: the Rust
# lane's release tests link the lib, and the final vitest shard hosts
# the server's native-runner suite, which builds the binary in its
# beforeAll. The static and first vitest lanes run no cargo at all.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
if: ${{ matrix.release_crate == true }}
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Restore the Runner release crate (read only)
id: runner_release_crate
if: ${{ matrix.release_crate == true }}
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-process-table-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_process_table-*
packages/paperclip-runner/runner/target/release/deps/paperclip_process_table-*.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/acpx-profiles.json', 'packages/paperclip-runner/antigravity-distributions.json', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Mark the restored Runner release crate current
if: ${{ steps.runner_release_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-release-crate-cache.mjs mark-current
- name: Compute the Runner debug crate cache key
id: runner_debug_crate_key
if: hashFiles('scripts/runner-debug-crate-cache.mjs') != ''
run: node scripts/runner-debug-crate-cache.mjs plan
- name: Restore the Runner debug crate (read only)
id: runner_debug_crate
if: hashFiles('scripts/runner-debug-crate-cache.mjs') != ''
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.runner_debug_crate_key.outputs.paths }}
key: ${{ steps.runner_debug_crate_key.outputs.key }}
- name: Mark the restored Runner debug crate current
if: ${{ steps.runner_debug_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-debug-crate-cache.mjs mark-current
- name: Verify Runner Rust through Turbo
if: matrix.command == 'check:runner' && hashFiles('turbo.json') != ''
uses: ./.github/actions/run-ci-task
with:
task: ci:runner-rust
args: '[]'
partition: ${{ strategy.job-index || 0 }}
force: ${{ needs.gate.outputs.profile_ci == 'true' || needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).mode != 'enabled' || fromJSON(needs.gate.outputs.plan).forceReason != null }}
- name: Verify Paperclip Runner
if: matrix.command != 'check:runner' || hashFiles('turbo.json') == ''
# pnpm appends trailing args to the end of the script's shell chain,
# so the vitest lanes' --shard lands on `vitest run`. Do not add a
# `--` separator: pnpm forwards it literally and vitest would then
# read the shard flag as a test filter.
run: pnpm --filter @paperclipai/paperclip-runner ${{ matrix.command }}
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
build:
name: Build
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.build.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: Build
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Share the checked-in lockfile key with master. PR merge refs must not
# save full copies of the store or evict the post-merge build caches.
- name: Restore materialized dependencies and shared builds
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
uses: ./.github/actions/setup-ci-dependencies
- name: Locate pnpm store
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
id: pnpm_store
run: |
echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
- name: Restore pnpm store (read only)
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.pnpm_store.outputs.path }}
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Install dependencies
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
# The public-install verifier at the end of this lane downloads the whole
# public registry graph into an empty npm cache: its consumer install
# took 119-129s of the 226-241s step, more than half of this ~415-446s
# lane, in runs 38426120144, 38432086560 and 38423713537 (2026-10-13).
# That graph is fixed by source manifests, so this fetches it in the
# background while the Rust restore, Evalbook and `pnpm build` run, about
# 150s before the verifier installs anything. Only the cache is handed
# over, and the verify step waits for this first. A failed or partial
# warm-up leaves the verifier to fetch what is missing as before.
- name: Start warming the public-install npm cache
if: ${{ hashFiles('scripts/warm-public-install-npm-cache.mjs') != '' }}
run: |
(
# The default `bash -e` would skip the status line on a failure.
set +e
PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache" node scripts/warm-public-install-npm-cache.mjs
echo $? > "$RUNNER_TEMP/public-install-npm-cache.status"
) > "$RUNNER_TEMP/public-install-npm-cache.log" 2>&1 &
# pnpm build reaches the Runner package's build:binary step; without the
# shared Rust cache that is a ~3m40s cold compile of all third-party
# crates (run 35036001734, 2026-09-15). Same restore-only contract as
# Verify Paperclip Runner.
# rust-cache hashes its absolute cache paths into the entry's version,
# and GitHub only serves an entry whose key and version both match. The
# target directory sits under the checkout, and the checkout root
# differs by runner: /home/runner/_work on the RunsOn fleets that write
# this cache against /home/runner/work on GitHub-hosted runners. Every
# key input agreed, yet all 25 completed pull requests on 2026-09-28
# logged "No cache found" (run 36424309181) and cold-compiled every
# crate for ~4 minutes in four lanes. Point rust-cache at the same
# directory through a checkout-independent path so both layouts hash
# the same version. Keep this block identical in both workflows: the
# reader and the writer must agree or the version matches nothing.
- name: Pin the Runner Rust workspace path
id: runner_rust_workspace
run: |
set -euo pipefail
pinned="$HOME/paperclip-runner-rust"
# A symlink here is removed as a link, never followed into the checkout.
rm -rf "$pinned"
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
echo "path=$pinned" >> "$GITHUB_OUTPUT"
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -uo pipefail
# release-verify.yml runs on a single post-merge fleet image; the
# gate here can route to ubuntu-latest or the public PR fleet, so
# this tolerates an image without rustup instead of failing every
# pull request. Without the pin the cache key simply will not match.
if ! command -v rustup >/dev/null 2>&1; then
echo '::notice title=Runner Rust cache::rustup is unavailable; building with the image default toolchain'
exit 0
fi
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
# rust-cache hashes every installed toolchain into the cache key, not
# only the active one. Each runner image also ships its own stable
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
# therefore still computed different keys, and every GitHub-hosted
# pull request missed this cache while the fleet hit it. Remove every
# toolchain except the pin, so the key depends on the pinned compiler
# and the lockfile alone rather than on what the image happens to
# carry. Keep this block identical in both workflows: the reader and
# the writer must agree or the key matches nothing.
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [ -n "$extra_toolchains" ]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Restore Runner Rust dependencies (read only)
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
shared-key: release-runner-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
# Mirror the master writer: these also feed the cache key.
cache-workspace-crates: false
cache-bin: false
# Restore only. Never let a pull request evict master's entry.
save-if: false
# The Runner's own crate is the one thing the entry above leaves out:
# rust-cache drops workspace crates when it saves, and a fresh checkout
# would invalidate cargo's mtime fingerprints for them anyway. So this
# job compiled paperclip-runner-core in release mode on every run,
# 28-49s (runs 38169436532 to 38183149920, 2026-10-11), while 221 of
# the 239 master commits of the preceding week left its inputs alone.
# The master Rust lane in release-verify.yml saves that crate's release
# artifacts (about 70 MB) under a key made from the compiler, the
# checkout root and a hash of every input the build reads. A pull
# request whose merge tree carries byte-identical inputs restores them
# and marks them newer than the checkout, so cargo keeps them; any
# other pull request misses and compiles exactly as before. Exact key
# only: a prefix match would hand cargo another revision's output. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the writer's; a guard test holds them together.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Restore the Runner release crate (read only)
id: runner_release_crate
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-process-table-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_process_table-*
packages/paperclip-runner/runner/target/release/deps/paperclip_process_table-*.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/acpx-profiles.json', 'packages/paperclip-runner/antigravity-distributions.json', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Mark the restored Runner release crate current
if: ${{ steps.runner_release_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-release-crate-cache.mjs mark-current
- name: Build Runner Evalbook viewer
run: pnpm --filter @paperclipai/paperclip-runner build:issue-thread
- name: Build
run: pnpm build
# The production image starts the server from this bundle with plain
# node (Dockerfile CMD), so prove it boots before the image is built:
# bundle the `pnpm build` output, start it against a throwaway embedded
# instance, and wait for /api/health to report ok.
- name: Boot the production runtime bundle
run: |
pnpm --filter @paperclipai/server build:runtime
scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs
# This verification only needs `pnpm build` output (the verifier stages
# the public packages' dist and reuses ui/dist itself), yet it ran at the
# end of Canary Dry Run, serially after the ~210-295s release.sh dry run.
# That made Canary Dry Run the slowest check in every green PR run:
# 484s, 572s and 483s in runs 37310951026, 37309272947 and 37308895471
# (2026-10-05), with this step alone at 210-223s of each. The Build lane
# finished its `pnpm build` in ~206s total in those same runs, so it
# absorbs the step at ~416s while Canary Dry Run drops to ~275-350s and
# neither remains the critical path (server 2/12 at ~437-467s is).
# docker-runner-check.yml runs the same verifier after `pnpm build`.
- name: Verify built-in Grok from a clean public npm install
if: ${{ hashFiles('scripts/verify-grok-npm-install.mjs') != '' }}
run: |
status="$RUNNER_TEMP/public-install-npm-cache.status"
if [ -f "$RUNNER_TEMP/public-install-npm-cache.log" ]; then
# Normally finished long before this; never let it hold the lane.
for _ in $(seq 180); do [ -f "$status" ] && break; sleep 1; done
if [ "$(cat "$status" 2>/dev/null)" = 0 ]; then
tail -n 1 "$RUNNER_TEMP/public-install-npm-cache.log"
else
echo '::notice title=Public-install npm cache::warm-up did not finish cleanly; the verifier fetches what is missing'
tail -n 20 "$RUNNER_TEMP/public-install-npm-cache.log"
fi
export PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache"
fi
node scripts/verify-grok-npm-install.mjs
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
verify_serialized_server:
name: Verify serialized server suites (${{ matrix.shard_label }})
needs: [gate, test_selection]
if: ${{ always() && !cancelled() && needs.gate.result == 'success' && needs.gate.outputs.full_ci == 'true' && ((needs.test_selection.outputs.plan || needs.gate.outputs.plan) == '' || fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan)).jobs.verify_serialized_server.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
strategy:
fail-fast: false
max-parallel: ${{ needs.gate.outputs.runner == 'ubuntu-latest' && 4 || 256 }}
matrix: ${{ fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan) != '' && toJSON(fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan)).jobs.verify_serialized_server.matrix) || '{"include":[{"shard_index":0,"shard_count":9,"shard_label":"1/9","task":"ci:serialized"},{"shard_index":1,"shard_count":9,"shard_label":"2/9","task":"ci:serialized"},{"shard_index":2,"shard_count":9,"shard_label":"3/9","task":"ci:serialized"},{"shard_index":3,"shard_count":9,"shard_label":"4/9","task":"ci:serialized"},{"shard_index":4,"shard_count":9,"shard_label":"5/9","task":"ci:serialized"},{"shard_index":5,"shard_count":9,"shard_label":"6/9","task":"ci:serialized"},{"shard_index":6,"shard_count":9,"shard_label":"7/9","task":"ci:serialized"},{"shard_index":7,"shard_count":9,"shard_label":"8/9","task":"ci:serialized"},{"shard_index":8,"shard_count":9,"shard_label":"9/9","task":"ci:serialized"}]}') }}
env:
PAPERCLIP_CI_TEST_SELECTION: ${{ matrix.selection && '.ci-test-selection/manifest.json' || '' }}
PAPERCLIP_CI_SELECTION_HASH: ${{ matrix.selection && fromJSON(needs.test_selection.outputs.plan).testSelection.hash || '' }}
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: Verify serialized server suites (${{ matrix.shard_label }})
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Download selected test files
if: matrix.selection
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
artifact-ids: ${{ fromJSON(needs.test_selection.outputs.plan).testSelection.artifactId }}
path: .ci-test-selection
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Share the checked-in lockfile key with master. PR merge refs must not
# save full copies of the store or evict the post-merge build caches.
- name: Restore materialized dependencies and shared builds
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
uses: ./.github/actions/setup-ci-dependencies
- name: Locate pnpm store
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
id: pnpm_store
run: |
echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
- name: Restore pnpm store (read only)
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.pnpm_store.outputs.path }}
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Install dependencies
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
- name: Run serialized server test shard through Turbo
if: hashFiles('turbo.json', '.github/actions/run-ci-task/action.yml') != ''
uses: ./.github/actions/run-ci-task
with:
task: ci:serialized
args: ${{ format('["--shard-index","{0}","--shard-count","{1}"]', matrix.shard_index, matrix.shard_count) }}
partition: ${{ strategy.job-index || 0 }}
force: ${{ needs.gate.outputs.profile_ci == 'true' || (needs.test_selection.outputs.plan || needs.gate.outputs.plan) == '' || fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan)).mode != 'enabled' || fromJSON((needs.test_selection.outputs.plan || needs.gate.outputs.plan)).forceReason != null }}
- name: Run serialized server test shard
if: hashFiles('turbo.json') == ''
run: pnpm test:run:serialized -- --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
canary_dry_run:
name: Canary Dry Run
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.canary_dry_run.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: Canary Dry Run
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Share the checked-in lockfile key with master. PR merge refs must not
# save full copies of the store or evict the post-merge build caches.
- name: Restore materialized dependencies and shared builds
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
uses: ./.github/actions/setup-ci-dependencies
- name: Locate pnpm store
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
id: pnpm_store
run: |
echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
- name: Restore pnpm store (read only)
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.pnpm_store.outputs.path }}
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Install dependencies
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
# release.sh's Step 2/7 workspace build reaches the Runner package's
# build:binary step; without the shared Rust cache that is a ~3m40s cold
# compile of all third-party crates (run 35036001734, 2026-09-15). Same
# restore-only contract as Verify Paperclip Runner.
# rust-cache hashes its absolute cache paths into the entry's version,
# and GitHub only serves an entry whose key and version both match. The
# target directory sits under the checkout, and the checkout root
# differs by runner: /home/runner/_work on the RunsOn fleets that write
# this cache against /home/runner/work on GitHub-hosted runners. Every
# key input agreed, yet all 25 completed pull requests on 2026-09-28
# logged "No cache found" (run 36424309181) and cold-compiled every
# crate for ~4 minutes in four lanes. Point rust-cache at the same
# directory through a checkout-independent path so both layouts hash
# the same version. Keep this block identical in both workflows: the
# reader and the writer must agree or the version matches nothing.
- name: Pin the Runner Rust workspace path
id: runner_rust_workspace
run: |
set -euo pipefail
pinned="$HOME/paperclip-runner-rust"
# A symlink here is removed as a link, never followed into the checkout.
rm -rf "$pinned"
ln -s "$GITHUB_WORKSPACE/packages/paperclip-runner/runner" "$pinned"
echo "path=$pinned" >> "$GITHUB_OUTPUT"
- name: Select the pinned Runner Rust toolchain
working-directory: packages/paperclip-runner
run: |
set -uo pipefail
# release-verify.yml runs on a single post-merge fleet image; the
# gate here can route to ubuntu-latest or the public PR fleet, so
# this tolerates an image without rustup instead of failing every
# pull request. Without the pin the cache key simply will not match.
if ! command -v rustup >/dev/null 2>&1; then
echo '::notice title=Runner Rust cache::rustup is unavailable; building with the image default toolchain'
exit 0
fi
rustup show
toolchain="$(rustup show active-toolchain | awk '{print $1}')"
echo "RUSTUP_TOOLCHAIN=$toolchain" >> "$GITHUB_ENV"
# rust-cache hashes every installed toolchain into the cache key, not
# only the active one. Each runner image also ships its own stable
# Rust, and those disagree across images: 1.98.0 on the RunsOn fleets
# against 1.98.1 on ubuntu-latest. Two runners that agreed on the pin
# therefore still computed different keys, and every GitHub-hosted
# pull request missed this cache while the fleet hit it. Remove every
# toolchain except the pin, so the key depends on the pinned compiler
# and the lockfile alone rather than on what the image happens to
# carry. Keep this block identical in both workflows: the reader and
# the writer must agree or the key matches nothing.
extra_toolchains="$(rustup toolchain list | awk '{print $1}' | grep -vx "$toolchain" || true)"
if [ -n "$extra_toolchains" ]; then
echo "$extra_toolchains" | xargs -n1 rustup toolchain uninstall \
|| echo '::notice title=Runner Rust cache::could not remove an extra toolchain; the cache key may not match'
fi
rustup toolchain list
- name: Restore Runner Rust dependencies (read only)
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: ${{ steps.runner_rust_workspace.outputs.path }} -> target
shared-key: release-runner-v2-${{ hashFiles('packages/paperclip-runner/runner/Cargo.toml', '.cargo/config', '.cargo/config.toml', 'packages/.cargo/config', 'packages/.cargo/config.toml', 'packages/paperclip-runner/.cargo/config', 'packages/paperclip-runner/.cargo/config.toml', 'packages/paperclip-runner/runner/.cargo/config', 'packages/paperclip-runner/runner/.cargo/config.toml') }}
# Mirror the master writer: these also feed the cache key.
cache-workspace-crates: false
cache-bin: false
# Restore only. Never let a pull request evict master's entry.
save-if: false
# The Runner's own crate is the one thing the entry above leaves out:
# rust-cache drops workspace crates when it saves, and a fresh checkout
# would invalidate cargo's mtime fingerprints for them anyway. So this
# job compiled paperclip-runner-core in release mode on every run,
# 28-49s (runs 38169436532 to 38183149920, 2026-10-11), while 221 of
# the 239 master commits of the preceding week left its inputs alone.
# The master Rust lane in release-verify.yml saves that crate's release
# artifacts (about 70 MB) under a key made from the compiler, the
# checkout root and a hash of every input the build reads. A pull
# request whose merge tree carries byte-identical inputs restores them
# and marks them newer than the checkout, so cargo keeps them; any
# other pull request misses and compiles exactly as before. Exact key
# only: a prefix match would hand cargo another revision's output. The
# path list and the key live in scripts/runner-release-crate-cache.mjs
# and must match the writer's; a guard test holds them together.
- name: Compute the Runner release crate cache key
id: runner_release_crate_key
run: echo "scope=$(node scripts/runner-release-crate-cache.mjs scope)" >> "$GITHUB_OUTPUT"
- name: Restore the Runner release crate (read only)
id: runner_release_crate
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-runner-core-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_runner_core-*
packages/paperclip-runner/runner/target/release/deps/paperclip_runner_core-*.d
packages/paperclip-runner/runner/target/release/deps/paperclip_runnerd-*
packages/paperclip-runner/runner/target/release/paperclip-runnerd
packages/paperclip-runner/runner/target/release/paperclip-runnerd.d
packages/paperclip-runner/runner/target/release/.fingerprint/paperclip-process-table-*
packages/paperclip-runner/runner/target/release/deps/libpaperclip_process_table-*
packages/paperclip-runner/runner/target/release/deps/paperclip_process_table-*.d
key: runner-release-crate-v1-${{ steps.runner_release_crate_key.outputs.scope }}-${{ hashFiles('packages/paperclip-runner/rust-toolchain.toml', 'packages/paperclip-runner/acpx-profiles.json', 'packages/paperclip-runner/antigravity-distributions.json', 'packages/paperclip-runner/runner/Cargo.toml', 'packages/paperclip-runner/runner/Cargo.lock', 'packages/paperclip-runner/runner/crates/**', 'packages/paperclip-runner/protocol/**', 'packages/paperclip-runner/test-fixtures/**', 'packages/paperclip-runner/test/fixtures/**', 'tests/runner-e2e/fixtures/**') }}
- name: Mark the restored Runner release crate current
if: ${{ steps.runner_release_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-release-crate-cache.mjs mark-current
# `release.sh` always executes its Step 2/7 workspace build, even when
# `--skip-verify` bypasses the initial verification gate. release.sh
# also requires a clean working tree, and the install step may have
# resolved a stale lockfile in place (manifest-changing or stacked
# PRs), so stage any changed lockfile into an ephemeral local commit:
# release.sh then sees a clean tree and its workspace build sees a
# lockfile that matches the manifests.
- name: Release canary dry run via release.sh internal build
run: |
git checkout -B master HEAD
if git diff --quiet pnpm-lock.yaml; then
git checkout -- pnpm-lock.yaml
else
git add pnpm-lock.yaml
git -c user.email=ci@paperclip.local -c user.name=CI \
commit --no-verify -m "ci(canary): stage regenerated lockfile"
fi
./scripts/release.sh canary --skip-verify --dry-run
# The clean public npm install verification for the built-in Grok
# provider used to run here, after the dry run. It only needs `pnpm
# build` output, which the Build lane also produces, so it moved there:
# see "Verify built-in Grok from a clean public npm install" in `build`.
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
e2e_shards:
name: e2e shard (${{ matrix.shard_label }})
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.e2e_shards.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 30
strategy:
fail-fast: false
max-parallel: ${{ needs.gate.outputs.runner == 'ubuntu-latest' && 4 || 256 }}
matrix: ${{ fromJSON(needs.gate.outputs.plan != '' && toJSON(fromJSON(needs.gate.outputs.plan).jobs.e2e_shards.matrix) || '{"include":[{"shard_index":0,"shard_count":8,"shard_label":"1/8","task":"ci:e2e"},{"shard_index":1,"shard_count":8,"shard_label":"2/8","task":"ci:e2e"},{"shard_index":2,"shard_count":8,"shard_label":"3/8","task":"ci:e2e"},{"shard_index":3,"shard_count":8,"shard_label":"4/8","task":"ci:e2e"},{"shard_index":4,"shard_count":8,"shard_label":"5/8","task":"ci:e2e"},{"shard_index":5,"shard_count":8,"shard_label":"6/8","task":"ci:e2e"},{"shard_index":6,"shard_count":8,"shard_label":"7/8","task":"ci:e2e"},{"shard_index":7,"shard_count":8,"shard_label":"8/8","task":"ci:e2e"}]}') }}
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
PAPERCLIP_CI_PROFILE_JOB: e2e shard (${{ matrix.shard_label }})
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Initialize CI profile
continue-on-error: true
if: env.PAPERCLIP_CI_PROFILE == 'true'
run: node scripts/ci-profile/initialize.mjs
- name: Setup Node.js for pnpm bootstrap
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
package-manager-cache: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Share the checked-in lockfile key with master. PR merge refs must not
# save full copies of the store or evict the post-merge build caches.
- name: Restore materialized dependencies and shared builds
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') != ''
uses: ./.github/actions/setup-ci-dependencies
- name: Locate pnpm store
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
id: pnpm_store
run: |
echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
echo "arch=$(node -p 'process.arch')" >> "$GITHUB_OUTPUT"
- name: Restore pnpm store (read only)
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.pnpm_store.outputs.path }}
key: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: node-cache-${{ runner.os }}-${{ steps.pnpm_store.outputs.arch }}-pnpm-
- name: Install dependencies
if: hashFiles('.github/actions/setup-ci-dependencies/action.yml') == ''
# Manifest-changing and stacked PRs can hold a pnpm-lock.yaml that is
# stale for the merge tree. Resolve it inline instead of waiting on a
# policy-job artifact: that dependency put the policy job's queue and
# runtime (~60s) on every lane's critical path, and policy still
# validates resolution as a required check in parallel.
run: |
if ! pnpm install --frozen-lockfile; then
echo '::notice title=Lockfile::checked-in lockfile is stale for this merge tree; resolving it inline'
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
pnpm install --frozen-lockfile
fi
- name: Verify runner Chrome
# GitHub's Ubuntu runner image already ships Google Chrome, so use that
# directly for the headless e2e lane instead of downloading Playwright
# browser bundles inside the 30 minute job budget.
run: google-chrome --version
- name: Generate Paperclip config
run: |
mkdir -p ~/.paperclip/instances/default
cat > ~/.paperclip/instances/default/config.json << 'CONF'
{
"$meta": { "version": 1, "updatedAt": "2026-01-01T00:00:00.000Z", "source": "onboard" },
"database": { "mode": "embedded-postgres" },
"logging": { "mode": "file" },
"server": { "deploymentMode": "local_trusted", "host": "127.0.0.1", "port": 3100 },
"auth": { "baseUrlMode": "auto" },
"storage": { "provider": "local_disk" },
"secrets": { "provider": "local_encrypted", "strictMode": false }
}
CONF
- name: Run e2e tests
env:
PAPERCLIP_E2E_SKIP_LLM: "true"
PAPERCLIP_PLAYWRIGHT_CHANNEL: "chrome"
run: |
# Playwright's own --shard balances by test count, and one spec
# (smoke-lab) is ~40% of the lane's wall clock. Partition by recorded
# spec duration instead so both runners finish together.
specs="$(node ./scripts/e2e-shard.mjs \
--shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }})"
echo "shard ${{ matrix.shard_label }} specs: $specs"
# specs is an intentional argument list.
# shellcheck disable=SC2086
pnpm run test:e2e $specs
- name: Upload Playwright report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: always()
with:
name: playwright-report-${{ matrix.shard_index }}
path: |
tests/e2e/playwright-report/
tests/e2e/test-results/
retention-days: 14
- name: Upload CI profile timings
continue-on-error: true
if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
path: .ci-profile/*.ndjson
include-hidden-files: true
if-no-files-found: warn
retention-days: 14
e2e:
name: e2e
if: ${{ always() }}
needs: [gate, policy, e2e_shards]
runs-on: ${{ needs.gate.outputs.runner || 'ubuntu-latest' }}
timeout-minutes: 5
steps:
- name: Validate every planned suite result
env:
FULL_CI: ${{ needs.gate.outputs.full_ci }}
CI_PLAN: ${{ needs.gate.outputs.plan }}
CI_RESULTS: ${{ toJSON(needs) }}
run: |
node --input-type=module <<'NODE'
import assert from 'node:assert/strict';
const results = JSON.parse(process.env.CI_RESULTS);
assert.equal(results.gate.result, 'success', 'CI routing must succeed');
assert.equal(results.policy.result, 'success', 'Policy must succeed');
assert.ok(['true', 'false'].includes(process.env.FULL_CI), 'Valid scope required');
const plan = process.env.CI_PLAN ? JSON.parse(process.env.CI_PLAN) : null;
if (plan) assert.equal(plan.version, 1, 'Supported task plan required');
for (const job of ["e2e_shards"]) {
if (plan) assert.equal(typeof plan.jobs[job]?.run, 'boolean', 'Missing suite decision: ' + job);
const selected = process.env.FULL_CI === 'true' && (!plan || plan.jobs[job].run);
assert.equal(results[job].result, selected ? 'success' : 'skipped', job);
}
NODE