perf(release): number canaries by master position and confirm them outside the release lock (#21917)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every master push runs a canary, and only one canary at a time may
run the release workflow
> - After #21554 and #21555 a canary held that lock for 35.5 min, and
13-16 min of it was waiting for npm to show the new packages
> - The wait was required: the next canary computed `canary.N` from
npm's version list, so starting before npm showed the previous canary
would reuse its number
> - This pull request numbers a canary by its commit's position on
master and moves the registry confirmation, the git tag, the Docker
dispatch and the onboarding smoke to a separate `canary-finalize.yml`
run
> - The benefit is that the release lock is held for about 20-25 min per
canary instead of 35, with the same checks before a canary is tagged

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Canary throughput. In run 43715037497 (2026-11-15) `publish_canary` took
23.7 min, and 13.3 min of that was "Waiting for registry visibility"
(13:08:50 to 13:22:11). The next canary could not start until that run
ended.

**Current behavior**
`release.sh canary` sets `N` to (highest published `-canary.N` for
today) + 1, then publishes, waits for npm to show every package, checks
dist-tags, installs `paperclipai@canary`, and creates
`canary/v<version>`. `publish_canary` pushes the tag, dispatches
`docker.yml`, and `smoke_canary_onboarding` runs in the same run.

**Proposed behavior**
`N` = `git rev-list --count HEAD` (full history required), or the
registry number + 1 when that is higher (a rerun of the same commit).
`release.sh canary --defer-registry-confirmation` ends after npm accepts
every publish. `publish_canary` dispatches `canary-finalize.yml`, which
does the confirmation, tag, Docker dispatch and smoke outside the
release lock. Nightly, beta and stable numbering and flow are unchanged.

Refs #21554, #21555

## What Changed

- `scripts/release-lib.sh`: `next_canary_version_from_source`. A
publishing canary refuses a shallow checkout; a dry run (the PR Canary
Dry Run job) falls back to the registry number.
- `scripts/release.sh`: canary uses the new number; new canary-only
`--defer-registry-confirmation` flag skips the registry wait, the
dist-tag check and the local tag, and writes the version to
`PAPERCLIP_RELEASE_VERSION_FILE`.
- `scripts/confirm-canary-publication.sh` (new): waits for registry
visibility, checks provenance, checks dist-tags with
`--allow-newer-canary`, installs `paperclipai@<version>`, then creates
and pushes `canary/v<version>`. A tag already on the same commit is
accepted, so reruns are safe.
- `scripts/verify-canary-provenance.mjs` (new): every package's npm SLSA
provenance must name `.github/workflows/release.yml` on
`refs/heads/master` in `paperclipai/paperclip` at the source commit. A
dispatch with invented inputs cannot tag anything.
- `scripts/verify-release-registry-state.mjs`: `--allow-newer-canary`
accepts a `canary` dist-tag that a newer canary already moved; older,
foreign or missing tags still fail.
- `.github/workflows/canary-finalize.yml` (new, `workflow_dispatch`,
master only): `confirm_canary` (contents: write, actions: write) and
`smoke_canary_onboarding` (downloads the publisher's lockfile artifact
by run id). Inputs reach shell steps only through a validating step.
- `.github/workflows/release.yml`: `publish_canary` hands off to
`canary-finalize.yml` and drops to `contents: read`; its tag push,
Docker dispatch and smoke job move to `canary-finalize.yml`.
- Tests and `doc/RELEASING.md`.

## Verification

- `node --test .github/scripts/tests/*.test.mjs
scripts/release-lib.test.mjs
scripts/verify-release-registry-state.test.mjs
scripts/verify-canary-provenance.test.mjs`: 846 pass.
`scripts/__tests__/release-verify-workflow.test.mjs`: 73 pass.
- New tests: numbering by master position, rerun moves past a published
number, shallow refusal and dry-run fallback; newer/older/foreign
dist-tags; provenance from another commit, workflow, branch, repository
or package is rejected; finalize wiring, permissions and input handling.
- Real npm data: `verify-canary-provenance.mjs 2026.1116.0-canary.7
b4fa1c7f... paperclipai @paperclipai/shared @paperclipai/server` passes;
the same version with another commit fails.
- `confirm-canary-publication.sh 2026.1116.0-canary.7 b4fa1c7f...` from
a checkout of that commit ran all steps against npm in 61 s and ended
with "canary/v2026.1116.0-canary.7 already points at b4fa1c7f...",
pushing nothing.
- `release.sh canary --dry-run --defer-registry-confirmation` and
`release.sh nightly --defer-registry-confirmation` both stop with "only
applies to a published canary".
- `actionlint` reports no new findings compared with master.
- Not verified before merge: a full publish through
`canary-finalize.yml`. It can run only from master. The first canary
after merge is the check.

## Risks

- Canary numbers jump from `canary.7` to about `canary.8144` (master has
8,143 commits). This is valid semver and still higher than every earlier
canary; users install through the `canary` dist-tag.
- The `canary/v` tag, Docker image and smoke now finish after the
release run ends. Nightly selects only tagged canaries, as before. If
finalize fails, that canary is on npm without a tag, so nightly skips
it, as it skips any canary whose old in-run verification failed.
- A canary can now publish while the previous one is still being
confirmed. Publishing itself is still serialized, so the `canary`
dist-tag only moves forward.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-16 11:37:20 -06:00
1 parent 93111b175b
commit d16c06308d
15 files changed
+742 -179

No files matched your search

+151
View File
@@ -0,0 +1,151 @@
name: Canary finalize
run-name: Canary finalize ${{ inputs.version }}
on:
workflow_dispatch:
inputs:
version:
description: Canary version that publish_canary published
required: true
type: string
source_sha:
description: Full source commit SHA of that canary
required: true
type: string
release_run_id:
description: Release workflow run that published it
required: true
type: string
permissions: {}
concurrency:
group: canary-finalize-${{ inputs.version }}
cancel-in-progress: false
env:
NPM_PUBLISH_VERIFY_ATTEMPTS: "180"
NPM_PUBLISH_VERIFY_DELAY_SECONDS: "10"
jobs:
confirm_canary:
name: Confirm canary on npm and tag it
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master'
runs-on: ubuntu-latest
timeout-minutes: 45
permissions:
contents: write
actions: write
outputs:
version: ${{ steps.inputs.outputs.version }}
source_sha: ${{ steps.inputs.outputs.source_sha }}
release_run_id: ${{ steps.inputs.outputs.release_run_id }}
steps:
- name: Validate inputs
id: inputs
env:
VERSION: ${{ inputs.version }}
SOURCE_SHA: ${{ inputs.source_sha }}
RELEASE_RUN_ID: ${{ inputs.release_run_id }}
run: |
set -euo pipefail
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+-canary\.[0-9]+$ ]] || { echo "Error: not a canary version" >&2; exit 1; }
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]] || { echo "Error: not a full commit SHA" >&2; exit 1; }
[[ "$RELEASE_RUN_ID" =~ ^[1-9][0-9]*$ ]] || { echo "Error: not a run id" >&2; exit 1; }
{
echo "version=$VERSION"
echo "source_sha=$SOURCE_SHA"
echo "release_run_id=$RELEASE_RUN_ID"
} >> "$GITHUB_OUTPUT"
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ steps.inputs.outputs.source_sha }}
fetch-depth: 0
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- name: Configure git author
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
- name: Confirm the canary and push its tag
env:
VERSION: ${{ steps.inputs.outputs.version }}
SOURCE_SHA: ${{ steps.inputs.outputs.source_sha }}
PUBLISH_REMOTE: origin
run: ./scripts/confirm-canary-publication.sh "$VERSION" "$SOURCE_SHA"
- name: Build Docker images for the canary tag
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ steps.inputs.outputs.version }}
RELEASE_RUN_ID: ${{ steps.inputs.outputs.release_run_id }}
run: |
{
echo "## Canary published"
echo ""
echo "- Published canary: \`$VERSION\` ([release run](https://github.com/$GITHUB_REPOSITORY/actions/runs/$RELEASE_RUN_ID))"
echo "- Docker build dispatched at \`canary/v$VERSION\`"
} >> "$GITHUB_STEP_SUMMARY"
gh workflow run docker.yml --ref "refs/tags/canary/v$VERSION" --repo "$GITHUB_REPOSITORY"
smoke_canary_onboarding:
needs: confirm_canary
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: read
actions: read
env:
PAPERCLIPAI_VERSION: ${{ needs.confirm_canary.outputs.version }}
PAPERCLIP_PLAYWRIGHT_CHANNEL: chrome
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.confirm_canary.outputs.source_sha }}
persist-credentials: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- name: Restore canary smoke lockfile
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: canary-smoke-lockfile-${{ needs.confirm_canary.outputs.source_sha }}
run-id: ${{ needs.confirm_canary.outputs.release_run_id }}
github-token: ${{ github.token }}
- name: Install test dependencies
run: pnpm install --frozen-lockfile
- name: Show browser version
run: google-chrome --version
- name: Smoke exact published canary through onboarding
env:
PAPERCLIP_CANARY_SMOKE_SERVER_LOG: ${{ runner.temp }}/canary-onboarding-server.log
run: pnpm run test:canary-onboarding-smoke
- name: Upload failed canary onboarding diagnostics
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: canary-onboarding-smoke-${{ needs.confirm_canary.outputs.version }}
if-no-files-found: warn
retention-days: 14
path: |
${{ runner.temp }}/canary-onboarding-server.log
tests/canary-onboarding/playwright-report/
tests/canary-onboarding/test-results/
+2 -2
View File
@@ -9,8 +9,8 @@ on:
- "nightly/v*"
- "beta/v*"
# Release workflows push lane tags with GITHUB_TOKEN, and GitHub suppresses
# push-triggered runs for those, so release.yml dispatches this workflow at
# the new tag ref instead. The tag mapping below keys off github.ref either
# push-triggered runs for those, so release.yml (canary-finalize.yml for a
# canary) dispatches this workflow at the new tag ref instead. The tag mapping below keys off github.ref either
# way.
workflow_dispatch:
+4 -1
View File
@@ -249,7 +249,10 @@ jobs:
run: node "$CI_NODE_TEST" ./scripts/__tests__/e2e-shard.test.mjs
- name: Test release verify workflow wiring
run: node "$CI_NODE_TEST" ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/eval-schedule-monitor.test.mjs ./scripts/__tests__/prepare-runner-release-image.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/cloud-readiness-runner-loss.test.mjs ./scripts/standard-image-contract.test.mjs ./scripts/reuse-canary-image.test.mjs
run: |
tests=(./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/eval-schedule-monitor.test.mjs ./scripts/__tests__/prepare-runner-release-image.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/cloud-readiness-runner-loss.test.mjs ./scripts/standard-image-contract.test.mjs ./scripts/reuse-canary-image.test.mjs)
if [[ -f ./scripts/verify-canary-provenance.test.mjs ]]; then tests+=(./scripts/verify-canary-provenance.test.mjs); fi
node "$CI_NODE_TEST" "${tests[@]}"
- name: Test merge-queue workflow wiring
# This workflow loads from master and tests the caller's checkout. A
+16 -90
View File
@@ -396,9 +396,9 @@ jobs:
outputs:
canary_version: ${{ steps.canary_tag.outputs.version }}
permissions:
contents: write
contents: read
id-token: write
# For the explicit docker.yml dispatch below.
# For the canary-finalize.yml dispatch below.
actions: write
steps:
@@ -488,7 +488,8 @@ jobs:
- name: Publish canary
env:
GITHUB_ACTIONS: "true"
run: ./scripts/release.sh canary --skip-verify
PAPERCLIP_RELEASE_VERSION_FILE: ${{ runner.temp }}/canary-version
run: ./scripts/release.sh canary --skip-verify --defer-registry-confirmation
- name: Dump npm debug logs
if: failure()
@@ -500,100 +501,25 @@ jobs:
-e 's#((authorization|_authToken|_auth|node_auth_token|npm_token)"?[[:space:]]*[:=][[:space:]]*"?)(Bearer[[:space:]]+)?[^",[:space:]]+#\1***REDACTED***#Ig'
done
- name: Push canary tag
- name: Hand off npm confirmation, the tag and the image to canary-finalize
id: canary_tag
run: |
tag="$(git tag --points-at HEAD | grep '^canary/v' | head -1)"
if [ -z "$tag" ]; then
echo "Error: no canary tag points at HEAD after release." >&2
exit 1
fi
git push origin "refs/tags/${tag}"
echo "version=${tag#canary/v}" >> "$GITHUB_OUTPUT"
# Canary images previously relied on the master-push docker.yml run,
# whose single pending concurrency slot gets superseded by every
# newer push — on a busy day no canary image publishes at all (five
# consecutive canaries shipped npm packages with no cloud image on
# 2026-09-06, starving downstream managed deploys for ~18 hours).
# Tag pushes made with GITHUB_TOKEN do not fire docker.yml's
# triggers, so dispatch the image build at the canary tag
# explicitly, exactly like the nightly and beta lanes: the run keys
# its concurrency off the tag ref, so no master push can supersede
# it, and docker.yml's `type=sha` mapping publishes the
# standard sha-<short> images either way.
- name: Build Docker images for the canary tag
env:
GH_TOKEN: ${{ github.token }}
VERSION_FILE: ${{ runner.temp }}/canary-version
SOURCE_SHA: ${{ github.sha }}
run: |
set -euo pipefail
version="$(cat "$VERSION_FILE")"
[[ "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+-canary\.[0-9]+$ ]] || { echo "Error: release.sh wrote no canary version" >&2; exit 1; }
echo "version=$version" >> "$GITHUB_OUTPUT"
gh workflow run canary-finalize.yml --ref master --repo "$GITHUB_REPOSITORY" \
-f version="$version" -f source_sha="$SOURCE_SHA" -f release_run_id="$GITHUB_RUN_ID"
{
echo "## Canary published"
echo "## Canary published to npm"
echo ""
echo "- Published canary: \`${{ steps.canary_tag.outputs.version }}\`"
echo "- Docker build dispatched at \`canary/v${{ steps.canary_tag.outputs.version }}\`"
echo "- Version: \`$version\`"
echo "- npm confirmation, \`canary/v$version\`, the Docker build and the onboarding smoke run in Canary finalize."
} >> "$GITHUB_STEP_SUMMARY"
gh workflow run docker.yml --ref "refs/tags/canary/v${{ steps.canary_tag.outputs.version }}" --repo "$GITHUB_REPOSITORY"
# The package is already public when this gate runs. A red result leaves the
# immutable canary in npm, but makes the release workflow visibly fail before
# anyone mistakes an installable package for an onboardable one.
smoke_canary_onboarding:
needs: publish_canary
if: ${{ !cancelled() && needs.publish_canary.result == 'success' }}
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: read
env:
PAPERCLIPAI_VERSION: ${{ needs.publish_canary.outputs.canary_version }}
PAPERCLIP_PLAYWRIGHT_CHANNEL: chrome
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.sha }}
persist-credentials: false
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
# Default download scope is this workflow run, including job reruns.
# A missing artifact fails instead of resolving a different dependency set.
- name: Restore canary smoke lockfile
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: canary-smoke-lockfile-${{ github.sha }}
- name: Install test dependencies
run: pnpm install --frozen-lockfile
- name: Show browser version
run: google-chrome --version
- name: Smoke exact published canary through onboarding
env:
PAPERCLIP_CANARY_SMOKE_SERVER_LOG: ${{ runner.temp }}/canary-onboarding-server.log
run: pnpm run test:canary-onboarding-smoke
- name: Upload failed canary onboarding diagnostics
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: canary-onboarding-smoke-${{ needs.publish_canary.outputs.canary_version }}
if-no-files-found: warn
retention-days: 14
path: |
${{ runner.temp }}/canary-onboarding-server.log
tests/canary-onboarding/playwright-report/
tests/canary-onboarding/test-results/
# ----- Nightly lane -----------------------------------------------------
# Once a night (or on a forced nightly dispatch), promote the newest master