mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds, probes, gates and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after that image is published. > - Each master push rebuilds the image from scratch: about 10.6 min for amd64 and 7.8 min for arm64. > - The owner approved a merge queue for Core. A merge queue tests the exact commit that master will become, before the merge. > - This pull request builds and probes the image for that exact commit in the queue, with no registry credentials. The later master push pushes those exact bytes instead of rebuilding. When no matching verified build exists, master builds from source exactly as today. > - The benefit is that the image legs on master drop from about 8–11 min to about 3–4 min, and nothing changes until the owner turns the queue on. ## Linked Issues or Issue Description Refs #21013 (it measured the image path and proposed this plan). **What existing behavior does this improve?** Every master push rebuilds the standard image, even when the same commit could already have been built while it waited to merge. **Subsystem affected** `.github/workflows/docker.yml` (`build-and-push` only), new `.github/workflows/docker-merge-group.yml`, new `scripts/merge-group-image.mjs`. **Current behavior** `build-and-push` always runs the full BuildKit build on master: 639 s (amd64) and 468 s (arm64) in run 43133035209. **Proposed behavior** On a `merge_group` event, build both architectures for the merge-group commit, probe PID 1, and upload an OCI archive plus a record of commit, tree, architecture and digest. On the master push for the same commit, each leg pushes that archive by digest instead of building. Everything after the leg (probe, exact-source gate, signed-index check, attestation) is unchanged. **Reason and benefit** The build moves before the merge, where it runs in parallel with the queue's other checks. The master leg only downloads and pushes. **Breaking changes** None. Without `merge_group` events, `find` returns no run and the leg builds as before. **Additional context** This PR does not touch the exact-source gate jobs (`source_verification`, `publication_verification`, `merge-and-push`), `scripts/watch-release-source.mjs` or `cloud-readiness.yml`. Another change owns them. ## What Changed - **`docker-merge-group.yml` (new, `on: merge_group` only).** It runs only for this repository's master queue. Workflow permissions are `{}` and the job has `contents: read` only. There are no secrets, no registry login, no `cache-to`, no `push: true`, no environments, and checkout uses `persist-credentials: false`. It runs on GitHub-hosted runners, not the paid AWS fleets. Steps: build `target: production` with the same build args as master (`PAPERCLIP_BUILD_COMMIT` is the merge-group SHA, which becomes the master SHA). Read the public registry cache. Write `type=oci` to a tar. `docker load` it and run `scripts/assert-orphan-reaping.sh`. Only then write `record.json` and upload `merge-group-image-<arch>` (3-day retention). - **`scripts/merge-group-image.mjs` (new).** - `find` (master push only): selects a completed, successful run of this workflow, with event `merge_group`, the same `head_sha`, repository and head repository both `1170821064`, and a `gh-readonly-queue/master/` head branch. It needs exactly one unexpired artifact for the architecture. - `record` (merge group only): reads the archive's `index.json`, checks each blob's sha256 and that the image is `linux/<arch>`, and writes `{sha, tree, arch, digest}`. - `verify` (master push only): re-reads the archive the same way and requires record SHA = `GITHUB_SHA`, record tree = `git rev-parse HEAD^{tree}` of the master checkout, same architecture, and archive digest = recorded digest. - **`docker.yml` `build-and-push`.** - The leg gets `actions: read` so it can download the artifact. - New steps run after the registry login: find, download, then promote (`verify`, then `skopeo copy --all --preserve-digests oci-archive:… docker://ghcr.io/…@<digest>`). The step sets its `digest` output only after the push succeeds. - All three new steps are `continue-on-error`. Buildx, labels, metadata and "Build and push by digest" run when `steps.promote.outputs.digest == ''`, so any failure builds from source. - "Select image digest" feeds the existing PID 1 probe and digest export. - **Tests (`scripts/merge-group-image.test.mjs`, 8 tests):** - Run selection rejects another commit, another event, another workflow, a fork head, another repository, another queue base, and failed, cancelled or running builds. - Artifact selection. - Archive digest with and without provenance, a tampered blob, the wrong architecture, and a real tar read through the system `tar`. - Record checks for commit, tree, architecture and digest mismatch. - The CLI refuses `record` outside `merge_group` and `find`/`verify` outside a canonical master push. - Static checks of the merge-group workflow: credentials, triggers, permissions and runners. - Static checks of the `docker.yml` wiring: promotion is gated and falls back, the output is set only after the push, and every build step has the fallback condition. ### How promotion proves it publishes the verified tree 1. `find` accepts only a run whose `head_sha` equals `github.sha` of the push, meaning the queue landed the commit it tested (unverified: see "Merge method" in Risks). 2. `verify` compares the record's tree with the tree of the master checkout, and its commit with `GITHUB_SHA`. 3. The archive is content-addressed. The digest is recomputed from the archive blobs and must equal the recorded digest. `skopeo --preserve-digests` to that digest reference fails rather than produce other bytes. 4. The leg then runs the PID 1 probe on that registry digest. `merge-and-push` still signs only an index whose Linux images are the probed digests (#21013), after the unchanged exact-source gate. 5. The pipeline itself is trusted only as master already ran it. `find` and `verify` run the copy of `scripts/merge-group-image.mjs` from the push's `before` commit (`git show "$BEFORE:…"`), not the copy the push brings. `verify` refuses promotion if anything in `before..HEAD` (every commit of a multi-PR group, not only the tip) changes `docker-merge-group.yml`, `docker.yml`, the promotion script or the PID 1 probe. Such a push builds from source. So does the first push after this PR merges, because its `before` has no verifier. ## Verification - `node --test scripts/merge-group-image.test.mjs`: 10 pass. One builds a real git history in which a two-PR push changes the pipeline in its first commit only, and checks that the range check catches it. - `node --test` on the workflow and contract suites (cloud-readiness, docker-canary-promotion, post-merge-runner-routing, lockfile-refresh-workflows, release-verify-workflow, reuse-canary-image, standard-image-contract, cloud-source-verification, merge-group-image): 492 pass. - `vitest run server/src/__tests__/docker-build-stamp.test.ts server/src/__tests__/container-init-reaping.test.ts`: 22 pass. - `actionlint` on both workflows: no new findings (the 6 remaining are pre-existing `ls` and word-splitting notes in `docker.yml`). - Local: `skopeo 1.24.1 copy --all --preserve-digests` between OCI layouts kept the index digest. It accepted a `docker://…@sha256:` destination and failed only at the network step. Cannot be exercised before the queue exists, because no `merge_group` event fires: - the merge-group run itself: the hosted-runner build time, disk use with tar plus `docker load`, `docker load` of a BuildKit OCI tar, and a ~5 GB artifact upload; - the cross-run artifact download on master; - whether `skopeo` exists on the runs-on AMI and accepts digest destinations in its installed version. Each master-side failure falls back to the full build. A merge-group failure only blocks the queue if its check is required, which is why that check is required last in the steps below. ## Risks - **Inert until enabled.** No `merge_group` events fire before the owner adds a merge queue. On master, `find` lists runs (one API call per leg, about 1 s), finds none, and the leg builds as today. The only always-on change is the leg's `actions: read` permission. - **Build cache freshness.** A promoted master push writes no `buildcache-<arch>`, and merge-group builds read but never write it. With every push promoted, the cache ages, so dependency or weekly tool-layer changes rebuild in each merge-group run until a fallback build refreshes the cache. A follow-up can refresh the cache from master without blocking publication. - **Publication time is still gate-bound.** The exact-source gate (about 13–14 min after push) still decides when the image publishes. See "Expected timing". - **Merge method (unverified).** Promotion needs the master commit to be exactly the commit the queue tested, because the image stamps the commit SHA (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack revision). Matching only the tree would publish an image stamped with another SHA, so this PR does not relax the check. I expect the queue to fast-forward master to its tested commit. GitHub's merge-queue page does not state this, and it cannot be tested before the queue exists. If the queue lands a different SHA, promotion never matches and every push builds from source. That is safe but inert. Step 4 below checks this. - **skopeo on the AWS fleet.** The promote step checks for `skopeo` first and logs "Merge-queue promotion unavailable" if it is missing, then builds from source. If that notice appears, add skopeo to the fleet image. - **Hosted runner capacity.** amd64 merge-group builds run on 4-core hosted runners, colder and slower than the AWS fleet. That adds queue latency, not master latency. ### Settings the owner must enable, in order 1. **Prerequisite, a code change, not a setting.** The `master` ruleset (id 13619726) requires `ci / verify` and `ci / e2e`. Those come from `pr.yml`, which triggers only on `pull_request`. A merge queue waits for required checks on the `merge_group` event too, so every entry would stall until its timeout. First decide, and merge in a separate PR, how `ci / verify` and `ci / e2e` report on `merge_group`: a full CI run on the merge-group commit, or a fast pass that relies on the PR-head result. Today the ruleset does not require up-to-date branches either. 2. Merge this PR. Nothing changes yet. 3. In Settings → Rules → Rulesets → `master` (13619726), add **Require merge queue**: - merge method Squash; - build concurrency 5, minimum group size 1, maximum group size 5, wait 0 min; - "Require all queue entries to pass required checks"; - status check timeout 60 min. Do not add the image build to the required checks yet. Promotion is then opportunistic: if the merge-group build has not finished when master is pushed, master builds from source. 4. After the first queued merge, confirm promotion can match. The master commit SHA must equal the `head_sha` of its `Docker merge-queue build` run, and the next master `Docker` run's "Promote the merge-queue image" step must succeed. If the SHAs differ, promotion stays inert under this merge method. Stop here and report it. 5. After about a week of green `Docker merge-queue build` runs, add `build (linux/amd64, ubuntu-latest, amd64)` and `build (linux/arm64, ubuntu-24.04-arm, arm64)` (app: GitHub Actions) to the ruleset's required status checks. Then every queued commit has a verified image ready to promote. 6. The existing bypass actors (org admins, repository admins, integration 5534676) can still merge outside the queue. Those pushes have no merge-group run and build from source. ### Rollback - Remove the image checks from the required status checks (step 5), or remove **Require merge queue** (step 3). Master pushes then find no merge-group run and build from source. Nothing to migrate. - Code: revert this PR. Old merge-group artifacts expire after 3 days. Promoted images are ordinary digests under `ghcr.io/paperclipai/paperclip` and stay valid. ### Expected timing - Image leg on master: about 3–4 min (download about 5 GB, push the new layers by digest, PID 1 probe) instead of 10.6 min (amd64) and 7.8 min (arm64). - Push to published: still bounded by the exact-source gate on master, about 14–15 min (gate plus about 45 s for `merge-and-push`). Today it is the same bound, but with the legs as a second 11-min path and the serialized-run queue in front. - If the gate later accepts proof produced in the merge queue (owned by the gate change), push to published would be about 4–5 min (leg promotion plus `merge-and-push`). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only GitHub API and registry reads). Default reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
584 lines
24 KiB
Markdown
584 lines
24 KiB
Markdown
# Releasing Paperclip
|
|
|
|
Maintainer runbook for shipping Paperclip across npm, GitHub, and the website-facing changelog surface.
|
|
|
|
The release model is now commit-driven:
|
|
|
|
1. Every push to `master` publishes a canary automatically.
|
|
2. Once a night, the newest master commit with a green canary publish is
|
|
smoke-tested and republished as the nightly.
|
|
3. Betas are manual, human-approved promotions of a chosen nightly.
|
|
4. Stable releases promote a beta that has soaked for at least 3 days
|
|
(bypass requires a written justification).
|
|
5. Stable release notes live in `releases/vYYYY.MDD.P.md`. They are
|
|
drafted automatically when a beta is published (as
|
|
`releases/beta/v<beta-version>.md` on `master`), edited during the
|
|
soak, and moved to the versioned name after the stable ships.
|
|
6. Only stable releases get GitHub Releases.
|
|
|
|
The user-facing guide to the channels is [`CHANNELS.md`](CHANNELS.md).
|
|
|
|
## Versioning Model
|
|
|
|
Paperclip uses calendar versions that still fit semver syntax:
|
|
|
|
- stable: `YYYY.MDD.P`
|
|
- canary: `YYYY.MDD.P-canary.N`
|
|
- nightly: `YYYY.MDD.P-nightly.N`
|
|
- beta: `YYYY.MDD.P-beta.N`
|
|
|
|
Examples:
|
|
|
|
- first stable on March 18, 2026: `2026.318.0`
|
|
- second stable on March 18, 2026: `2026.318.1`
|
|
- fourth canary for the `2026.318.1` line: `2026.318.1-canary.3`
|
|
- first nightly cut on March 18, 2026: `2026.318.1-nightly.0`
|
|
- first beta promoted on March 18, 2026: `2026.318.1-beta.0`
|
|
|
|
A promotion republishes the exact source commit of the previous lane's build
|
|
(canary → nightly → beta); the version dates the promotion, not the source
|
|
build.
|
|
|
|
Important constraints:
|
|
|
|
- the middle numeric slot is `MDD`, where `M` is the UTC month and `DD` is the zero-padded UTC day
|
|
- use `2026.303.0` for March 3, not `2026.33.0`
|
|
- do not use leading zeroes such as `2026.0318.0`
|
|
- do not use four numeric segments such as `2026.3.18.1`
|
|
- the semver-safe canary form is `2026.318.0-canary.1`
|
|
|
|
## Release Surfaces
|
|
|
|
Every stable release has four separate surfaces:
|
|
|
|
1. **Verification** — the exact git SHA passes typecheck, tests, and build
|
|
2. **npm** — `paperclipai` and public workspace packages are published
|
|
3. **GitHub** — the stable release gets a git tag and GitHub Release
|
|
4. **Website / announcements** — the stable changelog is published externally and announced
|
|
|
|
A stable release is done only when all four surfaces are handled.
|
|
|
|
Canaries, nightlies, and betas only cover the first two surfaces plus an
|
|
internal traceability tag.
|
|
|
|
## Core Invariants
|
|
|
|
- canaries publish from `master`
|
|
- the canary publisher checks out the exact master push SHA. It verifies the
|
|
repository, event, branch, and checked-out commit before attaching local
|
|
`master` at that same SHA. It never switches to a newer branch tip. The local
|
|
release script still rejects detached or non-master checkouts.
|
|
- nightlies republish a commit that already shipped a canary (the commit must
|
|
carry a `canary/v*` tag), and only after the release smoke suite passes
|
|
against that exact published canary
|
|
- betas republish a commit that already shipped a nightly (the commit must
|
|
carry a `nightly/v*` tag), behind the `npm-beta` approval gate, and the
|
|
published beta is re-smoked
|
|
- stables publish from an explicitly chosen source ref, which must have
|
|
shipped as a beta at least 3 days earlier unless a written justification
|
|
is provided
|
|
- tags point at the original source commit, not a generated release commit
|
|
- stable notes are always `releases/vYYYY.MDD.P.md` in the end state: a
|
|
promoted beta's notes are drafted and edited at
|
|
`releases/beta/v<beta-version>.md` on `master` during the soak (the
|
|
promoted commit cannot carry a file named for a promotion date that was
|
|
unknown when it was created), and a post-stable canonicalization PR
|
|
moves them to the versioned name
|
|
- canaries, nightlies, and betas never create GitHub Releases
|
|
- canaries, nightlies, and betas never require changelog generation
|
|
- Docker `:latest` moves only on stable releases; canary dispatches promote
|
|
`:canary`, nightly builds `:nightly`, and beta builds `:beta`
|
|
|
|
## TL;DR
|
|
|
|
### Canary
|
|
|
|
Every push to `master` runs the canary path inside [`.github/workflows/release.yml`](../.github/workflows/release.yml).
|
|
|
|
It:
|
|
|
|
- verifies the pushed commit
|
|
- computes the canary version for the current UTC date
|
|
- publishes workspace packages dependency-first under npm dist-tag `canary`
|
|
- waits for each package version to become registry-visible before continuing
|
|
- publishes the user-facing `paperclipai` package last, so `paperclipai@canary` does not advance before the full package set exists
|
|
- verifies that `canary` resolves to the just-published version and that published internal dependencies exist on npm
|
|
- installs `paperclipai@canary` into a clean temporary prefix as the final npm gate
|
|
- starts the exact published canary through a separate fresh npm install for the onboarding smoke; npm `ETARGET` and verified canary tarball availability failures retry, up to three attempts within two minutes, before onboarding runs once under the existing five-minute startup deadline
|
|
- fails by default if npm leaves `latest` pointing at a canary; use `--allow-canary-latest` only when that state is intentional
|
|
- creates a git tag `canary/vYYYY.MDD.P-canary.N`
|
|
|
|
After publication, the onboarding smoke checks out the same source SHA and
|
|
installs its test dependencies from the publisher's resolved lockfile with
|
|
`--frozen-lockfile`. The same-run artifact is captured before tracked files are
|
|
restored or release versions are rewritten. This lets a source commit use its
|
|
current patches and manifests while the separate lockfile-refresh PR is still
|
|
pending, without resolving new dependencies in the smoke job. The smoke still installs and tests
|
|
the exact published canary version, not a workspace build.
|
|
|
|
The tarball retry applies only to an unambiguous npm `E404` for a canonical
|
|
public-registry tarball of a CI-published Paperclip package at that exact canary
|
|
version. The helper queries npm's effective global and applicable scoped registry
|
|
in the failed install's prefix, working directory, and environment. A custom or
|
|
unknown effective registry fails. Configuration output stays private in memory.
|
|
Fresh primary-registry metadata must match the package, version, and tarball URL
|
|
and announce a valid SHA512 integrity value. This establishes the announced
|
|
artifact identity; it does not download the tarball or independently verify its
|
|
contents. The actual npm install still checks tarball integrity.
|
|
|
|
Configuration and metadata checks share a five-second deadline, including the
|
|
response body, within the same two-minute acquisition budget. Metadata is limited
|
|
to 64 KiB. Redirects, authentication errors, missing metadata, conflicting npm
|
|
errors, and other artifacts fail. A persistent promised-tarball failure also
|
|
fails after three installs. Onboarding errors never restart acquisition or
|
|
repeat onboarding.
|
|
|
|
The lockfile artifact is retained for 14 days. A publisher-job rerun replaces its
|
|
source-named artifact; a smoke-only rerun uses the existing artifact. A missing
|
|
artifact fails the job rather than falling back to a different dependency set.
|
|
|
|
Users install canaries with:
|
|
|
|
```bash
|
|
npx paperclipai@canary onboard
|
|
# or
|
|
npx paperclipai@canary onboard --data-dir "$(mktemp -d /tmp/paperclip-canary.XXXXXX)"
|
|
```
|
|
|
|
### Nightly
|
|
|
|
A scheduled job in [`.github/workflows/release.yml`](../.github/workflows/release.yml)
|
|
runs once a night at 09:00 UTC.
|
|
|
|
It:
|
|
|
|
- selects the newest commit on `master` that carries a `canary/v*` tag (the
|
|
tag is pushed only after a successful canary publish, so it is the
|
|
green-publish signal)
|
|
- skips with a job-summary reason when there is no new candidate or the
|
|
candidate already shipped as a nightly
|
|
- runs the release smoke suite ([`release-smoke.yml`](../.github/workflows/release-smoke.yml))
|
|
against that exact published canary version — red smoke means no nightly
|
|
tonight
|
|
- republishes the same source commit as `YYYY.MDD.P-nightly.N` under the npm
|
|
dist-tag `nightly` (the commit was already verified by its canary run, so
|
|
verification is not repeated)
|
|
- creates and pushes the git tag `nightly/vYYYY.MDD.P-nightly.N`
|
|
- dispatches [`docker.yml`](../.github/workflows/docker.yml) at that tag to
|
|
publish the `:nightly` images
|
|
|
|
To force a nightly outside the schedule (recovery, or promoting a specific
|
|
canary), dispatch `release.yml` with `channel: nightly`. Leave
|
|
`source_version` empty for automatic selection, or set it to an exact
|
|
canary version. `dry_run: true` previews the publish and skips smoke, the tag
|
|
push, and the Docker dispatch.
|
|
|
|
Users install nightlies with:
|
|
|
|
```bash
|
|
npx paperclipai@nightly onboard
|
|
```
|
|
|
|
### Beta
|
|
|
|
Betas are manual promotions. Dispatch
|
|
[`release.yml`](../.github/workflows/release.yml) with `channel: beta`.
|
|
|
|
- leave `source_version` empty to promote the newest nightly, or set it to an
|
|
exact nightly version such as `2026.807.0-nightly.0`
|
|
- the selection job resolves the nightly's source commit and fails loudly if
|
|
it does not exist or already shipped as a beta
|
|
- the publish waits for approval in the **`npm-beta` environment** — its
|
|
required reviewers are the promotion gate
|
|
- promotions run the release tooling of the source commit, so the source
|
|
nightly must postdate the beta channel's introduction; the selection job
|
|
rejects older sources with a clear error (in practice every nightly cut
|
|
after the beta tooling merged qualifies)
|
|
- the same commit is republished as `YYYY.MDD.P-beta.N` under the npm
|
|
dist-tag `beta`, tagged `beta/vYYYY.MDD.P-beta.N`, and `docker.yml` is
|
|
dispatched at that tag to publish the `:beta` images
|
|
- after publishing, the release smoke suite runs against the exact published
|
|
beta version as verification
|
|
- a `draft_stable_notes` job also generates the eventual stable's notes
|
|
skeleton — `releases/beta/v<beta-version>.md`, grouped from
|
|
`git log <last-stable-tag>..<source-commit>` — and force-pushes it to the
|
|
machine-owned `release-notes/v<beta-version>` branch. Open the PR from
|
|
the job-summary link (a human opens it so CI runs) and edit the notes
|
|
during the soak; the stable promotion reads the merged file from
|
|
`master`
|
|
- `dry_run: true` previews the publish and skips the tag push, Docker
|
|
dispatch, and post-publish smoke
|
|
|
|
Users install betas with:
|
|
|
|
```bash
|
|
npx paperclipai@beta onboard
|
|
```
|
|
|
|
#### Beta fix path: candidate branches
|
|
|
|
When one or two targeted fixes are needed before beta and waiting for the
|
|
next nightly (or absorbing a whole day of `master`) is wrong, build the beta
|
|
from a short-lived candidate branch:
|
|
|
|
1. cut `candidate/beta-<target>` from the chosen nightly's source commit
|
|
(for example `candidate/beta-2026.811.0`)
|
|
2. cherry-pick only the required fix commits onto it and push the branch
|
|
3. dispatch `release.yml` with `channel: beta` and `candidate_branch:
|
|
candidate/beta-<target>`
|
|
4. selection validates the branch name, rejects heads that already shipped
|
|
as a beta, records the cherry-picked commits in the job summary, and the
|
|
head runs **full verification** before publishing (it never went through
|
|
a canary or nightly)
|
|
5. after the beta ships, land the fixes on `master` normally and delete the
|
|
candidate branch
|
|
|
|
Use this sparingly: the happy path is promoting a nightly. A candidate build
|
|
has its own `-beta.N` identity and is never pretended to be the nightly it
|
|
was cut from.
|
|
|
|
### Stable
|
|
|
|
Use [`.github/workflows/release.yml`](../.github/workflows/release.yml) from the Actions tab with the manual `workflow_dispatch` inputs.
|
|
|
|
[Run the action here](https://github.com/paperclipai/paperclip/actions/workflows/release.yml)
|
|
|
|
Inputs:
|
|
|
|
- `channel`
|
|
- `stable` (the default) for a stable release; `beta` and `nightly` run
|
|
those lanes instead (see above)
|
|
- `source_ref`
|
|
- commit SHA, branch, or tag
|
|
- `stable_date`
|
|
- optional UTC date override in `YYYY-MM-DD`
|
|
- enter a date like `2026-03-18`, not a version like `2026.318.0`
|
|
- `skip_soak_justification`
|
|
- written reason for releasing a stable whose source has not soaked as a
|
|
beta for 3 days; leave empty for normal releases
|
|
- `dry_run`
|
|
- preview only when true
|
|
|
|
The stable preflight enforces the beta soak: the source commit must carry a
|
|
`beta/v*` tag whose npm publish time is at least 3 days old. If it is not,
|
|
the run fails unless `skip_soak_justification` is provided; the justification
|
|
is echoed into the job summary. Dry runs report soak state without blocking.
|
|
|
|
For a cherry-picked stable (the release fix path), cut
|
|
`candidate/release-<target>` from the chosen beta's source commit,
|
|
cherry-pick the required fixes, push the branch, and use it as
|
|
`source_ref`. The candidate head carries no `beta/v*` tag, so the soak gate
|
|
requires `skip_soak_justification` — that is deliberate: the exact bits were
|
|
not soaked, and the justification is the recorded trade-off. Reconcile the
|
|
fixes back to `master` and delete the branch after shipping.
|
|
|
|
Before running stable:
|
|
|
|
1. pick the beta you are promoting (its source commit is the `source_ref`)
|
|
2. confirm the beta has soaked for 3 days with no open blockers
|
|
3. resolve the target stable version with `./scripts/release.sh stable --date "$(date +%F)" --print-version`
|
|
4. make sure the notes PR from the beta's draft branch
|
|
(`release-notes/v<beta-version>`, adding
|
|
`releases/beta/v<beta-version>.md`) is merged on `master` — or, for
|
|
candidate builds, that the candidate branch itself carries
|
|
`releases/vYYYY.MDD.P.md`
|
|
5. run the stable workflow from that source ref
|
|
|
|
Example:
|
|
|
|
- `source_ref`: `master`
|
|
- `stable_date`: `2026-03-18`
|
|
- resulting stable version: `2026.318.0`
|
|
|
|
The workflow:
|
|
|
|
- re-verifies the exact source ref
|
|
- computes the next stable patch slot for the chosen UTC date
|
|
- resolves the release notes in preflight: `releases/vYYYY.MDD.P.md` at
|
|
the source commit (the candidate fix path) takes precedence, otherwise
|
|
`releases/beta/v<beta-version>.md` on `master` (a promoted beta). When
|
|
neither exists the run fails before the `npm-stable` approval gate with
|
|
the missing path named
|
|
- publishes `YYYY.MDD.P` under npm dist-tag `latest`
|
|
- creates git tag `vYYYY.MDD.P`
|
|
- dispatches [`docker.yml`](../.github/workflows/docker.yml) at that tag to
|
|
publish `:latest` and the versioned stable images
|
|
- creates or updates the GitHub Release from the resolved notes file
|
|
- for master-side beta notes, pushes a `release-notes/v<version>-canonicalize`
|
|
branch that `git mv`s them to `releases/vYYYY.MDD.P.md` — open and merge
|
|
its PR to restore the canonical layout
|
|
|
|
## Docker Image Tags
|
|
|
|
Master image publication requires both exact-source verification and a successful
|
|
`publish` job from the canonical migrator workflow at the same source SHA and
|
|
latest attempt. A green admission-only run with skipped build and publish jobs
|
|
is pending work. The source observer retries only producers that were strictly
|
|
deferred, continues watching after build admission, and waits for publication.
|
|
The final publication step rechecks both proofs before it creates image tags.
|
|
Failed or cancelled producers require inspection and are not retried automatically.
|
|
|
|
[`docker.yml`](../.github/workflows/docker.yml) publishes the standard image:
|
|
|
|
| Build ref | Tags |
|
|
| --- | --- |
|
|
| `master` push | `:sha-<short>`, `:sha-<full>` |
|
|
| `canary/v*` dispatch | `:sha-<short>`, then `:canary` for the current npm canary |
|
|
| `nightly/v*` tag | `:nightly`, `:sha-<short>` |
|
|
| `beta/v*` tag | `:beta`, `:sha-<short>` |
|
|
| `v*` tag (stable) | `:latest`, `:YYYY.MDD.P`, `:YYYY.MDD`, `:sha-<short>` |
|
|
|
|
Lane tags are pushed by release workflows using `GITHUB_TOKEN`, and GitHub
|
|
suppresses push-triggered workflow runs for those pushes. The release jobs
|
|
therefore dispatch `docker.yml` explicitly at the new tag ref; the tag
|
|
mapping keys off `github.ref` either way.
|
|
|
|
When the merge queue is enabled for `master`,
|
|
[`docker-merge-group.yml`](../.github/workflows/docker-merge-group.yml) builds
|
|
and probes the production image for each merge-group commit. It has no registry
|
|
credentials and uploads an OCI archive with a record of the commit, tree,
|
|
architecture and digest. The `master` push for that same commit then pushes
|
|
those exact bytes by digest instead of rebuilding
|
|
(`scripts/merge-group-image.mjs`). A commit that changes that pipeline (the
|
|
merge-group workflow, `docker.yml`, the promotion script or the PID 1 probe)
|
|
anywhere in the pushed range is always built from source. `docker.yml` runs the
|
|
promotion script as master had it before the push, so promotion trusts only a
|
|
pipeline that master already ran.
|
|
Without a matching successful merge-group
|
|
run, or on any mismatch or error, the push builds from source. The PID 1 probe,
|
|
exact-source gate and signing apply on both paths.
|
|
|
|
Canary dispatches first look up the full-SHA master image. Reuse requires a
|
|
valid multi-platform index with both `linux/amd64` and `linux/arm64`, a matching
|
|
registry digest, and signed provenance from this repository's `docker.yml`
|
|
on `master` at the exact source commit. The image is retagged by immutable
|
|
digest. This preserves the original build inputs and skips both native builds
|
|
and the manifest merge. A missing full-SHA image runs the normal build instead;
|
|
registry errors or invalid provenance fail the run. Rerun after the master
|
|
image's attestation finishes if the image became visible before its proof.
|
|
|
|
Both paths retain the serialized promotion that resolves the current npm
|
|
`canary` dist-tag before moving Docker `:canary`. Nightly, beta, stable, and
|
|
master builds retain their existing build paths. Downstream image builds can
|
|
continue to consume the signed full-SHA master image independently of npm
|
|
canary publication.
|
|
|
|
The Docker build matrix and the release preview image builder configure
|
|
BuildKit to check `mirror.gcr.io` for Docker Hub images first. This covers
|
|
both the Dockerfile frontend and base images. BuildKit keeps Docker Hub as
|
|
the fallback when the cache has no usable copy. The image references,
|
|
build arguments, GHCR publication, and release gates stay the same.
|
|
|
|
This is an availability improvement, not an outage guarantee. Google can
|
|
evict cached images, and mutable tags can lag upstream changes or deletions.
|
|
Matching manifests at one point does not guarantee later tag freshness or
|
|
blob availability. The initial `moby/buildkit` bootstrap still uses the host
|
|
Docker daemon and can fail on Docker Hub before this configuration takes
|
|
effect. See the [BuildKit mirror configuration](https://docs.docker.com/build/ci/github-actions/configure-builder/#registry-mirror)
|
|
and [Google cache limits](https://docs.cloud.google.com/artifact-registry/docs/pull-cached-dockerhub-images).
|
|
|
|
## Local Commands
|
|
|
|
### Preview a canary locally
|
|
|
|
```bash
|
|
./scripts/release.sh canary --dry-run
|
|
```
|
|
|
|
### Preview a nightly locally
|
|
|
|
Requires HEAD to be a commit that already shipped a canary (it must carry a
|
|
`canary/v*` tag):
|
|
|
|
```bash
|
|
./scripts/release.sh nightly --dry-run
|
|
```
|
|
|
|
### Preview a beta locally
|
|
|
|
Requires HEAD to be a commit that already shipped a nightly (it must carry a
|
|
`nightly/v*` tag):
|
|
|
|
```bash
|
|
./scripts/release.sh beta --dry-run
|
|
```
|
|
|
|
### Preview a stable locally
|
|
|
|
```bash
|
|
./scripts/release.sh stable --dry-run
|
|
```
|
|
|
|
### Publish a stable locally
|
|
|
|
This is mainly for emergency/manual use. The normal path is the GitHub workflow.
|
|
|
|
```bash
|
|
./scripts/release.sh stable
|
|
git push public-gh refs/tags/vYYYY.MDD.P
|
|
PUBLISH_REMOTE=public-gh ./scripts/create-github-release.sh YYYY.MDD.P
|
|
```
|
|
|
|
## Stable Changelog Workflow
|
|
|
|
Stable changelog files live at:
|
|
|
|
- `releases/vYYYY.MDD.P.md`
|
|
|
|
Canaries do not get changelog files.
|
|
|
|
The `draft_stable_notes` job seeds a deterministic skeleton (grouped
|
|
commit subjects) on the `release-notes/v<beta-version>` branch at beta
|
|
publish; the flows below turn that skeleton into narrative release notes
|
|
during the soak. Run them against the draft branch's
|
|
`releases/beta/v<beta-version>.md` and push to the notes PR.
|
|
|
|
Recommended local generation flow:
|
|
|
|
```bash
|
|
VERSION="$(./scripts/release.sh stable --date 2026-03-18 --print-version)"
|
|
claude --print --output-format stream-json --verbose --dangerously-skip-permissions --model claude-opus-4-6 "Use the release-changelog skill to draft or update releases/v${VERSION}.md for Paperclip. Read doc/RELEASING.md and .agents/skills/release-changelog/SKILL.md, then generate the stable changelog for v${VERSION} from commits since the last stable tag. Do not create a canary changelog."
|
|
```
|
|
|
|
The repo intentionally does not run this through GitHub Actions because:
|
|
|
|
- canaries are too frequent
|
|
- stable notes are the only public narrative surface that needs LLM help
|
|
- maintainer LLM tokens should not live in Actions
|
|
|
|
## Smoke Testing
|
|
|
|
For a canary:
|
|
|
|
```bash
|
|
PAPERCLIPAI_VERSION=canary ./scripts/docker-onboard-smoke.sh
|
|
```
|
|
|
|
For the current stable:
|
|
|
|
```bash
|
|
PAPERCLIPAI_VERSION=latest ./scripts/docker-onboard-smoke.sh
|
|
```
|
|
|
|
Useful isolated variants:
|
|
|
|
```bash
|
|
HOST_PORT=3232 DATA_DIR=./data/release-smoke-canary PAPERCLIPAI_VERSION=canary ./scripts/docker-onboard-smoke.sh
|
|
HOST_PORT=3233 DATA_DIR=./data/release-smoke-stable PAPERCLIPAI_VERSION=latest ./scripts/docker-onboard-smoke.sh
|
|
```
|
|
|
|
Automated browser smoke is also available:
|
|
|
|
```bash
|
|
gh workflow run release-smoke.yml -f paperclip_version=canary
|
|
gh workflow run release-smoke.yml -f paperclip_version=nightly
|
|
gh workflow run release-smoke.yml -f paperclip_version=beta
|
|
gh workflow run release-smoke.yml -f paperclip_version=latest
|
|
```
|
|
|
|
The nightly lane runs this same suite automatically against its candidate
|
|
before publishing, and the beta lane runs it against the published beta as
|
|
post-publish verification.
|
|
|
|
Minimum checks:
|
|
|
|
- `npx paperclipai@canary onboard` installs
|
|
- onboarding completes without crashes
|
|
- authenticated login works with the smoke credentials
|
|
- the browser lands in onboarding on a fresh instance
|
|
- company creation succeeds
|
|
- the first CEO agent is created
|
|
- the first CEO heartbeat run is triggered
|
|
|
|
## Rollback
|
|
|
|
Rollback does not unpublish versions.
|
|
|
|
It only moves the `latest` dist-tag back to a previous stable:
|
|
|
|
```bash
|
|
./scripts/rollback-latest.sh 2026.318.0 --dry-run
|
|
./scripts/rollback-latest.sh 2026.318.0
|
|
```
|
|
|
|
Then fix forward with a new stable patch slot or release date.
|
|
|
|
## Failure Playbooks
|
|
|
|
### If the canary publishes but smoke testing fails
|
|
|
|
Do not run stable.
|
|
|
|
Instead:
|
|
|
|
1. fix the issue on `master`
|
|
2. merge the fix
|
|
3. wait for the next automatic canary
|
|
4. rerun smoke testing
|
|
|
|
### If the nightly skipped or failed
|
|
|
|
A skipped nightly is working as designed — the job summary names the reason
|
|
(no new green candidate, candidate already shipped, or red smoke). Nothing was
|
|
published, so there is nothing to clean up.
|
|
|
|
To recover after fixing the cause, either wait for the next scheduled run or
|
|
force one: dispatch `release.yml` with `channel: nightly` (optionally pinning
|
|
`source_version` to a specific canary).
|
|
|
|
If the nightly published to npm but the tag push or Docker dispatch failed,
|
|
push the `nightly/v*` tag manually and run `docker.yml` at that tag.
|
|
|
|
### If a tag push is rejected with a workflows-permission error
|
|
|
|
GITHUB_TOKEN may not create refs that point at commits which modify workflow
|
|
files when the run was started by dispatch or schedule (push-triggered runs
|
|
are exempt, which is why canary tags on the same commit succeed). The npm
|
|
publish is already complete and correct when this happens. The failed job's
|
|
summary contains the exact recovery commands: create and push the tag with
|
|
maintainer credentials, then dispatch `docker.yml` at the tag (and for
|
|
stable, run `create-github-release.sh`). This only occurs when a
|
|
release-infrastructure commit itself becomes a promotion source.
|
|
|
|
### If a beta looks bad during soak
|
|
|
|
Do not promote it to stable. Fix forward: land the fix on `master`, let it
|
|
ship through canary and nightly, and promote a new beta. The soak clock
|
|
starts over for the new beta.
|
|
|
|
If the published beta is actively harmful to beta users, move the `beta`
|
|
dist-tag back to the previous beta version with `npm dist-tag add` per
|
|
package, and re-point the `:beta` Docker tags at the previous beta's images.
|
|
|
|
### If stable npm publish succeeds but tag push or GitHub release creation fails
|
|
|
|
This is a partial release. npm is already live.
|
|
|
|
Do this immediately:
|
|
|
|
1. push the missing tag
|
|
2. rerun `PUBLISH_REMOTE=public-gh ./scripts/create-github-release.sh YYYY.MDD.P`
|
|
3. verify the GitHub Release notes point at `releases/vYYYY.MDD.P.md`
|
|
|
|
Do not republish the same version.
|
|
|
|
### If `latest` is broken after stable publish
|
|
|
|
Roll back the dist-tag:
|
|
|
|
```bash
|
|
./scripts/rollback-latest.sh YYYY.MDD.P
|
|
```
|
|
|
|
Then fix forward with a new stable release.
|
|
|
|
## Related Files
|
|
|
|
- [`scripts/release.sh`](../scripts/release.sh)
|
|
- [`scripts/release-package-map.mjs`](../scripts/release-package-map.mjs)
|
|
- [`scripts/create-github-release.sh`](../scripts/create-github-release.sh)
|
|
- [`scripts/rollback-latest.sh`](../scripts/rollback-latest.sh)
|
|
- [`doc/RELEASE-CHECKLIST.md`](RELEASE-CHECKLIST.md)
|
|
- [`doc/PUBLISHING.md`](PUBLISHING.md)
|
|
- [`doc/RELEASE-AUTOMATION-SETUP.md`](RELEASE-AUTOMATION-SETUP.md)
|