Files
PaperClipAI/doc/RELEASING.md
T
DottaandClaude Opus 5.5 2659ff1d22 fix(release): retry Cloud readiness jobs that lost their Spot runner (#21399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every push to `master` publishes a canary, and the canary's image
reaches staging
> - Each canary waits for `Cloud source verified v1` from the Cloud
readiness run of its exact SHA
> - Cloud readiness runs on AWS Spot workers. When a worker is
reclaimed, the gate failed and the image watcher cancelled the readiness
run
> - Nothing retried that attempt. On a busy master the newest commit's
release failed, and merged work did not reach staging until a person
re-ran the run
> - This pull request re-runs the failed jobs of a readiness attempt
whose every failure is a lost runner. A real check failure still fails
closed
> - The benefit is that a Spot reclaim no longer holds a release

## Linked Issues or Issue Description

**What happened?**
On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained
it published for more than an hour. The release workflow keeps one
running and one pending canary. A newer push replaces the pending one,
so the newest commit always runs next. That part works as designed. The
failures came from the readiness gate:

| Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause
|
| --- | --- | --- | --- |
| `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` |
Spot workers reclaimed at 13:49:56Z. The run was then cancelled by
`github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by
hand |
| `8af889f612` | 43434587048 / 2 | Release superseded while pending;
readiness failed in the same reclaim | Spot reclaim |
| `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` |
Spot reclaim |
| `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` |
Flaky test `decision-models.test.ts` (concurrent budget admission) |
| `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` |
Flaky test `fresh-continuation-fallback.test.ts` (truncate) |
| `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` |
Both flaky tests |

Over the last 200 master pushes, release runs ended 14 success, 33
failure, and 150 cancelled (superseded while pending, which is
expected).

**Expected behavior**
A lost runner is not a source failure. The gate retries the lost jobs
and publishes the newest commit that passes.

**Steps to reproduce**
See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master
simulation". It replays the 13:49:56Z reclaim through both waiters.

## What Changed

- `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed
readiness attempt with the existing PR classifier
(`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when
every causing job (`failure`, `timed_out`, `startup_failure`) lost its
runner. Cancelled siblings are fallout. A run that is still going is
cancelled first, then `rerun-failed-jobs` runs on the next poll. The
ceiling is readiness attempt 4. Writes go only to `cancel` or
`rerun-failed-jobs` of a canonical run, after an identity check.
- `scripts/cloud-source-verification.mjs` (canary gate in
`release.yml`): on `SourceVerificationFailed`, ask the retry first. The
source wait goes from 60 to 90 minutes.
- `scripts/watch-release-source.mjs` (master image watcher in
`docker.yml`): on a runner loss, keep the image attempt alive and wait
for the retried proof. Do not cancel it.
- `.github/workflows/release.yml`: `verify_canary` timeout goes from 70
to 100 minutes.
- `.github/workflows/pr-trusted.yml`: run the new test file.
- `doc/RELEASING.md`: new "Busy master and lost runners" section.

## Verification

- `node --test scripts/cloud-readiness-runner-loss.test.mjs
scripts/cloud-source-verification.test.mjs
.github/scripts/tests/*.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: all pass.
`release-verify-workflow.test.mjs` passes 51 of 51 but does not exit
locally. This also happens on `origin/master` without this change.
- Red/green: with the two waiter files reverted, 3 of the 6 simulation
tests fail (the canary gate, two reclaims in a row, and the image
watcher). With the change, all 20 pass.
- Dry run against real runs (read-only, writes captured): 43435593630/2,
43434587048/2, and 43423674511/2 give `rerun-failed-jobs`.
43439942063/1, 43421480302/1, and 43422042261/2 (real test failures)
give `failed` with no write.
- `pnpm verify` exits 2 because a workflow file changed. Required CI
supplies broad coverage.

## Risks

- A real failure classified as a runner loss would be retried. The
classifier needs the runner's own shutdown or lost-communication
`##[error]` line in the job log. It is the same classifier PR CI uses,
and unreadable evidence fails closed.
- On a reclaim, the waiter cancels the readiness run to retry sooner.
Jobs that were still healthy run again. This costs runner time, not
correctness.
- Flaky tests still fail a canary. This change does not retry them. They
need their own fixes.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:20:30 -06:00

607 lines
26 KiB
Markdown

# Releasing Paperclip
Maintainer runbook for shipping Paperclip across npm, GitHub, and the website-facing changelog surface.
The release model is now commit-driven:
1. Every push to `master` publishes a canary automatically.
2. Once a night, the newest master commit with a green canary publish is
smoke-tested and republished as the nightly.
3. Betas are manual, human-approved promotions of a chosen nightly.
4. Stable releases promote a beta that has soaked for at least 3 days
(bypass requires a written justification).
5. Stable release notes live in `releases/vYYYY.MDD.P.md`. They are
drafted automatically when a beta is published (as
`releases/beta/v<beta-version>.md` on `master`), edited during the
soak, and moved to the versioned name after the stable ships.
6. Only stable releases get GitHub Releases.
The user-facing guide to the channels is [`CHANNELS.md`](CHANNELS.md).
## Versioning Model
Paperclip uses calendar versions that still fit semver syntax:
- stable: `YYYY.MDD.P`
- canary: `YYYY.MDD.P-canary.N`
- nightly: `YYYY.MDD.P-nightly.N`
- beta: `YYYY.MDD.P-beta.N`
Examples:
- first stable on March 18, 2026: `2026.318.0`
- second stable on March 18, 2026: `2026.318.1`
- fourth canary for the `2026.318.1` line: `2026.318.1-canary.3`
- first nightly cut on March 18, 2026: `2026.318.1-nightly.0`
- first beta promoted on March 18, 2026: `2026.318.1-beta.0`
A promotion republishes the exact source commit of the previous lane's build
(canary → nightly → beta); the version dates the promotion, not the source
build.
Important constraints:
- the middle numeric slot is `MDD`, where `M` is the UTC month and `DD` is the zero-padded UTC day
- use `2026.303.0` for March 3, not `2026.33.0`
- do not use leading zeroes such as `2026.0318.0`
- do not use four numeric segments such as `2026.3.18.1`
- the semver-safe canary form is `2026.318.0-canary.1`
## Release Surfaces
Every stable release has four separate surfaces:
1. **Verification** — the exact git SHA passes typecheck, tests, and build
2. **npm** — `paperclipai` and public workspace packages are published
3. **GitHub** — the stable release gets a git tag and GitHub Release
4. **Website / announcements** — the stable changelog is published externally and announced
A stable release is done only when all four surfaces are handled.
Canaries, nightlies, and betas only cover the first two surfaces plus an
internal traceability tag.
## Core Invariants
- canaries publish from `master`
- the canary publisher checks out the exact master push SHA. It verifies the
repository, event, branch, and checked-out commit before attaching local
`master` at that same SHA. It never switches to a newer branch tip. The local
release script still rejects detached or non-master checkouts.
- nightlies republish a commit that already shipped a canary (the commit must
carry a `canary/v*` tag), and only after the release smoke suite passes
against that exact published canary
- betas republish a commit that already shipped a nightly (the commit must
carry a `nightly/v*` tag), behind the `npm-beta` approval gate, and the
published beta is re-smoked
- stables publish from an explicitly chosen source ref, which must have
shipped as a beta at least 3 days earlier unless a written justification
is provided
- tags point at the original source commit, not a generated release commit
- stable notes are always `releases/vYYYY.MDD.P.md` in the end state: a
promoted beta's notes are drafted and edited at
`releases/beta/v<beta-version>.md` on `master` during the soak (the
promoted commit cannot carry a file named for a promotion date that was
unknown when it was created), and a post-stable canonicalization PR
moves them to the versioned name
- canaries, nightlies, and betas never create GitHub Releases
- canaries, nightlies, and betas never require changelog generation
- Docker `:latest` moves only on stable releases; canary dispatches promote
`:canary`, nightly builds `:nightly`, and beta builds `:beta`
## TL;DR
### Canary
Every push to `master` runs the canary path inside [`.github/workflows/release.yml`](../.github/workflows/release.yml).
It:
- verifies the pushed commit
- computes the canary version for the current UTC date
- publishes workspace packages dependency-first under npm dist-tag `canary`
- waits for each package version to become registry-visible before continuing
- publishes the user-facing `paperclipai` package last, so `paperclipai@canary` does not advance before the full package set exists
- verifies that `canary` resolves to the just-published version and that published internal dependencies exist on npm
- installs `paperclipai@canary` into a clean temporary prefix as the final npm gate
- starts the exact published canary through a separate fresh npm install for the onboarding smoke; npm `ETARGET` and verified canary tarball availability failures retry, up to three attempts within two minutes, before onboarding runs once under the existing five-minute startup deadline
- fails by default if npm leaves `latest` pointing at a canary; use `--allow-canary-latest` only when that state is intentional
- creates a git tag `canary/vYYYY.MDD.P-canary.N`
#### Busy master and lost runners
Master often receives a commit every few minutes, faster than one canary
publishes (about 60-90 minutes). The release workflow keeps one running canary
and one pending canary. A newer push replaces the pending one, so the canary
that runs next is always the newest master commit. Commits in between are not
published on their own; they ship inside the next canary.
Each canary waits for `Cloud source verified v1` from the Cloud readiness run
of its exact SHA (up to 90 minutes). Cloud readiness runs on AWS Spot workers.
When an attempt fails only because its workers were reclaimed, the canary gate
and the master image watcher re-run that attempt's failed jobs instead of
failing the release
([`scripts/cloud-readiness-runner-loss.mjs`](../scripts/cloud-readiness-runner-loss.mjs)).
A job counts as lost when its log shows the runner's shutdown or lost
communication error. Retries continue up to attempt 4 of the readiness run. A
real check failure, including a flaky test, is never retried: that canary fails
and the next push publishes. If a canary is missing after a quiet period, look
at the failed `Reuse exact-source verification` job. It names the readiness run
and lists each failed job as a lost runner or a failure.
After publication, the onboarding smoke checks out the same source SHA and
installs its test dependencies from the publisher's resolved lockfile with
`--frozen-lockfile`. The same-run artifact is captured before tracked files are
restored or release versions are rewritten. This lets a source commit use its
current patches and manifests while the separate lockfile-refresh PR is still
pending, without resolving new dependencies in the smoke job. The smoke still installs and tests
the exact published canary version, not a workspace build.
The tarball retry applies only to an unambiguous npm `E404` for a canonical
public-registry tarball of a CI-published Paperclip package at that exact canary
version. The helper queries npm's effective global and applicable scoped registry
in the failed install's prefix, working directory, and environment. A custom or
unknown effective registry fails. Configuration output stays private in memory.
Fresh primary-registry metadata must match the package, version, and tarball URL
and announce a valid SHA512 integrity value. This establishes the announced
artifact identity; it does not download the tarball or independently verify its
contents. The actual npm install still checks tarball integrity.
Configuration and metadata checks share a five-second deadline, including the
response body, within the same two-minute acquisition budget. Metadata is limited
to 64 KiB. Redirects, authentication errors, missing metadata, conflicting npm
errors, and other artifacts fail. A persistent promised-tarball failure also
fails after three installs. Onboarding errors never restart acquisition or
repeat onboarding.
The lockfile artifact is retained for 14 days. A publisher-job rerun replaces its
source-named artifact; a smoke-only rerun uses the existing artifact. A missing
artifact fails the job rather than falling back to a different dependency set.
Users install canaries with:
```bash
npx paperclipai@canary onboard
# or
npx paperclipai@canary onboard --data-dir "$(mktemp -d /tmp/paperclip-canary.XXXXXX)"
```
### Nightly
A scheduled job in [`.github/workflows/release.yml`](../.github/workflows/release.yml)
runs once a night at 09:00 UTC.
It:
- selects the newest commit on `master` that carries a `canary/v*` tag (the
tag is pushed only after a successful canary publish, so it is the
green-publish signal)
- skips with a job-summary reason when there is no new candidate or the
candidate already shipped as a nightly
- runs the release smoke suite ([`release-smoke.yml`](../.github/workflows/release-smoke.yml))
against that exact published canary version — red smoke means no nightly
tonight
- republishes the same source commit as `YYYY.MDD.P-nightly.N` under the npm
dist-tag `nightly` (the commit was already verified by its canary run, so
verification is not repeated)
- creates and pushes the git tag `nightly/vYYYY.MDD.P-nightly.N`
- dispatches [`docker.yml`](../.github/workflows/docker.yml) at that tag to
publish the `:nightly` images
To force a nightly outside the schedule (recovery, or promoting a specific
canary), dispatch `release.yml` with `channel: nightly`. Leave
`source_version` empty for automatic selection, or set it to an exact
canary version. `dry_run: true` previews the publish and skips smoke, the tag
push, and the Docker dispatch.
Users install nightlies with:
```bash
npx paperclipai@nightly onboard
```
### Beta
Betas are manual promotions. Dispatch
[`release.yml`](../.github/workflows/release.yml) with `channel: beta`.
- leave `source_version` empty to promote the newest nightly, or set it to an
exact nightly version such as `2026.807.0-nightly.0`
- the selection job resolves the nightly's source commit and fails loudly if
it does not exist or already shipped as a beta
- the publish waits for approval in the **`npm-beta` environment** — its
required reviewers are the promotion gate
- promotions run the release tooling of the source commit, so the source
nightly must postdate the beta channel's introduction; the selection job
rejects older sources with a clear error (in practice every nightly cut
after the beta tooling merged qualifies)
- the same commit is republished as `YYYY.MDD.P-beta.N` under the npm
dist-tag `beta`, tagged `beta/vYYYY.MDD.P-beta.N`, and `docker.yml` is
dispatched at that tag to publish the `:beta` images
- after publishing, the release smoke suite runs against the exact published
beta version as verification
- a `draft_stable_notes` job also generates the eventual stable's notes
skeleton — `releases/beta/v<beta-version>.md`, grouped from
`git log <last-stable-tag>..<source-commit>` — and force-pushes it to the
machine-owned `release-notes/v<beta-version>` branch. Open the PR from
the job-summary link (a human opens it so CI runs) and edit the notes
during the soak; the stable promotion reads the merged file from
`master`
- `dry_run: true` previews the publish and skips the tag push, Docker
dispatch, and post-publish smoke
Users install betas with:
```bash
npx paperclipai@beta onboard
```
#### Beta fix path: candidate branches
When one or two targeted fixes are needed before beta and waiting for the
next nightly (or absorbing a whole day of `master`) is wrong, build the beta
from a short-lived candidate branch:
1. cut `candidate/beta-<target>` from the chosen nightly's source commit
(for example `candidate/beta-2026.811.0`)
2. cherry-pick only the required fix commits onto it and push the branch
3. dispatch `release.yml` with `channel: beta` and `candidate_branch:
candidate/beta-<target>`
4. selection validates the branch name, rejects heads that already shipped
as a beta, records the cherry-picked commits in the job summary, and the
head runs **full verification** before publishing (it never went through
a canary or nightly)
5. after the beta ships, land the fixes on `master` normally and delete the
candidate branch
Use this sparingly: the happy path is promoting a nightly. A candidate build
has its own `-beta.N` identity and is never pretended to be the nightly it
was cut from.
### Stable
Use [`.github/workflows/release.yml`](../.github/workflows/release.yml) from the Actions tab with the manual `workflow_dispatch` inputs.
[Run the action here](https://github.com/paperclipai/paperclip/actions/workflows/release.yml)
Inputs:
- `channel`
- `stable` (the default) for a stable release; `beta` and `nightly` run
those lanes instead (see above)
- `source_ref`
- commit SHA, branch, or tag
- `stable_date`
- optional UTC date override in `YYYY-MM-DD`
- enter a date like `2026-03-18`, not a version like `2026.318.0`
- `skip_soak_justification`
- written reason for releasing a stable whose source has not soaked as a
beta for 3 days; leave empty for normal releases
- `dry_run`
- preview only when true
The stable preflight enforces the beta soak: the source commit must carry a
`beta/v*` tag whose npm publish time is at least 3 days old. If it is not,
the run fails unless `skip_soak_justification` is provided; the justification
is echoed into the job summary. Dry runs report soak state without blocking.
For a cherry-picked stable (the release fix path), cut
`candidate/release-<target>` from the chosen beta's source commit,
cherry-pick the required fixes, push the branch, and use it as
`source_ref`. The candidate head carries no `beta/v*` tag, so the soak gate
requires `skip_soak_justification` — that is deliberate: the exact bits were
not soaked, and the justification is the recorded trade-off. Reconcile the
fixes back to `master` and delete the branch after shipping.
Before running stable:
1. pick the beta you are promoting (its source commit is the `source_ref`)
2. confirm the beta has soaked for 3 days with no open blockers
3. resolve the target stable version with `./scripts/release.sh stable --date "$(date +%F)" --print-version`
4. make sure the notes PR from the beta's draft branch
(`release-notes/v<beta-version>`, adding
`releases/beta/v<beta-version>.md`) is merged on `master` — or, for
candidate builds, that the candidate branch itself carries
`releases/vYYYY.MDD.P.md`
5. run the stable workflow from that source ref
Example:
- `source_ref`: `master`
- `stable_date`: `2026-03-18`
- resulting stable version: `2026.318.0`
The workflow:
- re-verifies the exact source ref
- computes the next stable patch slot for the chosen UTC date
- resolves the release notes in preflight: `releases/vYYYY.MDD.P.md` at
the source commit (the candidate fix path) takes precedence, otherwise
`releases/beta/v<beta-version>.md` on `master` (a promoted beta). When
neither exists the run fails before the `npm-stable` approval gate with
the missing path named
- publishes `YYYY.MDD.P` under npm dist-tag `latest`
- creates git tag `vYYYY.MDD.P`
- dispatches [`docker.yml`](../.github/workflows/docker.yml) at that tag to
publish `:latest` and the versioned stable images
- creates or updates the GitHub Release from the resolved notes file
- for master-side beta notes, pushes a `release-notes/v<version>-canonicalize`
branch that `git mv`s them to `releases/vYYYY.MDD.P.md` — open and merge
its PR to restore the canonical layout
## Docker Image Tags
Master image publication requires both exact-source verification and a successful
`publish` job from the canonical migrator workflow at the same source SHA and
latest attempt. A green admission-only run with skipped build and publish jobs
is pending work. The source observer retries only producers that were strictly
deferred, continues watching after build admission, and waits for publication.
The final publication step rechecks both proofs before it creates image tags.
Failed or cancelled producers require inspection and are not retried automatically,
with one exception: a Cloud readiness attempt that failed only because Spot
runners were reclaimed is retried (see "Busy master and lost runners" below).
[`docker.yml`](../.github/workflows/docker.yml) publishes the standard image:
| Build ref | Tags |
| --- | --- |
| `master` push | `:sha-<short>`, `:sha-<full>` |
| `canary/v*` dispatch | `:sha-<short>`, then `:canary` for the current npm canary |
| `nightly/v*` tag | `:nightly`, `:sha-<short>` |
| `beta/v*` tag | `:beta`, `:sha-<short>` |
| `v*` tag (stable) | `:latest`, `:YYYY.MDD.P`, `:YYYY.MDD`, `:sha-<short>` |
Lane tags are pushed by release workflows using `GITHUB_TOKEN`, and GitHub
suppresses push-triggered workflow runs for those pushes. The release jobs
therefore dispatch `docker.yml` explicitly at the new tag ref; the tag
mapping keys off `github.ref` either way.
When the merge queue is enabled for `master`,
[`docker-merge-group.yml`](../.github/workflows/docker-merge-group.yml) builds
and probes the production image for each merge-group commit. It has no registry
credentials and uploads an OCI archive with a record of the commit, tree,
architecture and digest. The `master` push for that same commit then pushes
those exact bytes by digest instead of rebuilding
(`scripts/merge-group-image.mjs`). A commit that changes that pipeline (the
merge-group workflow, `docker.yml`, the promotion script or the PID 1 probe)
anywhere in the pushed range is always built from source. `docker.yml` runs the
promotion script as master had it before the push, so promotion trusts only a
pipeline that master already ran.
Without a matching successful merge-group
run, or on any mismatch or error, the push builds from source. The PID 1 probe,
exact-source gate and signing apply on both paths.
Canary dispatches first look up the full-SHA master image. Reuse requires a
valid multi-platform index with both `linux/amd64` and `linux/arm64`, a matching
registry digest, and signed provenance from this repository's `docker.yml`
on `master` at the exact source commit. The image is retagged by immutable
digest. This preserves the original build inputs and skips both native builds
and the manifest merge. A missing full-SHA image runs the normal build instead;
registry errors or invalid provenance fail the run. Rerun after the master
image's attestation finishes if the image became visible before its proof.
Both paths retain the serialized promotion that resolves the current npm
`canary` dist-tag before moving Docker `:canary`. Nightly, beta, stable, and
master builds retain their existing build paths. Downstream image builds can
continue to consume the signed full-SHA master image independently of npm
canary publication.
The Docker build matrix and the release preview image builder configure
BuildKit to check `mirror.gcr.io` for Docker Hub images first. This covers
both the Dockerfile frontend and base images. BuildKit keeps Docker Hub as
the fallback when the cache has no usable copy. The image references,
build arguments, GHCR publication, and release gates stay the same.
This is an availability improvement, not an outage guarantee. Google can
evict cached images, and mutable tags can lag upstream changes or deletions.
Matching manifests at one point does not guarantee later tag freshness or
blob availability. The initial `moby/buildkit` bootstrap still uses the host
Docker daemon and can fail on Docker Hub before this configuration takes
effect. See the [BuildKit mirror configuration](https://docs.docker.com/build/ci/github-actions/configure-builder/#registry-mirror)
and [Google cache limits](https://docs.cloud.google.com/artifact-registry/docs/pull-cached-dockerhub-images).
## Local Commands
### Preview a canary locally
```bash
./scripts/release.sh canary --dry-run
```
### Preview a nightly locally
Requires HEAD to be a commit that already shipped a canary (it must carry a
`canary/v*` tag):
```bash
./scripts/release.sh nightly --dry-run
```
### Preview a beta locally
Requires HEAD to be a commit that already shipped a nightly (it must carry a
`nightly/v*` tag):
```bash
./scripts/release.sh beta --dry-run
```
### Preview a stable locally
```bash
./scripts/release.sh stable --dry-run
```
### Publish a stable locally
This is mainly for emergency/manual use. The normal path is the GitHub workflow.
```bash
./scripts/release.sh stable
git push public-gh refs/tags/vYYYY.MDD.P
PUBLISH_REMOTE=public-gh ./scripts/create-github-release.sh YYYY.MDD.P
```
## Stable Changelog Workflow
Stable changelog files live at:
- `releases/vYYYY.MDD.P.md`
Canaries do not get changelog files.
The `draft_stable_notes` job seeds a deterministic skeleton (grouped
commit subjects) on the `release-notes/v<beta-version>` branch at beta
publish; the flows below turn that skeleton into narrative release notes
during the soak. Run them against the draft branch's
`releases/beta/v<beta-version>.md` and push to the notes PR.
Recommended local generation flow:
```bash
VERSION="$(./scripts/release.sh stable --date 2026-03-18 --print-version)"
claude --print --output-format stream-json --verbose --dangerously-skip-permissions --model claude-opus-4-6 "Use the release-changelog skill to draft or update releases/v${VERSION}.md for Paperclip. Read doc/RELEASING.md and .agents/skills/release-changelog/SKILL.md, then generate the stable changelog for v${VERSION} from commits since the last stable tag. Do not create a canary changelog."
```
The repo intentionally does not run this through GitHub Actions because:
- canaries are too frequent
- stable notes are the only public narrative surface that needs LLM help
- maintainer LLM tokens should not live in Actions
## Smoke Testing
For a canary:
```bash
PAPERCLIPAI_VERSION=canary ./scripts/docker-onboard-smoke.sh
```
For the current stable:
```bash
PAPERCLIPAI_VERSION=latest ./scripts/docker-onboard-smoke.sh
```
Useful isolated variants:
```bash
HOST_PORT=3232 DATA_DIR=./data/release-smoke-canary PAPERCLIPAI_VERSION=canary ./scripts/docker-onboard-smoke.sh
HOST_PORT=3233 DATA_DIR=./data/release-smoke-stable PAPERCLIPAI_VERSION=latest ./scripts/docker-onboard-smoke.sh
```
Automated browser smoke is also available:
```bash
gh workflow run release-smoke.yml -f paperclip_version=canary
gh workflow run release-smoke.yml -f paperclip_version=nightly
gh workflow run release-smoke.yml -f paperclip_version=beta
gh workflow run release-smoke.yml -f paperclip_version=latest
```
The nightly lane runs this same suite automatically against its candidate
before publishing, and the beta lane runs it against the published beta as
post-publish verification.
Minimum checks:
- `npx paperclipai@canary onboard` installs
- onboarding completes without crashes
- authenticated login works with the smoke credentials
- the browser lands in onboarding on a fresh instance
- company creation succeeds
- the first CEO agent is created
- the first CEO heartbeat run is triggered
## Rollback
Rollback does not unpublish versions.
It only moves the `latest` dist-tag back to a previous stable:
```bash
./scripts/rollback-latest.sh 2026.318.0 --dry-run
./scripts/rollback-latest.sh 2026.318.0
```
Then fix forward with a new stable patch slot or release date.
## Failure Playbooks
### If the canary publishes but smoke testing fails
Do not run stable.
Instead:
1. fix the issue on `master`
2. merge the fix
3. wait for the next automatic canary
4. rerun smoke testing
### If the nightly skipped or failed
A skipped nightly is working as designed — the job summary names the reason
(no new green candidate, candidate already shipped, or red smoke). Nothing was
published, so there is nothing to clean up.
To recover after fixing the cause, either wait for the next scheduled run or
force one: dispatch `release.yml` with `channel: nightly` (optionally pinning
`source_version` to a specific canary).
If the nightly published to npm but the tag push or Docker dispatch failed,
push the `nightly/v*` tag manually and run `docker.yml` at that tag.
### If a tag push is rejected with a workflows-permission error
GITHUB_TOKEN may not create refs that point at commits which modify workflow
files when the run was started by dispatch or schedule (push-triggered runs
are exempt, which is why canary tags on the same commit succeed). The npm
publish is already complete and correct when this happens. The failed job's
summary contains the exact recovery commands: create and push the tag with
maintainer credentials, then dispatch `docker.yml` at the tag (and for
stable, run `create-github-release.sh`). This only occurs when a
release-infrastructure commit itself becomes a promotion source.
### If a beta looks bad during soak
Do not promote it to stable. Fix forward: land the fix on `master`, let it
ship through canary and nightly, and promote a new beta. The soak clock
starts over for the new beta.
If the published beta is actively harmful to beta users, move the `beta`
dist-tag back to the previous beta version with `npm dist-tag add` per
package, and re-point the `:beta` Docker tags at the previous beta's images.
### If stable npm publish succeeds but tag push or GitHub release creation fails
This is a partial release. npm is already live.
Do this immediately:
1. push the missing tag
2. rerun `PUBLISH_REMOTE=public-gh ./scripts/create-github-release.sh YYYY.MDD.P`
3. verify the GitHub Release notes point at `releases/vYYYY.MDD.P.md`
Do not republish the same version.
### If `latest` is broken after stable publish
Roll back the dist-tag:
```bash
./scripts/rollback-latest.sh YYYY.MDD.P
```
Then fix forward with a new stable release.
## Related Files
- [`scripts/release.sh`](../scripts/release.sh)
- [`scripts/release-package-map.mjs`](../scripts/release-package-map.mjs)
- [`scripts/create-github-release.sh`](../scripts/create-github-release.sh)
- [`scripts/rollback-latest.sh`](../scripts/rollback-latest.sh)
- [`doc/RELEASE-CHECKLIST.md`](RELEASE-CHECKLIST.md)
- [`doc/PUBLISHING.md`](PUBLISHING.md)
- [`doc/RELEASE-AUTOMATION-SETUP.md`](RELEASE-AUTOMATION-SETUP.md)