PaperClipAI/doc/RELEASE-AUTOMATION-SETUP.md

# Release Automation Setup

This document covers the GitHub and npm setup required for the current Paperclip release model:

- automatic canaries from `master`
- manual stable promotion from a chosen source ref
- npm trusted publishing via GitHub OIDC
- protected release infrastructure in a public repository

Repo-side files that depend on this setup:

- `.github/workflows/release.yml`
- `.github/CODEOWNERS`

Note:

- the release workflows intentionally use `pnpm install --no-frozen-lockfile`
- this matches the repo's current policy where `pnpm-lock.yaml` is refreshed by GitHub automation after manifest changes land on `master`
- the publish jobs then restore `pnpm-lock.yaml` before running `scripts/release.sh`, so the release script still sees a clean worktree

## 1. Merge the Repo Changes First

Before touching GitHub or npm settings, merge the release automation code so the referenced workflow filenames already exist on the default branch.

Required files:

- `.github/workflows/release.yml`
- `.github/CODEOWNERS`

## 2. Configure npm Trusted Publishing

Do this for every public package that Paperclip publishes.

At minimum that includes:

- `paperclipai`
- `@paperclipai/server`
- `@paperclipai/ui`
- public packages under `packages/`

### 2.1. In npm, open each package settings page

For each package:

1. open npm as an owner of the package
2. go to the package settings / publishing access area
3. add a trusted publisher for the GitHub repository `paperclipai/paperclip`

### 2.2. Add one trusted publisher entry per package

npm currently allows one trusted publisher configuration per package.

Configure:

- workflow: `.github/workflows/release.yml`

Repository:

- `paperclipai/paperclip`

Environment name:

- leave the npm trusted-publisher environment field blank

Why:

- the single `release.yml` workflow handles both canary and stable publishing
- GitHub environments `npm-canary` and `npm-stable` still enforce different approval rules on the GitHub side

### 2.2.1. Newly added public packages need a bootstrap phase

Trusted publishing is configured on the npm package itself, not at the repo scope.
That means a brand-new public package must not be auto-enrolled into CI publishing until its npm package exists and its trusted publisher has been configured.

Repo policy:

1. add every non-private package to [`scripts/release-package-manifest.json`](../scripts/release-package-manifest.json)
2. set `"publishFromCi": true` only when CI is expected to publish that package
3. if the package is not ready for CI publishing yet, keep `"publishFromCi": false`
4. complete the package bootstrap before merging any PR that changes a release-enabled new package

Bootstrap sequence for a new package:

1. publish the package once from a trusted maintainer machine using normal npm auth
2. open that package on npm and add the `paperclipai/paperclip` trusted publisher for `.github/workflows/release.yml`
3. rerun or dry-run the release flow as needed to confirm CI publishing now works
4. only then enable `"publishFromCi": true`

PR CI enforces this by checking changed release-enabled package manifests against npm. That keeps `master` canary publishing healthy while preserving the no-long-lived-token model for normal CI releases.

### 2.3. Verify trusted publishing before removing old auth

After the workflows are live:

1. run a canary publish
2. confirm npm publish succeeds without any `NPM_TOKEN`
3. run a stable dry-run
4. run one real stable publish

Only after that should you remove old token-based access.

## 3. Remove Legacy npm Tokens

After trusted publishing works:

1. revoke any repository or organization `NPM_TOKEN` secrets used for publish
2. revoke any personal automation token that used to publish Paperclip
3. if npm offers a package-level setting to restrict publishing to trusted publishers, enable it

Goal:

- no long-lived npm publishing token should remain in GitHub Actions

## 4. Create GitHub Environments

Create three environments in the GitHub repository:

- `npm-canary`
- `npm-beta`
- `npm-stable`

Path:

1. GitHub repository
2. `Settings`
3. `Environments`
4. `New environment`

## 5. Configure `npm-canary`

Recommended settings for `npm-canary`:

- environment name: `npm-canary`
- required reviewers: none
- wait timer: none
- deployment branches and tags:
  - selected branches only
  - allow `master`

Reasoning:

- every push to `master` should be able to publish a canary automatically
- no human approval should be required for canaries

The scheduled nightly lane also publishes under `npm-canary`: it is the same
trust level (fully automated, no human gate), its runs execute on `master` so
the branch rule is satisfied, and reusing the environment means the nightly
lane required no new environments and no npm trusted-publisher changes
(publishing still happens from `release.yml`, see section 2.2).

## 5.1. Configure `npm-beta`

Recommended settings for `npm-beta`:

- environment name: `npm-beta`
- required reviewers: at least one maintainer
- prevent self-review: enabled when your team size allows it
- wait timer: none
- deployment branches and tags:
  - selected branches only
  - allow `master`

Reasoning:

- beta promotions are deliberate human decisions; the required reviewer on
  this environment is the promotion gate
- create this environment before the first `channel: beta` dispatch. If the
  workflow runs first, GitHub auto-creates the environment with no
  protection rules, and that first beta would publish without approval

Like nightly, beta publishing lives in `release.yml`, so no npm
trusted-publisher changes are needed (see section 2.2).

## 6. Configure `npm-stable`

Recommended settings for `npm-stable`:

- environment name: `npm-stable`
- required reviewers: at least one maintainer other than the person triggering the workflow when possible
- prevent self-review: enabled
- admin bypass: disabled if your team can tolerate it
- wait timer: optional
- deployment branches and tags:
  - selected branches only
  - allow `master`

Reasoning:

- stable publishes should require an explicit human approval gate
- the workflow is manual, but the environment should still be the real control point

## 7. Protect `master`

Open the branch protection settings for `master`.

Recommended rules:

1. require pull requests before merging
2. require status checks to pass before merging
3. require review from code owners
4. dismiss stale approvals when new commits are pushed
5. restrict who can push directly to `master`

At minimum, make sure workflow and release script changes cannot land without review.

## 8. Enforce CODEOWNERS Review

This repo now includes `.github/CODEOWNERS`, but GitHub only enforces it if branch protection requires code owner reviews.

In branch protection for `master`, enable:

- `Require review from Code Owners`

Then verify the owner entries are correct for your actual maintainer set.

Current file:

- `.github/CODEOWNERS`

If `@cryppadotta` is not the right reviewer identity in the public repo, change it before enabling enforcement.

## 8.1. Let the Lockfile Bot Merge Lockfile Refreshes

`.github/workflows/refresh-lockfile.yml` regenerates `pnpm-lock.yaml` from the manifests on `master` and merges the result without CI or review. Contributors never commit the lockfile, so the bot is the only path for lockfile changes.

Resolution keeps locked versions, but it picks the newest matching version for an added or widened dependency. A release-age gate (`.github/scripts/check-lockfile-release-age.mjs`) therefore checks every package version that the refresh adds. The bot merges only when each one has been on npm for at least 3 days. If any version is newer, or the gate cannot verify it, the bot leaves a comment on the PR and does not merge. A maintainer can review and merge it, or the daily scheduled refresh merges it after the versions are old enough. A held run saves its lockfile, and the hash of the `master` lockfile it started from, as an artifact. The next run resolves from that lockfile only while `master` still has the same lockfile. Held versions therefore stay put while their manifests are unchanged, even when newer releases appear, and a lockfile change on `master` replaces them. The gate also holds any added entry whose resolution is not a plain npm registry tarball. Change the age with `--min-age-days` in the workflow.

The merge uses a dedicated GitHub App. Do not reuse `commitperclip` or bypass the built-in `github-actions` app. Those identities are available to other workflows, including release jobs that run dependency install scripts, so a bypass for them would let any of those workflows land code on `master`.

1. Create an organization-owned GitHub App, for example `paperclip-lockfile-bot`:
   - Disable the webhook.
   - Repository permissions: `Contents: Read and write`, `Pull requests: Read and write`. Leave everything else at `No access`.
   - Allow installation only on this account.
2. Install the app on `paperclipai/paperclip` only.
3. Create a GitHub environment named `lockfile-refresh`. Under deployment branches and tags, allow only `master`. Add no required reviewers.
4. In that environment, add:
   - variable `LOCKFILE_BOT_CLIENT_ID`: the app's client ID
   - secret `LOCKFILE_BOT_PRIVATE_KEY`: a private key generated for the app
5. In the `master` ruleset, add the app as a bypass actor with bypass mode `For pull requests only`. The app can then merge PRs past required checks and code owner review, but it cannot push to `master` directly.

The workflow mints the app token after dependency resolution finishes and merges only the commit it pushed (`--match-head-commit`). A commit anyone else pushes to `chore/refresh-lockfile` makes the merge fail.

## 9. Protect Release Infrastructure Specifically

These files should always trigger code owner review:

- `.github/workflows/release.yml`
- `scripts/release.sh`
- `scripts/release-lib.sh`
- `scripts/release-package-map.mjs`
- `scripts/create-github-release.sh`
- `scripts/rollback-latest.sh`
- `doc/RELEASING.md`
- `doc/PUBLISHING.md`

If you want stronger controls, add a repository ruleset that explicitly blocks direct pushes to:

- `.github/workflows/**`
- `scripts/release*`

## 10. Do Not Store a Claude Token in GitHub Actions

Do not add a personal Claude or Anthropic token for automatic changelog generation.

Recommended policy:

- stable changelog generation happens locally from a trusted maintainer machine
- canaries never generate changelogs

This keeps LLM spending intentional and avoids a high-value token sitting in Actions.

## 11. Verify the Canary Workflow

After setup:

1. merge a harmless commit to `master`
2. open the `Release` workflow run triggered by that push
3. confirm it passes verification
4. confirm publish succeeds under the `npm-canary` environment
5. confirm npm now shows a new `canary` release
6. confirm a git tag named `canary/vYYYY.MDD.P-canary.N` was pushed

Install-path check:

```bash
npm install --prefix "$(mktemp -d)" paperclipai@canary --no-audit --no-fund
```

The release script runs this clean-prefix install after publishing every workspace
package dependency-first and publishing `paperclipai` last. A package that is not
yet registry-visible stops the train before the channel entrypoint can advance.

## 12. Verify the Stable Workflow

After at least one good canary exists:

1. resolve the target stable version with `./scripts/release.sh stable --date YYYY-MM-DD --print-version`
2. prepare `releases/vYYYY.MDD.P.md` on the source commit you want to promote
3. open `Actions` -> `Release`
4. run it with:
   - `source_ref`: the tested commit SHA or canary tag source commit
   - `stable_date`: leave blank or set the intended UTC date like `2026-03-18`
     do not enter a version like `2026.318.0`; the workflow computes that from the date
   - `dry_run`: `true`
5. confirm the dry-run succeeds
6. rerun with `dry_run: false`
7. approve the `npm-stable` environment when prompted
8. confirm npm `latest` points to the new stable version
9. confirm git tag `vYYYY.MDD.P` exists
10. confirm the GitHub Release was created

Implementation note:

- the GitHub Actions stable workflow calls `create-github-release.sh` with `PUBLISH_REMOTE=origin`
- local maintainer usage can still pass `PUBLISH_REMOTE=public-gh` explicitly when needed

## 13. Suggested Maintainer Policy

Use this policy going forward:

- canaries are automatic and cheap
- stables are manual and approved
- only stables get public notes and announcements
- release notes are committed before stable publish
- rollback uses `npm dist-tag`, not unpublish

## 14. Troubleshooting

### Trusted publishing fails with an auth error

Check:

1. the workflow filename on GitHub exactly matches the filename configured in npm
2. the package has the trusted publisher entry for the correct repository
3. the job has `id-token: write`
4. the job is running from the expected repository, not a fork

### Stable workflow runs but never asks for approval

Check:

1. the `publish` job uses environment `npm-stable`
2. the environment actually has required reviewers configured
3. the workflow is running in the canonical repository, not a fork

### CODEOWNERS does not trigger

Check:

1. `.github/CODEOWNERS` is on the default branch
2. branch protection on `master` requires code owner review
3. the owner identities in the file are valid reviewers with repository access

## Related Docs

- [doc/RELEASING.md](RELEASING.md)
- [doc/PUBLISHING.md](PUBLISHING.md)
- [doc/plans/2026-03-17-release-automation-and-versioning.md](plans/2026-03-17-release-automation-and-versioning.md)

## Runner verification dependency cache

`release-verify.yml` runs `Verify Paperclip Runner` on two independent runners.
The protocol lane runs `check:eval-kernel` and `check:protocol`. The Rust lane
runs `check:runner` and `check:api-authority`. Together they retain every check
in `check:all`; both lanes must pass before Cloud source verification or
readiness can succeed. A failed lane does not cancel the other lane.

Both lanes restore Cargo dependencies with the pinned Rust Cache action. The
compiler comes from the Runner package's `rust-toolchain.toml` before the action
computes its key. Compiler and Cargo metadata changes select a new cache. The
`release-runner-v2` shared key avoids separate copies for these lanes. Its suffix
hashes the virtual workspace's `runner/Cargo.toml` and ancestor Cargo configuration.
Rust Cache discovers member manifests through Cargo metadata, which omits this
virtual manifest. Without the suffix, changing `[profile.dev]` can restore an
exact but obsolete dependency cache on every run. GitHub caches are immutable,
so the successful rebuild cannot replace that old entry. The profile/configuration
hash also partitions fallback restores. Every reader and writer uses the same
inputs, including the separate typecheck and cross-target caches.
Only the Rust lane saves this cache. After verification it also runs `build:rust`
to warm the debug dependencies used by the protocol lane; its own tests already
warm release dependencies. The cache writer is shorter than the protocol lane.

GitHub matches a cache entry on its key and on a version hash of the absolute
paths in the entry. The master writer runs on the RunsOn fleet, where the
checkout is `/home/runner/_work/paperclip/paperclip`. The trusted PR workflow
restores the same entry read-only on GitHub-hosted runners, where the checkout
is `/home/runner/work/paperclip/paperclip`. A `Pin the Runner Rust workspace
path` step in both workflows links `$HOME/paperclip-runner-rust` to the Runner
crate and passes that path to the cache action, so both layouts hash the same
paths and the PR lanes can restore master's entry. The guard tests under
`.github/scripts/tests/` require this step, with identical text, before every
Rust cache step in both workflows.

Workspace crates and installed Cargo binaries are excluded from this entry.
Every run runs all assigned checks, including on a cache hit. In release
verification, only an own-repository master-push run verifying that push's exact
SHA can restore the cache, and only a successful Rust lane saves it. The PR
workflow restores master's cache read only. Tag and manual candidate release
verification compile without this cache. A miss or eviction costs compilation
time but does not change the checks. To discard old dependency caches, increment
the shared-key version and let the next successful master verification warm it.

### Installed JavaScript dependencies

`setup-ci-dependencies` can restore exact `node_modules` trees and separately
certified shared/SDK build outputs. `scripts/ci-dependency-cache.mjs` hashes
manifests, resolution, patches, install scripts, and the runtime/checkout layout.
The root workspace-link hook and the server/Runner Copilot asset hooks are known
postinstall commands. An exact dependency restore replays these hooks before
validating workspace links and build outputs. Copilot defers installation in an
unbuilt source checkout; its normal build installs and verifies the assets.
Provider assets are not included in the dependency cache. Any unrecognized
workspace lifecycle hook still disables materialized reuse. Only a frozen install
from the canonical master push can publish the shared entries.

For CI cache changes, the `profile-ci` label runs the paired setup benchmark
and exercises restored dependencies with the shared-package tests. The longer
server test comparison runs when its measured scripts or fixtures change;
cache-only changes do not require unchanged test commands to run faster.
It also builds Rust units on one GitHub-hosted runner and restores the dependency
and workspace caches on a second runner with a fresh checkout. Both restores
must be exact, and Cargo must report every unit as fresh. These qualification
caches are scoped to the PR and run; they cannot populate master's caches.

### Runner release crate cache

The dependency entry leaves the Runner's own crate out, and a fresh checkout
invalidates cargo's mtime fingerprints for workspace crates anyway, so every
pull request build compiled `paperclip-runner-core` in release mode from
scratch (28-49s per job on 2026-10-11) although most master commits leave its
inputs untouched. A second, content-addressed entry holds only that crate's
release artifacts: its fingerprints, the lib's rlib and rmeta, the
`paperclip-runnerd` binary and their dep-info, about 70 MB. The path list and
the key inputs live in `scripts/runner-release-crate-cache.mjs`, which both
workflows call, and `scripts/__tests__/runner-release-crate-cache.test.mjs`
holds the readers and the writer to them.

The key is `runner-release-crate-v1-<scope>-<hashFiles>`. The scope names the
platform, the pinned compiler (`rustc --version --verbose`: release, commit and
host) and a hash of the checkout root, because cargo's dep-info records the
protocol files the crate includes by relative path as absolute paths. The
`hashFiles` part covers every input the build reads: the toolchain pin, the
manifests and lockfile, the crate sources, the protocol schemas and fixtures,
and the three fixture trees the test modules include (two under the Runner
package and `tests/runner-e2e/fixtures` at the repository root). Any change
to any of them selects a new key. A guard test resolves every `include_str!`
and `include_bytes!` in the crate, whether its argument is a literal or a
`concat!` of literals, `env!("CARGO_MANIFEST_DIR")` and macro metavariables,
and fails on an include that lands outside these patterns or uses a form it
cannot resolve. Readers pass no `restore-keys`: a prefix match would hand
cargo another revision's output.

The writer is the Rust lane of the same master-push verification, after its
checks and the debug warm-up. It looks the key up first and, on a miss, runs
`cargo clean --release -p paperclip-runner-core` and then the package's
`build:binary` (the exact command the readers run) before saving, so the entry
carries no test-profile variants. In the trusted PR workflow, the Typecheck,
Build and Canary Dry Run jobs and the two Verify Paperclip Runner lanes that
build the crate (`rust` and the final vitest shard) restore the entry
read-only right after the dependency restore. On a hit, `mark-current` sets
the mtime of every restored artifact to now, so cargo's freshness check sees
every source as older than the build and keeps the artifacts. A restore that
lacks any file cargo reads or checks for either unit (the fingerprint hash
and dep-info of each unit, the lib's rlib and rmeta, the binary under
`deps`; measured with cargo 1.97.1, which recompiles without any one of
them and recreates the uplifted binary without any of the rest) is removed
instead, the job log names the missing file, and cargo compiles as on a
miss. `pnpm build`
then prints `Finished release profile` in well under a second for that crate.
On a miss nothing changes: cargo compiles the crate as before.

The trust boundary is unchanged: only master writes, a pull request can only
restore master's entry, and it restores only artifacts built from inputs
byte-identical to its own merge tree.

The trust boundary is the protected master branch, not the cache-key text.
GitHub does not let master restore caches created by a child branch, sibling
branch, tag, or PR merge ref. Both permitted restore scopes (current branch and
default branch) are master here. A workflow with authority to execute arbitrary
code on master can affect verification directly and is already trusted. The
cache contains dependency build artifacts, not credentials or workspace output.
See [GitHub cache access restrictions](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache).

## Chat integration test shards

Release verification runs the large chat integration file on three independent
runners. Five other server shards cover every remaining general server file.
The ordinary local test command and trusted PR workflow keep their complete
`general-server` group. Each chat case shuts down its services, pauses its own
still-active endpoints, and retires its active/waiting conversations after
assertions. This keeps workers in later cases from claiming earlier
fixtures in the shared test database. Application assertions stay unchanged.

Each chat job collects active tests with Vitest, groups cases by source line,
and balances those groups by case count. Parameterized cases and loop-generated
cases on one line stay together. The job re-collects with the exact line filters
it will execute and fails if the selected case identities differ. Hooks and test
execution remain sequential inside each runner with its own temporary home.

Run one shard locally with:

```sh
pnpm test:run:general -- --group general-chat --shard-index 0 --shard-count 3
```

Use indexes 0, 1, and 2 to run the complete chat suite. The CLI validates that
each shard has work and that collection includes usable source locations. A
Vitest collection or filtering change fails verification instead of dropping
tests. Splitting adds three release-verification jobs and repeats collection and
fixture setup; it does not make a single test faster.

The file-duration manifest also records the native Codex Runner integration
suite's measured import and execution cost, so the existing file balancer
accounts for it in both ordinary PR and release verification.


## Cloud readiness runner placement

When AWS routing is enabled, Cloud image builds use `paperclip-cloud-build-x64`
and source verification uses `paperclip-post-merge-x64`. The artifact wait and
the `Cloud source verified v1` and `Cloud deployable v1` marker jobs run on
GitHub-hosted runners. These small jobs must not hold or wait for capacity in
the source-verification fleet. During a merge
burst, even a completed build must wait for its marker before consumers can
recognize readiness.

Runner placement does not change readiness requirements: exact-source artifacts,
all source checks, and the image verification must still pass. The versioned
markers and their dependency gates are unchanged.