Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-15 12:28:10 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
9611da2b38c080e7608d760facba51ad3feed4e4
PaperClipAI/doc/PUBLISHING.md
T
Devin Foley b5a3a863c3 feat(release): bootstrap new npm packages with a placeholder publish (#11757) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Its release pipeline publishes a set of npm packages from CI with
npm trusted publishing (GitHub OIDC), gated by
`scripts/release-package-manifest.json`
> - A brand-new package name cannot be published by CI directly: the PR
bootstrap gate requires the name to resolve on npm, and a
trusted-publisher rule can only be configured after the package page
exists
> - The current bootstrap helper closes that gap by building the package
locally and publishing its real output from a maintainer machine —
before the PR that adds the package has passed CI or review
> - This pull request replaces that flow: the helper now publishes a
minimal deprecated placeholder at version `0.0.0` that only reserves the
name, so every real version ships from CI
> - The benefit is that unreviewed build output never reaches npm, and
the bootstrap runs from any checkout (including `master`, before the new
package's PR merges) with no local build

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The one-time npm bootstrap for a brand-new release package (`pnpm run
release:bootstrap-package`).

**Current behavior**

The helper builds the target package locally and publishes the real
build output from a maintainer machine. That content has not passed
repository CI or review at publish time. The helper also requires the
new package to exist in the local workspace, so it must run from the
(unmerged) PR branch that adds the package.

**Proposed behavior**

The helper publishes a three-file placeholder at version `0.0.0`
(manifest, README, and an `index.js` that throws a descriptive error),
waits for the registry to show the package, then deprecates it. The PR
bootstrap gate (`scripts/check-release-package-bootstrap.mjs`) only
requires the name to resolve on the registry, so the placeholder
satisfies it. The first real calver release from CI supersedes the
placeholder, and a stable release moves `latest` off it — the same
`latest` window that existed under the old flow, but containing an
explicit inert stub instead of unreviewed code.

**Reason and benefit**

Real package content only ever reaches npm from CI, after review and
merge. The bootstrap becomes safer (scope guard refuses names outside
`@paperclipai/`, already-published names are rejected) and simpler (no
local build, no workspace state, runs from any checkout).

**Breaking changes**

None at runtime. The helper's CLI surface changes: it now takes a
package name only (no directory selector) and drops `--skip-build`.
`doc/PUBLISHING.md` is updated to match.

## What Changed

- `scripts/bootstrap-npm-package.mjs`: replaced the build-and-publish
flow with a placeholder publish — stages `package.json` + `README.md` +
throwing `index.js` at version `0.0.0` in a temp directory, previews
with `npm publish --dry-run`, and publishes only with `--publish`.
One-time passwords are prompted interactively (never passed as
arguments, since they are single-use and would land in shell history),
with re-prompt on a rejected or expired code. After publishing, the
helper polls the registry until the package is visible (a first publish
can lag by minutes; verified live at ~5 minutes), requiring two
consecutive sightings before prompting for a second code and deprecating
the placeholder so accidental installs warn loudly; on timeout or
failure it prints the exact manual `npm deprecate` command. Added an
`@paperclipai/`-scope guard and a fail-fast error when `--publish` runs
without an interactive terminal. Removed the workspace-plan dependency
so it runs from any checkout.
- `scripts/bootstrap-npm-package.test.mjs`: rewrote for the new
interface — argument parsing, scope validation, the generated
placeholder files (manifest shape, throwing entry point, README), the
OTP re-prompt loop, and the registry poll (consecutive-sighting
requirement, timeout, transient-error tolerance) via injected fakes.
- `doc/PUBLISHING.md`: rewrote the "One-time bootstrap sequence for a
new package" section for the placeholder flow, including the `latest`
dist-tag window and the trusted-publishing setup ordering (placeholder
publish → trusted publisher rule → `"publishFromCi": true`).
- `.github/scripts/check-pr-release-bootstrap.mjs` (+ test, + wiring in
`run-quality-gates.mjs`): new informational commitperclip notice on PRs
that need this bootstrap. It fires when the PR newly release-enables a
package that is missing from npm, or adds an unpublished `publishFromCi:
false` package that published packages declare a `workspace:*`
dependency on, and names the exact maintainer command — so contributors
know the red `policy` check is not theirs to fix. It never fails the
gate (the `policy` job remains the enforcer), only looks up
scope-validated names on the registry, and stays quiet on registry
errors.

## Verification

- `node --test scripts/bootstrap-npm-package.test.mjs`: 13/13 pass
- `node --test .github/scripts/tests/*.test.mjs`: 147/147 pass (10 new
for the PR notice)
- `pnpm run test:release-registry`: 82/82 pass
- Replayed the new PR notice against a real historical PR's live API
data (files, manifest at base and head refs): with the registry in its
pre-bootstrap state it produces the exact maintainer instruction; with
the package bootstrapped it stays silent
- Full live end-to-end run: the flow bootstrapped
`@paperclipai/adapter-kimi-local` for real — dry-run preview (634-byte,
3-file tarball), publish, registry visibility after ~5 minutes of
propagation lag, deprecation confirmed via `npm view ... deprecated`
- Guards verified live: an already-published name is rejected, an
out-of-scope name (`left-pad`) is rejected, unknown options (including
the removed `--otp`) are rejected, and `--publish` in a non-interactive
shell fails fast before any network call

## Risks

- The `latest` dist-tag points at the deprecated `0.0.0` placeholder
until the first stable release supersedes it. This window also existed
under the old flow (which parked `latest` at a locally built version);
internal consumers are unaffected because release version rewrites pin
exact calver versions.
- The registry poll caps at ~10 minutes. If propagation is slower than
that, the helper prints the exact `npm deprecate ... --otp <code>`
command to run manually once `npm view` resolves.
- The helper no longer validates the name against the workspace release
plan, so a typo within the `@paperclipai/` scope would reserve a wrong
name. The dry-run preview shows the exact name before any publish.

## Model Used

- Anthropic, **Claude Fable 5** (`claude-fable-5`) via Claude Code, with
repository, shell, and Git tooling. It analyzed the existing bootstrap
flow and the release scripts (`release-package-map.mjs`,
`check-release-package-bootstrap.mjs`, `release.sh` dist-tag handling),
wrote the replacement script and tests, updated the documentation, and
ran the verification above.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-19 19:38:17 -07:00

267 lines
11 KiB
Markdown
Raw Blame History

# Publishing to npm
Low-level reference for how Paperclip packages are prepared and published to npm.
For the maintainer workflow, use [doc/RELEASING.md](RELEASING.md). This document focuses on packaging internals.
## Current Release Entry Points
Use these scripts:
- [`scripts/release.sh`](../scripts/release.sh) for canary and stable publish flows
- [`scripts/create-github-release.sh`](../scripts/create-github-release.sh) after pushing a stable tag
- [`scripts/rollback-latest.sh`](../scripts/rollback-latest.sh) to repoint `latest`
- [`scripts/build-npm.sh`](../scripts/build-npm.sh) for the CLI packaging build
Paperclip no longer uses release branches or Changesets for publishing.
## Why the CLI needs special packaging
The CLI package, `paperclipai`, imports code from workspace packages such as:
- `@paperclipai/server`
- `@paperclipai/db`
- `@paperclipai/shared`
- adapter packages under `packages/adapters/`
Those workspace references are valid in development but not in a publishable npm package. The release flow rewrites versions temporarily, then builds a publishable CLI bundle.
## `build-npm.sh`
Run:
```bash
./scripts/build-npm.sh
```
This script:
1. runs the forbidden token check unless `--skip-checks` is supplied
2. runs `pnpm -r typecheck`
3. bundles the CLI entrypoint with esbuild into `cli/dist/index.js`
4. verifies the bundled entrypoint with `node --check`
5. rewrites `cli/package.json` into a publishable npm manifest and stores the dev copy as `cli/package.dev.json`
6. copies the repo `README.md` into `cli/README.md` for npm metadata
After the release script exits, the dev manifest and temporary files are restored automatically.
## Package discovery and versioning
Public packages are discovered from:
- `packages/`
- `server/`
- `ui/`
- `cli/`
The version rewrite step now uses [`scripts/release-package-map.mjs`](../scripts/release-package-map.mjs), which:
- finds all public packages
- sorts them topologically by internal dependencies
- rewrites each package version to the target release version
- rewrites internal `workspace:*` dependency references to the exact target version
- updates the CLI's displayed version string
Those rewrites are temporary. The working tree is restored after publish or dry-run.
## `@paperclipai/ui` packaging
The UI package publishes prebuilt static assets, not the source workspace.
The `ui` package uses [`scripts/generate-ui-package-json.mjs`](../scripts/generate-ui-package-json.mjs) during `prepack` to swap in a lean publish manifest that:
- keeps the release-managed `name` and `version`
- publishes only `dist/`
- omits the source-only dependency graph from downstream installs
After packing or publishing, `postpack` restores the development manifest automatically.
### Manual first publish for `@paperclipai/ui`
If you need to publish only the UI package once by hand, use the real package name:
- `@paperclipai/ui`
Recommended flow from the repo root:
```bash
# optional sanity check: this 404s until the first publish exists
npm view @paperclipai/ui version
# make sure the dist payload is fresh
pnpm --filter @paperclipai/ui build
# confirm your local npm auth before the real publish
npm whoami
# safe preview of the exact publish payload
cd ui
pnpm publish --dry-run --no-git-checks --access public
# real publish
pnpm publish --no-git-checks --access public
```
Notes:
- Publish from `ui/`, not the repo root.
- `prepack` automatically rewrites `ui/package.json` to the lean publish manifest, and `postpack` restores the dev manifest after the command finishes.
- If `npm view @paperclipai/ui version` already returns the same version that is in [`ui/package.json`](../ui/package.json), do not republish. Bump the version or use the normal repo-wide release flow in [`scripts/release.sh`](../scripts/release.sh).
If the first real publish returns npm `E404`, check npm-side prerequisites before retrying:
- `npm whoami` must succeed first. An expired or missing npm login will block the publish.
- For an organization-scoped package like `@paperclipai/ui`, the `paperclipai` npm organization must exist and the publisher must be a member with permission to publish to that scope.
- The initial publish must include `--access public` for a public scoped package.
- npm also requires either account 2FA for publishing or a granular token that is allowed to bypass 2FA.
## Version formats
Paperclip uses calendar versions:
- stable: `YYYY.MDD.P`
- canary: `YYYY.MDD.P-canary.N`
Examples:
- stable: `2026.318.0`
- canary: `2026.318.1-canary.2`
## Publish model
### Canary
Canaries publish under the npm dist-tag `canary`.
Example:
- `paperclipai@2026.318.1-canary.2`
This keeps the default install path unchanged while allowing explicit installs with:
```bash
npx paperclipai@canary onboard
```
The release script now verifies two things after a canary publish:
- the `canary` dist-tag resolves to the version that was just published
- every published internal `@paperclipai/*` dependency referenced by that manifest exists on npm
It also treats `latest -> canary` as a failure by default, because npm metadata can otherwise leave the default install path pointing at an unreleased canary dependency graph. Only pass `./scripts/release.sh canary --allow-canary-latest` when that `latest` behavior is explicitly intended.
### Stable
Stable publishes use the npm dist-tag `latest`.
Example:
- `paperclipai@2026.318.0`
Stable publishes do not create a release commit. Instead:
- package versions are rewritten temporarily
- packages are published from the chosen source commit
- git tag `vYYYY.MDD.P` points at that original commit
## Trusted publishing
The intended CI model is npm trusted publishing through GitHub OIDC.
That means:
- no long-lived `NPM_TOKEN` in repository secrets
- GitHub Actions obtains short-lived publish credentials
- trusted publisher rules are configured per workflow file
See [doc/RELEASE-AUTOMATION-SETUP.md](RELEASE-AUTOMATION-SETUP.md) for the GitHub/npm setup steps.
## Release enrollment for new public packages
Paperclip does not auto-publish every non-private workspace package anymore.
CI publishing is controlled by [`scripts/release-package-manifest.json`](../scripts/release-package-manifest.json).
When you add a new public package:
1. add it to the manifest and decide whether CI should publish it immediately
2. if CI should publish it, reserve the name on npm with the placeholder bootstrap before merge
3. if CI should not publish it yet, keep `"publishFromCi": false`
4. only enable `"publishFromCi": true` after npm trusted publishing is configured for that package
PR CI now checks changed release-enabled package manifests against npm. That catches a missing first-publish bootstrap before the change reaches `master`. When a PR needs this bootstrap, commitperclip also posts an informational notice on the PR naming the exact command, so contributors know a maintainer action is pending rather than something they can fix.
### One-time bootstrap sequence for a new package
Creating a brand-new package name on npm still needs one human maintainer with npm write access.
After that, trusted publishing takes over — and CI publishes the only real content the package ever gets.
The bootstrap intentionally does **not** publish the package's real build output.
It publishes a tiny placeholder at version `0.0.0` (a manifest, a README, and an
`index.js` that throws a descriptive error), because:
- real package content should only ever reach npm from CI, after the PR that adds the package has been reviewed and merged
- the PR CI gate only requires the name to resolve on the registry
- the trusted publisher rule can only be configured once the package page exists
- the placeholder needs no local build and no workspace state, so it can be published from any checkout (including `master`, before the new package's PR merges)
Example for a newly added public package from the repo root:
```bash
# safe preview (stages the placeholder and runs npm publish --dry-run)
pnpm run release:bootstrap-package -- @paperclipai/new-package
# one-time placeholder publish from an authenticated maintainer machine
# (prompts for npm one-time passwords; they are never passed as arguments)
pnpm run release:bootstrap-package -- @paperclipai/new-package --publish
```
The helper script:
- refuses names outside the `@paperclipai/` scope
- checks that the package does not already exist on npm
- stages the placeholder in a temporary directory and previews it with `npm publish --dry-run --access public`
- with `--publish`, prompts for a one-time password and publishes. Codes are entered interactively and handed to npm through its environment (`npm_config_otp`), so they never appear on a command line, in shell history, or in a process listing; a rejected or expired code re-prompts
- then waits for the registry to show the package (a first publish can take a few minutes to become visible on the read/write endpoints) and prompts for a second code to deprecate the placeholder, so accidental installs warn loudly. If the wait times out or the deprecation fails, it prints the exact `npm deprecate` command to run manually
Until the first stable release supersedes it, the `latest` dist-tag points at the
deprecated placeholder. Internal consumers are unaffected: release version
rewrites pin exact calver versions, so nothing inside the release package set
resolves through `latest`.
For the real `--publish` step, the maintainer machine must already be authenticated to npm.
If `npm whoami` returns `401`, first run `npm logout --registry=https://registry.npmjs.org/` to clear any stale local auth, then run `npm login` or `npm adduser` locally as an npm org member, and finally rerun the helper.
That local human auth is fine for the one-time bootstrap publish; we just do not want the same auth model inside CI.
`--publish` requires an interactive terminal: the helper prompts for the one-time password right before the publish and again before the deprecation, handing each code to npm through its environment (`npm_config_otp`), so codes never appear in command arguments, shell history, or process listings.
After the placeholder publish succeeds:
1. open `https://www.npmjs.com/package/@paperclipai/new-package`
2. go to `Settings` → `Trusted publishing`
3. add repository `paperclipai/paperclip`
4. set workflow filename to `release.yml`
5. optionally go to `Settings` → `Publishing access` and enable `Require two-factor authentication and disallow tokens`
6. only then set `"publishFromCi": true` in [`scripts/release-package-manifest.json`](../scripts/release-package-manifest.json)
Once those steps are done, future canary and stable publishes for that package are automated through GitHub OIDC. The manual step only reserves the name on npm; every real version ships from CI.
## Rollback model
Rollback does not unpublish anything.
It repoints the `latest` dist-tag to a prior stable version:
```bash
./scripts/rollback-latest.sh 2026.318.0
```
This is the fastest way to restore the default install path if a stable release is bad.
## Related Files
- [`scripts/build-npm.sh`](../scripts/build-npm.sh)
- [`scripts/generate-npm-package-json.mjs`](../scripts/generate-npm-package-json.mjs)
- [`scripts/generate-ui-package-json.mjs`](../scripts/generate-ui-package-json.mjs)
- [`scripts/release-package-map.mjs`](../scripts/release-package-map.mjs)
- [`cli/esbuild.config.mjs`](../cli/esbuild.config.mjs)
- [`doc/RELEASING.md`](RELEASING.md)
Reference in new issue View Git Blame Copy Permalink