Files
PaperClipAI/scripts/release-lib.sh
T
Devin Foley f9173782cd feat(release): add smoke-gated nightly channel and lane-separated Docker tags (#11006)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The release subsystem publishes the `paperclipai` npm package set
and the Docker images on two lanes: canary on every master push, and
stable on manual promotion
> - There is no middle ground between those lanes. Users must track
every merge or wait weeks for a stable. Docker `:latest` also tracks
master, so Docker users have no stable image at all
> - A calm prerelease lane needs to exist, and it must never ship a
build that failed its checks
> - This pull request adds the nightly channel: a scheduled job that
selects the newest master commit with a green canary publish, runs the
full release smoke suite against that exact published canary, and only
then republishes it as the nightly. It also separates Docker tags by
lane, so `:latest` finally means stable
> - The benefit is that users can follow prereleases at a nightly
cadence with a smoke-tested guarantee, and Docker users get real
`:canary`, `:nightly`, and stable image tags

## Linked Issues or Issue Description

**Subsystem affected**

Release automation: `scripts/release.sh`, `scripts/release-lib.sh`,
`.github/workflows/release.yml`, `.github/workflows/docker.yml`,
`.github/workflows/release-smoke.yml`.

**Problem or motivation**

The project publishes only `canary` (every master push) and `latest`
(manual stable). Users who want prereleases without per-merge churn have
no option. Docker has a second problem: master builds overwrite
`:latest`, and CI-published stables never produced Docker images,
because tags pushed with `GITHUB_TOKEN` do not fire the `v*` tag trigger
in `docker.yml`. No stable-versioned image exists in ghcr today.

**Proposed solution**

Add a `nightly` channel. A scheduled job selects the newest
canary-tagged master commit, smoke-tests that exact published canary,
and republishes the same commit as `YYYY.MDD.P-nightly.N` under the
`nightly` dist-tag. Separate Docker tags by lane (`:canary` for master,
`:nightly` for nightly tags, `:latest` plus version tags for stable tags
only), and have the release jobs dispatch `docker.yml` at the new tag so
lane images actually build.

**Alternatives considered**

Moving the `nightly` dist-tag to the existing canary version without a
republish. Rejected: the version string would say `canary` while the
user is on nightly, which breaks at-a-glance lane identification in bug
reports and `--version` output.

## What Changed

- `scripts/release-lib.sh`: channel-parameterized
`next_prerelease_version` and `prerelease_tag_name` helpers (canary
helpers delegate to them), a `require_channel_tag_at_head` guard, and
the no-provenance retry for Sigstore transparency-log duplicates now
covers the `nightly` dist-tag as well as `canary`
- `scripts/release.sh`: new `nightly` channel. It requires HEAD to carry
a `canary/v*` tag, publishes the full public package set as
`YYYY.MDD.P-nightly.N` under dist-tag `nightly`, and tags the source
commit `nightly/vYYYY.MDD.P-nightly.N`
- `.github/workflows/release.yml`: scheduled nightly chain (09:00 UTC) —
select candidate, smoke it via `release-smoke.yml`, publish on green
under the existing `npm-canary` environment, push the tag, dispatch
`docker.yml`. New `channel` dispatch input (default `stable`, so
existing stable dispatches are unchanged) with `nightly_source_version`
and `dry_run` support for forced runs. The stable path now also
dispatches `docker.yml` at the new `v*` tag
- `.github/workflows/docker.yml`: lane tag mapping for both image jobs —
master pushes publish `:canary` and no longer move `:latest`;
`nightly/v*` tags publish `:nightly`; only stable `v*` tags publish
`:latest` and the versioned tags. New `workflow_dispatch` trigger for
the release-job dispatches. Build-version stamping uses the exact
nightly version on nightly tag builds
- `.github/workflows/release-smoke.yml`: `nightly` added to the dispatch
choice list
- `doc/CHANNELS.md` (new): user-facing guide to the channels
- `doc/RELEASING.md`: nightly lane documentation, Docker tag mapping
table, and a nightly failure playbook
- `doc/RELEASE-AUTOMATION-SETUP.md`: note that nightly reuses
`npm-canary` and needs no npm trusted-publisher changes
- Tests: channel-parameterized version helper coverage in
`scripts/release-registry-versions.test.mjs`, and nightly flow coverage
(publish identity, notes not required, canary-tag guard) in
`scripts/__tests__/release-dry-run-notes.test.mjs`

## Verification

- `node --test` on the release script suites: 68 pass, including 6 new
tests. The only failure, `acpx-patch-packaging.test.mjs`, needs
installed `node_modules` and fails identically on a pristine checkout of
master in the same environment
- `bash -n` on both shell scripts and YAML parse of all three workflows
- Live fail-path check: `./scripts/release.sh nightly --print-version`
from a master tip with no canary tag fails with `HEAD has no canary/v*
tag`
- Live success-path check: the same command from the
`canary/v2026.806.0-canary.7` commit prints `2026.806.0-nightly.0`
- Live selection check: the candidate-selection shell logic run against
the real repository selects the commit of `canary/v2026.806.0-canary.7`,
which matches the current npm `canary` dist-tag exactly
- After merge: dispatch `release.yml` with `channel: nightly` and
`dry_run: true` to preview, then a real forced run to validate end to
end before the first scheduled run

## Risks

- Docker `:latest` changes meaning from "latest master build" to "latest
stable release". This is deliberate and will be announced. Users who
want the old behavior pull `:canary`. Until the first stable release
after this change, `:latest` stays at its current (master-built) image
- The nightly is a rebuild of the same source commit, not the
byte-identical canary artifact that was smoked. The lockfile pins
dependencies, and the publish path's registry-visibility and
clean-prefix install gates still run on the nightly artifacts
- All npm publishing must stay inside `release.yml` because npm trusted
publishing pins that workflow file per package. The nightly jobs were
added to `release.yml` for exactly that reason; this constraint is now
documented in `RELEASING.md`
- The stable-lane Docker dispatch fails gracefully (a warning with
manual instructions) when the source ref predates `docker.yml`'s
`workflow_dispatch` trigger

## Model Used

Claude Fable 5 (`claude-fable-5`, Anthropic) in Claude Code, with
extended thinking and full tool use (repository exploration, local test
execution, live registry and git verification). All code, tests, and
docs in this PR were model-authored under human direction.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending — will confirm before
merge)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending — will confirm before merge)
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-10 12:16:55 -07:00

550 lines
14 KiB
Bash

#!/usr/bin/env bash
if [ -z "${REPO_ROOT:-}" ]; then
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
fi
release_info() {
echo "$@"
}
release_warn() {
echo "Warning: $*" >&2
}
release_fail() {
echo "Error: $*" >&2
exit 1
}
git_remote_exists() {
git -C "$REPO_ROOT" remote get-url "$1" >/dev/null 2>&1
}
github_repo_from_remote() {
local remote_url
remote_url="$(git -C "$REPO_ROOT" remote get-url "$1" 2>/dev/null || true)"
[ -n "$remote_url" ] || return 1
remote_url="${remote_url%.git}"
remote_url="${remote_url#ssh://}"
node - "$remote_url" <<'NODE'
const remoteUrl = process.argv[2];
const patterns = [
/^https?:\/\/github\.com\/([^/]+\/[^/]+)$/,
/^git@github\.com:([^/]+\/[^/]+)$/,
/^[^:]+:([^/]+\/[^/]+)$/
];
for (const pattern of patterns) {
const match = remoteUrl.match(pattern);
if (!match) continue;
process.stdout.write(match[1]);
process.exit(0);
}
process.exit(1);
NODE
}
resolve_release_remote() {
local remote="${RELEASE_REMOTE:-${PUBLISH_REMOTE:-}}"
if [ -n "$remote" ]; then
git_remote_exists "$remote" || release_fail "git remote '$remote' does not exist."
printf '%s\n' "$remote"
return
fi
if git_remote_exists public-gh; then
printf 'public-gh\n'
return
fi
if git_remote_exists public; then
printf 'public\n'
return
fi
if git_remote_exists origin; then
printf 'origin\n'
return
fi
release_fail "no git remote found. Configure RELEASE_REMOTE or PUBLISH_REMOTE."
}
fetch_release_remote() {
git -C "$REPO_ROOT" fetch "$1" --prune --tags
}
git_current_branch() {
git -C "$REPO_ROOT" symbolic-ref --quiet --short HEAD 2>/dev/null || true
}
git_local_tag_exists() {
git -C "$REPO_ROOT" show-ref --verify --quiet "refs/tags/$1"
}
git_remote_tag_exists() {
git -C "$REPO_ROOT" ls-remote --exit-code --tags "$2" "refs/tags/$1" >/dev/null 2>&1
}
get_last_stable_tag() {
git -C "$REPO_ROOT" tag --list 'v*' --sort=-version:refname | head -1
}
get_current_stable_version() {
local tag
tag="$(get_last_stable_tag)"
if [ -z "$tag" ]; then
printf '0.0.0\n'
else
printf '%s\n' "${tag#v}"
fi
}
stable_version_slot_for_date() {
node - "${1:-}" <<'NODE'
const input = process.argv[2];
const date = input ? new Date(`${input}T00:00:00Z`) : new Date();
if (Number.isNaN(date.getTime())) {
console.error(`invalid date: ${input}`);
process.exit(1);
}
const month = String(date.getUTCMonth() + 1);
const day = String(date.getUTCDate()).padStart(2, '0');
process.stdout.write(`${date.getUTCFullYear()}.${month}${day}`);
NODE
}
utc_date_iso() {
node <<'NODE'
const date = new Date();
const y = date.getUTCFullYear();
const m = String(date.getUTCMonth() + 1).padStart(2, '0');
const d = String(date.getUTCDate()).padStart(2, '0');
process.stdout.write(`${y}-${m}-${d}`);
NODE
}
next_stable_version() {
local release_date="$1"
shift
node - "$release_date" "$@" <<'NODE'
const input = process.argv[2];
const packageNames = process.argv.slice(3);
const { execSync } = require("node:child_process");
const { readFileSync } = require("node:fs");
const date = input ? new Date(`${input}T00:00:00Z`) : new Date();
if (Number.isNaN(date.getTime())) {
console.error(`invalid date: ${input}`);
process.exit(1);
}
// Optional pre-fetched version data (see release-registry-versions.mjs).
// Avoids one serial `npm view` round-trip per package.
let versionsCache = null;
if (process.env.RELEASE_PACKAGE_VERSIONS_FILE) {
try {
versionsCache = JSON.parse(readFileSync(process.env.RELEASE_PACKAGE_VERSIONS_FILE, "utf8"));
} catch {
versionsCache = null;
}
}
const stableSlot = `${date.getUTCFullYear()}.${date.getUTCMonth() + 1}${String(date.getUTCDate()).padStart(2, "0")}`;
const pattern = new RegExp(`^${stableSlot.replace(/\./g, '\\.')}\.(\\d+)$`);
let max = -1;
for (const packageName of packageNames) {
let versions = [];
if (versionsCache && Array.isArray(versionsCache[packageName])) {
versions = versionsCache[packageName];
} else {
try {
const raw = execSync(`npm view ${JSON.stringify(packageName)} versions --json`, {
encoding: "utf8",
stdio: ["ignore", "pipe", "ignore"],
}).trim();
if (raw) {
const parsed = JSON.parse(raw);
versions = Array.isArray(parsed) ? parsed : [parsed];
}
} catch {
versions = [];
}
}
for (const version of versions) {
const match = version.match(pattern);
if (!match) continue;
max = Math.max(max, Number(match[1]));
}
}
process.stdout.write(`${stableSlot}.${max + 1}`);
NODE
}
require_prerelease_channel() {
case "$1" in
canary|nightly) ;;
*) release_fail "unknown prerelease channel: $1" ;;
esac
}
next_prerelease_version() {
local channel="$1"
local stable_version="$2"
shift 2
require_prerelease_channel "$channel"
node - "$channel" "$stable_version" "$@" <<'NODE'
const channel = process.argv[2];
const stable = process.argv[3];
const packageNames = process.argv.slice(4);
const { execSync } = require("node:child_process");
const { readFileSync } = require("node:fs");
// Optional pre-fetched version data (see release-registry-versions.mjs).
// Avoids one serial `npm view` round-trip per package.
let versionsCache = null;
if (process.env.RELEASE_PACKAGE_VERSIONS_FILE) {
try {
versionsCache = JSON.parse(readFileSync(process.env.RELEASE_PACKAGE_VERSIONS_FILE, "utf8"));
} catch {
versionsCache = null;
}
}
const pattern = new RegExp(`^${stable.replace(/\./g, '\\.')}-${channel}\\.(\\d+)$`);
let max = -1;
for (const packageName of packageNames) {
let versions = [];
if (versionsCache && Array.isArray(versionsCache[packageName])) {
versions = versionsCache[packageName];
} else {
try {
const raw = execSync(`npm view ${JSON.stringify(packageName)} versions --json`, {
encoding: "utf8",
stdio: ["ignore", "pipe", "ignore"],
}).trim();
if (raw) {
const parsed = JSON.parse(raw);
versions = Array.isArray(parsed) ? parsed : [parsed];
}
} catch {
versions = [];
}
}
for (const version of versions) {
const match = version.match(pattern);
if (!match) continue;
max = Math.max(max, Number(match[1]));
}
}
process.stdout.write(`${stable}-${channel}.${max + 1}`);
NODE
}
next_canary_version() {
local stable_version="$1"
shift
next_prerelease_version canary "$stable_version" "$@"
}
release_notes_file() {
printf '%s/releases/v%s.md\n' "$REPO_ROOT" "$1"
}
stable_tag_name() {
printf 'v%s\n' "$1"
}
prerelease_tag_name() {
require_prerelease_channel "$1"
printf '%s/v%s\n' "$1" "$2"
}
canary_tag_name() {
prerelease_tag_name canary "$1"
}
npm_package_version_exists() {
local package_name="$1"
local version="$2"
local resolved
resolved="$(npm view "${package_name}@${version}" version 2>/dev/null || true)"
[ "$resolved" = "$version" ]
}
wait_for_npm_package_version() {
local package_name="$1"
local version="$2"
local attempts="${3:-12}"
local delay_seconds="${4:-5}"
local attempt=1
while [ "$attempt" -le "$attempts" ]; do
if npm_package_version_exists "$package_name" "$version"; then
return 0
fi
if [ "$attempt" -lt "$attempts" ]; then
sleep "$delay_seconds"
fi
attempt=$((attempt + 1))
done
return 1
}
is_npm_tlog_duplicate_error() {
local output="$1"
grep -q "TLOG_CREATE_ENTRY_ERROR" <<< "$output" &&
grep -q "equivalent entry already exists in the transparency log" <<< "$output"
}
package_publish_tool() {
node -e '
const pkg = require(process.cwd() + "/package.json");
const bundled = pkg.bundleDependencies ?? pkg.bundledDependencies ?? [];
process.stdout.write(bundled.length > 0 ? "npm" : "pnpm");
'
}
BUNDLED_NPM_PACK_VERSION="10.9.7"
BUNDLED_NPM_PUBLISH_VERSION="11.18.0"
run_bundled_npm_pack() {
npx --yes "npm@$BUNDLED_NPM_PACK_VERSION" "$@"
}
run_bundled_npm_publish() {
npx --yes "npm@$BUNDLED_NPM_PUBLISH_VERSION" "$@" --loglevel verbose
}
run_package_publish() {
local publish_tool="$1"
local dist_tag="$2"
local disable_provenance="${3:-false}"
if [ "$publish_tool" = "npm" ]; then
if [ "$disable_provenance" = "true" ]; then
run_bundled_npm_publish publish --tag "$dist_tag" --access public --provenance=false
else
run_bundled_npm_publish publish --tag "$dist_tag" --access public
fi
return
fi
if [ "$disable_provenance" = "true" ]; then
pnpm publish --no-git-checks --tag "$dist_tag" --access public --provenance=false
else
pnpm publish --no-git-checks --tag "$dist_tag" --access public
fi
}
publish_package_to_npm() {
local dist_tag="$1"
local package_name="$2"
local package_version="$3"
local publish_tool="${4:-pnpm}"
local publish_log
publish_log="$(mktemp "${TMPDIR:-/tmp}/paperclip-npm-publish.XXXXXX")"
if (set -o pipefail; run_package_publish "$publish_tool" "$dist_tag" false 2>&1 | tee "$publish_log"); then
rm -f "$publish_log"
return 0
fi
if ! is_npm_tlog_duplicate_error "$(cat "$publish_log")"; then
rm -f "$publish_log"
return 1
fi
release_warn "npm publish hit a duplicate Sigstore transparency-log entry for ${package_name}@${package_version}."
if npm_package_version_exists "$package_name" "$package_version"; then
release_warn "npm already exposes ${package_name}@${package_version}; continuing to registry verification."
rm -f "$publish_log"
return 0
fi
case "$dist_tag" in
canary|nightly) ;;
*)
release_warn "Not retrying ${package_name}@${package_version} without provenance for dist-tag ${dist_tag}."
rm -f "$publish_log"
return 1
;;
esac
release_warn "Retrying ${package_name}@${package_version} once with npm provenance disabled."
if run_package_publish "$publish_tool" "$dist_tag" true; then
rm -f "$publish_log"
return 0
fi
rm -f "$publish_log"
return 1
}
publish_package_to_npm_and_wait() {
local dist_tag="$1"
local package_name="$2"
local package_version="$3"
local publish_tool="${4:-pnpm}"
local attempts="${5:-12}"
local delay_seconds="${6:-5}"
publish_package_to_npm "$dist_tag" "$package_name" "$package_version" "$publish_tool" || return 1
if wait_for_npm_package_version "$package_name" "$package_version" "$attempts" "$delay_seconds"; then
return 0
fi
release_warn "npm accepted ${package_name}@${package_version}, but the version did not become registry-visible."
return 1
}
verify_npm_installable() {
local package_spec="$1"
local expected_version="$2"
local install_dir
local installed_version
install_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-release-install.XXXXXX")"
if ! npm install --prefix "$install_dir" "$package_spec" --no-audit --no-fund; then
rm -rf "$install_dir"
return 1
fi
installed_version="$(node -e "console.log(require(process.argv[1]).version)" "$install_dir/node_modules/paperclipai/package.json")"
rm -rf "$install_dir"
[ "$installed_version" = "$expected_version" ]
}
wait_for_release_registry_state() {
local attempts="${1:-12}"
local delay_seconds="${2:-5}"
shift 2
local attempt=1
local output
local status
while [ "$attempt" -le "$attempts" ]; do
if output="$(node "$REPO_ROOT/scripts/verify-release-registry-state.mjs" "$@" 2>&1)"; then
[ -n "$output" ] && printf '%s\n' "$output"
return 0
fi
status=$?
printf '%s\n' "$output" >&2
if [ "$status" -eq 2 ]; then
return "$status"
fi
if [ "$attempt" -lt "$attempts" ]; then
release_warn "npm registry metadata has not converged yet (attempt ${attempt}/${attempts}); retrying in ${delay_seconds}s."
sleep "$delay_seconds"
fi
attempt=$((attempt + 1))
done
return "${status:-1}"
}
require_clean_worktree() {
if [ -n "$(git -C "$REPO_ROOT" status --porcelain)" ]; then
release_fail "working tree is not clean. Commit, stash, or remove changes before releasing."
fi
}
require_on_master_branch() {
local current_branch
current_branch="$(git_current_branch)"
if [ "$current_branch" != "master" ]; then
release_fail "this release step must run from branch master, but current branch is ${current_branch:-<detached>}."
fi
}
# Promotion channels only republish commits that already shipped on the
# previous lane, so the source commit must carry that lane's release tag.
require_channel_tag_at_head() {
local channel="$1"
require_prerelease_channel "$channel"
if ! git -C "$REPO_ROOT" tag --points-at HEAD | grep -q "^${channel}/v"; then
release_fail "HEAD has no ${channel}/v* tag; this channel only publishes commits that already shipped a ${channel} release."
fi
}
# The inverse guard: a commit ships on a promotion channel at most once, so
# concurrent or repeated runs cannot double-publish it. Delete the lane tag
# first if a republish is genuinely intended.
require_channel_tag_absent_at_head() {
local channel="$1"
local existing
require_prerelease_channel "$channel"
existing="$(git -C "$REPO_ROOT" tag --points-at HEAD | grep "^${channel}/v" | head -1 || true)"
if [ -n "$existing" ]; then
release_fail "HEAD already shipped as ${existing}; delete that tag first if you really want to republish this commit on the ${channel} channel."
fi
}
require_npm_publish_auth() {
local dry_run="$1"
if [ "$dry_run" = true ]; then
return
fi
if npm whoami >/dev/null 2>&1; then
release_info " ✓ Logged in to npm as $(npm whoami)"
return
fi
if [ "${GITHUB_ACTIONS:-}" = "true" ]; then
release_info " ✓ npm publish auth will be provided by GitHub Actions trusted publishing"
return
fi
release_fail "npm publish auth is not available. Use 'npm login' locally or run from GitHub Actions with trusted publishing."
}
list_public_package_info() {
node "$REPO_ROOT/scripts/release-package-map.mjs" list
}
set_public_package_version() {
node "$REPO_ROOT/scripts/release-package-map.mjs" set-version "$1"
}