Files
PaperClipAI/scripts/verify-canary-provenance.test.mjs
T
DottaandClaude Opus 5.5 d16c06308d perf(release): number canaries by master position and confirm them outside the release lock (#21917)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every master push runs a canary, and only one canary at a time may
run the release workflow
> - After #21554 and #21555 a canary held that lock for 35.5 min, and
13-16 min of it was waiting for npm to show the new packages
> - The wait was required: the next canary computed `canary.N` from
npm's version list, so starting before npm showed the previous canary
would reuse its number
> - This pull request numbers a canary by its commit's position on
master and moves the registry confirmation, the git tag, the Docker
dispatch and the onboarding smoke to a separate `canary-finalize.yml`
run
> - The benefit is that the release lock is held for about 20-25 min per
canary instead of 35, with the same checks before a canary is tagged

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Canary throughput. In run 43715037497 (2026-11-15) `publish_canary` took
23.7 min, and 13.3 min of that was "Waiting for registry visibility"
(13:08:50 to 13:22:11). The next canary could not start until that run
ended.

**Current behavior**
`release.sh canary` sets `N` to (highest published `-canary.N` for
today) + 1, then publishes, waits for npm to show every package, checks
dist-tags, installs `paperclipai@canary`, and creates
`canary/v<version>`. `publish_canary` pushes the tag, dispatches
`docker.yml`, and `smoke_canary_onboarding` runs in the same run.

**Proposed behavior**
`N` = `git rev-list --count HEAD` (full history required), or the
registry number + 1 when that is higher (a rerun of the same commit).
`release.sh canary --defer-registry-confirmation` ends after npm accepts
every publish. `publish_canary` dispatches `canary-finalize.yml`, which
does the confirmation, tag, Docker dispatch and smoke outside the
release lock. Nightly, beta and stable numbering and flow are unchanged.

Refs #21554, #21555

## What Changed

- `scripts/release-lib.sh`: `next_canary_version_from_source`. A
publishing canary refuses a shallow checkout; a dry run (the PR Canary
Dry Run job) falls back to the registry number.
- `scripts/release.sh`: canary uses the new number; new canary-only
`--defer-registry-confirmation` flag skips the registry wait, the
dist-tag check and the local tag, and writes the version to
`PAPERCLIP_RELEASE_VERSION_FILE`.
- `scripts/confirm-canary-publication.sh` (new): waits for registry
visibility, checks provenance, checks dist-tags with
`--allow-newer-canary`, installs `paperclipai@<version>`, then creates
and pushes `canary/v<version>`. A tag already on the same commit is
accepted, so reruns are safe.
- `scripts/verify-canary-provenance.mjs` (new): every package's npm SLSA
provenance must name `.github/workflows/release.yml` on
`refs/heads/master` in `paperclipai/paperclip` at the source commit. A
dispatch with invented inputs cannot tag anything.
- `scripts/verify-release-registry-state.mjs`: `--allow-newer-canary`
accepts a `canary` dist-tag that a newer canary already moved; older,
foreign or missing tags still fail.
- `.github/workflows/canary-finalize.yml` (new, `workflow_dispatch`,
master only): `confirm_canary` (contents: write, actions: write) and
`smoke_canary_onboarding` (downloads the publisher's lockfile artifact
by run id). Inputs reach shell steps only through a validating step.
- `.github/workflows/release.yml`: `publish_canary` hands off to
`canary-finalize.yml` and drops to `contents: read`; its tag push,
Docker dispatch and smoke job move to `canary-finalize.yml`.
- Tests and `doc/RELEASING.md`.

## Verification

- `node --test .github/scripts/tests/*.test.mjs
scripts/release-lib.test.mjs
scripts/verify-release-registry-state.test.mjs
scripts/verify-canary-provenance.test.mjs`: 846 pass.
`scripts/__tests__/release-verify-workflow.test.mjs`: 73 pass.
- New tests: numbering by master position, rerun moves past a published
number, shallow refusal and dry-run fallback; newer/older/foreign
dist-tags; provenance from another commit, workflow, branch, repository
or package is rejected; finalize wiring, permissions and input handling.
- Real npm data: `verify-canary-provenance.mjs 2026.1116.0-canary.7
b4fa1c7f... paperclipai @paperclipai/shared @paperclipai/server` passes;
the same version with another commit fails.
- `confirm-canary-publication.sh 2026.1116.0-canary.7 b4fa1c7f...` from
a checkout of that commit ran all steps against npm in 61 s and ended
with "canary/v2026.1116.0-canary.7 already points at b4fa1c7f...",
pushing nothing.
- `release.sh canary --dry-run --defer-registry-confirmation` and
`release.sh nightly --defer-registry-confirmation` both stop with "only
applies to a published canary".
- `actionlint` reports no new findings compared with master.
- Not verified before merge: a full publish through
`canary-finalize.yml`. It can run only from master. The first canary
after merge is the check.

## Risks

- Canary numbers jump from `canary.7` to about `canary.8144` (master has
8,143 commits). This is valid semver and still higher than every earlier
canary; users install through the `canary` dist-tag.
- The `canary/v` tag, Docker image and smoke now finish after the
release run ends. Nightly selects only tagged canaries, as before. If
finalize fails, that canary is on npm without a tag, so nightly skips
it, as it skips any canary whose old in-run verification failed.
- A canary can now publish while the previous one is still being
confirmed. Publishing itself is still serialized, so the `canary`
dist-tag only moves forward.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 11:37:20 -06:00

74 lines
4.9 KiB
JavaScript

import assert from "node:assert/strict";
import test from "node:test";
import { packageProvenance, verifyCanaryProvenance } from "./verify-canary-provenance.mjs";
const sha = "b".repeat(40);
const version = "2026.1116.0-canary.2180000";
function attestation({ name = "%40paperclipai/shared", commit = sha, path = ".github/workflows/release.yml", ref = "refs/heads/master",
repository = "https://github.com/paperclipai/paperclip", predicateType = "https://slsa.dev/provenance/v1" } = {}) {
const statement = { subject: [{ name: `pkg:npm/${name}@${version}` }],
predicate: { buildDefinition: { externalParameters: { workflow: { repository, path, ref } }, resolvedDependencies: [{ digest: { gitCommit: commit } }] } } };
return { attestations: [{ predicateType, bundle: { dsseEnvelope: { payload: Buffer.from(JSON.stringify(statement)).toString("base64") } } }] };
}
const check = (doc, packageName = "@paperclipai/shared") => packageProvenance(doc, { packageName, version, sourceSha: sha });
const respond = (body, status = 200) => new Response(JSON.stringify(body), { status });
const noSleep = async () => {};
const registry = (byPackage) => async (url) => {
const name = decodeURIComponent(new URL(url).pathname.split("/attestations/")[1]).replace(/@[^@]+$/, "");
const entry = byPackage[name];
return typeof entry === "function" ? entry() : entry ?? respond({}, 404);
};
test("provenance from master's release workflow at the source commit matches, scoped or not", () => {
assert.equal(check(attestation()), "match");
assert.equal(check(attestation({ name: "paperclipai" }), "paperclipai"), "match");
});
test("provenance from another commit, workflow, branch, repository or package is a mismatch", () => {
for (const overrides of [{ commit: "c".repeat(40) }, { path: ".github/workflows/pr.yml" }, { ref: "refs/heads/feature" },
{ repository: "https://github.com/someone/paperclip" }, { name: "%40paperclipai/server" }]) {
assert.equal(check(attestation(overrides)), "mismatch", JSON.stringify(overrides));
}
});
test("a package published through the transparency-log retry has no provenance, which is allowed beside a match", async () => {
assert.equal(check({ attestations: [] }), "none");
assert.equal(check(attestation({ predicateType: "https://github.com/npm/attestation/tree/main/specs/publish/v0.1" })), "none");
const result = await verifyCanaryProvenance({ packages: ["@paperclipai/shared", "paperclipai"], version, sourceSha: sha, sleep: noSleep,
fetcher: registry({ paperclipai: respond(attestation({ name: "paperclipai" })) }) });
assert.deepEqual(result, { problems: [], matched: 1, withoutProvenance: ["@paperclipai/shared"] });
});
test("packages that all took the retry report no match and no problem, and any mismatch is a problem", async () => {
const none = await verifyCanaryProvenance({ packages: ["@paperclipai/shared", "paperclipai"], version, sourceSha: sha, sleep: noSleep, fetcher: registry({}) });
assert.deepEqual(none, { problems: [], matched: 0, withoutProvenance: ["@paperclipai/shared", "paperclipai"] });
const mixed = await verifyCanaryProvenance({ packages: ["@paperclipai/shared", "paperclipai"], version, sourceSha: sha, sleep: noSleep,
fetcher: registry({ "@paperclipai/shared": respond(attestation()), paperclipai: respond(attestation({ name: "paperclipai", commit: "c".repeat(40) })) }) });
assert.equal(mixed.problems.length, 1);
assert.match(mixed.problems[0], /^paperclipai@.*provenance does not name/);
});
test("a brief npm failure is retried, and a lasting or permanent one is a problem", async () => {
let calls = 0;
const flaky = await verifyCanaryProvenance({ packages: ["@paperclipai/shared"], version, sourceSha: sha, sleep: noSleep,
fetcher: registry({ "@paperclipai/shared": () => (++calls < 3 ? respond({}, 503) : respond(attestation())) }) });
assert.deepEqual(flaky.problems, []);
assert.equal(calls, 3);
const lasting = await verifyCanaryProvenance({ packages: ["@paperclipai/shared"], version, sourceSha: sha, sleep: noSleep, attempts: 3,
fetcher: async () => { throw new Error("socket hang up"); } });
assert.match(lasting.problems[0], /attestations unavailable \(socket hang up\)/);
let forbidden = 0;
const permanent = await verifyCanaryProvenance({ packages: ["@paperclipai/shared"], version, sourceSha: sha, sleep: noSleep,
fetcher: async () => { forbidden += 1; return respond({}, 403); } });
assert.match(permanent.problems[0], /HTTP 403/);
assert.equal(forbidden, 1);
});
test("inputs are validated before any request", async () => {
const fetcher = async () => { throw new Error("must not fetch"); };
await assert.rejects(verifyCanaryProvenance({ packages: ["paperclipai"], version: "2026.1116.0-nightly.1", sourceSha: sha, fetcher }), /not a canary version/);
await assert.rejects(verifyCanaryProvenance({ packages: ["paperclipai"], version, sourceSha: "abc", fetcher }), /not a full commit SHA/);
});