From 52d120f68d6e6a34b3d95002e677733c3951d0c2 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Mon, 14 Sep 2026 20:16:29 -0700 Subject: [PATCH] fix(release): wait 30 minutes for npm to expose a published version (#13436) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every release publishes a batch of npm packages and then waits for each one to become visible before continuing > - npm accepts a publish immediately but exposes it later, so that wait exists to keep a release from continuing past a package nobody can install yet > - Today that wait was too short twice in a row, and each timeout aborted the release with the batch half-published > - Version numbers are derived from what is already on npm, so every retry moves to a new number and meets the same lag > - Two canary attempts burned two versions this way and shipped nothing > - This pull request raises the per-package budget from ten minutes to thirty > - The benefit is that ordinary registry lag costs waiting instead of a failed, half-published release ## Linked Issues or Issue Description No existing issue. The problem, in the bug report format: **What happened** `publish_canary` failed twice in a row with the batch half-published: ``` Warning: npm accepted @paperclipai/server@2026.914.0-canary.2, but the version did not become registry-visible. Error: stopping release: npm did not publish and expose @paperclipai/server@2026.914.0-canary.2 ``` The version was accepted at 17:49:09 and became visible at 18:04:29 — about five minutes after the poll gave up. `shared`, `db` and `adapter-utils` published at that version; `server`, `paperclip-runner` and the root package did not. **Expected behavior** Ordinary registry propagation delay costs the release some waiting, not a failure. A package that becomes visible after 15 minutes must not abort the batch because of the old 10-minute wait. Longer registry outages can still leave a partial batch. **Steps to reproduce** 1. Publish any channel while npm is propagating slowly. 2. A package takes longer than `NPM_PUBLISH_VERIFY_ATTEMPTS * NPM_PUBLISH_VERIFY_DELAY_SECONDS` to become visible. 3. The release aborts, that version is half-published, and the retry picks a new version number and meets the same lag. **Paperclip version or commit** Present on master. Observed on 2026-09-14 across canary runs in workflow run 34869494325. ## What Changed - Increase npm visibility checks from 60 to 180, retaining the 10-second delay: about 30 minutes per package. - Increase canary, nightly, beta, and stable publish job timeouts from 90 to 150 minutes. - Add offline regression tests using the workflow's actual settings. They cover the observed 15-minute 20-second delay, immediate visibility, exhausted retries, and job timeout sizing. - Load the access router in test setup so its cold transform does not consume the first permission test's 10-second timeout. The permission assertions are unchanged. ## Verification - `node --test scripts/release-lib.test.mjs`: 14 passed. - `pnpm run test:release-registry`: 129 passed. - `pnpm exec vitest run server/src/__tests__/access-routes-permissions-upgrade.test.ts`: 3 passed. - Regression proof in temporary fixtures: restoring 60 attempts fails the observed-delay test; restoring 90-minute jobs fails the timeout-budget test. - [CI run 34916632804](https://github.com/paperclipai/paperclip/actions/runs/34916632804): all jobs passed, including typecheck/release registry, build, all general and serialized server shards, browser tests, runner verification, and canary dry run. The PR has 31 successful checks and two expected Storybook skips at `a27f5e896`. - [Previously failing serialized shard](https://github.com/paperclipai/paperclip/actions/runs/34916632804/job/104215809616): all three access-route permission tests passed in CI after preloading the router. - Greptile's final review is 5/5 with no outstanding findings. Both review threads are resolved. - Local limits: `pnpm -r typecheck` and `pnpm build` stop at the runner package because this machine has no Rust `cargo` executable. The duplicate full local `pnpm test:run` was interrupted while the complete CI matrix ran. Targeted local results are listed above; full validation is from CI. The previous CI failures were unrelated to npm propagation: - [PR review](https://github.com/paperclipai/paperclip/actions/runs/34891770396) required a test file for this fix. - [Serialized server shard 3](https://github.com/paperclipai/paperclip/actions/runs/34891773736/job/104136392196) timed out in the first access-route permission test at 10 seconds. The other two tests in that file passed. - The canary dry run passed in that same CI run. ## Risks Low risk. Production behavior changes only in release waiting budgets. - Polling exits as soon as npm exposes the version, so healthy publishes do not wait longer. - An unavailable version now takes about 30 minutes to report. Polls remain bounded and still fail the release on exhaustion. - The 150-minute jobs leave roughly 30 minutes for setup/build plus four full polling windows. npm command runtime and later release steps also consume that budget; a broader outage can still interrupt a batch. - The permission-test change moves module loading into a bounded setup hook; it does not relax authorization assertions. - No schema changes or operational migrations. ## Model Used - Claude Fable 5 (`claude-fable-5`), 1M context, extended thinking, run through Claude Code with tool use and code execution. - OpenAI GPT-6 (Codex), with reasoning, tool use, and code execution, for the CI follow-up. Context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted checks listed above; full validation passed in CI - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes — workflow comments and verification details - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/workflows/release.yml | 24 ++++- scripts/release-lib.test.mjs | 95 ++++++++++++++++++- .../access-routes-permissions-upgrade.test.ts | 6 ++ 3 files changed, 116 insertions(+), 9 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a11e7967fc..878899cb24 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -71,7 +71,21 @@ env: # 07:07:40.) The publish jobs' timeout-minutes are sized for several # laggard packages; if most of a batch lags the full budget, npm is having # a real incident and the job failing is correct. - NPM_PUBLISH_VERIFY_ATTEMPTS: "60" + # + # Raised to 30 minutes on 2026-09-14, when 10 was not enough twice in a + # row: @paperclipai/server was accepted at 17:49:09 and became visible at + # 18:04:29 — five minutes after the poll gave up. Each timeout aborts the + # release mid-batch, leaving that version half-published, and because the + # next version number is derived from what is already on npm the retry + # moves to a new number and meets the same lag. Waiting is cheap; a + # half-published release is not. + # + # Packages are published and verified one at a time, so the publish jobs' + # timeout-minutes went to 150 alongside this: a 30-minute build plus four + # packages each lagging the full budget still finishes inside the job, + # instead of the job timing out mid-batch and leaving the same + # half-published state this budget exists to avoid. + NPM_PUBLISH_VERIFY_ATTEMPTS: "180" NPM_PUBLISH_VERIFY_DELAY_SECONDS: "10" jobs: @@ -326,7 +340,7 @@ jobs: if: github.event_name == 'push' needs: verify_canary runs-on: ubuntu-latest - timeout-minutes: 90 + timeout-minutes: 150 environment: npm-canary outputs: canary_version: ${{ steps.canary_tag.outputs.version }} @@ -582,7 +596,7 @@ jobs: (needs.smoke_nightly.result == 'success' || (needs.smoke_nightly.result == 'skipped' && github.event_name == 'workflow_dispatch' && inputs.dry_run)) runs-on: ubuntu-latest - timeout-minutes: 90 + timeout-minutes: 150 environment: npm-canary # The workflow-level concurrency group is per event, so a forced dispatch # nightly could otherwise overlap the scheduled one and race it to the @@ -848,7 +862,7 @@ jobs: (needs.verify_beta_candidate.result == 'success' || (needs.verify_beta_candidate.result == 'skipped' && needs.select_beta.outputs.mode == 'promote')) runs-on: ubuntu-latest - timeout-minutes: 90 + timeout-minutes: 150 environment: npm-beta # Serialize beta publishes so two dispatches cannot race to the same next # -beta.N version; release.sh additionally refuses to double-publish a @@ -1271,7 +1285,7 @@ jobs: if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' && !inputs.dry_run needs: [preflight_stable, verify_stable] runs-on: ubuntu-latest - timeout-minutes: 90 + timeout-minutes: 150 environment: npm-stable permissions: contents: write diff --git a/scripts/release-lib.test.mjs b/scripts/release-lib.test.mjs index cc51aa7b15..4d8da63ab2 100644 --- a/scripts/release-lib.test.mjs +++ b/scripts/release-lib.test.mjs @@ -6,6 +6,14 @@ import { join } from "node:path"; import test from "node:test"; const repoRoot = new URL("..", import.meta.url).pathname.replace(/\/$/, ""); +const releaseWorkflow = readFileSync(new URL("../.github/workflows/release.yml", import.meta.url), "utf8"); + +function workflowVerifyBudget() { + return { + verifyAttempts: Number(releaseWorkflow.match(/^ NPM_PUBLISH_VERIFY_ATTEMPTS: "(\d+)"$/m)?.[1]), + verifyDelaySeconds: Number(releaseWorkflow.match(/^ NPM_PUBLISH_VERIFY_DELAY_SECONDS: "(\d+)"$/m)?.[1]), + }; +} function writeExecutable(path, body) { writeFileSync(path, body, { mode: 0o755 }); @@ -18,6 +26,9 @@ function runPublishHelper({ callerPipefail = true, publishTool = "pnpm", waitForRegistry = false, + npmVersionExistsAfterChecks = 0, + verifyAttempts = 1, + verifyDelaySeconds = 0, }) { const fixtureDir = mkdtempSync(join(tmpdir(), "paperclip-release-lib-")); const binDir = join(fixtureDir, "bin"); @@ -74,9 +85,18 @@ exit 1 `#!/usr/bin/env bash set -euo pipefail printf 'npm %s\\n' "$*" >> "$FAKE_CALL_LOG" -if [ "$1" = "view" ] && [ "$NPM_VERSION_EXISTS" = "true" ]; then - echo "1.2.3" - exit 0 +if [ "$1" = "view" ]; then + checks=0 + if [ -f "$FAKE_STATE_DIR/view-checks" ]; then + read -r checks < "$FAKE_STATE_DIR/view-checks" + fi + checks=$((checks + 1)) + echo "$checks" > "$FAKE_STATE_DIR/view-checks" + if [ "$NPM_VERSION_EXISTS" = "true" ] || + { [ "$NPM_VERSION_EXISTS_AFTER_CHECKS" -gt 0 ] && [ "$checks" -ge "$NPM_VERSION_EXISTS_AFTER_CHECKS" ]; }; then + echo "1.2.3" + exit 0 + fi fi if [ "$1" = "publish" ]; then case "$PNPM_MODE" in @@ -130,9 +150,11 @@ exec npm "$@" const script = ` ${shellOptions} source "${repoRoot}/scripts/release-lib.sh" +# Record virtual waiting so registry-delay tests remain fast and offline. +sleep() { printf 'sleep %s\\n' "$*" >> "$FAKE_CALL_LOG"; } ${ waitForRegistry - ? `publish_package_to_npm_and_wait ${distTag} @paperclipai/example 1.2.3 ${publishTool} 1 0` + ? `publish_package_to_npm_and_wait ${distTag} @paperclipai/example 1.2.3 ${publishTool} "$VERIFY_ATTEMPTS" "$VERIFY_DELAY_SECONDS"` : `publish_package_to_npm ${distTag} @paperclipai/example 1.2.3 ${publishTool}` } `; @@ -149,6 +171,9 @@ ${ FAKE_CALL_LOG: callLog, FAKE_STATE_DIR: stateDir, NPM_VERSION_EXISTS: npmVersionExists ? "true" : "false", + NPM_VERSION_EXISTS_AFTER_CHECKS: String(npmVersionExistsAfterChecks), + VERIFY_ATTEMPTS: String(verifyAttempts), + VERIFY_DELAY_SECONDS: String(verifyDelaySeconds), PNPM_MODE: pnpmMode, REPO_ROOT: fixtureDir, }, @@ -264,3 +289,65 @@ test("publish_package_to_npm_and_wait blocks the release when registry visibilit assert.match(result.calls, /^npm view @paperclipai\/example@1\.2\.3 version$/m); assert.match(result.output, /did not become registry-visible/); }); + +test("the workflow budget tolerates the observed 15-minute 20-second registry delay", () => { + const budget = workflowVerifyBudget(); + assert.ok(budget.verifyDelaySeconds > 0); + const delayedCheck = Math.ceil((15 * 60 + 20) / budget.verifyDelaySeconds) + 1; + const result = runPublishHelper({ + pnpmMode: "success", + waitForRegistry: true, + npmVersionExistsAfterChecks: delayedCheck, + ...budget, + }); + + assert.equal(result.status, 0, result.output); + assert.equal(result.calls.match(/^pnpm publish /gm)?.length, 1); + assert.equal(result.calls.match(/^npm view /gm)?.length, delayedCheck); + assert.deepEqual( + result.calls.split("\n").filter((call) => call.startsWith("sleep ")), + Array(delayedCheck - 1).fill(`sleep ${budget.verifyDelaySeconds}`), + ); +}); + +test("the workflow budget does not delay an immediately visible publish", () => { + const result = runPublishHelper({ + pnpmMode: "success", + npmVersionExists: true, + waitForRegistry: true, + ...workflowVerifyBudget(), + }); + + assert.equal(result.status, 0, result.output); + assert.equal(result.calls.match(/^npm view /gm)?.length, 1); + assert.doesNotMatch(result.calls, /^sleep /m); +}); + +test("the workflow budget fails closed after the last registry check", () => { + const budget = workflowVerifyBudget(); + assert.ok(budget.verifyAttempts > 0); + const result = runPublishHelper({ + pnpmMode: "success", + waitForRegistry: true, + npmVersionExistsAfterChecks: budget.verifyAttempts + 1, + ...budget, + }); + + assert.notEqual(result.status, 0); + assert.match(result.output, /did not become registry-visible/); + assert.equal(result.calls.match(/^pnpm publish /gm)?.length, 1); + assert.equal(result.calls.match(/^npm view /gm)?.length, budget.verifyAttempts); + assert.equal(result.calls.match(/^sleep /gm)?.length, budget.verifyAttempts - 1); +}); + +test("every publish job budgets for build time and four delayed packages", () => { + const { verifyAttempts, verifyDelaySeconds } = workflowVerifyBudget(); + const pollingSeconds = (verifyAttempts - 1) * verifyDelaySeconds; + const requiredSeconds = 30 * 60 + 4 * pollingSeconds; + + for (const job of ["publish_canary", "publish_nightly", "publish_beta", "publish_stable"]) { + const body = releaseWorkflow.split(`\n ${job}:\n`)[1]?.split(/\n [a-z_]+:\n/)[0] ?? ""; + const timeoutMinutes = Number(body.match(/^ timeout-minutes: (\d+)$/m)?.[1]); + assert.ok(timeoutMinutes * 60 > requiredSeconds, `${job} must leave time beyond build and polling`); + } +}); diff --git a/server/src/__tests__/access-routes-permissions-upgrade.test.ts b/server/src/__tests__/access-routes-permissions-upgrade.test.ts index 7f9d440cfe..799701e108 100644 --- a/server/src/__tests__/access-routes-permissions-upgrade.test.ts +++ b/server/src/__tests__/access-routes-permissions-upgrade.test.ts @@ -88,6 +88,12 @@ describeEmbeddedPostgres("access routes permissions upgrade compatibility", () = let db!: Db; let tempDb: Awaited> | null = null; + // Load the large router graph during setup so a cold CI transform does not + // consume the first permission assertion's timeout budget. + beforeAll(async () => { + await import("../routes/access.js"); + }, 30_000); + beforeAll(async () => { tempDb = await startEmbeddedPostgresTestDatabase("paperclip-access-routes-permissions-upgrade-"); db = createDb(tempDb.connectionString);