From 1ac48aff8dc13926539b085ca2e1d67913bcf7dc Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 11 Oct 2026 12:02:19 -0700 Subject: [PATCH] perf(ci): pack the verifier's staged packages concurrently (#15888) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the `Build` check. Its last step is the public npm install verifier, `scripts/verify-grok-npm-install.mjs`, which stages the eighteen public packages, packs them, and installs them in a clean consumer. > - The verifier packs the eighteen packages one after another with `npm pack`. Each pack is a Node start-up plus gzip of that package's files. The timing line added in #15868 shows this loop at 35.6-37.9s on four `Build` runs, the largest part of the staging phase now that the bundled installs overlap. > - `Build` at 433-530s is the slowest check in most PR runs once the server shards drop, so its staging phase sets the wall-clock time of the run. > - Each pack reads only its own `package-` directory and writes one tarball. The runner has four cores and the packs are CPU-bound. There is no reason to run them one after the other. > - This pull request runs the packs as concurrent children through the prefetch helper, one per core, bundled packages first, each into its own directory, and puts the results back into dependency order before every tarball is checked. > - The benefit is a shorter `Build` job, and a timing line that reports the packs' serial cost next to their wall time, so the saving is measured on this PR's own run instead of estimated. ## Linked Issues or Issue Description Refs #15868, #15861. This branch is stacked on the branch of #15868, which is stacked on #15861, because the same execution branch holds both open PRs. The diff here is this PR's one commit. After #15868 merges, this PR is retargeted to `master` and rebased. **What existing behavior does this improve?** The `Build` PR check. Its public npm install verifier packs the eighteen public packages in one serial loop. **Subsystem affected** CI and release scripts (`scripts/verify-grok-npm-install.mjs`, `scripts/prefetch-child.mjs`). **Current behavior** After the bundled installs are joined, the verifier runs `npm pack --pack-destination ` for each of the eighteen packages in turn, finds the new tarball by listing the shared root, and checks it with `tar -tzf`. `stage_pack_ms` was 35.8s, 37.1s, 35.6s and 37.9s on the four runs of #15868. The parent is idle apart from the one pack it waits on, and three of the four cores are idle. **Proposed behavior** Start the packs as children through `startPrefetch`, at most one per core (`os.availableParallelism()`), bundled packages first because they make the largest tarballs. Each child packs into its own `pack-` directory, so the tarball a package produced is read from there instead of guessed from a shared directory. Register each child for cleanup as it starts. Put the results back into dependency order, then check every tarball with `tar -tzf` as before, then copy them into the assets directory as before. Report `stage_pack_check_ms`, `pack_concurrency`, `pack_serial_ms` and `pack_longest_ms` on the timing line, and keep `stage_pack_ms` as the whole step. **Reason and benefit** With four slots the step takes about the longest single pack or a quarter of the serial sum, whichever is larger. The expected saving is 20-25s of the 36-38s, measured on this PR's own `Build` job by `pack_serial_ms` against `stage_pack_ms`. **Measured on this PR's first run ([38146108750](https://github.com/paperclipai/paperclip/actions/runs/38146108750), `Build` 433s, verifier 214s):** `stage_pack_ms` 29.9s against 35.6-37.9s before, so the step shed 6-8s, not the 20-25s estimated. `pack_serial_ms` is 44.4s with `pack_concurrency` 4, and `pack_longest_ms` is 28.3s: one pack (the server with its 513 MB bundled closure, gzip level 9 inside npm) runs for almost the whole step, so the step is bounded by that pack and not by the slot count. The seventeen other packs fit inside it. The serial sum is 7-9s higher than the old serial step, which is the four processes contending for the runner's cores. `stage_packages_ms` 75.8s against 80.4-89.6s on #15868's runs; `image_wait_ms` and `grok_wait_ms` stay 0. **Breaking changes** None. The same packages are packed with the same `npm pack` arguments, the tarballs are checked the same way, and the consumer receives them in the same order. ## What Changed - `scripts/pack-staged-packages.mjs` (new): `packStagedPackages` launches the packs in the order given with at most `concurrency` children at once, through the prefetch helper, and resolves with each entry's tarball path and wall time in the same order. Every child is passed to `register` as it starts. The first failure rejects with the tail of that child's log and nothing further is launched. A child that leaves anything other than one tarball in its directory is a failure. `packConcurrency` is one per core. `packTimings` gives the timing-line keys. - `scripts/verify-grok-npm-install.mjs`: each entry gets its `index` and its own `pack-` destination. The pack loop is replaced by one `packStagedPackages` call, launched bundled-first and registered into the `prefetches` list as each child starts, so an earlier failure stops any pack still running before the scratch root is removed. The results are sorted back into dependency order, every tarball is checked in that order, and the copy into the assets directory is unchanged. Only the lifecycle sentinel is still packed in-process. - Timing line: `stage_pack_check_ms`, `pack_concurrency`, `pack_serial_ms` and `pack_longest_ms` are added. `stage_pack_ms` still covers the whole step, so the before and after numbers compare directly. - `scripts/__tests__/pack-staged-packages.test.mjs` (new): five tests with a local stand-in for npm and no network. The default concurrency and the pack arguments; four packs with two slots run the two slow packs together, start a third only once a slot is free, receive the exact `npm pack` argument list, register every child, and come back in entry order with their tarballs and timing keys although the fast packs finish first; a failed pack rejects with its log tail, launches nothing more, and leaves the running pack to be stopped; a pack that produces two tarballs or none is a failure; and no entries resolve at once. - `scripts/__tests__/grok-public-install-sandbox.test.mjs`: the guard test on the verifier source now checks the import, the bundled-first launch with cleanup registration, the reorder before the checks, the checks before the copy, that only the sentinel is packed in-process, that each pack has its own directory, and the new timing keys. - `package.json`: the new test file is in `test:release-registry`. ## Verification - `node --test scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/prefetch-child.test.mjs`: 35 tests pass (30 on #15868's head plus the five new ones). The new file passes three consecutive runs. - `node --check` passes on both scripts. - With the real npm 11.12.1 on Node 25.9: two small packages packed through `packStagedPackages` with two slots produce one tarball each in their own `pack-` directory, with the expected `package/` listing from `tar -tzf`. - The verifier itself needs Docker and a network, which this workspace does not have. This PR's own `Build` job runs the branch's verifier; its timing line carries `stage_pack_ms`, `pack_serial_ms`, `pack_longest_ms` and `pack_concurrency`, which is the measurement. Before, on #15868's four runs: `stage_pack_ms` 35.8s / 37.1s / 35.6s / 37.9s. - `docker-runner-check.yml` runs the same script after `pnpm build` and needs no change. ## Risks - Four `npm pack` processes run at once and share the host npm configuration and cache. `npm pack --ignore-scripts` reads the package directory and writes one tarball; it does not write to the cache beyond its per-process log file. If a shared-state race appears, the failing pack's log tail is in the error and the fix is to lower the concurrency. - Memory and CPU: four Node processes instead of one on the four-core `Build` runner, for about thirty seconds. The largest pack (the server with its bundled dependencies) is the same process as before. - The pack output moves from the CI log into per-pack log files in the scratch root. A failure reports the last 2000 characters of the failing pack's log; a successful pack's output is discarded with the scratch root. - The tarball checks (`tar -tzf`) now run after all packs instead of after each pack. A bundled Codex payload is still rejected before any tarball reaches the consumer; the only change is that the error arrives a few seconds later. - Low risk otherwise. The bundled installs, the consumer install, and every probe are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.1 (Anthropic), model ID `claude-fable-5-1`, run as a Paperclip agent through the Claude Code local adapter with tool use (shell, file edits, GitHub CLI). Reasoning mode: extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (run 38146108750: `server (11/12)` failed on its first attempt on a teardown `EACCES` while removing the per-worker `PAPERCLIP_HOME` after all 74 files and 1257 tests had passed, in a file this PR does not touch; the re-run of that job passed and every check is green) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 --- package.json | 2 +- .../grok-public-install-sandbox.test.mjs | 23 ++-- .../__tests__/pack-staged-packages.test.mjs | 107 ++++++++++++++++++ scripts/pack-staged-packages.mjs | 64 +++++++++++ scripts/verify-grok-npm-install.mjs | 31 +++-- 5 files changed, 207 insertions(+), 20 deletions(-) create mode 100644 scripts/__tests__/pack-staged-packages.test.mjs create mode 100644 scripts/pack-staged-packages.mjs diff --git a/package.json b/package.json index 29b999b28e..1d8f7fb078 100644 --- a/package.json +++ b/package.json @@ -61,7 +61,7 @@ "smoke:posthog-live": "node scripts/smoke/posthog-live.mjs", "smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh", "smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs", - "test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", + "test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", "storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs", "test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts", "test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack", diff --git a/scripts/__tests__/grok-public-install-sandbox.test.mjs b/scripts/__tests__/grok-public-install-sandbox.test.mjs index 11fde52433..3cf43712a1 100644 --- a/scripts/__tests__/grok-public-install-sandbox.test.mjs +++ b/scripts/__tests__/grok-public-install-sandbox.test.mjs @@ -120,19 +120,28 @@ test('the bundled-dependency installs run together while the other packages are // Only the bundled packages' sources are staged before the installs start; // the installs are registered for cleanup at once, before anything else can // fail; the other packages are copied while the installs run; the join comes - // before the first pack; and the pack loop walks every package in order. + // before the packs start; and the packs run through the helper covered by + // pack-staged-packages.test.mjs, bundled packages first, each registered for + // cleanup as it starts, with the results put back into dependency order + // before every tarball is checked. const bundledSources = position('for (const entry of entries.filter(entry => entry.bundled)) stageSource(entry)'); const start = position("startBundledInstalls(entries.filter(entry => entry.bundled), { logDir: prefetched, cwd: repo })"); const registered = position('prefetches.push(...installs.map(({ handle }) => handle))'); const otherSources = position('for (const entry of entries.filter(entry => !entry.bundled))'); const join = position("await awaited('bundled', Promise.all(installs.map(({ handle }) => handle.done)))"); - const packLoop = position('for (const { target } of entries)'); - const firstPack = position("run('npm', ['pack', '--ignore-scripts', '--pack-destination', root], target)"); - assert.ok(bundledSources < start && start < registered && registered < otherSources && otherSources < join && join < packLoop && packLoop < firstPack); - assert.equal(source.match(/run\('npm', \['pack'/g).length, 2, 'one package pack loop plus the lifecycle sentinel'); - // The timing line reports the split staging phase and each install. - for (const key of ['stage_sources_ms', 'stage_bundled_ms', 'stage_pack_ms']) assert.ok(source.includes(`timing.${key} =`)); + assert.ok(source.includes("import { packConcurrency, packStagedPackages, packTimings } from './pack-staged-packages.mjs'")); + const packs = position("await packStagedPackages([...entries].sort((a, b) => Number(b.bundled) - Number(a.bundled)), { logDir: prefetched, env, concurrency, register: handle => prefetches.push(handle) })"); + const reordered = position('packs.sort((a, b) => a.index - b.index)'); + const checks = position("for (const tarball of tarballs) assertNoBundledCodexPayloads(run('tar', ['-tzf', tarball])"); + const copied = position('for (const tarball of tarballs) {'); + assert.ok(bundledSources < start && start < registered && registered < otherSources && otherSources < join && join < packs && packs < reordered && reordered < checks && checks < copied); + assert.equal(source.match(/run\('npm', \['pack'/g).length, 1, 'only the lifecycle sentinel is packed in-process'); + assert.ok(source.includes("destination: join(root, `pack-${index}`)"), 'each pack writes into its own directory'); + // The timing line reports the split staging phase, each install, and the + // pack step with what the packs would have cost one after another. + for (const key of ['stage_sources_ms', 'stage_bundled_ms', 'stage_pack_ms', 'stage_pack_check_ms']) assert.ok(source.includes(`timing.${key} =`)); assert.ok(source.includes('...bundledInstallTimings(installs)')); + assert.ok(source.includes('Object.assign(timing, packTimings(packs, concurrency))')); }); test('the installed Codex probe exercises the public export and rejects incomplete or mismatched packages', () => { diff --git a/scripts/__tests__/pack-staged-packages.test.mjs b/scripts/__tests__/pack-staged-packages.test.mjs new file mode 100644 index 0000000000..09c19cf64a --- /dev/null +++ b/scripts/__tests__/pack-staged-packages.test.mjs @@ -0,0 +1,107 @@ +import assert from 'node:assert/strict'; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import test from 'node:test'; +import { stopPrefetches } from '../prefetch-child.mjs'; +import { PACK_ARGS, packConcurrency, packStagedPackages, packTimings } from '../pack-staged-packages.mjs'; + +// A small local script stands in for npm; no test here needs npm or a network. +// It is run as `node