From b7f41ee51db5982f0b936f9365d41f13112def07 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Tue, 13 Oct 2026 17:18:44 -0700 Subject: [PATCH] perf(ci): warm the public-install npm cache during pnpm build (#16204) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each pull request runs the Trusted PR CI. The `ci / Build` lane is the longest check, or close to the longest, in each green run. > - On 2026-10-13, `ci / Build` took 415-446s in runs 38426120144, 38432086560 and 38423713537. Of that, the public-install verifier took 226-241s. > - In the verifier, the consumer `npm install` took 119-129s. It downloads the full public registry graph into an empty npm cache. > - Source manifests set that graph. Thus CI can download it before `pnpm build` makes any tarball. > - This pull request fills the npm cache in the background while the Rust restore, Evalbook and `pnpm build` run. Then the verifier reads that cache. > - The benefit is a shorter `ci / Build` lane. The verifier keeps all its checks. ## Linked Issues or Issue Description Refs #15213, #15849, #15868, #15888 (earlier changes that made this verifier faster). **Enhancement description** The `ci / Build` lane runs the public-install verifier after `pnpm build`. The verifier starts with an empty npm cache. Its consumer install then downloads the full public registry graph: 119-129s of a 226-241s step. **Proposed change** Download the same registry graph into an npm cache in the background, during the earlier part of the lane. Give that cache to the verifier. **Why it helps** The download moves off the critical path of the lane. A local test of the same 628-package install took 51s with a cold cache and 21s with a warm cache. ## What Changed - Add `scripts/public-install-closure.mjs`. It holds the one definition of the public package closure: `@paperclipai/server`, `paperclipai`, and each public workspace package that they reach through `workspace:` specs. It also gives the registry dependencies of that closure. - Add `scripts/warm-public-install-npm-cache.mjs`. It installs those registry dependencies into a temporary consumer, with scripts disabled, to fill `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE`. Then it removes the temporary consumer. - `scripts/verify-grok-npm-install.mjs` now gets its closure from the shared module. When `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE` is set, it uses that directory as the consumer npm cache. When the variable is not set, it uses a new empty cache, as before. `docker-runner-check.yml` and `release-smoke.yml` do not change. - `pr-trusted.yml` (Build job): a new step starts the warm-up in the background after `Install dependencies`. The verify step waits for the warm-up for a maximum of 180s and then gives the cache to the verifier. If the warm-up fails, the step writes a notice and the verifier continues as before. - Add `scripts/__tests__/public-install-closure.test.mjs` and add it to `test:release-registry`. Update the closure assertions in `grok-public-install-sandbox.test.mjs`. ## Verification - `node --test scripts/__tests__/public-install-closure.test.mjs`: 4/4 pass. - `node --test scripts/__tests__/grok-public-install-sandbox.test.mjs`: 24/24 pass. - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs`: all pass. - `release-verify-workflow.test.mjs`: the two "publisher lockfile" tests fail in my sandbox on clean master too, because they need pnpm. Two canary git-fixture tests timed out once under load. They pass when they run alone. - I ran the warm-up against current master: 62 direct dependencies, 628 packages, 51s cold. The same install with the warm cache took 21s. - Important: `pr.yml` calls `pr-trusted.yml@master`. Thus the CI of this pull request runs the refactored verifier on its default path (empty cache), but not the new warm-up steps. The warm-up steps start after merge. To measure them, compare the `consumer_install_ms` value in the `paperclip.public-npm-install.timing.v1` line of a Build job before and after merge. - Final review: all latest-head CI checks pass (54 successes, one neutral result, two intentional skips). Fresh Apex is 5/5 on `64d091b8b0c5359fa3dc0edb1fd7ef9d24294477` with no unresolved findings. The two relevant local script suites pass all 28 tests. Current master combines cleanly. ## Risks - Low risk. The verifier result does not change. npm checks each cached tarball against the integrity of its own resolution, and the consumer directory and lock start empty. - npm can use a cached registry packument that is a maximum of about 5 minutes old (registry `max-age`). This has the same effect as starting the verifier a few minutes earlier. - The warm-up runs at the same time as `pnpm build`. It uses network, disk and some CPU. In the measured runs, `pnpm build` took 76-96s. - If the warm-up is slow or fails, the verify step waits a maximum of 180s and then continues with what is in the cache. - The verifier receipt still reports `cleanNpmInstall: true`. The consumer install is clean. Only the download cache is shared. ## Expected time savings - `ci / Build`: an estimated 65-85s less, from 415-446s to about 340-370s. The consumer install goes from 119-129s to about 40-55s, from the local cold/warm ratio. - Full PR time: after this change, `ci / Build` is no longer the longest lane. The longest lanes become `General tests (server N/12)` and `Verify Paperclip Runner (vitest 1/2)` at about 405-496s. Thus the full PR time decreases only when Build is the critical path. That is approximately 0-30s in the runs above. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in a Paperclip agent run. OpenAI GPT-6 (`gpt-6`) through Codex assisted the final repository review and targeted script validation. This session does not expose an exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip --- .github/workflows/pr-trusted.yml | 34 +++++++++- package.json | 2 +- .../grok-public-install-sandbox.test.mjs | 18 ++---- .../__tests__/public-install-closure.test.mjs | 62 +++++++++++++++++++ scripts/public-install-closure.mjs | 49 +++++++++++++++ scripts/verify-grok-npm-install.mjs | 25 ++++---- scripts/warm-public-install-npm-cache.mjs | 37 +++++++++++ 7 files changed, 199 insertions(+), 28 deletions(-) create mode 100644 scripts/__tests__/public-install-closure.test.mjs create mode 100644 scripts/public-install-closure.mjs create mode 100755 scripts/warm-public-install-npm-cache.mjs diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index 65f72fa8c56..2ff512638af 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -788,6 +788,25 @@ jobs: pnpm install --frozen-lockfile fi + # The public-install verifier at the end of this lane downloads the whole + # public registry graph into an empty npm cache: its consumer install + # took 119-129s of the 226-241s step, more than half of this ~415-446s + # lane, in runs 38426120144, 38432086560 and 38423713537 (2026-10-13). + # That graph is fixed by source manifests, so this fetches it in the + # background while the Rust restore, Evalbook and `pnpm build` run, about + # 150s before the verifier installs anything. Only the cache is handed + # over, and the verify step waits for this first. A failed or partial + # warm-up leaves the verifier to fetch what is missing as before. + - name: Start warming the public-install npm cache + if: ${{ hashFiles('scripts/warm-public-install-npm-cache.mjs') != '' }} + run: | + ( + # The default `bash -e` would skip the status line on a failure. + set +e + PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache" node scripts/warm-public-install-npm-cache.mjs + echo $? > "$RUNNER_TEMP/public-install-npm-cache.status" + ) > "$RUNNER_TEMP/public-install-npm-cache.log" 2>&1 & + # pnpm build reaches the Runner package's build:binary step; without the # shared Rust cache that is a ~3m40s cold compile of all third-party # crates (run 35036001734, 2026-09-15). Same restore-only contract as @@ -914,7 +933,20 @@ jobs: # docker-runner-check.yml runs the same verifier after `pnpm build`. - name: Verify built-in Grok from a clean public npm install if: ${{ hashFiles('scripts/verify-grok-npm-install.mjs') != '' }} - run: node scripts/verify-grok-npm-install.mjs + run: | + status="$RUNNER_TEMP/public-install-npm-cache.status" + if [ -f "$RUNNER_TEMP/public-install-npm-cache.log" ]; then + # Normally finished long before this; never let it hold the lane. + for _ in $(seq 180); do [ -f "$status" ] && break; sleep 1; done + if [ "$(cat "$status" 2>/dev/null)" = 0 ]; then + tail -n 1 "$RUNNER_TEMP/public-install-npm-cache.log" + else + echo '::notice title=Public-install npm cache::warm-up did not finish cleanly; the verifier fetches what is missing' + tail -n 20 "$RUNNER_TEMP/public-install-npm-cache.log" + fi + export PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache" + fi + node scripts/verify-grok-npm-install.mjs verify_serialized_server: name: Verify serialized server suites (${{ matrix.shard_label }}) diff --git a/package.json b/package.json index df1a2323f53..66cf20785c7 100644 --- a/package.json +++ b/package.json @@ -61,7 +61,7 @@ "smoke:posthog-live": "node scripts/smoke/posthog-live.mjs", "smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh", "smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs", - "test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", + "test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", "storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs", "test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts", "test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack", diff --git a/scripts/__tests__/grok-public-install-sandbox.test.mjs b/scripts/__tests__/grok-public-install-sandbox.test.mjs index 8c43c72a133..a3cbfbd7f2b 100644 --- a/scripts/__tests__/grok-public-install-sandbox.test.mjs +++ b/scripts/__tests__/grok-public-install-sandbox.test.mjs @@ -9,6 +9,7 @@ import { fileURLToPath } from 'node:url'; import { isWorkspacePackage, parseWorkspaceEntries } from '../build-standalone-public-packages.mjs'; import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertInstalledCodexPlatformProvenance, assertInstalledNativeConsumerContinuity, assertNoBundledCodexPayloads, grokConsumerDockerArgs, installedCodexProbeSource, installedDaemonProbeSource, macNodeHeadersDirectory, macPublicInstallLifecycleProfile, macInstalledStartupProbeSource, runMacInitialInstallWithDiagnostics } from '../grok-public-install-sandbox.mjs'; import { verifyPiPackagedDaemon } from '../pi-public-install-probe.mjs'; +import { publicInstallClosure } from '../public-install-closure.mjs'; import { runnerBinaryTarget } from '../../packages/paperclip-runner/src/live/runner-binary.ts'; const paths = { assets: '/private/staging/assets', consumer: '/private/staging/consumer', cache: '/private/staging/cache', uid: 1001, gid: 1001 }; @@ -209,7 +210,7 @@ test('the image and Grok downloads start before staging and join without weakeni assert.ok(source.includes("const stagedGrok = join(prefetched, 'grok')")); assert.doesNotMatch(source, /provision-grok\.mjs'\), prerequisite\]/); // Both downloads start before any package is staged and before the first sandbox runs. - const staging = position("visit('@paperclipai/server')"); + const staging = position('publicInstallClosure(repo)'); const firstSandbox = position("isolated(['npm', 'install', '--ignore-scripts', '--omit=dev'"); assert.ok(imagePull < staging && grokProvision < staging); assert.ok(position("await awaited('image', imageReady)") < firstSandbox); @@ -284,17 +285,8 @@ test('the verifier stages only workspace packages, so it skips the standalone bu const retention = readFileSync(new URL('../retain-runner-qualification-packages.mjs', import.meta.url), 'utf8'); assert.ok(retention.includes("const pluginRelative = 'packages/plugins/sandbox-providers/daytona';") && retention.includes("cpSync(join(repo, pluginRelative, 'dist'), join(plugin, 'dist'), { recursive: true })"), 'retention still packs the Daytona plugin from its dist, which is why the gated build exists'); const repo = new URL('../../', import.meta.url); - const listing = spawnSync(process.execPath, [fileURLToPath(new URL('scripts/release-package-map.mjs', repo)), 'list'], { cwd: fileURLToPath(repo), encoding: 'utf8' }); - assert.equal(listing.status, 0, listing.stderr); - const packages = new Map(listing.stdout.trim().split('\n').map(line => line.split('\t')).map(([dir, name]) => [name, { dir, manifest: JSON.parse(readFileSync(new URL(`${dir}/package.json`, repo), 'utf8')) }])); - const needed = new Set(); - const visit = name => { - if (needed.has(name)) return; - const entry = packages.get(name); assert.ok(entry, `missing public workspace dependency ${name}`); needed.add(name); - for (const [dep, spec] of Object.entries({ ...entry.manifest.dependencies, ...entry.manifest.optionalDependencies })) if (spec.startsWith('workspace:')) visit(dep); - }; - assert.ok(source.includes("visit('@paperclipai/server');\n visit('paperclipai');")); - visit('@paperclipai/server'); visit('paperclipai'); + assert.ok(source.includes('const { packages, needed } = publicInstallClosure(repo);')); + const { packages, needed } = publicInstallClosure(fileURLToPath(repo)); const workspace = parseWorkspaceEntries(readFileSync(new URL('pnpm-workspace.yaml', repo), 'utf8')); const standalone = [...packages.values()].filter(({ dir }) => !isWorkspacePackage(dir, workspace)).map(({ dir }) => dir); assert.ok(standalone.length > 0, 'the public package map must still carry standalone packages for this check to mean anything'); @@ -307,7 +299,7 @@ test('the verifier stages only workspace packages, so it skips the standalone bu assert.deepEqual(JSON.parse(readFileSync(new URL('server/package.json', repo), 'utf8')).files, ['dist', 'ui-dist', 'skills']); assert.ok(source.includes("PAPERCLIP_RELEASE_REUSE_UI_DIST: '1'")); const uiDist = position("run('bash', [join(repo, 'scripts/prepare-server-ui-dist.sh')], repo)"); - assert.ok(position("visit('paperclipai')") < gate && gate < uiDist && uiDist < position('timing.stage_ui_dist_ms =') && uiDist < position('const stageSource = ')); + assert.ok(position('publicInstallClosure(repo)') < gate && gate < uiDist && uiDist < position('timing.stage_ui_dist_ms =') && uiDist < position('const stageSource = ')); }); test('the installed Codex probe exercises the public export and rejects incomplete or mismatched packages', () => { diff --git a/scripts/__tests__/public-install-closure.test.mjs b/scripts/__tests__/public-install-closure.test.mjs new file mode 100644 index 00000000000..9dd9ac2ddf5 --- /dev/null +++ b/scripts/__tests__/public-install-closure.test.mjs @@ -0,0 +1,62 @@ +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { readFileSync } from 'node:fs'; +import test from 'node:test'; +import { fileURLToPath } from 'node:url'; +import { PUBLIC_INSTALL_ROOTS, publicInstallClosure, publicRegistryDependencies } from '../public-install-closure.mjs'; + +const repo = fileURLToPath(new URL('../../', import.meta.url)); + +test('the closure starts at the server and the CLI and follows only workspace specs', () => { + assert.deepEqual(PUBLIC_INSTALL_ROOTS, ['@paperclipai/server', 'paperclipai']); + const { packages, needed } = publicInstallClosure(repo); + const order = [...needed]; + assert.equal(order[0], '@paperclipai/server'); + assert.ok(needed.has('paperclipai')); + for (const name of needed) { + const { manifest } = packages.get(name); + for (const [dep, spec] of Object.entries({ ...manifest.dependencies, ...manifest.optionalDependencies })) { + assert.equal(needed.has(dep), spec.startsWith('workspace:'), `${name} -> ${dep}@${spec}`); + } + } +}); + +test('the warm-up graph is the closure\'s registry dependencies, first spec wins', () => { + const pkg = (dependencies, optionalDependencies) => ({ dir: 'x', manifest: { dependencies, optionalDependencies } }); + const packages = new Map([ + ['a', pkg({ b: 'workspace:*', left: '^1.0.0', shared: '^2.0.0', local: 'file:../local', linked: 'link:../linked' }, { native: '1.2.3' })], + ['b', pkg({ shared: '^3.0.0', right: '~4.0.0' }, { left: '^9.0.0' })], + ['unvisited', pkg({ never: '1.0.0' })], + ]); + assert.deepEqual(publicRegistryDependencies({ packages, needed: new Set(['a', 'b']) }), { + dependencies: { left: '^1.0.0', shared: '^2.0.0', right: '~4.0.0' }, + optionalDependencies: { native: '1.2.3' }, + }); + const real = publicRegistryDependencies(publicInstallClosure(repo)); + const names = [...Object.keys(real.dependencies), ...Object.keys(real.optionalDependencies)]; + assert.ok(names.length > 0); + for (const spec of [...Object.values(real.dependencies), ...Object.values(real.optionalDependencies)]) { + assert.doesNotMatch(spec, /^(workspace|file|link):/); + } +}); + +test('the warm-up refuses to run without the cache the verifier will read', () => { + const env = { ...process.env }; + delete env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE; + const result = spawnSync(process.execPath, [fileURLToPath(new URL('../warm-public-install-npm-cache.mjs', import.meta.url))], { env, encoding: 'utf8' }); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE/); +}); + +test('the Build lane warms the cache before pnpm build and hands the same directory to the verifier', () => { + const workflow = readFileSync(new URL('../../.github/workflows/pr-trusted.yml', import.meta.url), 'utf8'); + const build = workflow.slice(workflow.indexOf('\n build:\n'), workflow.indexOf('\n verify_serialized_server:\n')); + const position = fragment => { const index = build.indexOf(fragment); assert.notEqual(index, -1, `Build job must contain ${fragment}`); return index; }; + const warm = position('PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache" node scripts/warm-public-install-npm-cache.mjs'); + assert.ok(position('- name: Install dependencies') < warm && warm < position('- name: Build\n'), 'the warm-up overlaps the Rust restore and pnpm build'); + assert.ok(build.includes('set +e\n'), 'a failed warm-up still records its status, so the verifier never waits it out'); + const verify = position('export PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache"'); + assert.ok(verify < position('node scripts/verify-grok-npm-install.mjs')); + const verifier = readFileSync(new URL('../verify-grok-npm-install.mjs', import.meta.url), 'utf8'); + assert.ok(verifier.includes("const cache = process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE ? resolve(process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE) : join(root, 'cache');")); +}); diff --git a/scripts/public-install-closure.mjs b/scripts/public-install-closure.mjs new file mode 100644 index 00000000000..f1e375f6404 --- /dev/null +++ b/scripts/public-install-closure.mjs @@ -0,0 +1,49 @@ +// The public packages the public-install verifier stages: `@paperclipai/server` +// and `paperclipai` plus every public workspace package they reach through +// `workspace:` specs, in the depth-first order the verifier packs them. The +// npm cache warm-up reads the same closure from source manifests before +// `pnpm build` has produced anything, so both share this one definition. +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; + +export const PUBLIC_INSTALL_ROOTS = ['@paperclipai/server', 'paperclipai']; + +// Returns `{ packages, needed }`: `packages` maps every public workspace +// package name to `{ dir, manifest }`, and `needed` is the closure in visit +// order. +export function publicInstallClosure(repo, roots = PUBLIC_INSTALL_ROOTS) { + const listing = execFileSync(process.execPath, [join(repo, 'scripts/release-package-map.mjs'), 'list'], { cwd: repo, stdio: 'pipe' }) + .toString().trim().split('\n').map(line => line.split('\t')); + const packages = new Map(listing.map(([dir, name]) => [name, { dir, manifest: JSON.parse(readFileSync(join(repo, dir, 'package.json'), 'utf8')) }])); + const needed = new Set(); + function visit(name) { + if (needed.has(name)) return; + const entry = packages.get(name); assert.ok(entry, `Missing public workspace dependency ${name}`); needed.add(name); + for (const [dep, spec] of Object.entries({ ...entry.manifest.dependencies, ...entry.manifest.optionalDependencies })) { + if (spec.startsWith('workspace:')) visit(dep); + } + } + for (const root of roots) visit(root); + return { packages, needed }; +} + +// The registry dependencies the closure's packages declare, as one consumer +// manifest. A name two packages declare with different specs keeps the first +// spec: the warm-up only fills a cache, and the real install fetches whatever +// it still lacks. +export function publicRegistryDependencies({ packages, needed }) { + const dependencies = {}; + const optionalDependencies = {}; + for (const name of needed) { + const { manifest } = packages.get(name); + for (const [section, target] of [['dependencies', dependencies], ['optionalDependencies', optionalDependencies]]) { + for (const [dep, spec] of Object.entries(manifest[section] ?? {})) { + if (needed.has(dep) || /^(workspace|file|link):/.test(spec) || dep in dependencies || dep in optionalDependencies) continue; + target[dep] = spec; + } + } + } + return { dependencies, optionalDependencies }; +} diff --git a/scripts/verify-grok-npm-install.mjs b/scripts/verify-grok-npm-install.mjs index f7af462f8a3..ea01fa01bb0 100644 --- a/scripts/verify-grok-npm-install.mjs +++ b/scripts/verify-grok-npm-install.mjs @@ -13,6 +13,7 @@ import { basename, dirname, join, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; import { packConcurrency, packStagedPackages, packTimings } from './pack-staged-packages.mjs'; import { materializePublishManifest } from './prepare-bundled-package.mjs'; +import { publicInstallClosure } from './public-install-closure.mjs'; import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertInstalledNativeConsumerContinuity, assertNoBundledCodexPayloads, grokConsumerDockerArgs, installedCodexProbeSource, installedDaemonProbeSource, macNodeHeadersDirectory, macPublicInstallLifecycleProfile, macInstalledStartupProbeSource, runMacInitialInstallWithDiagnostics } from './grok-public-install-sandbox.mjs'; import { startPrefetch, stopPrefetches } from './prefetch-child.mjs'; import { retainRunnerQualificationPackages } from './retain-runner-qualification-packages.mjs'; @@ -229,18 +230,10 @@ try { // Provision as the unprivileged verification user, never into the host's /opt. // Only the positive probe sees this file at the canonical sandbox path. const grokReady = prefetch('grok', process.execPath, [join(repo, 'packages/paperclip-runner/scripts/provision-grok.mjs'), stagedGrok]); - const listing = run(process.execPath, [join(repo, 'scripts/release-package-map.mjs'), 'list'], repo).toString().trim().split('\n').map(line => line.split('\t')); - const packages = new Map(listing.map(([dir, name]) => [name, { dir, manifest: JSON.parse(readFileSync(join(repo, dir, 'package.json'), 'utf8')) }])); - const needed = new Set(); - function visit(name) { - if (needed.has(name)) return; - const entry = packages.get(name); assert.ok(entry, `Missing public workspace dependency ${name}`); needed.add(name); - for (const [dep, spec] of Object.entries({ ...entry.manifest.dependencies, ...entry.manifest.optionalDependencies })) { - if (spec.startsWith('workspace:')) visit(dep); - } - } - visit('@paperclipai/server'); - visit('paperclipai'); + // The server, the CLI and every public workspace package they reach through + // `workspace:` specs. The npm cache warm-up reads the same closure, so both + // take it from one module. + const { packages, needed } = publicInstallClosure(repo); // release.sh also builds the standalone public packages (the sandbox-provider // plugins outside the pnpm workspace) before it stages anything, because it // publishes them. Nothing staged here reaches them: `needed` is closed under @@ -340,7 +333,13 @@ try { } const assets = join(root, 'assets'); mkdirSync(assets, { mode: 0o755 }); const consumer = join(root, 'consumer'); mkdirSync(consumer); - const cache = join(root, 'cache'); mkdirSync(cache); + // CI can point this at a cache that scripts/warm-public-install-npm-cache.mjs + // filled with the same public registry graph while `pnpm build` ran. npm + // checks every cached tarball against the integrity its own resolution + // records, so a warm cache changes where bytes come from, never what is + // installed; the consumer directory and its lock still start empty. + const cache = process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE ? resolve(process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE) : join(root, 'cache'); + mkdirSync(cache, { recursive: true }); writeFileSync(join(consumer, 'package.json'), JSON.stringify({ private: true, type: 'module' })); for (const tarball of tarballs) { const destination = join(assets, basename(tarball)); cpSync(tarball, destination); chmodSync(destination, 0o644); diff --git a/scripts/warm-public-install-npm-cache.mjs b/scripts/warm-public-install-npm-cache.mjs new file mode 100755 index 00000000000..a6ee8f3b194 --- /dev/null +++ b/scripts/warm-public-install-npm-cache.mjs @@ -0,0 +1,37 @@ +#!/usr/bin/env node +// Fill an npm cache with the public registry graph of the packages the +// public-install verifier stages, while `pnpm build` still runs. The verifier +// starts after the build with an empty cache, and its consumer install then +// downloads the whole graph (106-127s in runs 38241687458 to 38250487646, +// 2026-10-12). The graph is fixed by source manifests, so it can be fetched +// before any tarball exists. +// +// This only downloads. Scripts are disabled, nothing here is verified, and the +// scratch consumer is removed afterwards. npm checks every cached tarball +// against the integrity its own resolution records, so a warm cache changes +// where bytes come from, never what the verifier installs. Any failure leaves +// a partial cache that the verifier completes from the registry as before. +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { publicInstallClosure, publicRegistryDependencies } from './public-install-closure.mjs'; + +const repo = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +const cache = process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE; +assert.ok(cache, 'Set PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE to the cache directory the verifier will use'); +mkdirSync(cache, { recursive: true }); +const started = performance.now(); +const scratch = mkdtempSync(join(tmpdir(), 'paperclip-public-install-warm-')); +try { + const { dependencies, optionalDependencies } = publicRegistryDependencies(publicInstallClosure(repo)); + writeFileSync(join(scratch, 'package.json'), JSON.stringify({ private: true, dependencies, optionalDependencies })); + execFileSync('npm', ['install', '--ignore-scripts', '--omit=dev', '--no-audit', '--no-fund', '--no-package-lock'], { + cwd: scratch, stdio: 'inherit', env: { ...process.env, npm_config_cache: cache, npm_config_update_notifier: 'false' }, + }); + console.log(JSON.stringify({ schema: 'paperclip.public-npm-install.cache-warm.v1', dependencyCount: Object.keys(dependencies).length + Object.keys(optionalDependencies).length, warm_ms: Math.round(performance.now() - started) })); +} finally { + rmSync(scratch, { recursive: true, force: true }); +}