mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 16:27:18 +02:00
perf(ci): warm the public-install npm cache during pnpm build (#16204)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each pull request runs the Trusted PR CI. The `ci / Build` lane is the longest check, or close to the longest, in each green run. > - On 2026-10-13, `ci / Build` took 415-446s in runs 38426120144, 38432086560 and 38423713537. Of that, the public-install verifier took 226-241s. > - In the verifier, the consumer `npm install` took 119-129s. It downloads the full public registry graph into an empty npm cache. > - Source manifests set that graph. Thus CI can download it before `pnpm build` makes any tarball. > - This pull request fills the npm cache in the background while the Rust restore, Evalbook and `pnpm build` run. Then the verifier reads that cache. > - The benefit is a shorter `ci / Build` lane. The verifier keeps all its checks. ## Linked Issues or Issue Description Refs #15213, #15849, #15868, #15888 (earlier changes that made this verifier faster). **Enhancement description** The `ci / Build` lane runs the public-install verifier after `pnpm build`. The verifier starts with an empty npm cache. Its consumer install then downloads the full public registry graph: 119-129s of a 226-241s step. **Proposed change** Download the same registry graph into an npm cache in the background, during the earlier part of the lane. Give that cache to the verifier. **Why it helps** The download moves off the critical path of the lane. A local test of the same 628-package install took 51s with a cold cache and 21s with a warm cache. ## What Changed - Add `scripts/public-install-closure.mjs`. It holds the one definition of the public package closure: `@paperclipai/server`, `paperclipai`, and each public workspace package that they reach through `workspace:` specs. It also gives the registry dependencies of that closure. - Add `scripts/warm-public-install-npm-cache.mjs`. It installs those registry dependencies into a temporary consumer, with scripts disabled, to fill `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE`. Then it removes the temporary consumer. - `scripts/verify-grok-npm-install.mjs` now gets its closure from the shared module. When `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE` is set, it uses that directory as the consumer npm cache. When the variable is not set, it uses a new empty cache, as before. `docker-runner-check.yml` and `release-smoke.yml` do not change. - `pr-trusted.yml` (Build job): a new step starts the warm-up in the background after `Install dependencies`. The verify step waits for the warm-up for a maximum of 180s and then gives the cache to the verifier. If the warm-up fails, the step writes a notice and the verifier continues as before. - Add `scripts/__tests__/public-install-closure.test.mjs` and add it to `test:release-registry`. Update the closure assertions in `grok-public-install-sandbox.test.mjs`. ## Verification - `node --test scripts/__tests__/public-install-closure.test.mjs`: 4/4 pass. - `node --test scripts/__tests__/grok-public-install-sandbox.test.mjs`: 24/24 pass. - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs`: all pass. - `release-verify-workflow.test.mjs`: the two "publisher lockfile" tests fail in my sandbox on clean master too, because they need pnpm. Two canary git-fixture tests timed out once under load. They pass when they run alone. - I ran the warm-up against current master: 62 direct dependencies, 628 packages, 51s cold. The same install with the warm cache took 21s. - Important: `pr.yml` calls `pr-trusted.yml@master`. Thus the CI of this pull request runs the refactored verifier on its default path (empty cache), but not the new warm-up steps. The warm-up steps start after merge. To measure them, compare the `consumer_install_ms` value in the `paperclip.public-npm-install.timing.v1` line of a Build job before and after merge. - Final review: all latest-head CI checks pass (54 successes, one neutral result, two intentional skips). Fresh Apex is 5/5 on `64d091b8b0c5359fa3dc0edb1fd7ef9d24294477` with no unresolved findings. The two relevant local script suites pass all 28 tests. Current master combines cleanly. ## Risks - Low risk. The verifier result does not change. npm checks each cached tarball against the integrity of its own resolution, and the consumer directory and lock start empty. - npm can use a cached registry packument that is a maximum of about 5 minutes old (registry `max-age`). This has the same effect as starting the verifier a few minutes earlier. - The warm-up runs at the same time as `pnpm build`. It uses network, disk and some CPU. In the measured runs, `pnpm build` took 76-96s. - If the warm-up is slow or fails, the verify step waits a maximum of 180s and then continues with what is in the cache. - The verifier receipt still reports `cleanNpmInstall: true`. The consumer install is clean. Only the download cache is shared. ## Expected time savings - `ci / Build`: an estimated 65-85s less, from 415-446s to about 340-370s. The consumer install goes from 119-129s to about 40-55s, from the local cold/warm ratio. - Full PR time: after this change, `ci / Build` is no longer the longest lane. The longest lanes become `General tests (server N/12)` and `Verify Paperclip Runner (vitest 1/2)` at about 405-496s. Thus the full PR time decreases only when Build is the critical path. That is approximately 0-30s in the runs above. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in a Paperclip agent run. OpenAI GPT-6 (`gpt-6`) through Codex assisted the final repository review and targeted script validation. This session does not expose an exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed
1 parent
e0f30a4c71
commit
b7f41ee51d
7 files changed
+199
-28
No files matched your search
@@ -788,6 +788,25 @@ jobs:
|
||||
pnpm install --frozen-lockfile
|
||||
fi
|
||||
|
||||
# The public-install verifier at the end of this lane downloads the whole
|
||||
# public registry graph into an empty npm cache: its consumer install
|
||||
# took 119-129s of the 226-241s step, more than half of this ~415-446s
|
||||
# lane, in runs 38426120144, 38432086560 and 38423713537 (2026-10-13).
|
||||
# That graph is fixed by source manifests, so this fetches it in the
|
||||
# background while the Rust restore, Evalbook and `pnpm build` run, about
|
||||
# 150s before the verifier installs anything. Only the cache is handed
|
||||
# over, and the verify step waits for this first. A failed or partial
|
||||
# warm-up leaves the verifier to fetch what is missing as before.
|
||||
- name: Start warming the public-install npm cache
|
||||
if: ${{ hashFiles('scripts/warm-public-install-npm-cache.mjs') != '' }}
|
||||
run: |
|
||||
(
|
||||
# The default `bash -e` would skip the status line on a failure.
|
||||
set +e
|
||||
PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache" node scripts/warm-public-install-npm-cache.mjs
|
||||
echo $? > "$RUNNER_TEMP/public-install-npm-cache.status"
|
||||
) > "$RUNNER_TEMP/public-install-npm-cache.log" 2>&1 &
|
||||
|
||||
# pnpm build reaches the Runner package's build:binary step; without the
|
||||
# shared Rust cache that is a ~3m40s cold compile of all third-party
|
||||
# crates (run 35036001734, 2026-09-15). Same restore-only contract as
|
||||
@@ -914,7 +933,20 @@ jobs:
|
||||
# docker-runner-check.yml runs the same verifier after `pnpm build`.
|
||||
- name: Verify built-in Grok from a clean public npm install
|
||||
if: ${{ hashFiles('scripts/verify-grok-npm-install.mjs') != '' }}
|
||||
run: node scripts/verify-grok-npm-install.mjs
|
||||
run: |
|
||||
status="$RUNNER_TEMP/public-install-npm-cache.status"
|
||||
if [ -f "$RUNNER_TEMP/public-install-npm-cache.log" ]; then
|
||||
# Normally finished long before this; never let it hold the lane.
|
||||
for _ in $(seq 180); do [ -f "$status" ] && break; sleep 1; done
|
||||
if [ "$(cat "$status" 2>/dev/null)" = 0 ]; then
|
||||
tail -n 1 "$RUNNER_TEMP/public-install-npm-cache.log"
|
||||
else
|
||||
echo '::notice title=Public-install npm cache::warm-up did not finish cleanly; the verifier fetches what is missing'
|
||||
tail -n 20 "$RUNNER_TEMP/public-install-npm-cache.log"
|
||||
fi
|
||||
export PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache"
|
||||
fi
|
||||
node scripts/verify-grok-npm-install.mjs
|
||||
|
||||
verify_serialized_server:
|
||||
name: Verify serialized server suites (${{ matrix.shard_label }})
|
||||
|
||||
+1
-1
@@ -61,7 +61,7 @@
|
||||
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
|
||||
"smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh",
|
||||
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
|
||||
"test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
|
||||
"test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
|
||||
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
|
||||
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
|
||||
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
|
||||
|
||||
@@ -9,6 +9,7 @@ import { fileURLToPath } from 'node:url';
|
||||
import { isWorkspacePackage, parseWorkspaceEntries } from '../build-standalone-public-packages.mjs';
|
||||
import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertInstalledCodexPlatformProvenance, assertInstalledNativeConsumerContinuity, assertNoBundledCodexPayloads, grokConsumerDockerArgs, installedCodexProbeSource, installedDaemonProbeSource, macNodeHeadersDirectory, macPublicInstallLifecycleProfile, macInstalledStartupProbeSource, runMacInitialInstallWithDiagnostics } from '../grok-public-install-sandbox.mjs';
|
||||
import { verifyPiPackagedDaemon } from '../pi-public-install-probe.mjs';
|
||||
import { publicInstallClosure } from '../public-install-closure.mjs';
|
||||
import { runnerBinaryTarget } from '../../packages/paperclip-runner/src/live/runner-binary.ts';
|
||||
|
||||
const paths = { assets: '/private/staging/assets', consumer: '/private/staging/consumer', cache: '/private/staging/cache', uid: 1001, gid: 1001 };
|
||||
@@ -209,7 +210,7 @@ test('the image and Grok downloads start before staging and join without weakeni
|
||||
assert.ok(source.includes("const stagedGrok = join(prefetched, 'grok')"));
|
||||
assert.doesNotMatch(source, /provision-grok\.mjs'\), prerequisite\]/);
|
||||
// Both downloads start before any package is staged and before the first sandbox runs.
|
||||
const staging = position("visit('@paperclipai/server')");
|
||||
const staging = position('publicInstallClosure(repo)');
|
||||
const firstSandbox = position("isolated(['npm', 'install', '--ignore-scripts', '--omit=dev'");
|
||||
assert.ok(imagePull < staging && grokProvision < staging);
|
||||
assert.ok(position("await awaited('image', imageReady)") < firstSandbox);
|
||||
@@ -284,17 +285,8 @@ test('the verifier stages only workspace packages, so it skips the standalone bu
|
||||
const retention = readFileSync(new URL('../retain-runner-qualification-packages.mjs', import.meta.url), 'utf8');
|
||||
assert.ok(retention.includes("const pluginRelative = 'packages/plugins/sandbox-providers/daytona';") && retention.includes("cpSync(join(repo, pluginRelative, 'dist'), join(plugin, 'dist'), { recursive: true })"), 'retention still packs the Daytona plugin from its dist, which is why the gated build exists');
|
||||
const repo = new URL('../../', import.meta.url);
|
||||
const listing = spawnSync(process.execPath, [fileURLToPath(new URL('scripts/release-package-map.mjs', repo)), 'list'], { cwd: fileURLToPath(repo), encoding: 'utf8' });
|
||||
assert.equal(listing.status, 0, listing.stderr);
|
||||
const packages = new Map(listing.stdout.trim().split('\n').map(line => line.split('\t')).map(([dir, name]) => [name, { dir, manifest: JSON.parse(readFileSync(new URL(`${dir}/package.json`, repo), 'utf8')) }]));
|
||||
const needed = new Set();
|
||||
const visit = name => {
|
||||
if (needed.has(name)) return;
|
||||
const entry = packages.get(name); assert.ok(entry, `missing public workspace dependency ${name}`); needed.add(name);
|
||||
for (const [dep, spec] of Object.entries({ ...entry.manifest.dependencies, ...entry.manifest.optionalDependencies })) if (spec.startsWith('workspace:')) visit(dep);
|
||||
};
|
||||
assert.ok(source.includes("visit('@paperclipai/server');\n visit('paperclipai');"));
|
||||
visit('@paperclipai/server'); visit('paperclipai');
|
||||
assert.ok(source.includes('const { packages, needed } = publicInstallClosure(repo);'));
|
||||
const { packages, needed } = publicInstallClosure(fileURLToPath(repo));
|
||||
const workspace = parseWorkspaceEntries(readFileSync(new URL('pnpm-workspace.yaml', repo), 'utf8'));
|
||||
const standalone = [...packages.values()].filter(({ dir }) => !isWorkspacePackage(dir, workspace)).map(({ dir }) => dir);
|
||||
assert.ok(standalone.length > 0, 'the public package map must still carry standalone packages for this check to mean anything');
|
||||
@@ -307,7 +299,7 @@ test('the verifier stages only workspace packages, so it skips the standalone bu
|
||||
assert.deepEqual(JSON.parse(readFileSync(new URL('server/package.json', repo), 'utf8')).files, ['dist', 'ui-dist', 'skills']);
|
||||
assert.ok(source.includes("PAPERCLIP_RELEASE_REUSE_UI_DIST: '1'"));
|
||||
const uiDist = position("run('bash', [join(repo, 'scripts/prepare-server-ui-dist.sh')], repo)");
|
||||
assert.ok(position("visit('paperclipai')") < gate && gate < uiDist && uiDist < position('timing.stage_ui_dist_ms =') && uiDist < position('const stageSource = '));
|
||||
assert.ok(position('publicInstallClosure(repo)') < gate && gate < uiDist && uiDist < position('timing.stage_ui_dist_ms =') && uiDist < position('const stageSource = '));
|
||||
});
|
||||
|
||||
test('the installed Codex probe exercises the public export and rejects incomplete or mismatched packages', () => {
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import test from 'node:test';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { PUBLIC_INSTALL_ROOTS, publicInstallClosure, publicRegistryDependencies } from '../public-install-closure.mjs';
|
||||
|
||||
const repo = fileURLToPath(new URL('../../', import.meta.url));
|
||||
|
||||
test('the closure starts at the server and the CLI and follows only workspace specs', () => {
|
||||
assert.deepEqual(PUBLIC_INSTALL_ROOTS, ['@paperclipai/server', 'paperclipai']);
|
||||
const { packages, needed } = publicInstallClosure(repo);
|
||||
const order = [...needed];
|
||||
assert.equal(order[0], '@paperclipai/server');
|
||||
assert.ok(needed.has('paperclipai'));
|
||||
for (const name of needed) {
|
||||
const { manifest } = packages.get(name);
|
||||
for (const [dep, spec] of Object.entries({ ...manifest.dependencies, ...manifest.optionalDependencies })) {
|
||||
assert.equal(needed.has(dep), spec.startsWith('workspace:'), `${name} -> ${dep}@${spec}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('the warm-up graph is the closure\'s registry dependencies, first spec wins', () => {
|
||||
const pkg = (dependencies, optionalDependencies) => ({ dir: 'x', manifest: { dependencies, optionalDependencies } });
|
||||
const packages = new Map([
|
||||
['a', pkg({ b: 'workspace:*', left: '^1.0.0', shared: '^2.0.0', local: 'file:../local', linked: 'link:../linked' }, { native: '1.2.3' })],
|
||||
['b', pkg({ shared: '^3.0.0', right: '~4.0.0' }, { left: '^9.0.0' })],
|
||||
['unvisited', pkg({ never: '1.0.0' })],
|
||||
]);
|
||||
assert.deepEqual(publicRegistryDependencies({ packages, needed: new Set(['a', 'b']) }), {
|
||||
dependencies: { left: '^1.0.0', shared: '^2.0.0', right: '~4.0.0' },
|
||||
optionalDependencies: { native: '1.2.3' },
|
||||
});
|
||||
const real = publicRegistryDependencies(publicInstallClosure(repo));
|
||||
const names = [...Object.keys(real.dependencies), ...Object.keys(real.optionalDependencies)];
|
||||
assert.ok(names.length > 0);
|
||||
for (const spec of [...Object.values(real.dependencies), ...Object.values(real.optionalDependencies)]) {
|
||||
assert.doesNotMatch(spec, /^(workspace|file|link):/);
|
||||
}
|
||||
});
|
||||
|
||||
test('the warm-up refuses to run without the cache the verifier will read', () => {
|
||||
const env = { ...process.env };
|
||||
delete env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE;
|
||||
const result = spawnSync(process.execPath, [fileURLToPath(new URL('../warm-public-install-npm-cache.mjs', import.meta.url))], { env, encoding: 'utf8' });
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(result.stderr, /PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE/);
|
||||
});
|
||||
|
||||
test('the Build lane warms the cache before pnpm build and hands the same directory to the verifier', () => {
|
||||
const workflow = readFileSync(new URL('../../.github/workflows/pr-trusted.yml', import.meta.url), 'utf8');
|
||||
const build = workflow.slice(workflow.indexOf('\n build:\n'), workflow.indexOf('\n verify_serialized_server:\n'));
|
||||
const position = fragment => { const index = build.indexOf(fragment); assert.notEqual(index, -1, `Build job must contain ${fragment}`); return index; };
|
||||
const warm = position('PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache" node scripts/warm-public-install-npm-cache.mjs');
|
||||
assert.ok(position('- name: Install dependencies') < warm && warm < position('- name: Build\n'), 'the warm-up overlaps the Rust restore and pnpm build');
|
||||
assert.ok(build.includes('set +e\n'), 'a failed warm-up still records its status, so the verifier never waits it out');
|
||||
const verify = position('export PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE="$RUNNER_TEMP/public-install-npm-cache"');
|
||||
assert.ok(verify < position('node scripts/verify-grok-npm-install.mjs'));
|
||||
const verifier = readFileSync(new URL('../verify-grok-npm-install.mjs', import.meta.url), 'utf8');
|
||||
assert.ok(verifier.includes("const cache = process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE ? resolve(process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE) : join(root, 'cache');"));
|
||||
});
|
||||
@@ -0,0 +1,49 @@
|
||||
// The public packages the public-install verifier stages: `@paperclipai/server`
|
||||
// and `paperclipai` plus every public workspace package they reach through
|
||||
// `workspace:` specs, in the depth-first order the verifier packs them. The
|
||||
// npm cache warm-up reads the same closure from source manifests before
|
||||
// `pnpm build` has produced anything, so both share this one definition.
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
|
||||
export const PUBLIC_INSTALL_ROOTS = ['@paperclipai/server', 'paperclipai'];
|
||||
|
||||
// Returns `{ packages, needed }`: `packages` maps every public workspace
|
||||
// package name to `{ dir, manifest }`, and `needed` is the closure in visit
|
||||
// order.
|
||||
export function publicInstallClosure(repo, roots = PUBLIC_INSTALL_ROOTS) {
|
||||
const listing = execFileSync(process.execPath, [join(repo, 'scripts/release-package-map.mjs'), 'list'], { cwd: repo, stdio: 'pipe' })
|
||||
.toString().trim().split('\n').map(line => line.split('\t'));
|
||||
const packages = new Map(listing.map(([dir, name]) => [name, { dir, manifest: JSON.parse(readFileSync(join(repo, dir, 'package.json'), 'utf8')) }]));
|
||||
const needed = new Set();
|
||||
function visit(name) {
|
||||
if (needed.has(name)) return;
|
||||
const entry = packages.get(name); assert.ok(entry, `Missing public workspace dependency ${name}`); needed.add(name);
|
||||
for (const [dep, spec] of Object.entries({ ...entry.manifest.dependencies, ...entry.manifest.optionalDependencies })) {
|
||||
if (spec.startsWith('workspace:')) visit(dep);
|
||||
}
|
||||
}
|
||||
for (const root of roots) visit(root);
|
||||
return { packages, needed };
|
||||
}
|
||||
|
||||
// The registry dependencies the closure's packages declare, as one consumer
|
||||
// manifest. A name two packages declare with different specs keeps the first
|
||||
// spec: the warm-up only fills a cache, and the real install fetches whatever
|
||||
// it still lacks.
|
||||
export function publicRegistryDependencies({ packages, needed }) {
|
||||
const dependencies = {};
|
||||
const optionalDependencies = {};
|
||||
for (const name of needed) {
|
||||
const { manifest } = packages.get(name);
|
||||
for (const [section, target] of [['dependencies', dependencies], ['optionalDependencies', optionalDependencies]]) {
|
||||
for (const [dep, spec] of Object.entries(manifest[section] ?? {})) {
|
||||
if (needed.has(dep) || /^(workspace|file|link):/.test(spec) || dep in dependencies || dep in optionalDependencies) continue;
|
||||
target[dep] = spec;
|
||||
}
|
||||
}
|
||||
}
|
||||
return { dependencies, optionalDependencies };
|
||||
}
|
||||
@@ -13,6 +13,7 @@ import { basename, dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { packConcurrency, packStagedPackages, packTimings } from './pack-staged-packages.mjs';
|
||||
import { materializePublishManifest } from './prepare-bundled-package.mjs';
|
||||
import { publicInstallClosure } from './public-install-closure.mjs';
|
||||
import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertInstalledNativeConsumerContinuity, assertNoBundledCodexPayloads, grokConsumerDockerArgs, installedCodexProbeSource, installedDaemonProbeSource, macNodeHeadersDirectory, macPublicInstallLifecycleProfile, macInstalledStartupProbeSource, runMacInitialInstallWithDiagnostics } from './grok-public-install-sandbox.mjs';
|
||||
import { startPrefetch, stopPrefetches } from './prefetch-child.mjs';
|
||||
import { retainRunnerQualificationPackages } from './retain-runner-qualification-packages.mjs';
|
||||
@@ -229,18 +230,10 @@ try {
|
||||
// Provision as the unprivileged verification user, never into the host's /opt.
|
||||
// Only the positive probe sees this file at the canonical sandbox path.
|
||||
const grokReady = prefetch('grok', process.execPath, [join(repo, 'packages/paperclip-runner/scripts/provision-grok.mjs'), stagedGrok]);
|
||||
const listing = run(process.execPath, [join(repo, 'scripts/release-package-map.mjs'), 'list'], repo).toString().trim().split('\n').map(line => line.split('\t'));
|
||||
const packages = new Map(listing.map(([dir, name]) => [name, { dir, manifest: JSON.parse(readFileSync(join(repo, dir, 'package.json'), 'utf8')) }]));
|
||||
const needed = new Set();
|
||||
function visit(name) {
|
||||
if (needed.has(name)) return;
|
||||
const entry = packages.get(name); assert.ok(entry, `Missing public workspace dependency ${name}`); needed.add(name);
|
||||
for (const [dep, spec] of Object.entries({ ...entry.manifest.dependencies, ...entry.manifest.optionalDependencies })) {
|
||||
if (spec.startsWith('workspace:')) visit(dep);
|
||||
}
|
||||
}
|
||||
visit('@paperclipai/server');
|
||||
visit('paperclipai');
|
||||
// The server, the CLI and every public workspace package they reach through
|
||||
// `workspace:` specs. The npm cache warm-up reads the same closure, so both
|
||||
// take it from one module.
|
||||
const { packages, needed } = publicInstallClosure(repo);
|
||||
// release.sh also builds the standalone public packages (the sandbox-provider
|
||||
// plugins outside the pnpm workspace) before it stages anything, because it
|
||||
// publishes them. Nothing staged here reaches them: `needed` is closed under
|
||||
@@ -340,7 +333,13 @@ try {
|
||||
}
|
||||
const assets = join(root, 'assets'); mkdirSync(assets, { mode: 0o755 });
|
||||
const consumer = join(root, 'consumer'); mkdirSync(consumer);
|
||||
const cache = join(root, 'cache'); mkdirSync(cache);
|
||||
// CI can point this at a cache that scripts/warm-public-install-npm-cache.mjs
|
||||
// filled with the same public registry graph while `pnpm build` ran. npm
|
||||
// checks every cached tarball against the integrity its own resolution
|
||||
// records, so a warm cache changes where bytes come from, never what is
|
||||
// installed; the consumer directory and its lock still start empty.
|
||||
const cache = process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE ? resolve(process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE) : join(root, 'cache');
|
||||
mkdirSync(cache, { recursive: true });
|
||||
writeFileSync(join(consumer, 'package.json'), JSON.stringify({ private: true, type: 'module' }));
|
||||
for (const tarball of tarballs) {
|
||||
const destination = join(assets, basename(tarball)); cpSync(tarball, destination); chmodSync(destination, 0o644);
|
||||
|
||||
Executable
+37
@@ -0,0 +1,37 @@
|
||||
#!/usr/bin/env node
|
||||
// Fill an npm cache with the public registry graph of the packages the
|
||||
// public-install verifier stages, while `pnpm build` still runs. The verifier
|
||||
// starts after the build with an empty cache, and its consumer install then
|
||||
// downloads the whole graph (106-127s in runs 38241687458 to 38250487646,
|
||||
// 2026-10-12). The graph is fixed by source manifests, so it can be fetched
|
||||
// before any tarball exists.
|
||||
//
|
||||
// This only downloads. Scripts are disabled, nothing here is verified, and the
|
||||
// scratch consumer is removed afterwards. npm checks every cached tarball
|
||||
// against the integrity its own resolution records, so a warm cache changes
|
||||
// where bytes come from, never what the verifier installs. Any failure leaves
|
||||
// a partial cache that the verifier completes from the registry as before.
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { publicInstallClosure, publicRegistryDependencies } from './public-install-closure.mjs';
|
||||
|
||||
const repo = resolve(dirname(fileURLToPath(import.meta.url)), '..');
|
||||
const cache = process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE;
|
||||
assert.ok(cache, 'Set PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE to the cache directory the verifier will use');
|
||||
mkdirSync(cache, { recursive: true });
|
||||
const started = performance.now();
|
||||
const scratch = mkdtempSync(join(tmpdir(), 'paperclip-public-install-warm-'));
|
||||
try {
|
||||
const { dependencies, optionalDependencies } = publicRegistryDependencies(publicInstallClosure(repo));
|
||||
writeFileSync(join(scratch, 'package.json'), JSON.stringify({ private: true, dependencies, optionalDependencies }));
|
||||
execFileSync('npm', ['install', '--ignore-scripts', '--omit=dev', '--no-audit', '--no-fund', '--no-package-lock'], {
|
||||
cwd: scratch, stdio: 'inherit', env: { ...process.env, npm_config_cache: cache, npm_config_update_notifier: 'false' },
|
||||
});
|
||||
console.log(JSON.stringify({ schema: 'paperclip.public-npm-install.cache-warm.v1', dependencyCount: Object.keys(dependencies).length + Object.keys(optionalDependencies).length, warm_ms: Math.round(performance.now() - started) }));
|
||||
} finally {
|
||||
rmSync(scratch, { recursive: true, force: true });
|
||||
}
|
||||
Reference in new issue
Block a user