perf(ci): reuse package downloads and overlap runtime checks (#16474)

## Thinking Path

> - Paperclip contributors wait for CI before delivering changes.
> - Build still finishes last after #16460.
> - CI already downloads the public npm graph while the build runs.
> - Bundled-package preparation ignores that cache and downloads the
same archives again.
> - Reuse the existing cache and overlap independent runtime checks
while preserving fresh installs, integrity, offline lifecycle and
admission verification.
> - Measure staging and the full workflow separately.

## Linked Issues or Issue Description

Refs #16460 and #16434. This is a focused follow-up to merged #16460,
now based directly on master.

**Current behavior:**
The Build workflow warms `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE`, but
`startBundledInstalls` inherits the host's default npm cache. Server
staging took 31.17 seconds in the latest passing run, before packing and
the clean consumer install.

**Proposed behavior:**
Pass the verifier environment, including the explicitly configured warm
cache, to the three existing staging children. Without an explicit
shared cache, keep the usual producer npm configuration.

**Alternatives considered:**
Copying an installed dependency tree would bypass normal fresh
installation. Adding jobs would add setup cost. This change reuses only
npm's integrity-checked download cache. It does not remove coverage or
change product behavior and does not overlap a roadmap feature. Open PR
title search found no equivalent cache wiring change.

## What Changed

- Initialize the existing consumer cache before package staging and
expose it to producer npm only when explicitly configured.
- Pass that environment to the existing bundled-install children.
- Extend the child orchestration check to verify cache propagation and
document the cache boundary.
- Start Pi setup after the offline lifecycle, in its own download
sandbox, while Codex and Grok probes run in separate offline sandboxes.
Join successful Pi setup before lock verification and Pi admission. Name
and clean up only the verifier-owned setup container.

## Verification

- 33 focused public-install, closure and child-orchestration tests pass
on Node 24.21.0.
- A real server dependency staging experiment used
`prepare-bundled-package.mjs`, its actual manifests, provider overrides
and release patches. Built application payloads were omitted from this
dependency-only measurement.
- Cold cache: **16.139s**. After the actual CI warm-up script:
**6.980s**. A fully warm producer cache took **6.127s**. These are local
sequential measurements, not a CI wall-clock claim; filesystem warmth
and registry timing vary.
- All **28,329 files/links**, including both lockfiles, were
byte-identical between the cold output and the actual-workflow-warm
output.
- The warm-up itself took 49.631s; CI already runs it during Rust
restore/build. This change adds no warm-up operation.
- First follow-up run [passed all 48
jobs](https://github.com/paperclipai/paperclip/actions/runs/38791277425),
and Greptile gave the cache-only head 5/5. Server staging fell **31.172s
→ 17.565s** and total staging **70.169s → 54.869s**. But complete CI was
**8m20s**, slower than the parent's 8m04s; longer Codex/Pi probe times
outweighed that saving. This is not a full-run speed improvement claim.
- The next revision overlaps the independent probes instead of
serializing Pi setup behind them. All 46 focused sandbox, staging,
closure and process-supervision tests pass, including failed background
children, parent exit and cancellation. Pre-retarget head `590873a44`
passes all 48 jobs and Greptile 5/5 with no review findings. After
#16460 merged, head `fe4f8d38e` incorporates master and retargets this
PR directly to master. Its diff remains only four
optimization/documentation/test files; 46 focused checks pass again.
Fresh full CI and review are pending. The unchanged parent
conflict-resolution run passed all 48 jobs in 8m04s:
https://github.com/paperclipai/paperclip/actions/runs/38789785045.

Final paired runs against the same updated master, both on
`m7i-flex.xlarge`:

| Measurement | Parent #16460 | This PR |
| --- | ---: | ---: |
| Bundled server staging | 22.220s | 14.202s |
| All package staging | 61.277s | 48.304s |
| Clean consumer install | 67.617s | 73.490s |
| Pi setup remaining wait | 25.428s (serial setup) | 20.719s |
| Build job | 385s | 368s |
| Build start after workflow creation | 23s | 136s |
| Entire workflow, including admission and aggregates | 452s (7m32s) |
552s (9m12s) |

[Parent
run](https://github.com/paperclipai/paperclip/actions/runs/38792983313),
[final
run](https://github.com/paperclipai/paperclip/actions/runs/38792986392).

Build execution improved by **17s (4.4%)** in this pair. The total
workflow did **not** improve: it waited 113 seconds longer to start
Build, and other lanes/aggregates also contribute. The overlapping setup
ran for 25.667s while Codex/Grok took 4.946s, leaving 20.719s to wait.
Actual installed Codex, Grok and Pi checks all passed with zero provider
calls. These are individual samples with filesystem/network/CPU
variance, not a claimed steady-state average. The earlier slower
cache-only run is retained above. No tests were removed and no jobs were
added.

## Risks

Shared-cache contention can reduce the benefit if warm-up is still
running. npm handles concurrent content-addressed cache writers and
checks integrity; misses still download normally. Two runtime containers
can now overlap and contend for host resources; the full CI run measures
that tradeoff. They retain separate network policies and per-container
bounds. Pi writes only its provider assets and a private receipt; the
other probes inspect Codex/Grok. No installed tree, lock, test result or
build output is reused. The consumer still starts empty and performs the
full lifecycle and runtime checks. Normal unconfigured producer cache
behavior is preserved.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, execution and
GitHub tools. The serving model suffix and context capacity are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in
this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no private ticket
identifier
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [ ] All Paperclip CI gates are green (prior head passed; rerunning
after master merge)
- [ ] Greptile is 5/5 with no open P2s, recommendations or follow-ups
(prior head 5/5; current head pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/ci-profiling.md

@@ -205,3 +205,19 @@ the public npm registry; public tarball URLs remain unchanged. The temporary
 registry closes before the existing offline lifecycle rebuild and runtime
 probes. No lifecycle, network-isolation, lock-preservation or provider checks
 are skipped. Native ARM and Mac qualification keep their existing file installs.
+
+The Linux verifier also shares `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE` with the
+producer installs that prepare bundled dependencies. These run after the existing
+cache warm-up and reuse its downloaded archives instead of populating a second
+host cache. Each producer still installs into a separate empty directory, applies
+the release patches, and packs fresh output. npm resolution and integrity checks
+remain enabled; cache misses download normally. Without an explicit shared cache,
+producer installs keep their normal npm cache configuration.
+
+After the offline lifecycle finishes, Pi setup runs in its own download sandbox
+while the independent Codex/Grok probes run offline. Pi admission still waits
+for successful setup and an unchanged consumer lock. The setup container has a
+unique verifier-owned name and is removed on early failure as well as completion.
+`pi_setup_prefetch_ms` measures setup wall time; `pi_setup_wait_ms` measures only
+the remaining wait after the other probes. These overlapping durations must not
+be added when interpreting total verification time.

scripts/__tests__/grok-public-install-sandbox.test.mjs

@@ -244,7 +244,7 @@ test('the bundled-dependency installs run together while the other packages are
   // cleanup as it starts, with the results put back into dependency order
   // before every tarball is checked.
   const bundledSources = position('for (const entry of entries.filter(entry => entry.bundled)) stageSource(entry)');
-  const start = position("startBundledInstalls(entries.filter(entry => entry.bundled), { logDir: prefetched, cwd: repo })");
+  const start = position("startBundledInstalls(entries.filter(entry => entry.bundled), { logDir: prefetched, cwd: repo, env })");
   const registered = position('prefetches.push(...installs.map(({ handle }) => handle))');
   const otherSources = position('for (const entry of entries.filter(entry => !entry.bundled))');
   const join = position("await awaited('bundled', Promise.all(installs.map(({ handle }) => handle.done)))");
@@ -596,3 +596,19 @@ test('Pi admission verifies assembled host daemon identity and preserves the fla
     await assert.rejects(verifyPiPackagedDaemon(server, flat, target, runnerBinaryTarget), /manifest must be an object/);
   } finally { rmSync(root, { recursive: true, force: true }); }
 });
+
+
+test('Pi setup overlaps independent probes only after the offline lifecycle and joins before admission', () => {
+  const source = readFileSync(new URL('../verify-grok-npm-install.mjs', import.meta.url), 'utf8');
+  const position = fragment => { const index = source.indexOf(fragment); assert.notEqual(index, -1, fragment); return index; };
+  const lifecycle = position("phase('lifecycle_rebuild')");
+  const start = position("prefetch('pi_setup', 'docker', piSetupArgs)");
+  const codex = position("const codex = JSON.parse(isolated(['node', '/packages/codex-probe.mjs'])");
+  const grok = position("phase('grok_probes')");
+  const join = position("await awaited('pi_setup', piSetupReady)");
+  const receipt = position("assert.equal(receipt.status, 'installed_verified')");
+  const probe = position("console.log(isolated(['node', '/packages/pi-public-install-probe.mjs'");
+  assert.ok(lifecycle < start && start < codex && codex < grok && grok < join && join < receipt && receipt < probe);
+  assert.ok(source.includes("piSetupArgs.splice(1, 0, '--name', piSetupContainer)"));
+  assert.ok(position("await stopPrefetches(prefetches)") < position("['rm', '--force', piSetupContainer]"));
+});

scripts/__tests__/stage-bundled-packages.test.mjs

@@ -47,18 +47,18 @@ test('the installs run at the same time, each with its own source and target', a
       if (Date.now() > deadline) { console.error('the other install never started'); process.exit(3); }
       Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 20);
     }
-    writeFileSync(join(target, 'package.json'), '{}');
+    writeFileSync(join(target, 'package.json'), JSON.stringify({ cache: process.env.npm_config_cache }));
   `);
   const entries = entriesFor(root, ['@paperclipai/server', '@paperclipai/db']);
   for (const { target } of entries) mkdirSync(target);
-  const installs = startBundledInstalls(entries, { logDir: root, cwd: root, script });
+  const installs = startBundledInstalls(entries, { logDir: root, cwd: root, script, env: { ...process.env, npm_config_cache: join(root, 'shared-cache') } });
   assert.deepEqual(installs.map(({ slug }) => slug), ['server', 'db']);
   await Promise.all(installs.map(({ handle }) => handle.done));
   for (const [index, { stagedSource, target, slug, handle }] of installs.entries()) {
     assert.equal(handle.child.exitCode, 0);
     assert.equal(readFileSync(join(root, `bundled-${slug}.log`), 'utf8'), `install ${stagedSource} -> ${target}\n`);
     assert.equal(target, join(root, `package-${index}`));
-    assert.ok(existsSync(join(target, 'package.json')));
+    assert.equal(JSON.parse(readFileSync(join(target, 'package.json'), 'utf8')).cache, join(root, 'shared-cache'));
   }
   const timings = bundledInstallTimings(installs);
   assert.deepEqual(Object.keys(timings), ['stage_bundled_server_ms', 'stage_bundled_db_ms']);

scripts/verify-grok-npm-install.mjs

@@ -197,7 +197,9 @@ if (process.argv[2] === '--linux-arm64-consumer') {
 assert.equal(process.platform, 'linux', 'Run this verification on disposable EC2 Linux, not a developer host');
 const root = mkdtempSync(join(tmpdir(), 'paperclip-grok-public-install-'));
 const prerequisite = join(root, 'native/grok');
-const env = { ...process.env, NODE_PATH: '', PAPERCLIP_RELEASE_REUSE_UI_DIST: '1', npm_config_ignore_scripts: 'false', npm_config_audit: 'false', npm_config_fund: 'false' };
+const cache = process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE ? resolve(process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE) : join(root, 'cache');
+mkdirSync(cache, { recursive: true });
+const env = { ...process.env, NODE_PATH: '', ...(process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE ? { npm_config_cache: cache } : {}), PAPERCLIP_RELEASE_REUSE_UI_DIST: '1', npm_config_ignore_scripts: 'false', npm_config_audit: 'false', npm_config_fund: 'false' };
 const run = (cmd, args, cwd = root) => execFileSync(cmd, args, { cwd, env, stdio: 'pipe', maxBuffer: 32 * 1024 * 1024 });
 const sourceRevision = run('git', ['rev-parse', 'HEAD'], repo).toString().trim();
 const releaseVersion = `0.0.0-grok-verify.${sourceRevision.slice(0, 12)}`;
@@ -215,6 +217,8 @@ const stagedGrok = join(prefetched, 'grok');
 const prefetches = [];
 const downloads = [];
 const timing = {};
+const piSetupContainer = `paperclip-pi-setup-${basename(root)}`;
+let piSetupStarted = false;
 // A prefetch failure surfaces where its result is awaited, never as an
 // unhandled rejection that would mask an earlier staging error.
 const prefetch = (name, cmd, args) => {
@@ -291,7 +295,7 @@ try {
   const sourcesStarted = performance.now();
   for (const entry of entries.filter(entry => entry.bundled)) stageSource(entry);
   const installsStarted = performance.now();
-  const installs = startBundledInstalls(entries.filter(entry => entry.bundled), { logDir: prefetched, cwd: repo });
+  const installs = startBundledInstalls(entries.filter(entry => entry.bundled), { logDir: prefetched, cwd: repo, env });
   prefetches.push(...installs.map(({ handle }) => handle));
   for (const entry of entries.filter(entry => !entry.bundled)) {
     const releaseManifest = stageSource(entry);
@@ -339,8 +343,6 @@ try {
   // checks every cached tarball against the integrity its own resolution
   // records, so a warm cache changes where bytes come from, never what is
   // installed; the consumer directory and its lock still start empty.
-  const cache = process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE ? resolve(process.env.PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE) : join(root, 'cache');
-  mkdirSync(cache, { recursive: true });
   writeFileSync(join(consumer, 'package.json'), JSON.stringify({ private: true, type: 'module' }));
   for (const tarball of tarballs) {
     const destination = join(assets, basename(tarball)); cpSync(tarball, destination); chmodSync(destination, 0o644);
@@ -374,6 +376,13 @@ try {
   assert.equal(readFileSync(sentinel, 'utf8'), 'ok', 'Offline lifecycle scripts must actually execute');
   assert.equal(readFileSync(join(consumer, 'package-lock.json'), 'utf8'), consumerLock, 'Lifecycle execution must preserve the resolved consumer lock');
   phase('lifecycle_rebuild');
+  const piSetupReceipt = join(consumer, '.paperclip-pi-setup.json');
+  const piSetupArgs = grokConsumerDockerArgs({ assets, consumer, cache, uid: consumerUid, gid: process.getgid(),
+    command: ['sh', '-c', 'exec node /consumer/node_modules/paperclipai/dist/index.js runtime setup pi > /consumer/.paperclip-pi-setup.json'],
+    download: true, temporarySizeMiB: 2048 });
+  piSetupArgs.splice(1, 0, '--name', piSetupContainer);
+  const piSetupReady = prefetch('pi_setup', 'docker', piSetupArgs);
+  piSetupStarted = true;
   for (const name of needed) {
     const installedManifest = JSON.parse(readFileSync(join(consumer, 'node_modules', name, 'package.json'), 'utf8'));
     assert.equal(installedManifest.version, releaseVersion, `Installed release version for ${name}`);
@@ -422,8 +431,9 @@ try {
   // Pi assembles a bundled runtime in scratch space before atomic publication.
   // Keep the existing sandbox and memory bound; only this download needs more
   // temporary capacity than the smaller offline lifecycle probes.
-  const setup = isolated(['node', '/consumer/node_modules/paperclipai/dist/index.js', 'runtime', 'setup', 'pi'], { download: true, temporarySizeMiB: 2048 }).toString();
-  const receipt = JSON.parse(setup.trim());
+  await awaited('pi_setup', piSetupReady);
+  const receipt = JSON.parse(readFileSync(piSetupReceipt, 'utf8'));
+  rmSync(piSetupReceipt);
   phase('pi_setup');
   assert.equal(receipt.status, 'installed_verified');
   assert.equal(receipt.target, 'linux-x64');
@@ -453,6 +463,9 @@ try {
   // An earlier failure must not leave a prefetch running, and scratch is
   // removed only once both children have stopped writing into it.
   await stopPrefetches(prefetches);
+  if (piSetupStarted) {
+    try { execFileSync('docker', ['rm', '--force', piSetupContainer], { timeout: 10_000, stdio: 'ignore' }); } catch {}
+  }
   rmSync(root, { recursive: true, force: true });
 }
 }