Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-12 18:06:11 +02:00
Code Issues Packages Projects Releases Wiki Activity

perf(ci): prefetch the consumer image and Grok binary during package staging (#15849)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Pull requests to this repository run a large CI matrix. The slowest
check sets the time a contributor waits for a green run.
> - After #15826 leveled the server shards and #15831 split
`workspaces-b`, the `Build` lane is the slowest check: 634s in run
38107578322 (2026-10-11 03:11Z), ahead of the next server shard at 574s.
It was 533s in run 38107430861 and 543s to 568s in earlier runs that
night.
> - `Build` is about 100s of setup, 148s to 195s of `pnpm build`, and
278s to 320s of `scripts/verify-grok-npm-install.mjs`. The verifier
stages and packs the public packages first (84s to 100s of CPU work),
and only then pulls the pinned `node:24-trixie` consumer image (8
layers, 440 MB compressed) and downloads the Grok binary (65 MB gzip,
166 MB verified). Both network waits sit after the staging work on the
critical path.
> - Neither download depends on the packed packages. The image is pinned
by digest and the Grok binary is checked against its pinned sha256 by
the provisioning script, so starting them early changes no verified
property.
> - This pull request starts both downloads as background children
before staging and joins them where they are needed: the image before
the first sandbox, the binary after the negative probe. It also prints
one timing line per phase so the next slow part is visible in the CI
log.
> - The benefit is that the whole pull and the whole Grok download hide
behind the staging phase. The verifier should shed the full pull time on
a fresh runner, and the next cycle can read the phase split instead of
inferring it from log gaps.

## Linked Issues or Issue Description

**Describe the enhancement**

The `Build` PR check is now the slowest check in the run. Its last step,
the public npm install verifier, pulls the consumer Docker image and
downloads the Grok binary only after it has finished staging and packing
the eighteen public packages. Both downloads could run while the
packages are staged.

**Why is this needed**

The slowest check decides the wall-clock time of every PR run. In the
most recent fully green run (38107578322), `Build` took 634s: `pnpm
build` 195s and the verifier 320s. The verifier's log shows the staging
phase ends at +100s and the Grok result arrives at +267s. The image pull
and the Grok download are inside that second window, in series with the
consumer install and the probes.

**Proposed solution**

Start `docker pull --platform linux/amd64 <pinned image>` and
`provision-grok.mjs` as background children at the top of the verifier,
before the staging loop. Await the image right before the first sandbox
call. Provision Grok into a prefetch directory and rename it into
`native/grok` only after the "npm must not provision Grok" assertion and
the negative probe, so that assertion keeps its meaning and only the
positive probe sees the binary. Print a
`paperclip.public-npm-install.timing.v1` line with wall time per phase
and the time the main flow waited on each prefetch.

**Alternatives considered**

- Run the verifier as its own job in parallel with `Build`. It needs the
full `pnpm build` output, so that job would be as long as `Build` is
now.
- Warm the consumer npm cache from an Actions cache. That changes the
"clean public npm install" property the verifier reports. Not done here;
the new timing line will show whether the consumer install is worth that
discussion.
- Pack the eighteen packages concurrently. The three bundled packages
run their own npm installs in series inside `prepareBundledPackage`,
which is synchronous. A larger restructure; left for a later cycle with
timing data.
- Overlap the Pi runtime setup with the Grok probes. Both use the same
consumer directory from separate containers, and the probes after
provisioning are short. Not worth the concurrency on shared state.

## What Changed

- `scripts/prefetch-child.mjs` (new): `startPrefetch({ name, cmd, args,
cwd, env, logPath })` spawns a background child with no inherited
terminal and its stdout and stderr in a log file, and returns `{ name,
child, done, durationMs }`. `done` resolves on exit 0 and otherwise
rejects with the tail of the log; the rejection is observed inside the
helper, so an unawaited failure never becomes an unhandled rejection.
`durationMs` ends at the child's last write to its log (the exit event
is observed late when the caller is blocked in synchronous work; both
downloads print their final status line as they complete).
`stopPrefetches(prefetches, { graceMs })` sends `SIGTERM` to every child
still running, escalates to `SIGKILL` after the grace period, and
resolves only once all of them have exited.
- `scripts/verify-grok-npm-install.mjs`: the image pull and the Grok
provisioning start through the helper inside the `try` block, before the
staging loop. The main flow awaits the image before the first `npm
install` sandbox and awaits Grok after the negative probe, then renames
the staged binary into `native/grok` before the positive probe. The
`finally` block awaits `stopPrefetches` and only then removes the
scratch root, so cleanup never races a child's writes. The script prints
one timing JSON line after the Pi probe, with the per-phase wall times,
the two `*_wait_ms` values, and the corrected `*_prefetch_ms` values.
- `scripts/__tests__/prefetch-child.test.mjs` (new): seven tests run
small local Node children, with no Docker or network: a successful
prefetch resolves and keeps its output; the duration ends at the
download even when the parent is blocked for 1.5s; a nonzero exit
rejects with the log tail; a command that cannot start rejects instead
of hanging the join or the stop; stopping a running child waits for its
exit; a child that ignores `SIGTERM` is killed after the grace period;
an unawaited failure does not produce an unhandled rejection in a fresh
process.
- `scripts/__tests__/grok-public-install-sandbox.test.mjs`: the source
test asserts that exactly the pinned image is pulled for `linux/amd64`,
that Grok is provisioned into the prefetch directory and never straight
into `native/grok`, that both prefetches start before staging, that the
image is joined before the first sandbox, that the rename happens after
the npm assertion and the negative probe and before the positive probe,
that the verifier imports the helper, and that `stopPrefetches` is
awaited before the single `rmSync(root, ...)`.
- `package.json`: `test:release-registry` runs the new test file.

## Verification

- `node --test scripts/__tests__/prefetch-child.test.mjs
scripts/__tests__/grok-public-install-sandbox.test.mjs
scripts/__tests__/cursor-public-install-sandbox.test.mjs
scripts/__tests__/retain-runner-qualification-packages.test.mjs`: 22
pass (7 new in `prefetch-child.test.mjs`, 12 in the sandbox file). The
new file passed on three consecutive runs.
- `node --check scripts/verify-grok-npm-install.mjs` passes.
- `node packages/paperclip-runner/scripts/provision-grok.mjs
<scratch>/prefetch/grok` on this host: `Verified Grok 1.0.13
(linux-x64)` in 1.0s, `native/grok` absent before the move, 166079904
bytes in place after it.
- Docker is not installed on the host that produced this PR, so the
verifier was not run end to end locally. This PR's own `Build` job runs
`node scripts/verify-grok-npm-install.mjs` from the branch checkout, so
it exercises the change in full.

**Measured on this PR's first CI run**
([38108964834](https://github.com/paperclipai/paperclip/actions/runs/38108964834),
created 03:37Z), against two `master` runs created in the same twelve
minutes:

| | 38108878166 (`master`, 03:35Z) | 38108995800 (`master`, 03:37Z) |
38108964834 (this PR, 03:37Z) |
| --- | --- | --- | --- |
| Verifier step | 307s | 248s | 219s |
| `Build` job | 555s | 484s | 459s |
| Slowest check in the run | `Build` 555s | server (8/12) 535s | server
(12/12) 547s |

The PR run's timing line: `stage_packages_ms 84464`, `image_wait_ms 1`,
`consumer_install_ms 97305`, `lifecycle_rebuild_ms 3545`,
`codex_probe_ms 1211`, `grok_wait_ms 0`, `grok_probes_ms 626`,
`pi_setup_ms 22194`, `pi_probe_ms 7753`. Both waits are at most 1ms, so
the staging phase hid both downloads in full. The window from the end of
staging to the Grok result went from 161s to 167s on the two runs that
motivated this PR to 102s here, so the hidden downloads were about 60s
on a fresh `runs-on` spot instance. Against the two same-window `master`
runs, the verifier step is 29s to 88s shorter and `Build` is 25s to 96s
shorter; the spread is runner variance in `pnpm build` and the consumer
install.

The `image_prefetch_ms` and `grok_prefetch_ms` values on that first run
(84550 and 186696) were the parent's late observation of each exit, not
download times; that is the first Greptile finding, fixed by the second
commit, which measures from the download's last output instead.

## Risks

- Low risk. The verifier proves the same properties in the same order.
The only ordering change is that the downloads start earlier; every
assertion, probe, and sandbox argument is unchanged.
- The Grok binary is written under the scratch root in a `prefetch`
directory during staging. That directory is never mounted into a
container, and the `native/grok` assertion still runs against an absent
path.
- A prefetch that fails early now fails the step at its join point,
where the old code would have failed at the same download a few seconds
later. Same outcome, same message source.
- `*_prefetch_ms` is a last-output time. A child that printed early and
then kept working silently would be under-reported; neither `docker
pull` nor `provision-grok.mjs` behaves that way (both print their final
status as they finish), and a silent child falls back to the observed
exit time.
- Cleanup now waits up to 5s for a child that ignores `SIGTERM` before
killing it, so a failing run can end up to 5s later than before. Both
children exit promptly on `SIGTERM` in practice.
- The timing line is one more stdout line. Nothing in the repository
parses the verifier's stdout; the two workflows that run it write the
output to the job log or to a file.

## Model Used

- Claude (Anthropic), model ID `claude-fable-5-1`, running as a
Paperclip agent through the Claude local adapter with tool use (shell,
file edits, GitHub CLI). The agent collected the CI data, made the
change, ran the tests, and wrote this description.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Devin FoleydevinfoleyClaude Fable 5.1
 authored and GitHub committed 2026-10-10 21:10:13 -07:00
1 parent 8dd381f5dd
commit 42ecc96556
5 files changed
+232 -5
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

package.json
+1 -1
View File
@@ -61,7 +61,7 @@
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
"smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh",
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
"test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
"test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
scripts/__tests__/grok-public-install-sandbox.test.mjs
+29
View File
@@ -81,6 +81,35 @@ test('verification never elevates PR-controlled provisioning or cleanup on the h
assert.ok(source.includes("const prerequisite = join(root, 'native/grok')"));
});
test('the image and Grok downloads start before staging and join without weakening the npm checks', () => {
const source = readFileSync(new URL('../verify-grok-npm-install.mjs', import.meta.url), 'utf8');
const position = fragment => { const index = source.indexOf(fragment); assert.notEqual(index, -1, `verifier must contain ${fragment}`); return index; };
// Exactly the pinned consumer image is pulled, for the platform the probes run on.
const imagePull = position("prefetch('image', 'docker', ['pull', '--platform', 'linux/amd64', GROK_PUBLIC_INSTALL_IMAGE])");
// Grok is provisioned into the prefetch directory, never straight into `native/grok`.
const grokProvision = position("prefetch('grok', process.execPath, [join(repo, 'packages/paperclip-runner/scripts/provision-grok.mjs'), stagedGrok])");
assert.ok(source.includes("const stagedGrok = join(prefetched, 'grok')"));
assert.doesNotMatch(source, /provision-grok\.mjs'\), prerequisite\]/);
// Both downloads start before any package is staged and before the first sandbox runs.
const staging = position("visit('@paperclipai/server')");
const firstSandbox = position("isolated(['npm', 'install', '--ignore-scripts', '--omit=dev'");
assert.ok(imagePull < staging && grokProvision < staging);
assert.ok(position("await awaited('image', imageReady)") < firstSandbox);
// The binary moves to the canonical host path only after npm is proven not to
// have provisioned it, after the negative probe, and before the positive probe.
const npmCheck = position("'npm must not provision Grok'");
const negativeProbe = position("isolated(['node', '/packages/probe.mjs', 'missing'])");
const grokJoin = position("await awaited('grok', grokReady)");
const move = position('renameSync(stagedGrok, prerequisite)');
const positiveProbe = position("isolated(['node', '/packages/probe.mjs', 'present'], { prerequisite })");
assert.ok(npmCheck < negativeProbe && negativeProbe < grokJoin && grokJoin < move && move < positiveProbe);
// The children run through the helper covered by prefetch-child.test.mjs, and
// cleanup waits for them before removing the directory they write into.
assert.ok(source.includes("import { startPrefetch, stopPrefetches } from './prefetch-child.mjs'"));
assert.ok(position('await stopPrefetches(prefetches)') < position('rmSync(root, { recursive: true, force: true })'));
assert.equal(source.match(/rmSync\(root/g).length, 1);
});
test('the installed Codex probe exercises the public export and rejects incomplete or mismatched packages', () => {
const root = realpathSync(mkdtempSync(join(tmpdir(), 'installed-codex-probe-')));
try {
scripts/__tests__/prefetch-child.test.mjs
+93
View File
@@ -0,0 +1,93 @@
import assert from 'node:assert/strict';
import { spawnSync } from 'node:child_process';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import test from 'node:test';
import { startPrefetch, stopPrefetches } from '../prefetch-child.mjs';
// Small local Node children stand in for the image pull and the Grok download;
// none of these tests needs Docker or a network.
const withScratch = async body => {
const root = mkdtempSync(join(tmpdir(), 'prefetch-child-'));
try { await body(root); } finally { rmSync(root, { recursive: true, force: true }); }
};
const start = (root, name, script, cmd = process.execPath, args = ['-e', script]) =>
startPrefetch({ name, cmd, args, cwd: root, env: process.env, logPath: join(root, `${name}.log`) });
const blockFor = ms => Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
const untilLogHas = async (path, text) => {
for (let attempt = 0; attempt < 400; attempt++) {
try { if (readFileSync(path, 'utf8').includes(text)) return; } catch {}
await new Promise(resolve => setTimeout(resolve, 25));
}
throw new Error(`log never contained ${text}`);
};
test('a successful prefetch resolves, keeps its output, and is left alone by stop', async () => withScratch(async root => {
const handle = start(root, 'ok', "setTimeout(() => process.stdout.write('fetched\\n'), 150)");
await handle.done;
assert.equal(handle.child.exitCode, 0);
assert.equal(readFileSync(join(root, 'ok.log'), 'utf8'), 'fetched\n');
assert.ok(handle.durationMs >= 100 && handle.durationMs < 5000, `durationMs ${handle.durationMs}`);
await stopPrefetches([handle]);
assert.equal(handle.child.signalCode, null);
}));
test('the duration ends at the download, not when a blocked parent notices the exit', async () => withScratch(async root => {
const handle = start(root, 'early', "setTimeout(() => process.stdout.write('Status: Downloaded\\n'), 100)");
// Synchronous work in the parent, as package staging is in the verifier.
blockFor(1500);
const noticed = Date.now();
await handle.done;
assert.ok(Date.now() - noticed < 500, 'the exit was pending by the time the parent yielded');
assert.ok(handle.durationMs < 900, `durationMs ${handle.durationMs} must not include the 1500ms the parent was blocked`);
}));
test('a nonzero exit rejects with the log tail where the result is awaited', async () => withScratch(async root => {
const handle = start(root, 'bad', "process.stderr.write('registry unreachable\\n'); process.exit(3)");
await assert.rejects(handle.done, /^Error: bad prefetch failed \(exit 3\): registry unreachable$/);
await stopPrefetches([handle]);
}));
test('a command that cannot start rejects instead of hanging the join or the stop', async () => withScratch(async root => {
const handle = start(root, 'missing', '', join(root, 'no-such-command'), []);
await assert.rejects(handle.done, /missing prefetch failed \(.*ENOENT/);
await stopPrefetches([handle]);
}));
test('stopping a running prefetch waits for it to exit so its directory can be removed safely', async () => withScratch(async root => {
const handle = start(root, 'slow', "process.stdout.write('started\\n'); setInterval(() => {}, 1000)");
await untilLogHas(join(root, 'slow.log'), 'started');
const stopping = Date.now();
await stopPrefetches([handle]);
assert.ok(Date.now() - stopping < 4000);
assert.equal(handle.child.signalCode, 'SIGTERM');
assert.equal(handle.child.exitCode, null);
await assert.rejects(handle.done, /slow prefetch failed \(SIGTERM\): started/);
}));
test('a prefetch that ignores SIGTERM is killed after the grace period', async () => withScratch(async root => {
const handle = start(root, 'stubborn', "process.on('SIGTERM', () => {}); process.stdout.write('ignoring\\n'); setInterval(() => {}, 1000)");
await untilLogHas(join(root, 'stubborn.log'), 'ignoring');
const stopping = Date.now();
await stopPrefetches([handle], { graceMs: 200 });
assert.ok(Date.now() - stopping >= 200 && Date.now() - stopping < 4000);
assert.equal(handle.child.signalCode, 'SIGKILL');
}));
test('an unawaited failure never becomes an unhandled rejection', () => {
const script = `
import { startPrefetch } from ${JSON.stringify(new URL('../prefetch-child.mjs', import.meta.url).href)};
import { mkdtempSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path';
const root = mkdtempSync(join(tmpdir(), 'prefetch-unawaited-'));
try {
startPrefetch({ name: 'ignored', cmd: process.execPath, args: ['-e', 'process.exit(2)'], cwd: root, env: process.env, logPath: join(root, 'ignored.log') });
await new Promise(resolve => setTimeout(resolve, 500));
console.log('still running');
} finally { rmSync(root, { recursive: true, force: true }); }
`;
const result = spawnSync(process.execPath, ['--input-type=module', '-e', script], { encoding: 'utf8', timeout: 10_000 });
assert.equal(result.status, 0, result.stderr);
assert.match(result.stdout, /still running/);
assert.doesNotMatch(result.stderr, /unhandled/i);
});
scripts/prefetch-child.mjs
+60
View File
@@ -0,0 +1,60 @@
// Background downloads for verification scripts: start a child early with its
// output in a log file, join it where the result is needed, and stop it before
// the scratch directory it writes into is removed. No shell, no inherited
// terminal, and a failure is reported where the caller awaits it.
import { spawn } from 'node:child_process';
import { closeSync, openSync, readFileSync, statSync } from 'node:fs';
const running = child => child.pid !== undefined && child.exitCode === null && child.signalCode === null;
// Returns { name, child, done, durationMs }. `done` resolves on exit code 0
// and otherwise rejects with the tail of the log; the rejection is observed
// here so an unawaited failure never surfaces as an unhandled rejection.
export function startPrefetch({ name, cmd, args, cwd, env, logPath }) {
const startedAt = Date.now();
const log = openSync(logPath, 'w');
let child;
try {
child = spawn(cmd, args, { cwd, env, stdio: ['ignore', log, log] });
} finally {
closeSync(log); // the child holds its own descriptor
}
const handle = { name, child, done: null, durationMs: null };
let settled = false;
handle.done = new Promise((settle, reject) => {
const finish = failure => {
if (settled) return;
settled = true;
// The caller may be blocked in synchronous work when the child exits, so
// this event can arrive long after the fact. When the child wrote
// anything, its last write is the end time; both downloads this serves
// print their final status line as they complete. A silent child falls
// back to the time the exit was observed.
let finishedAt = Date.now();
try {
const { mtimeMs, size } = statSync(logPath);
if (size > 0) finishedAt = Math.min(finishedAt, mtimeMs);
} catch {}
handle.durationMs = Math.max(0, Math.round(finishedAt - startedAt));
if (failure === null) return settle();
let output = '';
try { output = readFileSync(logPath, 'utf8').trim().slice(-2000); } catch {}
reject(new Error(`${name} prefetch failed (${failure}): ${output}`));
};
child.once('error', error => finish(error.message));
child.once('exit', (code, signal) => finish(code === 0 ? null : (signal ?? `exit ${code}`)));
});
handle.done.catch(() => {});
return handle;
}
// Stops every prefetch that is still running and resolves only once all of
// them have exited, so the caller can remove their working directory without
// racing a write. A child that ignores SIGTERM is killed after `graceMs`.
export async function stopPrefetches(prefetches, { graceMs = 5000 } = {}) {
await Promise.all(prefetches.map(async ({ child, done }) => {
if (running(child)) child.kill('SIGTERM');
const escalate = setTimeout(() => { if (running(child)) child.kill('SIGKILL'); }, graceMs);
try { await done; } catch {} finally { clearTimeout(escalate); }
}));
}
scripts/verify-grok-npm-install.mjs
+49 -4
View File
@@ -4,12 +4,13 @@
// explicit and separate from npm installation, and is removed in finally.
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import { chmodSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import { chmodSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, renameSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { basename, dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { materializePublishManifest, prepareBundledPackage } from './prepare-bundled-package.mjs';
import { GROK_PUBLIC_INSTALL_IMAGE, GROK_PUBLIC_INSTALL_LIFECYCLE, assertNoBundledCodexPayloads, grokConsumerDockerArgs, installedCodexProbeSource } from './grok-public-install-sandbox.mjs';
import { startPrefetch, stopPrefetches } from './prefetch-child.mjs';
import { retainRunnerQualificationPackages } from './retain-runner-qualification-packages.mjs';
const repo = resolve(dirname(fileURLToPath(import.meta.url)), '..');
assert.equal(process.platform, 'linux', 'Run this verification on disposable EC2 Linux, not a developer host');
@@ -19,7 +20,31 @@ const env = { ...process.env, NODE_PATH: '', PAPERCLIP_RELEASE_REUSE_UI_DIST: '1
const run = (cmd, args, cwd = root) => execFileSync(cmd, args, { cwd, env, stdio: 'pipe', maxBuffer: 32 * 1024 * 1024 });
const sourceRevision = run('git', ['rev-parse', 'HEAD'], repo).toString().trim();
const releaseVersion = `0.0.0-grok-verify.${sourceRevision.slice(0, 12)}`;
// The two network downloads the probes need do not depend on anything packed
// below, so they overlap the CPU-bound staging instead of running after it.
// The consumer image is pinned by digest and the Grok binary is checked against
// its pinned sha256 by the provisioning script, so an early start changes no
// verified property. Neither touches the consumer, the assets, nor the
// `native/grok` path that the post-install assertion inspects.
const prefetched = join(root, 'prefetch'); mkdirSync(prefetched);
const stagedGrok = join(prefetched, 'grok');
const prefetches = [];
const timing = {};
// A prefetch failure surfaces where its result is awaited, never as an
// unhandled rejection that would mask an earlier staging error.
const prefetch = (name, cmd, args) => {
const handle = startPrefetch({ name, cmd, args, cwd: root, env, logPath: join(prefetched, `${name}.log`) });
prefetches.push(handle);
return handle.done;
};
let phaseStarted = performance.now();
const phase = name => { const now = performance.now(); timing[`${name}_ms`] = Math.round(now - phaseStarted); phaseStarted = now; };
const awaited = async (name, promise) => { const started = performance.now(); await promise; timing[`${name}_wait_ms`] = Math.round(performance.now() - started); };
try {
const imageReady = prefetch('image', 'docker', ['pull', '--platform', 'linux/amd64', GROK_PUBLIC_INSTALL_IMAGE]);
// Provision as the unprivileged verification user, never into the host's /opt.
// Only the positive probe sees this file at the canonical sandbox path.
const grokReady = prefetch('grok', process.execPath, [join(repo, 'packages/paperclip-runner/scripts/provision-grok.mjs'), stagedGrok]);
const listing = run(process.execPath, [join(repo, 'scripts/release-package-map.mjs'), 'list'], repo).toString().trim().split('\n').map(line => line.split('\t'));
const packages = new Map(listing.map(([dir, name]) => [name, { dir, manifest: JSON.parse(readFileSync(join(repo, dir, 'package.json'), 'utf8')) }]));
const needed = new Set();
@@ -81,9 +106,12 @@ try {
const sentinel = join(consumer, 'node_modules/paperclip-verification-lifecycle-sentinel/lifecycle-ran');
const consumerUid = process.getuid();
const isolated = (command, options = {}) => run('docker', grokConsumerDockerArgs({ assets, consumer, cache, uid: consumerUid, gid: process.getgid(), command, ...options }));
phase('stage_packages');
await awaited('image', imageReady);
// npm resolution is intentionally the public consumer graph, not the pnpm
// workspace graph. Freeze that result before any lifecycle script can run.
isolated(['npm', 'install', '--ignore-scripts', '--omit=dev', ...readdirSync(assets).filter(file => file.endsWith('.tgz')).map(file => `/packages/${file}`)], { download: true });
phase('consumer_install');
assert.equal(existsSync(sentinel), false, 'Dependency download must not run lifecycle scripts');
const consumerLock = readFileSync(join(consumer, 'package-lock.json'), 'utf8');
// npm ci rejects bundled optional platform dependencies absent from its own
@@ -92,6 +120,7 @@ try {
isolated(GROK_PUBLIC_INSTALL_LIFECYCLE);
assert.equal(readFileSync(sentinel, 'utf8'), 'ok', 'Offline lifecycle scripts must actually execute');
assert.equal(readFileSync(join(consumer, 'package-lock.json'), 'utf8'), consumerLock, 'Lifecycle execution must preserve the resolved consumer lock');
phase('lifecycle_rebuild');
for (const name of needed) {
const installedManifest = JSON.parse(readFileSync(join(consumer, 'node_modules', name, 'package.json'), 'utf8'));
assert.equal(installedManifest.version, releaseVersion, `Installed release version for ${name}`);
@@ -104,6 +133,7 @@ try {
writeFileSync(join(assets, 'codex-probe.mjs'), installedCodexProbeSource(
'/consumer/node_modules/@paperclipai/server/dist/vendor/paperclip-runner/index.js', '/consumer', codexVersion), { mode: 0o644 });
const codex = JSON.parse(isolated(['node', '/packages/codex-probe.mjs']).toString().trim());
phase('codex_probe');
assert.equal(codex.codexConsumerHostOnly, true);
assert.equal(codex.codexPlatformPackageSource, 'official npm');
assert.ok(existsSync(join(installed, 'providers/grok/launcher.cjs')));
@@ -122,10 +152,14 @@ try {
`;
writeFileSync(join(assets, 'probe.mjs'), probe, { mode: 0o644 });
isolated(['node', '/packages/probe.mjs', 'missing']);
// Provision as the unprivileged verification user, never into the host's /opt.
// Only the positive probe sees this file at the canonical sandbox path.
run(process.execPath, [join(repo, 'packages/paperclip-runner/scripts/provision-grok.mjs'), prerequisite]);
// The binary was provisioned outside `native/grok` while the packages were
// staged, so the assertion above still proves npm put nothing there. It moves
// into place only now, so only the positive probe sees it.
await awaited('grok', grokReady);
mkdirSync(dirname(prerequisite), { recursive: true });
renameSync(stagedGrok, prerequisite);
isolated(['node', '/packages/probe.mjs', 'present'], { prerequisite });
phase('grok_probes');
console.log(JSON.stringify({ schema: 'paperclip.grok.public-npm-install.v1', sourceRevision, releaseVersion, lifecycleScriptsEnabled: true, lifecycleSentinelVerified: true, lifecycleNetwork: 'none', consumerImage: GROK_PUBLIC_INSTALL_IMAGE, consumerUid, consumerLockPreserved: true, cleanNpmInstall: true, packageCount: needed.size, codexPackedPayloadsOmitted: true, codexPackedPackagesChecked: tarballs.length, ...codex, builtinLauncherPresent: true, separateGrokPackage: false, npmProvisionedBinary: false, missingPrerequisiteRejected: true, provisionedBinaryVerified: true, commandLeaseVerified: true, providerCalls: 0 }));
// Exercise Pi's public CLI and installed server, never a private workspace
// package or binary override. Public dependency downloads are explicit and
@@ -137,6 +171,7 @@ try {
// temporary capacity than the smaller offline lifecycle probes.
const setup = isolated(['node', '/consumer/node_modules/paperclipai/dist/index.js', 'runtime', 'setup', 'pi'], { download: true, temporarySizeMiB: 2048 }).toString();
const receipt = JSON.parse(setup.trim());
phase('pi_setup');
assert.equal(receipt.status, 'installed_verified');
assert.equal(receipt.target, 'linux-x64');
assert.equal(readFileSync(join(consumer, 'package-lock.json'), 'utf8'), consumerLock, 'Pi setup must preserve the consumer dependency graph');
@@ -144,6 +179,13 @@ try {
// Docker defaults tmpfs to noexec. The offline admission probe must execute
// its verified private snapshot while keeping lifecycle/download scratch noexec.
console.log(isolated(['node', '/packages/pi-public-install-probe.mjs', '/consumer/node_modules/@paperclipai/server'], { temporarySizeMiB: 2048, temporaryExecutable: true }).toString().trim());
phase('pi_probe');
// Wall time per phase, so the slow part of this step is visible in the CI log
// without re-running it. `*_prefetch_ms` runs from a download's start to its
// last output, and `*_wait_ms` is how long the main flow waited for a
// prefetch that had not finished yet; zero means the overlap hid it entirely.
const downloads = Object.fromEntries(prefetches.map(({ name, durationMs }) => [`${name}_prefetch_ms`, durationMs]));
console.log(JSON.stringify({ schema: 'paperclip.public-npm-install.timing.v1', ...timing, ...downloads }));
if (process.env.PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR) {
const retained = retainRunnerQualificationPackages({ repo, output: process.env.PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR, sourceRevision, releaseVersion, env,
publicArchives: [...needed].map((name, index) => ({ name, file: tarballs[index] })),
@@ -151,5 +193,8 @@ try {
console.log(JSON.stringify(retained));
}
} finally {
// An earlier failure must not leave a prefetch running, and scratch is
// removed only once both children have stopped writing into it.
await stopPrefetches(prefetches);
rmSync(root, { recursive: true, force: true });
}