Files
PaperClipAI/packages/paperclip-runner/scripts/materialize-pi-distribution.mjs
T
Devin FoleyandClaude Opus 5.5 522c653d56 fix(runner): retry stalled Pi Node archive downloads (#18126)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/230/changes/paperclip/230
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The master Docker image contains a runner provider pack, and the
pack contains the Pi agent runtime
> - The Pi materializer downloads a hash-pinned Node archive from
nodejs.org during the image build
> - That download has one 60 s timeout and no retry, so one stalled
transfer stops the full amd64 image build
> - Master Docker builds failed this way two times in two days, and the
failure did not identify the request
> - This pull request retries only transport failures of that download
and keeps every security gate
> - The benefit is fewer red master image builds from short nodejs.org
stalls, and a clear error when all attempts fail

## Linked Issues or Issue Description

No public issue exists. This is a bug report.

**What happened?**
The `Docker` workflow on `master` failed in the `build-and-push
(linux/amd64)` job at `Dockerfile:163` (`runner-provider-pack`).
`build-provider-pack.mjs` stopped with an uncaught `DOMException
[TimeoutError]: The operation was aborted due to timeout`. The arm64 job
of the same commit passed. The next master build passed without a code
change.
- https://github.com/paperclipai/paperclip/actions/runs/40686298026
(2026-10-28)
- https://github.com/paperclipai/paperclip/actions/runs/40808852599
(2026-10-29)

**Steps to reproduce**
1. Run `node
packages/paperclip-runner/scripts/materialize-pi-distribution.mjs
/tmp/pi-out` on linux-x64.
2. Make the transfer from nodejs.org stall for more than 60 s.
3. The script stops at once with an anonymous `TimeoutError`.

**Expected behavior**
A short network stall does not stop the build. If all attempts fail, the
error identifies the Pi Node download.

**Evidence**
In both failed jobs, the uncaught timeout came about 60 s after the last
provider-pack output line (61.18 s → 121.6 s, and 38.72 s → 99.15 s).
The Pi Node archive fetch is the only provider download with a 60 s
timeout. The Cursor and Copilot downloads use 120–180 s.

## What Changed

- `materialize-pi-distribution.mjs`: add `downloadPinnedPiNodeArchive`.
It tries the download up to 3 times. Each attempt has a new 60 s
timeout. The timeout covers the headers and the full body. The backoff
is 5 s, then 10 s.
- Only transport failures cause a retry: timeout, network `TypeError`,
HTTP 5xx and HTTP 429.
- These failures stop at once, with no retry: SHA-256 pin mismatch, the
128 MiB size bound, HTTP 4xx, a missing body, and cancellation. The
SHA-256 pin still checks every byte.
- The body of each rejected (non-2xx) response is canceled before the
next attempt, so a 5xx/429 reply does not hold its connection.
- Each failed attempt writes a warning to the build log, so a later
successful retry does not hide the failure. When all attempts fail, the
error is `Pinned Pi Node download failed after 3 attempts: <cause>`.
- `materialize-pi-distribution.test.mjs`: add tests for stalled headers
and a stalled body that end only when the per-attempt deadline signal
aborts, release of rejected responses before the next attempt, exhausted
attempts, HTTP 503, and the fail-closed cases.
- `package.json`: add `scripts/materialize-pi-distribution.test.mjs` to
the `test:typescript:prep` node:test list. Before this change, no CI
lane ran this test file.

## Verification

- `node --test
packages/paperclip-runner/scripts/materialize-pi-distribution.test.mjs`
→ 13 pass, 0 fail.
- Red checks: with the old materializer, the new tests fail. When the
body cancel is removed, the release test fails. When the deadline signal
is removed from the fetch call, the deadline tests fail.
- `cd packages/paperclip-runner && node
../../scripts/ci-profile/node-test.mjs
scripts/candidate-provider-pack.test.mjs
scripts/materialize-pi-distribution.test.mjs` → 20 pass, 0 fail.
- `node --check` passes on the changed script.
- I did not run the full `pnpm -r typecheck`, `pnpm test:run`, or `pnpm
build` locally, because the sandbox has no installed workspace
dependencies. The changed files are `.mjs` scripts and are not in the
runner `tsconfig.json`. CI runs the full checks.
- A reviewer can read the `Build and push by digest` log of the next
master Docker run. A transient stall now shows `Pinned Pi Node download
attempt 1/3 failed: ...; retrying`.

## Risks

- Low risk. The change affects only the Pi Node archive download. Two
callers use it: the Docker provider-pack build
(`candidate-provider-pack.mjs`) and `paperclipai runtime setup pi`
(`provision-pi.mjs`).
- The worst-case time before failure increases from 60 s to about 195 s
(3 × 60 s + 15 s of backoff). The Docker job has more time than this.
`provision-pi.mjs` has a 900 s setup deadline that it enforces through
`checkCancelled`. The download calls `checkCancelled` before each
attempt, so the deadline and SIGINT/SIGTERM still stop setup. A cancel
can wait for one backoff of at most 10 s.
- For `runtime setup pi`, a stall now prints one warning line on stderr
for each failed attempt. Normal operation prints nothing new.
- The retry does not weaken integrity. The same SHA-256 pin, size bound,
and `redirect: "error"` apply to each attempt. A mismatch does not cause
a retry.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, GitHub connector). It read the failed CI
logs, wrote the change and tests, and ran the tests locally.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:31:36 -07:00

318 lines
23 KiB
JavaScript

import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { chmod, copyFile, lstat, mkdir, mkdtemp, readFile, readdir, realpath, rename, rm, unlink, writeFile } from "node:fs/promises";
import { stripTypeScriptTypes } from "node:module";
import { dirname, isAbsolute, join, resolve } from "node:path";
import { setTimeout as sleep } from "node:timers/promises";
import { fileURLToPath, pathToFileURL } from "node:url";
import { promisify } from "node:util";
import { PI_DISTRIBUTION_CLOSURE_SHA256 } from "../src/drivers/acpx/pi-closure-pins.ts";
import { PI_NODE_DISTRIBUTIONS, PI_NODE_VERSION } from "../src/drivers/acpx/pi-node-pins.ts";
import { buildNodeStartupTimeout } from "./build-node-startup-timeout.mjs";
import acpxProfiles from "../acpx-profiles.json" with { type: "json" };
import { inventoryPiRuntimeFiles, verifyPiRuntimeManifest } from "../src/drivers/acpx/pi-verified-runtime.ts";
const run = promisify(execFile);
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
const workspaceRoot = resolve(packageRoot, "../..");
const lockDirectory = join(packageRoot, "scripts/pi-distribution");
const patchPath = join(workspaceRoot, "patches/pi-acp@0.0.33.patch");
const securityPatchPath = join(workspaceRoot, "patches/brace-expansion@5.0.9.patch");
export const PI_BRACE_SECURITY_PATCH_SHA256 = "5273a195b952f233336122842faaf650193c04f36b9359d20d4d63079b0ce814";
const supportedTargets = new Set(["darwin-arm64", "darwin-x64", "linux-x64"]);
export const PI_DISTRIBUTION_PINS = Object.freeze({
wrapper: "0.0.33", runtime: "1.0.0", sdk: "0.26.0", zod: "3.25.76", nodeVersion: PI_NODE_VERSION, undici: "8.10.2", nodeBundledUndici: "7.29.1",
wrapperSha256: "9d129b3d38772e93e97080aa6c4e574ac5df4e47ce5bc331a484a9fbf8188b36",
helperSha256: "41e0490b617da0d60e0c8ec58ef311236f945129d99f816cff6897f78da7f91d",
});
const hash = (bytes) => createHash("sha256").update(bytes).digest("hex");
/** Install the published Pi graph; then apply the one pinned security correction. */
export function piDistributionInstallCommand() {
return ["ci", "--ignore-scripts", "--include=optional", "--omit=dev", "--no-audit", "--no-fund", "--registry=https://registry.npmjs.org", "--userconfig=.npmrc", "--globalconfig=.npmrc-global"];
}
function supports(list, current) {
return !Array.isArray(list) || (list.includes(current) || !list.some((value) => !value.startsWith("!"))) && !list.includes(`!${current}`);
}
/** Prove installed package versions and the complete published Pi graph. */
export async function verifyLockedPiPackageGraph(root, lock, target = { platform: process.platform, architecture: process.arch }) {
if (lock.lockfileVersion !== 3 || !lock.packages || typeof lock.packages !== "object") throw new Error("Pi distribution lock is invalid");
let verified = 0;
for (const [path, entry] of Object.entries(lock.packages)) {
if (!path) continue;
if (!path.startsWith("node_modules/") || path.split("/").some((part) => part === ".." || part === "." || !part) || path.includes("\\") || entry.link || typeof entry.version !== "string" || !/^https:\/\/registry\.npmjs\.org\//.test(entry.resolved ?? "") || !/^sha512-[A-Za-z0-9+/]+=*$/.test(entry.integrity ?? "")) throw new Error("Pi distribution lock contains an unpinned package");
const compatible = supports(entry.os, target.platform) && supports(entry.cpu, target.architecture);
let metadata;
try { metadata = JSON.parse(await readFile(join(root, path, "package.json"), "utf8")); }
catch (error) {
if (error.code === "ENOENT" && entry.optional === true && !compatible) continue;
throw new Error(`Pi distribution is missing locked package ${path}`);
}
if (metadata.version !== entry.version) throw new Error(`Pi distribution package differs from its lock: ${path}`);
verified++;
}
const piPath = "node_modules/@earendil-works/pi-coding-agent";
const shrinkwrap = JSON.parse(await readFile(join(root, piPath, "npm-shrinkwrap.json"), "utf8"));
if (shrinkwrap.version !== PI_DISTRIBUTION_PINS.runtime || shrinkwrap.lockfileVersion !== 3) throw new Error("Pi upstream shrinkwrap is missing or changed");
for (const [path, entry] of Object.entries(shrinkwrap.packages)) {
if (!path) continue;
const pinned = lock.packages[`${piPath}/${path}`];
// Pi bundles its dependency payload, so npm overrides do not replace it.
// Only this reviewed build-owned patch may differ from the upstream graph.
if (path === "node_modules/brace-expansion" && entry.version === "5.0.9"
&& entry.integrity === "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg=="
&& entry.resolved === "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz"
&& pinned?.version === "5.0.12"
&& pinned.integrity === "sha512-YovQ3rzhaLMIrDjNDMkNS01tea93qhEhG5xy8f6+R0l+dw3Ki+5sCoIoI942iuLZTHWogWktgwVDhU09iNEimQ=="
&& pinned.resolved === "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.12.tgz") continue;
if (!pinned || pinned.version !== entry.version || (entry.integrity !== undefined && pinned.integrity !== entry.integrity) || pinned.resolved !== entry.resolved) throw new Error(`Pi distribution dropped or changed shrinkwrapped package ${path}`);
}
return verified;
}
/** Only OS libraries may remain outside the inventoried Node executable. */
export function assertPiNodeSystemDependencies(listing, platform) {
const lines = listing.split("\n").map((line) => line.trim()).filter(Boolean);
if (platform === "darwin") {
for (const line of lines.slice(1)) {
const path = line.split(" (", 1)[0];
if (!path.startsWith("/usr/lib/") && !path.startsWith("/System/Library/")) throw new Error("Pi Node requires an unbundled non-system library");
}
if (lines.length < 2) throw new Error("Pi Node dependency inspection is empty");
} else {
if (listing.includes("not found")) throw new Error("Pi Node requires a missing system library");
for (const line of lines) {
if (line.startsWith("linux-vdso.") || line === "statically linked") continue;
const path = line.includes(" => ") ? line.split(" => ")[1].split(" ")[0] : line.split(" ")[0];
const name = path.slice(path.lastIndexOf("/") + 1);
if (!/^\/(?:usr\/)?lib(?:64)?\//.test(path) || !/^(?:lib(?:c|m|dl|pthread|gcc_s|stdc\+\+|rt|atomic)\.so(?:\.[0-9]+)*|ld-linux[^/]*\.so(?:\.[0-9]+)*)$/.test(name)) throw new Error("Pi Node requires an unbundled non-system library");
}
if (!lines.length) throw new Error("Pi Node dependency inspection is empty");
}
}
export async function writePiDistributionManifest(runtimeRoot) {
const manifest = {
schema: "paperclip.pi-runtime-files.v1",
node: "node/bin/node",
piEntrypoint: "node_modules/@earendil-works/pi-coding-agent/dist/cli.js",
extension: "extensions/paperclip.js",
wrapperEntrypoint: "node_modules/pi-acp/dist/index.js",
files: await inventoryPiRuntimeFiles(runtimeRoot),
};
const verified = await verifyPiRuntimeManifest(runtimeRoot, manifest);
return { manifest, ...verified };
}
export function piDistributionBootstrapSource() {
return [
'const path=require("node:path");',
'process.env.PAPERCLIP_PI_NODE_EXECUTABLE=path.join(__dirname,"node/bin/node");',
'process.env.PAPERCLIP_PI_ENTRYPOINT=path.join(__dirname,"node_modules/@earendil-works/pi-coding-agent/dist/cli.js");',
'process.env.PAPERCLIP_PI_EXTENSION_PATH=path.join(__dirname,"extensions/paperclip.js");',
'process.env.PAPERCLIP_PI_MODULE_GUARD_PATH=path.join(__dirname,".paperclip-native-module-guard.cjs");',
'const protectedRoots=JSON.parse(process.env.PAPERCLIP_PI_PROTECTED_ROOTS??"[]");if(!Array.isArray(protectedRoots))throw new Error("Invalid Pi protected roots");',
'process.env.PAPERCLIP_PI_PROTECTED_ROOTS=JSON.stringify([...protectedRoots,__dirname]);',
'import(require("node:url").pathToFileURL(path.join(__dirname,"node_modules/pi-acp/dist/index.js")).href).catch(()=>{process.exitCode=1;});',
'',
].join("\n");
}
/** The already hash-verified Node archive also pins setup's package manager. */
export async function resolvePiBundledNpm(nodeRoot) {
const npmRoot = join(nodeRoot, "lib/node_modules/npm");
const manifest = join(npmRoot, "package.json");
const entry = join(npmRoot, "bin/npm-cli.js");
for (const path of [manifest, entry]) {
const info = await lstat(path);
if (!info.isFile() || info.isSymbolicLink() || info.nlink !== 1 || await realpath(path) !== path) throw new Error("Pinned Pi npm has an invalid archive entry");
}
if (JSON.parse(await readFile(manifest, "utf8")).version !== "11.19.0") throw new Error("Pinned Pi Node archive has an unexpected npm version");
return entry;
}
class RetryablePiNodeDownloadError extends Error {}
async function readPiNodeArchive(url, fetchImpl, timeoutMs) {
let response;
try { response = await fetchImpl(url, { redirect: "error", signal: AbortSignal.timeout(timeoutMs) }); }
catch (error) { throw error?.name === "TimeoutError" || error instanceof TypeError ? new RetryablePiNodeDownloadError(error.message, { cause: error }) : error; }
if (!response.ok) {
// Release the rejected response's connection before any retry starts.
await response.body?.cancel().catch(() => {});
const message = `Pinned Pi Node download returned HTTP ${response.status}`;
throw response.status >= 500 || response.status === 429 ? new RetryablePiNodeDownloadError(message) : new Error(message);
}
if (!response.body) throw new Error("Pinned Pi Node download failed");
const chunks = []; let length = 0;
try {
for await (const chunk of response.body) {
length += chunk.length; if (length > 128 * 1024 * 1024) throw new Error("Pi Node archive exceeds its bound");
chunks.push(Buffer.from(chunk));
}
} catch (error) {
// The timeout also bounds the body; a stalled or reset stream is transport loss.
throw error?.name === "TimeoutError" || error instanceof TypeError ? new RetryablePiNodeDownloadError(error.message, { cause: error }) : error;
}
return Buffer.concat(chunks);
}
/**
* nodejs.org transfers occasionally stall. Retry only transport failures; the
* release hash still gates every byte, and a mismatch or 4xx fails closed at once.
*/
export async function downloadPinnedPiNodeArchive({ url, archiveSha256, fetchImpl = fetch, attempts = 3, timeoutMs = 60_000, retryDelayMs = 5_000, warn = (message) => console.warn(message), checkCancelled = () => {} }) {
for (let attempt = 1; ; attempt++) {
checkCancelled();
let bytes;
try { bytes = await readPiNodeArchive(url, fetchImpl, timeoutMs); }
catch (error) {
if (!(error instanceof RetryablePiNodeDownloadError)) throw error;
if (attempt >= attempts) throw new Error(`Pinned Pi Node download failed after ${attempt} attempts: ${error.message}`, { cause: error });
// Keep each transient failure in the build log even when a retry succeeds.
warn(`Pinned Pi Node download attempt ${attempt}/${attempts} failed: ${error.message}; retrying`);
await sleep(retryDelayMs * attempt);
continue;
}
if (hash(bytes) !== archiveSha256) throw new Error("Pi Node archive does not match its release pin");
return bytes;
}
}
/** Build on the target platform. No lifecycle scripts, model requests, or auth. */
export async function materializePiDistribution({ outputRoot, nodeExecutable, npmExecutable = "npm", inputs, checkCancelled = () => {} }) {
// Cancellation is observed between bounded subprocesses. A setup signal must
// not abandon an npm/tar child while it still owns staging files.
const runOwned = async (...args) => { checkCancelled(); const result = await run(...args); checkCancelled(); return result; };
checkCancelled();
const inputLockDirectory = inputs?.lockDirectory ?? lockDirectory;
const inputPatchPath = inputs?.patchPath ?? patchPath;
const inputSecurityPatchPath = inputs?.securityPatchPath ?? securityPatchPath;
const helperSourcePath = inputs?.helperSourcePath ?? join(packageRoot, "src/drivers/acpx/pi-acp-runtime.ts");
const extensionSourcePath = inputs?.extensionSourcePath ?? join(packageRoot, "src/drivers/acpx/pi-runtime-extension.ts");
if (typeof outputRoot !== "string" || !outputRoot || !isAbsolute(outputRoot)) throw new Error("Pi distribution output must be absolute");
const output = resolve(outputRoot);
if (["/", workspaceRoot, packageRoot].includes(output)) throw new Error("Refusing unsafe Pi distribution output");
if (!supportedTargets.has(`${process.platform}-${process.arch}`)) throw new Error("Pi distribution target is unsupported");
try { await lstat(output); throw new Error("Pi distribution output already exists"); } catch (error) { if (error.code !== "ENOENT") throw error; }
await mkdir(dirname(output), { recursive: true });
const staging = await mkdtemp(join(await realpath(dirname(output)), ".paperclip-pi-distribution-"));
const runtimeRoot = join(staging, "runtime");
try {
const target = `${process.platform}-${process.arch}`;
const nodePin = PI_NODE_DISTRIBUTIONS[target];
let node;
let bundledNpm;
if (nodeExecutable) node = await realpath(nodeExecutable);
else {
const archiveName = `node-v${PI_NODE_VERSION}-${target}.tar.gz`;
const bytes = await downloadPinnedPiNodeArchive({ url: `https://nodejs.org/dist/v${PI_NODE_VERSION}/${archiveName}`, archiveSha256: nodePin.archiveSha256, checkCancelled });
const archivePath = join(staging, archiveName); await writeFile(archivePath, bytes);
const nodeRoot = join(staging, "node-extract"); await mkdir(nodeRoot);
await runOwned("tar", ["-xzf", archivePath, "-C", nodeRoot, "--strip-components=1", `node-v${PI_NODE_VERSION}-${target}/bin/node`, `node-v${PI_NODE_VERSION}-${target}/lib/node_modules/npm`], { timeout: 30_000 });
node = join(nodeRoot, "bin/node");
bundledNpm = await resolvePiBundledNpm(nodeRoot);
}
if (hash(await readFile(node)) !== nodePin.executableSha256 || (await lstat(node)).size !== nodePin.executableSize) throw new Error("Pi Node executable does not match its target release pin");
const version = (await runOwned(node, ["--version"], { env: {}, timeout: buildNodeStartupTimeout() })).stdout.trim();
if (version !== `v${PI_NODE_VERSION}`) throw new Error("Pi distribution requires exact pinned Node version");
if ((await runOwned(node, ["-p", "process.versions.undici"], { env: {}, timeout: buildNodeStartupTimeout() })).stdout.trim() !== PI_DISTRIBUTION_PINS.nodeBundledUndici) throw new Error("Pi Node bundled Undici differs from its reviewed security pin");
await mkdir(runtimeRoot);
await Promise.all(["package.json", "package-lock.json"].map((name) => copyFile(join(inputLockDirectory, name), join(runtimeRoot, name))));
await writeFile(join(runtimeRoot, ".npmrc"), "registry=https://registry.npmjs.org/\nignore-scripts=true\naudit=false\nfund=false\n");
await writeFile(join(runtimeRoot, ".npmrc-global"), "");
const buildHome = join(staging, "build-home"); await mkdir(buildHome);
// Do not inherit NPM_TOKEN, npm_config_*, NODE_OPTIONS, provider keys or user
// .npmrc. Public registry downloads need no private application credential.
const environment = Object.fromEntries(["PATH", "LANG", "LC_ALL", "HTTPS_PROXY", "HTTP_PROXY", "NO_PROXY", "https_proxy", "http_proxy", "no_proxy", "SSL_CERT_FILE", "SSL_CERT_DIR", "NODE_EXTRA_CA_CERTS"].flatMap((key) => typeof process.env[key] === "string" ? [[key, process.env[key]]] : []));
environment.HOME = buildHome;
// npm 10 prunes non-host packages bundled by upstream Pi, unlike the npm
// 11.19.0 used to qualify this complete closure. Public setup must use the
// npm pinned by the verified Node archive, never whichever npm is on PATH.
await runOwned(bundledNpm ? node : npmExecutable, [...(bundledNpm ? [bundledNpm] : []), ...piDistributionInstallCommand()], { cwd: runtimeRoot, env: environment, timeout: 300_000, maxBuffer: 4 * 1024 * 1024 });
const installedLockBytes = await readFile(join(runtimeRoot, "package-lock.json"));
if (!installedLockBytes.equals(await readFile(join(inputLockDirectory, "package-lock.json")))) throw new Error("Pi installation changed its committed lock");
const lock = JSON.parse(installedLockBytes.toString("utf8"));
if (hash(await readFile(inputSecurityPatchPath)) !== PI_BRACE_SECURITY_PATCH_SHA256) throw new Error("Pi dependency security patch differs from its reviewed pin");
const braceRoot = join(runtimeRoot, "node_modules/@earendil-works/pi-coding-agent/node_modules/brace-expansion");
await runOwned("git", ["apply", "--check", inputSecurityPatchPath], { cwd: braceRoot, env: environment, timeout: 10_000 });
await runOwned("git", ["apply", inputSecurityPatchPath], { cwd: braceRoot, env: environment, timeout: 10_000 });
const packageCount = await verifyLockedPiPackageGraph(runtimeRoot, lock);
const wrapper = join(runtimeRoot, "node_modules/pi-acp");
await runOwned("git", ["apply", "--check", inputPatchPath], { cwd: wrapper, env: environment, timeout: 10_000 });
await runOwned("git", ["apply", inputPatchPath], { cwd: wrapper, env: environment, timeout: 10_000 });
const [wrapperBytes, helperBytes, helperSource] = await Promise.all([
readFile(join(wrapper, "dist/index.js")), readFile(join(wrapper, "dist/paperclip-runtime.js")),
readFile(helperSourcePath, "utf8"),
]);
// Installed CLI users may run another supported Node patch. Generate the
// exact pinned closure with its verified Node, not the host's TS stripper.
const stripPinnedSource = async (path, source) => inputs
? (await runOwned(node, ["--input-type=module", "-e", 'import {readFileSync} from "node:fs"; import {stripTypeScriptTypes} from "node:module"; process.stdout.write(stripTypeScriptTypes(readFileSync(process.argv[1],"utf8")));', path], { env: {}, timeout: buildNodeStartupTimeout(), maxBuffer: 4 * 1024 * 1024 })).stdout
: stripTypeScriptTypes(source);
const stripped = (await stripPinnedSource(helperSourcePath, helperSource)).split("\n").map((line) => line.trimEnd()).join("\n");
if (hash(wrapperBytes) !== PI_DISTRIBUTION_PINS.wrapperSha256 || hash(helperBytes) !== PI_DISTRIBUTION_PINS.helperSha256 || helperBytes.toString("utf8") !== stripped) throw new Error("Pi wrapper patch does not match its qualified source");
await mkdir(join(runtimeRoot, "extensions"));
await writeFile(join(runtimeRoot, "extensions/paperclip.js"), (await stripPinnedSource(extensionSourcePath, await readFile(extensionSourcePath, "utf8"))).replace('from "./pi-acp-runtime.js"', 'from "../node_modules/pi-acp/dist/paperclip-runtime.js"'));
await mkdir(join(runtimeRoot, "node/bin"), { recursive: true });
const copiedNode = join(runtimeRoot, "node/bin/node");
await copyFile(node, copiedNode); await chmod(copiedNode, 0o755);
const dependencyListing = process.platform === "darwin"
? (await runOwned("/usr/bin/otool", ["-L", copiedNode], { env: {}, timeout: 10_000 })).stdout
: (await runOwned("ldd", [copiedNode], { env: { PATH: "/usr/bin:/bin" }, timeout: 10_000 })).stdout;
assertPiNodeSystemDependencies(dependencyListing, process.platform);
if ((await runOwned(copiedNode, ["--version"], { env: {}, timeout: buildNodeStartupTimeout() })).stdout.trim() !== version) throw new Error("Copied Pi Node cannot execute after relocation");
await rm(buildHome, { recursive: true });
// npm-generated .bin links and hidden lock metadata are not package payload
// files. No launch uses PATH; excluding them makes the closure regular-only.
for (const file of await inventoryPiRuntimeFiles(runtimeRoot)) {
if (file.kind === "symlink") {
if (!/(?:^|\/)node_modules\/\.bin\/[^/]+$/.test(file.path)) throw new Error("Pi package payload contains an unsupported symbolic link");
await unlink(join(runtimeRoot, file.path));
}
}
await rm(join(runtimeRoot, "node_modules/.package-lock.json"), { force: true });
await writeFile(join(runtimeRoot, "pi-entry.cjs"), piDistributionBootstrapSource());
for (const item of await readdir(staging)) if (item !== "runtime") await rm(join(staging, item), { recursive: true, force: true });
const { manifest, manifestDigest } = await writePiDistributionManifest(runtimeRoot);
const entries = await Promise.all(manifest.files.map(async (file) => {
if (file.kind !== "file") throw new Error("Pi native closure must contain regular files only");
const stat = await lstat(join(runtimeRoot, file.path));
return { path: file.path, sha256: file.sha256.slice("sha256:".length), size: stat.size, executable: Boolean(stat.mode & 0o111) };
}));
entries.sort((a, b) => a.path < b.path ? -1 : a.path > b.path ? 1 : 0);
const nativeClosureSha256 = hash(JSON.stringify(entries));
if (nativeClosureSha256 !== PI_DISTRIBUTION_CLOSURE_SHA256[target]) throw new Error("Pi native closure differs from its trusted target pin");
await writeFile(join(staging, "native-closure.json"), `${JSON.stringify({ entries })}\n`);
const metadata = {
schema: "paperclip.pi-distribution.v1", pins: PI_DISTRIBUTION_PINS,
target: { platform: process.platform, architecture: process.arch, nodeVersion: version.slice(1) },
packageCount, lockSha256: hash(await readFile(join(runtimeRoot, "package-lock.json"))), manifestDigest,
runtimeRoot: "runtime", nativeClosureSha256, manifest,
};
await writeFile(join(staging, "pi-distribution.json"), `${JSON.stringify(metadata, null, 2)}\n`);
checkCancelled();
await rename(staging, output);
const finalRoot = join(output, "runtime");
const binding = await verifyPiRuntimeManifest(finalRoot, manifest);
return {
version: PI_DISTRIBUTION_PINS.runtime,
profileDigest: acpxProfiles.profiles.pi.commandDigest,
closureDigest: `sha256:${nativeClosureSha256}`,
outputRoot: output, runtimeRoot: finalRoot, manifestPath: join(output, "pi-distribution.json"), metadata, ...binding,
};
} catch (error) { await rm(staging, { recursive: true, force: true }); throw error; }
}
if (import.meta.url.endsWith("/materialize-pi-distribution.mjs") && process.argv[1] && pathToFileURL(resolve(process.argv[1])).href === import.meta.url) {
const args = process.argv.slice(2).filter((arg) => arg !== "--");
const outputArgs = args.filter((arg) => !arg.startsWith("--node="));
const nodeArgs = args.filter((arg) => arg.startsWith("--node="));
if (outputArgs.length !== 1 || nodeArgs.length > 1) throw new Error("Usage: node scripts/materialize-pi-distribution.mjs /absolute/output-directory [--node=/absolute/portable-node]");
materializePiDistribution({ outputRoot: outputArgs[0], ...(nodeArgs.length ? { nodeExecutable: nodeArgs[0].slice("--node=".length) } : {}) }).then(result => {
process.stdout.write(`${JSON.stringify({ outputRoot: result.outputRoot, manifestPath: result.manifestPath, manifestDigest: result.manifestDigest, packageCount: result.metadata.packageCount })}\n`);
}).catch(error => { console.error(error.message); process.exitCode = 1; });
}