fix(runner): retry stalled Pi Node archive downloads (#18126)

<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/230/changes/paperclip/230
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The master Docker image contains a runner provider pack, and the
pack contains the Pi agent runtime
> - The Pi materializer downloads a hash-pinned Node archive from
nodejs.org during the image build
> - That download has one 60 s timeout and no retry, so one stalled
transfer stops the full amd64 image build
> - Master Docker builds failed this way two times in two days, and the
failure did not identify the request
> - This pull request retries only transport failures of that download
and keeps every security gate
> - The benefit is fewer red master image builds from short nodejs.org
stalls, and a clear error when all attempts fail

## Linked Issues or Issue Description

No public issue exists. This is a bug report.

**What happened?**
The `Docker` workflow on `master` failed in the `build-and-push
(linux/amd64)` job at `Dockerfile:163` (`runner-provider-pack`).
`build-provider-pack.mjs` stopped with an uncaught `DOMException
[TimeoutError]: The operation was aborted due to timeout`. The arm64 job
of the same commit passed. The next master build passed without a code
change.
- https://github.com/paperclipai/paperclip/actions/runs/40686298026
(2026-10-28)
- https://github.com/paperclipai/paperclip/actions/runs/40808852599
(2026-10-29)

**Steps to reproduce**
1. Run `node
packages/paperclip-runner/scripts/materialize-pi-distribution.mjs
/tmp/pi-out` on linux-x64.
2. Make the transfer from nodejs.org stall for more than 60 s.
3. The script stops at once with an anonymous `TimeoutError`.

**Expected behavior**
A short network stall does not stop the build. If all attempts fail, the
error identifies the Pi Node download.

**Evidence**
In both failed jobs, the uncaught timeout came about 60 s after the last
provider-pack output line (61.18 s → 121.6 s, and 38.72 s → 99.15 s).
The Pi Node archive fetch is the only provider download with a 60 s
timeout. The Cursor and Copilot downloads use 120–180 s.

## What Changed

- `materialize-pi-distribution.mjs`: add `downloadPinnedPiNodeArchive`.
It tries the download up to 3 times. Each attempt has a new 60 s
timeout. The timeout covers the headers and the full body. The backoff
is 5 s, then 10 s.
- Only transport failures cause a retry: timeout, network `TypeError`,
HTTP 5xx and HTTP 429.
- These failures stop at once, with no retry: SHA-256 pin mismatch, the
128 MiB size bound, HTTP 4xx, a missing body, and cancellation. The
SHA-256 pin still checks every byte.
- The body of each rejected (non-2xx) response is canceled before the
next attempt, so a 5xx/429 reply does not hold its connection.
- Each failed attempt writes a warning to the build log, so a later
successful retry does not hide the failure. When all attempts fail, the
error is `Pinned Pi Node download failed after 3 attempts: <cause>`.
- `materialize-pi-distribution.test.mjs`: add tests for stalled headers
and a stalled body that end only when the per-attempt deadline signal
aborts, release of rejected responses before the next attempt, exhausted
attempts, HTTP 503, and the fail-closed cases.
- `package.json`: add `scripts/materialize-pi-distribution.test.mjs` to
the `test:typescript:prep` node:test list. Before this change, no CI
lane ran this test file.

## Verification

- `node --test
packages/paperclip-runner/scripts/materialize-pi-distribution.test.mjs`
→ 13 pass, 0 fail.
- Red checks: with the old materializer, the new tests fail. When the
body cancel is removed, the release test fails. When the deadline signal
is removed from the fetch call, the deadline tests fail.
- `cd packages/paperclip-runner && node
../../scripts/ci-profile/node-test.mjs
scripts/candidate-provider-pack.test.mjs
scripts/materialize-pi-distribution.test.mjs` → 20 pass, 0 fail.
- `node --check` passes on the changed script.
- I did not run the full `pnpm -r typecheck`, `pnpm test:run`, or `pnpm
build` locally, because the sandbox has no installed workspace
dependencies. The changed files are `.mjs` scripts and are not in the
runner `tsconfig.json`. CI runs the full checks.
- A reviewer can read the `Build and push by digest` log of the next
master Docker run. A transient stall now shows `Pinned Pi Node download
attempt 1/3 failed: ...; retrying`.

## Risks

- Low risk. The change affects only the Pi Node archive download. Two
callers use it: the Docker provider-pack build
(`candidate-provider-pack.mjs`) and `paperclipai runtime setup pi`
(`provision-pi.mjs`).
- The worst-case time before failure increases from 60 s to about 195 s
(3 × 60 s + 15 s of backoff). The Docker job has more time than this.
`provision-pi.mjs` has a 900 s setup deadline that it enforces through
`checkCancelled`. The download calls `checkCancelled` before each
attempt, so the deadline and SIGINT/SIGTERM still stop setup. A cancel
can wait for one backoff of at most 10 s.
- For `runtime setup pi`, a stall now prints one warning line on stderr
for each failed attempt. Normal operation prints nothing new.
- The retry does not weaken integrity. The same SHA-256 pin, size bound,
and `redirect: "error"` apply to each attempt. A mismatch does not cause
a retry.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, GitHub connector). It read the failed CI
logs, wrote the change and tests, and ran the tests locally.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Devin FoleyandClaude Opus 5.5 authored and GitHub committed 2026-10-28 22:31:36 -07:00
1 parent a8a038e659
commit 522c653d56
3 files changed
+144 -11

No files matched your search

+1 -1
View File
@@ -85,7 +85,7 @@
"typecheck:browser": "tsc -p tsconfig.browser.json --noEmit",
"test": "pnpm run test:typescript && pnpm run test:rust",
"test:typescript": "pnpm run test:typescript:prep && vitest run",
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/grok-profile-identity.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/controlled-evals/controller.test.mjs scripts/controlled-evals/portable.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs",
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/materialize-pi-distribution.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/grok-profile-identity.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/controlled-evals/controller.test.mjs scripts/controlled-evals/portable.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs",
"test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs",
"test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace",
"test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider",
@@ -3,6 +3,7 @@ import { createHash } from "node:crypto";
import { chmod, copyFile, lstat, mkdir, mkdtemp, readFile, readdir, realpath, rename, rm, unlink, writeFile } from "node:fs/promises";
import { stripTypeScriptTypes } from "node:module";
import { dirname, isAbsolute, join, resolve } from "node:path";
import { setTimeout as sleep } from "node:timers/promises";
import { fileURLToPath, pathToFileURL } from "node:url";
import { promisify } from "node:util";
import { PI_DISTRIBUTION_CLOSURE_SHA256 } from "../src/drivers/acpx/pi-closure-pins.ts";
@@ -132,6 +133,54 @@ export async function resolvePiBundledNpm(nodeRoot) {
return entry;
}
class RetryablePiNodeDownloadError extends Error {}
async function readPiNodeArchive(url, fetchImpl, timeoutMs) {
let response;
try { response = await fetchImpl(url, { redirect: "error", signal: AbortSignal.timeout(timeoutMs) }); }
catch (error) { throw error?.name === "TimeoutError" || error instanceof TypeError ? new RetryablePiNodeDownloadError(error.message, { cause: error }) : error; }
if (!response.ok) {
// Release the rejected response's connection before any retry starts.
await response.body?.cancel().catch(() => {});
const message = `Pinned Pi Node download returned HTTP ${response.status}`;
throw response.status >= 500 || response.status === 429 ? new RetryablePiNodeDownloadError(message) : new Error(message);
}
if (!response.body) throw new Error("Pinned Pi Node download failed");
const chunks = []; let length = 0;
try {
for await (const chunk of response.body) {
length += chunk.length; if (length > 128 * 1024 * 1024) throw new Error("Pi Node archive exceeds its bound");
chunks.push(Buffer.from(chunk));
}
} catch (error) {
// The timeout also bounds the body; a stalled or reset stream is transport loss.
throw error?.name === "TimeoutError" || error instanceof TypeError ? new RetryablePiNodeDownloadError(error.message, { cause: error }) : error;
}
return Buffer.concat(chunks);
}
/**
* nodejs.org transfers occasionally stall. Retry only transport failures; the
* release hash still gates every byte, and a mismatch or 4xx fails closed at once.
*/
export async function downloadPinnedPiNodeArchive({ url, archiveSha256, fetchImpl = fetch, attempts = 3, timeoutMs = 60_000, retryDelayMs = 5_000, warn = (message) => console.warn(message), checkCancelled = () => {} }) {
for (let attempt = 1; ; attempt++) {
checkCancelled();
let bytes;
try { bytes = await readPiNodeArchive(url, fetchImpl, timeoutMs); }
catch (error) {
if (!(error instanceof RetryablePiNodeDownloadError)) throw error;
if (attempt >= attempts) throw new Error(`Pinned Pi Node download failed after ${attempt} attempts: ${error.message}`, { cause: error });
// Keep each transient failure in the build log even when a retry succeeds.
warn(`Pinned Pi Node download attempt ${attempt}/${attempts} failed: ${error.message}; retrying`);
await sleep(retryDelayMs * attempt);
continue;
}
if (hash(bytes) !== archiveSha256) throw new Error("Pi Node archive does not match its release pin");
return bytes;
}
}
/** Build on the target platform. No lifecycle scripts, model requests, or auth. */
export async function materializePiDistribution({ outputRoot, nodeExecutable, npmExecutable = "npm", inputs, checkCancelled = () => {} }) {
// Cancellation is observed between bounded subprocesses. A setup signal must
@@ -159,15 +208,7 @@ export async function materializePiDistribution({ outputRoot, nodeExecutable, np
if (nodeExecutable) node = await realpath(nodeExecutable);
else {
const archiveName = `node-v${PI_NODE_VERSION}-${target}.tar.gz`;
const response = await fetch(`https://nodejs.org/dist/v${PI_NODE_VERSION}/${archiveName}`, { redirect: "error", signal: AbortSignal.timeout(60_000) });
if (!response.ok || !response.body) throw new Error("Pinned Pi Node download failed");
const chunks = []; let length = 0;
for await (const chunk of response.body) {
length += chunk.length; if (length > 128 * 1024 * 1024) throw new Error("Pi Node archive exceeds its bound");
chunks.push(Buffer.from(chunk));
}
const bytes = Buffer.concat(chunks);
if (hash(bytes) !== nodePin.archiveSha256) throw new Error("Pi Node archive does not match its release pin");
const bytes = await downloadPinnedPiNodeArchive({ url: `https://nodejs.org/dist/v${PI_NODE_VERSION}/${archiveName}`, archiveSha256: nodePin.archiveSha256, checkCancelled });
const archivePath = join(staging, archiveName); await writeFile(archivePath, bytes);
const nodeRoot = join(staging, "node-extract"); await mkdir(nodeRoot);
await runOwned("tar", ["-xzf", archivePath, "-C", nodeRoot, "--strip-components=1", `node-v${PI_NODE_VERSION}-${target}/bin/node`, `node-v${PI_NODE_VERSION}-${target}/lib/node_modules/npm`], { timeout: 30_000 });
@@ -5,7 +5,7 @@ import { mkdtemp, mkdir, readFile, readdir, realpath, rm, symlink, writeFile } f
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import test from "node:test";
import { assertPiNodeSystemDependencies, PI_BRACE_SECURITY_PATCH_SHA256, PI_DISTRIBUTION_PINS, materializePiDistribution, resolvePiBundledNpm, piDistributionInstallCommand, verifyLockedPiPackageGraph, writePiDistributionManifest } from "./materialize-pi-distribution.mjs";
import { assertPiNodeSystemDependencies, downloadPinnedPiNodeArchive, PI_BRACE_SECURITY_PATCH_SHA256, PI_DISTRIBUTION_PINS, materializePiDistribution, resolvePiBundledNpm, piDistributionInstallCommand, verifyLockedPiPackageGraph, writePiDistributionManifest } from "./materialize-pi-distribution.mjs";
import { verifyPiRuntimeManifest } from "../src/drivers/acpx/pi-verified-runtime.ts";
async function fixture(t) {
@@ -137,3 +137,95 @@ test("setup selects the exact archive npm and rejects wrong versions or linked e
await symlink(join(root, prefix, "package.json"), join(root, prefix, "bin/npm-cli.js"));
await assert.rejects(resolvePiBundledNpm(root), /invalid archive entry/);
});
test("Node archive download retries transport stalls and records each failed attempt", async () => {
const archive = Buffer.from("pinned node archive");
const archiveSha256 = createHash("sha256").update(archive).digest("hex");
const stalledBody = new ReadableStream({ start(controller) { controller.enqueue(new Uint8Array([1])); controller.error(new DOMException("The operation was aborted due to timeout", "TimeoutError")); } });
const replies = [
() => Promise.reject(new DOMException("The operation was aborted due to timeout", "TimeoutError")),
() => Promise.resolve(new Response(stalledBody)),
() => Promise.resolve(new Response(archive)),
];
const signals = []; const warnings = [];
const bytes = await downloadPinnedPiNodeArchive({
url: "https://nodejs.org/dist/fixture.tar.gz", archiveSha256, retryDelayMs: 0, warn: (message) => warnings.push(message),
fetchImpl: (url, init) => { signals.push(init.signal); assert.equal(init.redirect, "error"); return replies.shift()(); },
});
assert.deepEqual(bytes, archive);
assert.equal(signals.length, 3);
assert.equal(new Set(signals).size, 3, "each attempt needs a fresh timeout");
assert.equal(warnings.length, 2);
assert.match(warnings[0], /attempt 1\/3 failed: The operation was aborted due to timeout; retrying/);
});
test("Node archive download deadline aborts stalled headers and stalled bodies", async () => {
const archive = Buffer.from("pinned node archive");
const archiveSha256 = createHash("sha256").update(archive).digest("hex");
const abortWith = (signal, fail) => signal.addEventListener("abort", () => fail(signal.reason), { once: true });
const replies = [
// Headers never arrive; only the supplied deadline can end this attempt.
(signal) => new Promise((_, reject) => abortWith(signal, reject)),
// Headers arrive, then the body stalls after one chunk until the same deadline fires.
(signal) => Promise.resolve(new Response(new ReadableStream({ start(controller) { controller.enqueue(new Uint8Array([1])); abortWith(signal, (reason) => controller.error(reason)); } }))),
() => Promise.resolve(new Response(archive)),
];
const warnings = [];
const bytes = await downloadPinnedPiNodeArchive({
url: "https://nodejs.org/dist/fixture.tar.gz", archiveSha256, timeoutMs: 20, retryDelayMs: 0, warn: (message) => warnings.push(message),
fetchImpl: (url, init) => replies.shift()(init.signal),
});
assert.deepEqual(bytes, archive);
assert.equal(warnings.length, 2);
for (const warning of warnings) assert.match(warning, /The operation was aborted due to timeout; retrying/);
});
test("Node archive download releases a rejected response before the next attempt", async () => {
const archive = Buffer.from("pinned node archive");
const archiveSha256 = createHash("sha256").update(archive).digest("hex");
const events = [];
const busy = () => new Response(new ReadableStream({ cancel() { events.push("cancel"); } }), { status: 503 });
const replies = [busy, () => new Response(archive)];
await downloadPinnedPiNodeArchive({
url: "https://nodejs.org/dist/fixture.tar.gz", archiveSha256, retryDelayMs: 0, warn: () => {},
fetchImpl: () => { events.push("fetch"); return Promise.resolve(replies.shift()()); },
});
assert.deepEqual(events, ["fetch", "cancel", "fetch"]);
events.length = 0;
await assert.rejects(downloadPinnedPiNodeArchive({
url: "https://nodejs.org/dist/fixture.tar.gz", archiveSha256, retryDelayMs: 0, warn: () => assert.fail("must not retry"),
fetchImpl: () => { events.push("fetch"); return Promise.resolve(new Response(new ReadableStream({ cancel() { events.push("cancel"); } }), { status: 404 })); },
}), /returned HTTP 404/);
assert.deepEqual(events, ["fetch", "cancel"]);
});
test("Node archive download names the exhausted request instead of an anonymous timeout", async () => {
let calls = 0;
await assert.rejects(downloadPinnedPiNodeArchive({
url: "https://nodejs.org/dist/fixture.tar.gz", archiveSha256: "0".repeat(64), retryDelayMs: 0, warn: () => {},
fetchImpl: () => { calls++; return Promise.reject(new TypeError("fetch failed")); },
}), /^Error: Pinned Pi Node download failed after 3 attempts: fetch failed$/);
assert.equal(calls, 3);
calls = 0;
await assert.rejects(downloadPinnedPiNodeArchive({
url: "https://nodejs.org/dist/fixture.tar.gz", archiveSha256: "0".repeat(64), retryDelayMs: 0, warn: () => {},
fetchImpl: () => { calls++; return Promise.resolve(new Response("busy", { status: 503 })); },
}), /failed after 3 attempts: Pinned Pi Node download returned HTTP 503/);
assert.equal(calls, 3);
});
test("Node archive download fails closed without retrying pin, bound, client or cancellation failures", async () => {
const options = { url: "https://nodejs.org/dist/fixture.tar.gz", archiveSha256: "0".repeat(64), retryDelayMs: 0, warn: () => assert.fail("must not retry") };
let calls = 0;
const once = (response) => () => { calls++; return Promise.resolve(response()); };
await assert.rejects(downloadPinnedPiNodeArchive({ ...options, fetchImpl: once(() => new Response("tampered")) }), /does not match its release pin/);
await assert.rejects(downloadPinnedPiNodeArchive({ ...options, fetchImpl: once(() => new Response("missing", { status: 404 })) }), /returned HTTP 404/);
const oversized = new ReadableStream({ start(controller) { controller.enqueue(new Uint8Array(128 * 1024 * 1024 + 1)); controller.close(); } });
await assert.rejects(downloadPinnedPiNodeArchive({ ...options, fetchImpl: once(() => new Response(oversized)) }), /exceeds its bound/);
assert.equal(calls, 3);
let checks = 0;
await assert.rejects(downloadPinnedPiNodeArchive({
...options, warn: () => {}, checkCancelled: () => { if (++checks > 1) throw new Error("cancelled"); },
fetchImpl: () => Promise.reject(new DOMException("The operation was aborted due to timeout", "TimeoutError")),
}), /cancelled/);
});