Files
PaperClipAI/scripts/prepare-bundled-package.mjs
T
DottaandPaperclip 0cc796b7bd Build isolated preview artifacts for exact-source deployments (#13041)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Managed deployments need a cloud image and a database migration
package.
> - Branch commits can lack both artifacts until a normal release runs.
> - Operators need to test an exact commit without advancing release
aliases.
> - This pull request adds a preview build mode to the existing release
workflow.
> - Builds use an immutable source SHA and publish isolated, reusable
artifacts.

## Linked Issues or Issue Description

**Subsystem affected**

Release automation, cloud Docker images, and shared/database npm
packages.

**Problem or motivation**

An operator cannot deploy an unpublished branch with new migrations
using only
the normal release artifacts. Publishing it through a normal lane would
also
advance shared release aliases.

**Proposed solution**

Dispatch the trusted release workflow on master with a full source SHA
and a
request UUID. Build missing SHA images and, when needed, deterministic
preview
shared/DB packages. Publish packages under the preview dist-tag with
exact
workspace pins. Reuse matching artifacts on retries.

**Roadmap alignment**

This extends release tooling for operator validation. It does not add a
core
product feature or duplicate a planned product capability. Related PR
searches
found no duplicate preview deployment workflow.

## What Changed

- Add the preview channel, request correlation, artifact checks, and
result artifact.
- Compile source packages in a separate job from the npm publisher. The
publisher
  uses trusted master code and disables package lifecycle scripts.
- Publish only SHA cloud image tags. Preserve release aliases. Use
full-SHA tags and no shared build cache.
- Verify full source identity for reused packages and images. Both image
and npm publishers
use isolated jobs and the externally master-restricted npm-canary
environment. Fail on registry
  authentication errors, outages, or artifact identity mismatches.
- Let bundled-package preparation use patches from the requested source
checkout.
- Document publishing configuration, artifact contracts, and deployment
order.

## Verification

- Passed `pnpm -r typecheck` and `pnpm build`.
- Passed `pnpm test:release-registry`: 107 tests, including eight
preview tests.
- Passed `actionlint -shellcheck= .github/workflows/release.yml`.
- Built real shared and DB preview tarballs from an isolated exact-SHA
checkout.
Verified package source identity and all 244 SQL files and journal
entries.
- Verified the full revision behind an existing published SHA cloud
image.
- `pnpm test:run` exposed missing local embedded PostgreSQL library
symlinks.
The package's postinstall repair restored initdb; all 12 previously
affected
suites passed on rerun (95 tests). Additional local matrix reruns are in
progress.
The complete PR CI matrix is green, including general/serialized tests,
e2e,
typecheck, build, release registry, canary dry run, and the required
verify gate.
- Live preview publication and staging deployment require this workflow
on master
and the compatible control-plane backend. They have not run yet. No
production
  deployment was performed.

## Risks

Preview npm versions are immutable public artifacts. Both packages must
retain
their trusted publisher for release.yml in environment npm-canary.
Source builds
must remain separated from privileged npm publishing. The deploying
control plane
must verify source identity, integrity, and migration compatibility
before use.

Normal release jobs retain their existing conditions. Roll back by
stopping preview
dispatches and reverting the workflow/tooling. Published preview
versions remain
isolated from normal release tags.

## Model Used

OpenAI GPT-6 through Codex, with repository tools, code execution, and
test runs.
The session does not expose a more specific model version or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with the available version and
capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue in-PR following the feature template
- [x] I have not referenced internal or instance-local issues or links
- [x] My branch name describes the change and contains no internal
ticket identifier
- [ ] I have run the full tests locally and they pass
- [x] I have added tests for the new behavior
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open recommendations or follow-ups
- [x] I will address review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-08 09:21:58 -05:00

223 lines
8.6 KiB
JavaScript

#!/usr/bin/env node
import { execFileSync } from "node:child_process";
import { cpSync, existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { resolve } from "node:path";
import { fileURLToPath } from "node:url";
const repoRoot = resolve(fileURLToPath(new URL("..", import.meta.url)));
export function materializePublishManifest(pkg) {
const publishConfig = pkg.publishConfig ?? {};
const publishManifest = { ...pkg };
for (const key of ["main", "types", "exports", "bin"]) {
if (publishConfig[key] !== undefined) publishManifest[key] = publishConfig[key];
}
for (const section of ["dependencies", "optionalDependencies", "peerDependencies"]) {
if (!publishManifest[section]) continue;
publishManifest[section] = Object.fromEntries(
Object.entries(publishManifest[section]).map(([name, specifier]) => {
if (typeof specifier !== "string" || !specifier.startsWith("workspace:")) return [name, specifier];
const range = specifier.slice("workspace:".length);
const prefix = range === "^" || range === "~" ? range : "";
return [name, `${prefix}${pkg.version}`];
}),
);
}
delete publishManifest.publishConfig;
return publishManifest;
}
export function createBundledInstallManifest(publishManifest, bundledDependencies) {
const bundledDependencyNames = new Set(bundledDependencies);
const installManifest = structuredClone(publishManifest);
delete installManifest.devDependencies;
for (const section of ["dependencies", "optionalDependencies", "peerDependencies"]) {
if (!installManifest[section]) continue;
installManifest[section] = Object.fromEntries(
Object.entries(installManifest[section]).filter(([name]) => bundledDependencyNames.has(name)),
);
if (Object.keys(installManifest[section]).length === 0) delete installManifest[section];
}
return installManifest;
}
function patchedDependencyPackageName(specifier) {
const versionSeparator = specifier.lastIndexOf("@");
const packageNameEnd = specifier.startsWith("@") ? specifier.indexOf("/") : 0;
if (packageNameEnd < 0) return specifier;
return versionSeparator > packageNameEnd ? specifier.slice(0, versionSeparator) : specifier;
}
export function selectBundledDependencyPatches(
destinationDir,
bundledDependencies,
patchedDependencies,
) {
const patchesByPackageName = new Map();
for (const [specifier, patchPath] of Object.entries(patchedDependencies)) {
const packageName = patchedDependencyPackageName(specifier);
const packagePatches = patchesByPackageName.get(packageName) ?? new Map();
packagePatches.set(specifier, patchPath);
patchesByPackageName.set(packageName, packagePatches);
}
const selectedPatches = [];
for (const packageName of new Set(bundledDependencies)) {
const packagePatches = patchesByPackageName.get(packageName);
if (!packagePatches) continue;
const installedManifestPath = resolve(
destinationDir,
"node_modules",
packageName,
"package.json",
);
let installedManifest;
try {
installedManifest = JSON.parse(readFileSync(installedManifestPath, "utf8"));
} catch (cause) {
throw new Error(
`Cannot select a patch for bundled dependency ${packageName}: failed to read ${installedManifestPath}`,
{ cause },
);
}
if (
installedManifest.name !== packageName ||
typeof installedManifest.version !== "string" ||
installedManifest.version.length === 0
) {
throw new Error(
`Cannot select a patch for bundled dependency ${packageName}: installed package manifest must declare the expected name and a version`,
);
}
const installedSpecifier = `${packageName}@${installedManifest.version}`;
const patchPath = packagePatches.get(installedSpecifier);
if (patchPath === undefined) {
const configuredSpecifiers = [...packagePatches.keys()].sort().join(", ");
throw new Error(
`Cannot select a patch for bundled dependency ${packageName}: installed ${installedSpecifier}, but configured patches are ${configuredSpecifiers}`,
);
}
if (typeof patchPath !== "string" || patchPath.length === 0) {
throw new Error(`Patch path for ${installedSpecifier} must be a non-empty string`);
}
selectedPatches.push({ packageName, specifier: installedSpecifier, patchPath });
}
return selectedPatches;
}
export function applyBundledDependencyPatches(destinationDir, bundledDependencies, sourceRoot = repoRoot) {
const rootPackage = JSON.parse(readFileSync(resolve(sourceRoot, "package.json"), "utf8"));
const patchedDependencies = rootPackage.pnpm?.patchedDependencies ?? {};
for (const { packageName, patchPath } of selectBundledDependencyPatches(
destinationDir,
bundledDependencies,
patchedDependencies,
)) {
execFileSync(
"patch",
["-p1", "--forward", "-d", resolve(destinationDir, "node_modules", packageName)],
{
input: readFileSync(resolve(sourceRoot, patchPath)),
stdio: ["pipe", "inherit", "inherit"],
},
);
}
}
export function prepareBundledPackage(sourceDir, destinationDir, { sourceRoot = repoRoot } = {}) {
const sourcePackagePath = resolve(sourceDir, "package.json");
const sourcePackage = JSON.parse(readFileSync(sourcePackagePath, "utf8"));
const bundledDependencies = sourcePackage.bundleDependencies ?? sourcePackage.bundledDependencies ?? [];
if (bundledDependencies.length === 0) {
throw new Error(`${sourcePackage.name} does not declare bundled dependencies`);
}
rmSync(destinationDir, { recursive: true, force: true });
mkdirSync(destinationDir, { recursive: true });
for (const entry of sourcePackage.files ?? []) {
cpSync(resolve(sourceDir, entry), resolve(destinationDir, entry), { recursive: true });
}
for (const entry of ["README.md", "LICENSE", "LICENSE.md"]) {
const sourcePath = resolve(sourceDir, entry);
if (existsSync(sourcePath)) cpSync(sourcePath, resolve(destinationDir, entry));
}
const deployedPackagePath = resolve(destinationDir, "package.json");
const publishManifest = materializePublishManifest(sourcePackage);
const installManifest = createBundledInstallManifest(publishManifest, bundledDependencies);
writeFileSync(deployedPackagePath, `${JSON.stringify(installManifest, null, 2)}\n`);
execFileSync(
"npm",
["install", "--omit=dev", "--ignore-scripts", "--no-audit", "--no-fund"],
{ cwd: destinationDir, stdio: "inherit" },
);
writeFileSync(deployedPackagePath, `${JSON.stringify(publishManifest, null, 2)}\n`);
applyBundledDependencyPatches(destinationDir, bundledDependencies, sourceRoot);
if (bundledDependencies.includes("acpx")) {
const acpxPackage = JSON.parse(
readFileSync(resolve(destinationDir, "node_modules/acpx/package.json"), "utf8"),
);
const expectedPatchMarker = {
"0.12.0": "onAgentStderr",
"0.13.1": "spawnEnvironment",
}[acpxPackage.version];
const acpxRuntime = readFileSync(
resolve(destinationDir, "node_modules/acpx/dist/runtime.js"),
"utf8",
);
if (!expectedPatchMarker || !acpxRuntime.includes(expectedPatchMarker)) {
throw new Error(
`staged acpx@${acpxPackage.version} runtime is missing the repository patch`,
);
}
}
if (bundledDependencies.includes("embedded-postgres")) {
const embeddedPostgresSource = readFileSync(
resolve(destinationDir, "node_modules/embedded-postgres/dist/index.js"),
"utf8",
);
if (
!embeddedPostgresSource.includes("const LC_MESSAGES_LOCALE = 'C';") ||
!embeddedPostgresSource.includes("globalThis.process.env")
) {
throw new Error("staged embedded-postgres runtime is missing the repository patch");
}
const embeddedPostgresPackage = JSON.parse(
readFileSync(resolve(destinationDir, "node_modules/embedded-postgres/package.json"), "utf8"),
);
const stagedPackage = JSON.parse(readFileSync(deployedPackagePath, "utf8"));
stagedPackage.optionalDependencies = {
...(stagedPackage.optionalDependencies ?? {}),
...(embeddedPostgresPackage.optionalDependencies ?? {}),
};
writeFileSync(deployedPackagePath, `${JSON.stringify(stagedPackage, null, 2)}\n`);
rmSync(resolve(destinationDir, "node_modules/@embedded-postgres"), { recursive: true, force: true });
}
}
if (process.argv[1] === fileURLToPath(import.meta.url)) {
const [sourceDir, destinationDir] = process.argv.slice(2);
if (!sourceDir || !destinationDir) {
console.error("Usage: prepare-bundled-package.mjs <source-dir> <destination-dir>");
process.exit(1);
}
prepareBundledPackage(resolve(sourceDir), resolve(destinationDir));
}