mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The repository has a supply-chain rule: contributors do not commit lockfiles. A bot and verified Dependabot commits write them. > - The rule checks only the root `pnpm-lock.yaml`. > - The sandbox providers are outside the root workspace, so they have two more pnpm lockfiles. Pull requests edited them by hand, for example #12791, #18839 and #19489. > - This pull request makes the same rule and the same bot cover those two lockfiles. > - The benefit is one lockfile rule for every managed pnpm lockfile, and no lockfile edits in contributor pull requests. ## Linked Issues or Issue Description Refs #19489 **What existing behavior does this improve?** The lockfile policy and the lockfile refresh bot. **Subsystem affected** CI: `.github/scripts/lockfile-trust.mjs`, `.github/workflows/refresh-lockfile.yml`, and the sandbox provider installs. **Current behavior** `lockfile-trust.mjs` blocks contributor edits to `pnpm-lock.yaml` only. A contributor can edit `scripts/typecheck-sandbox-providers/pnpm-lock.yaml` or `packages/plugins/sandbox-providers/daytona/pnpm-lock.yaml`, and CI passes. The refresh bot does not update these files. `typecheck:test-programs` fails on a stale provider lockfile and tells the contributor to commit a new one. **Proposed behavior** The policy blocks contributor edits to all three lockfiles. The refresh bot regenerates all three after merge. CI resolves a stale provider lockfile in place for the run, as it does for the root lockfile. **Reason and benefit** Lockfile content comes only from the bot or from verified Dependabot commits. Contributors do not need to run pnpm in special folders. **Breaking changes** None. A pull request that edits a provider lockfile now fails the policy check. Remove the lockfile change from the pull request. The bot commits it after merge. ## What Changed - `lockfile-trust.mjs` now exports `LOCKFILES`: the root lockfile, the shared sandbox provider lockfile, and the Daytona lockfile. The policy blocks contributor edits to each of them. A Dependabot pull request must match a verified Dependabot commit for each lockfile that it changes. The `merged` command and the reporting-lock proof still use only the root lockfile. - `check-pr-lockfile.mjs` (advisory check) uses the same list. - `refresh-lockfile.yml`: - A job-level `LOCKFILES` env lists the same three files. A test checks that it equals the policy list. - A new step regenerates the two provider lockfiles with `--resolution-only --ignore-scripts --no-frozen-lockfile`. - The unexpected-changes check, the change check, and `git add` cover all three lockfiles. - The release-age gate checks the added versions in every lockfile. - A held refresh saves all three lockfiles at their repository paths, with a `source-locks.sha256` file. The next run reuses them only if every master lockfile is unchanged. An artifact in the old format is not reused. - The PR title is now `chore(lockfile): refresh pnpm lockfiles`. - `check-lockfile-release-age.mjs` accepts more than one `--base`/`--head` pair. - `scripts/typecheck-test-programs.mjs`: if the frozen install of the provider workspace fails, the script resolves the lockfile in place and installs frozen again. - `scripts/build-standalone-public-packages.mjs`: the same fallback applies to a package with a committed lockfile (Daytona). `release.sh` already restores tracked files when it exits. - `doc/DEVELOPING.md` and `doc/RELEASE-AUTOMATION-SETUP.md` describe the managed lockfiles. They also say that the Pi and Kimi distribution lockfiles stay hand-managed, because they hold attested provider pins. ## Verification - `node --test .github/scripts/tests/*.test.mjs scripts/__tests__/build-standalone-concurrency.test.mjs`: 731 tests pass. New tests cover: - the policy for provider lockfiles, including a real git history with a Dependabot commit and a later hand edit - the workflow and policy lists match - the unexpected-changes check, run with bash against a fixture - a held refresh round trip, and its invalidation when a provider lockfile changes on master - the release-age gate across lockfile pairs - the inline fallback in both install scripts - `node --test scripts/typecheck-test-programs.test.mjs`: the install tests pass. Two tests fail locally only because this worktree has no root `node_modules` (no `tsc`). They also fail without this change. - I ran the new bot commands on master with pnpm 9.15.4. No package version changed. pnpm 9.15.4 removes `libc:` fields and adds `deprecated:` notes, so the first refresh is a metadata-only change. The release-age gate passes it because it adds no versions. - I ran the release-age and held-lock steps locally with bash against this commit. The gate reported `eligible=true`, and `sha256sum --check` accepted the saved hashes. - The workflow YAML parses, and `bash -n` accepts every `run` block. - Not testable before merge: the policy job loads `lockfile-trust.mjs` from the base branch, so the new rule applies only to pull requests opened after this merges. The refresh workflow runs only on master. ## Risks - The release build now resolves a stale Daytona lockfile in place, from the registry. The root release install already resolves without a frozen lockfile, so this is the same level of trust. - The paid runner e2e job still installs Daytona with `--frozen-lockfile`. After a Daytona manifest change merges, its Daytona cases fail until the bot's refresh merges. This matches the rule that paid jobs refuse a new root lockfile until its refresh merges. - Dependabot pull requests for the provider entries in #19489 change a provider lockfile. They pass only if each lockfile matches a verified Dependabot commit, the same as for the root lockfile. - Low risk otherwise: the bot keeps its existing token, environment, and merge rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, and git). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
167 lines
5.0 KiB
JavaScript
167 lines
5.0 KiB
JavaScript
import assert from "node:assert/strict";
|
|
import test from "node:test";
|
|
|
|
import {
|
|
installStandalonePackage,
|
|
RESOLVE_LOCKFILE_ARGS,
|
|
resolveConcurrency,
|
|
runWithConcurrency,
|
|
} from "../build-standalone-public-packages.mjs";
|
|
|
|
function deferred() {
|
|
let resolve;
|
|
let reject;
|
|
const promise = new Promise((res, rej) => {
|
|
resolve = res;
|
|
reject = rej;
|
|
});
|
|
return { promise, resolve, reject };
|
|
}
|
|
|
|
test("runs every item and preserves result order", async () => {
|
|
const items = [1, 2, 3, 4, 5];
|
|
const results = await runWithConcurrency(items, 2, async (value) => value * 10);
|
|
assert.deepEqual(results, [10, 20, 30, 40, 50]);
|
|
});
|
|
|
|
test("never exceeds the concurrency limit", async () => {
|
|
const items = Array.from({ length: 8 }, (_, index) => index);
|
|
const limit = 3;
|
|
let active = 0;
|
|
let peak = 0;
|
|
const gates = items.map(() => deferred());
|
|
|
|
const run = runWithConcurrency(items, limit, async (value) => {
|
|
active += 1;
|
|
peak = Math.max(peak, active);
|
|
assert.ok(active <= limit, `active ${active} exceeded limit ${limit}`);
|
|
await gates[value].promise;
|
|
active -= 1;
|
|
return value;
|
|
});
|
|
|
|
// Release gates progressively; the pool must keep at most `limit` in flight.
|
|
for (const gate of gates) {
|
|
await Promise.resolve();
|
|
gate.resolve();
|
|
}
|
|
|
|
await run;
|
|
assert.equal(peak, limit);
|
|
});
|
|
|
|
test("aggregates failures and still runs remaining items", async () => {
|
|
const items = [0, 1, 2, 3];
|
|
const processed = [];
|
|
|
|
await assert.rejects(
|
|
() =>
|
|
runWithConcurrency(items, 2, async (value) => {
|
|
processed.push(value);
|
|
if (value === 1) {
|
|
throw new Error(`boom-${value}`);
|
|
}
|
|
return value;
|
|
}),
|
|
(error) => {
|
|
assert.match(error.message, /1 standalone package build\(s\) failed/);
|
|
assert.equal(error.failures.length, 1);
|
|
assert.equal(error.failures[0].index, 1);
|
|
assert.match(error.failures[0].error.message, /boom-1/);
|
|
return true;
|
|
},
|
|
);
|
|
|
|
// A single failure must not abort the rest of the queue.
|
|
assert.deepEqual(processed.sort((a, b) => a - b), [0, 1, 2, 3]);
|
|
});
|
|
|
|
test("reports failures sorted by original index", async () => {
|
|
const items = [0, 1, 2, 3, 4];
|
|
|
|
await assert.rejects(
|
|
() =>
|
|
runWithConcurrency(items, 5, async (value) => {
|
|
if (value === 3 || value === 1) {
|
|
throw new Error(`fail-${value}`);
|
|
}
|
|
return value;
|
|
}),
|
|
(error) => {
|
|
assert.deepEqual(
|
|
error.failures.map(({ index }) => index),
|
|
[1, 3],
|
|
);
|
|
return true;
|
|
},
|
|
);
|
|
});
|
|
|
|
test("resolveConcurrency honors a valid env override, capped by package count", () => {
|
|
const previous = process.env.STANDALONE_BUILD_CONCURRENCY;
|
|
try {
|
|
process.env.STANDALONE_BUILD_CONCURRENCY = "2";
|
|
assert.equal(resolveConcurrency(7), 2);
|
|
assert.equal(resolveConcurrency(1), 1);
|
|
assert.equal(resolveConcurrency(0), 1);
|
|
|
|
process.env.STANDALONE_BUILD_CONCURRENCY = "0";
|
|
assert.ok(resolveConcurrency(7) >= 1);
|
|
|
|
process.env.STANDALONE_BUILD_CONCURRENCY = "not-a-number";
|
|
assert.ok(resolveConcurrency(7) >= 1);
|
|
} finally {
|
|
if (previous === undefined) {
|
|
delete process.env.STANDALONE_BUILD_CONCURRENCY;
|
|
} else {
|
|
process.env.STANDALONE_BUILD_CONCURRENCY = previous;
|
|
}
|
|
}
|
|
});
|
|
|
|
test("resolveConcurrency never returns more than the package count", () => {
|
|
const previous = process.env.STANDALONE_BUILD_CONCURRENCY;
|
|
try {
|
|
delete process.env.STANDALONE_BUILD_CONCURRENCY;
|
|
assert.equal(resolveConcurrency(0), 1);
|
|
assert.ok(resolveConcurrency(3) <= 3);
|
|
} finally {
|
|
if (previous !== undefined) {
|
|
process.env.STANDALONE_BUILD_CONCURRENCY = previous;
|
|
}
|
|
}
|
|
});
|
|
|
|
test("a package with a current lockfile installs it frozen once", async () => {
|
|
const calls = [];
|
|
await installStandalonePackage({ hasLockfile: true, lockfile: "pkg/pnpm-lock.yaml", log: () => {}, run: async (args) => { calls.push(args); } });
|
|
assert.deepEqual(calls, [["install", "--ignore-workspace", "--frozen-lockfile"]]);
|
|
});
|
|
|
|
test("a stale committed lockfile is resolved in place, then installed frozen", async () => {
|
|
const calls = [];
|
|
const logs = [];
|
|
await installStandalonePackage({
|
|
hasLockfile: true,
|
|
lockfile: "pkg/pnpm-lock.yaml",
|
|
log: (line) => logs.push(line),
|
|
run: async (args) => {
|
|
calls.push(args);
|
|
if (calls.length === 1) throw new Error("ERR_PNPM_OUTDATED_LOCKFILE");
|
|
},
|
|
});
|
|
assert.deepEqual(calls, [
|
|
["install", "--ignore-workspace", "--frozen-lockfile"],
|
|
RESOLVE_LOCKFILE_ARGS,
|
|
["install", "--ignore-workspace", "--frozen-lockfile"],
|
|
]);
|
|
assert.ok(RESOLVE_LOCKFILE_ARGS.includes("--resolution-only") && RESOLVE_LOCKFILE_ARGS.includes("--ignore-scripts"));
|
|
assert.match(logs[0], /pkg\/pnpm-lock\.yaml/);
|
|
});
|
|
|
|
test("a package without a lockfile installs without writing one", async () => {
|
|
const calls = [];
|
|
await installStandalonePackage({ hasLockfile: false, lockfile: "pkg/pnpm-lock.yaml", log: () => {}, run: async (args) => { calls.push(args); } });
|
|
assert.deepEqual(calls, [["install", "--ignore-workspace", "--no-lockfile"]]);
|
|
});
|