mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The repository has a supply-chain rule: contributors do not commit lockfiles. A bot and verified Dependabot commits write them. > - The rule checks only the root `pnpm-lock.yaml`. > - The sandbox providers are outside the root workspace, so they have two more pnpm lockfiles. Pull requests edited them by hand, for example #12791, #18839 and #19489. > - This pull request makes the same rule and the same bot cover those two lockfiles. > - The benefit is one lockfile rule for every managed pnpm lockfile, and no lockfile edits in contributor pull requests. ## Linked Issues or Issue Description Refs #19489 **What existing behavior does this improve?** The lockfile policy and the lockfile refresh bot. **Subsystem affected** CI: `.github/scripts/lockfile-trust.mjs`, `.github/workflows/refresh-lockfile.yml`, and the sandbox provider installs. **Current behavior** `lockfile-trust.mjs` blocks contributor edits to `pnpm-lock.yaml` only. A contributor can edit `scripts/typecheck-sandbox-providers/pnpm-lock.yaml` or `packages/plugins/sandbox-providers/daytona/pnpm-lock.yaml`, and CI passes. The refresh bot does not update these files. `typecheck:test-programs` fails on a stale provider lockfile and tells the contributor to commit a new one. **Proposed behavior** The policy blocks contributor edits to all three lockfiles. The refresh bot regenerates all three after merge. CI resolves a stale provider lockfile in place for the run, as it does for the root lockfile. **Reason and benefit** Lockfile content comes only from the bot or from verified Dependabot commits. Contributors do not need to run pnpm in special folders. **Breaking changes** None. A pull request that edits a provider lockfile now fails the policy check. Remove the lockfile change from the pull request. The bot commits it after merge. ## What Changed - `lockfile-trust.mjs` now exports `LOCKFILES`: the root lockfile, the shared sandbox provider lockfile, and the Daytona lockfile. The policy blocks contributor edits to each of them. A Dependabot pull request must match a verified Dependabot commit for each lockfile that it changes. The `merged` command and the reporting-lock proof still use only the root lockfile. - `check-pr-lockfile.mjs` (advisory check) uses the same list. - `refresh-lockfile.yml`: - A job-level `LOCKFILES` env lists the same three files. A test checks that it equals the policy list. - A new step regenerates the two provider lockfiles with `--resolution-only --ignore-scripts --no-frozen-lockfile`. - The unexpected-changes check, the change check, and `git add` cover all three lockfiles. - The release-age gate checks the added versions in every lockfile. - A held refresh saves all three lockfiles at their repository paths, with a `source-locks.sha256` file. The next run reuses them only if every master lockfile is unchanged. An artifact in the old format is not reused. - The PR title is now `chore(lockfile): refresh pnpm lockfiles`. - `check-lockfile-release-age.mjs` accepts more than one `--base`/`--head` pair. - `scripts/typecheck-test-programs.mjs`: if the frozen install of the provider workspace fails, the script resolves the lockfile in place and installs frozen again. - `scripts/build-standalone-public-packages.mjs`: the same fallback applies to a package with a committed lockfile (Daytona). `release.sh` already restores tracked files when it exits. - `doc/DEVELOPING.md` and `doc/RELEASE-AUTOMATION-SETUP.md` describe the managed lockfiles. They also say that the Pi and Kimi distribution lockfiles stay hand-managed, because they hold attested provider pins. ## Verification - `node --test .github/scripts/tests/*.test.mjs scripts/__tests__/build-standalone-concurrency.test.mjs`: 731 tests pass. New tests cover: - the policy for provider lockfiles, including a real git history with a Dependabot commit and a later hand edit - the workflow and policy lists match - the unexpected-changes check, run with bash against a fixture - a held refresh round trip, and its invalidation when a provider lockfile changes on master - the release-age gate across lockfile pairs - the inline fallback in both install scripts - `node --test scripts/typecheck-test-programs.test.mjs`: the install tests pass. Two tests fail locally only because this worktree has no root `node_modules` (no `tsc`). They also fail without this change. - I ran the new bot commands on master with pnpm 9.15.4. No package version changed. pnpm 9.15.4 removes `libc:` fields and adds `deprecated:` notes, so the first refresh is a metadata-only change. The release-age gate passes it because it adds no versions. - I ran the release-age and held-lock steps locally with bash against this commit. The gate reported `eligible=true`, and `sha256sum --check` accepted the saved hashes. - The workflow YAML parses, and `bash -n` accepts every `run` block. - Not testable before merge: the policy job loads `lockfile-trust.mjs` from the base branch, so the new rule applies only to pull requests opened after this merges. The refresh workflow runs only on master. ## Risks - The release build now resolves a stale Daytona lockfile in place, from the registry. The root release install already resolves without a frozen lockfile, so this is the same level of trust. - The paid runner e2e job still installs Daytona with `--frozen-lockfile`. After a Daytona manifest change merges, its Daytona cases fail until the bot's refresh merges. This matches the rule that paid jobs refuse a new root lockfile until its refresh merges. - Dependabot pull requests for the provider entries in #19489 change a provider lockfile. They pass only if each lockfile matches a verified Dependabot commit, the same as for the root lockfile. - Low risk otherwise: the bot keeps its existing token, environment, and merge rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, and git). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
175 lines
7.1 KiB
JavaScript
175 lines
7.1 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* check-lockfile-release-age.mjs
|
|
* Decides whether the lockfile bot may merge refreshed pnpm lockfiles without review.
|
|
* Every package version that the refresh adds must be published on npm for at least
|
|
* the minimum age. A version that is too new, or that cannot be verified, holds the
|
|
* PR for a human.
|
|
*
|
|
* Exports: parseLockfilePackages(text), addedPackageVersions(baseText, headText),
|
|
* addedPackageVersionsAcross(pairs),
|
|
* checkReleaseAge(entries, { fetchPublishTime, now, minAgeMs, concurrency })
|
|
*
|
|
* CLI: node check-lockfile-release-age.mjs --base <file> --head <file> [--base <file> --head <file> ...]
|
|
* --min-age-days <n>
|
|
* Pass one --base/--head pair per lockfile, in the same order. Writes `eligible=true|false`
|
|
* to GITHUB_OUTPUT and a Markdown report to --report.
|
|
*/
|
|
import { appendFileSync, readFileSync, writeFileSync } from 'node:fs';
|
|
import { fileURLToPath } from 'node:url';
|
|
import { parseArgs } from 'node:util';
|
|
|
|
const SEMVER = /^\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?$/;
|
|
// Every 2-space-indented line in `packages:` is an entry key. Keys can contain
|
|
// colons (URL and git dependencies), so match up to the final colon.
|
|
const PACKAGE_KEY = /^ {2}(\S.*):\s*$/;
|
|
const RESOLUTION = /^ {4}resolution: (.+?)\s*$/;
|
|
// A registry tarball is locked by integrity alone. A tarball URL, git repo or
|
|
// directory resolution names its own source and needs a human.
|
|
const REGISTRY_RESOLUTION = /^\{integrity: [^,{}]+\}$/;
|
|
const DAY_MS = 24 * 60 * 60 * 1000;
|
|
|
|
/** Maps each key of the lockfile's top-level `packages:` section to its resolution. */
|
|
export function parseLockfilePackages(text) {
|
|
const packages = new Map();
|
|
let inPackages = false;
|
|
let key = null;
|
|
for (const line of text.split('\n')) {
|
|
if (/^\S/.test(line)) {
|
|
inPackages = line.trimEnd() === 'packages:';
|
|
key = null;
|
|
continue;
|
|
}
|
|
if (!inPackages) continue;
|
|
const keyMatch = PACKAGE_KEY.exec(line);
|
|
if (keyMatch) {
|
|
key = keyMatch[1].replace(/^'(.*)'$/, '$1').replace(/\(.*$/, '');
|
|
packages.set(key, '');
|
|
continue;
|
|
}
|
|
const resolutionMatch = key && RESOLUTION.exec(line);
|
|
if (resolutionMatch) packages.set(key, resolutionMatch[1]);
|
|
}
|
|
return packages;
|
|
}
|
|
|
|
export function splitPackageKey(key) {
|
|
const at = key.lastIndexOf('@');
|
|
if (at <= 0) return { name: key, version: '' };
|
|
return { name: key.slice(0, at), version: key.slice(at + 1) };
|
|
}
|
|
|
|
/** Package entries in headText that baseText lacks or resolves differently. */
|
|
export function addedPackageVersions(baseText, headText) {
|
|
const base = parseLockfilePackages(baseText);
|
|
return [...parseLockfilePackages(headText)]
|
|
.filter(([key, resolution]) => base.get(key) !== resolution)
|
|
.sort(([a], [b]) => a.localeCompare(b))
|
|
.map(([key, resolution]) => ({ key, ...splitPackageKey(key), resolution }));
|
|
}
|
|
|
|
/** Added versions across several base/head lockfile pairs, each listed once. */
|
|
export function addedPackageVersionsAcross(pairs) {
|
|
const entries = new Map();
|
|
for (const [baseText, headText] of pairs) {
|
|
for (const entry of addedPackageVersions(baseText, headText)) entries.set(`${entry.key}\0${entry.resolution}`, entry);
|
|
}
|
|
return [...entries.values()].sort((a, b) => a.key.localeCompare(b.key));
|
|
}
|
|
|
|
export async function fetchNpmPublishTime(name, version, fetchImpl = fetch) {
|
|
const res = await fetchImpl(`https://registry.npmjs.org/${name.replace('/', '%2f')}`, {
|
|
headers: { Accept: 'application/json' },
|
|
signal: AbortSignal.timeout(30_000),
|
|
});
|
|
if (!res.ok) throw new Error(`npm registry returned ${res.status}`);
|
|
const doc = await res.json();
|
|
const published = doc.time?.[version];
|
|
return published ? new Date(published) : null;
|
|
}
|
|
|
|
/**
|
|
* Checks each added version. Fails closed: a lookup error, a missing publish time or a
|
|
* non-registry version holds the merge.
|
|
*/
|
|
export async function checkReleaseAge(entries, {
|
|
fetchPublishTime = fetchNpmPublishTime,
|
|
now = new Date(),
|
|
minAgeMs,
|
|
concurrency = 8,
|
|
}) {
|
|
const held = [];
|
|
const queue = [...entries];
|
|
const worker = async () => {
|
|
for (let entry = queue.shift(); entry; entry = queue.shift()) {
|
|
const reason = await holdReason(entry, fetchPublishTime, now, minAgeMs);
|
|
if (reason) held.push({ ...entry, reason });
|
|
}
|
|
};
|
|
await Promise.all(Array.from({ length: concurrency }, worker));
|
|
held.sort((a, b) => a.key.localeCompare(b.key));
|
|
return { eligible: held.length === 0, checked: entries.length, held };
|
|
}
|
|
|
|
async function holdReason({ name, version, resolution }, fetchPublishTime, now, minAgeMs) {
|
|
if (!SEMVER.test(version)) return 'not a registry version';
|
|
if (!REGISTRY_RESOLUTION.test(resolution ?? '')) return `not a registry resolution: ${resolution || 'missing'}`;
|
|
let published;
|
|
try {
|
|
published = await fetchPublishTime(name, version);
|
|
} catch (error) {
|
|
return `registry lookup failed: ${error.message}`;
|
|
}
|
|
if (!published || Number.isNaN(published.getTime())) return 'no publish time on npm';
|
|
const ageMs = now.getTime() - published.getTime();
|
|
if (ageMs < minAgeMs) {
|
|
return `published ${published.toISOString()} (${(ageMs / DAY_MS).toFixed(1)} days ago)`;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
export function formatReport(result, minAgeDays) {
|
|
const lines = ['<!-- lockfile-release-age -->'];
|
|
if (result.eligible) {
|
|
lines.push(`All ${result.checked} added package versions are at least ${minAgeDays} days old. The lockfile bot merges this PR.`);
|
|
} else {
|
|
lines.push(
|
|
`The lockfile bot did not merge this PR. These added package versions are newer than ${minAgeDays} days or could not be verified:`,
|
|
'',
|
|
...result.held.map(({ key, reason }) => `- \`${key}\`: ${reason}`),
|
|
'',
|
|
'A maintainer can review and merge this PR. If no one does, the daily refresh tries again and merges it when every version is old enough.',
|
|
);
|
|
}
|
|
return `${lines.join('\n')}\n`;
|
|
}
|
|
|
|
async function main() {
|
|
const { values } = parseArgs({
|
|
options: {
|
|
base: { type: 'string', multiple: true },
|
|
head: { type: 'string', multiple: true },
|
|
'min-age-days': { type: 'string' },
|
|
report: { type: 'string' },
|
|
},
|
|
});
|
|
const minAgeDays = Number(values['min-age-days']);
|
|
const bases = values.base ?? [];
|
|
const heads = values.head ?? [];
|
|
if (bases.length === 0 || bases.length !== heads.length || !Number.isFinite(minAgeDays) || minAgeDays < 0) {
|
|
console.error('Usage: check-lockfile-release-age.mjs --base <file> --head <file> [--base <file> --head <file> ...] --min-age-days <n> [--report <file>]');
|
|
process.exit(2);
|
|
}
|
|
|
|
const entries = addedPackageVersionsAcross(bases.map((base, index) => [readFileSync(base, 'utf8'), readFileSync(heads[index], 'utf8')]));
|
|
const result = await checkReleaseAge(entries, { minAgeMs: minAgeDays * DAY_MS });
|
|
const report = formatReport(result, minAgeDays);
|
|
console.log(report);
|
|
if (values.report) writeFileSync(values.report, report);
|
|
if (process.env.GITHUB_OUTPUT) appendFileSync(process.env.GITHUB_OUTPUT, `eligible=${result.eligible}\n`);
|
|
}
|
|
|
|
if (process.argv[1] === fileURLToPath(import.meta.url)) {
|
|
main().catch(e => { console.error(e.message); process.exit(1); });
|
|
}
|