Files
PaperClipAI/.github/scripts/lockfile-trust.mjs
T
DottaandClaude Fable 5.2 ac114d9479 refactor: cut DEVELOPING.md to a setup and workflow guide (#21670)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` tells every contributor and agent to read
`doc/DEVELOPING.md` for setup, commands, testing, and worktrees
> - That file grew to 5,495 lines. Most of it was notes on how one
subsystem behaves, added one pull request at a time
> - Readers could not find the commands. Agents also read behavior notes
as rules
> - This pull request cuts `doc/DEVELOPING.md` to a 660-line setup and
workflow guide. It moves the subsystem notes to focused docs
> - The benefit is a guide that a person or an agent can read in full,
and a clear place for each kind of note

## Linked Issues or Issue Description

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/DEVELOPING.md`

**What's wrong?**

The file was 5,495 lines. It opened with 112 lines about case handoffs
before it reached Prerequisites. Unrelated sections sat below the
"OpenClaw Docker UI One-Command Script" heading. Statements about
current behavior read like policy.

**Suggested fix**

Keep setup, commands, and workflow in `doc/DEVELOPING.md`. Move each
subsystem note to a doc for that subsystem. Add an index.

## What Changed

- `doc/DEVELOPING.md` is now 660 lines: prerequisites, start dev, health
checks, local run, test drive, Docker, instance layout, database,
storage, secrets, worktrees, test commands, Storybook, CLI, and an index
of the other docs.
- 15 new docs hold the moved text without wording changes:
`ci-workflows`, `test-infrastructure`, `worktrees`,
`hot-restart-deploys`, `heartbeat-internals`, `runner-internals`,
`workspace-repositories`, `workspace-storage`,
`skills-and-teams-catalog`, `ui-internals`, `case-handoffs`,
`chat-connector-development`, `instance-configuration`,
`integration-api-notes`, `database-backups`.
- "Publish a branch Storybook" moved to `doc/STORYBOOK-DEPLOYMENT.md`.
The privacy rollout notes moved to `doc/ISSUE-PRIVACY.md`.
- Deleted about 280 lines: the story-by-story Storybook notes, "Agent
Artifact Uploads" (a copy of `doc/AGENT-ARTIFACTS.md`), two pointer-only
sections, "Workspace feature checks (2026-10-09)", and "Chat and task
navigation caching". The test-drive build-lock paragraph is now four
lines.
- Repointed every inbound link and every script or workflow comment that
named a moved section.
- `AGENTS.md` now says where a behavior note goes, so the guide does not
grow again.

## Verification

- A script split the file by line range. It asserts that each of the
5,495 source lines is kept, moved, or dropped exactly one time.
- A link check on every changed Markdown file found 0 missing files and
0 missing anchors.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `node --check` passes on the three scripts with changed comments.
- `pnpm verify` was not run. This worktree has no installed
dependencies, and the change has no runtime code.

## Risks

- Low risk. The change is documentation and comments only.
- A link to a moved section from outside this repository will land on
the top of `doc/DEVELOPING.md`.
- The new docs are not in the audited prose list in
`.github/scripts/ci-inputs.mjs`. An edit to one of them runs full CI. An
edit to `doc/DEVELOPING.md` does not. This PR does not change that list.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 18:02:27 -06:00

238 lines
11 KiB
JavaScript

#!/usr/bin/env node
/**
* lockfile-trust.mjs
*
* Decides who may change the dependency locks (LOCKFILES). Only trusted environments write them:
* - the lockfile refresh workflow (`chore/refresh-lockfile`, opened by github-actions[bot]),
* - Dependabot, when the exact bytes of each lockfile it changes come from a verified Dependabot commit.
*
* The reporting-lock proof (`tests/runner-e2e/report-lock.json` and `.patch`) can rewrite the
* lock that paid evaluation jobs install, so it counts as lock content too. Only the refresh
* workflow writes it. After a verified Dependabot lockfile merges, the refresh workflow renews
* the proof for it.
*
* Keep this file free of imports outside node:*. The PR policy job runs the copy from the current
* base-branch tip, so a pull request cannot rewrite the check that judges it.
*
* Usage:
* node .github/scripts/lockfile-trust.mjs pr # PR policy job
* node .github/scripts/lockfile-trust.mjs merged # refresh workflow on master
*/
import { createHash } from 'node:crypto';
import { execFileSync } from 'node:child_process';
import { appendFileSync, realpathSync } from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
// The root lockfile. The reporting-lock proof and the `merged` command follow this one only.
export const LOCKFILE = 'pnpm-lock.yaml';
// Sandbox providers stay out of the root workspace (see doc/test-infrastructure.md), so they have their own
// pnpm lockfiles. The refresh workflow regenerates these too, and its LOCKFILES env lists the same
// paths. Hand-managed pins, such as the Pi and Kimi distributions, are not listed.
export const PROVIDER_LOCKFILES = [
'scripts/typecheck-sandbox-providers/pnpm-lock.yaml',
'packages/plugins/sandbox-providers/daytona/pnpm-lock.yaml',
];
export const LOCKFILES = [LOCKFILE, ...PROVIDER_LOCKFILES];
export const REPORT_LOCK_FILES = ['tests/runner-e2e/report-lock.json', 'tests/runner-e2e/report-lock.patch'];
export const DEPENDABOT_LOGIN = 'dependabot[bot]';
export const REFRESH_BRANCH = 'chore/refresh-lockfile';
export const REFRESH_PR_AUTHOR = 'github-actions[bot]';
// GitHub commits and signs Dependabot commits as web-flow. A verified signature proves only who
// committed, and anyone can set the author field, so require GitHub as the committer as well.
export const GITHUB_COMMITTER_LOGIN = 'web-flow';
const RECREATE_HINT = 'Comment `@dependabot recreate` so Dependabot rebuilds the lockfile, instead of merging or editing it by hand.';
/**
* True for a commit that Dependabot authored and GitHub committed and signed.
* @param {{ login: string | null, committer: string | null, verified: boolean }} commit
*/
export function isDependabotCommit(commit) {
return commit.login === DEPENDABOT_LOGIN && commit.committer === GITHUB_COMMITTER_LOGIN && commit.verified === true;
}
/**
* Find a Dependabot commit (see isDependabotCommit) in the pull request that holds exactly the
* lockfile under review.
* @param {{ lockBlob: string | null,
* commits: Array<{ sha: string, login: string | null, committer: string | null, verified: boolean }>,
* lockBlobAt: (sha: string) => string | null }} input
*/
export function findDependabotLockCommit({ lockBlob, commits, lockBlobAt }) {
if (!lockBlob) return null;
for (const commit of [...commits].reverse()) {
if (!isDependabotCommit(commit)) continue;
if (lockBlobAt(commit.sha) === lockBlob) return commit.sha;
}
return null;
}
/**
* Classify the lock-content change in a pull request.
* `dependabotLockCommits` maps each changed lockfile to the verified Dependabot commit that holds
* its exact bytes, or to null.
* @returns {{ allowed: boolean, writer: 'none' | 'dependabot' | 'refresh-bot', reason: string }}
*/
export function classifyPullRequest({ changedPaths, author, headRepo, baseRepo, headRef, dependabotLockCommits = {} }) {
const changedLockfiles = LOCKFILES.filter((file) => changedPaths.includes(file));
const proofChanged = changedPaths.some((file) => REPORT_LOCK_FILES.includes(file));
if (changedLockfiles.length === 0 && !proofChanged) {
return { allowed: true, writer: 'none', reason: 'The pull request does not change lock content.' };
}
const sameRepo = Boolean(headRepo) && headRepo === baseRepo;
if (sameRepo && author === REFRESH_PR_AUTHOR && headRef === REFRESH_BRANCH) {
return { allowed: true, writer: 'refresh-bot', reason: 'The lockfile refresh workflow opened this pull request.' };
}
if (sameRepo && author === DEPENDABOT_LOGIN) {
if (proofChanged) {
return { allowed: false, writer: 'dependabot',
reason: `Dependabot pull requests must not change ${REPORT_LOCK_FILES.join(' or ')}. The lockfile refresh renews the reporting-lock proof after merge.` };
}
const unverified = changedLockfiles.filter((file) => !dependabotLockCommits[file]);
if (unverified.length > 0) {
return { allowed: false, writer: 'dependabot',
reason: `${unverified.join(', ')} does not match any verified Dependabot commit in this pull request. ${RECREATE_HINT}` };
}
return { allowed: true, writer: 'dependabot',
reason: changedLockfiles.map((file) => `${file} matches verified Dependabot commit ${dependabotLockCommits[file]}.`).join(' ') };
}
const files = [...changedLockfiles, ...REPORT_LOCK_FILES.filter((file) => changedPaths.includes(file))];
return { allowed: false, writer: 'none',
reason: `Do not commit ${files.join(', ')} in pull requests. CI owns lockfile updates.` };
}
/**
* True when GitHub created and signed the commit, as it does when it merges a pull request.
* @param {any} commit Response of GET /repos/{repo}/commits/{sha}.
*/
export function isGitHubSignedCommit(commit) {
return commit?.committer?.login === GITHUB_COMMITTER_LOGIN && commit?.commit?.verification?.verified === true;
}
/**
* Pick the merged Dependabot pull request whose squash or merge commit is `commitSha`.
* @param {Array<any>} pulls Response of GET /repos/{repo}/commits/{sha}/pulls.
*/
export function findMergedDependabotPull(pulls, commitSha, defaultBranch) {
return pulls.find((pull) => pull?.user?.login === DEPENDABOT_LOGIN && Boolean(pull.merged_at) &&
pull.merge_commit_sha === commitSha && pull?.base?.ref === defaultBranch) ?? null;
}
function run(command, args, options = {}) {
return execFileSync(command, args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], ...options }).trim();
}
function tryRun(command, args) {
try { return run(command, args); } catch { return null; }
}
function ghJsonLines(endpoint, jq) {
const output = run('gh', ['api', '--paginate', endpoint, '--jq', jq]);
return output ? output.split('\n').map((line) => JSON.parse(line)) : [];
}
function pullCommits(repository, number) {
return ghJsonLines(`repos/${repository}/pulls/${number}/commits?per_page=100`,
'.[] | {sha, login: .author.login, committer: .committer.login, verified: .commit.verification.verified}');
}
function writeOutputs(values) {
const lines = Object.entries(values).map(([key, value]) => `${key}=${value}\n`).join('');
if (process.env.GITHUB_OUTPUT) appendFileSync(process.env.GITHUB_OUTPUT, lines);
}
function required(name) {
const value = process.env[name];
if (!value) throw new Error(`${name} is required`);
return value;
}
function checkPullRequest() {
const repository = required('GITHUB_REPOSITORY');
const baseSha = required('BASE_SHA');
const headSha = required('HEAD_SHA');
const author = required('PR_AUTHOR');
// Diff against the merge base so recent base-branch commits are not counted as PR changes.
const changedPaths = run('git', ['diff', '--name-only', `${baseSha}...${headSha}`]).split('\n').filter(Boolean);
const dependabotLockCommits = {};
const changedLockfiles = LOCKFILES.filter((file) => changedPaths.includes(file));
if (author === DEPENDABOT_LOGIN && changedLockfiles.length > 0) {
const commits = pullCommits(repository, required('PR_NUMBER'));
for (const file of changedLockfiles) {
dependabotLockCommits[file] = findDependabotLockCommit({
lockBlob: tryRun('git', ['rev-parse', `${headSha}:${file}`]),
commits,
lockBlobAt: (sha) => tryRun('git', ['rev-parse', `${sha}:${file}`]),
});
}
}
const result = classifyPullRequest({
changedPaths, author, headRepo: process.env.HEAD_REPO ?? '', baseRepo: repository,
headRef: process.env.HEAD_REF ?? '', dependabotLockCommits,
});
console.log(result.reason);
writeOutputs({ writer: result.writer });
if (!result.allowed) {
console.log(`::error title=Untrusted lockfile change::${result.reason}`);
process.exitCode = 1;
}
}
/**
* On master: report the committed lockfile's SHA-256 when its last change is a GitHub-signed merge
* of a Dependabot pull request whose head lockfile bytes come from a verified Dependabot commit.
* When master's lockfile moved after Dependabot's commit, GitHub's merge combines the two trusted
* lockfiles, so the merged bytes can differ from Dependabot's. The refresh then records the
* resolution of that merged lock as the reviewed correction. Otherwise report nothing.
*/
function checkMerged() {
const repository = required('GITHUB_REPOSITORY');
const head = required('GITHUB_SHA');
const defaultBranch = process.env.DEFAULT_BRANCH || 'master';
const lockCommit = run('gh', ['api', `repos/${repository}/commits?sha=${head}&path=${LOCKFILE}&per_page=1`, '--jq', '.[0].sha // empty']);
const pull = lockCommit
? findMergedDependabotPull(JSON.parse(run('gh', ['api', `repos/${repository}/commits/${lockCommit}/pulls`])), lockCommit, defaultBranch)
: null;
if (pull && !isGitHubSignedCommit(JSON.parse(run('gh', ['api', `repos/${repository}/commits/${lockCommit}`])))) {
console.log(`::warning title=Unverified Dependabot merge::${lockCommit} is not a GitHub-signed merge of #${pull.number}. The reporting-lock proof is not renewed.`);
writeOutputs({ trusted_lock_sha256: '', dependabot_pr: String(pull.number) });
return;
}
if (!pull) {
console.log(`The last ${LOCKFILE} change on ${defaultBranch} is not a merged Dependabot pull request.`);
writeOutputs({ trusted_lock_sha256: '', dependabot_pr: '' });
return;
}
// The checkout is shallow, so read each candidate's lockfile blob hash from the API.
const lockBlobAt = (sha) => tryRun('gh', ['api', `repos/${repository}/git/trees/${sha}`,
'--jq', `.tree[] | select(.path == "${LOCKFILE}") | .sha`]) || null;
const verified = findDependabotLockCommit({
lockBlob: lockBlobAt(pull.head.sha),
commits: pullCommits(repository, pull.number),
lockBlobAt,
});
if (!verified) {
console.log(`::warning title=Unverified Dependabot lockfile::The ${LOCKFILE} at the head of #${pull.number} matches no verified Dependabot commit. The reporting-lock proof is not renewed.`);
writeOutputs({ trusted_lock_sha256: '', dependabot_pr: String(pull.number) });
return;
}
const lockSha256 = createHash('sha256').update(execFileSync('git', ['show', `HEAD:${LOCKFILE}`])).digest('hex');
console.log(`${LOCKFILE} on ${defaultBranch} is GitHub's merge of verified Dependabot commit ${verified} from #${pull.number}.`);
writeOutputs({ trusted_lock_sha256: lockSha256, dependabot_pr: String(pull.number) });
}
// Compare real paths: the runner's temporary directory can sit behind a symlink.
if (process.argv[1] && realpathSync(path.resolve(process.argv[1])) === realpathSync(fileURLToPath(import.meta.url))) {
const mode = process.argv[2];
if (mode === 'pr') checkPullRequest();
else if (mode === 'merged') checkMerged();
else throw new Error('Usage: lockfile-trust.mjs pr|merged');
}