Files
PaperClipAI/tests/runner-e2e/source-sdk-public-inputs.mjs
T
DottaandClaude Fable 5.2 db3d7223e0 fix(verify): report stale SDK public input pins locally, before CI (#21283)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types against
pinned source. The pins are in
`tests/runner-e2e/source-sdk-public-inputs.json`.
> - The pins cover every file in `packages/shared/src` and
`packages/plugins/sdk/src`, and the SDK build inputs (for example
`pnpm-lock.yaml`).
> - A PR that edits a pinned file and does not refresh the registry
fails three files in the "Runner support suite" lane.
> - Between 2026-11-12 10:27Z and 2026-11-13 01:14Z, this failure
occurred in 14 CI runs on 12 PRs.
> - The local check, `pnpm verify`, returned exit 2 ("let CI cover it")
for these changes. It did not mention the pins. Authors pushed, and CI
failed.
> - This pull request makes `pnpm verify` find stale pins before the
push. It also makes the opaque fixture error name the file and the fix.
> - The benefit is that authors find and fix the problem in seconds,
locally, instead of after a 6-minute CI lane.

## Linked Issues or Issue Description

**What happened?**
Three files in the "Runner support suite" lane failed together on many
unrelated PRs: `source-sdk-evidence.test.ts`,
`source-sdk-public-inputs.test.mjs` and
`source-sdk-public-type.test.ts`. Two of the three failed with the
opaque error `Reviewed SDK reference source changed`.

**Root cause**
The pins on master are not stale. The tests pass on master (58/58). Each
failing PR edited a pinned input and did not refresh the registry. I
compared the PR heads with their merge base:

- All 14 failing heads (12 PRs) changed a pinned input. None of them
changed `source-sdk-public-inputs.json`.
- All 19 heads in the same window that changed a pinned input and passed
this lane also refreshed the registry. This includes later heads of
three PRs that failed first (#21119, #21121, #21156).
- Two of the 12 PRs are Dependabot PRs that change only `pnpm-lock.yaml`
(#19593, #19594). The lockfile refresh bot already resyncs the pins for
lock changes on master.

**Steps to reproduce**
1. Start from `origin/master`.
2. Add one line to `packages/shared/src/types/heartbeat.ts` and commit
it.
3. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts source-sdk-evidence.test.ts
source-sdk-public-inputs.test.mjs source-sdk-public-type.test.ts`.
4. The result is the same as CI: 3 files fail, with the same 4 failures.
5. Run `pnpm verify --explain`. It returns exit 2 and does not mention
the pins.

**Expected behavior**
`pnpm verify` reports the stale pins and the refresh command before the
author pushes.

## What Changed

- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputsInWorkingTree(root)`. It applies the existing drift
guard to the working tree (tracked and untracked files that git does not
ignore). Add `isSdkPublicInput(file)` and the shared failure message
`staleSdkPublicInputsMessage(stale)`. The committed-tree guard keeps its
behavior and now uses a shared comparison.
- `scripts/local-verify.mjs`: when the diff touches a pinned input, the
registry or `report-lock.json`, `pnpm verify` checks the pins first. If
they are stale, it exits 1 and prints the files and the refresh command.
The check reads files only. It does not write the registry.
- `tests/runner-e2e/source-sdk-reviewed-fixture.ts`: the error `Reviewed
SDK reference source changed` now names the file and gives the refresh
command.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: new test for the
working-tree guard (uncommitted edit, untracked file, ignored file). The
CI drift test uses the shared message.
- `doc/DEVELOPING.md`: document the new check under "Incremental local
verification".

No assertion is weakened. No pin is changed. No file under `.github/` is
changed.

## Verification

- Master, without this change: the three files pass (58/58).
- Red: on master plus a one-line edit to
`packages/shared/src/types/heartbeat.ts`, the three files fail with the
CI signature. `pnpm verify --explain` returns exit 2 and does not
mention the pins.
- With this change on the same red commit: `pnpm verify` exits 1 in
about 15 s and prints `tests/runner-e2e/source-sdk-public-inputs.json is
stale for 1 pinned input(s): packages/shared/src/types/heartbeat.ts
(changed). Commit your change, run ... --write, and commit the updated
registry.`
- Green: I followed the printed command (`node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`, then
commit). The three files then pass (59/59, with the new test).
- `node --test scripts/__tests__/local-verify.test.mjs`: 14/14 pass.
- `pnpm verify` on this branch returns exit 2 because
`scripts/local-verify.mjs` is a global CI input. Required CI supplies
the broad coverage.

## Risks

- Low risk. The new check only reads files. It runs only when the diff
touches a pinned input.
- `pnpm verify` now fails (exit 1) for a change that also fails in CI.
Before, it returned exit 2.
- The check needs a working `git ls-files` in the repository. `pnpm
verify` already needs git.
- Dependabot lock PRs still fail this lane until the lockfile bot
refreshes the pins. This PR does not change that path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent with tool use (shell, git,
GitHub CLI). Commits are attributed to Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:03:39 -06:00

161 lines
10 KiB
JavaScript

// Offline source-pin maintenance only. Compiler/parser/zod pins are never regenerated here.
// lockSha256 is not chosen here either: it is copied from the committed reviewed report-lock
// policy (tests/runner-e2e/report-lock.json), which only the trusted lockfile refresh writes.
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { lstat, readFile, readdir, realpath, writeFile } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
const registryPath = "tests/runner-e2e/source-sdk-public-inputs.json";
const reportLockPath = "tests/runner-e2e/report-lock.json";
const sourceRoots = ["packages/shared/src", "packages/plugins/sdk/src"];
const buildInputs = ["package.json", "packages/plugins/sdk/package.json", "packages/plugins/sdk/tsconfig.json",
"packages/shared/package.json", "packages/shared/tsconfig.json", "pnpm-lock.yaml",
"scripts/ensure-plugin-build-deps.mjs", "tsconfig.base.json", "tsconfig.json"];
const sourcePath = file => sourceRoots.some(root => file.startsWith(`${root}/`));
const blob = bytes => createHash("sha1").update(`blob ${bytes.length}\0`).update(bytes).digest("hex");
const sha256 = bytes => createHash("sha256").update(bytes).digest("hex");
const fail = () => { throw new Error("SDK public input synchronization refused"); };
const git = (root, args) => execFileSync("git", args, { cwd: root, encoding: "utf8", maxBuffer: 4 * 1024 * 1024, stdio: ["ignore", "pipe", "pipe"] });
const committedBytes = (root, revision, file) => execFileSync("git", ["show", `${revision}:${file}`], { cwd: root, maxBuffer: 64 * 1024 * 1024, stdio: ["ignore", "pipe", "pipe"] });
const hex64 = value => typeof value === "string" && /^[a-f0-9]{64}$/.test(value);
/**
* The reviewed lock identity the SDK proof must use at this revision: the committed
* report-lock policy's output hash, admitted only while the committed lock is that
* policy's reviewed input or output (the same rule the reviewed SDK reference applies).
*/
export function reviewedSdkLockSha256(root, revision) {
const policy = JSON.parse(committedBytes(root, revision, reportLockPath).toString("utf8"));
if (!hex64(policy.inputSha256) || !hex64(policy.outputSha256)) fail();
const committedLock = sha256(committedBytes(root, revision, "pnpm-lock.yaml"));
if (![policy.inputSha256, policy.outputSha256].includes(committedLock)) {
throw new Error(`SDK public input synchronization refused: the committed pnpm-lock.yaml is not reviewed by ${reportLockPath}`);
}
return policy.outputSha256;
}
export async function synchronizeSdkPublicInputs(root, revision, { write = false } = {}) {
if (!/^[a-f0-9]{40}$/.test(revision) || await realpath(root) !== root || git(root, ["rev-parse", "HEAD"]).trim() !== revision) fail();
// The reviewed committed registry supplies every non-source identity. Never trust a locally edited pin.
const baselineBytes = git(root, ["show", `${revision}:${registryPath}`]);
const baseline = JSON.parse(baselineBytes);
if (!baseline.inputs || Object.keys(baseline.inputs).filter(file => !sourcePath(file)).sort().join("\0") !== [...buildInputs].sort().join("\0")) fail();
const entries = git(root, ["ls-tree", "-rz", revision, "--", ...sourceRoots, ...buildInputs]).split("\0").filter(Boolean);
const inputs = {};
for (const entry of entries) {
const match = /^(100644|100755) blob ([a-f0-9]{40})\t([A-Za-z0-9._/-]+)$/.exec(entry);
if (!match || match[3].split("/").includes("..") || Object.hasOwn(inputs, match[3])) fail();
inputs[match[3]] = match[2];
}
const lockSha256 = reviewedSdkLockSha256(root, revision);
if (buildInputs.some(file => !inputs[file]) || sourceRoots.some(directory => !Object.keys(inputs).some(file => file.startsWith(`${directory}/`)))) fail();
const actualSources = [];
const collect = async directory => {
if (await realpath(path.join(root, directory)) !== path.join(root, directory)) fail();
for (const entry of await readdir(path.join(root, directory), { withFileTypes: true })) {
const file = `${directory}/${entry.name}`;
if (entry.isDirectory()) await collect(file);
else if (entry.isFile()) actualSources.push(file);
else fail();
}
};
for (const directory of sourceRoots) await collect(directory);
if (actualSources.sort().join("\0") !== Object.keys(inputs).filter(sourcePath).sort().join("\0")) fail();
for (const [file, expected] of Object.entries(inputs)) {
const filename = path.join(root, file), stat = await lstat(filename);
if (!stat.isFile() || stat.isSymbolicLink() || await realpath(filename) !== filename || stat.size > 8 * 1024 * 1024) fail();
const bytes = await readFile(filename);
// Preserve the existing, separately authenticated restored-lock exception.
if (blob(bytes) !== expected && !(file === "pnpm-lock.yaml" && sha256(bytes) === lockSha256)) fail();
}
const updated = { ...baseline, lockSha256, inputs: Object.fromEntries(Object.entries(inputs).sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0)) };
const output = `${JSON.stringify(updated, null, 2)}\n`;
const filename = path.join(root, registryPath), stat = await lstat(filename);
if (!stat.isFile() || stat.isSymbolicLink() || await realpath(filename) !== filename) fail();
const current = await readFile(filename, "utf8");
// A second invocation may check the exact generated bytes, but may not replace unrelated edits.
if (current !== baselineBytes && current !== output) fail();
if (git(root, ["rev-parse", "HEAD"]).trim() !== revision) fail();
const result = { revision, inputs: Object.keys(inputs).length, changed: current !== output,
registrySha256: sha256(output), lockPinChanged: baseline.lockSha256 !== lockSha256, runtimePinsChanged: false };
if (write && result.changed) await writeFile(filename, output);
return result;
}
/**
* Cheap drift guard: the source pins in the registry file must equal the blobs
* this checkout committed, and its lockSha256 must equal the committed reviewed
* report-lock output. Compares Git trees only (no working-tree reads), so a
* stale registry fails with the inputs that changed instead of the opaque
* "SDK public source proof unavailable" or "Reviewed SDK reference policy does
* not match public proof" raised later by the source proof.
*/
export async function staleSdkPublicInputs(root, revision = "HEAD") {
const registry = JSON.parse(await readFile(path.join(root, registryPath), "utf8"));
const committed = {};
for (const entry of git(root, ["ls-tree", "-rz", revision, "--", ...sourceRoots, ...buildInputs]).split("\0").filter(Boolean)) {
const match = /^\d+ blob ([a-f0-9]{40})\t(.+)$/.exec(entry);
if (match) committed[match[2]] = match[1];
}
// A lockfile refresh rewrites report-lock.json; lockSha256 must follow its output hash.
const policy = JSON.parse(git(root, ["show", `${revision}:${reportLockPath}`]));
return compareSdkPublicInputs(registry, committed, policy);
}
function compareSdkPublicInputs(registry, actual, policy) {
const pinned = registry.inputs ?? {};
const stale = [...new Set([...Object.keys(pinned), ...Object.keys(actual)])].sort()
.filter(file => pinned[file] !== actual[file])
.map(file => ({ file, change: !pinned[file] ? "unpinned" : !actual[file] ? "removed" : "changed" }));
if (registry.lockSha256 !== policy.outputSha256) stale.push({ file: `${reportLockPath} outputSha256`, change: "lockSha256 changed" });
return stale;
}
const committedLockBlob = root => {
try { return git(root, ["rev-parse", "--verify", "--quiet", "HEAD:pnpm-lock.yaml"]).trim(); }
catch { return null; }
};
/** True when a repository path is one of the inputs the registry pins. */
export const isSdkPublicInput = file => sourcePath(file) || buildInputs.includes(file);
/**
* The same drift guard over the working tree (tracked and untracked, non-ignored
* files), so local verification reports a missing registry refresh before the
* change is pushed. CI runs the committed-tree guard in the Runner support suite.
*/
export async function staleSdkPublicInputsInWorkingTree(root) {
const registry = JSON.parse(await readFile(path.join(root, registryPath), "utf8"));
const actual = {};
const listed = git(root, ["ls-files", "-z", "--cached", "--others", "--exclude-standard", "--", ...sourceRoots, ...buildInputs]);
for (const file of [...new Set(listed.split("\0").filter(Boolean))]) {
try {
const bytes = await readFile(path.join(root, file));
// Preserve the authenticated restored-lock exception: prepare-report-lock.mjs may leave the
// reviewed output lock in the working tree while the registry keeps the committed input pin.
// It applies only as an uncommitted overlay over a committed lock that matches its pin.
actual[file] = file === "pnpm-lock.yaml" && hex64(registry.lockSha256) && sha256(bytes) === registry.lockSha256
&& committedLockBlob(root) === registry.inputs?.[file] ? registry.inputs[file] : blob(bytes);
} catch (error) { if (error.code !== "ENOENT") throw error; }
}
const policy = JSON.parse(await readFile(path.join(root, reportLockPath), "utf8"));
return compareSdkPublicInputs(registry, actual, policy);
}
/** The actionable failure text shared by the CI drift guard and local verification. */
export function staleSdkPublicInputsMessage(stale) {
return `${registryPath} is stale for ${stale.length} pinned input(s): ${stale.slice(0, 10).map(row => `${row.file} (${row.change})`).join(", ")}${stale.length > 10 ? ", ..." : ""}. `
+ "Commit your change, run `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --write`, and commit the updated registry.";
}
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
const [revision, mode] = process.argv.slice(2);
if (process.argv.length !== 4 || !["--check", "--write"].includes(mode)) throw new Error("Usage: node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD-40hex> --check|--write");
const root = await realpath(path.resolve(fileURLToPath(new URL("../..", import.meta.url))));
const result = await synchronizeSdkPublicInputs(root, revision, { write: mode === "--write" });
console.log(JSON.stringify(result));
if (mode === "--check" && result.changed) process.exitCode = 1;
}