Files
PaperClipAI/tests/runner-e2e/source-sdk-reviewed-fixture.ts
T
DottaandClaude Fable 5.2 db3d7223e0 fix(verify): report stale SDK public input pins locally, before CI (#21283)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types against
pinned source. The pins are in
`tests/runner-e2e/source-sdk-public-inputs.json`.
> - The pins cover every file in `packages/shared/src` and
`packages/plugins/sdk/src`, and the SDK build inputs (for example
`pnpm-lock.yaml`).
> - A PR that edits a pinned file and does not refresh the registry
fails three files in the "Runner support suite" lane.
> - Between 2026-11-12 10:27Z and 2026-11-13 01:14Z, this failure
occurred in 14 CI runs on 12 PRs.
> - The local check, `pnpm verify`, returned exit 2 ("let CI cover it")
for these changes. It did not mention the pins. Authors pushed, and CI
failed.
> - This pull request makes `pnpm verify` find stale pins before the
push. It also makes the opaque fixture error name the file and the fix.
> - The benefit is that authors find and fix the problem in seconds,
locally, instead of after a 6-minute CI lane.

## Linked Issues or Issue Description

**What happened?**
Three files in the "Runner support suite" lane failed together on many
unrelated PRs: `source-sdk-evidence.test.ts`,
`source-sdk-public-inputs.test.mjs` and
`source-sdk-public-type.test.ts`. Two of the three failed with the
opaque error `Reviewed SDK reference source changed`.

**Root cause**
The pins on master are not stale. The tests pass on master (58/58). Each
failing PR edited a pinned input and did not refresh the registry. I
compared the PR heads with their merge base:

- All 14 failing heads (12 PRs) changed a pinned input. None of them
changed `source-sdk-public-inputs.json`.
- All 19 heads in the same window that changed a pinned input and passed
this lane also refreshed the registry. This includes later heads of
three PRs that failed first (#21119, #21121, #21156).
- Two of the 12 PRs are Dependabot PRs that change only `pnpm-lock.yaml`
(#19593, #19594). The lockfile refresh bot already resyncs the pins for
lock changes on master.

**Steps to reproduce**
1. Start from `origin/master`.
2. Add one line to `packages/shared/src/types/heartbeat.ts` and commit
it.
3. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts source-sdk-evidence.test.ts
source-sdk-public-inputs.test.mjs source-sdk-public-type.test.ts`.
4. The result is the same as CI: 3 files fail, with the same 4 failures.
5. Run `pnpm verify --explain`. It returns exit 2 and does not mention
the pins.

**Expected behavior**
`pnpm verify` reports the stale pins and the refresh command before the
author pushes.

## What Changed

- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputsInWorkingTree(root)`. It applies the existing drift
guard to the working tree (tracked and untracked files that git does not
ignore). Add `isSdkPublicInput(file)` and the shared failure message
`staleSdkPublicInputsMessage(stale)`. The committed-tree guard keeps its
behavior and now uses a shared comparison.
- `scripts/local-verify.mjs`: when the diff touches a pinned input, the
registry or `report-lock.json`, `pnpm verify` checks the pins first. If
they are stale, it exits 1 and prints the files and the refresh command.
The check reads files only. It does not write the registry.
- `tests/runner-e2e/source-sdk-reviewed-fixture.ts`: the error `Reviewed
SDK reference source changed` now names the file and gives the refresh
command.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: new test for the
working-tree guard (uncommitted edit, untracked file, ignored file). The
CI drift test uses the shared message.
- `doc/DEVELOPING.md`: document the new check under "Incremental local
verification".

No assertion is weakened. No pin is changed. No file under `.github/` is
changed.

## Verification

- Master, without this change: the three files pass (58/58).
- Red: on master plus a one-line edit to
`packages/shared/src/types/heartbeat.ts`, the three files fail with the
CI signature. `pnpm verify --explain` returns exit 2 and does not
mention the pins.
- With this change on the same red commit: `pnpm verify` exits 1 in
about 15 s and prints `tests/runner-e2e/source-sdk-public-inputs.json is
stale for 1 pinned input(s): packages/shared/src/types/heartbeat.ts
(changed). Commit your change, run ... --write, and commit the updated
registry.`
- Green: I followed the printed command (`node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`, then
commit). The three files then pass (59/59, with the new test).
- `node --test scripts/__tests__/local-verify.test.mjs`: 14/14 pass.
- `pnpm verify` on this branch returns exit 2 because
`scripts/local-verify.mjs` is a global CI input. Required CI supplies
the broad coverage.

## Risks

- Low risk. The new check only reads files. It runs only when the diff
touches a pinned input.
- `pnpm verify` now fails (exit 1) for a change that also fails in CI.
Before, it returned exit 2.
- The check needs a working `git ls-files` in the repository. `pnpm
verify` already needs git.
- Dependabot lock PRs still fail this lane until the lockfile bot
refreshes the pins. This PR does not change that path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent with tool use (shell, git,
GitHub CLI). Commits are attributed to Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:03:39 -06:00

75 lines
5.6 KiB
TypeScript

// Test-only reviewed SDK reference. Actual CI builds may use an unreviewed
// resolved lock; neither this fixture nor production admits that as public proof.
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { copyFile, mkdir, readFile, readdir, realpath, symlink, writeFile } from "node:fs/promises";
import { createRequire } from "node:module";
import path from "node:path";
import pins from "./source-sdk-public-inputs.json" with { type: "json" };
import { prepareReportLock } from "./prepare-report-lock.mjs";
import { verifySdkPublicSource } from "./source-sdk-public-type.js";
const sha=(bytes:Buffer)=>createHash("sha256").update(bytes).digest("hex");
const blob=(bytes:Buffer)=>createHash("sha1").update(`blob ${bytes.length}\0`).update(bytes).digest("hex");
const gitFile=(root:string,file:string)=>execFileSync("git",["show",`HEAD:${file}`],{cwd:root,maxBuffer:8*1024*1024,stdio:["ignore","pipe","pipe"]});
// Create owned directories, link only read-only installed third-party inputs.
// Never link workspace compiled exports or donor node_modules/.cache: the
// normal producer's lock and every Shared/SDK emit stay in this fixture root.
async function thirdPartyLinks(from:string,to:string,installedStore:string){
const link=async(source:string,destination:string)=>{
const target=await realpath(source);
// A local workspace may have an unscoped name too. Admit only installed
// third-party store entries, never another checkout/package's exports.
if(target!==installedStore&&!target.startsWith(installedStore+path.sep))return;
await symlink(target,destination);
};
await mkdir(to,{recursive:true});
for(const entry of await readdir(from,{withFileTypes:true})){
if([".cache",".bin","@paperclipai"].includes(entry.name))continue;
const source=path.join(from,entry.name),destination=path.join(to,entry.name);
if(entry.name.startsWith("@")){
await mkdir(destination,{recursive:true});
for(const child of await readdir(source))await link(path.join(source,child),path.join(destination,child));
}else await link(source,destination);
}
}
export async function prepareReviewedSdkReference(original:string,directory:string){
if(await realpath(directory)!==directory||(await readdir(directory)).length!==0)throw new Error("Reviewed SDK reference needs a fresh owned directory");
const policyFile="tests/runner-e2e/report-lock.json",patchFile="tests/runner-e2e/report-lock.patch";
const policyBytes=gitFile(original,policyFile),policy=JSON.parse(policyBytes.toString("utf8"));
if(!(await readFile(path.join(original,policyFile))).equals(policyBytes)||policy.outputSha256!==pins.lockSha256)
throw new Error("Reviewed SDK reference policy does not match public proof: source-sdk-public-inputs.json lockSha256 must equal the committed report-lock.json outputSha256. "
+"Commit the lock change, then run `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --write` and commit the updated registry.");
const committed=gitFile(original,"pnpm-lock.yaml");
if(![policy.inputSha256,policy.outputSha256].includes(sha(committed)))throw new Error("Committed SDK reference lock is not reviewed");
for(const [file,expected]of Object.entries(pins.inputs)){
const bytes=file==="pnpm-lock.yaml"?committed:await readFile(path.join(original,file));
if(file!=="pnpm-lock.yaml"&&blob(bytes)!==expected)throw new Error(`Reviewed SDK reference source changed: ${file} does not match its pin in tests/runner-e2e/source-sdk-public-inputs.json. `
+"Commit your change, run `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --write`, and commit the updated registry.");
const target=path.join(directory,file);await mkdir(path.dirname(target),{recursive:true});
if(file==="pnpm-lock.yaml")await writeFile(target,bytes);else await copyFile(path.join(original,file),target);
}
for(const file of [policyFile,patchFile]){
const bytes=gitFile(original,file);
if(!(await readFile(path.join(original,file))).equals(bytes))throw new Error("Reviewed SDK lock policy changed");
const target=path.join(directory,file);await mkdir(path.dirname(target),{recursive:true});
await writeFile(target,bytes);
}
// Same existing authenticated input->output correction. No hash refresh,
// changed-lock admission, install, resolver or compiler-option substitution.
await prepareReportLock(directory);
if(sha(await readFile(path.join(directory,"pnpm-lock.yaml")))!==policy.outputSha256)throw new Error("Reviewed SDK reference lock changed");
const installedStore=await realpath(path.join(original,"node_modules/.pnpm"));
await thirdPartyLinks(path.join(original,"node_modules"),path.join(directory,"node_modules"),installedStore);
for(const name of ["shared","plugins/sdk"])await thirdPartyLinks(path.join(original,"packages",name,"node_modules"),path.join(directory,"packages",name,"node_modules"),installedStore);
for(const modules of ["node_modules","packages/plugins/sdk/node_modules"]){
const scope=path.join(directory,modules,"@paperclipai");await mkdir(scope,{recursive:true});
await symlink(path.join(directory,"packages/shared"),path.join(scope,"shared"));
}
const require=createRequire(path.join(directory,"packages/plugins/sdk/package.json"));
const sharedEntry=await realpath(require.resolve("@paperclipai/shared"));
if(!sharedEntry.startsWith(path.join(directory,"packages/shared")+path.sep))throw new Error("SDK reference resolved a donor workspace");
// Authenticate original compiler/parser/Zod/source pins BEFORE normal build.
await verifySdkPublicSource(directory);
}