mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(verify): report stale SDK public input pins locally, before CI (#21283)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types against
pinned source. The pins are in
`tests/runner-e2e/source-sdk-public-inputs.json`.
> - The pins cover every file in `packages/shared/src` and
`packages/plugins/sdk/src`, and the SDK build inputs (for example
`pnpm-lock.yaml`).
> - A PR that edits a pinned file and does not refresh the registry
fails three files in the "Runner support suite" lane.
> - Between 2026-11-12 10:27Z and 2026-11-13 01:14Z, this failure
occurred in 14 CI runs on 12 PRs.
> - The local check, `pnpm verify`, returned exit 2 ("let CI cover it")
for these changes. It did not mention the pins. Authors pushed, and CI
failed.
> - This pull request makes `pnpm verify` find stale pins before the
push. It also makes the opaque fixture error name the file and the fix.
> - The benefit is that authors find and fix the problem in seconds,
locally, instead of after a 6-minute CI lane.
## Linked Issues or Issue Description
**What happened?**
Three files in the "Runner support suite" lane failed together on many
unrelated PRs: `source-sdk-evidence.test.ts`,
`source-sdk-public-inputs.test.mjs` and
`source-sdk-public-type.test.ts`. Two of the three failed with the
opaque error `Reviewed SDK reference source changed`.
**Root cause**
The pins on master are not stale. The tests pass on master (58/58). Each
failing PR edited a pinned input and did not refresh the registry. I
compared the PR heads with their merge base:
- All 14 failing heads (12 PRs) changed a pinned input. None of them
changed `source-sdk-public-inputs.json`.
- All 19 heads in the same window that changed a pinned input and passed
this lane also refreshed the registry. This includes later heads of
three PRs that failed first (#21119, #21121, #21156).
- Two of the 12 PRs are Dependabot PRs that change only `pnpm-lock.yaml`
(#19593, #19594). The lockfile refresh bot already resyncs the pins for
lock changes on master.
**Steps to reproduce**
1. Start from `origin/master`.
2. Add one line to `packages/shared/src/types/heartbeat.ts` and commit
it.
3. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts source-sdk-evidence.test.ts
source-sdk-public-inputs.test.mjs source-sdk-public-type.test.ts`.
4. The result is the same as CI: 3 files fail, with the same 4 failures.
5. Run `pnpm verify --explain`. It returns exit 2 and does not mention
the pins.
**Expected behavior**
`pnpm verify` reports the stale pins and the refresh command before the
author pushes.
## What Changed
- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputsInWorkingTree(root)`. It applies the existing drift
guard to the working tree (tracked and untracked files that git does not
ignore). Add `isSdkPublicInput(file)` and the shared failure message
`staleSdkPublicInputsMessage(stale)`. The committed-tree guard keeps its
behavior and now uses a shared comparison.
- `scripts/local-verify.mjs`: when the diff touches a pinned input, the
registry or `report-lock.json`, `pnpm verify` checks the pins first. If
they are stale, it exits 1 and prints the files and the refresh command.
The check reads files only. It does not write the registry.
- `tests/runner-e2e/source-sdk-reviewed-fixture.ts`: the error `Reviewed
SDK reference source changed` now names the file and gives the refresh
command.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: new test for the
working-tree guard (uncommitted edit, untracked file, ignored file). The
CI drift test uses the shared message.
- `doc/DEVELOPING.md`: document the new check under "Incremental local
verification".
No assertion is weakened. No pin is changed. No file under `.github/` is
changed.
## Verification
- Master, without this change: the three files pass (58/58).
- Red: on master plus a one-line edit to
`packages/shared/src/types/heartbeat.ts`, the three files fail with the
CI signature. `pnpm verify --explain` returns exit 2 and does not
mention the pins.
- With this change on the same red commit: `pnpm verify` exits 1 in
about 15 s and prints `tests/runner-e2e/source-sdk-public-inputs.json is
stale for 1 pinned input(s): packages/shared/src/types/heartbeat.ts
(changed). Commit your change, run ... --write, and commit the updated
registry.`
- Green: I followed the printed command (`node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`, then
commit). The three files then pass (59/59, with the new test).
- `node --test scripts/__tests__/local-verify.test.mjs`: 14/14 pass.
- `pnpm verify` on this branch returns exit 2 because
`scripts/local-verify.mjs` is a global CI input. Required CI supplies
the broad coverage.
## Risks
- Low risk. The new check only reads files. It runs only when the diff
touches a pinned input.
- `pnpm verify` now fails (exit 1) for a change that also fails in CI.
Before, it returned exit 2.
- The check needs a working `git ls-files` in the repository. `pnpm
verify` already needs git.
- Dependabot lock PRs still fail this lane until the lockfile bot
refreshes the pins. This PR does not change that path.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude (Anthropic), Claude Code agent with tool use (shell, git,
GitHub CLI). Commits are attributed to Claude Fable 5.2.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
0c63f0fdb3
commit
db3d7223e0
6 files changed
+151
-10
No files matched your search
@@ -1,5 +1,6 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync, spawn, spawnSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { chmodSync, cpSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, symlinkSync, utimesSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
@@ -288,6 +289,49 @@ process.stdout.write(result);
|
||||
assert.doesNotMatch(result.text, /Local Vitest verification passed/);
|
||||
});
|
||||
|
||||
test("stale SDK public input pins stop verification before selection; fresh pins and unrelated changes continue", async t => {
|
||||
const f = fixture(t);
|
||||
const sdkInputs = "tests/runner-e2e/source-sdk-public-inputs.mjs", registry = "tests/runner-e2e/source-sdk-public-inputs.json";
|
||||
const blob = text => createHash("sha1").update(`blob ${Buffer.byteLength(text)}\0`).update(text).digest("hex");
|
||||
const outputSha256 = "a".repeat(64);
|
||||
f.write(sdkInputs, readFileSync(path.join(repository, sdkInputs), "utf8"));
|
||||
f.write("tests/runner-e2e/report-lock.json", `${JSON.stringify({ inputSha256: "b".repeat(64), outputSha256 })}\n`);
|
||||
f.write("packages/shared/src/schema.ts", "export const schema = 1;\n");
|
||||
const writeRegistry = schema => f.write(registry, `${JSON.stringify({ lockSha256: outputSha256, inputs: {
|
||||
"package.json": blob(readFileSync(path.join(f.root, "package.json"), "utf8")), "packages/shared/src/schema.ts": blob(schema) } }, null, 2)}\n`);
|
||||
writeRegistry("export const schema = 1;\n");
|
||||
f.git("add", "."); f.git("-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "sdk pin baseline");
|
||||
const base = f.git("rev-parse", "HEAD");
|
||||
const run = args => {
|
||||
const result = spawnSync(process.execPath, [path.join(f.root, "scripts/local-verify.mjs"), "--base", base, ...args], {
|
||||
cwd: f.root, env: f.env, encoding: "utf8", timeout: 120_000, maxBuffer: 8 * 1024 * 1024 });
|
||||
assert.equal(result.error, undefined);
|
||||
return { ...result, text: stripVTControlCharacters(result.stdout + result.stderr) };
|
||||
};
|
||||
// Unrelated change: the pin check does not run, even with a stale registry in the tree.
|
||||
f.write("ui/src/leaf.ts", "export const value = 3;\n");
|
||||
f.write("packages/shared/src/schema.ts", "export const schema = 2;\n");
|
||||
f.git("add", "packages/shared/src/schema.ts"); f.git("-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "untracked drift");
|
||||
const unrelatedBase = f.git("rev-parse", "HEAD");
|
||||
const unrelated = spawnSync(process.execPath, [path.join(f.root, "scripts/local-verify.mjs"), "--base", unrelatedBase, "--explain"], {
|
||||
cwd: f.root, env: f.env, encoding: "utf8", timeout: 120_000, maxBuffer: 8 * 1024 * 1024 });
|
||||
assert.doesNotMatch(unrelated.stdout + unrelated.stderr, /is stale/);
|
||||
// Pinned input changed without a registry refresh: exit 1, name the input and the command, write nothing.
|
||||
const before = readFileSync(path.join(f.root, registry));
|
||||
const stale = run(["--explain"]);
|
||||
assert.equal(stale.status, 1, stale.text);
|
||||
assert.match(stale.text, /source-sdk-public-inputs\.json is stale for 1 pinned input\(s\): packages\/shared\/src\/schema\.ts \(changed\)/);
|
||||
assert.match(stale.text, /source-sdk-public-inputs\.mjs \$\(git rev-parse HEAD\) --write/);
|
||||
assert.doesNotMatch(stale.text, /Vitest test files selected|Broader verification required/);
|
||||
assert.deepEqual(readFileSync(path.join(f.root, registry)), before);
|
||||
// Refreshed pins: verification continues to selection.
|
||||
writeRegistry("export const schema = 2;\n");
|
||||
const fresh = run(["--explain"]);
|
||||
assert.doesNotMatch(fresh.text, /is stale/);
|
||||
assert.match(fresh.text, /Vitest test files selected|Broader verification required/);
|
||||
assert.notEqual(fresh.status, 1, fresh.text);
|
||||
});
|
||||
|
||||
test("documentation-only changes and empty diffs do not start a full suite", async t => {
|
||||
const f = fixture(t);
|
||||
f.write("ui/src/leaf.ts", "export const value = 1;\n");
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#!/usr/bin/env node
|
||||
import { realpathSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { existsSync, realpathSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { createHash } from "node:crypto";
|
||||
import { comparison, localEnvironment, protocolVersion, requestKey, snapshot } from "./local-verify/inputs.mjs";
|
||||
@@ -16,12 +17,31 @@ function describe(plan, cached) {
|
||||
console.log("[verify] Local Vitest coverage does not replace required CI, typecheck/build, browser, release, or platform-specific checks.");
|
||||
}
|
||||
|
||||
// A change to packages/shared/src, the plugin SDK source, or an SDK build input must
|
||||
// refresh the committed SDK public input pins, or every SDK-dependent Runner suite
|
||||
// fails in CI. The check reads only the working tree, so report it before selection.
|
||||
const sdkPublicInputs = "tests/runner-e2e/source-sdk-public-inputs.mjs";
|
||||
async function staleSdkPublicInputPins(root, changed) {
|
||||
const module = path.join(root, sdkPublicInputs);
|
||||
if (!existsSync(module)) return null;
|
||||
const { isSdkPublicInput, staleSdkPublicInputsInWorkingTree, staleSdkPublicInputsMessage } = await import(pathToFileURL(module).href);
|
||||
const pins = ["tests/runner-e2e/source-sdk-public-inputs.json", "tests/runner-e2e/report-lock.json"];
|
||||
if (!changed.some(file => isSdkPublicInput(file) || pins.includes(file))) return null;
|
||||
const stale = await staleSdkPublicInputsInWorkingTree(root);
|
||||
return stale.length ? staleSdkPublicInputsMessage(stale) : null;
|
||||
}
|
||||
|
||||
export async function verify({ root = process.cwd(), base, full = false, explain = false, noCache = false, env = localEnvironment() } = {}) {
|
||||
root = realpathSync(root);
|
||||
const before = snapshot(root, env), diff = comparison(root, base), key = requestKey(before, diff, full);
|
||||
const canReuse = !noCache && before.cacheable && mayReuseAttempt(root, key);
|
||||
const cached = canReuse ? readPass(root, key) : null;
|
||||
console.log(`[verify] Base ${diff.base.slice(0, 12)}; ${diff.changed.length} changed paths, including local edits`);
|
||||
const staleSdkPins = await staleSdkPublicInputPins(root, diff.changed);
|
||||
if (staleSdkPins) {
|
||||
console.error(`[verify] ${staleSdkPins}`);
|
||||
return 1;
|
||||
}
|
||||
if (cached) {
|
||||
if (snapshot(root, env).hash !== before.hash) throw new Error("Inputs changed during cache lookup. Run verification again.");
|
||||
describe(cached.plan, true);
|
||||
|
||||
Reference in new issue
Block a user