mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(ci): skip reviewed-lock checks for a pending Dependabot lock in the Runner support suite (#21942)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Dependabot pull requests that change `pnpm-lock.yaml` are the only way, other than the lockfile refresh, that lock content changes > - #19356 let a verified Dependabot lockfile pass CI. The server's reporting-lock policy test skips when `PAPERCLIP_CI_PR_AUTHOR` is `dependabot[bot]`, and the lockfile refresh renews the proof after the merge > - #19537 then made the Runner support suite a required check. That job does not set `PAPERCLIP_CI_PR_AUTHOR`, and the suite's own reviewed-lock checks have no Dependabot skip > - So every Dependabot lockfile pull request fails `ci / Runner support suite` and `ci / verify`. The last one that merged, #17364, merged before #19537 > - This pull request gives the support suite the same Dependabot skip as the server test > - The benefit is that Dependabot dependency updates can merge again, with the same lock trust rules as before ## Linked Issues or Issue Description Refs #19356, #19537, #19614, #17172, #19575. Rebased onto `master` after #21943, which fixed the unrelated `qualify_test_selection` failure that a CI file change triggers. **What happened?** Dependabot pull request #17172 changes `pnpm-lock.yaml`. `ci / Runner support suite` fails in four test files, and `ci / verify` fails because of it: - `prepare-report-lock.test.ts` > installs only the reviewed dependency correction (4 cases): the committed lock hash is not in `report-lock.json`. - `source-sdk-public-type.test.ts`: the shared setup throws "Committed SDK reference lock is not reviewed". - `source-sdk-evidence.test.ts` > seals a normal producer-certified reviewed-reference SDK: the same error. - `source-sdk-public-inputs.test.mjs` > the committed SDK public source pins match this checkout's committed tree: `pnpm-lock.yaml` is stale in `source-sdk-public-inputs.json`. A Dependabot pull request may not change `report-lock.json`, `report-lock.patch` or the SDK pins. `lockfile-trust.mjs` rejects that change, and `refresh-lockfile.yml` renews those files after the merge. So the pull request can never pass. **Expected behavior** A verified Dependabot lockfile passes the Runner support suite, as it already passes the server's reporting-lock policy test. Other authors keep every check. **Steps to reproduce** 1. Open a Dependabot pull request that changes `pnpm-lock.yaml`, such as #17172. 2. Let CI run. `ci / Runner support suite` fails in the four files above. ## What Changed - `.github/workflows/pr-trusted.yml`: the `verify_runner_e2e_support` job sets `PAPERCLIP_CI_PR_AUTHOR` from the pull request author, like the general test job. - `tests/runner-e2e/prepare-report-lock.test.ts`, `source-sdk-evidence.test.ts`, `source-sdk-public-inputs.test.mjs`: the test cases that need a reviewed committed lock skip when `PAPERCLIP_CI_PR_AUTHOR` is `dependabot[bot]`. This uses the same check as `server/scripts/reporting-lock-policy.test.mjs`. The other cases in these files still run. - `tests/runner-e2e/vitest.config.ts`: on a Dependabot pull request, the suite excludes `source-sdk-public-type.test.ts`. Its shared `beforeAll` prepares the reviewed SDK reference, so every case in the file needs a reviewed lock. - `tests/runner-e2e/source-sdk-public-type.ts`: the installed TypeScript, Babel parser and Zod package hash checks move into an exported `verifySdkProofPackages`, which `verifySdkPublicSource` calls. The lockfile refresh renews the lock and source pins after a merge, but keeps these package hashes. - `tests/runner-e2e/source-sdk-proof-packages.test.ts` (new): runs `verifySdkProofPackages` against the installed packages on every pull request, including Dependabot ones, and checks that a changed package pin is rejected. So a Dependabot update of one of those packages still fails until its pins are reviewed. ## Verification - Red, then green, with a real Dependabot lock: I committed the `pnpm-lock.yaml` and `package.json` changes of #17172 on top of this branch in a local commit (not pushed). Then I ran `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts` on the four files: - Without `PAPERCLIP_CI_PR_AUTHOR`: 4 files failed and the same 6 tests failed as in CI. - With `PAPERCLIP_CI_PR_AUTHOR=dependabot[bot]`: 3 files passed, 33 tests passed and 6 skipped. `source-sdk-public-type.test.ts` was excluded. - On this branch with master's lock: `pnpm test:e2e:runner:unit` passed (Vitest: 687 files, 11640 tests; node tests passed). - `node --test .github/scripts/tests/*.test.mjs`: 808 passed. - After the package check change: the five affected files (`source-sdk-proof-packages`, `source-sdk-public-type`, `source-sdk-evidence`, `source-sdk-public-inputs`, `prepare-report-lock`) pass, 71 tests. `tsc --noEmit -p tests/runner-e2e/tsconfig.json` passes. - `pr.yml` calls `pr-trusted.yml@master`, so this pull request's own CI does not use the new job variable. The first Dependabot pull request after the merge checks it. ## Risks - A Dependabot update of TypeScript, the Babel parser or Zod still fails the suite until a maintainer reviews the new package pins. That is deliberate. - On a Dependabot pull request, the suite no longer checks the committed lock against the reviewed proof. The policy job still requires that the lock comes from a verified Dependabot commit (`lockfile-trust.mjs pr`), and the lockfile refresh still admits the lock and renews the proof and the SDK pins after the merge. That is the same trust as #19356. - `PAPERCLIP_CI_PR_AUTHOR` is the pull request author from the GitHub event. A contributor cannot set it to `dependabot[bot]`. Other commits pushed to a Dependabot branch are still checked by the policy job. - No runtime code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
b2b2c1e60b
commit
6d117998c8
7 files changed
+42
-10
No files matched your search
@@ -756,6 +756,7 @@ jobs:
|
||||
env:
|
||||
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
|
||||
PAPERCLIP_CI_PROFILE_JOB: Runner support suite
|
||||
PAPERCLIP_CI_PR_AUTHOR: ${{ github.event.pull_request.user.login }}
|
||||
|
||||
# The paid Runner workflows depend on this suite: workflow security
|
||||
# policy, evidence oracles, cost receipts and controller recovery. It
|
||||
|
||||
@@ -7,7 +7,9 @@ import { expect, it } from 'vitest';
|
||||
import { parse } from 'yaml';
|
||||
import { prepareReportLock, refreshReportLock } from './prepare-report-lock.mjs';
|
||||
|
||||
it.each(['valid', 'unknown-lock', 'changed-patch', 'wrong-output'])(
|
||||
const pendingDependabotLock = process.env.PAPERCLIP_CI_PR_AUTHOR === 'dependabot[bot]';
|
||||
|
||||
it.skipIf(pendingDependabotLock).each(['valid', 'unknown-lock', 'changed-patch', 'wrong-output'])(
|
||||
'installs only the reviewed dependency correction: %s', async variant => {
|
||||
const root = await mkdtemp(path.join(tmpdir(), 'eval-report-lock-test-'));
|
||||
const here = import.meta.dirname, relative = path.join(root, 'tests/runner-e2e');
|
||||
|
||||
@@ -18,6 +18,7 @@ import { MANUAL_PRIVATE_EXECUTIONS, MANUAL_PRIVATE_PROTOCOL } from "./manual-pri
|
||||
|
||||
const roots: string[] = [], executionId = "chat-features.runner-opencode.local.plugins.install-plugin", sourceRevision = "a".repeat(40);
|
||||
const hash = (bytes: Buffer | string) => createHash("sha256").update(bytes).digest("hex");
|
||||
const pendingDependabotLock = process.env.PAPERCLIP_CI_PR_AUTHOR === "dependabot[bot]";
|
||||
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
|
||||
async function fixture(secrets: string[] = []) {
|
||||
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "source-sdk-synthetic-unit-"))); roots.push(root);
|
||||
@@ -116,7 +117,7 @@ it("retains declarations and source maps under names the private result sealer a
|
||||
// CI lock can be regenerated for unrelated PR patches and must not be relabeled reviewed.
|
||||
const builtSdk = ["packages/shared/dist", "packages/plugins/sdk/dist"].every(target =>
|
||||
existsSync(new URL(`../../${target}/.paperclip-build-complete`, import.meta.url)));
|
||||
it("seals a normal producer-certified reviewed-reference SDK, keeping key-shaped declarations out", async () => {
|
||||
it.skipIf(pendingDependabotLock)("seals a normal producer-certified reviewed-reference SDK, keeping key-shaped declarations out", async () => {
|
||||
const originalRoot=await realpath(path.resolve(import.meta.dirname,"../.."));
|
||||
const repositoryRoot=await realpath(await mkdtemp(path.join(os.tmpdir(),"source-sdk-reviewed-build-")));roots.push(repositoryRoot);
|
||||
await prepareReviewedSdkReference(originalRoot,repositoryRoot);
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
import { realpath } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import pins from "./source-sdk-public-inputs.json" with { type: "json" };
|
||||
import { verifySdkProofPackages } from "./source-sdk-public-type.js";
|
||||
|
||||
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
|
||||
|
||||
it("installed TypeScript, parser and Zod packages match the reviewed SDK proof pins", async () => {
|
||||
await expect(verifySdkProofPackages(await realpath(repositoryRoot))).resolves.toBeTruthy();
|
||||
});
|
||||
|
||||
it("rejects an installed proof package that differs from its pin", async () => {
|
||||
await expect(verifySdkProofPackages(await realpath(repositoryRoot), { ...pins, zodPackageSha256: "0".repeat(64) }))
|
||||
.rejects.toThrow("SDK public source proof unavailable");
|
||||
});
|
||||
@@ -9,6 +9,7 @@ import { fileURLToPath } from "node:url";
|
||||
import { isSdkPublicInput, staleSdkPublicInputs, staleSdkPublicInputsInWorkingTree, staleSdkPublicInputsMessage, synchronizeSdkPublicInputs } from "./source-sdk-public-inputs.mjs";
|
||||
|
||||
const registry = "tests/runner-e2e/source-sdk-public-inputs.json";
|
||||
const pendingDependabotLock = process.env.PAPERCLIP_CI_PR_AUTHOR === "dependabot[bot]";
|
||||
const reportLock = "tests/runner-e2e/report-lock.json";
|
||||
const reportPolicy = (inputSha256, outputSha256) => `${JSON.stringify({ inputSha256, outputSha256, patchSha256: "0".repeat(64), basis: "synthetic" }, null, 2)}\n`;
|
||||
const commit = (root, message) => { git(root, "add", "-A"); git(root, "-c", "user.name=Synthetic unit", "-c", "user.email=synthetic@example.invalid", "commit", "-qm", message); return git(root, "rev-parse", "HEAD").trim(); };
|
||||
@@ -113,7 +114,7 @@ test("refuses a committed lock that the report-lock policy does not review", asy
|
||||
// source. A PR that edits packages/shared/src (or another pinned input) without
|
||||
// refreshing them breaks every SDK-dependent Runner cell before its server
|
||||
// starts, so fail here, cheaply, with the inputs and the fix command.
|
||||
test("the committed SDK public source pins match this checkout's committed tree", async () => {
|
||||
test.skipIf(pendingDependabotLock)("the committed SDK public source pins match this checkout's committed tree", async () => {
|
||||
const root = fileURLToPath(new URL("../..", import.meta.url));
|
||||
const stale = await staleSdkPublicInputs(root);
|
||||
assert.deepEqual(stale, [], staleSdkPublicInputsMessage(stale));
|
||||
|
||||
@@ -15,6 +15,19 @@ interface PublicSource {
|
||||
inputs: SdkPublicInputs; identity: { inputs: number; manifestSha256: string; lockSha256: string; compilerPackageSha256: string; parserPackageSha256: string; parserEntrySha256: string; zodPackageSha256: string; zodEntrySha256: string; zodTypeGraphSha256: string };
|
||||
}
|
||||
const verified = new WeakSet<PublicSource>();
|
||||
const matches = async (file: string, expected: string) => { if (hash(await readFile(file)) !== expected) throw new Error("SDK public source proof unavailable"); };
|
||||
export async function verifySdkProofPackages(root: string, inputs: SdkPublicInputs = defaultInputs) {
|
||||
await matches(path.join(root, "node_modules/typescript/package.json"), inputs.compilerPackageSha256);
|
||||
const parserRoot = path.join(root, "node_modules/.pnpm/@babel+parser@7.29.8/node_modules/@babel/parser");
|
||||
await matches(path.join(parserRoot, "package.json"), inputs.parserPackageSha256); await matches(path.join(parserRoot, "lib/index.js"), inputs.parserEntrySha256);
|
||||
const require = createRequire(path.join(root, "packages/shared/package.json")), zodPackage = require.resolve("zod/package.json"), zodEntry = require.resolve("zod");
|
||||
await matches(zodPackage, inputs.zodPackageSha256); await matches(zodEntry, inputs.zodEntrySha256);
|
||||
for (const [relative, expected] of Object.entries(inputs.zodTypeFiles)) {
|
||||
if (!/^[A-Za-z0-9._/-]+$/.test(relative) || relative.split("/").includes("..")) throw new Error("SDK public source proof unavailable");
|
||||
await matches(path.join(path.dirname(zodPackage), relative), expected);
|
||||
}
|
||||
return { parserRoot, require, zodPackage };
|
||||
}
|
||||
export async function verifySdkPublicSource(root: string, expectations?: SdkPublicInputs): Promise<PublicSource> {
|
||||
const supplied = expectations ?? defaultInputs;
|
||||
const inputs: SdkPublicInputs = Object.freeze({ ...supplied, inputs: Object.freeze({ ...supplied.inputs }), zodTypeFiles: Object.freeze({ ...supplied.zodTypeFiles }) });
|
||||
@@ -39,13 +52,8 @@ export async function verifySdkPublicSource(root: string, expectations?: SdkPubl
|
||||
for (const directory of ["packages/shared/src", "packages/plugins/sdk/src"]) await collect(directory);
|
||||
const expectedSources = Object.keys(inputs.inputs).filter(file => file.startsWith("packages/shared/src/") || file.startsWith("packages/plugins/sdk/src/"));
|
||||
if (JSON.stringify(actualSources.sort()) !== JSON.stringify(expectedSources.sort())) throw new Error("SDK public source proof unavailable");
|
||||
const matches = async (file: string, expected: string) => { if (hash(await readFile(file)) !== expected) throw new Error("SDK public source proof unavailable"); };
|
||||
await matches(path.join(root, "pnpm-lock.yaml"), inputs.lockSha256);
|
||||
await matches(path.join(root, "node_modules/typescript/package.json"), inputs.compilerPackageSha256);
|
||||
const parserRoot = path.join(root, "node_modules/.pnpm/@babel+parser@7.29.8/node_modules/@babel/parser");
|
||||
await matches(path.join(parserRoot, "package.json"), inputs.parserPackageSha256); await matches(path.join(parserRoot, "lib/index.js"), inputs.parserEntrySha256);
|
||||
const require = createRequire(path.join(root, "packages/shared/package.json")), zodPackage = require.resolve("zod/package.json"), zodEntry = require.resolve("zod");
|
||||
await matches(zodPackage, inputs.zodPackageSha256); await matches(zodEntry, inputs.zodEntrySha256);
|
||||
const { parserRoot, require, zodPackage } = await verifySdkProofPackages(root, inputs);
|
||||
const parser = createRequire(path.join(parserRoot, "package.json"))("./lib/index.js");
|
||||
const publicTypes = new Map<string, any>();
|
||||
for (const [relative, expected] of Object.entries(inputs.zodTypeFiles)) {
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { configDefaults, defineConfig } from "vitest/config";
|
||||
|
||||
const pendingDependabotLock = process.env.PAPERCLIP_CI_PR_AUTHOR === "dependabot[bot]";
|
||||
|
||||
export default defineConfig({
|
||||
root: import.meta.dirname,
|
||||
test: {
|
||||
@@ -12,7 +14,8 @@ export default defineConfig({
|
||||
globalSetup: ["./verified-provider-bundles.setup.mjs"],
|
||||
include: ["**/*.test.ts", "**/*.test.mjs"],
|
||||
// These use node:test and run separately in test:e2e:runner:unit.
|
||||
exclude: [...configDefaults.exclude, "product-regression-testops.test.mjs", "reliability-testops.test.mjs", "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs", "chat-completion-testops-{api,contract}.test.mjs", "chat-full-catalog-testops-*.test.mjs", "chat-catalog-testops-streaming.test.mjs"],
|
||||
exclude: [...configDefaults.exclude, "product-regression-testops.test.mjs", "reliability-testops.test.mjs", "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs", "chat-completion-testops-{api,contract}.test.mjs", "chat-full-catalog-testops-*.test.mjs", "chat-catalog-testops-streaming.test.mjs",
|
||||
...(pendingDependabotLock ? ["source-sdk-public-type.test.ts"] : [])],
|
||||
testTimeout: 30_000,
|
||||
},
|
||||
});
|
||||
Reference in new issue
Block a user