Files
PaperClipAI/tests/runner-e2e/source-sdk-public-type.ts
T
Devin FoleyandPaperclip 6d117998c8 fix(ci): skip reviewed-lock checks for a pending Dependabot lock in the Runner support suite (#21942)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Dependabot pull requests that change `pnpm-lock.yaml` are the only
way, other than the lockfile refresh, that lock content changes
> - #19356 let a verified Dependabot lockfile pass CI. The server's
reporting-lock policy test skips when `PAPERCLIP_CI_PR_AUTHOR` is
`dependabot[bot]`, and the lockfile refresh renews the proof after the
merge
> - #19537 then made the Runner support suite a required check. That job
does not set `PAPERCLIP_CI_PR_AUTHOR`, and the suite's own reviewed-lock
checks have no Dependabot skip
> - So every Dependabot lockfile pull request fails `ci / Runner support
suite` and `ci / verify`. The last one that merged, #17364, merged
before #19537
> - This pull request gives the support suite the same Dependabot skip
as the server test
> - The benefit is that Dependabot dependency updates can merge again,
with the same lock trust rules as before

## Linked Issues or Issue Description

Refs #19356, #19537, #19614, #17172, #19575. Rebased onto `master` after
#21943, which fixed the unrelated `qualify_test_selection` failure that
a CI file change triggers.

**What happened?**

Dependabot pull request #17172 changes `pnpm-lock.yaml`. `ci / Runner
support suite` fails in four test files, and `ci / verify` fails because
of it:

- `prepare-report-lock.test.ts` > installs only the reviewed dependency
correction (4 cases): the committed lock hash is not in
`report-lock.json`.
- `source-sdk-public-type.test.ts`: the shared setup throws "Committed
SDK reference lock is not reviewed".
- `source-sdk-evidence.test.ts` > seals a normal producer-certified
reviewed-reference SDK: the same error.
- `source-sdk-public-inputs.test.mjs` > the committed SDK public source
pins match this checkout's committed tree: `pnpm-lock.yaml` is stale in
`source-sdk-public-inputs.json`.

A Dependabot pull request may not change `report-lock.json`,
`report-lock.patch` or the SDK pins. `lockfile-trust.mjs` rejects that
change, and `refresh-lockfile.yml` renews those files after the merge.
So the pull request can never pass.

**Expected behavior**

A verified Dependabot lockfile passes the Runner support suite, as it
already passes the server's reporting-lock policy test. Other authors
keep every check.

**Steps to reproduce**

1. Open a Dependabot pull request that changes `pnpm-lock.yaml`, such as
#17172.
2. Let CI run. `ci / Runner support suite` fails in the four files
above.

## What Changed

- `.github/workflows/pr-trusted.yml`: the `verify_runner_e2e_support`
job sets `PAPERCLIP_CI_PR_AUTHOR` from the pull request author, like the
general test job.
- `tests/runner-e2e/prepare-report-lock.test.ts`,
`source-sdk-evidence.test.ts`, `source-sdk-public-inputs.test.mjs`: the
test cases that need a reviewed committed lock skip when
`PAPERCLIP_CI_PR_AUTHOR` is `dependabot[bot]`. This uses the same check
as `server/scripts/reporting-lock-policy.test.mjs`. The other cases in
these files still run.
- `tests/runner-e2e/vitest.config.ts`: on a Dependabot pull request, the
suite excludes `source-sdk-public-type.test.ts`. Its shared `beforeAll`
prepares the reviewed SDK reference, so every case in the file needs a
reviewed lock.
- `tests/runner-e2e/source-sdk-public-type.ts`: the installed
TypeScript, Babel parser and Zod package hash checks move into an
exported `verifySdkProofPackages`, which `verifySdkPublicSource` calls.
The lockfile refresh renews the lock and source pins after a merge, but
keeps these package hashes.
- `tests/runner-e2e/source-sdk-proof-packages.test.ts` (new): runs
`verifySdkProofPackages` against the installed packages on every pull
request, including Dependabot ones, and checks that a changed package
pin is rejected. So a Dependabot update of one of those packages still
fails until its pins are reviewed.

## Verification

- Red, then green, with a real Dependabot lock: I committed the
`pnpm-lock.yaml` and `package.json` changes of #17172 on top of this
branch in a local commit (not pushed). Then I ran `node
node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts` on the four files:
- Without `PAPERCLIP_CI_PR_AUTHOR`: 4 files failed and the same 6 tests
failed as in CI.
- With `PAPERCLIP_CI_PR_AUTHOR=dependabot[bot]`: 3 files passed, 33
tests passed and 6 skipped. `source-sdk-public-type.test.ts` was
excluded.
- On this branch with master's lock: `pnpm test:e2e:runner:unit` passed
(Vitest: 687 files, 11640 tests; node tests passed).
- `node --test .github/scripts/tests/*.test.mjs`: 808 passed.
- After the package check change: the five affected files
(`source-sdk-proof-packages`, `source-sdk-public-type`,
`source-sdk-evidence`, `source-sdk-public-inputs`,
`prepare-report-lock`) pass, 71 tests. `tsc --noEmit -p
tests/runner-e2e/tsconfig.json` passes.
- `pr.yml` calls `pr-trusted.yml@master`, so this pull request's own CI
does not use the new job variable. The first Dependabot pull request
after the merge checks it.

## Risks

- A Dependabot update of TypeScript, the Babel parser or Zod still fails
the suite until a maintainer reviews the new package pins. That is
deliberate.
- On a Dependabot pull request, the suite no longer checks the committed
lock against the reviewed proof. The policy job still requires that the
lock comes from a verified Dependabot commit (`lockfile-trust.mjs pr`),
and the lockfile refresh still admits the lock and renews the proof and
the SDK pins after the merge. That is the same trust as #19356.
- `PAPERCLIP_CI_PR_AUTHOR` is the pull request author from the GitHub
event. A contributor cannot set it to `dependabot[bot]`. Other commits
pushed to a Dependabot branch are still checked by the policy job.
- No runtime code changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-16 18:03:41 -08:00

170 lines
15 KiB
TypeScript

import defaultInputs from "./source-sdk-public-inputs.json" with { type: "json" };
import { createHash } from "node:crypto";
import { createRequire } from "node:module";
import { lstat, readFile, readdir, realpath } from "node:fs/promises";
import path from "node:path";
import { findSecretLeakMatch, secretShapeObservations } from "./redaction.js";
const hash = (v: Buffer | string) => createHash("sha256").update(v).digest("hex");
export interface SdkPublicInputs {
inputs: Record<string, string>; lockSha256: string; compilerPackageSha256: string;
parserPackageSha256: string; parserEntrySha256: string; zodPackageSha256: string; zodEntrySha256: string; zodTypeFiles: Record<string, string>;
}
interface PublicSource {
root: string; parser: { parse(text: string, options: unknown): any }; constructors: Record<string, unknown>; factories: Record<string, unknown>;
inputs: SdkPublicInputs; identity: { inputs: number; manifestSha256: string; lockSha256: string; compilerPackageSha256: string; parserPackageSha256: string; parserEntrySha256: string; zodPackageSha256: string; zodEntrySha256: string; zodTypeGraphSha256: string };
}
const verified = new WeakSet<PublicSource>();
const matches = async (file: string, expected: string) => { if (hash(await readFile(file)) !== expected) throw new Error("SDK public source proof unavailable"); };
export async function verifySdkProofPackages(root: string, inputs: SdkPublicInputs = defaultInputs) {
await matches(path.join(root, "node_modules/typescript/package.json"), inputs.compilerPackageSha256);
const parserRoot = path.join(root, "node_modules/.pnpm/@babel+parser@7.29.8/node_modules/@babel/parser");
await matches(path.join(parserRoot, "package.json"), inputs.parserPackageSha256); await matches(path.join(parserRoot, "lib/index.js"), inputs.parserEntrySha256);
const require = createRequire(path.join(root, "packages/shared/package.json")), zodPackage = require.resolve("zod/package.json"), zodEntry = require.resolve("zod");
await matches(zodPackage, inputs.zodPackageSha256); await matches(zodEntry, inputs.zodEntrySha256);
for (const [relative, expected] of Object.entries(inputs.zodTypeFiles)) {
if (!/^[A-Za-z0-9._/-]+$/.test(relative) || relative.split("/").includes("..")) throw new Error("SDK public source proof unavailable");
await matches(path.join(path.dirname(zodPackage), relative), expected);
}
return { parserRoot, require, zodPackage };
}
export async function verifySdkPublicSource(root: string, expectations?: SdkPublicInputs): Promise<PublicSource> {
const supplied = expectations ?? defaultInputs;
const inputs: SdkPublicInputs = Object.freeze({ ...supplied, inputs: Object.freeze({ ...supplied.inputs }), zodTypeFiles: Object.freeze({ ...supplied.zodTypeFiles }) });
if (await realpath(root) !== root) throw new Error("SDK public source proof unavailable");
for (const [file, expected] of Object.entries(inputs.inputs)) {
if (!/^[A-Za-z0-9._/-]+$/.test(file) || file.split("/").includes("..")) throw new Error("SDK public source proof unavailable");
const filename = path.join(root, file), stat = await lstat(filename);
const bytes = await readFile(filename);
// The authenticated restored lock is the sole overlay on frozen Git inputs.
if (!stat.isFile() || stat.isSymbolicLink() || await realpath(filename) !== filename
|| (file === "pnpm-lock.yaml" ? hash(bytes) !== inputs.lockSha256 : createHash("sha1").update(`blob ${bytes.length}\0`).update(bytes).digest("hex") !== expected)) throw new Error("SDK public source proof unavailable");
}
const actualSources: string[] = [];
const collect = async (directory: string) => {
for (const entry of await readdir(path.join(root, directory), { withFileTypes: true })) {
const file = `${directory}/${entry.name}`;
if (entry.isDirectory()) await collect(file);
else if (entry.isFile()) actualSources.push(file);
else throw new Error("SDK public source proof unavailable");
}
};
for (const directory of ["packages/shared/src", "packages/plugins/sdk/src"]) await collect(directory);
const expectedSources = Object.keys(inputs.inputs).filter(file => file.startsWith("packages/shared/src/") || file.startsWith("packages/plugins/sdk/src/"));
if (JSON.stringify(actualSources.sort()) !== JSON.stringify(expectedSources.sort())) throw new Error("SDK public source proof unavailable");
await matches(path.join(root, "pnpm-lock.yaml"), inputs.lockSha256);
const { parserRoot, require, zodPackage } = await verifySdkProofPackages(root, inputs);
const parser = createRequire(path.join(parserRoot, "package.json"))("./lib/index.js");
const publicTypes = new Map<string, any>();
for (const [relative, expected] of Object.entries(inputs.zodTypeFiles)) {
if (!/^[A-Za-z0-9._/-]+$/.test(relative) || relative.split("/").includes("..")) throw new Error("SDK public source proof unavailable");
const file = path.join(path.dirname(zodPackage), relative); await matches(file, expected);
const ast = parser.parse(await readFile(file, "utf8"), { sourceType: "module", plugins: [["typescript", { dts: true }]], errorRecovery: true });
if (ast.errors.length) throw new Error("SDK public source proof unavailable"); publicTypes.set(relative, ast);
}
const resolves = (parent: string, specifier: unknown, child: string) => typeof specifier === "string" && path.posix.normalize(path.posix.join(path.posix.dirname(parent), specifier)).replace(/\.(?:cjs|js)$/, ".d.cts") === child;
const entry = publicTypes.get("index.d.cts"), external = publicTypes.get("v4/classic/external.d.cts"), schemas = publicTypes.get("v4/classic/schemas.d.cts");
if (!entry || !external || !schemas || !entry.program.body.some((n: any) => n.type === "ExportAllDeclaration" && resolves("index.d.cts", n.source?.value, "v4/classic/external.d.cts"))
|| !external.program.body.some((n: any) => n.type === "ExportAllDeclaration" && resolves("v4/classic/external.d.cts", n.source?.value, "v4/classic/schemas.d.cts"))) throw new Error("SDK public source proof unavailable");
const namespaces = entry.program.body.filter((n: any) => n.type === "ImportDeclaration" && resolves("index.d.cts", n.source?.value, "v4/classic/external.d.cts")).flatMap((n: any) => n.specifiers.filter((s: any) => s.type === "ImportNamespaceSpecifier").map((s: any) => s.local.name));
if (!entry.program.body.some((n: any) => n.type === "ExportNamedDeclaration" && n.specifiers?.some((s: any) => s.exported?.name === "z" && namespaces.includes(s.local?.name)))) throw new Error("SDK public source proof unavailable");
const declaredTypes = new Set(schemas.program.body.filter((n: any) => n.type === "ExportNamedDeclaration" && ["TSInterfaceDeclaration", "TSTypeAliasDeclaration", "ClassDeclaration"].includes(n.declaration?.type)).map((n: any) => n.declaration.id.name));
const declaredFactories = new Set(schemas.program.body.filter((n: any) => n.type === "ExportNamedDeclaration" && ["TSDeclareFunction", "FunctionDeclaration"].includes(n.declaration?.type)).map((n: any) => n.declaration.id.name));
const zod = require("zod"), namespace = zod.z ?? zod;
const constructors = Object.freeze(Object.fromEntries(Object.entries(namespace).filter(([name, value]) => declaredTypes.has(name) && typeof value === "function")));
const factories = Object.freeze(Object.fromEntries(Object.entries(namespace).filter(([name, value]) => declaredFactories.has(name) && typeof value === "function")));
const source = { root, parser, constructors, factories, inputs, identity: { inputs: Object.keys(inputs.inputs).length, manifestSha256: hash(JSON.stringify(inputs)),
lockSha256: inputs.lockSha256, compilerPackageSha256: inputs.compilerPackageSha256, parserPackageSha256: inputs.parserPackageSha256,
parserEntrySha256: inputs.parserEntrySha256, zodPackageSha256: inputs.zodPackageSha256, zodEntrySha256: inputs.zodEntrySha256, zodTypeGraphSha256: hash(JSON.stringify(inputs.zodTypeFiles)) } };
Object.freeze(source.identity); Object.freeze(source); verified.add(source); return source;
}
function nodes(node: any, fn: (node: any) => void) {
if (!node || typeof node !== "object") return;
if (Array.isArray(node)) { for (const child of node) nodes(child, fn); return; }
if (typeof node.type !== "string") return;
fn(node);
for (const [key, value] of Object.entries(node)) if (!["loc", "extra", "errors", "tokens", "comments", "leadingComments", "trailingComments", "innerComments"].includes(key)) nodes(value, fn);
}
function aliases(ast: any): string[] {
const found: string[] = [];
for (const row of ast.program.body) if (row.type === "ImportDeclaration" && row.source.value === "zod") {
for (const s of row.specifiers) if (s.type === "ImportNamespaceSpecifier" || s.type === "ImportSpecifier" && s.imported.name === "z") found.push(s.local.name);
}
if (new Set(found).size !== found.length) return [];
let shadowed = false;
nodes(ast, n => {
if (n.type === "TSModuleDeclaration") shadowed = true;
if (["VariableDeclarator", "FunctionDeclaration", "ClassDeclaration", "TSModuleDeclaration", "TSTypeAliasDeclaration", "TSInterfaceDeclaration", "TSEnumDeclaration"].includes(n.type) && found.includes(n.id?.name)) shadowed = true;
if (n.type === "TSTypeParameter" && found.includes(typeof n.name === "string" ? n.name : n.name?.name)) shadowed = true;
if (Array.isArray(n.params) && n.params.some((p: any) => p.type === "Identifier" && found.includes(p.name))) shadowed = true;
});
return shadowed ? [] : found;
}
function parsed(source: PublicSource, text: string, declaration: boolean) {
const ast = source.parser.parse(text, { sourceType: "module", plugins: [["typescript", { dts: declaration }]], errorRecovery: true });
if (ast.errors.length) throw new Error("SDK public source parse unavailable"); return ast;
}
const primitive = (node: any) => node?.type === "NumericLiteral" && Number.isFinite(node.value) || ["BooleanLiteral", "NullLiteral"].includes(node?.type) || node?.type === "StringLiteral" && node.value === "";
function publicSourceProperty(source: PublicSource, ast: any, key: string) {
const publicAliases = aliases(ast), properties: any[] = [];
nodes(ast, n => { if (n.type === "ObjectProperty" && !n.computed && (n.key?.name ?? n.key?.value) === key) properties.push(n); });
if (properties.length !== 1) return false;
let value = properties[0].value, factory: string | undefined;
while (value?.type === "CallExpression" && value.callee?.type === "MemberExpression" && !value.callee.computed && value.callee.property?.type === "Identifier") {
if (!value.arguments.every(primitive)) return false;
factory = value.callee.property.name; value = value.callee.object;
}
return value?.type === "Identifier" && publicAliases.includes(value.name) && !!factory && Object.hasOwn(source.factories, factory) && typeof source.factories[factory] === "function";
}
export async function sdkPublicTypeSyntax(source: PublicSource, file: string, text: string, span: { start: number; end: number }) {
if (!verified.has(source) || Buffer.byteLength(text) > 8 * 1024 * 1024) return false;
const original = file.replace("/dist/", "/src/").replace(/\.d\.ts$/, ".ts");
if (original === file || !source.inputs.inputs[original]) return false;
const filename = path.join(source.root, original), sourceText = await readFile(filename, "utf8");
if (createHash("sha1").update(`blob ${Buffer.byteLength(sourceText)}\0`).update(sourceText).digest("hex") !== source.inputs.inputs[original]) return false;
try {
const ast = parsed(source, text, true), publicAliases = aliases(ast), candidates: any[] = [];
nodes(ast, n => { if (n.type === "TSPropertySignature" && n.start <= span.start && n.end >= span.end) candidates.push(n); });
candidates.sort((a, b) => (a.end - a.start) - (b.end - b.start)); const property = candidates[0], type = property?.typeAnnotation?.typeAnnotation;
if (!property || property.computed || property.key?.type !== "Identifier" || property.initializer || property.value || type?.type !== "TSTypeReference"
|| span.start < property.key.start || span.start >= property.key.end || type.typeName?.type !== "TSQualifiedName" || span.end !== type.typeName.end
|| !/^\s*:\s*$/.test(text.slice(property.key.end, type.typeName.start))
|| (ast.comments ?? []).some((c: any) => c.start < property.end && c.end > property.start)) return false;
let bound = true, references = 0;
nodes(property.typeAnnotation, n => {
if (!["TSTypeAnnotation", "TSTypeReference", "TSQualifiedName", "TSTypeParameterInstantiation", "Identifier"].includes(n.type)) bound = false;
if (n.type === "TSTypeReference") {
references++; const name = n.typeName;
if (name?.type !== "TSQualifiedName" || name.left?.type !== "Identifier" || name.right?.type !== "Identifier" || !publicAliases.includes(name.left.name)
|| !Object.hasOwn(source.constructors, name.right.name) || typeof source.constructors[name.right.name] !== "function") bound = false;
}
});
return bound && references > 0 && publicSourceProperty(source, parsed(source, sourceText, false), property.key.name);
} catch { return false; }
}
export interface SdkScanObservation { pathSha256: string; fileSha256: string; detector: number; start: number; end: number; classification: "private_exact" | "direct_token" | "unresolved_context" | "public_schema_type" }
export function createSdkArtifactScanner(repositoryRoot: string, secrets: readonly string[], expectations?: SdkPublicInputs) {
const observations: SdkScanObservation[] = []; let bounded = true; let proof: Promise<PublicSource | null> | undefined;
const source = () => proof ??= verifySdkPublicSource(repositoryRoot, expectations).catch(() => null);
return {
async scan(file: string, bytes: Buffer) {
const base = { pathSha256: hash(file), fileSha256: hash(bytes) };
if (findSecretLeakMatch(bytes, secrets, { includeShapes: false })) { observations.push({ ...base, detector: -1, start: 0, end: bytes.length, classification: "private_exact" }); return false; }
const raw = secretShapeObservations(bytes.toString("utf8")); if (!raw.bounded) { bounded = false; for (const span of raw.matches) observations.push({ ...base, ...span, classification: span.detector === 3 ? "unresolved_context" : "direct_token" }); return false; }
if (raw.matches.some(span => span.detector !== 3)) {
for (const span of raw.matches) observations.push({ ...base, ...span, classification: span.detector === 3 ? "unresolved_context" : "direct_token" });
return false;
}
let safe = true;
for (const span of raw.matches) {
const publicSource = span.detector === 3 ? await source() : null;
const syntax = !!publicSource && await sdkPublicTypeSyntax(publicSource, file, bytes.toString("utf8"), span);
const classification = span.detector !== 3 ? "direct_token" : syntax ? "public_schema_type" : "unresolved_context";
observations.push({ ...base, ...span, classification }); if (!syntax) safe = false;
}
return safe;
},
async audit() { const p = proof ? await proof : null; return { schema: "paperclip.source-sdk-scan/v1", suppressedFindings: 0, bounded, publicSource: p?.identity ?? null, observations: structuredClone(observations).sort((a, b) => a.pathSha256.localeCompare(b.pathSha256) || a.detector - b.detector || a.start - b.start) }; },
};
}