mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 23:43:16 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - PR CI selects affected tests before allocating parallel runners. > - Recent audits retained about 96% of test files despite narrow changes. > - Unchanged consumer packages kept full coverage, and source scans confused data fields and message text with runtime loaders. > - File reads need explicit edges that imports alone cannot provide. > - This change follows consumer graphs and known file inputs while retaining unknown effects. > - The AWS CODEOWNERS gate stays in front of runner allocation. ## Linked Issues or Issue Description Refs #18376 and #18362. Narrow source changes still selected most tests because consumer packages kept full coverage and known file reads were treated as arbitrary input. Expected behavior is to retain affected import/file consumers and unknown inputs, then partition only the selected files. The shared storage failure found during the audit was fixed upstream in #18357. The candidate merge ran all 18 managed-skill storage tests successfully in [Linux qualification](https://github.com/paperclipai/paperclip/actions/runs/41140209697/job/124352365547). This branch includes that upstream commit; no additional storage implementation or permanent diagnostic job is included. ## What Changed - Assess every Vitest project in the coarse plan, including unchanged consumer workspaces. - Add dependency edges for statically resolved module-relative filesystem reads and directory listings. Keep unknown paths, writes, escaped APIs, mutable path objects and untracked inputs conservative. - Distinguish data keys, erased types and message strings from runtime globals, loaders and mock bypasses. Retain destructured globals, aliased object reads and mocked filesystem/path resolution. - Normalize `fs` and `node:fs` before recording coverage so a known read cannot hide an unknown one. - Review the Drizzle SQL/schema entry points by hashes of their complete import closures. CommonJS/dynamic loaders reject the fingerprint; drivers and migration readers remain opaque, and dependency updates revoke the review. - Record file input edges and per-test retention reasons in selection audit artifacts. - Add cross-workspace, mixed-import, symlink retargeting, setup-input and dependency-review regression cases. Object-valued URL constants are explicitly opaque with and without mutation. Update the developer guide. - Preserve the AWS CODEOWNERS authorization gate, runner-group restrictions and existing lane ownership. ## Verification - Final head `eedad362e84d6b68f0f048c92db6032e45622c46` passed [complete Actions CI](https://github.com/paperclipai/paperclip/actions/runs/41144233520) on attempt 1. Overall status: 57 successful checks, three intended skips, and one neutral Paperclip review. Both Greptile and Paperclip rated this head 5/5; zero unresolved review threads. - All 17 local selector tests passed: `node --test scripts/ci/test-selection*.test.mjs`. Final [Linux qualification](https://github.com/paperclipai/paperclip/actions/runs/41144233203/job/124365428832) passed all 18 selector/impact tests without skips, plus real workspace selection and cache/routing checks. - Same-tree local comparison against the previous selector at `d331d8481a`: Inbox changes selected 466 rather than 700 files (33% fewer); auth middleware changes selected 2,141 rather than 2,788 files (23% fewer). The sampled plans retained the same 2 and 24 jobs under existing weighted budgets. Final Linux qualification selected 466 files for Inbox and 1,949 for auth/config-test changes; platform-dependent graphs differ, so these are not a Linux before/after comparison. Chat/native Runner lanes retain their existing ownership and are excluded from these counts. - Local `pnpm -r typecheck`, `pnpm build` and `git diff --check` passed. The complete local `pnpm test:run` was attempted and stopped after repeated embedded PostgreSQL startup failures; initdb reported exhausted macOS semaphore capacity. Full Linux CI passed on the final combined tree. - The storage regression was corrected upstream in #18357. [Focused Linux verification](https://github.com/paperclipai/paperclip/actions/runs/41140209697/job/124352365547) ran all 18 managed-skill tests successfully, with no skips; the final full CI run also passed its server lane. ## Risks - Unknown effects must retain their consumers. Regression cases cover mixed builtin spellings, dynamic paths, escaped/mocked APIs, global aliases, reviewed dependency changes and cross-package file reads. - Graph analysis now visits unchanged consumers. Its added work must be included in rollout latency measurements; fewer test files do not by themselves prove shorter end-to-end PR time. - The weighted shard budget deliberately remains unchanged. These samples reduce file execution without reducing allocated jobs. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code execution and GitHub tools. The exact serving variant and context window are not exposed in this session. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted selector suite; full local suite limitation documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
148 lines
7.8 KiB
JavaScript
148 lines
7.8 KiB
JavaScript
import path from "node:path";
|
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
|
|
const filesystem = /^(?:node:)?fs(?:\/promises)?$/;
|
|
const readers = new Set(["readFile", "readFileSync", "existsSync", "stat", "statSync", "lstat", "lstatSync", "readdir", "readdirSync"]);
|
|
|
|
export function repositoryFileInputs(file, program) {
|
|
if (!program.body.some(node => node.type === "ImportDeclaration" && filesystem.test(node.source.value))) {
|
|
return { covered: [], inputs: [] };
|
|
}
|
|
const parents = new Map(), nodes = [], declarations = new Map(), constants = new Map(), imports = new Map();
|
|
const bind = (node) => {
|
|
if (!node) return;
|
|
if (node.type === "Identifier") declarations.set(node.name, (declarations.get(node.name) ?? 0) + 1);
|
|
else if (node.type === "ObjectPattern") node.properties.forEach(p => bind(p.value ?? p.argument));
|
|
else if (node.type === "ArrayPattern") node.elements.forEach(bind);
|
|
else if (node.type === "AssignmentPattern") bind(node.left);
|
|
else if (node.type === "RestElement") bind(node.argument);
|
|
};
|
|
function visit(node, parent) {
|
|
if (!node || typeof node !== "object" || !node.type) return;
|
|
parents.set(node, parent); nodes.push(node);
|
|
if (node.type === "ImportDeclaration") {
|
|
for (const spec of node.specifiers) {
|
|
bind(spec.local);
|
|
const source = filesystem.test(node.source.value) ? node.source.value.replace(/^node:/, "") : node.source.value;
|
|
imports.set(spec.local.name, { source, name: spec.type === "ImportSpecifier" ? spec.imported.name : "*" });
|
|
}
|
|
}
|
|
if (node.type === "VariableDeclarator") {
|
|
bind(node.id);
|
|
if (parent.kind === "const" && node.id.type === "Identifier") constants.set(node.id.name, node.init);
|
|
}
|
|
if (/Function/.test(node.type) || node.type === "ArrowFunctionExpression") {
|
|
bind(node.id); node.params?.forEach(bind);
|
|
}
|
|
if (node.type === "ClassDeclaration" || node.type === "ClassExpression") bind(node.id);
|
|
if (node.type === "CatchClause") bind(node.param);
|
|
if (node.type === "AssignmentExpression") bind(node.left);
|
|
if (node.type === "UpdateExpression") bind(node.argument);
|
|
for (const value of Object.values(node)) {
|
|
if (Array.isArray(value)) value.forEach(child => visit(child, node));
|
|
else if (value && typeof value === "object" && value.type) visit(value, node);
|
|
}
|
|
}
|
|
visit(program);
|
|
const tainted = new Set();
|
|
for (const node of nodes) {
|
|
if (node.type === "MetaProperty" && node.meta.name === "import" && node.property.name === "meta") {
|
|
const member = parents.get(node), use = parents.get(member);
|
|
if (member?.type !== "MemberExpression" || member.computed || !["url", "dirname", "filename"].includes(member.property.name)
|
|
|| (use?.type === "AssignmentExpression" && use.left === member)
|
|
|| use?.type === "UpdateExpression" || (use?.type === "UnaryExpression" && use.operator === "delete")) tainted.add("import.meta");
|
|
}
|
|
if (node.type !== "Identifier") continue;
|
|
const parent = parents.get(node);
|
|
const binding = imports.get(node.name);
|
|
if (binding?.name === "*" && /^(?:node:)?(?:path|url)$/.test(binding.source)
|
|
&& !parent?.type.startsWith("Import")) {
|
|
if (parent?.type !== "MemberExpression" || parent.object !== node
|
|
|| parents.get(parent)?.type !== "CallExpression" || parents.get(parent).callee !== parent) tainted.add(node.name);
|
|
}
|
|
if (node.name === "URL" && !(parent?.type === "NewExpression" && parent.callee === node)) tainted.add("URL");
|
|
}
|
|
const unique = (name) => declarations.get(name) === 1;
|
|
const member = (node, name) => node?.type === "MemberExpression" && !node.computed && node.property.name === name;
|
|
const importMeta = node => node?.type === "MetaProperty" && node.meta.name === "import" && node.property.name === "meta";
|
|
function imported(node, source, name) {
|
|
if (node?.type === "Identifier" && unique(node.name) && !tainted.has(node.name)) {
|
|
const entry = imports.get(node.name);
|
|
return entry && source.test(entry.source) && entry.name === name;
|
|
}
|
|
if (member(node, name) && node.object.type === "Identifier" && unique(node.object.name) && !tainted.has(node.object.name)) {
|
|
const entry = imports.get(node.object.name);
|
|
return entry && source.test(entry.source) && entry.name === "*";
|
|
}
|
|
return false;
|
|
}
|
|
function evaluate(node, seen = new Set()) {
|
|
if (!node) return undefined;
|
|
if (node.type === "Literal" && typeof node.value === "string") return node.value;
|
|
if (node.type === "TemplateLiteral" && !node.expressions.length) return node.quasis[0].value.cooked;
|
|
if (node.type === "Identifier" && unique(node.name) && constants.has(node.name) && !seen.has(node.name)) {
|
|
const value = evaluate(constants.get(node.name), new Set([...seen, node.name]));
|
|
return typeof value === "string" ? value : undefined;
|
|
}
|
|
if (node.type === "Identifier" && !declarations.has(node.name)) {
|
|
if (node.name === "__dirname") return path.dirname(file);
|
|
if (node.name === "__filename") return file;
|
|
}
|
|
if (node.type === "MemberExpression" && importMeta(node.object) && !node.computed && !tainted.has("import.meta")) {
|
|
if (node.property.name === "url") return pathToFileURL(file).href;
|
|
if (node.property.name === "dirname") return path.dirname(file);
|
|
if (node.property.name === "filename") return file;
|
|
}
|
|
if (node.type === "NewExpression" && node.callee.name === "URL" && !declarations.has("URL") && !tainted.has("URL") && node.arguments.length === 2) {
|
|
const args = node.arguments.map(arg => evaluate(arg, seen));
|
|
if (args.every(arg => typeof arg === "string")) {
|
|
try { return new URL(...args); } catch { return undefined; }
|
|
}
|
|
}
|
|
if (node.type === "CallExpression") {
|
|
const args = node.arguments.map(arg => evaluate(arg, seen));
|
|
if (imported(node.callee, /^(?:node:)?url$/, "fileURLToPath") && args.length === 1) {
|
|
try { return fileURLToPath(args[0]); } catch { return undefined; }
|
|
}
|
|
if (args.every(arg => typeof arg === "string")) {
|
|
for (const method of ["resolve", "join", "dirname"]) {
|
|
if (!imported(node.callee, /^(?:node:)?path$/, method)) continue;
|
|
if (!args.length || !path.isAbsolute(args[0]) || (method === "dirname" && args.length !== 1)) return undefined;
|
|
return path[method](...args);
|
|
}
|
|
}
|
|
}
|
|
return undefined;
|
|
}
|
|
const covered = new Set(), invalid = new Set(), inputs = new Map();
|
|
for (const node of nodes) {
|
|
if (node.type !== "Identifier" || !filesystem.test(imports.get(node.name)?.source ?? "")) continue;
|
|
const binding = imports.get(node.name), parent = parents.get(node);
|
|
if (parent?.type.startsWith("Import")) continue;
|
|
let target = node, method = binding.name;
|
|
if (!unique(node.name)) { invalid.add(binding.source); continue; }
|
|
if (method === "*" || method === "promises") {
|
|
target = parent;
|
|
if (method === "*" && member(target, "promises")) target = parents.get(target);
|
|
if (target?.type !== "MemberExpression" || target.computed) { invalid.add(binding.source); continue; }
|
|
method = target.property.name;
|
|
}
|
|
const call = parents.get(target);
|
|
if (!readers.has(method) || call?.type !== "CallExpression" || call.callee !== target) {
|
|
invalid.add(binding.source); continue;
|
|
}
|
|
let input = evaluate(call.arguments[0]);
|
|
if (input instanceof URL) {
|
|
try { input = fileURLToPath(input); } catch { input = undefined; }
|
|
}
|
|
if (typeof input !== "string" || !path.isAbsolute(input)) { invalid.add(binding.source); continue; }
|
|
covered.add(binding.source);
|
|
const values = inputs.get(binding.source) ?? new Set();
|
|
values.add(path.resolve(input)); inputs.set(binding.source, values);
|
|
}
|
|
return {
|
|
covered: [...covered].filter(source => !invalid.has(source)),
|
|
inputs: [...new Set([...inputs.values()].flatMap(values => [...values]))],
|
|
};
|
|
}
|