mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-08 21:03:51 +02:00
Dotta
5458940a6e
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner needs repeatable evaluation contracts. > - Evaluation code must stay separate from provider launch and production orchestration. > - Offline fixtures need stable compatibility, scoring, traceability, and report rules. > - Published Runner consumers need only the supported evaluation contract surface. > - This pull request adds offline evaluation tooling and a workspace-private matrix kernel. > - The benefit is deterministic evaluation without credentials or paid provider calls. ## Linked Issues or Issue Description Refs #11297 This pull request extracts the offline evaluation unit from the earlier aggregate Runner work. ## What Changed - Add a workspace-private, provider-neutral evaluation matrix kernel. - Add the public `@paperclipai/paperclip-runner/evals` compatibility and native execution contracts. - Add fail-closed runnerd artifact and protocol compatibility checks. - Add deterministic workflow catalogs, scoring, traceability, and report generation. - Add sanitized Codex, OpenCode, and ACPX fixtures. - Add package-boundary and clean-consumer checks. - Add the eval package manifest to the Docker dependency stage. - Add the generated protocol fixture digest without changing the lockfile. ## Verification GitHub Actions must run: - Runner TypeScript and Rust type checks. - Runner unit and protocol tests. - Evaluation kernel tests. - Workflow traceability checks. - Clean-consumer and package-boundary checks. - Repository test, type-check, build, policy, and security gates. No local test command was run. The repository owner requested GitHub-only verification. ## Risks This is a large greenfield review surface with 51 files. The code does not launch a live provider or load credentials. Package and protocol drift fail closed. The workspace lockfile remains under the existing CI-owned process. ## Model Used OpenAI Codex with the GPT-5 agent model. The work used high reasoning, repository inspection, tool use, and parallel code review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
60 lines
2.6 KiB
JavaScript
60 lines
2.6 KiB
JavaScript
import { readFile } from "node:fs/promises";
|
|
import { dirname, resolve } from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
|
|
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
|
const evalKernelRoot = resolve(packageRoot, "../paperclip-eval-kernel");
|
|
const runnerManifest = JSON.parse(await readFile(resolve(packageRoot, "package.json"), "utf8"));
|
|
const evalKernelManifest = JSON.parse(await readFile(resolve(evalKernelRoot, "package.json"), "utf8"));
|
|
const runtimeIndex = await readFile(resolve(packageRoot, "src/index.ts"), "utf8");
|
|
|
|
const violations = [];
|
|
if (runnerManifest.exports?.["./testing"] === undefined) {
|
|
violations.push("runner package must declare the ./testing export");
|
|
}
|
|
if (runnerManifest.exports?.["./evals"] === undefined) {
|
|
violations.push("runner package must declare the ./evals integration export");
|
|
}
|
|
if (evalKernelManifest.private !== true) {
|
|
violations.push("generic eval kernel must remain a workspace-private development package");
|
|
}
|
|
for (const privatePath of ["mock-core", "conformance", "scenarios", "src/tools", "./tools/"]) {
|
|
if (runtimeIndex.includes(privatePath)) {
|
|
violations.push(`runtime root must not export test/eval path: ${privatePath}`);
|
|
}
|
|
}
|
|
|
|
const runtimeDependencyGroups = [
|
|
runnerManifest.dependencies ?? {},
|
|
runnerManifest.optionalDependencies ?? {},
|
|
runnerManifest.peerDependencies ?? {},
|
|
];
|
|
for (const dependency of runtimeDependencyGroups.flatMap((group) => Object.keys(group))) {
|
|
if (dependency.includes("eval")) {
|
|
violations.push(`runner runtime must not depend on eval package: ${dependency}`);
|
|
}
|
|
}
|
|
if (runnerManifest.dependencies?.ajv === undefined || runnerManifest.devDependencies?.ajv !== undefined) {
|
|
violations.push("ajv must be declared as a runtime dependency because the public dispatcher imports it");
|
|
}
|
|
|
|
for (const dependency of Object.keys({
|
|
...(evalKernelManifest.dependencies ?? {}),
|
|
...(evalKernelManifest.optionalDependencies ?? {}),
|
|
...(evalKernelManifest.peerDependencies ?? {}),
|
|
})) {
|
|
if (dependency === "@paperclipai/paperclip-runner") {
|
|
violations.push("generic eval kernel must use structural callbacks, not a runner runtime dependency");
|
|
}
|
|
}
|
|
if (evalKernelManifest.files?.some((entry) => /scenario|fixture|corpus/.test(entry))) {
|
|
violations.push("generic eval kernel package inventory must not include scenario content");
|
|
}
|
|
|
|
if (violations.length > 0) {
|
|
process.stderr.write(`Package boundary check failed:\n${violations.map((item) => `- ${item}`).join("\n")}\n`);
|
|
process.exitCode = 1;
|
|
} else {
|
|
process.stdout.write("Package boundary check passed: runtime, testing, and eval ownership is acyclic.\n");
|
|
}
|