mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 11:23:40 +01:00
fix(evals): report controlled protocol cases from safe records (#19070)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner protocol qualification records real provider behavior in TestOps. > - Controlled attempts keep raw records and accounting in private storage. > - A live case passed, but a later public-render command rejected its controlled context. > - The missing public report caused the final result to be stored as broken. > - This pull request creates the closed TestOps result from existing sanitized records inside the private report job. > - The benefit is correct result reporting while the privacy guard stays effective. ## Linked Issues or Issue Description Refs #18985 and #19011. The related-PR search found no separate repair for this failure. **What happened?** [The live canary](https://github.com/paperclipai/paperclip/actions/runs/41916513378) passed all four provider checks. Its controller verified cleanup and closed its request accounting. The private canonical report recorded one passed case. The next command used `--public-viewer` while the controlled manifest remained set. The eval program rejected it with `controlled campaign evidence must remain private`. The closed exporter then had no public payload and correctly stored a broken result. **Expected behavior** Report the original case grade from sanitized structured records. Keep raw controlled records private. Keep the original checks, source bindings, and missing-evidence rules. **Steps to reproduce** Run one controlled protocol case with TestOps reporting enabled. Let the private canonical report finish. The following public-render step fails before the closed result can read the grade. **Paperclip version or commit** Observed at `4740d251b7032e714d33039ba66a56948238c38f`. This repair is based on `c6cb7e2a2d42743c0c3c4ead2cbad4a299f9a2e7`. **Deployment mode** GitHub-hosted Linux, controlled ACPX Claude protocol qualification, first workflow attempt, zero automatic retries. ## What Changed - Add a narrow `export-runs` path to the existing TestOps helper. Bind the sanitized score and artifact to the exact attempt, case, model, provider, driver, and public-projection schema. Reuse the existing strict grade and check validation. - Keep sanitization inside the private report job. Use structured records for controlled TestOps reporting. Keep the public viewer for legacy publication. - Add record identity, privacy, grade, command, and workflow routing controls. Wire the existing helper and workflow-security files into the required Node registry. Document the reporting path. ## Verification - 43 focused campaign/sanitizer, TestOps and workflow security tests passed. The command regression writes a raw attempt, calls the production sanitizer, and runs `export-runs` on its output. It also repeats that chain after a failed-check mutation. Both new-control files now run through required `test:typescript:prep`, not only the optional publisher script. - The actual required prep ran locally with provider credentials excluded. The eval-kernel and Rust builds passed. Its Node registry executed the new producer-to-export and security controls by name: 403 passed, 2 failed, 11 existing skips. The two original package-contract failures cannot resolve this isolated clone's Codex ACP and Claude ACP package links. This is not a green local full-prep claim. The first attempt also stopped at a missing Node-types link, before an ignored clone-local link to the locked cached version restored that prerequisite. - 72 release workflow tests passed with the existing locked pnpm 9.15.4 wrapper and local loopback permission. The first sandbox run failed because loopback fixtures could not listen; the default local tool path also failed two install fixtures. No test assertions were changed to resolve these failures. - The actual repaired workflow shell ran locally against the retained canary campaign and sanitized records. It emitted one closed passed result with all four original checks. No provider request or upload occurred. - Missing or failed checks still reject a pass. Private-only markers stay absent from the closed output. Wrong model, provider, driver, attempt, case, and schemas reject. - The unmodified, pinned eval program still rejects a controlled public-viewer command. - Actionlint and `git diff --check` passed. [Full final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41927878886) passed on its first attempt at `0d0c090b1b`. Required verify/e2e gates, all four Runner lanes, typecheck, build, backend and browser shards passed. Package suite: 4951 passed, 10 existing skips across 297 passed files and one existing file skip. - The required static log contains passing lines for both the production sanitizer-to-export command regression and the closed-projection security control. The prior optional-script registration did not run those tests in required CI; the one script-entry fix now enforces them. Provider/runtime and workflow bytes are unchanged by the verification-only follow-ups. - Greptile: fresh 5/5 on `0d0c090b1b`, with no actionable new issues. The separate Paperclip Review requested a producer-to-consumer regression. That regression and its required-CI wiring are complete. The addressed thread is resolved; no open review threads remain. - No fresh deployed reporting pass is claimed. The failed workflow and its stored broken result remain unchanged. ## Risks The exporter reads structured records inside the private job. It copies only the closed result fields and requires the existing sanitized marker and identity checks. The final reporting job still receives only those files and verifies the saved result bindings. The Python privacy guard, provider path, case assertions, budgets, deadlines, and retries are unchanged. ## Model Used OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort. The inherited runtime reports a 475,000-token context window. Capabilities used: reasoning, code changes, and repository, shell, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
06826915d7
commit
85e4d12856
6 files changed
+147
-19
No files matched your search
@@ -1294,12 +1294,16 @@ jobs:
|
||||
fi
|
||||
exit "$status"
|
||||
|
||||
- name: Render the same canonical grid from a public-safe evidence projection
|
||||
- name: Project safe run records for reporting
|
||||
if: needs.authorize.outputs.accounting_mode == 'legacy' || needs.authorize.outputs.testops_enabled == 'true'
|
||||
run: |
|
||||
node packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs sanitize \
|
||||
--runs-root runner-protocol-merged/runs \
|
||||
--output runner-protocol-merged/public-runs
|
||||
|
||||
- name: Render the same canonical grid from a public-safe evidence projection
|
||||
if: needs.authorize.outputs.accounting_mode == 'legacy'
|
||||
run: |
|
||||
python3 .paperclip-evals/evals/paperclip-runner/tools/eval_program.py report \
|
||||
--runs-root runner-protocol-merged/public-runs \
|
||||
--output runner-protocol-merged/public-report \
|
||||
@@ -1312,11 +1316,24 @@ jobs:
|
||||
- name: Create the closed TestOps projection inside the report boundary
|
||||
if: always() && !cancelled() && needs.authorize.outputs.testops_enabled == 'true' && needs.testops_selection.result == 'success'
|
||||
id: testops_projection
|
||||
env:
|
||||
ACCOUNTING_MODE: ${{ needs.authorize.outputs.accounting_mode }}
|
||||
run: |
|
||||
mkdir -m 700 runner-protocol-testops-projection
|
||||
node packages/paperclip-runner/scripts/runner-protocol-testops.mjs export \
|
||||
runner-protocol-testops-selection/runner-protocol-testops-selection.json \
|
||||
runner-protocol-merged/public-report runner-protocol-testops-projection/results
|
||||
case "$ACCOUNTING_MODE" in
|
||||
controlled)
|
||||
node packages/paperclip-runner/scripts/runner-protocol-testops.mjs export-runs \
|
||||
runner-protocol-testops-selection/runner-protocol-testops-selection.json \
|
||||
runner-protocol-merged/campaign.json runner-protocol-merged/public-runs \
|
||||
runner-protocol-testops-projection/results
|
||||
;;
|
||||
legacy)
|
||||
node packages/paperclip-runner/scripts/runner-protocol-testops.mjs export \
|
||||
runner-protocol-testops-selection/runner-protocol-testops-selection.json \
|
||||
runner-protocol-merged/public-report runner-protocol-testops-projection/results
|
||||
;;
|
||||
*) exit 1 ;;
|
||||
esac
|
||||
|
||||
- name: Transfer only the closed TestOps results and receipt
|
||||
if: always() && steps.testops_projection.outcome == 'success'
|
||||
|
||||
@@ -1017,9 +1017,13 @@ A TestOps-triggered job uses its requested selection. Targeted selections retain
|
||||
`full-run` launch tag. Full selections use `full-run` and `test-plan-3`.
|
||||
|
||||
The report job creates a closed projection of case identities, checks, numeric
|
||||
usage and statuses. For controlled mode this happens inside the existing
|
||||
private-history boundary; raw attempts, controller evidence and chat reports
|
||||
remain there. Only the closed Allure results and receipt cross into the final
|
||||
usage and statuses. For controlled mode, `runner-protocol-testops.mjs export-runs`
|
||||
reads sanitized structured score/artifact records inside the existing
|
||||
private-history boundary. It binds the original grade, complete checks,
|
||||
attempt, model, provider and driver before it can emit a pass. Raw attempts,
|
||||
controller evidence and canonical chat reports remain private. Controlled
|
||||
reporting never invokes the public viewer; legacy publication keeps that path.
|
||||
Only the closed Allure results and receipt cross into the final
|
||||
protected reporting job. It checks source and attempt identity, uploads one result
|
||||
per selected case, and verifies the saved case and status bindings. Missing
|
||||
execution/report evidence becomes a broken result. A passing campaign row
|
||||
|
||||
@@ -85,7 +85,7 @@
|
||||
"typecheck:browser": "tsc -p tsconfig.browser.json --noEmit",
|
||||
"test": "pnpm run test:typescript && pnpm run test:rust",
|
||||
"test:typescript": "pnpm run test:typescript:prep && vitest run",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/materialize-pi-distribution.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/acpx-release-attestations.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/materialize-kimi-distribution.test.mjs scripts/provision-kimi-package.test.mjs scripts/kimi-metadata-probe.test.mjs scripts/patch-kimi-runtime.test.mjs test/kimi-installed-runtime.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-provider-node.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/contracts/contract-fingerprints.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/controlled-evals/controller.test.mjs scripts/controlled-evals/portable.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs",
|
||||
"test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/materialize-pi-distribution.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/acpx-release-attestations.test.mjs scripts/provider-pack-runtime-tree.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/materialize-kimi-distribution.test.mjs scripts/provision-kimi-package.test.mjs scripts/kimi-metadata-probe.test.mjs scripts/patch-kimi-runtime.test.mjs test/kimi-installed-runtime.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-runtime-dependency-roots.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-provider-node.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs scripts/contracts/contract-fingerprints.test.mjs scripts/provider-pack-executable-shims.test.mjs scripts/controlled-evals/controller.test.mjs scripts/controlled-evals/portable.test.mjs scripts/runner-protocol-testops.test.mjs scripts/runner-protocol-eval-workflow-security.test.mjs && node --test scripts/build-copilot-distribution.test.mjs scripts/copilot-inner-distribution.test.mjs scripts/copilot-smoke-resources.test.mjs scripts/discover-copilot-acp.test.mjs scripts/materialize-copilot-binary.test.mjs scripts/install-copilot-assets.test.mjs scripts/probe-copilot-acp.test.mjs scripts/qualify-copilot-acp.test.mjs test/acpx-message-boundaries-package-contract.test.mjs",
|
||||
"test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs",
|
||||
"test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace",
|
||||
"test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider",
|
||||
|
||||
@@ -406,6 +406,15 @@ test('TestOps receives only a closed projection created inside the report bounda
|
||||
assert.equal(admits(false, true, false), false);
|
||||
assert.match(report, /environment:.*runner-e2e-private-history/u);
|
||||
assert.match(report, /Create the closed TestOps projection inside the report boundary[\s\S]*runner-protocol-testops\.mjs export/u);
|
||||
const publicGrid = report.slice(report.indexOf(' - name: Render the same canonical grid'), report.indexOf(' - name: Create the closed TestOps projection'));
|
||||
assert.match(publicGrid, /if: needs\.authorize\.outputs\.accounting_mode == 'legacy'/u);
|
||||
assert.doesNotMatch(publicGrid, /testops_enabled|env -u|unset /u);
|
||||
const sanitize = report.slice(report.indexOf(' - name: Project safe run records'), report.indexOf(' - name: Render the same canonical grid'));
|
||||
assert.match(sanitize, /accounting_mode == 'legacy' \|\| needs\.authorize\.outputs\.testops_enabled == 'true'/u);
|
||||
assert.match(sanitize, /runner-protocol-eval-campaign\.mjs sanitize/u);
|
||||
const projection = workflowShell(report, 'Create the closed TestOps projection inside the report boundary');
|
||||
assert.match(projection, /controlled\)[\s\S]*export-runs[\s\S]*campaign\.json runner-protocol-merged\/public-runs/u);
|
||||
assert.match(projection, /legacy\)[\s\S]*export \\[\s\S]*runner-protocol-merged\/public-report/u);
|
||||
assert.match(report, /Transfer only the closed TestOps results and receipt[\s\S]*path: runner-protocol-testops-projection\//u);
|
||||
assert.match(publisher, /name: runner-protocol-testops-projection-/u);
|
||||
assert.doesNotMatch(publisher, /runner-protocol-eval-public-|runner-protocol-merged|private-history|AWS_ACCESS_KEY_ID|BRAINTRUST_API_KEY/u);
|
||||
|
||||
@@ -139,6 +139,15 @@ export function filterCatalog(catalog, selection) {
|
||||
caseIds: [...selectedIds], allureJobRunId: selection.allureJobRunId } };
|
||||
}
|
||||
|
||||
export function payloadFromSanitizedRun(score, artifact, cell) {
|
||||
if (score?.schema !== "paperclip-runner/eval-score/v1" || artifact?.schema !== "paperclip-runner/eval-session-artifact/v1"
|
||||
|| score.attemptId !== cell.finalAttemptId || artifact.attemptId !== cell.finalAttemptId || score.caseId !== cell.caseId
|
||||
|| typeof score.passed !== "boolean" || artifact.publication?.schema !== "paperclip.runner-protocol-eval.public-chat/v1"
|
||||
|| artifact.requestedModel !== contract.model || artifact.provider !== contract.provider || artifact.driver !== contract.driver) fail("sanitized_payload_binding");
|
||||
return { attemptId: score.attemptId, caseId: score.caseId, disposition: score.disposition, passed: score.passed,
|
||||
publication: { schema: artifact.publication.schema }, checks: score.checks };
|
||||
}
|
||||
|
||||
export function projectResults({ selection, campaign, payloads = new Map() }) {
|
||||
validateSelection(selection);
|
||||
const campaignUrl = runUrl(selection.runId);
|
||||
@@ -244,27 +253,36 @@ async function main() {
|
||||
await githubOutputs({ matrix_0: JSON.stringify(catalog.matrices[0]), matrix_1: JSON.stringify(catalog.matrices[1]),
|
||||
matrix_1_present: String(catalog.matrices[1].include.length > 0), selected: catalog.cells.length });
|
||||
console.log(json({ selected: catalog.cells.length, fullPlan: catalog.selection.fullPlan }));
|
||||
} else if (command === "export") {
|
||||
} else if (command === "export" || command === "export-runs") {
|
||||
const structured = command === "export-runs";
|
||||
if (args.length !== (structured ? 4 : 3)) fail("arguments");
|
||||
const selection = await readJson(args[0]);
|
||||
const output = args[structured ? 3 : 2];
|
||||
let campaign;
|
||||
try { campaign = await readJson(join(args[1], "campaign.json")); } catch (error) { if (error.code !== "ENOENT") throw error; }
|
||||
try { campaign = await readJson(structured ? args[1] : join(args[1], "campaign.json")); } catch (error) { if (error.code !== "ENOENT") throw error; }
|
||||
const payloads = new Map();
|
||||
for (const row of campaign?.results ?? []) {
|
||||
if (!/^[A-Za-z0-9][A-Za-z0-9._-]{0,199}$/.test(row.finalAttemptId)) fail("attempt_path");
|
||||
try {
|
||||
const html = await readFile(join(args[1], "attempts", row.finalAttemptId, "index.html"), "utf8");
|
||||
const match = /<script type="application\/json" id="paperclip-eval-report">([^<]*)<\/script>/.exec(html);
|
||||
if (!match) fail("public_payload_missing");
|
||||
payloads.set(row.finalAttemptId, JSON.parse(match[1]));
|
||||
if (structured) {
|
||||
const directory = join(args[2], row.finalAttemptId);
|
||||
payloads.set(row.finalAttemptId, payloadFromSanitizedRun(await readJson(join(directory, "score.json")),
|
||||
await readJson(join(directory, "artifact.json")), row));
|
||||
} else {
|
||||
const html = await readFile(join(args[1], "attempts", row.finalAttemptId, "index.html"), "utf8");
|
||||
const match = /<script type="application\/json" id="paperclip-eval-report">([^<]*)<\/script>/.exec(html);
|
||||
if (!match) fail("public_payload_missing");
|
||||
payloads.set(row.finalAttemptId, JSON.parse(match[1]));
|
||||
}
|
||||
} catch (error) { if (error.code !== "ENOENT") throw error; }
|
||||
}
|
||||
const rows = projectResults({ selection, campaign, payloads });
|
||||
await mkdir(args[2], { recursive: false, mode: 0o700 });
|
||||
await mkdir(output, { recursive: false, mode: 0o700 });
|
||||
for (const row of rows) {
|
||||
await writeFile(join(args[2], row.file), json(row.result));
|
||||
await writeFile(join(args[2], row.attachment), json(row.evidence));
|
||||
await writeFile(join(output, row.file), json(row.result));
|
||||
await writeFile(join(output, row.attachment), json(row.evidence));
|
||||
}
|
||||
await writeFile(`${args[2]}-receipt.json`, json({ selection, results: rows.map(row => ({
|
||||
await writeFile(`${output}-receipt.json`, json({ selection, results: rows.map(row => ({
|
||||
testCaseId: Number(row.result.labels[0].value), fullName: row.result.fullName, status: row.result.status })) }));
|
||||
console.log(json({ reported: rows.length, passed: rows.filter(row => row.result.status === "passed").length }));
|
||||
} else if (command === "verify") {
|
||||
|
||||
@@ -1,14 +1,94 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
import { mkdtemp, mkdir, readFile, readdir, rm, writeFile } from "node:fs/promises";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { contract, selectCases, inspectSelection, filterCatalog, projectResults, verifyPublication, standingPlanCases, protocolTestOpsFailure,
|
||||
captureManualJobRunContext, parseOfficialJobRunContext } from "./runner-protocol-testops.mjs";
|
||||
captureManualJobRunContext, parseOfficialJobRunContext, payloadFromSanitizedRun } from "./runner-protocol-testops.mjs";
|
||||
import { testopsClient } from "../../../tests/runner-e2e/chat-feature-testops-api.mjs";
|
||||
import { sanitizeProtocolEvalRuns } from "./runner-protocol-eval-campaign.mjs";
|
||||
|
||||
const context = { runId: "123", workflowSha: "a".repeat(40), sourceSha: "a".repeat(40), evalsSha: "b".repeat(40),
|
||||
allureJobRunId: "7", jobId: 9, launchId: 11, accountingMode: "legacy" };
|
||||
const planned = contract.cases.map(row => ({ id: row.testCaseId, selector: row.fullName }));
|
||||
const selection = rows => selectCases(rows, context);
|
||||
|
||||
function sanitizedRun() {
|
||||
const expected = contract.cases[0];
|
||||
const cell = { rosterId: contract.rosterId, caseId: expected.caseId, model: contract.model, provider: contract.provider,
|
||||
driver: contract.driver, finalAttemptId: "attempt-1", disposition: "passed", passed: true };
|
||||
const score = { schema: "paperclip-runner/eval-score/v1", attemptId: cell.finalAttemptId, caseId: cell.caseId,
|
||||
disposition: cell.disposition, passed: true, checks: Object.entries(expected.checks).map(([id, kind]) => ({ id, kind, passed: true })) };
|
||||
const artifact = { schema: "paperclip-runner/eval-session-artifact/v1", attemptId: cell.finalAttemptId,
|
||||
requestedModel: contract.model, provider: contract.provider, driver: contract.driver,
|
||||
publication: { schema: "paperclip.runner-protocol-eval.public-chat/v1" }, privateExtra: "PRIVATE-SECRET" };
|
||||
const campaign = { schema: "paperclip.runner-protocol-eval.campaign/v1", source: { paperclip: { sha: context.sourceSha },
|
||||
evals: { sha: context.evalsSha }, workflowRunUrl: "https://github.com/paperclipai/paperclip/actions/runs/123" }, results: [cell] };
|
||||
return { cell, score, artifact, campaign };
|
||||
}
|
||||
|
||||
test("binds sanitized structured records without reading or forwarding private report content", () => {
|
||||
const { cell, score, artifact, campaign } = sanitizedRun();
|
||||
const input = { selection: selection(planned.slice(0, 1)), campaign,
|
||||
payloads: new Map([[cell.finalAttemptId, payloadFromSanitizedRun(score, artifact, cell)]]) };
|
||||
assert.equal(projectResults(input)[0].result.status, "passed");
|
||||
assert.ok(!JSON.stringify(input.payloads.get(cell.finalAttemptId)).includes("PRIVATE-SECRET"));
|
||||
for (const mutate of [value => { delete value.publication; }, value => { value.requestedModel = "other-model"; },
|
||||
value => { value.provider = "other-provider"; }, value => { value.driver = "other-driver"; },
|
||||
value => { value.attemptId = "other-attempt"; }, value => { value.schema = "raw-private-record"; }]) {
|
||||
const changed = structuredClone(artifact); mutate(changed);
|
||||
assert.throws(() => payloadFromSanitizedRun(score, changed, cell), /sanitized_payload_binding/);
|
||||
}
|
||||
for (const mutate of [value => { value.attemptId = "other-attempt"; }, value => { value.caseId = "other-case"; },
|
||||
value => { value.passed = "true"; }, value => { value.schema = "other-score"; }]) {
|
||||
const changed = structuredClone(score); mutate(changed);
|
||||
assert.throws(() => payloadFromSanitizedRun(changed, artifact, cell), /sanitized_payload_binding/);
|
||||
}
|
||||
for (const change of [checks => checks.pop(), checks => { checks[0].passed = false; }]) {
|
||||
const changed = structuredClone(input); change(changed.payloads.get(cell.finalAttemptId).checks);
|
||||
assert.throws(() => projectResults(changed), /unsupported_pass/);
|
||||
}
|
||||
});
|
||||
|
||||
test("exports the production sanitizer output as closed TestOps files under a controlled context", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "runner-testops-structured-"));
|
||||
try {
|
||||
const { cell, score, artifact, campaign } = sanitizedRun();
|
||||
const run = join(root, "raw-runs", cell.finalAttemptId);
|
||||
await mkdir(run, { recursive: true });
|
||||
delete artifact.publication;
|
||||
artifact.providerSessionId = "PRIVATE-SECRET";
|
||||
for (const [name, data] of [["selection.json", selectCases(planned.slice(0, 1), { ...context, accountingMode: "controlled" })], ["campaign.json", campaign],
|
||||
[`raw-runs/${cell.finalAttemptId}/score.json`, score], [`raw-runs/${cell.finalAttemptId}/artifact.json`, artifact],
|
||||
[`raw-runs/${cell.finalAttemptId}/case.json`, { id: cell.caseId, checks: score.checks }],
|
||||
[`raw-runs/${cell.finalAttemptId}/config.json`, { schema: "paperclip-runner/eval-config/v1", id: "config",
|
||||
model: contract.model, provider: contract.provider, driver: contract.driver }]]) {
|
||||
await writeFile(join(root, name), JSON.stringify(data));
|
||||
}
|
||||
const sanitize = () => sanitizeProtocolEvalRuns({ runsRoot: join(root, "raw-runs"), publicRunsRoot: join(root, "runs") });
|
||||
await sanitize();
|
||||
const execute = output => spawnSync(process.execPath, [fileURLToPath(new URL("./runner-protocol-testops.mjs", import.meta.url)),
|
||||
"export-runs", join(root, "selection.json"), join(root, "campaign.json"), join(root, "runs"), join(root, output)], {
|
||||
encoding: "utf8", env: { PATH: process.env.PATH, PAPERCLIP_RUNNER_EVAL_CONTROL_MANIFEST: "private-manifest" } });
|
||||
const positive = execute("positive");
|
||||
assert.equal(positive.status, 0, positive.stderr);
|
||||
assert.deepEqual(JSON.parse(positive.stdout), { reported: 1, passed: 1 });
|
||||
const files = await readdir(join(root, "positive"));
|
||||
assert.equal(files.length, 2);
|
||||
const values = await Promise.all(files.map(file => readFile(join(root, "positive", file), "utf8")));
|
||||
assert.ok(!values.join("").includes("PRIVATE-SECRET"));
|
||||
assert.equal(JSON.parse(values.find(value => JSON.parse(value).status)).status, "passed");
|
||||
score.checks[0].passed = false;
|
||||
await writeFile(join(run, "score.json"), JSON.stringify(score));
|
||||
await sanitize();
|
||||
const failed = execute("failed");
|
||||
assert.equal(failed.status, 1);
|
||||
assert.match(failed.stderr, /runner_testops_unsupported_pass/);
|
||||
} finally { await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
function catalog() {
|
||||
const roster = { rosterId: contract.rosterId, model: contract.model, provider: contract.provider,
|
||||
driver: contract.driver, cases: contract.cases.map(row => row.caseId) };
|
||||
|
||||
Reference in new issue
Block a user