From e7ea2e2961863171956e70fb05d3d1f3790f7909 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 28 Oct 2026 07:50:59 -0500 Subject: [PATCH] fix(evals): export grades for expanded qualification batches (#18010) ## Thinking Path > - Paperclip manages AI agents and their work. > - Product E2E campaigns retain each attempt and its independent grades. > - Braintrust receives product and cleanup grades for each retained attempt. > - A six-case qualification produces twelve scores. The export allowed only eight. > - Experiment creation succeeded, but grade insertion stopped before dispatch. > - This fix allows fourteen scores so the six- and seven-case batches retain every result. > - Operators can inspect failed and successful grades without running an agent again. ## Linked Issues or Issue Description Refs: #17044 **What happened?** A retained six-case Product E2E campaign created its Braintrust experiment, then failed with `Experiment ingestion allowance exhausted`. Its twelve deterministic scores exceeded the eight-score cap. No insert request was sent. Agent results and model accounting remained saved. **Expected behavior** The bounded qualification batches must export both grades for every selected attempt. Failed grades must remain visible. **Steps to reproduce** Create an experiment through `openExperimentIngestion`. Insert six or seven events with `product_pass` and `cleanup_pass` scores. The old limit rejects the insert before it reaches the transport. **Paperclip version or commit** Reproduced on master `cff331f6ca0a652b5e41641c68dfa5fa584c22ae`. **Deployment mode** Local Docker controller built from source. The failure is in the shared Product export path. ## What Changed - Raise the finite experiment score cap from eight to fourteen. Keep the three-request, 4 MiB and 60-second limits. - Add regressions for the six- and seven-case batches. Keep the failed product grade in each test batch. - Verify that the receipt collector accepts the old eight-score policy and the new fourteen-score policy. It rejects a fifteen-score policy. - Document the seven-attempt export ceiling. This change does not add agent calls or automatic retries. ## Verification - Before the fix: both new batch regressions failed with `Experiment ingestion allowance exhausted`. - After the fix: 67 tests pass across experiment ingestion, ingestion accounting, experiment projection and scheduled finalization. - Command: `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/experiment-ingestion.test.ts tests/runner-e2e/ingestion-usage.test.ts tests/runner-e2e/scheduled-finalization.test.ts tests/runner-e2e/braintrust-experiment.test.ts --configLoader runner --cache=false`. - `git diff --check` passes. All 58 current-head checks are acceptable, including typecheck, build and tests. Both reviews are 5/5, with no unresolved threads. - Live agent attempts will not be repeated to verify this reporting fix. Recovery uses the saved results and a separate bounded ingestion receipt. ## Risks - New exports can submit at most six more deterministic scores per experiment. These are saved grades, not model judge calls. - Larger campaigns still fail closed. Existing receipt files keep their original policies and traffic counts. - This fix does not resolve the plan-lifecycle behavior failures observed in the qualification. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- tests/runner-e2e/OPERATIONS.md | 6 +++++- tests/runner-e2e/experiment-ingestion.test.ts | 15 ++++++++++++++- tests/runner-e2e/experiment-ingestion.ts | 2 +- tests/runner-e2e/ingestion-usage.test.ts | 10 ++++++++++ 4 files changed, 30 insertions(+), 3 deletions(-) diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md index b99c85ec149..e193652ba13 100644 --- a/tests/runner-e2e/OPERATIONS.md +++ b/tests/runner-e2e/OPERATIONS.md @@ -399,12 +399,16 @@ This path currently supports the admitted local cells; remote transport remains unqualified. Scheduled experiment creation, grade insertion and summary retrieval share a -60-second window, three-request cap, 4 MiB total payload allowance and eight-score +60-second window, three-request cap, 4 MiB total payload allowance and fourteen-score cap. They run serially without automatic retry. Admission is persisted before dispatch in `.experiment-ingestion.jsonl`; uncertain responses retain their allowance and unresolved status. Reusing that file fails instead of resetting the budget. These limits cover the selected daily/weekly campaigns; the standalone experiment-import command does not use this transport budget. +Each retained attempt contributes two deterministic grades, including failed +grades. The cap supports seven attempts without dropping rows. Larger campaigns +still fail closed. Historical receipts with the eight-score policy remain +readable and keep their original limits and traffic counts. Private `ingestion-usage.json` binds all three receipt paths to the campaign's admitted attempts and ledger hash. Missing, malformed, unclosed or unresolved diff --git a/tests/runner-e2e/experiment-ingestion.test.ts b/tests/runner-e2e/experiment-ingestion.test.ts index 984bcb91a78..1e8fe2a2aec 100644 --- a/tests/runner-e2e/experiment-ingestion.test.ts +++ b/tests/runner-e2e/experiment-ingestion.test.ts @@ -52,6 +52,19 @@ it("keeps failed forwarding charged to the finite allowance and records an unres expect((await f.records()).at(-1)).toMatchObject({ admittedRequests: 3, responses: 0, unresolvedRequests: 3 }); }); +it.each([6, 7])("exports both deterministic grades for every retained case in a %i-case qualification", async cases => { + const upstream = vi.fn(async () => Response.json({ ok: true })); + const f = await fixture(upstream); + await f.send({ project_id: projectId }); + await f.send({ events: Array.from({ length: cases }, (_, index) => ({ + id: `attempt-${index}`, scores: { product_pass: Number(index !== 0), cleanup_pass: 1 }, + })) }, `experiment/${experimentId}/insert`); + await f.transport(`https://api.braintrust.dev/v1/experiment/${experimentId}/summarize`, { method: "GET" }); + await f.close(); + expect(upstream).toHaveBeenCalledTimes(3); + expect((await f.records()).at(-1)).toMatchObject({ admittedRequests: 3, admittedScores: cases * 2, unresolvedRequests: 0 }); +}); + it("retains an unresolved allowance when response headers arrive but the body disconnects", async () => { const f = await fixture(async () => new Response(new ReadableStream({ pull(controller) { controller.error(new Error("private-disconnect-error")); @@ -85,7 +98,7 @@ it("refuses excess bytes, scores, foreign project and unqualified endpoints befo const f = await fixture(upstream); await expect(f.send({ project_id: experimentId })).rejects.toThrow("project mismatch"); await expect(f.send({ project_id: projectId, data: "x".repeat(EVAL_EXPERIMENT_LIMITS.maxTotalBytes) })).rejects.toThrow("byte limit"); - await expect(f.send({ events: Array.from({ length: 9 }, () => ({ scores: { pass: 1 } })) }, `experiment/${experimentId}/insert`)) + await expect(f.send({ events: Array.from({ length: EVAL_EXPERIMENT_LIMITS.maxScores + 1 }, () => ({ scores: { pass: 1 } })) }, `experiment/${experimentId}/insert`)) .rejects.toThrow("allowance exhausted"); await expect(f.send({}, "project")).rejects.toThrow("Unqualified"); expect(upstream).not.toHaveBeenCalled(); diff --git a/tests/runner-e2e/experiment-ingestion.ts b/tests/runner-e2e/experiment-ingestion.ts index 7cb1c102bbe..1e221c9d934 100644 --- a/tests/runner-e2e/experiment-ingestion.ts +++ b/tests/runner-e2e/experiment-ingestion.ts @@ -3,7 +3,7 @@ import { mkdir, open } from "node:fs/promises"; import path from "node:path"; import { isEvalProjectId } from "./eval-otlp-payload.js"; -export const EVAL_EXPERIMENT_LIMITS = { maxRequests: 3, maxTotalBytes: 4 * 1024 * 1024, maxScores: 8 } as const; +export const EVAL_EXPERIMENT_LIMITS = { maxRequests: 3, maxTotalBytes: 4 * 1024 * 1024, maxScores: 14 } as const; export async function openExperimentIngestion(input: { receiptPath: string; campaignId: string; projectId: string; deadlineAt: number; transport?: typeof fetch; diff --git a/tests/runner-e2e/ingestion-usage.test.ts b/tests/runner-e2e/ingestion-usage.test.ts index 38e921b79f6..69633994a0e 100644 --- a/tests/runner-e2e/ingestion-usage.test.ts +++ b/tests/runner-e2e/ingestion-usage.test.ts @@ -90,6 +90,16 @@ it("keeps historical native receipts with the larger policy limits readable", as totalAdmittedPayloadBytes: f.expectedBytes, billedCostUsd: null }); }); +it.each([8, 14, 15])("validates an experiment receipt with a %i-score policy against the supported ceiling", async maxScores => { + const f = await fixture(), file = `${f.input.ledgerPath}.experiment-ingestion.jsonl`; + const rows = (await readFile(file, "utf8")).trim().split("\n").map(line => JSON.parse(line)); + rows[0].limits.maxScores = maxScores; + await writeFile(file, rows.map(row => JSON.stringify(row)).join("\n") + "\n"); + const usage = await collectScheduledIngestion(f.input); + expect(usage.status).toBe(maxScores <= 14 ? "complete" : "incomplete"); + expect(usage.receipts.at(-1)!.status).toBe(maxScores <= 14 ? "complete" : "unavailable"); +}); + it("keeps version 1 transport receipts readable without inventing expanded data measurements", async () => { const f = await fixture(), file = f.files[0]!; const rows = (await readFile(file, "utf8")).trim().split("\n").map(line => JSON.parse(line));