fix(evals): export grades for expanded qualification batches (#18010)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Product E2E campaigns retain each attempt and its independent
grades.
> - Braintrust receives product and cleanup grades for each retained
attempt.
> - A six-case qualification produces twelve scores. The export allowed
only eight.
> - Experiment creation succeeded, but grade insertion stopped before
dispatch.
> - This fix allows fourteen scores so the six- and seven-case batches
retain every result.
> - Operators can inspect failed and successful grades without running
an agent again.

## Linked Issues or Issue Description

Refs: #17044

**What happened?**

A retained six-case Product E2E campaign created its Braintrust
experiment, then failed with `Experiment ingestion allowance exhausted`.
Its twelve deterministic scores exceeded the eight-score cap. No insert
request was sent. Agent results and model accounting remained saved.

**Expected behavior**

The bounded qualification batches must export both grades for every
selected attempt. Failed grades must remain visible.

**Steps to reproduce**

Create an experiment through `openExperimentIngestion`. Insert six or
seven events with `product_pass` and `cleanup_pass` scores. The old
limit rejects the insert before it reaches the transport.

**Paperclip version or commit**

Reproduced on master `cff331f6ca0a652b5e41641c68dfa5fa584c22ae`.

**Deployment mode**

Local Docker controller built from source. The failure is in the shared
Product export path.

## What Changed

- Raise the finite experiment score cap from eight to fourteen. Keep the
three-request, 4 MiB and 60-second limits.
- Add regressions for the six- and seven-case batches. Keep the failed
product grade in each test batch.
- Verify that the receipt collector accepts the old eight-score policy
and the new fourteen-score policy. It rejects a fifteen-score policy.
- Document the seven-attempt export ceiling. This change does not add
agent calls or automatic retries.

## Verification

- Before the fix: both new batch regressions failed with `Experiment
ingestion allowance exhausted`.
- After the fix: 67 tests pass across experiment ingestion, ingestion
accounting, experiment projection and scheduled finalization.
- Command: `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts
tests/runner-e2e/experiment-ingestion.test.ts
tests/runner-e2e/ingestion-usage.test.ts
tests/runner-e2e/scheduled-finalization.test.ts
tests/runner-e2e/braintrust-experiment.test.ts --configLoader runner
--cache=false`.
- `git diff --check` passes. All 58 current-head checks are acceptable,
including typecheck, build and tests. Both reviews are 5/5, with no
unresolved threads.
- Live agent attempts will not be repeated to verify this reporting fix.
Recovery uses the saved results and a separate bounded ingestion
receipt.

## Risks

- New exports can submit at most six more deterministic scores per
experiment. These are saved grades, not model judge calls.
- Larger campaigns still fail closed. Existing receipt files keep their
original policies and traffic counts.
- This fix does not resolve the plan-lifecycle behavior failures
observed in the qualification.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution.
The exact serving variant and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

tests/runner-e2e/OPERATIONS.md

@@ -399,12 +399,16 @@ This path currently supports the admitted local cells; remote transport remains
 unqualified.
 
 Scheduled experiment creation, grade insertion and summary retrieval share a
-60-second window, three-request cap, 4 MiB total payload allowance and eight-score
+60-second window, three-request cap, 4 MiB total payload allowance and fourteen-score
 cap. They run serially without automatic retry. Admission is persisted before
 dispatch in `<ledgerPath>.experiment-ingestion.jsonl`; uncertain responses retain
 their allowance and unresolved status. Reusing that file fails instead of
 resetting the budget. These limits cover the selected daily/weekly campaigns;
 the standalone experiment-import command does not use this transport budget.
+Each retained attempt contributes two deterministic grades, including failed
+grades. The cap supports seven attempts without dropping rows. Larger campaigns
+still fail closed. Historical receipts with the eight-score policy remain
+readable and keep their original limits and traffic counts.
 
 Private `ingestion-usage.json` binds all three receipt paths to the campaign's
 admitted attempts and ledger hash. Missing, malformed, unclosed or unresolved

tests/runner-e2e/experiment-ingestion.test.ts

@@ -52,6 +52,19 @@ it("keeps failed forwarding charged to the finite allowance and records an unres
   expect((await f.records()).at(-1)).toMatchObject({ admittedRequests: 3, responses: 0, unresolvedRequests: 3 });
 });
 
+it.each([6, 7])("exports both deterministic grades for every retained case in a %i-case qualification", async cases => {
+  const upstream = vi.fn<typeof fetch>(async () => Response.json({ ok: true }));
+  const f = await fixture(upstream);
+  await f.send({ project_id: projectId });
+  await f.send({ events: Array.from({ length: cases }, (_, index) => ({
+    id: `attempt-${index}`, scores: { product_pass: Number(index !== 0), cleanup_pass: 1 },
+  })) }, `experiment/${experimentId}/insert`);
+  await f.transport(`https://api.braintrust.dev/v1/experiment/${experimentId}/summarize`, { method: "GET" });
+  await f.close();
+  expect(upstream).toHaveBeenCalledTimes(3);
+  expect((await f.records()).at(-1)).toMatchObject({ admittedRequests: 3, admittedScores: cases * 2, unresolvedRequests: 0 });
+});
+
 it("retains an unresolved allowance when response headers arrive but the body disconnects", async () => {
   const f = await fixture(async () => new Response(new ReadableStream({ pull(controller) {
     controller.error(new Error("private-disconnect-error"));
@@ -85,7 +98,7 @@ it("refuses excess bytes, scores, foreign project and unqualified endpoints befo
   const f = await fixture(upstream);
   await expect(f.send({ project_id: experimentId })).rejects.toThrow("project mismatch");
   await expect(f.send({ project_id: projectId, data: "x".repeat(EVAL_EXPERIMENT_LIMITS.maxTotalBytes) })).rejects.toThrow("byte limit");
-  await expect(f.send({ events: Array.from({ length: 9 }, () => ({ scores: { pass: 1 } })) }, `experiment/${experimentId}/insert`))
+  await expect(f.send({ events: Array.from({ length: EVAL_EXPERIMENT_LIMITS.maxScores + 1 }, () => ({ scores: { pass: 1 } })) }, `experiment/${experimentId}/insert`))
     .rejects.toThrow("allowance exhausted");
   await expect(f.send({}, "project")).rejects.toThrow("Unqualified");
   expect(upstream).not.toHaveBeenCalled();

tests/runner-e2e/experiment-ingestion.ts

@@ -3,7 +3,7 @@ import { mkdir, open } from "node:fs/promises";
 import path from "node:path";
 import { isEvalProjectId } from "./eval-otlp-payload.js";
 
-export const EVAL_EXPERIMENT_LIMITS = { maxRequests: 3, maxTotalBytes: 4 * 1024 * 1024, maxScores: 8 } as const;
+export const EVAL_EXPERIMENT_LIMITS = { maxRequests: 3, maxTotalBytes: 4 * 1024 * 1024, maxScores: 14 } as const;
 
 export async function openExperimentIngestion(input: {
   receiptPath: string; campaignId: string; projectId: string; deadlineAt: number; transport?: typeof fetch;

tests/runner-e2e/ingestion-usage.test.ts

@@ -90,6 +90,16 @@ it("keeps historical native receipts with the larger policy limits readable", as
     totalAdmittedPayloadBytes: f.expectedBytes, billedCostUsd: null });
 });
 
+it.each([8, 14, 15])("validates an experiment receipt with a %i-score policy against the supported ceiling", async maxScores => {
+  const f = await fixture(), file = `${f.input.ledgerPath}.experiment-ingestion.jsonl`;
+  const rows = (await readFile(file, "utf8")).trim().split("\n").map(line => JSON.parse(line));
+  rows[0].limits.maxScores = maxScores;
+  await writeFile(file, rows.map(row => JSON.stringify(row)).join("\n") + "\n");
+  const usage = await collectScheduledIngestion(f.input);
+  expect(usage.status).toBe(maxScores <= 14 ? "complete" : "incomplete");
+  expect(usage.receipts.at(-1)!.status).toBe(maxScores <= 14 ? "complete" : "unavailable");
+});
+
 it("keeps version 1 transport receipts readable without inventing expanded data measurements", async () => {
   const f = await fixture(), file = f.files[0]!;
   const rows = (await readFile(file, "utf8")).trim().split("\n").map(line => JSON.parse(line));