mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 04:25:52 +01:00
DottaandPaperclip
e7ea2e2961
## Thinking Path > - Paperclip manages AI agents and their work. > - Product E2E campaigns retain each attempt and its independent grades. > - Braintrust receives product and cleanup grades for each retained attempt. > - A six-case qualification produces twelve scores. The export allowed only eight. > - Experiment creation succeeded, but grade insertion stopped before dispatch. > - This fix allows fourteen scores so the six- and seven-case batches retain every result. > - Operators can inspect failed and successful grades without running an agent again. ## Linked Issues or Issue Description Refs: #17044 **What happened?** A retained six-case Product E2E campaign created its Braintrust experiment, then failed with `Experiment ingestion allowance exhausted`. Its twelve deterministic scores exceeded the eight-score cap. No insert request was sent. Agent results and model accounting remained saved. **Expected behavior** The bounded qualification batches must export both grades for every selected attempt. Failed grades must remain visible. **Steps to reproduce** Create an experiment through `openExperimentIngestion`. Insert six or seven events with `product_pass` and `cleanup_pass` scores. The old limit rejects the insert before it reaches the transport. **Paperclip version or commit** Reproduced on master `cff331f6ca0a652b5e41641c68dfa5fa584c22ae`. **Deployment mode** Local Docker controller built from source. The failure is in the shared Product export path. ## What Changed - Raise the finite experiment score cap from eight to fourteen. Keep the three-request, 4 MiB and 60-second limits. - Add regressions for the six- and seven-case batches. Keep the failed product grade in each test batch. - Verify that the receipt collector accepts the old eight-score policy and the new fourteen-score policy. It rejects a fifteen-score policy. - Document the seven-attempt export ceiling. This change does not add agent calls or automatic retries. ## Verification - Before the fix: both new batch regressions failed with `Experiment ingestion allowance exhausted`. - After the fix: 67 tests pass across experiment ingestion, ingestion accounting, experiment projection and scheduled finalization. - Command: `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/experiment-ingestion.test.ts tests/runner-e2e/ingestion-usage.test.ts tests/runner-e2e/scheduled-finalization.test.ts tests/runner-e2e/braintrust-experiment.test.ts --configLoader runner --cache=false`. - `git diff --check` passes. All 58 current-head checks are acceptable, including typecheck, build and tests. Both reviews are 5/5, with no unresolved threads. - Live agent attempts will not be repeated to verify this reporting fix. Recovery uses the saved results and a separate bounded ingestion receipt. ## Risks - New exports can submit at most six more deterministic scores per experiment. These are saved grades, not model judge calls. - Larger campaigns still fail closed. Existing receipt files keep their original policies and traffic counts. - This fix does not resolve the plan-lifecycle behavior failures observed in the qualification. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>