fix(evals): report controlled protocol cases from safe records (#19070)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner protocol qualification records real provider behavior in
TestOps.
> - Controlled attempts keep raw records and accounting in private
storage.
> - A live case passed, but a later public-render command rejected its
controlled context.
> - The missing public report caused the final result to be stored as
broken.
> - This pull request creates the closed TestOps result from existing
sanitized records inside the private report job.
> - The benefit is correct result reporting while the privacy guard
stays effective.

## Linked Issues or Issue Description

Refs #18985 and #19011. The related-PR search found no separate repair
for this failure.

**What happened?**

[The live
canary](https://github.com/paperclipai/paperclip/actions/runs/41916513378)
passed all four provider checks. Its controller verified cleanup and
closed its request accounting. The private canonical report recorded one
passed case. The next command used `--public-viewer` while the
controlled manifest remained set. The eval program rejected it with
`controlled campaign evidence must remain private`. The closed exporter
then had no public payload and correctly stored a broken result.

**Expected behavior**

Report the original case grade from sanitized structured records. Keep
raw controlled records private. Keep the original checks, source
bindings, and missing-evidence rules.

**Steps to reproduce**

Run one controlled protocol case with TestOps reporting enabled. Let the
private canonical report finish. The following public-render step fails
before the closed result can read the grade.

**Paperclip version or commit**

Observed at `4740d251b7032e714d33039ba66a56948238c38f`. This repair is
based on `c6cb7e2a2d42743c0c3c4ead2cbad4a299f9a2e7`.

**Deployment mode**

GitHub-hosted Linux, controlled ACPX Claude protocol qualification,
first workflow attempt, zero automatic retries.

## What Changed

- Add a narrow `export-runs` path to the existing TestOps helper. Bind
the sanitized score and artifact to the exact attempt, case, model,
provider, driver, and public-projection schema. Reuse the existing
strict grade and check validation.
- Keep sanitization inside the private report job. Use structured
records for controlled TestOps reporting. Keep the public viewer for
legacy publication.
- Add record identity, privacy, grade, command, and workflow routing
controls. Wire the existing helper and workflow-security files into the
required Node registry. Document the reporting path.

## Verification

- 43 focused campaign/sanitizer, TestOps and workflow security tests
passed. The command regression writes a raw attempt, calls the
production sanitizer, and runs `export-runs` on its output. It also
repeats that chain after a failed-check mutation. Both new-control files
now run through required `test:typescript:prep`, not only the optional
publisher script.
- The actual required prep ran locally with provider credentials
excluded. The eval-kernel and Rust builds passed. Its Node registry
executed the new producer-to-export and security controls by name: 403
passed, 2 failed, 11 existing skips. The two original package-contract
failures cannot resolve this isolated clone's Codex ACP and Claude ACP
package links. This is not a green local full-prep claim. The first
attempt also stopped at a missing Node-types link, before an ignored
clone-local link to the locked cached version restored that
prerequisite.
- 72 release workflow tests passed with the existing locked pnpm 9.15.4
wrapper and local loopback permission. The first sandbox run failed
because loopback fixtures could not listen; the default local tool path
also failed two install fixtures. No test assertions were changed to
resolve these failures.
- The actual repaired workflow shell ran locally against the retained
canary campaign and sanitized records. It emitted one closed passed
result with all four original checks. No provider request or upload
occurred.
- Missing or failed checks still reject a pass. Private-only markers
stay absent from the closed output. Wrong model, provider, driver,
attempt, case, and schemas reject.
- The unmodified, pinned eval program still rejects a controlled
public-viewer command.
- Actionlint and `git diff --check` passed. [Full final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41927878886)
passed on its first attempt at `0d0c090b1b`. Required verify/e2e gates,
all four Runner lanes, typecheck, build, backend and browser shards
passed. Package suite: 4951 passed, 10 existing skips across 297 passed
files and one existing file skip.
- The required static log contains passing lines for both the production
sanitizer-to-export command regression and the closed-projection
security control. The prior optional-script registration did not run
those tests in required CI; the one script-entry fix now enforces them.
Provider/runtime and workflow bytes are unchanged by the
verification-only follow-ups.
- Greptile: fresh 5/5 on `0d0c090b1b`, with no actionable new issues.
The separate Paperclip Review requested a producer-to-consumer
regression. That regression and its required-CI wiring are complete. The
addressed thread is resolved; no open review threads remain.
- No fresh deployed reporting pass is claimed. The failed workflow and
its stored broken result remain unchanged.

## Risks

The exporter reads structured records inside the private job. It copies
only the closed result fields and requires the existing sanitized marker
and identity checks. The final reporting job still receives only those
files and verifies the saved result bindings. The Python privacy guard,
provider path, case assertions, budgets, deadlines, and retries are
unchanged.

## Model Used

OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort. The
inherited runtime reports a 475,000-token context window. Capabilities
used: reasoning, code changes, and repository, shell, and GitHub tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-04 22:03:59 -06:00
1 parent 06826915d7
commit 85e4d12856
6 files changed
+147 -19

No files matched your search

@@ -1294,12 +1294,16 @@ jobs:
fi
exit "$status"
- name: Render the same canonical grid from a public-safe evidence projection
- name: Project safe run records for reporting
if: needs.authorize.outputs.accounting_mode == 'legacy' || needs.authorize.outputs.testops_enabled == 'true'
run: |
node packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs sanitize \
--runs-root runner-protocol-merged/runs \
--output runner-protocol-merged/public-runs
- name: Render the same canonical grid from a public-safe evidence projection
if: needs.authorize.outputs.accounting_mode == 'legacy'
run: |
python3 .paperclip-evals/evals/paperclip-runner/tools/eval_program.py report \
--runs-root runner-protocol-merged/public-runs \
--output runner-protocol-merged/public-report \
@@ -1312,11 +1316,24 @@ jobs:
- name: Create the closed TestOps projection inside the report boundary
if: always() && !cancelled() && needs.authorize.outputs.testops_enabled == 'true' && needs.testops_selection.result == 'success'
id: testops_projection
env:
ACCOUNTING_MODE: ${{ needs.authorize.outputs.accounting_mode }}
run: |
mkdir -m 700 runner-protocol-testops-projection
node packages/paperclip-runner/scripts/runner-protocol-testops.mjs export \
runner-protocol-testops-selection/runner-protocol-testops-selection.json \
runner-protocol-merged/public-report runner-protocol-testops-projection/results
case "$ACCOUNTING_MODE" in
controlled)
node packages/paperclip-runner/scripts/runner-protocol-testops.mjs export-runs \
runner-protocol-testops-selection/runner-protocol-testops-selection.json \
runner-protocol-merged/campaign.json runner-protocol-merged/public-runs \
runner-protocol-testops-projection/results
;;
legacy)
node packages/paperclip-runner/scripts/runner-protocol-testops.mjs export \
runner-protocol-testops-selection/runner-protocol-testops-selection.json \
runner-protocol-merged/public-report runner-protocol-testops-projection/results
;;
*) exit 1 ;;
esac
- name: Transfer only the closed TestOps results and receipt
if: always() && steps.testops_projection.outcome == 'success'