Files
PaperClipAI/tests
Dotta 9203ab52f0 fix(evals): bind complete unassigned TestOps plan buckets (#18608)
## Thinking Path

> - Paperclip manages agents that do real work.
> - Product evals check the saved outcome of that work.
> - TestOps plans select the exact automated cases to run.
> - A new plan can report its unassigned cases as a job-statistics
bucket.
> - The pilot registration guard rejects that bucket as a foreign job.
> - This change binds only a complete, known unassigned bucket and
records assignment diagnostics before writes.

## Linked Issues or Issue Description

Refs #18516 and #18542. The metadata-only bootstrap in
[41464514278](https://github.com/paperclipai/paperclip/actions/runs/41464514278)
created the dedicated plan and verified its GitHub job definition, then
stopped at `plan_job_mismatch` before job or case-assignment mutations.
No model work ran.

The retained receipt did not include the assignment rows. The live
failure's exact shape remains unconfirmed. The API's job-statistics
carrier permits an absent job. The old guard rejects a complete
unassigned bucket. This PR covers that case and records the actual rows
in the next closed inspection.

## What Changed

- Accept one closed unassigned bucket with the exact selected-case
count. Support an explicit null job or an omitted optional job field.
- Assign the exact selected cases once. Keep the existing owned
assignment without assigning it twice.
- Reject foreign jobs, mixed buckets, wrong counts and unknown
unassigned fields before mutations.
- Retain only bounded assignment kinds, numeric job IDs and counts in
the existing observation receipt.
- Merge current master, including the full-catalog adapter from #18542.
Retain its context-aware plan and job checks alongside the new pre-write
assignment inspection. The helper consumes that context’s expected case
count.
- Document the inspection and binding rules. The helper takes the
expected count from its owner. This does not import or run the
full-catalog route from #18542.

## Verification

- Red: both complete unassigned carriers fail on the old guard; the
other 70 dispatch/API controls pass.
- Green: all 72 dispatch/API controls passed before the master sync.
After the sync, all 104 related pilot and full-catalog Node controls
pass. They include ordinary registration, no duplicate assignment,
foreign-job rejection, incomplete and unknown bucket rejection, launch
fences, uncertain-response protection and result readback.
- Diff check passes. No TypeScript or compiled product code changes.
Full repository checks were not run locally. All required CI checks pass
on current head `a8bec6418065a7faae28c8daab3efdfc9b184797` ([run
41473510972](https://github.com/paperclipai/paperclip/actions/runs/41473510972)).
Both current-head reviews are 5/5 with no unresolved threads. Before the
master sync, one unchanged server timing test failed on attempt one and
passed on the only targeted rerun; that original failure remains in CI
attempt one. No rerun was needed on the current head.
- Live acceptance still requires another metadata-only inspection,
followed by a genuine three-case TestOps launch. This PR does not claim
a new passing measurement.

## Risks

An overly broad unassigned rule could replace an existing job. The new
rule permits only one known bucket with the exact selected-case count
and no foreign job. All final binding checks still apply. Unknown
assignment shapes stop before writes. Original case queries, model,
clocks, graders, evidence rules and retries are unchanged.

## Model Used

OpenAI GPT-6 through Codex assisted with repository inspection, code
execution and tool use. The host does not expose the exact model
revision or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run the relevant local controls and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-02 09:56:36 -06:00
..