Commit Graph
4 Commits
Author SHA1 Message Date
DottaandPaperclip b8a8d005ab fix(evals): close failed TestOps runs and verify each verdict
Keep official run-start context available to always-on cleanup after selection failure or cancellation. Validate manual selectors before opening a run and bind pending TestOps jobs before adoption. Compare every uploaded case verdict against its sealed projection, including a swapped-status negative control.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 10:02:28 -06:00
DottaandPaperclip d908af01dc test(evals): bind official aliases for the full-stack workflow
Accept only the named, master blob and independently verified numeric workflow URLs for the same full-stack workflow. Keep foreign workflow and repository aliases rejected.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 09:51:55 -06:00
DottaandPaperclip 0da58a64d7 test(evals): retain authoritative attempt verdict receipts
Use normalized campaign accounting for earlier failed attempts. Preserve provider model identifiers with slashes and validate the actual string status returned by TestOps. Cover missing-artifact placeholder rows and server-side status/identity controls.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 09:51:15 -06:00
DottaandPaperclip d4b11b2024 test(evals): run Product regression selections from TestOps
Reuse the paid full-stack workflow for the standing 20-case Product regression plan. Bind TestOps case identities, original job runs, source revisions and retained verdicts in separate trusted reporting jobs. Preserve existing paid admission and make missing evidence explicitly broken.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 09:49:07 -06:00