Files
PaperClipAI/tests
DottaandPaperclip 8595efc20b fix(evals): support official TestOps job metadata (#18385)
## Thinking Path

> - Paperclip manages AI agents and verifies real work through Product
E2E tests.
> - The fixed Chat feature workflow uses TestOps to select and report
one backlog case.
> - Official registration exposes a workflow-query URL and a Branch
parameter.
> - The existing reader accepts only workflow-file or numeric URLs, and
the admin launcher requires an operation parameter.
> - These assumptions reject valid official metadata before the feature
can run.
> - This pull request supports the exact observed form while preserving
source, selection, and once-only execution checks.

## Linked Issues or Issue Description

Refs: #18322, #18375.

**What happened?**
The second zero-model bootstrap stopped with a generic job-mismatch
code. Ordinary TestOps UI inspection then showed an official
workflow-query URL outside the accepted literals and a Branch-only job.
The exact first failed predicate in that bootstrap remains unknown.

**Expected behavior**
Accept that exact fixed-workflow URL. Preserve the existing master
Branch value when the job has no operation parameter. Keep rejecting
foreign URLs, unknown parameters, wrong branches, and conflicting
operation IDs before mutation.

**Steps to reproduce**
Pass the official query URL and Branch-only metadata through the
existing bootstrap, selection, and admin-launch owners. The new pure
controls exercise those actual helpers without a CLI, backend, or model
call.

**Paperclip version or commit**
This change starts from `0e15c92073a09e50df99190a37f69a86286b989b` after
#18375.

**Deployment mode**
GitHub Actions and the protected TestOps reporting environment.

## What Changed

- Share one exact URL predicate across bootstrap, evaluation selection,
and admin preparation. Add only the observed official workflow-query
literal.
- Support Branch-only, operation-only, and Branch-plus-operation jobs
with strict definition IDs, names, values, and duplicate checks.
- Preserve the exact master Branch value. Set an existing operation
parameter to `evaluate`; otherwise use the fixed workflow's existing
default.
- Use the same checked parameter vector for job and plan writes and
their readbacks.
- Report distinct fixed bootstrap predicate codes without response
values.
- Document when the protected operation-parameter ID is required.
- Align one existing Cloud-maintenance test with the inherited
managed-Cloud S3 policy. Preserve the same-version local-storage proof
in the parent, use valid synthetic Cloud settings in the child, and
check the expected provider mismatch without S3 requests. Retain the
no-seed and unchanged persisted-row assertions.

## Verification

- URL controls: three failures on the parent; all pass with the
correction.
- Branch controls: two failures and one correct rejection on the
predecessor; all pass with the correction.
- `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs`: 40
owning pure/mock controls pass on Node 26.4.0.
- Two MJS syntax checks and `git diff --check` pass. TypeScript noEmit
was not run for this MJS/documentation change.
- The first [normal CI
run](https://github.com/paperclipai/paperclip/actions/runs/41131094974)
failed only in the managed-skill Cloud fixture and its dependent
verification gate: 46 jobs passed, two failed, and one skipped. The
tested merge included a new upstream policy that rejects the test's
local-disk Cloud configuration. The exact caught runtime exception
remains unknown.
- Three pure controls using the actual policy validator pass: the old
invalid Cloud fixture rejects, the new synthetic S3 fixture admits, and
an invalid master key rejects. An initial tsx wrapper IPC permission
error executed zero controls and remains a separate setup failure.
- The PostgreSQL/maintenance-child test was not run locally. On new-head
AWS CI, `managed-skill-content.test.ts` passed all 17 tests in server
shard 9.
- [Normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41133876636)
is green on exact head `b76d5dd40d7711f1b5f6b6316b1114c0aae02d57`: 48
jobs passed and one skipped. Both required `ci / e2e` and `ci / verify`
pass.
- Greptile reviewed this exact head at 5/5 with no actionable findings
or unresolved review threads.
- No actual CLI, backend, model, provider, app, database, build, or
bootstrap was run to validate this correction locally.

## Risks

The exact original bootstrap predicate is still unknown. This fixes a
demonstrated metadata false negative and preserves the prior failed run.
Actual backend parameter types and integration dispatch still need
verification. Unknown metadata stops before the once marker and writes.
This PR changes no workflow, credential, case, model, grader, clock, or
private-evidence policy. It does not claim a passing feature result.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool
use, and pure test review. The runtime does not expose the exact
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 17:04:59 -05:00
..