Files
PaperClipAI/tests
DottaandPaperclip 410b0e2701 fix(evals): accept refreshed catalog workflow URL (#18666)
## Thinking Path

> - Paperclip lets people manage AI agents and inspect their work.
> - The chat catalog reports original Product E2E grades to TestOps.
> - TestOps can refresh its job metadata from GitHub.
> - That operation returns a canonical master workflow-file URL.
> - The catalog guard currently rejects that exact URL before case
selection completes.
> - This PR accepts that fixed URL and keeps the existing source and
identity checks.
> - The catalog can use the refreshed job without changing its cases or
grades.

## Linked Issues or Issue Description

Refs #18542, #18647.

**What happened?**

A supported TestOps job refresh changed the catalog job URL to the
GitHub `blob/master/.github/workflows/chat-catalog-testops.yml` form.
The selection guard rejects it as `job_mismatch`.

**Expected behavior**

The guard accepts that exact parent workflow URL. It still rejects
another owner, branch or workflow.

**Steps to reproduce**

Run the new adapter regression against the parent commit. It fails in
`validatePilotSelection` before the job-run request.

## What Changed

- Add one exact canonical master URL to the full-catalog workflow URL
guard.
- Add a regression through the real selection validator with refreshed
job metadata. Keep negative checks for another owner, branch and the
called case workflow.

## Verification

- The new selection regression fails on the parent with `job_mismatch`
and passes with this change.
- All 73 focused controls pass: `node --test
tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs
tests/runner-e2e/chat-completion-testops-api.test.mjs
tests/runner-e2e/chat-catalog-testops-streaming.test.mjs`.
- Both changed files pass Node syntax checks. `git diff --check` passes.
- These controls use synthetic metadata and mocked HTTP. No local app,
database, native build or provider runs.
- Removing the one added predicate restores the parent API file exactly.
The branch, job, case, source, evidence and grade checks remain
unchanged. The pilot URL policy also stays unchanged.
- Current-head GitHub CI passes: 48 jobs pass and one is skipped,
including the required typecheck, test, build and final summaries. Live
streaming qualification remains pending. No documentation change is
needed for this internal URL compatibility fix.

## Risks

- The alias accepts only this repository, master branch and parent
workflow in the full-catalog lane. It grants no alternate source or
execution authority.
- An earlier TestOps dispatch failed with an integration HTTP 500 before
GitHub created a run. Its cause remains unknown. This patch addresses
the separate, reproduced metadata compatibility failure; it does not
claim to cause or cure that HTTP 500.
- Existing in-flight workflows keep their original source. This PR does
not patch, rerun or upload into them.

## Model Used

OpenAI `gpt-6.1-sol`, with tool use and code execution. The session does
not expose a context-window size. The independent source peer uses the
same model family.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 14:53:30 -06:00
..