Files
PaperClipAI/tests
DottaandPaperclip e5e22148e4 test(evals): pin the reviewed chat catalog fixture repairs (#18710)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat catalog uses a trusted TestOps workflow and an immutable
candidate source.
> - The retained first pass found fixture failures before real feature
work could start.
> - Candidate repairs must use the same source checks as the original
lane.
> - This pull request advances only the fixed catalog target and its
source note.
> - New AWS measurements keep the original case queries, limits, graders
and reporting path.

## Linked Issues or Issue Description

Refs: #18542, #18647, #18680, #18693.

**What happened?**

The full-catalog lane remains pinned to the original candidate. A repair
candidate cannot run through the trusted TestOps selection without a
reviewed pin change.

**Expected behavior**

TestOps must select the reviewed immutable repair source. It must keep
the existing case identities, model, clocks, private evidence controls
and original grading.

**Steps to reproduce**

Inspect the full-catalog target in the manifest. The trusted contract
uses that exact commit for selection, source checks, build and export.

## What Changed

- Advance the immutable full-catalog target to
[`9ab0f951a5b5990c266ef4456610e0a0775aa334`](https://github.com/paperclipai/paperclip/commit/9ab0f951a5b5990c266ef4456610e0a0775aa334).
[Candidate-only
delta](https://github.com/paperclipai/paperclip/compare/3b66c3c144360ddf32714a0742fd774b8ceafb16...9ab0f951a5b5990c266ef4456610e0a0775aa334).
- Preserve all original queries and limits. Record company observer
source ownership and the secret-entry v2 oracle as explicit metadata
overrides against the original definition source.
- Explain SDK staging, scoped human membership, original-clock
observation and exact controller-secret baseline conditions. Preserve
the original failed measurements.
- Leave the application and held core fixes off master. This change does
not copy the candidate app or catalog implementation to master.

## Verification

- Candidate integration: 167 focused pure tests passed for SDK custody,
membership, feature and privacy controls.
- Trusted target consumers: 46 manifest, full-catalog adapter and
dispatch contract tests passed.
- The candidate has no diff to the original YAML queries or descriptor
bytes.
- `git diff --check` passed.
- Local full typecheck, application build, DB and model execution were
not run for this two-file pin change. Worker typecheck attempts hit
existing missing workspace dependencies and are not recorded as passes.
The trusted AWS source check and build must pass before any native
query. It imports the real candidate catalog and checks every descriptor
before provider access. It does not typecheck all harness TypeScript
files.
- Initial CI41563164012 passed48 jobs with1 skip. A later review
requested the exact target assertion; the valid-SHA mutant is now
rejected. Corrected-head CI41564795118 passed48 jobs with1 skip
on2226f093. Both required checks passed. Greptile and the follow-up
source review are5/5 with no unresolved threads. No new candidate case
result is claimed here.

## Risks

A new fixture condition can change a measured outcome. Each result
records the actual candidate and trusted workflow revision. Original
failed measurements remain unchanged. The model, case bounds,
independent feature requirements, private custody, reporting identities
and automatic retry count remain fixed. This pin applies to future
full-catalog runs, so operators must not infer that every catalog case
has been qualified. The inherited source-ref label is not source
authority; the exact checked commit is authoritative.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, source editing and
tool use. The exact context window is not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable; the existing
target consumers cover this data-only change
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 20:28:06 -06:00
..