Files
PaperClipAI/tests/runner-e2e/fixtures
DottaandPaperclip 32e5eada06 test(evals): pin reviewed attribution and secure question repairs (#18974)
## Thinking Path

> - Paperclip must complete work through ordinary human and agent
controls.
> - The remote chat campaign exposed observer and setup defects as well
as product discovery gaps.
> - Case 466 rejected a valid handoff that used headings, pronouns and
inline HTML.
> - Case 487 seeded three daily runs although its declared journey needs
six.
> - Secure-entry replies used a real bound question card that the
observer did not support.
> - Team installation omitted its existing strict request schemas from
the API search result.
> - This PR pins the reviewed coherent repairs and keeps their metadata
in sync.

## Linked Issues or Issue Description

Refs #18921. Refs #18964. Refs #18967. Refs #18968. Refs #18970. Refs
#18971. Refs #18972.

The previous launch remains unchanged. Case 53 passed its latest native
attempt. Cases 466, 487, 580 and 610 still retain their prior failures.
The next representative run must test the repaired conditions. The
company-secret case follows only after the personal-secret journey
proves real secure entry and worker use. Its separate case-local
follow-up is held for a later pin; case 578 is excluded from the next
four-case selection and has no pass credit.

## What Changed

- Pin immutable candidate `e757e447371c4781a3eff3a0ea9dc9c4eceb060a` and
its existing descriptor snapshot.
- Record the explicit oracle versions and source-file changes for
timeline attribution, review fixture allowance, and bound
personal/company secret questions.
- Keep all 244 case identities, human requests, original run and action
counts, time allowances, provider/model, private evidence limits and
broader qualification holds unchanged.
- Preserve the actual request-body schema affordance for team preview
and installation, existing response links and prior help.

## Verification

- The final integrated candidate passed 442 owning checks across 18
suites. All 242 declarations loaded with zero fixture callbacks; the
existing manifest joins those declarations and two legacy tasks into 244
cases.
- The team service fixture passed 29 tests. The frozen candidate API
consumer passed 2045 tests after the three-line schema exposure change.
These are component controls, not a new native pass.
- The stale inventory expectation reproduced one failure with 16 passes.
Its unchanged derivation traces five previously reviewed YAML
corrections. Updating only the expected constant passed 17 controls, and
the existing definition-drift and catalog-tamper negatives still passed.
No scenario YAML changed in this candidate.
- The existing SDK guard confirms all 506 inputs are unchanged and the
reviewed registry remains
`efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. No
compiler build or registry renewal was needed.
- The existing pin, TestOps contract, adapter, reconciliation and
streaming controls passed locally. Current master also passed the
existing reporting-lock guard with its actual current lock.
- The final source includes the reviewed
immediate-actor/consecutive-pronoun controls and checks every rendered
question title, header and submit label for unsafe instructions. The
remaining fixed scanner reasons are diagnostic only. The combined source
passed 442 checks; all 131 pin/contract/API/streaming controls passed.
- Fresh exact-head [normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41847423336)
passed: 48 successful jobs and one selector skip, including both
required aggregates. Greptile reports 5/5 for this head, with no
actionable findings or unresolved review threads. The current optional
Paperclip review is queued and is not credited as an assessment pass;
its prior-head 5/5 remains historical. The finite source peer review
found no material issue and did not claim native execution. The prior CI
attempt is retained. No new native pass is claimed by this PR.

## Risks

The new source repairs observation and setup, but it does not prove the
next native outcome. The secure-entry accounting records one real chat
request and one authenticated UI answer as the two declared human
actions; it keeps the raw comment count separate. The question path
requires the bound pending card, actual resolver audit and delivery
facts. Ordinary cases retain their existing accounting. Private scans
and archive limits can still reject evidence. Prior failed and unknown
attempts remain immutable.

## Model Used

OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis,
integration and local checks. The host does not expose the
context-window limit. The evaluated model stays OpenCode/OpenRouter
`deepseek/deepseek-v4.1-flash-1017`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 08:40:01 -06:00
..