Files
PaperClipAI/tests
DottaandPaperclip 088e959d7b test(evals): pin qualified chat scenario follow-up repairs (#19155)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - The protected TestOps workflow admits one fixed catalog source.
> - The retained uniform attempt exposed scenario observation and
grading gaps.
> - Focused source PRs now contain the reviewed repairs and local
controls.
> - A new immutable candidate combines those changes without a runtime
refresh.
> - This PR binds that qualified candidate for a future normal TestOps
measurement.

## Linked Issues or Issue Description

Refs: #19135, #19148, #19086, #19084, #19093, #19095, #19100, #19112.

The previous uniform attempt finished **3 PASS / 7 FAIL**. Its original
results and the separate failed control result4587 remain retained. The
source repairs correct visible multipart summary evidence, canonical
skill navigation, bound approval history, timeline attribution,
credential guidance, company configuration questions, and team-install
observation. Local qualification is component evidence. It does not
establish a new passing uniform-ten attempt.

## What Changed

- Pin immutable source `2ec214914cdaa11148ab2527d2203303f1485e2f`, tree
`f1dc0c47668e1ed1db1b4d1ead4d3d262edefc0a`.
- Update eleven existing oracle-version declarations in the manifest and
its frozen descriptor snapshot.
- Keep all 244 identities, 242 canonical declarations, requests, setup,
human/native counts, clocks, model, profile, retries, permissions, and
the 196/244 readiness gate.
- Preserve historical manifest provenance and prior failure records.
- Keep the launcher and all 75 private protocol files byte-identical to
the prior source. This PR changes no controller helper or workflow.

## Verification

- Existing focused pin, selection, admission, streaming, API, and
reconciliation tests: **131 passed**.
- Source owner qualification on the exact published candidate: **675
Node controls and 20 hire DOM controls passed**; all six affected
domains are included. The 242-declaration gate passed.
- Fresh server declarations and full E2E typecheck passed on the exact
candidate under local Node **26.4.0**. The temporary Node24/output tree
disappeared; the source owner restored exact frozen dependencies and
rebuilt only missing identical-source outputs under Root authorization.
Actual AWS Node and the model profile remain unchanged.
- Existing SDK guard: **506 registered inputs unchanged**, registry
`efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`; no
SDK renewal or runtime build.
- Production server, packages, CLI, UI, YAML, and lock bytes equal the
prior source. Only eleven oracle-version descriptor leaves change.
Query, clock, count, and policy comparisons passed.
- Launcher SHA256 stays
`2b9a6953ee51f0ea0dbbd475b0491d9a6c39a1d6835eeaa912e0dbb7cf987724`.
Descriptor SHA256 is
`41038d984f045f593923da53ac036a9a6adefed5edcaedc609e55991e2fa03b7`.
- The next controlled acceptance selection is **only [53, 466, 564,
580]**. Their current source reviews and manual contract reviews are
clear. Cases 74, 578, and 610 have separate open observation findings
and remain held. This pin does not qualify the whole catalog or a
uniform-ten result.
- Root approved the final three-SHA pin delta. Exact current-head pin
review is 5/5 with no unresolved threads, and Windows passed. Normal CI
is terminal and blocked: 42 jobs passed, one expected selector skipped,
and six failed. Both direct-task Cloud recovery variants failed the
existing initial heading check. The native Runner, package Runner, and
server1 test leaves lost communication with their self-hosted runners,
as the GitHub annotations confirm. Their host cause is unproven; Spot is
not established. The dependent E2E and verify aggregates also failed.
Original CI results are retained. One same-head failed-jobs-only
recovery was accepted after a fresh terminal/head guard. The independent
unchanged direct-task control passed locally in 4.9s; the CI trace shows
its five-second heading wait began before the Cloud roundtrip, with the
expected page visible in the failure screenshot shortly afterward. This
supports a test-boundary hypothesis, not a fixed race or reliability
claim. Attempt 2 is pending; no second unchanged retry is authorized.
Merge remains blocked until all required gates pass. Individual source
PRs retain their separate CI and base-policy limitations; this PR does
not claim that all source PRs are merge-ready.
- No model call, TestOps submission, archive read, permission change,
core merge, or catalog merge occurred during this pin work.

## Risks

The candidate contains unmerged catalog changes. The protected source
pin keeps that boundary explicit. Component controls cannot replace a
real product attempt. Prior failures and unknown historical grades
remain unchanged. A future measurement must use the admitted source and
original bounds.

## Model Used

OpenAI GPT-6.1 (`gpt-6.1-sol`), with reasoning, code execution, and
repository tools. The host did not expose a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched for and linked related public PRs above
- [x] I have linked related public PRs and described the issue
- [x] I have not referenced internal or instance-local Paperclip links
- [x] My branch name describes the change
- [x] I have run the relevant pin tests locally and they pass
- [x] I have updated the source pin regression and descriptor snapshot
- [x] I have updated the owning source provenance conditions
- [x] I have considered and documented the risks
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 22:32:45 -06:00
..