Files
PaperClipAI/tests
DottaandPaperclip b918ee165d test(evals): pin coherent chat feature follow-ups (#19411)
## Thinking Path

> - Paperclip manages agent work and its permission boundaries.
> - The chat catalog runs feature checks from an immutable source.
> - The last full-ten attempt had five reported passes, four failures
and one missing original.
> - Local evidence identified narrow assertion, setup and
evidence-retention repairs.
> - This PR updates the trusted source pin and its exact descriptor
snapshot.
> - The next five-case measurement can test the affected features under
the same model, clocks and native cap.

## Linked Issues or Issue Description

Refs: #19395, #19405, #19093, #19084, #19112.

The previous source remains preserved. This PR selects reviewed source
`f5692c57dafaf9afbef479c25fcd277815932516` on
`codex/chat-full-ten-coherent-qualified-f5692c57-20261106`, tree
`b9e302d2ea7b605a60125dec8c81672eac7a3c57`. The full catalog and all
core PRs remain unmerged. Core #19405 is included only as a source
overlay.

## What Changed

- Update the source target, exact descriptor snapshot and the matching
pin checks. Preserve historical conditions, frozen source anchors and
the prior query clarification.
- Record Company ACK handling and Company/Personal negation corrections,
plus the skill-update preview negation correction. A denied or unapplied
action still fails when the agent claims it completed.
- Disclose the Team setup change: a disposable catalog package selects
the approved OpenCode runtime and same-company account. Other file bytes
and non-runtime sidecar fields stay unchanged. Original and effective
package identities and hashes are recorded. A real PostgreSQL install
without caller overrides proves the saved runtime. This gives no
end-to-end runtime-migration or override-handling credit.
- Include defensive failed-turn evidence retirement and generic,
schema-valid human-input examples. The old case 53 missing-original
ENOENT cause is not diagnosed. The examples are a usability hypothesis,
not a schema repair or native-success claim.

All 244 YAML definition fields (query, assumed context and expected
outcome) remain byte-identical versus source 936c. The Team derived
runtime fixture and task setup context change is disclosed above.
Native/human bounds, clocks, DeepSeek 1017, credentials, SDK506/efa,
launcher, private interfaces, native cap two and zero automatic retries
remain unchanged. Only descriptor `sourceFiles` and oracle versions
change. The 196/244 gate stays unchanged.

## Verification

Pin-local checks:

```sh
node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs tests/runner-e2e/chat-completion-testops-contract.test.mjs
```

- All 49 pin/admission controls pass locally. They join all 244
identities, historical query overrides, native bounds and the exact
242-row descriptor snapshot.
- Source qualification: 447 controls across 32 suites, full E2E
TypeScript checks and normal contract generators pass. The separate 21
core contract controls are carried-forward component evidence; they are
not added to 447. Local source checks used Node 26.4.0. AWS runtime
settings are unchanged.
- The selected source descriptor SHA256 is
`24fe88af167724c6fb120d3984296aab47c993a65782158e03d80a936fbd94ed`.
- The actual source launcher is byte-identical to 936c: SHA256
`2b9a6953ee51f0ea0dbbd475b0491d9a6c39a1d6835eeaa912e0dbb7cf987724`, Git
blob `c88df15f509b2ac8638af826647a97d45f16bd1f`.
- All 506 registered SDK inputs and the ten trusted helper bytes are
unchanged. Five required source-helper joins pass. No SDK renewal or
native build is part of this PR.
- The current controller's 67 dependency inputs match the qualified
frozen reporting graph. No reporting-lock, package, permission or
workflow change is included.
- Exact head `b833036fb477b55b05cbdf1f4feb2e2c5387b420` is green: CI run
42166216108 attempt 2 succeeded, Windows succeeded, and both current
reviews are 5/5 with zero unresolved threads. The check rollup has 56
successes, three configured skips and one neutral result. New
representative cases 53, 578, 580, 585 and 610 are unrun. A later
uniform-ten regression is separate.
- CI attempt 1 remains retained: one cloud-auth browser fixture read a
trailing health response after context teardown began, after all
task-recovery assertions had passed. One authorized rerun of only that
failed job and its dependent checks succeeded on the unchanged head.
This does not fix or qualify fixture reliability; no eval or model run
was retried.

The previous full-ten outcome remains five reported passes, four
failures and one missing-original/reporting gap. Case 585 reached only
its first preview before the false rejection; its update phase is
unqualified. Case 610's old noncompletion and runtime-override journey
remain unresolved. All original failures and evidence remain retained.
No old result is regraded or replaced.

## Risks

The local repairs may not produce a native pass. The new Team setup
tests installation with an already selected runtime package; it does not
qualify runtime migration. Failed-turn retirement has local regression
coverage but does not establish the cause of the old case 53 reporting
gap. The core guidance overlay remains unmerged and its native benefit
is unproved. Normal protected merge and a new authenticated measurement
are still required.

## Model Used

OpenAI `gpt-6.1-sol` through Codex. It used code inspection, local test
execution and GitHub tooling. The context window size is not exposed. No
model inference or provider call was made for this PR's local checks.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 23:32:34 -06:00
..