Files
PaperClipAI/tests/e2e
DottaandPaperclip c6fe9b143c test(evals): pin reviewed chat repairs and restore UI test setup (#18921)
## Thinking Path

> - Paperclip agents must describe recorded work and respect human
review decisions.
> - The catalog must test the user's request without adding an
unrequested format rule.
> - Timeline case 466 wrongly required date and timezone tokens in every
valid answer.
> - Missing private evidence and stalled review admission also need
bounded failure facts.
> - The reviewed changes correct that contract and add diagnostics
without changing product behavior.
> - A new retained secure-entry reply also exposed an observer false
positive for an explicitly excluded chat destination.
> - A retained team-install reply also omitted every resource link, so
the reviewed service now returns canonical hrefs.
> - This PR pins the coherent corrections and fixes the observed UI-test
interruption helper.

## Linked Issues or Issue Description

Refs #18914. Refs #18923. Refs #18927. Refs #18780.

This prepares the next diagnostic selection. It does not change the
three-case run already in progress. The prior case-466 failure remains a
failure because its answer omitted the required handoff. The prior
case-53 original grade remains unknown.

## What Changed

- Pin immutable candidate `25cd5e4ebb6b955701778d2d1d702f86a573cfaa`.
- Update the existing descriptor snapshot and matching timeline oracle
declaration to v3. Only case 466's expected outcome is clarified:
respect a timezone when explicitly requested.
- Update case 487's existing source-file declaration to include its
diagnostic helper. Its oracle, request, run counts, and clocks stay
unchanged.
- The candidate adds fixed carrier/reason counts for case-53 evidence
failures. It also records bounded, issue-bound delivery/wake/run facts
only when case 487's existing settled check fails, then rethrows that
original error.
- Correct the shared chat-credential matcher for secure-form
instructions that explicitly exclude chat. Mixed or later chat requests
still fail. Give case 580 an explicit personal-secret v2 oracle identity
and update its existing snapshot and manifest declaration.
- Include the reviewed additive team-install resourceLinks response, its
shared type and actual 201 OpenAPI schema. Preserve the full existing
partial-install/repeat help and synchronize the existing typed Storybook
result. The core PR remains unmerged.
- Fix the UI test announcement handler that timed out during the two-tab
attachment check. Only a detached target with no current replacement is
ignored; real remaining-target errors still fail. The original
functional assertions and deadlines remain. This changes no catalog case
550 behavior.
- Keep all 244 case identities, original requests, model, deadlines,
private scanner limits, and broader runtime qualification holds
unchanged.

## Verification

- All 47 existing TestOps contract, adapter, and manifest controls
passed on this pin.
- Integrated timeline and private-diagnostic controls passed: 162
timeline checks, 14 evidence checks, and full 242-declaration admission
with zero fixture callbacks.
- The source author ran 53 real PostgreSQL/review controls and the
declaration-backed runner typecheck on the exact accepted review
diagnostic. Those passed.
- The integrated replay passed 179 tests. Its three PostgreSQL observer
cases initially could not start because the local native package lacked
a shared library. After the existing same-version runtime was resolved
through owned dependency links, those three real PostgreSQL cases
passed. The setup failures are retained; no source, test, timeout,
scanner, or assertion was weakened.
- The retained secure-entry consumer replay accepts the truthful
secure-form/not-chat handoff with the correction. All 25 shared consumer
checks and the fresh full 242-declaration admission passed (28 tests
total). The earlier failed result is unchanged. This does not yet prove
human entry or worker delivery.
- A cheap prefreeze check on actual master passed the existing
reporting-lock guard with unchanged reviewed lock bytes.
- The resource-link type changes exactly one of 506 SDK inputs. The
existing owning helper freshly compiled shared and plugin SDK outputs.
Measured SDK runtime JavaScript is unchanged; the type declaration, its
map and two build receipts changed. The emitted resource-link consumer
compiles and a foreign resource kind fails with TS2322. The old
registry/default proof rejected the change; the existing generator
updated only that input and the new default proof passed. All 51 SDK and
20 input/build custody controls passed. Compiler, parser, Zod and canary
certificate pins are unchanged; held runtime cases remain held.
- Core service/wire/OpenAPI controls passed (72 tests), plus four real
PostgreSQL checks. The typed story synchronization passed UI typecheck,
catalog checks and 30 controls. Candidate token gates passed.
- The prior CI two-tab check failed in the announcement locator handler.
A controlled real-browser disappearance reproduces it before the fix and
passes after it; a genuine remaining-target error still propagates. That
prior failed check is retained. Fresh normal CI and review run on this
actual correction.
- The review-requested real detached-ElementHandle control now covers
the catch branch and asserts the target is absent. Removing that
predicate fails with the real browser detachment error; the restored
three controls pass. This is test coverage only; helper behavior and
deadlines are unchanged.
- Fresh exact-head normal [PR
CI](https://github.com/paperclipai/paperclip/actions/runs/41774814211)
and [Windows
Runner](https://github.com/paperclipai/paperclip/actions/runs/41774814064)
passed. Greptile and the completed Paperclip review both report 5/5 with
no actionable feedback; all review threads are resolved. The optional
security scan is neutral, not a security assessment PASS. No new catalog
runtime PASS is claimed for this candidate.

## Risks

The timeline contract correction accepts valid answers without extra
date labels. It still rejects explicit dates outside the requested
window, missing attribution, failed-run misstatements, hidden work, and
unsupported claims. Balanced owning controls retain those negatives.

The diagnostics add evidence rather than changing delivery behavior.
Case 53 can still fail closed at the existing archive limit. Case 487
can still fail its original settled check. The additive resource links
are a product affordance and still need a real agent rerun. Earlier
failed grades and attempts remain unchanged. The full 244 runtime
catalog is not merged into master; its existing 196/244 gate remains.

## Model Used

OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis,
code integration, and local validation. The host does not expose the
context-window limit. The model under evaluation remains
OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 22:56:20 -06:00
..